Question

使用sqoop 1.3

尝试将hdfs输出导出到mysql表

加载大小超过300MB的未压缩文件时，一切都按预期工作

但是在加载大小为75 MB或79 MB的压缩文件（.gz和.lzo）时，我看到加载到表中的行的两倍。当压缩文件的大小为60MB或更小（猜测与64 MB相关的内容，块大小）时，不会发生这种情况。我在上面提到的一些操作：

bash-3.2$ ls -ltr
-rw-r--r-- 1 bhargavn bhargavn 354844413 Nov 16 02:27 large_file
-rw-rw-r-- 1 bhargavn bhargavn  15669507 Nov 21 03:41 small_file.lzo
-rw-rw-r-- 1 bhargavn bhargavn  75173037 Nov 21 03:46 large_file.lzo

bash-3.2$ wc -l large_file
247060 large_file

bash-3.2$ sqoop export --connect 'jdbc:mysql://db.com/test?zeroDateTimeBehavior=round&    rewriteBatchedStatements=true' 
--table table_with_large_data 
--username sqoopuser 
--password sqoop 
--export-dir /user/bhargavn/workspace/data/sqoop-test/large_file.lzo 
--fields-terminated-by '\001' -m 1
[21/11/2012:05:52:28 PST] main      INFO org.apache.hadoop.mapred.JobClient:  map 0% reduce 0%
[21/11/2012:05:57:03 PST] main      INFO com.cloudera.sqoop.mapreduce.ExportJobBase:     Transferred 143.3814 MB in 312.2832 seconds (470.1584 KB/sec)
[21/11/2012:05:57:03 PST] main      INFO com.cloudera.sqoop.mapreduce.ExportJobBase:     Exported 494120 records.

mysql> select count(1) from table_with_large_data;
+----------+
| count(1) |
+----------+
|   494120 |
+----------+

mysql> truncate table_with_large_data;
bash-3.2$ sqoop export --connect 'jdbc:mysql://db.com/test?zeroDateTimeBehavior=round&    rewriteBatchedStatements=true' 
--table table_with_large_data 
--uername sqoopuser 
--password sqoop 
--export-dir /user/bhargavn/workspace/data/sqoop-test/large_file
--fields-terminated-by '\001' 
-m 1
[21/11/2012:06:05:35 PST] main      INFO org.apache.hadoop.mapred.JobClient:  map 0%     reduce 0%
[21/11/2012:06:08:06 PST] main      INFO org.apache.hadoop.mapred.JobClient:  map 100%     reduce 0%
[21/11/2012:06:08:06 PST] main      INFO com.cloudera.sqoop.mapreduce.ExportJobBase:     Transferred 338.4573 MB in 162.5891 seconds (2.0817 MB/sec)
[21/11/2012:06:08:06 PST] main      INFO com.cloudera.sqoop.mapreduce.ExportJobBase:     Exported 247060 records.
mysql> select count(1) from table_with_large_data;
+----------+
| count(1) |
+----------+
|   247060 |
+----------+

Answer 1

你可能会遇到Sqoop需要修复的已知错误[1]。

你介意签到Sqoop用户邮件列表[2]并在那里描述你的问题吗？我非常有信心Sqoop的开发人员会参与解决这一特定问题。

Jarcec

链接：

1：https://issues.apache.org/jira/browse/SQOOP-721

2：http://sqoop.apache.org/mail-lists.html

Sqoop导出到RDBMS .lzo .gz文件超过64 MB加载重复项

1 个答案: