只能复制到0个节点而不是minReplication(= 1)。有2个数据节点在运行,并且在此操作中不排除任何节点

时间:2017-04-13 07:39:30

标签: hadoop hive sqoop bigdata

我做了这个错误" sqoop import ..."蜂巢。

namenode log
java.io.IOException: File /input/xxxx/_temporary/1/_temporary/attempt_1492073551248_0012_m_000002_1/part-m-00002 could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.
datanode logs
slave1 :2017-04-13 19:58:59,444 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.149.141:50010, dest: /192.168.149.141:42764, bytes: 451, op: HDFS_READ, cliID: DFSClient_attempt_1492073551248_0012_m_000001_2_785964301_1, offset: 0, srvID: f274418e-04b6-4109-9521-e3c384c21ad0, blockid: BP-219683118-192.168.149.138-1491539013447:blk_1073742751_1927, duration: 160511 

datanode logs
slave2: 2017-04-13 19:58:02,389 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.149.141:34576, dest: /192.168.149.142:50010, bytes: 127362723, op: HDFS_WRITE, cliID: DFSClient_attempt_1492073551248_0012_m_000000_0_-417808976_1, offset: 0, srvID: 7f9110ab-8a1d-4a32-8219-aff6e3cd29b2, blockid: BP-219683118-192.168.149.138-1491539013447:blk_1073742761_1937, duration: 64254909353
2017-04-13 19:58:02,389 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-219683118-192.168.149.138-1491539013447:blk_1073742761_1937, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2017-04-13 19:58:11,269 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.149.141:34588, dest: /192.168.149.142:50010, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_attempt_1492073551248_0012_m_000002_1_-2031862368_1, offset: 0, srvID: 7f9110ab-8a1d-4a32-8219-aff6e3cd29b2, blockid: BP-219683118-192.168.149.138-1491539013447:blk_1073742762_1938, duration: 63824306914
2017-04-13 19:58:11,270 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-219683118-192.168.149.138-1491539013447:blk_1073742762_1938, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2017-04-13 19:58:15,441 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:349ms (threshold=300ms)
2017-04-13 19:58:15,769 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:328ms (threshold=300ms)
2017-04-13 19:58:28,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.149.142:51700, dest: /192.168.149.142:50010, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_attempt_1492073551248_0012_m_000003_1_-395038848_1, offset: 0, srvID: 7f9110ab-8a1d-4a32-8219-aff6e3cd29b2, blockid: BP-219683118-192.168.149.138-1491539013447:blk_1073742763_1939, duration: 52247885321
2017-04-13 19:58:28,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-219683118-192.168.149.138-1491539013447:blk_1073742763_1939, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2017-04-13 19:58:28,689 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-219683118-192.168.149.138-1491539013447:blk_1073742764_1940 src: /192.168.149.142:51718 dest: /192.168.149.142:50010   

有什么想法修复此错误?谢谢!

1 个答案:

答案 0 :(得分:1)

检查https://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo, 我在短时间内执行大量查询时遇到同样的错误,因此增加了datanode上的线程,并通过设置" dfs.datanode.handler.count"来解决问题。出于多种原因抛出此异常,请访问链接以查看您提到的情况。