Question

我正在使用hadoop distcp -update将目录从一个HDFS集群复制到另一个HDFS集群。有时（经常）我会遇到这种异常：

13/07/03 00:20:03 INFO tools.DistCp: srcPaths=[hdfs://HDFS1:51175/directory_X]
13/07/03 00:20:03 INFO tools.DistCp: destPath=hdfs://HDFS2:51175/directory_X
13/07/03 00:25:27 WARN hdfs.DFSClient: src=directory_X, datanodes[0].getName()=***.***.***.***:8550
java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/***.***.***.***:35872 remote=/***.***.***.***:8550]
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:116)
        at java.io.DataInputStream.readShort(DataInputStream.java:295)
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:885)
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:822)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:541)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:53)
        at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1230)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1110)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
13/07/03 00:26:40 INFO tools.DistCp: sourcePathsCount=8542
13/07/03 00:26:40 INFO tools.DistCp: filesToCopyCount=0
13/07/03 00:26:40 INFO tools.DistCp: bytesToCopyCount=0.0

有谁知道它会是什么？使用Hadoop 0.20.205.0

Answer 1

建议增加dfs.socket.timeout的超时，以便读取超时。和dfs.datanode.socket.write.timeout，用于写入超时。

默认值：

// Timeouts for communicating with DataNode for streaming writes/reads
public static int READ_TIMEOUT = 60 * 1000; // here, 69000 millis > 60000
public static int WRITE_TIMEOUT = 8 * 60 * 1000;

在hadoop-site.xml或hdfs-site.xml

中添加以下内容

<property>
  <name>dfs.datanode.socket.write.timeout</name>
  <value>3000000</value>
</property>

<property>
  <name>dfs.socket.timeout</name>
  <value>3000000</value>
</property>

希望有所帮助。

Answer 2

我想你也想设置dfs.client.socket-timeout

Answer 3

Here就是原因。

不推荐使用的属性名称 - ＆gt;新属性名称

dfs.socket.timeout - ＆gt; dfs.client.socket-timeout

在集群之间运行hadoop distcp -update时出现SocketTimeoutException

3 个答案:

不推荐使用的属性名称 - ＆gt;新属性名称