Question

这是我的错误消息。

~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal README.txt /input/wex <br/>
~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal RandomString.txt /input/wex<br/>
~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal 2M.TITLE.ID /input/wex<br/>
~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal 2M.SRCID.DSTID /input/wex<br/>
~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal 10K.ID.CONTENTS /input/wex<br/>
~/hadoop-1.0.3/data$ hadoop fs -copyFromLocal 2M.ID.CONTENTS /input/wex<br/>

14/11/30 17:57:46 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/wex/2M.ID.CONTENTS could only be replicated to 0 nodes, instead of 1
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
        at org.apache.hadoop.ipc.Client.call(Client.java:1070)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)

正如您所看到的，我只在Big File中失败了。（2M.ID.CONTENTS）

ls -al
total 2710312<br\>
drwxr-xr-x  2 ubuntu ubuntu       4096 Jan  4  2013 .<br/>
drwxr-xr-x 17 ubuntu ubuntu       4096 Nov 30 16:42 ..<br/>
-rw-r--r--  1 ubuntu ubuntu    9735683 Sep 24  2012 10K.ID.CONTENT<br/>
-rw-r--r--  1 ubuntu ubuntu 2506926541 Sep 24  2012 2M.ID.CONTENTS<br/>
-rwxrwxr-x  1 ubuntu ubuntu  151641370 Aug  8  2012 2M.SRCID.DSTID<br/>
-rw-r--r--  1 ubuntu ubuntu   73550219 Jul 25  2012 2M.TITLE.ID<br/>
-rw-r--r--  1 ubuntu ubuntu   33481780 Aug  9  2012 RandomString.txt<br/>
-rw-r--r--  1 ubuntu ubuntu       1366 Sep 23  2012 README.txt<br/>

我尝试了以下内容：

1）我检查了hadoop.tmp.dir目录权限 2）我多次重启所有hadoop进程（包括重新格式化namenode） 3）我将hadoop.tmp.dir从/tmp/my-user-name/dfs/data更改为/home/my-user-name/data/hdfstmp。

为什么我不能将大文件上传到HDFS？

Answer 1

您的hadoop群集似乎正在运行;但它没有看到数据节点。您可以通过
检查datanode是否正在运行 1.运行命令jps你的终端。您必须看到datanode和namenode活动
2.如果您在ubuntu计算机（而非EC2）上安装了hadoop，请转到浏览器并尝试使用此地址http://localhost:50070/dfshealth.jsp，您必须看到live node =1和not zero。如果为零，则表示存在安装问题。

Hadoop：File ...只能复制到0个节点，而不是1个

1 个答案: