Hadoop(MapReduce)问题。请勿处理此错误

时间:2019-11-12 00:47:24

标签: hadoop mapreduce hdfs yarn

我正在将Linux用于大数据主题的大学项目。我使用要通过Hadoop分析的Intellij构建了一个jar。该项目包括3个职位。分析完第一份工作后,shell上出现以下错误:

> > 2019-11-12 01:29:18,203 INFO mapreduce.Job: Job job_1573503622064_0002 completed successfully 2019-11-12 01:29:33,594
> INFO mapreduce.Job: Counters: 55
>     File System Counters
>         FILE: Number of bytes read=1116078848
>         FILE: Number of bytes written=2283517013
>         FILE: Number of read operations=0
>         FILE: Number of large read operations=0
>         FILE: Number of write operations=0
>         HDFS: Number of bytes read=27162033259
>         HDFS: Number of bytes written=38
>         HDFS: Number of read operations=686
>         HDFS: Number of large read operations=0
>         HDFS: Number of write operations=2
>         HDFS: Number of bytes read erasure-coded=0
>     Job Counters 
>         Killed map tasks=1
>         Launched map tasks=227
>         Launched reduce tasks=1
>         Data-local map tasks=227
>         Total time spent by all maps in occupied slots (ms)=7571323
>         Total time spent by all reduces in occupied slots (ms)=1386257
>         Total time spent by all map tasks (ms)=7571323
>         Total time spent by all reduce tasks (ms)=1386257
>         Total vcore-milliseconds taken by all map tasks=7571323
>         Total vcore-milliseconds taken by all reduce tasks=1386257
>         Total megabyte-milliseconds taken by all map tasks=7753034752
>         Total megabyte-milliseconds taken by all reduce tasks=1419527168
>     Map-Reduce Framework
>         Map input records=305975157
>         Map output records=111607883
>         Map output bytes=892863064
>         Map output materialized bytes=1116080192
>         Input split bytes=29026
>         Combine input records=0
>         Combine output records=0
>         Reduce input groups=1
>         Reduce shuffle bytes=1116080192
>         Reduce input records=111607883
>         Reduce output records=1
>         Spilled Records=223215766
>         Shuffled Maps =227
>         Failed Shuffles=0
>         Merged Map outputs=227
>         GC time elapsed (ms)=140246
>         CPU time spent (ms)=457710
>         Physical memory (bytes) snapshot=51324133376
>         Virtual memory (bytes) snapshot=576444985344
>         Total committed heap usage (bytes)=38742728704
>         Peak Map Physical memory (bytes)=233918464
>         Peak Map Virtual memory (bytes)=2528382976
>         Peak Reduce Physical memory (bytes)=718159872
>         Peak Reduce Virtual memory (bytes)=2536824832
>     Shuffle Errors
>         BAD_ID=0
>         CONNECTION=0
>         IO_ERROR=0
>         WRONG_LENGTH=0
>         WRONG_MAP=0
>         WRONG_REDUCE=0
>     File Input Format Counters 
>         Bytes Read=27162004233
>     File Output Format Counters 
>         Bytes Written=38 2019-11-12 01:29:35,986 INFO ipc.Client: Retrying connect to server: ubuntu/127.0.1.1:44561. Already tried 0
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:36,988 INFO ipc.Client: Retrying
> connect to server: ubuntu/127.0.1.1:44561. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:37,989 INFO ipc.Client:
> Retrying connect to server: ubuntu/127.0.1.1:44561. Already tried 2
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:40,523 INFO
> mapred.ClientServiceDelegate: Application state is completed.
> FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2019-11-12 01:29:41,858 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:42,860 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:43,863 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:44,866 INFO ipc.Client: Retrying
> connect to server: 0.
> 0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:45,867 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:46,869 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:47,872 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:48,874 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:49,876 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:50,878 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:51,012 INFO
> mapred.ClientServiceDelegate: Application state is completed.
> FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2019-11-12 01:29:52,013 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:53,016 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:54,021 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:55,023 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:56,025 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:57,026 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:29:58,028 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:29:59,030 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:00,031 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:30:01,032 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:03,208 INFO
> mapred.ClientServiceDelegate: Application state is completed.
> FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2019-11-12 01:30:04,214 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:3 0:05,217 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:06,218 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:30:07,219 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:08,221 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:30:09,223 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:10,225 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:30:11,228 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) 2019-11-12 01:30:12,232 INFO ipc.Client:
> Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8
> time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS) 2019-11-12 01:30:13,233 INFO ipc.Client: Retrying
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) Exception in thread "main"
> java.io.IOException: java.net.ConnectException: Your endpoint
> configuration is wrong; For more details see: 
> http://wiki.apache.org/hadoop/UnsetHostnameOrPort
>     at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:345)
>     at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:430)
>     at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:871)
>     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:331)
>     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:328)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>     at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:328)
>     at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:624)
>     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1603)
>     at FirstProject.main(FirstProject.java:33)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:236) Caused by: java.net.ConnectException: Your endpoint configuration is wrong; For
> more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:753)
>     at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
>     at org.apache. hadoop.ipc.Client.call(Client.java:1491)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1388)
>     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
>     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
>     at com.sun.proxy.$Proxy14.getJobReport(Unknown Source)
>     at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
>     at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:326)
>     ... 17 more Caused by: java.net.ConnectException: Connection refused
>     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
>     at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
>     at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
>     at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1435)
>     ... 26 more

Hadoop配置是正确的,因为我分析了同一个jar和其他数据。此外,YARN和DFS也将启动。我删除输出文件夹,然后创建另一个。你能帮我理解吗?

1 个答案:

答案 0 :(得分:0)

能否再次检查Hadoop配置? 这项工作正在尝试连接0.0.0.0/0.0.0.0:10020,这似乎不是正确的模式。应该像IP:Port

也许此作业将覆盖您的默认配置。