运行Hadoop管道时出错程序:“服务器无法验证”

时间:2011-11-25 12:30:40

标签: c++ hadoop pipe

尝试在我的hadoop集群上运行引用此(link)的C ++程序时。我收到了下面提到的错误。

我提到了有关此错误的相关帖子(this),并尝试调整我的Makefile,但仍然无法解决此问题。

(我使用的是fedora c12,我也无法找到要添加的.configure文件。)

我的Makefile看起来像这样:

CC = g++
HADOOP_INSTALL = /home/hadoop/Desktop/Cloudera/hadoop-0.20.2-cdh3u1
SSL_INSTALL = /usr/include/openssl
PLATFORM = Linux-i386-32
CPPFLAGS = -m32 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include -I$(SSL_INSTALL)


wordcount: wordcount.cpp
    $(CC) $(CPPFLAGS) $< -Wall -Wextra -L$(SSL_INSTALL) -lssl -lcrypto -L$(HADOOP_INSTALL)/c++/$(PLATFORM)/lib -lhadooppipes \
     -lhadooputils -lpthread -g -O2 -o $@ 

我无法弄清楚如何解决此错误。

任何帮助将不胜感激。

hadoop@01HW394491 Desktop]$ hadoop pipes -D hadoop.pipes.java.recordreader=true  -D hadoop.pipes.java.recordwriter=true -input /user/hadoop/dtest  -output /user/hadoop/dipeshtryWC  -program /user/hadoop/dbin/wordcount
    11/11/25 15:55:07 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
    11/11/25 15:55:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    11/11/25 15:55:07 WARN snappy.LoadSnappy: Snappy native library not loaded
    11/11/25 15:55:07 INFO mapred.FileInputFormat: Total input paths to process : 3
    11/11/25 15:55:07 INFO mapred.JobClient: Running job: job_201111161101_0145
    11/11/25 15:55:08 INFO mapred.JobClient:  map 0% reduce 0%
    11/11/25 15:55:14 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000000_0, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000000_0: Server failed to authenticate. Exiting
    11/11/25 15:55:14 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000002_0, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000002_0: Server failed to authenticate. Exiting
    11/11/25 15:55:15 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000001_0, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000001_0: Server failed to authenticate. Exiting
    11/11/25 15:55:19 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000000_1, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000000_1: Server failed to authenticate. Exiting
    11/11/25 15:55:19 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000002_1, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000002_1: Server failed to authenticate. Exiting
    11/11/25 15:55:19 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000001_1, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000001_1: Server failed to authenticate. Exiting
    11/11/25 15:55:24 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000000_2, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000000_2: Server failed to authenticate. Exiting
    11/11/25 15:55:24 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000002_2, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000002_2: Server failed to authenticate. Exiting
    11/11/25 15:55:25 INFO mapred.JobClient: Task Id : attempt_201111161101_0145_m_000001_2, Status : FAILED
    java.io.IOException
        at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
        at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

    attempt_201111161101_0145_m_000001_2: Server failed to authenticate. Exiting
    11/11/25 15:55:30 INFO mapred.JobClient: Job complete: job_201111161101_0145
    11/11/25 15:55:30 INFO mapred.JobClient: Counters: 8
    11/11/25 15:55:30 INFO mapred.JobClient:   Job Counters 
    11/11/25 15:55:30 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=56297
    11/11/25 15:55:30 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
    11/11/25 15:55:30 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
    11/11/25 15:55:30 INFO mapred.JobClient:     Rack-local map tasks=9
    11/11/25 15:55:30 INFO mapred.JobClient:     Launched map tasks=12
    11/11/25 15:55:30 INFO mapred.JobClient:     Data-local map tasks=3
    11/11/25 15:55:30 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
    11/11/25 15:55:30 INFO mapred.JobClient:     Failed map tasks=1
    11/11/25 15:55:30 INFO mapred.JobClient: Job Failed: NA
    Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1246)
        at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
        at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
        at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
    [hadoop@01HW394491 Desktop]$

0 个答案:

没有答案