请帮助理解在Hadoop上运行的不成功的Scalding的输出。
我从git获得了最新的Scalding发行版: git clone https://github.com/twitter/scalding.git
从sbt assembly
目录scalding
之后,我尝试使用命令运行教程:
scripts/scald.rb --hdfs tutorial/Tutorial0.scala
结果我收到了以下错误:
scripts/scald.rb:194: warning: already initialized constant SCALA_LIB_DIR
rsyncing 19.8M from scalding-core-assembly-0.10.0.jar to my.host.here in background...
downloading hadoop-core-1.1.2.jar from http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar...
ssh: Could not resolve hostname my.host.here: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Successfully downloaded hadoop-core-1.1.2.jar!
downloading commons-codec-1.8.jar from http://repo1.maven.org/maven2/commons-codec/commons-codec/1.8/commons-codec-1.8.jar...
Successfully downloaded commons-codec-1.8.jar!
downloading commons-configuration-1.9.jar from http://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.9/commons-configuration-1.9.jar...
Successfully downloaded commons-configuration-1.9.jar!
downloading jackson-asl-0.9.5.jar from http://repo1.maven.org/maven2/org/codehaus/jackson/jackson-asl/0.9.5/jackson-asl-0.9.5.jar...
Successfully downloaded jackson-asl-0.9.5.jar!
downloading jackson-mapper-asl-1.9.13.jar from http://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar...
Successfully downloaded jackson-mapper-asl-1.9.13.jar!
downloading commons-lang-2.6.jar from http://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar...
Successfully downloaded commons-lang-2.6.jar!
downloading slf4j-log4j12-1.6.6.jar from http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.6/slf4j-log4j12-1.6.6.jar...
Successfully downloaded slf4j-log4j12-1.6.6.jar!
downloading log4j-1.2.15.jar from http://repo1.maven.org/maven2/log4j/log4j/1.2.15/log4j-1.2.15.jar...
Successfully downloaded log4j-1.2.15.jar!
downloading commons-httpclient-3.1.jar from http://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar...
Successfully downloaded commons-httpclient-3.1.jar!
downloading commons-cli-1.2.jar from http://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar...
Successfully downloaded commons-cli-1.2.jar!
downloading commons-logging-1.1.1.jar from http://repo1.maven.org/maven2/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar...
Successfully downloaded commons-logging-1.1.1.jar!
downloading zookeeper-3.3.4.jar from http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.3.4/zookeeper-3.3.4.jar...
Successfully downloaded zookeeper-3.3.4.jar!
compiling tutorial/Tutorial0.scala
scalac -classpath /tmp/temp_scala_home_2.9.3_654763/scala-library-2.9.3.jar:/tmp/temp_scala_home_2.9.3_654763/scala-compiler-2.9.3.jar:/home/test/Cascading/scalding/scalding-core/target/scala-2.9.3/scalding-core-assembly-0.10.0.jar:/tmp/maven/hadoop-core-1.1.2.jar:/tmp/maven/commons-codec-1.8.jar:/tmp/maven/commons-configuration-1.9.jar:/tmp/maven/jackson-asl-0.9.5.jar:/tmp/maven/jackson-mapper-asl-1.9.13.jar:/tmp/maven/commons-lang-2.6.jar:/tmp/maven/slf4j-log4j12-1.6.6.jar:/tmp/maven/log4j-1.2.15.jar:/tmp/maven/commons-httpclient-3.1.jar:/tmp/maven/commons-cli-1.2.jar:/tmp/maven/commons-logging-1.1.1.jar:/tmp/maven/zookeeper-3.3.4.jar -d /tmp/script-build tutorial/Tutorial0.scala
ssh: Could not resolve hostname my.host.here: Name or service not known
rsyncing 1.5K from job-jars/Tutorial0.jar to my.host.here in background...
Waiting for 2 background threads...
ssh: Could not resolve hostname my.host.here: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]
Could not rsync: /home/test/Cascading/scalding/scalding-core/target/scala-2.9.3/scalding-core-assembly-0.10.0.jar to my.host.here:scalding-core-assembly-0.10.0.jar
Could not rsync: /tmp/Tutorial0.jar to my.host.here:job-jars/Tutorial0.jar
*更新*
在scald.rb
中更改主机后,我收到了以下身份验证问题:
$ scripts/scald.rb --hdfs tutorial/Tutorial0.scala
scripts/scald.rb:194: warning: already initialized constant SCALA_LIB_DIR
rsyncing 19.8M from scalding-core-assembly-0.10.0.jar to node7.test.net in background...
The authenticity of host 'node7.test.net (10.1.21.32)' can't be established.
RSA key fingerprint is fa:41:31:ab:b0:46:08:8f:2b:75:0a:18:24:f9:d5:ec.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'node7.test.net (10.1.21.32)' can't be established.
RSA密钥指纹是fa:41:31:ab:b0:46:08:8f:2b:75:0a:18:24:f9:d5:ec。 您确定要继续连接(是/否)吗?是 警告:永久性地将“node7.test.net”(RSA)添加到已知主机列表中。 test@node7.test.net的密码:请输入“是”或“否”: 许可被拒绝,请再试一次。 test@node7.test.net的密码:
我输入了正确的路径,但身份验证错误仍然存在。我该如何配置rsync
?
答案 0 :(得分:1)
你确实改变了这个 https://github.com/twitter/scalding/blob/develop/scripts/scald.rb#l27 正确?
默认主机是:my.host.here。