spark-ec2和Tachyon hadoop版本悬殊

时间:2015-02-11 20:27:31

标签: apache-spark hadoop amazon-ec2 alluxio spark-ec2

我尝试使用spark-ec2来启动带有hadoop 2.x版的ec2集群,所以我试过了:

./spark-ec2 -k spark -i ~/.ssh/spark.pem -s 1 --hadoop-major-version=2 launch my-spark-cluster

然后我发现tachyon设置过程中有错误:

Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
ec2-52-1-147-16.compute-1.amazonaws.com
ec2-52-1-147-16.compute-1.amazonaws.com: Formatting Tachyon Worker @ ip-172-31-21-86.ec2.internal
ec2-52-1-147-16.compute-1.amazonaws.com: Removing local data under folder: /mnt/ramdisk/tachyonworker/
Formatting Tachyon Master @ ec2-52-1-14-186.compute-1.amazonaws.com
Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
    at tachyon.util.CommonUtils.runtimeException(CommonUtils.java:246)
    at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:73)
    at tachyon.UnderFileSystemHdfs.getClient(UnderFileSystemHdfs.java:53)
    at tachyon.UnderFileSystem.get(UnderFileSystem.java:53)
    at tachyon.Format.main(Format.java:54)
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
    at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:69)
    ... 3 more

我搜索了一些相关的问题,似乎Server IPC version 7 cannot communicate with client version 4表示服务器正在使用hadoop 2.x而客户端正在使用hadoop 1.x.但是,我用hadoop 2.4.0构建了我的火花,我也尝试了带有hadoop 2.4.0及更高版本的官方火花预制版本,两者都导致同样的错误。

顺便说一下,通过设置--hadoop-major-version=2创建的hadoop版本为Hadoop 2.0.0-cdh4.2.0。这是一个问题吗?但是我试着在这里使用2.4或2.4.0,它们都没有被认为是有效的hadoop版本

0 个答案:

没有答案