Spark Streaming ERROR TransportRequestHandler:为单向消息调用RpcHandler#receive()时出错

时间:2016-12-16 17:56:12

标签: hadoop apache-spark streaming

我正在使用: Spark 2.0.0 斯卡拉2.11.8 Java 1.7 行家

我正在尝试在5台机器的集群中运行Streaming Kmeans。 我不知道是什么导致了这个错误,有人可以帮我吗?

控制台日志下方:

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
starting streaming context
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/12/16 15:49:31 INFO SparkContext: Running Spark version 2.0.0
16/12/16 15:49:31 INFO SecurityManager: Changing view acls to: master
16/12/16 15:49:31 INFO SecurityManager: Changing modify acls to: master
16/12/16 15:49:31 INFO SecurityManager: Changing view acls groups to: 
16/12/16 15:49:31 INFO SecurityManager: Changing modify acls groups to: 
16/12/16 15:49:31 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(master); groups with view permissions: Set(); users  with modify permissions: Set(master); groups with modify permissions: Set()
16/12/16 15:49:31 INFO Utils: Successfully started service 'sparkDriver' on port 42691.
16/12/16 15:49:31 INFO SparkEnv: Registering MapOutputTracker
16/12/16 15:49:31 INFO SparkEnv: Registering BlockManagerMaster
16/12/16 15:49:31 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-bfe445b5-2ee0-4209-9778-6750809a7b53
16/12/16 15:49:32 INFO MemoryStore: MemoryStore started with capacity 1890.6 MB
16/12/16 15:49:32 INFO SparkEnv: Registering OutputCommitCoordinator
16/12/16 15:49:32 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/12/16 15:49:32 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://200.17.78.23:4040
16/12/16 15:49:32 INFO SparkContext: Added JAR /home/master/scala%20projects/testeKmeans/testeKmeans.jar at spark://200.17.78.23:42691/jars/testeKmeans.jar with timestamp 1481910572510
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
16/12/16 15:49:32 INFO TransportClientFactory: Successfully created connection to master/200.17.78.23:7077 after 26 ms (0 ms spent in bootstraps)
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20161216154932-0009
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20161216154932-0009/0 on worker-20161215083739-200.17.78.23-54671 (200.17.78.23:54671) with 6 cores
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20161216154932-0009/0 on hostPort 200.17.78.23:54671 with 6 cores, 1024.0 MB RAM
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20161216154932-0009/1 on worker-20161215083737-200.17.78.26-33946 (200.17.78.26:33946) with 4 cores
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20161216154932-0009/1 on hostPort 200.17.78.26:33946 with 4 cores, 1024.0 MB RAM
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20161216154932-0009/2 on worker-20161215084302-200.17.78.27-57926 (200.17.78.27:57926) with 4 cores
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20161216154932-0009/2 on hostPort 200.17.78.27:57926 with 4 cores, 1024.0 MB RAM
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20161216154932-0009/3 on worker-20161215083727-200.17.78.25-56713 (200.17.78.25:56713) with 4 cores
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20161216154932-0009/3 on hostPort 200.17.78.25:56713 with 4 cores, 1024.0 MB RAM
16/12/16 15:49:32 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48945.
16/12/16 15:49:32 INFO NettyBlockTransferService: Server created on 200.17.78.23:48945
16/12/16 15:49:32 INFO StandaloneAppClient$ClientEndpoint: 
Executor added: app-20161216154932-0009/4 on worker-20161215084213-200.17.78.24-54792 (200.17.78.24:54792) with 4 cores
16/12/16 15:49:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20161216154932-0009/4 on hostPort 200.17.78.24:54792 with 4 cores, 1024.0 MB RAM
16/12/16 15:49:32 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 200.17.78.23, 48945)
16/12/16 15:49:32 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
java.io.InvalidClassException: org.apache.spark.deploy.DeployMessages$ExecutorUpdated; local class incompatible: stream classdesc serialVersionUID = 1654279024112373855, local class serialVersionUID = 3598161183190952796
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:612)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1779)

0 个答案:

没有答案