独立Spark上的JavaWordCount - 未找到类异常

时间:2014-09-01 17:10:33

标签: apache-spark

将WordCount示例从Apache-Hadoop2迁移到Apache-Spark时出现以下错误。不确定需要添加什么额外的类路径,我得到了ClassNotFoundException,如下所示。 JavaWordCount.java是一个内部带有main()的Java类,我在eclipse中将它作为Java Application运行。有没有人有任何想法?非常感谢你!


14/09/01 09:42:19 INFO SecurityManager: Using Spark's default log4j profile:   org/apache/spark/log4j-defaults.properties
14/09/01 09:42:19 INFO SecurityManager: Changing view acls to: edamame
14/09/01 09:42:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(edamame)
14/09/01 09:42:19 INFO Slf4jLogger: Slf4jLogger started
14/09/01 09:42:19 INFO Remoting: Starting remoting
14/09/01 09:42:19 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@10.0.0.19:57558]
14/09/01 09:42:19 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@10.0.0.19:57558]
14/09/01 09:42:19 INFO SparkEnv: Registering MapOutputTracker
14/09/01 09:42:19 INFO SparkEnv: Registering BlockManagerMaster
14/09/01 09:42:19 INFO DiskBlockManager: Created local directory at /var/folders/wv/tqlc_6hj399d0b_bnw68x94h0000gp/T/spark-local-20140901094219-def0
14/09/01 09:42:19 INFO MemoryStore: MemoryStore started with capacity 2.4 GB.
14/09/01 09:42:19 INFO ConnectionManager: Bound socket to port 57559 with id = ConnectionManagerId(10.0.0.19,57559)
14/09/01 09:42:19 INFO BlockManagerMaster: Trying to register BlockManager
14/09/01 09:42:19 INFO BlockManagerInfo: Registering block manager 10.0.0.19:57559 with 2.4 GB RAM
14/09/01 09:42:19 INFO BlockManagerMaster: Registered BlockManager
14/09/01 09:42:19 INFO HttpServer: Starting HTTP Server
14/09/01 09:42:19 INFO HttpBroadcast: Broadcast server started at http://10.0.0.19:57560
14/09/01 09:42:19 INFO HttpFileServer: HTTP File server directory is /var/folders/wv/tqlc_6hj399d0b_bnw68x94h0000gp/T/spark-e4d032b5-d039-488e-b902-2cd970a0b06f
14/09/01 09:42:19 INFO HttpServer: Starting HTTP Server
14/09/01 09:42:19 INFO SparkUI: Started SparkUI at http://10.0.0.19 :4040
14-09-01 09:42:20.108 java[11223:1003] Unable to load realm info from SCDynamicStore
14/09/01 09:42:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/09/01 09:42:20 INFO AppClient$ClientActor: Connecting to master spark://myMachine.local:7077...
14/09/01 09:42:20 INFO MemoryStore: ensureFreeSpace(146579) called with curMem=0, maxMem=2566520832
14/09/01 09:42:20 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 143.1 KB, free 2.4 GB)
14/09/01 09:42:20 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140901094220-0000
14/09/01 09:42:20 INFO AppClient$ClientActor: Executor added: app-20140901094220-0000/0 on worker-20140901094208-10.0.0.19-57554 (10.0.0.19:57554) with 8 cores
14/09/01 09:42:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140901094220-0000/0 on hostPort 10.0.0.19:57554 with 8 cores, 512.0 MB RAM
14/09/01 09:42:20 INFO AppClient$ClientActor: Executor updated: app-20140901094220-0000/0 is now RUNNING
14/09/01 09:42:20 INFO FileInputFormat: Total input paths to process : 4
14/09/01 09:42:20 INFO SparkContext: Starting job: collect at JavaWordCount.java:47
14/09/01 09:42:20 INFO DAGScheduler: Registering RDD 4 (reduceByKey at JavaWordCount.java:41)
14/09/01 09:42:20 INFO DAGScheduler: Got job 0 (collect at JavaWordCount.java:47) with 4 output partitions (allowLocal=false)
14/09/01 09:42:20 INFO DAGScheduler: Final stage: Stage 0(collect at JavaWordCount.java:47)
14/09/01 09:42:20 INFO DAGScheduler: Parents of final stage: List(Stage 1)
14/09/01 09:42:20 INFO DAGScheduler: Missing parents: List(Stage 1)

14/09/01 09:42:20 INFO DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[4] at reduceByKey at JavaWordCount.java:41), which has no missing parents
14/09/01 09:42:20 INFO DAGScheduler: Submitting 4 missing tasks from Stage 1 (MapPartitionsRDD[4] at reduceByKey at JavaWordCount.java:41)
14/09/01 09:42:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
14/09/01 09:42:22 INFO SparkDeploySchedulerBackend: Registered executor: Actor[    akka.tcp://sparkExecutor@10.0.0.19:57565/user/Executor#1444462950] with ID 0
14/09/01 09:42:22 INFO TaskSetManager: Starting task 1.0:0 as TID 0 on executor 0: 10.0.0.19 (PROCESS_LOCAL)
14/09/01 09:42:22 INFO TaskSetManager: Serialized task 1.0:0 as 2355 bytes in 1 ms
14/09/01 09:42:22 INFO TaskSetManager: Starting task 1.0:1 as TID 1 on executor 0: 10.0.0.19 (PROCESS_LOCAL)
14/09/01 09:42:22 INFO TaskSetManager: Serialized task 1.0:1 as 2356 bytes in 0 ms
14/09/01 09:42:22 INFO TaskSetManager: Starting task 1.0:2 as TID 2 on executor 0: 10.0.0.19 (PROCESS_LOCAL)
14/09/01 09:42:22 INFO TaskSetManager: Serialized task 1.0:2 as 2355 bytes in 1 ms
14/09/01 09:42:22 INFO TaskSetManager: Starting task 1.0:3 as TID 3 on executor 0: 10.0.0.19 (PROCESS_LOCAL)
14/09/01 09:42:22 INFO TaskSetManager: Serialized task 1.0:3 as 2355 bytes in 0 ms
14/09/01 09:42:22 INFO BlockManagerInfo: Registering block manager 10.0.0.19:57570 with 303.4 MB RAM
14/09/01 09:42:23 WARN TaskSetManager: Lost TID 3 (task 1.0:3)
14/09/01 09:42:23 WARN TaskSetManager: Loss was due to java.lang.ClassNotFoundException

java.lang.ClassNotFoundException: examples.JavaWordCount$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

1 个答案:

答案 0 :(得分:2)

你应该发布你的命令,但我想你没有输入完整的类名。您似乎从examples包开始。