我每次遇到此错误时都尝试运行一个字数计数spark应用程序请帮助,以下是wordcount.scala
文件,sbt
包后我运行了spark-submit
命令
package main
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object WordCount {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Word Count")
val sc = new SparkContext(conf)
val textfile = sc.textFile("file:///usr/local/spark/README.md")
val tokenizeddata = textfile.flatMap(line => line.split(" "))
val countprep = tokenizeddata.map(word => (word,1))
val counts = countprep.reduceByKey((accumvalue,newvalue)=>(accumvalue+newvalue))
val sortedcount = counts.sortBy(kvpair=>kvpair._2,false)
sortedcount.saveAsTextFile("file:///usr/local/wordcount")
}
}
我跑了下一个命令。
bin/spark-submit --class "main.WordCount" --master "local[*]" "/home/hadoop/SparkApps/target/scala-2.10/word-count_2.10-1.0.jar"
Spark程序集已经使用Hive构建,包括Datanucleus jar classpath Java HotSpot(TM)64位服务器VM警告:
忽略选项MaxPermSize = 128m;支持已在8.0中删除15/11/28 07:38:51错误执行程序:阶段1.0中任务1.0中的异常
(TID 1)java.net.NoRouteToHostException:没有到主机的路由 at java.net.PlainSocketImpl.socketConnect(Native Method) 在java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 在java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 在java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 在java.net.Socket.connect(Socket.java:589) 在sun.net.NetworkClient.doConnect(NetworkClient.java:175) 在sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 在sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 在sun.net.www.http.HttpClient。(HttpClient.java:211) 在sun.net.www.http.HttpClient.New(HttpClient.java:308) 在sun.net.www.http.HttpClient.New(HttpClient.java:326) 在sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169) 在sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) 在sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933) 在org.apache.spark.util.Utils $ .fetchFile(Utils.scala:375) 在org.apache.spark.executor.Executor $$ anonfun $ org $ apache $ spark $ executor $ Executor $$ updateDependencies $ 6.apply(Executor.scala:325) at org.apache.spark.executor.Executor $$ anonfun $ org $ apache $ spark $ executor $ Executor $$ updateDependencies $ 6.apply(Executor.scala:323) 在scala.collection.TraversableLike $ WithFilter $$ anonfun $ foreach $ 1.apply(TraversableLike.scala:772) 在scala.collection.mutable.HashMap $$ anonfun $ foreach $ 1.apply(HashMap.scala:98) 在scala.collection.mutable.HashMap $$ anonfun $ foreach $ 1.apply(HashMap.scala:98) 在scala.collection.mutable.HashTable $ class.foreachEntry(HashTable.scala:226) 在scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 在scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 在scala.collection.TraversableLike $ WithFilter.foreach(TraversableLike.scala:771) 在org.apache.spark.executor.Executor.org $ apache $ spark $ executor $ Executor $$ updateDependencies(Executor.scala:323) 在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:158) 在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617) 在java.lang.Thread.run(Thread.java:745)
答案 0 :(得分:0)
也许您应该添加.setMaster(“ local”)