Question

在我的spark应用程序中，我需要从本地文件系统中查找一个小的输入文件。

所以，我尝试将文件加载为sc.textFile（“file：/// AbsolutePathToFile”）;

当我使用spark-submit（在本地模式下）运行此程序时，它会抛出以下异常：

14/11/25 17:27:30 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.IllegalArgumentException: For input string: "0"
    at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
    at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
    at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
    at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:184)
    at org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:184)
    at scala.Option.map(Option.scala:145)
    at org.apache.spark.SparkConf.getBoolean(SparkConf.scala:184)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:356)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:325)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:323)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:323)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
14/11/25 17:27:30 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.lang.IllegalArgumentException: For input string: "0"
        scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
        scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
        scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
        org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:184)
        org.apache.spark.SparkConf$$anonfun$getBoolean$2.apply(SparkConf.scala:184)
        scala.Option.map(Option.scala:145)
        org.apache.spark.SparkConf.getBoolean(SparkConf.scala:184)
        org.apache.spark.util.Utils$.fetchFile(Utils.scala:356)
        org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:325)
        org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:323)
        scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
        scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
        scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
        scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
        scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
        scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
        scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
        org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:323)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        java.lang.Thread.run(Thread.java:679)

但是相同的代码在使用spark-scala的eclipse配置中工作正常。

请帮我解决这个问题。

另外，让我知道从eclipse IDE执行程序和spark submit之间有什么区别。

带有localfile系统的SparkContext textFile抛出IllegalArgumentException

0 个答案: