Question

我在 Mac （jupyter笔记本电脑）而不是 Windows 上运行Spark。我正在尝试读取txt文件：

val text = sc.textFile("shakespeare.txt")
val relevant_lines = text.filter(l => l.contains("Music"))
val result = relevant_lines.count()

我收到以下错误：

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: Module 3:%20Apache%20Spark
  at org.apache.hadoop.fs.Path.initialize(Path.java:205)
  at org.apache.hadoop.fs.Path.<init>(Path.java:171)
  at org.apache.hadoop.fs.Path.<init>(Path.java:93)
  at org.apache.hadoop.fs.Globber.glob(Globber.java:211)
  at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1676)
  at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
  at org.apache.spark.rdd.RDD.count(RDD.scala:1168)
  ... 37 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI: Module 3:%20Apache%20Spark
  at java.base/java.net.URI.checkPath(URI.java:1941)
  at java.base/java.net.URI.<init>(URI.java:757)
  at org.apache.hadoop.fs.Path.initialize(Path.java:202)
  ... 61 more

您能帮我解决它吗？

谢谢

Answer 1

Give the complete path where the text file is located in your MAC.
eg -: "/user/name/shakespeare.txt" 

For multiple text files 
Syntax-: sc.textFile("/user/name/*")

val text = sc.textFile("/user/name/shakespeare.txt")
val relevant_lines = text.filter(l => l.contains("Music"))
val result = relevant_lines.count()

绝对URI中的相对路径：txt Spark mac

1 个答案: