将RDD写入CSV文件

Question

我有一个格式为RDD[(Int, Double)]的相对较小的RDD，我希望将其写入csv文件。按照Writing a RDD to a csv的逻辑，我最终得到了以下代码：

val myRdd.map{case(a, b) =>
  var line = a.toString + "," + b.toString
  line
}.saveAsTextFile

但是，我收到以下错误：

Main.scala:111: ambiguous reference to overloaded definition,
[error] both method saveAsTextFile in class RDD of type (path: String, codec: Class[_ <: org.apache.hadoop.io.compress.CompressionCodec])Unit
[error] and  method saveAsTextFile in class RDD of type (path: String)Unit
[error] match expected type ?
[error]   }.saveAsTextFile
[error]     ^

有什么建议吗？

Answer 1

您必须提供路径：

val myRdd.map{case(a, b) =>
  var line = a.toString + "," + b.toString
  line
}.saveAsTextFile("path");

本地和HDFS路径都是正确的。 Here是文档