Question

Spark RDD具有saveAsTxtFile功能。但是，我如何打开一个文件并将一个简单的字符串写入hadoop商店？

val sparkConf: SparkConf = new SparkConf().setAppName("example")
val sc: SparkContext = new SparkContext(sparkConf)

sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "...")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "...")

val lines: RDD[String] = sc.textFile("s3n://your-output-bucket/lines.txt")
val lengths: RDD[Int] = lines.map(_.length)
lengths.saveAsTextFile("s3n://your-output-bucket/lenths.txt")

val numLines: Long = lines.count
val resultString: String = s"numLines: $numLines"
// how to save resultString to "s3n://your-output-bucket/result.txt"

sc.stop()

Answer 1

为什么不这样做？

val strings = sc.parallelize(Seq("hello", "there"), <numPartitions>)
strings.saveAsTextFile("<path-to-file>")

否则，您可能需要查看hadoop API来编写文件并从驱动程序中明确调用该代码。

Answer 2

假设您SparkContext绑定到sc：

import java.io.{BufferedWriter, OutputStreamWriter}

val hdfs = org.apache.hadoop.fs.FileSystem.get(sc.hadoopConfiguration)

val outputPath = 
  new org.apache.hadoop.fs.Path("hdfs://localhost:9000//tmp/hello.txt")

val overwrite = true

val bw = 
  new BufferedWriter(new OutputStreamWriter(hdfs.create(outputPath, overwrite)))
bw.write("Hello, world")
bw.close()

注意：为了简单起见，在发生异常时没有代码可以关闭编写器。

在Spark中，如何在没有RDD的情况下在Hadoop上编写文件？

2 个答案: