火花累加器不起作用

时间:2015-07-26 23:24:47

标签: apache-spark

我使用累加器计算记录编号,但它总是打印出来0.请检查我的示例代码如下:

  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("test")
    val sc = new SparkContext(sparkConf)
    val lines = sc.textFile(filePath)
    val recordNumber = sc.accumulator(0)
    lines.foreachPartition{ partitionOfRecords =>
        println("====================recordNumber:"+recordNumber)//always print 0
        partitionOfRecords.foreach { line =>
          recordNumber+=1
          //recordNumber.add(1)//I also tried this way to add up the count
        }
    }
} 

0 个答案:

没有答案