Question

我是Hadoop / Spark的新手，并尝试将多行输入blob处理为csv或制表符分隔格式以供进一步处理。

示例输入

------------------------------------------------------------------------
AAA=someValueAAA1
BBB=someValueBBB1
CCC=someValueCCC1
DDD=someValueDDD1
EEE=someValueEEE1
FFF=someValueFFF1
ENDOFRECORD
------------------------------------------------------------------------
AAA=someValueAAA2
BBB=someValueBBB2
CCC=someValueCCC2
DDD=someValueDDD2
EEE=someValueEEE2
FFF=someValueFFF2
ENDOFRECORD
------------------------------------------------------------------------
AAA=someValueAAA3
BBB=someValueBBB3
CCC=someValueCCC3
DDD=someValueDDD3
EEE=someValueEEE3
FFF=someValueFFF3
GGG=someValueGGG3
HHH=someValueHHH3
ENDOFRECORD
------------------------------------------------------------------------

需要的输出

someValueAAA1, someValueBBB1, someValueCCC1, someValueDDD1, someValueEEE1, someValueFFF1
someValueAAA2, someValueBBB2, someValueCCC2, someValueDDD2, someValueEEE2, someValueFFF2
someValueAAA3, someValueBBB3, someValueCCC3, someValueDDD3, someValueEEE3, someValueFFF3

代码香港专业教育学院迄今为止尝试过 -

#inputRDD
val inputRDD = sc.textFile("/somePath/someFile.gz")

#transform
val singleRDD = inputRDD.map(x=>x.split("ENDOFRECORD")).filter(x=>x.trim.startsWith("AAA"))


val logData = singleRDD.map(x=>{
  val rowData = x.split("\n")

  var AAA = ""
  var BBB = ""
  var CCC = ""
  var DDD = ""
  var EEE = ""
  var FFF = ""

  for (data <- rowData){
    if(data.trim().startsWith("AAA")){
      AAA = data.split("AAA=")(1)
    }else if(data.trim().startsWith("BBB")){
      BBB = data.split("BBB=")(1)
    }else if(data.trim().startsWith("CCC=")){
      CCC = data.split("CCC=")(1)
    }else if(data.trim().startsWith("DDD=")){
      DDD = data.split("DDD=")(1)
    }else if(data.trim().startsWith("EEE=")){
      EEE = data.split("EEE=")(1)
    }else if(data.trim().startsWith("FFF=")){
      FFF = data.split("FFF=")(1)
    }
  }
  (AAA,BBB,CCC,DDD,EEE,FFF)
})

logData.take(10).foreach(println)

这似乎不起作用，我得到o / p，如

AAA,,,,,,
,BBB,,,,,
,,CCC,,,,
,,,DDD,,,

似乎无法弄清楚这里有什么不对。我是否必须编写自定义输入格式来解决此问题？

Answer 1

根据您的要求处理数据：

将数据集加载为wholeTextFiles，这会使您的数据集成为键值对
将键值对转换为FlatMap以获取单个文本集合。例如：

AAA=someValueAAA1 BBB=someValueBBB1 CCC=someValueCCC1 DDD=someValueDDD1 EEE=someValueEEE1 FFF=someValueFFF1 ENDOFRECORD
通过使用\n

请尝试以下代码：

// load your data set
val data = sc.wholeTextFiles("file:///path/to/file")

val data1 = data.flatMap(x => x._2.split("ENDOFRECORD"))

val logData = data1.map(x=>{
  val rowData = x.split("\n")

  var AAA = ""
  var BBB = ""
  var CCC = ""
  var DDD = ""
  var EEE = ""
  var FFF = ""

  for (data <- rowData){
    if(data.trim().contains("AAA")){
      AAA = data.split("AAA=")(1)
    }else if(data.trim().contains("BBB")){
      BBB = data.split("BBB=")(1)
    }else if(data.trim().contains("CCC=")){
      CCC = data.split("CCC=")(1)
    }else if(data.trim().contains("DDD=")){
      DDD = data.split("DDD=")(1)
    }else if(data.trim().contains("EEE=")){
      EEE = data.split("EEE=")(1)
    }else if(data.trim().contains("FFF=")){
      FFF = data.split("FFF=")(1)
    }
  }
  (AAA,BBB,CCC,DDD,EEE,FFF)
})

logData.foreach(println)

<强>输出：

(someValueAAA1,someValueBBB1,someValueCCC1,someValueDDD1,someValueEEE1,someValueFFF1)
(someValueAAA2,someValueBBB2,someValueCCC2,someValueDDD2,someValueEEE2,someValueFFF2)
(someValueAAA3,someValueBBB3,someValueCCC3,someValueDDD3,someValueEEE3,someValueFFF3)

Spark：处理多行输入blob

1 个答案: