Question

如何将RDD [Array [Row]]转换为RDD [Row]？

详细说明：

我有一些用例，我的解析函数为某些数据返回类型Array [Row]，为某些数据返回Row。如何将这两个转换为RDD [Row]以供进一步使用？

代码样本

2017-08-15T15:31:03.11-0700 [CELL/0]     OUT Creating container
2017-08-15T15:31:03.76-0700 [CELL/0]     OUT Successfully destroyed container
2017-08-15T15:31:03.89-0700 [CELL/0]     OUT Successfully created container
2017-08-15T15:31:10.55-0700 [CELL/0]     OUT Starting health monitoring of container
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR module.js:487
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     throw err;
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     ^
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR Error: Cannot find module '/home/vcap/app/supplier-vue.js'
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     at Function.Module._resolveFilename (module.js:485:15)
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     at Function.Module._load (module.js:437:25)
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     at Function.Module.runMain (module.js:605:10)
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     at startup (bootstrap_node.js:158:16)
2017-08-15T15:31:11.00-0700 [APP/PROC/WEB/0]ERR     at bootstrap_node.js:575:3
2017-08-15T15:31:11.02-0700 [APP/PROC/WEB/0]OUT Exit status 1
2017-08-15T15:31:11.02-0700 [CELL/0]     OUT Exit status 0
2017-08-15T15:31:11.04-0700 [CELL/0]     OUT Stopping instance d1d9f847-d52c-4c3c-5499-97a6
2017-08-15T15:31:11.04-0700 [CELL/0]     OUT Destroying container
2017-08-15T15:31:11.12-0700 [API/2]      OUT Process has crashed with type: "web"
2017-08-15T15:31:11.14-0700 [API/2]      OUT App instance exited with guid 3d2511fc-5209-437c-9c9e-377ed10f8b6d payload: {"instance"=>"d1d9f847-d52c-4c3c-5499-97a6", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>3, "crash_timestamp"=>1502836271075038027, "version"=>"a275f709-e0aa-4520-9e28-e44c345c5c45"}
2017-08-15T15:31:12.17-0700 [CELL/0]     OUT Successfully destroyed container

}

private def getRows(rdd: RDD[String], parser: Parser): RDD[Row] = {

var processedLines = rdd.map { line =>      
  map(p => parser.processBeacon(line) }

val rddOfRowsList = processedLines.map { x =>
  x match {
    case Right(obj) => obj.map { p =>          
      MyRow.getValue(p)
    }//I can use flatmap here
    case Left(obj) =>          
      MyRow.getValue(obj)
  }//Cant use flatmap here
}

 // Here I have to convert rddOfRowsList to RDD[Row]
 //?????
 val rowsRdd =?????
 //


rowsRdd

Answer 1

使用// Obtain your viewcontroller let viewController = ... // Initialize a navigation controller, with your view controller as its root let navigationController = UINavigationController(rootViewController: viewController) navigationController.navigationBar.barStyle = .blackTranslucent // Present it modally from the current controller present(navigationController, animated: true, completion: nil);

flatMap

Answer 2

您可以使用flatmap获取新的rdd，然后使用union来组合它们。

Answer 3

使用flatMap平面化RDD的内容

将RDD [Array [Row]]转换为RDD [Row]

3 个答案: