如何修复过滤器映射和mapWithState函数

时间:2018-06-28 08:08:55

标签: scala apache-spark spark-streaming

我是一个新的scala corder,我有一个FlatMap函数,它返回一个FlatMappedDStream对象,这是一个火花流作业,该句柄函数返回一个Map [String,Any],代码如下:

val parseAction = filterActions.flatMap(record => ParseOperation.parseMatch(categoryMap, record))

功能定义:

val parseMatch = (categoryMap: collection.Map[Int, String], record: Map[String, Any]) => {
record.get("operation").get.toString match {
  case "view" => parseView(categoryMap, record)
  case "impression" => parseRecord(record)
  case "click" => parseRecord(record)
  case _ => ListBuffer.apply(record)

}

}

parseMatch函数返回类型为Map [String,Any]的已处理流记录,现在我要打印所有结果并放入新的过滤器map函数和mapWithState函数,我尝试了一下,但是没有用。 错误的代码如下:

val finalActions = parseAction.filter(record => record.get("invalid").get == None)

val userModels = finalActions.map(record => (record.asInstanceOf[Map[String, Any]].getOrElse("deviceid", ""), record))
                       .mapWithState(StateSpec.function(stateUpdateFunction))

mapWithState函数为:

val stateUpdateFunction = (deviceId: Any, newRecord: Option[Map[String, Any]], stateData: State[Map[String, Any]]) => {
  XXXX

}

但是filter函数和mapWithState函数不正确,如何解决?

1 个答案:

答案 0 :(得分:0)

我已修复它,我将返回类型从Map [String,Any]修改为ListBuffer [Map [String,Any]],它确实有效!