Question

val SCHEMA : Schema = ....
dStream.map(b => deserialize(bytes))

def deserialize(b: Array[Byte]): GenericRecord = {
      new GenericDatumReader[GenericRecord](SCHEMA)
        .read(null, DecoderFactory.get().jsonDecoder(SCHEMA, new ByteArrayInputStream(b)))
  }

我需要将SCHEMA传递给地图功能。如何在反序列化方法中传递SCHEMA变量。

Answer 1

使用咖喱粉

deserialize(schema: Schema)(b: Array[Byte]): GenericRecord = { ... }
dStream.map(deserialize(SCHEMA))

或二进制函数：

def deserialize(b: Array[Byte], schema: Schema): GenericRecord = { ... }

dStream.map(b => deserialize(b, SCHEMA))

如何在Spark Scala中的地图函数上传递常量值

1 个答案: