Question

我正在尝试为以下json构建一个avro架构：

{
  "id":1234,
  "my_name_field": "my_name",
  "extra_data": {
      "my_long_value": 1234567890,
      "my_message_string": "Hello World!",
      "my_int_value":  777,
      "some_new_field": 1
  }
}

'id'和'my_name_field'的值已知，但'extra_data'中的字段会动态更改并且未知。

我想到的avro架构是：

{
    "name":"my_record",
    "type":"record",
    "fields":[
        {"name":"id", "type":"int", "default":0},
        {"name":"my_name_field", "type":"string", "default":"NoName"},
        { "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]}     }        
    ]
}

我的第一个想法是使用地图制作'extra_data'记录，但这不起作用：

{ "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]} }

我明白了：

AvroTypeException: Expected start-union. Got VALUE_NUMBER_INT

apache在https://cwiki.apache.org/confluence/display/Hive/AvroSerDe中给出了一些不错的例子，但似乎都没有做到这一点。

这是我要检查的单元测试：

公共类AvroTest {

@Test
public void readRecord() throws IOException {

    String event="{\"id\":1234,\"my_name_field\":\"my_name\",\"extra_data\":{\"my_long_value\":1234567890,\"my_message_string\":\"Hello World!\",\"my_int_value\":777,\"some_new_field\":1}}";

    SchemaRegistry<Schema> registry = new com.linkedin.camus.schema.MySchemaRegistry();
    DecoderFactory decoderFactory = DecoderFactory.get();

    ObjectMapper mapper = new ObjectMapper();
    GenericDatumReader<GenericData.Record> reader = new GenericDatumReader<GenericData.Record>();
    Schema schema = registry.getLatestSchemaByTopic("record_topic").getSchema();
    reader.setSchema(schema);

    HashMap hashMap = mapper.readValue(event, HashMap.class);
    long now = Long.valueOf(hashMap.get("now").toString())*1000;
    GenericData.Record read = reader.read(null, decoderFactory.jsonDecoder(schema, event));
}

非常感谢这方面的帮助，感谢。

Answer 1

如果使用多个可选值字段确实未知额外数据字段列表可能会有所帮助，如下所示：

{
    "name":"my_record",
    "type":"record",
    "fields":[
        {"name":"id", "type":"int", "default":0},
        {"name":"my_name_field", "type":"string", "default":"NoName"},
        {"name":"extra_data", "type": "array", "items": {
            {"name": "extra_data_entry", "type":"record", "fields": [
                {"name":"extra_data_field_name", "type": "string"},
                {"name":"extra_data_field_type", "type": "string"},
                {"name":"extra_data_field_value_string", "type": ["null", "string"]},
                {"name":"extra_data_field_value_int", "type": ["null", "int"]},
                {"name":"extra_data_field_value_long", "type": ["null", "long"]}
            ]}
        }}
    ]
}

然后，您可以根据该字段的extra_data_field_value_*选择extra_data_field_type值。

为简单的json创建Avro架构

1 个答案: