Question

我有一些文件记录，其中存储为纯文本Json。样本记录：

{
      "datasetID": "Orders",
      "recordID": "rid1",
      "recordGroupID":"asdf1",
      "recordType":"asdf1",
      "recordTimestamp": 100,
      "recordPartitionTimestamp": 100,
      "recordData":{
        "customerID": "cid1",
        "marketplaceID": "mid1",
        "quantity": 10,
        "buyingDate": "1481353448",
        "orderID" : "oid1"
      }
}

对于每条记录，recordData可能是null。如果recordData存在，则orderID可能为null。

我编写以下Avro架构来表示结构：

[{
  "namespace":"model",
  "name":"OrderRecordData",
  "type":"record",
  "fields":[
    {"name":"marketplaceID","type":"string"},
    {"name":"customerID","type":"string"},
    {"name":"quantity","type":"long"},
    {"name":"buyingDate","type":"string"},
    {"name":"orderID","type":["null", "string"]}
  ]
},
{
  "namespace":"model",
  "name":"Order",
  "type":"record",
  "fields":[
    {"name":"datasetID","type":"string"},
    {"name":"recordID","type":"string"},
    {"name":"recordGroupID","type":"string"},
    {"name":"recordType","type":"string"},
    {"name":"recordTimestamp","type":"long"},
    {"name":"recordPartitionTimestamp","type":"long"},
    {"name":"recordData","type": ["null", "model.OrderRecordData"]}
  ]
}]

Ans最后，我使用以下方法将每个String记录反序列化为我的Avro类：

Order jsonDecodeToAvro(String inputString) {
        return new SpecificDatumReader<Order>(Order.class)
           .read(null, DecoderFactory.get().jsonDecoder(Order.SCHEMA$, inputString));
}

但是我在尝试达到上述记录时仍然遇到异常：

org.apache.avro.AvroTypeException: Unknown union branch customerID
    at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:445)

我做错了什么？我使用的是JDK8和Avro 1.7.7

Answer 1

json输入必须采用

形式

{
  "datasetID": "Orders",
  "recordID": "rid1",
  "recordGroupID":"asdf1",
  "recordType":"asdf1",
  "recordTimestamp": 100,
  "recordPartitionTimestamp": 100,
  "recordData":{
  "model.OrderRecordData" :{
    "orderID" : null,
    "customerID": "cid1",
    "marketplaceID": "mid1",
    "quantity": 10,
    "buyingDate": "1481353448"
    } 
  }
}

这是因为Avro的JSON编码处理联合和空值的方式。

看看这个：

How to fix Expected start-union. Got VALUE_NUMBER_INT when converting JSON to Avro on the command line?

还有一个未解决的问题： https://issues.apache.org/jira/browse/AVRO-1582

将Json数据解码为Avro类

1 个答案: