Question

我正在尝试为以下json（针对Hadoop）构建Avro架构：

{
  "name_tag":"Guy",
  "known_nested_structure" : {
    "fieldA" : ["value1"],
    "fieldB" : ["value1","value2"],
    "fieldC" : [],
    "fieldD" : ["value1"]
  },
  "another_field" : "hi"
}

我的第一个想法是这个avro架构（包括hive命令）：

CREATE EXTERNAL TABLE IF NOT EXISTS record_table
     PARTITIONED BY (YEAR INT, MONTH INT, DAY INT, HOUR INT)
     ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
     STORED AS
     INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
     OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
     LOCATION 'hdfs://localhost/data/output/records_data/hourly'
     TBLPROPERTIES ('avro.schema.literal'='{
  "name": "myRecord",
  "type": "record",
  "fields": [
    {"name":"name_tag", "type":"string",c"default": ""},
    {
      "name": "known_nested_structure",
      "type": "record",
      "fields": [
          {"name":"fieldA", "type":{"type":"array","items":"string"},"default":null},
          {"name":"fieldB", "type":{"type":"array","items":"string"},"default":null},
          {"name":"fieldC", "type":{"type":"array","items":"string"},"default":null},
          {"name":"fieldD", "type":{"type":"array","items":"string"},"default":null}
        ],
        "default":null
    },
    {"name": "another_field","type":"string","default": ""}
  ]
}');

命令的配置结果：好来自反序列化程序的error_error_error_error_error_error_error字符串来自反序列化器的cannot_determine_schema字符串检查来自反序列化器的字符串来自反序列化器的模式字符串来自反序列化器的url字符串和反序列化器中的字符串来自反序列化器的文字字符串年int 月int 天int 小时int 所用时间：0.128秒

但由于某种原因，这是有效的avro架构。

{
  "name": "myRecord",
  "type": "record",
  "fields": [
    {"name":"name_tag", "type":"string","default": null},
    {
  "name": "known_nested_structure",
  "type": {
        "name": "known_nested_structure",
        "type": "record",
        "fields": [
                {"name":"fieldA", "type":{"type":"array","items":"string"},"default":null},
                {"name":"fieldB", "type":{"type":"array","items":"string"},"default":null},
                {"name":"fieldC", "type":{"type":"array","items":"string"},"default":null},
                {"name":"fieldD", "type":{"type":"array","items":"string"},"default":null}
              ],
              "default":null

       }
    },
        {"name": "another_field","type": "string","default": null}
  ]
}

结果：

OK
name_tag    string  from deserializer
known_nested_structure          struct<fielda:array<string>,fieldb:array<string>,fieldc:array<string>,fieldd:array<string>>         from deserializer
another_field   string  from deserializer
year    int 
month   int 
day int 
hour    int 
Time taken: 0.123 seconds

第一个avro架构不起作用的原因是什么？为什么我不能直接将记录作为字段（known_nested_structure在我的第二个模式示例中的known_nested_structure中）？

谢谢，

盖

Answer 1

正如我所看到的，AvroSerde使用Avro api并解析Schema，它使用org.apache.avro.Schema的parse（）方法。如果您查看该方法，您可以清楚地看到它在读取字段时进行了解析的递归调用。因此，如果您的字段中有“记录”，则需要遵循与（name，type =“record”，fields []）序列相同的约定。这就是你的第二个avro工作和第一个失败的可能原因。 org.apache.avro.Schema上的grepcode，它应该解释。

Answer 2

我可以在您的架构中看到一个错误（默认情况下 c ）：

{"name":"name_tag", "type":"string",c"default": ""},

应该是：

{"name":"name_tag", "type":"string","default": ""},

在Avro架构中为简单的json创建嵌套记录

2 个答案: