Question

Am使用avro 1.8.2和像这样的简单avro idl记录

record FooRecord {
       string fooString;
       int fooInt;
       union {null, date} fooDate = null;
}

无法写入镶木地板。从我看到的一些解决方案中，他们建议使用avro 1.9（但我可能不想这样做，因为我们当前的大多数技术堆栈都使用1.8.2）。

是否有某种方法可以使avro-1.8.2正常工作？

java.lang.ClassCastException: org.joda.time.LocalDate cannot be cast to java.lang.Number
    at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:323)
    at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:275)
    at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
    at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
    at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299)

Answer 1

需要添加逻辑数据类型时间支持

    GenericData timeSupport = new GenericData();
    // need to add logicalTime Support
    timeSupport.addLogicalTypeConversion(new TimeConversions.DateConversion());
    timeSupport.addLogicalTypeConversion(new TimeConversions.TimeConversion());
    timeSupport.addLogicalTypeConversion(new TimeConversions.TimestampConversion());
    ParquetWriter<GenericRecord> avroParquetWriter = AvroParquetWriter.<GenericRecord> builder(parquetPath)
                                                                      .withSchema(avro.getSchema())
                                                                      .withDataModel(timeSupport)
                                                                      .withCompressionCodec(CompressionCodecName.SNAPPY)
                                                                      .build();

avro 1.8.2日期类型和拼写形式

1 个答案: