Question

我尝试使用scala读取avro文件。

我使用avro-tools提取文件的架构并将其保存到文件中，然后尝试使用以下代码读取它：

 val zibi= scala.io.Source.fromFile("/home/wasabi/schema").mkString
 val schema_obj =  new Schema.Parser
 val schema2 = schema_obj.parse(zibi)
 val READER2 = new GenericDatumReader[GenericRecord](schema2)
 val myFile = Files.readAllBytes(Paths.get("/tmp/check/CMRF_80_1442744555901-1_1_2_1_1_1_4_10_1.avro"))

 val datum = READER2.read(null, DecoderFactory.defaultFactory.createBinaryDecoder(myFile,null))

但是我一直在点击IOExceptions：

java.io.IOException: Invalid int encoding
        at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:145)
        at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
        at org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:444)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:159)
        at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
        at org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:219)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
        at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)

当我通过avro-tools读取文件时，它读起来很好。

我做错了什么？

Answer 1

尝试使用DataFileReader而不是BinaryDecoder。

虽然使用编码器/解码器来编写和读取原始avros，但我怀疑它们会阻塞avro数据文件中的标题信息。

import org.apache.avro.generic.{ GenericDatumReader, GenericRecord }
import org.apache.avro.file.DataFileReader

val zibi= scala.io.Source.fromFile("/home/wasabi/schema").mkString
val schema_obj =  new Schema.Parser
val schema2 = schema_obj.parse(zibi)
val READER2 = new GenericDatumReader[GenericRecord](schema2)

val myFile = new File("/tmp/check/CMRF_80_1442744555901-1_1_2_1_1_1_4_10_1.avro")
val dataFileReader = new DataFileReader[GenericRecord](myFile, READER2)
val datum = dataFileReader.next()

从scala中读取Avro文件

1 个答案: