Question

如果有人熟悉Apache Avro的Java实现，那么这在黑暗中就是一个镜头。

我的高级目标是通过网络传输一些avro数据系列（例如，我们只说HTTP，但特定的协议对此并不重要）。在我的上下文中，我有一个HttpServletResponse，我需要以某种方式编写这些数据。

我最初尝试将数据写为avro容器文件的虚拟版本（假设“response”的类型为HttpServletResponse）：

response.setContentType("application/octet-stream");
response.setHeader("Content-transfer-encoding", "binary");
ServletOutputStream outStream = response.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream(outStream);

Schema someSchema = Schema.parse(".....some valid avro schema....");
GenericRecord someRecord = new GenericData.Record(someSchema);
someRecord.put("somefield", someData);
...

GenericDatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(someSchema);
DataFileWriter<GenericRecord> fileWriter = new DataFileWriter<GenericRecord>(datumWriter);
fileWriter.create(someSchema, bos);
fileWriter.append(someRecord);
fileWriter.close();
bos.flush();

这一切都很好，但是事实证明Avro并没有提供一种方法来读取除实际文件之外的容器文件：DataFileReader只有两个构造函数：

public DataFileReader(File file, DatumReader<D> reader);

和

public DataFileReader(SeekableInput sin, DatumReader<D> reader);

其中SeekableInput是一些特定于avro的自定义表单，其创建也最终从文件中读取。现在给出，除非有某种方法以某种方式将InputStream强制转换为文件（http://stackoverflow.com/questions/578305/create-a-java-file-object-or-equivalent-using-a-byte- array-in-memory-without-a表示没有，我也试过查看Java文档），如果OutputStream另一端的阅读器收到该avro容器文件，这种方法将无效（我不确定为什么他们允许一个人将avro二进制容器文件输出到任意OutputStream而不提供从另一端的相应InputStream读取它们的方法，但这不是重点。似乎容器文件阅读器的实现需要具体文件提供的“可搜索”功能。

好的，所以看起来这种做法看起来不像我想做的那样。如何创建模仿avro容器文件的JSON响应？

public static Schema WRAPPER_SCHEMA = Schema.parse(
  "{\"type\": \"record\", " +
   "\"name\": \"AvroContainer\", " +
   "\"doc\": \"a JSON avro container file\", " +
   "\"namespace\": \"org.bar.foo\", " +
   "\"fields\": [" +
     "{\"name\": \"schema\", \"type\": \"string\", \"doc\": \"schema representing the included data\"}, " +
     "{\"name\": \"data\", \"type\": \"bytes\", \"doc\": \"packet of data represented by the schema\"}]}"
  );

考虑到上述限制，我不确定这是否是解决此问题的最佳方法，但看起来这可能会成功。我将把模式（例如，来自上面的“Schema someSchema”）作为一个字符串放在“schema”字段中，然后放入avro-binary-serialized形式的记录拟合该模式（即“GenericRecord” someRecord“）在”数据“字段中。

我实际上想知道下面描述的具体细节，但我认为给出一个更大的背景是值得的，所以如果有一个更好的高级方法我可以采取（这种方法有效，但感觉不太理想）请告诉我。

我的问题是，假设我使用这种基于JSON的方法，如何将我的Record的avro二进制表示写入AvroContainer模式的“data”字段？例如，我到了这里：

ByteArrayOutputStream baos = new ByteArrayOutputStream();
GenericDatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(someSchema);
Encoder e = new BinaryEncoder(baos);
datumWriter.write(resultsRecord, e);
e.flush();

GenericRecord someRecord = new GenericData.Record(someSchema);
someRecord.put("schema", someSchema.toString());
someRecord.put("data", ByteBuffer.wrap(baos.toByteArray()));
datumWriter = new GenericDatumWriter<GenericRecord>(WRAPPER_SCHEMA);
JsonGenerator jsonGenerator = new JsonFactory().createJsonGenerator(baos, JsonEncoding.UTF8);
e = new JsonEncoder(WRAPPER_SCHEMA, jsonGenerator);
datumWriter.write(someRecord, e);
e.flush();

PrintWriter printWriter = response.getWriter(); // recall that response is the HttpServletResponse
response.setContentType("text/plain");
response.setCharacterEncoding("UTF-8");
printWriter.print(baos.toString("UTF-8"));

我最初尝试省略ByteBuffer.wrap子句，但接着是行

datumWriter.write(someRecord, e);

引发了一个异常，我无法将字节数组转换为ByteBuffer。很公平，看起来当调用Encoder类（其中JsonEncoder是子类）来编写avro Bytes对象时，它需要将ByteBuffer作为参数给出。因此，我尝试使用java.nio.ByteBuffer.wrap封装byte []，但是当打印出数据时，它被打印为一系列直接字节，而不是通过avro十六进制表示：

"data": {"bytes": ".....some gibberish other than the expected format...}

这似乎不对。根据avro文档，他们给出的示例字节对象说我需要放入一个json对象，其中一个例子看起来像“\ u00FF”，而我放在那里的内容显然不是那种格式。我现在想知道的是：

avro字节格式的示例是什么？它看起来像“\ uDEADBEEFDEADBEEF ......”吗？
如何将我的二进制avro数据（由BinaryEncoder输出到byte []数组中）强制转换为可以插入GenericRecord对象并在JSON中正确打印的格式？例如，我想要一个对象数据，我可以调用一些GenericRecord“someRecord.put（”data“，DATA）;”我的avro序列化数据在里面？
当给出文本JSON表示并希望重新创建由AvroContainer格式JSON表示的GenericRecord时，我如何将该数据读回另一个（消费者）端的字节数组？
（重申之前的问题）我有更好的方法可以做到这一切吗？

Answer 1

正如克努特所说，如果你想使用文件以外的东西，你可以：

使用SeekableByteArrayInput，就像Knut所说的那样，你可以将任何东西都塞进一个字节数组中
以您自己的方式实现SeekablInput - 例如，如果您从一些奇怪的数据库结构中获取它。
或者只是使用一个文件。为什么不呢？

这些是你的答案。

Answer 2

我解决这个问题的方法是将模式与数据分开运送。我设置了一个连接握手，从服务器向下传输模式，然后我来回发送编码数据。您必须创建一个外部包装器对象，如下所示：

{'name':'Wrapper','type':'record','fields':[
  {'name':'schemaName','type':'string'},
  {'name':'records','type':{'type':'array','items':'bytes'}}
]}

首先将您的记录数组逐个编码为编码字节数组。一个数组中的所有内容都应该具有相同的模式。然后使用上面的模式对包装器对象进行编码 - 设置＆＃34; schemaName＆＃34;是用于编码数组的模式的名称。

在服务器上，您将首先解码包装器对象。一旦解码了包装器对象，你就知道了schemaName，并且你有一个你知道如何解码的对象数组 - 使用就像你一样！

请注意，如果您使用WebSockets之类的协议和Socket.IO之类的引擎（对于Node.js），您可以在不使用包装器对象的情况下离开.Socket.io为您提供了一个通道 - 基于浏览器和服务器之间的通信层。在这种情况下，只需为每个通道使用特定的架构，在发送之前对每条消息进行编码。您仍然必须在连接启动时共享模式 - 但如果您使用WebSockets，这很容易实现。完成后，您在客户端和服务器之间拥有任意数量的强类型双向流。

Answer 3

在Java和Scala下，我们通过使用Scala nitro codegen生成的代码尝试使用初始化。初始是Javascript mtth/avsc库如何解决此问题problem。但是，我们使用Java库遇到了几个序列化问题，其中错误的字节被一致地注入到字节流中 - 我们可以不找出这些字节的来源。

当然这意味着使用ZigZag编码构建我们自己的Varint实现。 MEH。

这是：

package com.terradatum.query

import java.io.ByteArrayOutputStream
import java.nio.ByteBuffer
import java.security.MessageDigest
import java.util.UUID

import akka.actor.ActorSystem
import akka.stream.stage._
import akka.stream.{Attributes, FlowShape, Inlet, Outlet}
import com.nitro.scalaAvro.runtime.GeneratedMessage
import com.terradatum.diagnostics.AkkaLogging
import org.apache.avro.Schema
import org.apache.avro.generic.{GenericDatumWriter, GenericRecord}
import org.apache.avro.io.EncoderFactory
import org.elasticsearch.search.SearchHit

import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag

/*
* The original implementation of this helper relied exclusively on using the Header Avro record and inception to create
* the header. That didn't work for us because somehow erroneous bytes were injected into the output.
*
* Specifically:
* 1. 0x08 prepended to the magic
* 2. 0x0020 between the header and the sync marker
*
* Rather than continue to spend a large number of hours trying to troubleshoot why the Avro library was producing such
* erroneous output, we build the Avro Container File using a combination of our own code and Avro library code.
*
* This means that Terradatum code is responsible for the Avro Container File header (including magic, file metadata and
* sync marker) and building the blocks. We only use the Avro library code to build the binary encoding of the Avro
* records.
*
* @see https://avro.apache.org/docs/1.8.1/spec.html#Object+Container+Files
*/
object AvroContainerFileHelpers {

  val magic: ByteBuffer = {
    val magicBytes = "Obj".getBytes ++ Array[Byte](1.toByte)
    val mg = ByteBuffer.allocate(magicBytes.length).put(magicBytes)
    mg.position(0)
    mg
  }

  def makeSyncMarker(): Array[Byte] = {
    val digester = MessageDigest.getInstance("MD5")
    digester.update(s"${UUID.randomUUID}@${System.currentTimeMillis()}".getBytes)
    val marker = ByteBuffer.allocate(16).put(digester.digest()).compact()
    marker.position(0)
    marker.array()
  }

  /*
  * Note that other implementations of avro container files, such as the javascript library
  * mtth/avsc uses "inception" to encode the header, that is, a datum following a header
  * schema should produce valid headers. We originally had attempted to do the same but for
  * an unknown reason two bytes wore being inserted into our header, one at the very beginning
  * of the header before the MAGIC marker, and one right before the syncmarker of the header.
  * We were unable to determine why this wasn't working, and so this solution was used instead
  * where the record/map is encoded per the avro spec manually without the use of "inception."
  */
  def header(schema: Schema, syncMarker: Array[Byte]): Array[Byte] = {
    def avroMap(map: Map[String, ByteBuffer]): Array[Byte] = {
      val mapBytes = map.flatMap {
        case (k, vBuff) =>
          val v = vBuff.array()
          val byteStr = k.getBytes()
          Varint.encodeLong(byteStr.length) ++ byteStr ++ Varint.encodeLong(v.length) ++ v
      }
      Varint.encodeLong(map.size.toLong) ++ mapBytes ++ Varint.encodeLong(0)
    }

    val schemaBytes = schema.toString.getBytes
    val schemaBuffer = ByteBuffer.allocate(schemaBytes.length).put(schemaBytes)
    schemaBuffer.position(0)
    val metadata = Map("avro.schema" -> schemaBuffer)
    magic.array() ++ avroMap(metadata) ++ syncMarker
  }

  def block(binaryRecords: Seq[Array[Byte]], syncMarker: Array[Byte]): Array[Byte] = {
    val countBytes = Varint.encodeLong(binaryRecords.length.toLong)
    val sizeBytes = Varint.encodeLong(binaryRecords.foldLeft(0)(_+_.length).toLong)

    val buff: ArrayBuffer[Byte] = new scala.collection.mutable.ArrayBuffer[Byte]()

    buff.append(countBytes:_*)
    buff.append(sizeBytes:_*)
    binaryRecords.foreach { rec =>
      buff.append(rec:_*)
    }
    buff.append(syncMarker:_*)

    buff.toArray
  }

  def encodeBlock[T](schema: Schema, records: Seq[GenericRecord], syncMarker: Array[Byte]): Array[Byte] = {
    //block(records.map(encodeRecord(schema, _)), syncMarker)
    val writer = new GenericDatumWriter[GenericRecord](schema)
    val out = new ByteArrayOutputStream()
    val binaryEncoder = EncoderFactory.get().binaryEncoder(out, null)
    records.foreach(record => writer.write(record, binaryEncoder))
    binaryEncoder.flush()
    val flattenedRecords = out.toByteArray
    out.close()

    val buff: ArrayBuffer[Byte] = new scala.collection.mutable.ArrayBuffer[Byte]()

    val countBytes = Varint.encodeLong(records.length.toLong)
    val sizeBytes = Varint.encodeLong(flattenedRecords.length.toLong)

    buff.append(countBytes:_*)
    buff.append(sizeBytes:_*)
    buff.append(flattenedRecords:_*)
    buff.append(syncMarker:_*)

    buff.toArray
  }

  def encodeRecord[R <: GeneratedMessage with com.nitro.scalaAvro.runtime.Message[R]: ClassTag](
      entity: R
  ): Array[Byte] =
    encodeRecord(entity.companion.schema, entity.toMutable)

  def encodeRecord(schema: Schema, record: GenericRecord): Array[Byte] = {
    val writer = new GenericDatumWriter[GenericRecord](schema)
    val out = new ByteArrayOutputStream()
    val binaryEncoder = EncoderFactory.get().binaryEncoder(out, null)
    writer.write(record, binaryEncoder)
    binaryEncoder.flush()
    val bytes = out.toByteArray
    out.close()
    bytes
  }
}

/**
  * Encoding of integers with variable-length encoding.
  *
  * The avro specification uses a variable length encoding for integers and longs.
  * If the most significant bit in a integer or long byte is 0 then it knows that no
  * more bytes are needed, if the most significant bit is 1 then it knows that at least one
  * more byte is needed. In signed ints and longs the most significant bit is traditionally
  * used to represent the sign of the integer or long, but for us it's used to encode whether
  * more bytes are needed. To get around this limitation we zig-zag through whole numbers such that
  * negatives are odd numbers and positives are even numbers:
  *
  * i.e. -1, -2, -3 would be encoded as 1, 3, 5, and so on
  * while 1,  2,  3 would be encoded as 2, 4, 6, and so on.
  *
  * More information is available in the avro specification here:
  * @see http://lucene.apache.org/core/3_5_0/fileformats.html#VInt
  *      https://developers.google.com/protocol-buffers/docs/encoding?csw=1#types
  */
object Varint {

  import scala.collection.mutable

  def encodeLong(longVal: Long): Array[Byte] = {
    val buff = new ArrayBuffer[Byte]()
    Varint.zigZagSignedLong(longVal, buff)
    buff.toArray[Byte]
  }

  def encodeInt(intVal: Int): Array[Byte] = {
    val buff = new ArrayBuffer[Byte]()
    Varint.zigZagSignedInt(intVal, buff)
    buff.toArray[Byte]
  }

  def zigZagSignedLong[T <: mutable.Buffer[Byte]](x: Long, dest: T): Unit = {
    // sign to even/odd mapping: http://code.google.com/apis/protocolbuffers/docs/encoding.html#types
    writeUnsignedLong((x << 1) ^ (x >> 63), dest)
  }

  def writeUnsignedLong[T <: mutable.Buffer[Byte]](v: Long, dest: T): Unit = {
    var x = v
    while ((x & 0xFFFFFFFFFFFFFF80L) != 0L) {
      dest += ((x & 0x7F) | 0x80).toByte
      x >>>= 7
    }
    dest += (x & 0x7F).toByte
  }

  def zigZagSignedInt[T <: mutable.Buffer[Byte]](x: Int, dest: T): Unit = {
    writeUnsignedInt((x << 1) ^ (x >> 31), dest)
  }

  def writeUnsignedInt[T <: mutable.Buffer[Byte]](v: Int, dest: T): Unit = {
    var x = v
    while ((x & 0xFFFFF80) != 0L) {
      dest += ((x & 0x7F) | 0x80).toByte
      x >>>= 7
    }
    dest += (x & 0x7F).toByte
  }
}

在Java中，如何在不强制使用File作为媒介的情况下创建Apache Avro容器文件的等效文件？

3 个答案: