Question

我正在编写自定义SerDe，并且只会使用它来反序列化。基础数据是节俭的二进制文件，每一行都是事件日志。每个事件都有我可以访问的模式，但是我们将事件包装在另一个模式中，在存储之前将其称为Message。我写SerDe而不是使用ThriftDeserializer的原因是因为如上所述，基础事件被包装为Message。因此，我们首先需要使用Message的模式进行反序列化，然后对该事件的数据进行反序列化。

SerDe仅在执行SELECT *时可以工作，并且可以按预期反序列化数据，但是每当我从表中选择列而不是SELECT *时，行都是NULL。返回的对象检查器是ThriftStructObjectInspector，反序列化返回的对象是TBase。

什么会导致Hive在选择列时返回NULL，但在我执行SELECT *时返回列数据？

这是SerDe类（更改了一些类名）：

public class MyThriftSerde extends AbstractSerDe {

  private static final Log LOG = LogFactory.getLog(MyThriftSerde.class);

  /* Abstracting away the deserialization of the underlying event which is wrapped in a message */
  private static final MessageDeserializer myMessageDeserializer =
      MessageDeserializer.getInstance();

  /* Underlying event class which is wrapped in a Message */
  private String schemaClassName;
  private Class<?> schemaClass;

  /* Used to read the input row */
  public static List<String> inputFieldNames;
  public static List<ObjectInspector> inputFieldOIs;
  public static List<Integer> notSkipIDs;
  public static ObjectInspector inputRowObjectInspector;

  /* Output Object Inspector */
  public static ObjectInspector thriftStructObjectInspector;

  @Override
  public void initialize(Configuration conf, Properties tbl) throws SerDeException {
    try {

      logHeading("INITIALIZE MyThriftSerde");

      schemaClassName = tbl.getProperty(SERIALIZATION_CLASS);
      schemaClass = conf.getClassByName(schemaClassName);

      LOG.info(String.format("Building DDL for event: %s", schemaClass.getName()));

      inputFieldNames = new ArrayList<>();
      inputFieldOIs = new ArrayList<>();
      notSkipIDs = new ArrayList<>();

      /* Initialize the Input fields */

      // The underlying data is stored in RCFile format, and only has 1 column, event_binary
      // So we create a ColumnarStructBase for each row we deserialize.
      // This ColumnasStruct only has 1 column: event_binary
      inputFieldNames.add("event_binary");
      notSkipIDs.add(0);
      inputFieldOIs.add(LazyPrimitiveObjectInspectorFactory.LAZY_BINARY_OBJECT_INSPECTOR);
      inputRowObjectInspector =
          ObjectInspectorFactory.getColumnarStructObjectInspector(inputFieldNames, inputFieldOIs);

      /* Output Object Inspector*/

      // This is what the SerDe will return, it is a ThriftStructObjectInspector
      thriftStructObjectInspector =
          ObjectInspectorFactory.getReflectionObjectInspector(
              schemaClass, ObjectInspectorFactory.ObjectInspectorOptions.THRIFT);

      // Only for debugging
      logHeading("THRIFT OBJECT INSPECTOR");
      LOG.info("Output OI Class Name: " + thriftStructObjectInspector.getClass().getName());
      LOG.info(
          "OI Details: "
              + ObjectInspectorUtils.getObjectInspectorName(thriftStructObjectInspector));

    } catch (Exception e) {
      LOG.info("Exception while initializing SerDe", e);
    }
  }

  @Override
  public Object deserialize(Writable rowWritable) throws SerDeException {

    logHeading("START DESERIALIZATION");

    ColumnarStructBase inputLazyStruct =
        new ColumnarStruct(inputRowObjectInspector, notSkipIDs, null);
    LazyBinary eventBinary;
    Message rowAsMessage;
    TBase deserializedRow = null;

    try {
      inputLazyStruct.init((BytesRefArrayWritable) rowWritable);
      eventBinary = (LazyBinary) inputLazyStruct.getField(0);
      rowAsMessage =
          myMessageDeserializer.fromBytes(eventBinary.getWritableObject().copyBytes(), null);
      deserializedRow = rowAsMessage.getEvent();

      LOG.info("deserializedRow.getClass(): " + deserializedRow.getClass());
      LOG.info("deserializedRow.toString(): " + deserializedRow.toString());

    } catch (Exception e) {
      e.printStackTrace();
    }

    logHeading("END DESERIALIZATION");

    return deserializedRow;
  }

  private void logHeading(String s) {
    LOG.info(String.format("-------------------  %s  -------------------", s));
  }

  @Override
  public ObjectInspector getObjectInspector() {
    return thriftStructObjectInspector;
  }
}

代码上下文：

在基础数据中，每一行仅包含1列（称为event_binary），以二进制形式存储。二进制是一条消息，其中包含2个字段“ schema” +“ event_data”。即每行是一条消息，其中包含基础事件的架构+数据。我们使用Message中的模式对数据进行反序列化。
SerDe首先将行反序列化为消息，提取事件数据，然后反序列化事件。

我使用创建一个指向Thrift数据的EXTERNAL表

ADD JAR hdfs://my-jar.jar;

CREATE EXTERNAL TABLE dev_db.thrift_event_data_deserialized
ROW FORMAT SERDE 'com.test.only.MyThriftSerde'
WITH SERDEPROPERTIES (
  "serialization.class"="com.test.only.TestEvent"
) STORED AS RCFILE
LOCATION 'location/of/thrift/data';

MSCK REPAIR TABLE thrift_event_data_deserialized;

然后SELECT * FROM dev_db.thrift_event_data_deserialized LIMIT 10;会按预期工作但是，SELECT column1_name, column2_name FROM dev_db.thrift_event_data_deserialized LIMIT 10;不起作用。

知道我在这里缺少什么吗？希望对此有任何帮助！

自定义Hive SerDe无法选择列，但在执行SELECT时可以使用*

0 个答案: