Question

我尝试使用pyspark的HiveContext将外部表格加载为avro格式。外部表创建查询在配置单元中运行。但是，相同的查询在hive上下文中失败，错误为org.apache.hadoop.hive.serde2.SerDeException: Encountered exception determining schema. Returning signal schema to indicate problem: null

我的avro架构如下。

{
  "type" : "record",
  "name" : "test_table",
  "namespace" : "com.ent.dl.enh.test_table",
  "fields" : [ {
    "name" : "column1",
    "type" : [ "null", "string" ] , "default": null
  }, {
    "name" : "column2",
    "type" : [ "null", "string" ] , "default": null
  }, {
    "name" : "column3",
    "type" : [ "null", "string" ] , "default": null
  }, {
    "name" : "column4",
    "type" : [ "null", "string" ] , "default": null
  } ]
}

我的创建表脚本是

CREATE EXTERNAL TABLE test_table_enh ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION 's3://Staging/test_table/enh' TBLPROPERTIES ('avro.schema.url'='s3://Staging/test_table/test_table.avsc')

我使用spark-submit，

在代码下面运行

from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext

print "Start of program"
sc = SparkContext()
hive_context = HiveContext(sc)


hive_context.sql("CREATE EXTERNAL TABLE test_table_enh ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION 's3://Staging/test_table/enh' TBLPROPERTIES ('avro.schema.url'='s3://Staging/test_table/test_table.avsc')")

print "end"

Spark版本：2.2.0 OpenJDK版本：1.8.0 Hive版本：2.3.0

在写入avro格式时，在pyspark的HiveContext中查询失败

0 个答案: