Kudu与PySpark2:KuduStorageHandler出错

时间:2017-08-24 22:33:57

标签: hive cloudera-cdh pyspark-sql apache-spark-2.0 apache-kudu

我正在尝试使用PySpark 2.1.0读取存储为Kudu的数据

>>> from os.path import expanduser, join, abspath
>>> from pyspark.sql import SparkSession
>>> from pyspark.sql import Row
>>> spark = SparkSession.builder \
        .master("local") \
        .appName("HivePyspark") \
        .config("hive.metastore.warehouse.dir", "hdfs:///user/hive/warehouse") \
        .enableHiveSupport() \
        .getOrCreate()
>>> spark.sql("select count(*) from mySchema.myTable").show()

我在集群上安装了Kudu 1.2.0。那些是hive / Impala表。

当我执行最后一行时,出现以下错误:

.
.
.
: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.com.cloudera.kudu.hive.KuduStorageHandler
.
.
.
aused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.com.cloudera.kudu.hive.KuduStorageHandler
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:315)
    at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:284)
    ... 61 more
Caused by: java.lang.ClassNotFoundException: com.cloudera.kudu.hive.KuduStorageHandler

我指的是以下资源:

我很想知道如何将Kudu相关的依赖项包含到我的pyspark程序中,以便我可以移除此错误。

2 个答案:

答案 0 :(得分:0)

我解决这个问题的方法是将相应的jar for kudu-spark传递给pyspark2 shell或者spark2-submit命令

答案 1 :(得分:0)

Apache Spark 2.3

下面是供您参考的代码:


使用以下代码从pyspark读取kudu表:

kuduDF = spark.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE name").load()

kuduDF.show(5)

使用以下代码写入kudu表:

DF.write.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE name").mode("append").save()

参考链接:https://medium.com/@sciencecommitter/how-to-read-from-and-write-to-kudu-tables-in-pyspark-via-impala-c4334b98cf05

如果您想使用Scala,以下是参考链接:

https://kudu.apache.org/docs/developing.html