Question

我正在尝试使用spark从Greenplum上的表读取数据。我写了如下代码：

  val yearDF = spark.read.format("io.pivotal.greenplum.spark.GreenplumRelationProvider").option("url", connectionUrl)
    .option("server.port","5432")
    .option("dbtable", "tablename")
    .option("dbschema","schemaname")
    .option("user", devUserName)
    .option("password", devPassword)
    .option("partitionColumn","employeeLoc")
    .option("partitions",450)
    .load()
    .where("period_year=2017 and period_num=12")
    .select(gpColSeq map col:_*)
    .withColumn(flagCol, lit(0))

我正在使用greenplum.jar，它为数据直接jdbc驱动程序提供了使用Spark从greenplum表中读取数据的功能。我正在使用以下spark-submit命令：

SPARK_MAJOR_VERSION=2 spark-submit --class com.partition.source.YearPartition --master=yarn --conf spark.ui.port=4090 --driver-class-path /home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar --conf spark.jars=/home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar --executor-cores 3 --executor-memory 13G --keytab /home/hdpuser/hdpuser.keytab --principal hdpuser@devuser.COM --files /usr/hdp/current/spark2-client/conf/hive-site.xml,testconnection.properties --name Splinter --conf spark.executor.extraClassPath=/home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar splinter_2.11-0.1.jar

提交罐子时，出现异常：

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: io.pivotal.greenplum.spark.GreenplumRelationProvider. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:553)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:304)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
    at com.partition.source.YearPartition$.prepareFinalDF$1(YearPartition.scala:154)
    at com.partition.source.YearPartition$.main(YearPartition.scala:181)
    at com.partition.source.YearPartition.main(YearPartition.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: io.pivotal.greenplum.spark.GreenplumRelationProvider.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22$$anonfun$apply$14.apply(DataSource.scala:537)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22$$anonfun$apply$14.apply(DataSource.scala:537)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22.apply(DataSource.scala:537)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22.apply(DataSource.scala:537)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:537)

我了解这是由于在数据源语句即

中使用了 io.pivotal.greenplum.spark.GreenplumRelationProvider

spark.read.format（“ io.pivotal.greenplum.spark.GreenplumRelationProvider”）

我尝试了“ io.pivotal.greenplum.spark.GreenplumRelationProvider”和“ greenplum”，但两者均导致相同的异常，即：

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source:
io.pivotal.greenplum.spark.GreenplumRelationProvider。请找到 http://spark.apache.org/third-party-projects.html
中的软件包

在使用数据直接jdbc jar：greenplum.jar

时，我无法想到应该在语句spark.read.format中作为数据源提供什么？

任何人都可以让我知道如何解决此问题吗？

Answer 1

您正在使用什么版本的greenplum-spark连接器？

您应该能够在“驱动程序”选项中指定自定义jdbc驱动程序。请参阅http://greenplum-spark.docs.pivotal.io/160/using_the_connector.html#use_custom_jdbcdriver。

您可以如下指定数据源：

spark.read.format("greenplum")

使用Greenplum的数据直接jdbc驱动程序（greenplum.jar）读取greenplum表时，如何在spark.read.format中指定数据源？

1 个答案: