java.util.NoSuchElementException的修补程序是什么：运行Spark应用程序将数据移入HDFS时出现的None.get？

时间：2019-02-05 14:45:42

标签： apache-spark apache-spark-sql greenplum

我正在尝试使用greenplum spark连接器jar：greenplum-spark_2.11-1.3.0.jar将数据从GP迁移到Hive。 我读下表如下：

  val conf = new SparkConf().setAppName("Data_Read").set("spark.executor.heartbeatInterval", "1200s").set("spark.network.timeout", "12000s").set("spark.sql.inMemoryColumnarStorage.compressed", "true").set("spark.shuffle.compress", "true").set("spark.shuffle.spill.compress", "true").set("spark.sql.orc.filterPushdown", "true").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.max", "512m").set("spark.serializer", classOf[org.apache.spark.serializer.KryoSerializer].getName).set("spark.streaming.stopGracefullyOnShutdown", "true").set("spark.yarn.driver.memoryOverhead", "8192").set("spark.yarn.executor.memoryOverhead", "8192").set("spark.dynamicAllocation.enabled", "false").set("spark.shuffle.service.enabled", "true").set("spark.sql.tungsten.enabled", "true").set("spark.executor.instances", "1").set("spark.executor.memory", "1g").set("spark.executor.cores", "1").set("spark.files.maxPartitionBytes", "268435468").set("spark.sql.shuffle.partitions", "400")
  try {
    Class.forName("io.pivotal.greenplum.spark.GreenplumRelationProvider").newInstance()
  }
  catch {
    case cnf: ClassNotFoundException =>
      println("No class def found. Killing the application...." + cnf.printStackTrace())
      System.exit(1)
    case e: Exception =>
      println("No class def found. Killing the application...." + e.printStackTrace())
      System.exit(1)
  }
  val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
  val yearDF = spark.read.format("io.pivotal.greenplum.spark.GreenplumRelationProvider").option("url", "url").option("server.port","port").option("dbtable", "tablename").option("dbschema","schema").option("user", "userID").option("password", "12345678").option("partitionColumn","headers").option("partitions",64).load().where("period_year=2017 and period_num=12 and source_system_name='BANKERS'").select(splitSeq map col:_*).withColumn("delCol", lit(0))

  dataDF.write.format("csv").save("hdfs://location_of_the_file_on_HDFS")

我看到下面的查询也在打GP。

INSERT INTO "schema"."spark_c04d88c2015291d9_38ee758b9192317b_driver_1016" 
(creation_date,attribute50,headers,attribute12,drill_line_pk_id,sap_partner_func_area,attribute38,global_attribute2,global_attribute4,usd_mor_activity_amount,attribute5,journal_line,drill_debug_info,reference_8,last_updated_by,sap_reference,document_type,attribute47,attribute48,last_update_tms,created_by_name) 
SELECT creation_date,attribute50,headers,attribute12,drill_line_pk_id,sap_partner_func_area,attribute38,global_attribute2,global_attribute4,usd_mor_activity_amount,attribute5,journal_line,drill_debug_info,reference_8,last_updated_by,sap_reference,document_type,attribute47,attribute48,last_update_tms,created_by_name 
FROM "schema"."table" WHERE ("period_year" IS NOT NULL AND "period_num" IS NOT NULL AND "period_num" = 12 AND "source_system_name" = 'BANKERS' AND "period_year" = 2017) 
AND (gp_segment_id = 48)

但是该作业运行了一段时间，并且没有任何适当的结果/进度，但以java.util.NoSuchElementException: None.get为例外。完整的异常堆栈消息可以在下面看到。我们的greenplum数据库中有64个细分，因此它创建了64个任务，如下所示。 

Exception Message:
[Stage 5:> (0 + 48) / 64]18/12/27 10:29:10 WARN TaskSetManager: Lost task 6.0 in stage 5.0 (TID 11, executor 11): java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:347)
    at scala.None$.get(Option.scala:345)
    at io.pivotal.greenplum.spark.jdbc.Jdbc$.copyTable(Jdbc.scala:43)
    at io.pivotal.greenplum.spark.externaltable.GreenplumRowIterator.liftedTree1$1(GreenplumRowIterator.scala:110)
    at io.pivotal.greenplum.spark.externaltable.GreenplumRowIterator.<init>(GreenplumRowIterator.scala:109)
    at io.pivotal.greenplum.spark.GreenplumRDD.compute(GreenplumRDD.scala:49)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

已使用火花提交：

SPARK_MAJOR_VERSION=2 spark-submit --class com.partition.source.PartitionData --master=yarn --conf spark.ui.port=4090 --driver-class-path /home/devuser/jars/greenplum-spark_2.11-1.3.0.jar:/home/devuser/jars/postgresql-42.1.4.jar --conf spark.jars=/home/devuser/jars/greenplum-spark_2.11-1.3.0.jar,/home/devuser/jars/postgresql-42.1.4.jar --executor-cores 3 --executor-memory 13G --keytab /home/devuser/devuser.keytab --principal devuser@DEVENV.COM --files /usr/hdp/spark2-client/conf/hive-site.xml,testconnection.properties --name Splinter --conf spark.executor.extraClassPath=/home/devuser/jars/greenplum-spark_2.11-1.3.0.jar splinter_2.11-0.1.jar

jar文件：greenplum-spark_2.11-1.3.0.jar使用gpfdist协议。我正在使用greenplum关键documentation中提到的驱动程序类。 我已尽力调试该问题，但我不知道问题出在哪里。 有人可以让我知道我在这里做什么错误吗？

1 个答案:

答案 0 :(得分：0)

您使用的是哪个版本的greenplum？另外，您是否尝试过GP Spark Connector的最新版本（v.1.6）？最新版本提高了内存使用率，并改善了分区的创建方式。

我注意到您的DF使用了partitions选项，该选项仅在1.6版中引入。版本1.3中受支持的选项是partitionsPerSegment，默认值为1。您可以尝试将partitionsPerSegment的值设置为2，看看是否有任何区别。

val yearDF = spark.read
.format("io.pivotal.greenplum.spark.GreenplumRelationProvider")
...
.option("partitions", 64)
.load()...

希望此信息对您有帮助，请尝试上面的建议或Greenplum Spark连接器的最新版本（1.6）。

相关问题
最新问题