Question

在火花流媒体作业中，我使用下面的代码片段将我的rdd数据保存到Hadoop的HDFS中的镶木地板文件中：

readyToSave.foreachRDD((VoidFunction<JavaPairRDD<Void, MyProtoRecord>>) rdd -> {          
            Configuration configuration = rdd.context().hadoopConfiguration();
            Job job = Job.getInstance(configuration);
            ParquetOutputFormat.setWriteSupportClass(job, ProtoWriteSupport.class);
            ProtoParquetOutputFormat.setProtobufClass(job, MyProtoRecord.class);
            rdd.saveAsNewAPIHadoopFile("path-to-hdfs", Void.class, MyProtoRecord.class, ParquetOutputFormat.class, configuration);
    });

我在下面得到例外：

java.lang.NullPointerException: writeSupportClass should not be null
at parquet.Preconditions.checkNotNull(Preconditions.java:38)
at parquet.hadoop.ParquetOutputFormat.getWriteSupport(ParquetOutputFormat.java:326)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:272)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1112)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1095)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

我该如何解决这个问题？

Answer 1

发现问题！在调用“ saveAsNewAPIHadoopFile（）方法时，您可以指定作业的配置（ job.getConfiguration（））：

readyToSave.foreachRDD((VoidFunction<JavaPairRDD<Void, MyProtoRecord>>) rdd -> {
            Configuration configuration = rdd.context().hadoopConfiguration();
            Job job = Job.getInstance(configuration);
            ParquetOutputFormat.setWriteSupportClass(job, ProtoWriteSupport.class);
            ProtoParquetOutputFormat.setProtobufClass(job, MyProtoRecord.class);
            rdd.saveAsNewAPIHadoopFile("path-to-hdfs", Void.class, MyProtoRecord.class, ParquetOutputFormat.class, job.getConfiguration());
    });

java.lang.NullPointerException：在spark流媒体作业中编写镶木地板文件时，writeSupportClass不应为null

1 个答案: