Question

我正在使用的火花是2.3。

我有这个代码片段，可读取“ hdfspath”下的序列文件（此路径下约有20个文件，每个文件约60MB），

SparkSession spark = ...;
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(spark.sparkContext());
JavaPairRDD<BytesWritable, BytesWritable> temp = jsc.sequenceFile(hdfspath, BytesWritable.class, BytesWritable.class);
temp.take(1);

这给了我这个错误，

19/04/03 14:50:18 INFO CodecPool: Got brand-new decompressor [.gz]
19/04/03 14:50:18 INFO CodecPool: Got brand-new decompressor [.gz]
19/04/03 14:50:18 INFO CodecPool: Got brand-new decompressor [.gz]
19/04/03 14:50:18 INFO CodecPool: Got brand-new decompressor [.gz]
19/04/03 14:50:18 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NegativeArraySizeException
    at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
    at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
    at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
    at org.apache.hadoop.io.SequenceFile$Reader.deserializeKey(SequenceFile.java:2606)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2597)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:277)

我要读取的hdfs文件是具有这样输出设置的旧mapreduce作业的输出，

job.setOutputKeyClass(BytesWritable.class);
job.setOutputValueClass(BytesWritable.class);
job.setOutputFormatClass(SequenceFileAsBinaryOutputFormat.class);
SequenceFileAsBinaryOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK);

我查看了org.apache.hadoop.io.BytesWritable.setCapacity（...）方法，

public void setSize(int size) {
if (size > getCapacity()) {
  setCapacity(size * 3 / 2);
}
this.size = size;
}

以某种方式，size参数为808464432，并且在执行size * 3时导致溢出，最终导致NegativeArraySizeException。

任何人都可以帮助解释这种情况如何发生以及如何解决吗？

Answer 1

想通了。使用JavaSparkContext#newAPIHadoopFile代替JavaSparkContext#sequenceFile。

序列文件时出现Spark，NegativeArraySizeException

1 个答案: