从SequenceFile读取时,Protocol Buffers中的一个字段总是丢失

时间:2017-09-16 21:37:19

标签: hadoop apache-spark protocol-buffers sequencefile java-pair-rdd

我身上发生了一件神秘事:

我想做什么:

1. Save a Protocol Buffers object as SequenceFile format.
2. Read this SequenceFile text and extract the field that I need.

神秘的部分是: 我想要检索的一个字段始终为空。

Product_Perf是我想从SequencFiles中提取的字段,总是丢失。

这是我的协议缓冲模式:

message ProductJoin {
Signals signals = 1;
int64 id = 2;
}

message Signals {
ProductPerf product_perf = 1;
}

message ProductPerf {
    int64 impressions = 1;
}

以下是我将协议缓冲区保存为SequenceFiles的方法:

JavaPairRDD<BytesWritable, BytesWritable> bytesWritableJavaPairRdd =
                flattenedPjPairRdd.mapToPair(
                    new PairFunction<Tuple2<Long, ProductJoin>, BytesWritable, BytesWritable>() {

                        @Override
                        public Tuple2<BytesWritable, BytesWritable> call(Tuple2<Long, ProductJoin> longProductJoinTuple2) throws Exception {
                                return new Tuple2<>(
new BytesWritable(longProductJoinTuple2._2().getId().getBytes()),
new BytesWritable(longProductJoinTuple2._2().toByteArray()));
                            }
                        }
    //dump SequenceFiles
                bytesWritableJavaPairRdd.saveAsHadoopFile(
                    "/tmp/path/",
                    BytesWritable.class,
                    BytesWritable.class,
                    SequenceFileOutputFormat.class
                );

以下是我如何阅读SequenceFile的代码:

 sparkSession.sparkContext()
            .sequenceFile("tmp/path", BytesWritable.class, BytesWritable.class)
            .toJavaRDD()
            .mapToPair(
                bytesWritableBytesWritableTuple2 -> {
                    Method parserMethod = clazz.getDeclaredMethod("parser");
                    Parser<T> parser = (Parser<T>) parserMethod.invoke(null);
                    return new Tuple2<>(
                        Text.decode(bytesWritableBytesWritableTuple2._1().getBytes()),
                        parser.parseFrom(bytesWritableBytesWritableTuple2._2().getBytes()));
                }
            );

0 个答案:

没有答案