我使用以下方法在镶木地板文件中写入数据:
static void toParquet() {
String schema = "message spark_schema {\n optional binary stringField (UTF8);\n optional INT96 createdAt;\n}";
MessageType readSchema = MessageTypeParser.parseMessageType(schema);
Configuration configuration = new Configuration();
GroupWriteSupport.setSchema(readSchema, configuration);
SimpleGroupFactory sfg = new SimpleGroupFactory(readSchema);
Path file = new Path("/home/user/data-" + System.currentTimeMillis() + ".parquet");
try {
ParquetWriter<Group> writer = new ParquetWriter<Group>(file, new GroupWriteSupport(), CompressionCodecName.UNCOMPRESSED, 1024, 1024, 512,
true, false, ParquetProperties.WriterVersion.PARQUET_1_0, configuration);
for (int i = 0; i < 10000; ++i) {
writer.write(sfg.newGroup().append("stringField", "abc").append("createdAt", String.valueOf(System.currentTimeMillis())));
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
但它给了我以下例外:
java.lang.IllegalArgumentException:固定二进制大小13没有 匹配字段类型长度12 at org.apache.parquet.column.values.plain.FixedLenByteArrayPlainValuesWriter.writeBytes(FixedLenByteArrayPlainValuesWriter.java:53) 在 org.apache.parquet.column.values.dictionary.DictionaryValuesWriter $ PlainFixedLenArrayDictionaryValuesWriter.createDictionaryPage(DictionaryValuesWriter.java:324) 在 org.apache.parquet.column.values.fallback.FallbackValuesWriter.createDictionaryPage(FallbackValuesWriter.java:102) 在 org.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:242) 在 org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:126) 在 org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:164) 在 org.apache.parquet.hadoop.InternalParquetRecordWriter.checkBlockSizeReached(InternalParquetRecordWriter.java:141) 在 org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123) 在 org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:288) 在com.app.test.SimpleParquet.toParquet(SimpleParquet.java:498)at com.app.test.SimpleParquet.main(SimpleParquet.java:63)
由于System.currentTimeMillis()给出13字节值,epoc时间为10字节,如何为当前时间戳提供int96值,这是一个固定长度的12字节数组。
我找到了一些线程,他们说时间戳是julian日和纳秒的时间组合,我如何将当前时间戳转换为这种格式?
答案 0 :(得分:0)
下面的代码对我有用,其中我已经计算了NanoTime,然后在其上调用toBinary函数。
public static NanoTime getNanoTime(String time) {
Timestamp ts = Timestamp.valueOf(time);
Calendar calendar = getCalendar();
calendar.setTime(ts);
JDateTime jDateTime = new JDateTime(calendar.get(Calendar.YEAR), calendar.get(Calendar.MONTH) + 1, calendar.get(Calendar.DAY_OF_MONTH));
int days = jDateTime.getJulianDayNumber();
long hour = calendar.get(Calendar.HOUR_OF_DAY);
long minute = calendar.get(Calendar.MINUTE);
long second = calendar.get(Calendar.SECOND);
long nanos = ts.getNanos();
long nanosOfDay = nanos + NANOS_PER_SECOND * second + NANOS_PER_SECOND * SECONDS_PER_MINUTE * minute + NANOS_PER_SECOND * SECONDS_PER_MINUTE
* MINUTES_PER_HOUR * hour;
return new NanoTime(days, nanosOfDay);
}
在上述方法中使用镶木地板书写时-
writer.write(sfg.newGroup().append("stringField", "abc").append("createdAt", getNanoTime("2017-05-23 11:59:43.345717").toBinary()));