我正在尝试将xml文件转换为avro,然后转换为镶木地板文件格式,而不使用大数据工具。我能够将其转换为avro,但此后出现错误:
线程“主”中的异常java.lang.ClassCastException:sun.nio.fs.UnixPath无法转换为org.apache.parquet.io.OutputFile 在Parquet.ConversionToParquet.main(ConversionToParquet.java:65)
下面是我的代码:
import java.io.ByteArrayOutputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.hadoop.conf.Configuration;
import org.apache.parquet.avro.AvroParquetWriter;
import org.apache.parquet.hadoop.ParquetWriter;
import org.apache.parquet.hadoop.metadata.CompressionCodecName;
import org.apache.parquet.io.OutputFile;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.xml.XMLConstants;
import javax.xml.bind.JAXBContext;
import javax.xml.validation.SchemaFactory;
public class ConversionToParquet {
private static final Logger LOGGER = LoggerFactory.getLogger(ConversionToParquet.class);
private static final Path inputPath = Paths.get("/home/lucky/output.avro");
private static final Path outputPath = Paths.get("/home/lucky/Desktop/sample.parquet");
public static void main(String[] args) throws Exception {
JAXBContext jaxbContext = JAXBContext.newInstance(ObjectFactory.class);
javax.xml.validation.Schema newSchema = SchemaFactory
.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI)
.newSchema(ConversionToParquet.class.getResource("/question.xsd"));
XmlSerializer<Question> xmlSerializer = new XmlSerializer<>(
jaxbContext, newSchema);
InputStream questionStream = ConversionToParquet.class.getResourceAsStream(
"/question.xml");
Question question = xmlSerializer.readFromXml(questionStream);
AvroSchemaGenerator schemaGenerator = new AvroSchemaGenerator();
Schema questionSchema = schemaGenerator.generateSchema(Question.class);
AvroSerializer<Question> avroSerializer = schemaGenerator.createAvroSerializer(questionSchema);
ByteArrayOutputStream avroByteStream = new ByteArrayOutputStream();
avroSerializer.writeToAvro(avroByteStream, question);
byte[] avroBytes = avroByteStream.toByteArray();
avroSerializer.writeToAvro(new FileOutputStream("/home/lucky/output.avro"), question);
System.out.println("File Converted to Avro");
try (ParquetWriter writer = AvroParquetWriter
.builder((OutputFile) outputPath)
.withSchema(questionSchema)
.withConf(new Configuration())
.withCompressionCodec(CompressionCodecName.SNAPPY)
.build()){
for (Path record : inputPath) {
writer.write(record);
}
}
System.out.println("File Convereted Successfully");
}
}
答案 0 :(得分:0)
此行中的错误:
.builder((OutputFile) outputPath)
最简单的解决方案是使用不推荐使用的builder
方法,该方法带有一个org.apache.hadoop.fs.Path
参数,该参数不同于当前正在导入的java.nio.file.Path
类。