Question

我的应用程序以镶木地板格式写入AWS S3a。我的应用程序代码正在使用flink版本1.1-snapshot，除了快照之外，其版本1.1的抛出异常。

以下代码用于写入s3

Job job = Job.getInstance();
HadoopOutputFormat hadoopOutputFormat = new HadoopOutputFormat(new AvroParquetOutputFormat(), job);

FileOutputFormat.setOutputPath(job,getOutputPath(runDate));     
AvroParquetOutputFormat.setSchema(job, schema);
AvroParquetOutputFormat.setCompression(job, CompressionCodecName.SNAPPY);
AvroParquetOutputFormat.setEnableDictionary(job, true);
data.output(hadoopOutputFormat);

在设置架构AvroParquetOutputFormat.setSchema（作业，架构）时抛出以下异常;

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/ParquetRuntimeException
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.App.writeInS3(App.java:89)
    at com.App.execute(App.java:81)
    at com.App.main(App.java:53)
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.ParquetRuntimeException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 15 more

Answer 1

如果你在你的hadoop类路径中使用twitter groupId add

export HADOOP_CLASSPATH=~/.m2/repository/com/twitter/parquet-avro/1.5.0/parquet-avro-1.5.0.jar::~/.m2/repository/com/twitter/parquet-column/1.5.0/parquet-column-1.5.0.jar:~/.m2/repository/com/twitter/parquet-common/1.5.0/parquet-common-1.5.0.jar:~/.m2/repository/com/twitter/parquet-encoding/1.5.0/parquet-encoding-1.5.0.jar:~/.m2/repository/com/twitter/parquet-generator/1.5.0/parquet-generator-1.5.0.jar:~/.m2/repository/com/twitter/parquet-jackson/1.5.0/parquet-jackson-1.5.0.jar:~/.m2/repository/com/twitter/parquet-format/2.1.0/parquet-format-2.1.0.jar:~/.m2/repository/com/twitter/parquet-hadoop/1.5.0/parquet-hadoop-1.5.0.jar

或akk

中的其他Parquer依赖项

hadoop_classpath

AvroParquetOutputFormat setSchema方法抛出flink 1.1版的异常

1 个答案: