Question

在猪中，我正在使用我的udf，它正在使用外部jar文件例程，而在eclipse中我也导出了该jar文件。但是在运行pig -x local script1.pig时，它会给我外部jar文件例程的错误。

请帮忙！

感谢。

编辑1：

正如我的代码评论中所述：

script1.pig:

    REGISTER ./csv2arff.jar;

    csvraw = LOAD 'sample' USING PigStorage('\n') as (c);

    arffraws = FOREACH csvraw GENERATE pighw2java.CSV2ARFF(c);

pighw2java.CSV2ARFF：

public String exec(Tuple input) throws IOException {
    if (input == null || input.size() == 0)
        return null;
    try{
            System.out.println(">>> " + input.get(0).toString());
            // 1.1) csv to instances
            ByteArrayInputStream inputStream = new ByteArrayInputStream(input.get(0).toString().getBytes("UTF-8"));
            CSVLoader loader = new CSVLoader();    **HERE IS ERROR**

                 .....

        }
}

我得到的错误：

java.lang.NoClassDefFoundError: weka/core/converters/CSVLoader
at pighw2java.CSV2ARFF.exec(CSV2ARFF.java:24)
at pighw2java.CSV2ARFF.exec(CSV2ARFF.java:1)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:216)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:305)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:322)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
    Caused by: java.lang.ClassNotFoundException: weka.core.converters.CSVLoader
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 14 more

Answer 1

您需要注册自定义UDF中使用的所有第三方依赖项。这里：

register '/path/to/weka.jar'

使用外部jar时为什么“没有类发现错误”（但我也导出了外部jar！）

1 个答案: