Question

我正在尝试设置猪单元测试，我正在查看他们提供的文档。它似乎有点过时，所以我切换到svn主干。第一个奇怪的事实是它实际上需要更多的库，不仅是pigunit，pig和hadoop-commons才能工作（添加hadoop-hdfs，hadoop-mapreduce-client-core，hadoop-mapreduce-client-jobclient）。我不确定在我的依赖管理器中有这些是好事但这不是主要问题。所以这是我试图执行的测试：

 @Test
public void testNtoN() throws ParseException, IOException {
    String[] args = {
                    "n=3",
                    "reducers=1",
                    "input=top_queries_input_data.txt",
                    "output=top_3_queries",
    };
    test = new PigTest("script dir", args);

    String[] output = {
                    "(yahoo,25)",
                    "(facebook,15)",
                    "(twitter,7)",
    };

    test.assertOutput("queries_limit", output);
}

这是实际的脚本：

 data =
     LOAD '$input'
     AS (query:CHARARRAY, count:INT);

 queries_group = 
     GROUP data 
     BY query
     PARALLEL $reducers;

 queries_sum = 
     FOREACH queries_group 
     GENERATE 
         group AS query, 
         SUM(data.count) AS count;

 queries_ordered = 
     ORDER queries_sum 
     BY count DESC
     PARALLEL $reducers;

 queries_limit = LIMIT queries_ordered $n;

 STORE queries_limit INTO '$output';

这是堆栈跟踪：

 STORE queries_limit INTO 'top_3_queries';
 --> none

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias queries_limit

 at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
 at org.apache.pig.pigunit.PigTest.getAliasFromCache(PigTest.java:224)
 at org.apache.pig.pigunit.PigTest.getActualResults(PigTest.java:319)
 at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:409)
 at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:400)
 at BlaUnitTest.testBla(BlaUnitTest.java:24)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
 at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
 at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
 at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.io.IOException: Couldn't retrieve job.
 at org.apache.pig.PigServer.store(PigServer.java:1083)
 at org.apache.pig.PigServer.openIterator(PigServer.java:994)
 ... 34 more

我尝试调试它以查看实际发生的情况，当它尝试构建查询计划并获取ExecJob时会发生这种情况，但我无法弄明白。我甚至尝试简化脚本并删除除加载和存储数据的代码之外的所有内容。结果是一样的。

Answer 1

我成功地解决了这个问题。问题是我在类路径中包含了一些依赖项，这似乎搞乱了正确的执行。唯一需要的依赖是hadoop-core（我使用hadoop-aws，因为我将它与aws一起使用），hadoop-client，pig和pigunit。所以现在一切都正常运行。

运行pigunit样品测试时出现奇怪错误

1 个答案: