Question

尝试根据此处的文档（https://github.com/linkedin/WhereHows/blob/master/wherehows-docs/getting-started.md#build-the-source-code）在Centos7上安装Linkedin WhereHows，并在使用gradle构建时遇到与wherehows-common：test相关的构建错误。

> Task :wherehows-common:test
Putting task artifact state for task ':wherehows-common:test' into context took 0.001 secs.
Executing task ':wherehows-common:test' (up-to-date check took 0.041 secs) due to:
  No history is available.
Starting process 'Gradle Test Executor 2'. Working directory: /home/me/WhereHows/wherehows-common Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.7.1.201405082137.jar_r1v2o0qii4g3mwygeto49hza/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,dumponexit=true,output=file,jmx=false -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/me/.gradle/caches/4.0.2/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 2'
Successfully started process 'Gradle Test Executor 2'

Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
    hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }

Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs FAILED
    java.lang.AssertionError: expected [2] but found [47]
        at org.testng.Assert.fail(Assert.java:94)
        at org.testng.Assert.failNotEquals(Assert.java:513)
        at org.testng.Assert.assertEqualsImpl(Assert.java:135)
        at org.testng.Assert.assertEquals(Assert.java:116)
        at org.testng.Assert.assertEquals(Assert.java:389)
        at org.testng.Assert.assertEquals(Assert.java:399)
        at wherehows.common.utils.JobsUtilTest.testGetEnabledJobs(JobsUtilTest.java:91)
.....

更改所引用的JobsUtilTest.java（https://github.com/linkedin/WhereHows/blob/master/wherehows-common/src/test/java/wherehows/common/utils/JobsUtilTest.java#L77）中的testGetEnabledJobs（）函数看起来像（注意我添加了一些打印语句，以查看在运行测试时作业字典实际看到的内容）：

public void testGetEnabledJobs() throws IOException, ConfigurationException {
    String propertyStr1 = "job.class=test\n" + "job.cron.expr=0 0 1 * * ? *\n" + "#job.disabled=1\n" + "job.type=TEST1";
    String propertyStr2 = "job.class=test\n" + "#job.disabled=1\n";

    Path path1 = createPropertiesFile(propertyStr1);
    Path path2 = createPropertiesFile(propertyStr2);

    String filename1 = jobNameFromPath(path1);
    String filename2 = jobNameFromPath(path2);

    String dir = path1.getParent().toString();

    Map<String, Properties> jobs = getEnabledJobs(dir);

    System.out.println("Displaying jobs map");
    jobs.forEach((key, value) -> System.out.println(key + ":" + value));

    Assert.assertEquals(jobs.size(), 2);
    Assert.assertEquals(jobs.get(filename1).getProperty("job.class"), "test");
    Assert.assertEquals(jobs.get(filename1).getProperty("job.disabled", ""), "");
    Assert.assertEquals(jobs.get(filename2).getProperty("job.class"), "test");

    Files.deleteIfExists(path1);
    Files.deleteIfExists(path2);
  }

我看到输出

> Task :wherehows-common:test

Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
    hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }

Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs STANDARD_OUT
    temp1911939516358009277:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp6499017376697657088:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp5966987645065306263:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp2019798894975335038:{job.class=test}
    temp7378838424107087948:{job.class=test}
    temp8013153217448339252:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp4015785892016530294:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8886920588183826697:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp7303423395129181364:{job.class=test}
    temp7377443665642582850:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp8117994429105024533:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp1240481287927638671:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp7839672993673219735:{job.class=test}
    temp7259952657422506556:{job.class=test, job.type=TEST3}
    temp7162637183580461694:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp3707419465874500296:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp7235208969315338463:{job.class=test, job.type=TEST3}
    temp3853444188071964131:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp4656083272550419936:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp4446014595794536877:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp9150954469641724228:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp1130156704067895902:{job.class=test, job.type=TEST3}
    temp780755426601798559:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2434470881148405176:{job.class=test}
    temp3689536441531931750:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp5615428018432710013:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp8234020029970009016:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2936711116252973718:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2794065451765912556:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8170223748305820520:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8108053620028090977:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp7690999652582845774:{job.class=test, job.type=TEST3}
    temp8403527686775796761:{job.class=test}
    temp1264073914844226751:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp4824707996781445098:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp4743553641259724497:{job.class=test, job.type=TEST3}
    temp6976197742641904750:{job.class=test}
    temp1730302984164232378:{job.class=test, job.type=TEST3}
    temp1842348735636276811:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp1493041975452434193:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp7759939924299343315:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2004080563168505783:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp1339889112521909105:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp930690436274101959:{job.class=test, job.type=TEST3}
    temp8950010240587776997:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp1690669672245318742:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp7618814921091278264:{job.class=test}
    temp5153364346961309871:{job.class=test}
    temp2706126941897920939:{job.class=test, job.type=TEST3}
    temp7879422791151165293:{job.class=test, job.type=TEST3}
    temp3500152346117134646:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp749236937622369939:{job.class=test}
    temp2402271667647981403:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp3731569061193901687:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8046137399375181735:{job.class=test, job.type=TEST3}
    temp2787658903538848691:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp2549570320110302073:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp4717503488348299098:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp4558936186853777244:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp4977131264038603527:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp7320554126669992802:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8291109043177891111:{job.class=test}
    temp3078133305981963798:{job.class=test, job.type=TEST3}
    temp4497083049478812420:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp3224402460504056397:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8457758602058485743:{job.class=test, job.type=TEST3}
    temp1336150142768316636:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp577381353130877703:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp1077045599840249203:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp1960639765588269368:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp4407187188971325393:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2303367009745054129:{job.class=test, job.type=TEST3}
    temp8807915018871869160:{job.class=test}
    temp8784520992289559963:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2670455510663453671:{job.class=test}
    temp1910488462419291875:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp7195699649615422636:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp1493552070734553408:{job.class=test}
    temp9092541414990236719:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp6892241678814671661:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8422662429932217118:{job.class=test}
    temp7706138265443041124:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp5607003154298079581:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8785154796880304252:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp9219444719015202876:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
    temp1177833530220968687:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp8577396298139877100:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
    temp979103679152008854:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp2369961801884435918:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp3349449102692687060:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp430974220547583703:{job.class=test, job.type=TEST3}
    temp7799011812497499444:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}

Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs FAILED
    java.lang.AssertionError: expected [2] but found [92]
        at org.testng.Assert.fail(Assert.java:94)
        at org.testng.Assert.failNotEquals(Assert.java:513)
        at org.testng.Assert.assertEqualsImpl(Assert.java:135)
        at org.testng.Assert.assertEquals(Assert.java:116)
        at org.testng.Assert.assertEquals(Assert.java:389)
        at org.testng.Assert.assertEquals(Assert.java:399)
        at wherehows.common.utils.JobsUtilTest.testGetEnabledJobs(JobsUtilTest.java:93)

所以应该只有2个工作，但有一堆重复工作。这里似乎有什么问题？

是否有其他人使用此工具知道这里可能出现的问题？

**有用的系统和环境。信息：

[me@mapr07 WhereHows]$ tree -L 2 wherehows-etl/extralibs/
wherehows-etl/extralibs/
├── dist
│   ├── changes.txt
│   ├── compile_run_demo.txt
│   ├── demos
│   ├── GeneralSQLParser_UserGuide.pdf
│   ├── gsp.jar
│   ├── javadoc
│   └── test
├── gsp.jar
├── gsp_java_trial_1_9_2_3.zip
├── ojdbc7.jar
├── orai18n.jar
├── README.md
├── readme.txt
├── tdgssconfig.jar
├── terajdbc4.jar
└── TeraJDBC__indep_indep.16.20.00.06.tar

4 directories, 13 files

[me@mapr07 WhereHows]$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

[me@mapr07 WhereHows]$ javac -version
javac 1.8.0_171

[me@mapr07 WhereHows]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre

[me@mapr07 WhereHows]$ cat /proc/version
Linux version 3.10.0-693.21.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Wed Mar 7 19:03:37 UTC 2018

Answer 1

解决了这个问题。在测试函数的代码中添加一些print语句，如下所示：

 @Test
  public void testGetEnabledJobs() throws IOException, ConfigurationException {
    String propertyStr1 = "job.class=test\n" + "job.cron.expr=0 0 1 * * ? *\n" + "#job.disabled=1\n" + "job.type=TEST1";
    String propertyStr2 = "job.class=test\n" + "#job.disabled=1\n";

    Path path1 = createPropertiesFile(propertyStr1);
    Path path2 = createPropertiesFile(propertyStr2);

    String filename1 = jobNameFromPath(path1);
    String filename2 = jobNameFromPath(path2);
    System.out.println("filenames: " + filename1 + "\n" + filename2);

    String dir = path1.getParent().toString();
    System.out.println("dir="+dir);

    Map<String, Properties> jobs = getEnabledJobs(dir);

    System.out.println("Displaying jobs map");
    jobs.forEach((key, value) -> System.out.println(key + ":" + value));

    Assert.assertEquals(jobs.size(), 2);
    Assert.assertEquals(jobs.get(filename1).getProperty("job.class"), "test");
    Assert.assertEquals(jobs.get(filename1).getProperty("job.disabled", ""), "");
    Assert.assertEquals(jobs.get(filename2).getProperty("job.class"), "test");

    Files.deleteIfExists(path1);
    Files.deleteIfExists(path2);
  }

我看到了输出：

> Task :wherehows-common:test
Putting task artifact state for task ':wherehows-common:test' into context took 0.0 secs.
Executing task ':wherehows-common:test' (up-to-date check took 0.02 secs) due to:
  No history is available.
Starting process 'Gradle Test Executor 3'. Working directory: /home/rvillanueva/WhereHows/wherehows-common Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.7.1.201405082137.jar_r1v2o0qii4g3mwygeto49hza/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,dumponexit=true,output=file,jmx=false -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/rvillanueva/.gradle/caches/4.0.2/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 3'
Successfully started process 'Gradle Test Executor 3'

Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
    hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }

Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs STANDARD_OUT
    filenames: temp7961565693796243498
    temp1135071152526160059
    dir=/tmp
    Displaying jobs map
    temp7961565693796243498:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
    temp1135071152526160059:{job.class=test}
    ......

查看/ tmp目录。我看到了一堆旧的temp * .job文件。删除这些并重新运行似乎让我成功构建。

我认为这个问题是这样的：

最初运行了gradlew build命令，由于JAVA_HOME未设置而导致错误并退出。此时，我认为testGetEnabledJobs（）已经创建了其断言期望并成功传递的2个临时* .job文件。
然后设置JAVA_HOME并尝试重新构建。这次testGetEnabledJobs（）在/ tmp中创建了更多的temp * .job文件（在上一次构建尝试中未删除的文件之上）并得到了testGetEnabledJobs（）错误，因为现在/ tmp中有更多的temp * .job文件比预计会导致错误并退出。
此时，每次重建建筑都会显示断言计数越来越多预期值。

在任何情况下，我认为应该更改此测试以删除它创建的临时* .job文件，如果它在相关断言中出错，否则我们将在/ tmp目录中得到复合问题。我刚刚修好了。

WhereHows，wherehows-常见的构建错误（创建的作业太多）

1 个答案: