在Jar中运行map reduce作业与第三方依赖

Question

我正在尝试测试是否可以将第三方依赖项放入jar中，将jar连接到远程计算机，然后运行map reduce job。

过程：

通过我的项目，我运行mvn clean package并生成文件my-appy-1.0-SNAPSHOT.jar和original-my-appy-1.0-SNAPSHOT.jar。我将第一个文件scp到我的远程计算机并运行命令：

hadoop jar hadoop my-appy-1.0-SNAPSHOT.jar  /user/bli1/wordcount/input /user/bli1/wordcount/output

我也尝试过：

hadoop my-appy-1.0-SNAPSHOT.jar WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

我不确定为什么会收到此错误：

Error: Could not find or load main class my-appy-1.0-SNAPSHOT.jar

的pom.xml：

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.2</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <createDependencyReducedPom>false</createDependencyReducedPom>
              <artifactSet>
                <excludes>
                  <exclude>org.hamcrest:*</exclude>
                  <exclude>org.mockito:*</exclude>
                  <exclude>org.objenesis:*</exclude>
                </excludes>
              </artifactSet>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/LICENSE</exclude>
                    <exclude>META-INF/license</exclude>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <manifestEntries>
                    <Main-Class>com.mycompany.app.WordCount</Main-Class>
                    <Build-Number>1</Build-Number>
                  </manifestEntries>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Answer 1

打开* .jar文件并检查它的MANIFEST.mf文件。它应该包含“Main-Class”行，您的类使用“public static main（）”方法。如果没有“Main-Class”自己添加

此链接可帮助您执行此操作：https://docs.oracle.com/javase/tutorial/deployment/jar/appman.html

Answer 2

你需要提供主类的完整路径，包括包名。

hadoop my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

Answer 3

您通过jar文件运行的命令如下： hadoop jar jar_filename.jar package_name.class_name HDFS_inputfile_name HDFS_output_directory

你的命令应该是

hadoop jar my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount / user / bli1 / wordcount / input / user / bli1 / wordcount / output