Question

我是新来的火花。我需要在oozie中运行一个火花工作。个别我能够运行火花工作，但在工作启动后使用oozie我收到以下错误：

017-01-12 13:51:57,696 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
    at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)

Spark版本：spark-1.5.2-bin-hadoop2.6 Hadoop：hadoop-2.6.2 Hbase： hbase-1.1.5 Oozie：oozie-4.2.0

我的pom.xml的快照是：

<dependency>
   <groupId>org.apache.zookeeper</groupId>
   <artifactId>zookeeper</artifactId>
   <version>3.4.8</version>
   <type>pom</type>
</dependency>
<dependency>
   <groupId>org.apache.hbase</groupId>
   <artifactId>hbase-common</artifactId>
   <version>1.1.5</version>
   <exclusions>
      <exclusion>
         <groupId>org.slf4j</groupId>
         <artifactId>slf4j-log4j12</artifactId>
      </exclusion>
   </exclusions>
</dependency>

<dependency>
   <groupId>org.apache.hbase</groupId>
   <artifactId>hbase-client</artifactId>
   <version>1.1.5</version>
   <exclusions>
      <exclusion>
         <groupId>org.slf4j</groupId>
         <artifactId>slf4j-log4j12</artifactId>
      </exclusion>
   </exclusions>
</dependency>

<dependency>
   <groupId>org.apache.hbase</groupId>
   <artifactId>hbase-server</artifactId>
   <version>1.1.5</version>
   <exclusions>
      <exclusion>
         <groupId>org.slf4j</groupId>
         <artifactId>slf4j-log4j12</artifactId>
      </exclusion>
   </exclusions>
</dependency>
<dependency>
   <groupId>org.apache.hbase</groupId>
   <artifactId>hbase-testing-util</artifactId>
   <version>1.1.5</version>
</dependency>
<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.11</artifactId>
   <version>1.5.2</version>
   <exclusions>
      <exclusion>
         <artifactId>javax.servlet</artifactId>
         <groupId>org.eclipse.jetty.orbit</groupId>
      </exclusion>
   </exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-sql_2.11</artifactId>
   <version>1.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn_2.10 -->
<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-yarn_2.11</artifactId>
   <version>1.5.2</version>

</dependency>


<!-- https://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core -->
<dependency>
   <groupId>org.mongodb.mongo-hadoop</groupId>
   <artifactId>mongo-hadoop-core</artifactId>
   <version>1.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-common</artifactId>
   <version>2.6.2</version>
   <exclusions>
      <exclusion>
         <artifactId>servlet-api</artifactId>
         <groupId>javax.servlet</groupId>
      </exclusion>
      <exclusion>
         <artifactId>jetty-util</artifactId>
         <groupId>org.mortbay.jetty</groupId>
      </exclusion>
      <exclusion>
         <artifactId>jsp-api</artifactId>
         <groupId>javax.servlet.jsp</groupId>
      </exclusion>
   </exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-client</artifactId>
   <version>2.6.2</version>
   <exclusions>
      <exclusion>
         <artifactId>jetty-util</artifactId>
         <groupId>org.mortbay.jetty</groupId>
      </exclusion>
   </exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-mapreduce-client-core</artifactId>
   <version>2.6.2</version>
</dependency>
<dependency>
   <groupId>org.mongodb</groupId>
   <artifactId>mongo-java-driver</artifactId>
   <version>3.2.1</version>
</dependency>
<!-- hadoop dependency -->
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-core</artifactId>
   <version>1.2.1</version>
   <exclusions>
      <exclusion>
         <artifactId>jetty-util</artifactId>
         <groupId>org.mortbay.jetty</groupId>
      </exclusion>
   </exclusions>
</dependency>

直到现在我已经搜索了几个博客。我从阅读这些博客中了解到的是，存在一些嵌入spark-assembly-1.5.2-hadoop2.6.0.jar的tachyon jar存在问题。我尝试从oozie的共享库中删除tachyon-0.5.0.jar tachyon-client-0.5.0.jar（存在于spark库下）但是后来我开始收到错误：

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.spark.util.Utils$.DEFAULT_DRIVER_MEM_MB()I
java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.DEFAULT_DRIVER_MEM_MB()I

请帮我调试并解决它。

使用oozie TFS FileSystem实现错误的Spark作业

0 个答案: