Question

研究1周后，不得不提出以下要求：

环境：Azure HDInsight
Oozie版本：“ Oozie客户端内部版本：4.2.2.0.2.5.3004-13”
火花：Spark2
我的程序：简单的Scala程序读取一个文件i.csv，并将其写入o.csv
经过Spark-Submit测试：是

job.properties

nameNode=wasb://mycontainer@something.blob.core.windows.net
jobTracker=hn0-something.internal.cloudapp.net:8050
master=yarn-cluster
queueName=default
deployed_loc=zs_app
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/${deployed_loc}

workflow.xml：

<workflow-app xmlns='uri:oozie:workflow:0.3' name='zs-wf'>
    <start to="Loader" />
    <action name="Loader">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
               <delete path="${nameNode}/${deployed_loc}/output-data"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
            </configuration>
            <master>${master}</master>
            <mode>cluster</mode>
            <name>Spark-Loader</name>
            <class>zs.test</class>
            <jar>${nameNode}/${deployed_loc}/zs_app.jar</jar>                        
            <arg>--testId=1</arg>            
        </spark>
            <ok to="end" />
            <error to="fail" />
            </action>
            <kill name="fail">
            <message>Workflow failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}] </message>
            </kill>
            <end name='end' />
</workflow-app>

我得到以下例外：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:556)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:338)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:204)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:674)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:67)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:672)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 17 more

我得出以下结论：

在Spark会话的更高版本中引入Spark会话时，它如何指向
oozie也可以提交作业，因为我使用“ yarn logs -applicationId appid”提取了此错误，我从oozie日志中获取了appid。

现在，如果我在job.properties中添加此行

oozie.action.sharelib.for.spark=spark2

我得到以下例外：

JOB[0000115-181216154825160-oozie-oozi-W] ACTION[0000115-181216154825160-oozie-oozi-W@Loader] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2308)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:229)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2214)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2306)
    ... 9 more

我得出以下结论：

Oozie无法提交作业，因为我在oozie日志本身上发现了错误。

我不明白为什么这一定要这么复杂，如果Microsoft Azure将HDInsight与spark2，oozie打包在一起……这件事应该运行平稳或进行较小的更改，就应该在某处提供干净的文档。

Answer 1

尝试在job.properties中设置您的oozie共享库路径。例如我的是：

oozie.libpath=/user/oozie/share/lib/lib_20180312160954

虽然不确定在蔚蓝环境中的位置。

Answer 2

假设您已经使用了HDInsight 3.6，请在HDInsight 4.0环境中尝试使用Spark2与oozie。早期版本在使用oozie时似乎无法直接使用Spark2。

HDInsight 4.0使用HDP 3.0。这可能会有所帮助。 Spark2 with Oozie in HDP3.0

使用Ozzie，Azure HDInsight和Spark2的ClassNotFound

2 个答案: