运行Oozie工作流时出现异常

时间:2016-03-22 20:03:58

标签: hadoop oozie hue oozie-coordinator google-cloud-dataproc

我正在尝试运行一个我在Hue中创建的简单工作流程,它只调用没有参数的Spark作业,但在运行它时会遇到异常。这是在Google DataProc群集上运行,并使用带有Hue 3.9的Oozie 4.2.0进行用户界面并运行它。

我确保mapreduce.framework.name适用于纱线。到目前为止,我在网上找到了一些解决方案,但所有解决方案都围绕着工作流XML的问题。我已经手工重建了,它也是这样做的。

以下是我的工作流程的代码:

<workflow-app name="Drug_File_Price_Generation" xmlns="uri:oozie:workflow:0.5">
<start to="spark-22ca"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-22ca">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>local[*]</master>
        <mode>client</mode>
        <name>MySpark</name>
          <class>com.company.drug.GenerateDrugPriceFile</class>
        <jar>gs://customers/jar/DrugFileImport-assembly-1.0.jar</jar>
    </spark>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

以下是我收到的日志的输出:

2016-03-22 19:27:27,927  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] Start action [0000011-160322181838855-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action status=DONE

2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action updated in DB!

2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W

2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W@:start:

2016-03-22 19:27:27,976  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

2016-03-22 19:27:32,670  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
    at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
    at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
    at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
    ... 10 more

编辑: 这里要求的是我的mapred-site.xml文件:

<?xml version="1.0" ?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.job.maps</name>
    <value>20</value>
    <description>
      The default number of map tasks per job. Ignored when mapred.job.tracker
      is &amp;amp;amp;quot;local&amp;amp;amp;quot;.
    </description>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>3072</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>cluster-1-m:10020</value>
    <description>MapReduce JobHistory Server IPC host:port</description>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>cluster-1-m:19888</value>
    <description>MapReduce JobHistory Server Web UI host:port</description>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>6144</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx4915m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.tasktracker.map.tasks.maximum</name>
    <value>4</value>
    <description>
      Property from MapReduce version 1 still used for TeraGen sharding.
    </description>
  </property>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.input.fileinputformat.list-status.num-threads</name>
    <value>20</value>
    <description>
      The number of threads to use to list and fetch block locations for the
      specified input paths. Note: multiple threads should not be used if a
      custom non thread-safe path filter is used. Setting a larger value than
      the default of 1 can significantly improve job startup overhead,
      especially if using GCS as input with multi-level directories, such
      as in partitioned Hive tables.
    </description>
  </property>
  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx4915m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
    <value>2</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>2</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>6144</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.job.reduces</name>
    <value>8</value>
    <description>
      The default number of reduce tasks per job. Typically set to 99%of the
      cluster's reduce capacity, so that if a node fails the reduces can still
      be executed in a single wave. Ignored when mapred.job.tracker is
      &amp;amp;amp;quot;local&amp;amp;amp;quot;.
    </description>
  </property>
  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx2457m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.job.working.dir</name>
    <value>/user/${user.name}</value>
    <description>
      The FileSystem working directory to use for relative paths.
    </description>
  </property>
  <property>
    <name>mapreduce.fileoutputcommitter.algorithm.version</name>
    <value>2</value>
    <description>
      Updated file output committer algorithm in Hadoop 2.7+. Significantly
      improves commitJob times when using the Google Cloud Storage connector.
      See https://issues.apache.org/jira/browse/MAPEDUCE-4815 for more details.
    </description>
  </property>
  <property>
    <name>mapred.local.dir</name>
    <value>/hadoop/mapred/local</value>
    <description>
      Directories on the local machine in which to store mapreduce temp files.
    </description>
  </property>
  <property>
    <name>mapred.bq.project.id</name>
    <value>instant-bonbon-111304</value>
    <description>
      Google Cloud Project ID to use for BigQuery operations.
    </description>
  </property>
  <property>
    <name>mapred.bq.output.buffer.size</name>
    <value>67108864</value>
    <description>
      The size in bytes of the output buffer to use when writing to BigQuery.
    </description>
  </property>
  <property>
    <name>mapred.bq.gcs.bucket</name>
    <value>dataproc-efab4f81-9a3a-4e62-ae58-db967541284c-us</value>
    <description>
      The GCS bucket holding temporary BigQuery data for the input connector.
    </description>
  </property>
</configuration>

编辑2: 无法获取Hue导出job.properties文件,但这是我在命令行上测试时生成的文件。我确信生成的不会太差异。

nameNode=hdfs://localhost:8020
jobTracker=localhost:8050
master=local[*]
oozie.wf.application.path=/home/sayle_matthews/workflow
oozie.use.system.libpath=true

0 个答案:

没有答案