尝试使用Oozie
工作流运行java action
协调员,该工作流包含运行Camus
映射器作业。协调器似乎运行,并且每20分钟启动一次工作流程,但工作流程将无限期地运行,即使独立运行的作业很容易在几分钟内完成。我认为错误或者与我如何运行作业或者如何传递参数有关,但我不确定如何调试它。这是代码:
/coord/job.properties
oozie.coord.application.path=hdfs://10.0.2.15:8020/user/hue/app/coord/coordinator.xml
name=camus
frequency=20
start=2015-07-30T11:40Z
end=2016-07-30T11:40Z
timezone=GMT+0530
workflow=hdfs://10.0.2.15:8020/user/hue/app/workflow/workflow.xml
nameNode=hdfs://10.0.2.15:8020
jobTracker=10.0.2.15:8021
queueName=default
properties=${nameNode}/user/hue/app/workflows/lib/config.properties
坐标/ coordinator.xml
<coordinator-app name="${name}" frequency="${frequency}" start="${start}" end="${end}" timezone="${timezone}" xmlns="uri:oozie:coordinator:0.1">
<action>
<workflow>
<app-path>${workflow}</app-path>
</workflow>
</action>
</coordinator-app>
/workflow/workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.4' name='camus-wf'>
<start to='camus_job' />
<action name='camus_job'>
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<main-class>com.linkedin.camus.etl.kafka.CamusJob</main-class>
<arg>-P</arg>
<arg>${properties}</arg>
</java>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>Camus Job Failed</message>
</kill>
<end name='end' />
</workflow-app>
SHADED jar和config.properties位于/ workflow / lib /
中我正在运行HDP 2.2
协调员日志:
2015-08-03 06:43:43,820 INFO CoordSubmitXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0000000-150803063131195-oozie-oozi-C
2015-08-03 06:43:43,935 INFO CoordMaterializeTransitionXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
start=Thu Jul 30 11:40:00 UTC 2015, end=Thu Jul 30 15:40:00 UTC 2015,
timeUnit 12,
frequency :20:MINUTE,
lastActionNumber 0
2015-08-03 06:43:43,971 INFO CoordMaterializeTransitionXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[-] [0000000-150803063131195-oozie-oozi-C]: Update status from PREP to RUNNING
2015-08-03 06:43:44,113 INFO CoordActionInputCheckXCommand:543 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[0000000-150803063131195-oozie-oozi-C@1] [0000000-150803063131195-oozie-oozi-C@1]::CoordActionInputCheck:: Missing deps:
2015-08-03 06:43:44,209 INFO CoordActionNotificationXCommand:543 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[0000000-150803063131195-oozie-oozi-C@1] STARTED Coordinator Notification actionId=0000000-150803063131195-oozie-oozi-C@1 : WAITING
...
2015-08-03 06:43:44,267 INFO CoordActionNotificationXCommand:543 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[0000000-150803063131195-oozie-oozi-C@12] No Notification URL is defined. Therefore nothing to notify for job 0000000-150803063131195-oozie-oozi-C action ID 0000000-150803063131195-oozie-oozi-C@12
2015-08-03 06:43:44,268 INFO CoordActionNotificationXCommand:543 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[0000000-150803063131195-oozie-oozi-C@12] ENDED Coordinator Notification actionId=0000000-150803063131195-oozie-oozi-C@12
2015-08-03 06:43:44,433 WARN ParameterVerifier:546 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150803063131195-oozie-oozi-C] ACTION[0000000-150803063131195-oozie-oozi-C@1] The application does not define formal parameters in its XML definition
...
工作流程日志:
2015-08-03 06:43:44,672 INFO ActionStartXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus-wf] JOB[0000001-150803063131195-oozie-oozi-W] ACTION[0000001-150803063131195-oozie-oozi-W@:start:] Start action [0000001-150803063131195-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-08-03 06:43:44,673 INFO ActionStartXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus-wf] JOB[0000001-150803063131195-oozie-oozi-W] ACTION[0000001-150803063131195-oozie-oozi-W@:start:] [***0000001-150803063131195-oozie-oozi-W@:start:***]Action status=DONE
2015-08-03 06:43:44,673 INFO ActionStartXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus-wf] JOB[0000001-150803063131195-oozie-oozi-W] ACTION[0000001-150803063131195-oozie-oozi-W@:start:] [***0000001-150803063131195-oozie-oozi-W@:start:***]Action updated in DB!
2015-08-03 06:43:45,104 INFO ActionStartXCommand:543 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[camus-wf] JOB[0000001-150803063131195-oozie-oozi-W] ACTION[0000001-150803063131195-oozie-oozi-W@camus_job] Start action [0000001-150803063131195-oozie-oozi-W@camus_job] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]