在cloudera.quickstart虚拟机中,我正在学习oozie,并尝试通过sqoop操作运行简单的工作流程。 当我执行工作时,它似乎挂起了,我不知道那里发生了什么。 我没有注意到错误,还是缺少通知网址是问题所在?
2018-10-28 03:23:26,481 INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[] No results found
2018-10-28 03:23:26,575 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] Start action [0000003-181027074351697-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 03:23:26,576 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] [***0000003-181027074351697-oozie-oozi-W@:start:***]Action status=DONE
2018-10-28 03:23:26,576 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] [***0000003-181027074351697-oozie-oozi-W@:start:***]Action updated in DB!
2018-10-28 03:23:26,690 INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] No results found
2018-10-28 03:23:26,740 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000003-181027074351697-oozie-oozi-W@:start:
2018-10-28 03:23:26,740 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000003-181027074351697-oozie-oozi-W
2018-10-28 03:23:26,812 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@sqoop-node] Start action [0000003-181027074351697-oozie-oozi-W@sqoop-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
工作流程定义为
<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
<start to="sqoop-node"/>
<action name="sqoop-node">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/sqoop"/>
<mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<command>import --connect jdbc:mysql://localhost:3306/retail_db --username=retail_dba --password cloudera \
--table departments --as-textfile --target-dir /user/cloudera/departments</command>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
属性文件
nameNode=hdfs://localhost:8020
jobTracker=localhost:8023
queueName=default
examplesRoot=questions_oozie/q59
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/
添加:我创建了另一个oozie工作流,这次使用HUE和Pig脚本。因此,工作流是由HUE生成的,不应包含错误。我认为问题与我的集群配置有关?
2018-10-28 07:01:53,110 INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No results found
2018-10-28 07:01:53,240 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] Start action [0000007-181027074351697-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:01:53,244 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] [***0000007-181027074351697-oozie-oozi-W@:start:***]Action status=DONE
2018-10-28 07:01:53,244 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] [***0000007-181027074351697-oozie-oozi-W@:start:***]Action updated in DB!
2018-10-28 07:01:53,428 INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] No results found
2018-10-28 07:01:53,478 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@:start:
2018-10-28 07:01:53,479 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W
2018-10-28 07:01:53,564 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Start action [0000007-181027074351697-oozie-oozi-W@pig-ca9a] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:01:55,590 INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] checking action, hadoop job ID [job_1540651330075_0018] status [RUNNING]
2018-10-28 07:01:55,600 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] [***0000007-181027074351697-oozie-oozi-W@pig-ca9a***]Action status=RUNNING
2018-10-28 07:01:55,600 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] [***0000007-181027074351697-oozie-oozi-W@pig-ca9a***]Action updated in DB!
2018-10-28 07:01:55,666 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@pig-ca9a
2018-10-28 07:05:30,298 INFO CallbackServlet:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] callback for action [0000007-181027074351697-oozie-oozi-W@pig-ca9a]
2018-10-28 07:05:30,696 INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Hadoop Jobs launched : [job_1540651330075_0020,job_1540651330075_0021]
2018-10-28 07:05:30,701 INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] action completed, external ID [job_1540651330075_0018]
2018-10-28 07:05:30,725 WARN PigActionExecutor:523 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2018-10-28 07:05:30,772 INFO ActionEndXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] ERROR is considered as FAILED for SLA
2018-10-28 07:05:30,834 INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No results found
2018-10-28 07:05:30,875 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] Start action [0000007-181027074351697-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:05:30,875 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] [***0000007-181027074351697-oozie-oozi-W@Kill***]Action status=DONE
2018-10-28 07:05:30,875 INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] [***0000007-181027074351697-oozie-oozi-W@Kill***]Action updated in DB!
2018-10-28 07:05:31,069 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@Kill
2018-10-28 07:05:31,071 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W
2018-10-28 07:05:31,072 INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@pig-ca9a