Question

需要一些建议我试图在oozie中运行sqoop工作，但突然它被杀了，oozie-error.log中有这个警告

2018-01-21 17:30:12,473  WARN SqoopActionExecutor:523 - SERVER[edge01.domain.com] USER[linknet] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000006-180121122345026-oozie-link-W] ACTION[0000006-180121122345026-oozie-link-W@sqoop-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

job.properties

nameNode=hdfs://hadoop01.domain.com:8020
jobTracker=hadoop01.domain.com:18032
queueName=default
oozie.use.system.libpath=true
examplesRoot=examples
oozie.libpath=${nameNode}/share/lib/oozie
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/sqoop

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
    <start to="sqoop-node"/>

    <action name="sqoop-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/sqoop"/>
                <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <command>job --exec ingest_cpm_alarm</command>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

这就是我创建sqoop job ingest_cpm_alarm

的方法

$ sqoop job --create ingest_cpm_alarm -- import --connect jdbc:postgresql://xxx.xxx.xxx.xxx:5432/snapshot --username "extractor" -P \
--incremental append \
--check-column snapshot_date \
--table cpm_snr_history \
--as-avrodatafile \
--target-dir /tmp/trash/cpm_alarm

我可以成功运行此sqoop作业，但不能在Oozie调度程序中运行。另外，jar文件postgresql-42.1.4.jar和$ SQOOP_HOME / lib下的所有内容都已复制到libpath目录（/ share / lib / oozie）。

Oozie和sqoop位于同一台服务器上。在我的sqoop-site.xml中，我只设置了这些参数。

sqoop.metastore.client.enable.autoconnect=true
sqoop.metastore.client.record.password=true
sqoop.metastore.client.record.password=true

我在这里错过了什么吗？

Answer 1

它已解决，我错过了应该在HDFS中的同一个工作流目录中可用的sqoop-site.xml。

这篇文章有类似的问题： sqoop exec job in oozie is not working

感谢。

无法运行＆＃34; sqoop job --exec＆＃34;在oozie

1 个答案: