Question

我试图通过sqoop从mysql导入数据到hdfs。我能够通过shell运行sqoop，它工作正常。但是当我通过oozie调用时。它出现以下错误并失败。我还包括罐子。

现在我在家里尝试这个 - 虚拟盒中的CDH 5.3单节点但错误仍然相同

我知道这已经讨论过了，但仍然没有帮助..

主要错误是：

ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

作业属性文件如下所示，存在于本地环境

中

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/harry/

Workflow.xml文件如下所示，该文件存在于$ {nameNode} / user / harry /

下的HDFS中

<workflow-app name="Trend_Analysi_Scoop_Script" xmlns="uri:oozie:workflow:0.4">
<start to="sqoop-c68d"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-c68d">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <arg>import</arg>
        <arg> --connect</arg>
        <arg>jdbc:mysql://localhost/oozie_training</arg>
        <arg>--username</arg>
        <arg>root</arg>
        <arg>--password</arg>
        <arg>cloudera</arg>
        <arg>--table</arg>
        <arg>user</arg>
        <arg>--target-dir</arg>
        <arg>/training/sqoop</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

我在下面的路径中包含了所有必要的罐子

[cloudera@quickstart ~]$ hadoop fs -ls /user/oozie/share/lib/lib_20141218070949/sqoop
Found 75 items
-rw-r--r--   1 root  oozie     960374 2016-05-26 22:00 /user/oozie/share/lib/lib_20141218070949/sqoop/mysql-connector-java-5.1.34-bin.jar
-rw-r--r--   1 oozie oozie      16924 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/oozie-sharelib-sqoop-4.0.0-cdh5.3.0.jar
-rw-r--r--   1 oozie oozie    1010790 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/sqoop-1.4.5-cdh5.3.0.jar

我在添加罐子后也重新启动了oozie服务器。

我得到的错误日志如下。

2016-05-24 10:37:17,994  INFO ActionStartXCommand:520 - SERVER[***]             USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] Start action [0000001-160524103133053-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action status=DONE
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action updated in DB!
2016-05-24 10:37:18,164  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@:start:
2016-05-24 10:37:18,165  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:18,188  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Start action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:26,080  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:26,106  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] checking action, hadoop job ID [job_1462295072181_0163] status [RUNNING]
2016-05-24 10:37:26,108  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action status=RUNNING
2016-05-24 10:37:26,114  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action updated in DB!
2016-05-24 10:37:26,184  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@sqoop-c68d
2016-05-24 10:37:37,181  INFO CallbackServlet:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] callback for action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d]
2016-05-24 10:37:37,314  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:37,388  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] action completed, external ID [job_1462295072181_0163]
2016-05-24 10:37:37,395  WARN SqoopActionExecutor:523 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2016-05-24 10:37:37,484  INFO ActionEndXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] ERROR is considered as FAILED for SLA
2016-05-24 10:37:37,679  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] Start action [0000001-160524103133053-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action status=DONE
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action updated in DB!
2016-05-24 10:37:37,930  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@Kill
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi- W@sqoop-c68d

我认为oozie没有拿起份额/ lib，如果有的话，如果有人可以帮我解决这个问题会非常好

任何帮助都将不胜感激。

Answer 1

看起来sqoop操作中的“command”标记存在一些问题。由于它已经是sqoop动作，因此不需要“/usr/hdp/2.4.0.0-169/sqoop/bin/sqoop”。请参阅以下链接中的示例以更正它们。并尝试重新恢复工作。

https://oozie.apache.org/docs/4.0.0/DG_SqoopActionExtension.html#Sqoop_Action

Answer 2

您正在使用--query元素中的command选项，而oozie不支持该选项。您需要使用arg标记并单独提供所有参数。 command标记由空格拆分，因此不适用于--query选项。以下是示例示例：

        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://host:3306/oozie</arg>
        <arg>--username</arg>
        <arg>user</arg>
        <arg>--password</arg>
        <arg>pwd</arg>
        <arg>--driver</arg>
        <arg>com.mysql.jdbc.Driver</arg>
        <arg>--query</arg>
        <arg>"select BUNDLE_ACTIONS.* from BUNDLE_ACTIONS where $CONDITIONS"</arg>
        <arg>--hcatalog-table</arg>
        <arg>bundle_actions</arg>
        <arg>--hcatalog-database</arg>
        <arg>oozie</arg>
        <arg>--skip-dist-cache</arg>
        <arg>-m</arg>
        <arg>1</arg>

Oozie错误 - 从mysql导入数据到hdfs

2 个答案: