Question

我正在尝试一个sqoop shell操作。Shell脚本在本地边缘节点上运行良好，但是当它在oozie中运行时，显示未找到错误sqoop命令。

NM_AUX_SERVICE_spark_shuffle=:
LOCAL_USER_DIRS=/data/disk1/hadoop/yarn/local/usercache/206638483/,/data/disk2/hadoop/yarn/local/usercache/206638483/,/data/disk3/hadoop/yarn/local/usercache/206638483/,/data/disk4/hadoop/yarn/local/usercache/206638483/,/data/disk5/hadoop/yarn/local/usercache/206638483/,/data/disk6/hadoop/yarn/local/usercache/206638483/,/data/disk7/hadoop/yarn/local/usercache/206638483/,/data/disk8/hadoop/yarn/local/usercache/206638483/,/data/disk9/hadoop/yarn/local/usercache/206638483/,/data/disk10/hadoop/yarn/local/usercache/206638483/,/data/disk11/hadoop/yarn/local/usercache/206638483/,/data/disk12/hadoop/yarn/local/usercache/206638483/,/data/disk13/hadoop/yarn/local/usercache/206638483/,/data/disk14/hadoop/yarn/local/usercache/206638483/,/data/disk15/hadoop/yarn/local/usercache/206638483/,/data/disk16/hadoop/yarn/local/usercache/206638483/,/data/disk17/hadoop/yarn/local/usercache/206638483/,/data/disk18/hadoop/yarn/local/usercache/206638483/,/data/disk19/hadoop/yarn/local/usercache/206638483/,/data/disk20/hadoop/yarn/local/usercache/206638483/,/data/disk21/hadoop/yarn/local/usercache/206638483/,/data/disk22/hadoop/yarn/local/usercache/206638483/,/data/disk23/hadoop/yarn/local/usercache/206638483/,/data/disk24/hadoop/yarn/local/usercache/206638483/:
HADOOP_HOME=/usr/hdp/2.6.4.125-1/hadoop:
OOZIE_ACTION_CONF_XML=/data/disk1/hadoop/yarn/local/usercache/206638483/appcache/application_1559931840942_7387/container_e45_1559931840942_7387_01_000002/action.xml:
SHLVL=1:
HOME=/home/:
NM_AUX_SERVICE_spark2_shuffle=:
CONTAINER_ID=container_e45_1559931840942_7387_01_000002:
MALLOC_ARENA_MAX=4:
=================================================================

>>> Invoking Shell command line now >>

Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput 
Exit code of the Shell command 0
<<< Invocation of Shell command completed <<<


<<< Invocation of Main class completed <<<

Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@5a4bef8 to null

Oozie Launcher ends

Workflow.xml是：

   ?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app 
   xmlns="uri:oozie:workflow:0.5" name="Edm_Sqoop_full">
    <start to="shell_1"/>
    <action name="shell_1">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.use.system.libpath</name>
                    <value>true</value>
                </property>
            </configuration>
            <exec>/user/sparkcronsvc/edm/sqoop_full.sh</exec>
            <argument>uso12smktdb01</argument>
            <argument>unica</argument>
            <file>/user/sparkcronsvc/edm/sqoop_full.sh</file>
            <file>/user/sparkcronsvc/edm/fulllist.txt</file>
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
    </workflow-app>

请提出建议。 Shell脚本使用加密的密码冲击波和表名（通过表列表）调用sqoop导入。

Script is :

＃！/ bin / sh dmserver = $ 1 hdp_dir = $ 2

同时读-r行做 tbl = echo $line | cut -d "|" -f1 dmdb = echo $line | cut -d "|" -f2 sqoop import -Dmapreduce.job.user.classpath.first = true- Dhadoop.security.credential.provider.path = jceks： //hdfs/user/root/mssql.datamarthdpservice.pass.jceks \ --connect

  'jdbc:jtds:sqlserver://'${server}'.use.ucdp.net/'${dmdb}';
   integratedSecurity=true;domain=use.ucdp.net;database='${dmdb}';' \
                      --username dataid \
                     --password-alias mssql.data.pass \
                     --query "SELECT * from ${dmdb}.dbo.${tbl} WITH(NOLOCK) 
   WHERE \$CONDITIONS" -m 1 \
                    -----------

   done < <(tail -n +2 fulllist.txt | grep -v "#") 
```

Answer 1

似乎没有在每个数据节点中安装sqoop命令行，这一定是失败了。而是用sqoop动作做同样的事情。

找不到Oozie Sqoop shell操作sqoop命令

1 个答案: