oozie sqoop optionsfile提供File Not Found异常

时间：2015-02-17 17:50:45

标签： oozie hue hortonworks-data-platform

我有各种博客提到有关Sqoop作业所需的选项文件应放在HDFS位置内的事实。

我确实将文件放在HDFS位置，并在文件参数中提供了文件。

尽管如此，它还是会抛出FileNotFound异常。

oozie工作流程文件如下所示 - 

    <workflow-app name="TestFlow" xmlns="uri:oozie:workflow:0.4">
        <start to="TableImport"/>
        <action name="TableImport">
            <sqoop xmlns="uri:oozie:sqoop-action:0.2">
                <job-tracker>${jobTracker}</job-tracker>
                <name-node>${nameNode}</name-node>
                <prepare>
                      <delete path="${nameNode}/path/to/place/file"/>
                </prepare>
                  <arg>--options-file </arg>
                  <arg>/tmp/sqoopoptions.params</arg>
                  <arg>--target-dir</arg>
                  <arg>/path/to/place/file</arg>
                  <arg>--query</arg>
                  <arg>select employee_id, employeename, DEPT_ID from employee where $CONDITIONS</arg>
                  <arg>--split-by</arg>
                  <arg>DEPT_ID</arg>
               <file>/tmp/sqoopoptions.params#sqoopoptions.params</file>
            </sqoop>
            <ok to="end"/>
            <error to="kill"/>
        </action>
        <kill name="kill">
            <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
        <end name="end"/>
    </workflow-app>

选项文件存在于HDFS位置

/tmp/sqoopoptions.params

我可以从Hue查看文件，意味着该文件位于HDFS位置！。

我正在使用Hortonworks 2.1发行版

例外stakctrace是

    4046 [main] ERROR org.apache.sqoop.Sqoop  - Error while expanding arguments
    java.lang.Exception: Unable to read options file: /tmp/sqoopoptions.params
        at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:102)
        at com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:203)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
    Caused by: java.io.FileNotFoundException: /tmp/sqoopoptions.params (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at java.io.FileReader.<init>(FileReader.java:72)
        at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:70)

更新：

我尝试将选项文件放在运行oozie服务器的本地文件系统中。有用。当我查看代码时，它似乎也是正确的。 org.apache.sqoop.util.OptionsFileUtil的相关部分如下所示

     public static String[] expandArguments(String[] args) throws Exception {
        List<String> options = new ArrayList<String>();

        for (int i = 0; i < args.length; i++) {
          if (args[i].equals(Sqoop.SQOOP_OPTIONS_FILE_SPECIFIER)) {
            if (i == args.length - 1) {
              throw new Exception("Missing options file");
            }

            String fileName = args[++i];
            File optionsFile = new File(fileName);
            BufferedReader reader = null;
            StringBuilder buffer = new StringBuilder();
            try {
             **reader = new BufferedReader(new FileReader(optionsFile));**
              String nextLine = null;
              while ((nextLine = reader.readLine()) != null) {

现在，这适用于本地文件配置。这意味着，如果我将文件放在HDFS中，它就不会工作。除非，如果路径以hdfs：// namdnode / path / to / file给出，FileReader将从HDFS位置选择文件。这是预期的工作方式吗？

我将参数提供给{nameNode} /tmp/sqoopoptions.params。它抛出以下异常。如果我仔细注意到，/正在引起问题。它从异常堆栈跟踪中清楚。其中一个斜杠在传递给FileReader

时消失 
    java.lang.Exception: Unable to read options file: hdfs://ipofnamednode:port/tmp/sqoopoptions.params
        at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:102)
        at com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:203)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
    **Caused by: java.io.FileNotFoundException: hdfs:/ipofnamednode:port/tmp/sqoopoptions.params (No such file or directory)**
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at java.io.FileReader.<init>(FileReader.java:72)
        at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:70)
        ... 21 more

如在堆栈跟踪中所见，由于转义字符而删除了其中一个斜杠。

2 个答案:

答案 0 :(得分：0)

您需要在<file>属性中使用namenode作为文件路径的前缀，例如：

<file>hdfs://tmp/sqoopoptions.params#sqoopoptions.params</file>

答案 1 :(得分：0)

尝试将符号链接名称作为<arg>并将namenode添加到<file>标记的开头：

    ...
    <arg>--options-file </arg>
    <arg>sqoopoptions.params</arg>
    ...
    <file>${namenode}/tmp/sqoopoptions.params#sqoopoptions.params</file>
    ...
相关问题
最新问题