Question

我已安装oozie并按照此处的说明安装了共享库

http://mockus.in/forum/viewtopic.php?t=48

当我运行安装附带的示例MapReduce Workflow时，工作流程失败。在webconsole日志中，我可以看到此错误消息

2015-05-24 18:09:32,664  WARN ActionStartXCommand:544 - 
SERVER[mountain] USER[mountain] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[0000000-150524175852032-oozie-moun-W] ACTION[0000000-150524175852032-
oozie-moun-W@mr-node] Error starting action [mr-node]. ErrorType [FAILED], 
ErrorCode [It should never happen], Message [File /user/mountain/share/lib 
does not exist]

我可以在我的hdfs

中看到共享库

mountain@mountain:~/oo/conf$ hdfs dfs -ls share/lib/*
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/distcp
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/hcatalog
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/hive
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/mapreduce-streaming
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/oozie
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/pig
-rw-r--r--   3 mountain supergroup       1348 2015-05-24 21:47 share/lib/lib_20150524214701/sharelib.properties
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/sqoop

列出共享库的命令也返回空

mountain@mountain:~/oo/conf$ oozie admin -shareliblist -oozie http://localhost:11000/oozie
[Available ShareLib]

我错过了任何配置吗？

Answer 1

请注意，您的用户名是山，是oozie-site.xml的属性

<property>
        <name>oozie.service.WorkflowAppService.system.libpath</name>
        <value>/user/${user.name}/share/lib</value>
        <description>
            System library path to use for workflow applications.
            This path is added to workflow application if their job properties sets
            the property 'oozie.use.system.libpath' to true.
        </description>
    </property>

说明Share lib jar将被添加到上述位置 / user / mountain / share / lib 。注意： / user / $ {user.name} 是作业的用户名，应该根据计算机的不同而不同。确保您的共享库jar放在 / user / mountain 下。将oozie目录中的.gz（tar）文件解压缩为 oozie-sharelib-4.1.0 4.1.0是oozie的版本，并将共享文件夹上传到HDFS中，如上面的路径所述。

Answer 2

除了上述回复之外，您还可以从$OOZIE_HOME/bin

运行以下脚本

oozie-setup.sh sharelib create -fs hdfs://localhost:9000

oozie-setup.sh推送oozie-sharelib-4.x.x.tar.gz（位于$OOZIE_HOME dir内）解压缩内容（即distcp，hcatalog，hive，hive2，mapreduce-streaming，oozie，pig，spark，sqoop和sharelib。已定义的/user/${user.name}/share/lib（默认情况下）HDFS位置

但有一点需要注意，首先它会在HDFS cluster share/lib/（lib_截至时间格式）中创建一个目录，例如lib_20171230083804

并将lib的内容放在新创建的目录中，因此时间用户运行oozie作业，因为配置oozie转到oozie.service.WorkflowAppService.system.libpath定义的值，但由于上面提到的新类型目录，它没有'找到任何包含库的内容并抛出这种错误 -

ACTION[0000000-171230085230873-oozie-arif-W@sqoop-node] Error starting action [sqoop-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/arif/share/lib does not exist]

因此，您可以在这里定义oozie.service.WorkflowAppService.system.libpath /user/${user.name}/share/lib/lib_20171230083804或使用hadoop fs -mv将目录lib_20171230083804内的所有内容直接移至share/lib/，此后 - 重新开始工作，

并且可以验证sharelib列表如下 -

bin/oozie admin -oozie http://localhost:11000/oozie -shareliblist

它会解决你的问题，希望这会对某人有所帮助，谢谢。

命令'oozie admin sharedliblist ...'清空共享库

2 个答案: