命令'oozie admin sharedliblist ...'清空共享库

时间:2015-05-25 05:16:56

标签: hadoop oozie

我已安装oozie并按照此处的说明安装了共享库

http://mockus.in/forum/viewtopic.php?t=48

当我运行安装附带的示例MapReduce Workflow时,工作流程失败。在webconsole日志中,我可以看到此错误消息

2015-05-24 18:09:32,664  WARN ActionStartXCommand:544 - 
SERVER[mountain] USER[mountain] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[0000000-150524175852032-oozie-moun-W] ACTION[0000000-150524175852032-
oozie-moun-W@mr-node] Error starting action [mr-node]. ErrorType [FAILED], 
ErrorCode [It should never happen], Message [File /user/mountain/share/lib 
does not exist]

我可以在我的hdfs

中看到共享库
mountain@mountain:~/oo/conf$ hdfs dfs -ls share/lib/*
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/distcp
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/hcatalog
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/hive
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/mapreduce-streaming
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/oozie
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/pig
-rw-r--r--   3 mountain supergroup       1348 2015-05-24 21:47 share/lib/lib_20150524214701/sharelib.properties
drwxr-xr-x   - mountain supergroup          0 2015-05-24 21:47 share/lib/lib_20150524214701/sqoop

列出共享库的命令也返回空

mountain@mountain:~/oo/conf$ oozie admin -shareliblist -oozie http://localhost:11000/oozie
[Available ShareLib]

我错过了任何配置吗?

2 个答案:

答案 0 :(得分:0)

请注意,您的用户名是山,是oozie-site.xml的属性

<property>
        <name>oozie.service.WorkflowAppService.system.libpath</name>
        <value>/user/${user.name}/share/lib</value>
        <description>
            System library path to use for workflow applications.
            This path is added to workflow application if their job properties sets
            the property 'oozie.use.system.libpath' to true.
        </description>
    </property>

说明Share lib jar将被添加到上述位置 / user / mountain / share / lib 。注意: / user / $ {user.name} 是作业的用户名,应该根据计算机的不同而不同。 确保您的共享库jar放在 / user / mountain 下。将oozie目录中的.gz(tar)文件解压缩为 oozie-sharelib-4.1.0 4.1.0是oozie的版本,并将共享文件夹上传到HDFS中,如上面的路径所述。

答案 1 :(得分:0)

除了上述回复之外,您还可以从$OOZIE_HOME/bin

运行以下脚本
oozie-setup.sh sharelib create -fs hdfs://localhost:9000

oozie-setup.sh推送oozie-sharelib-4.x.x.tar.gz(位于$OOZIE_HOME dir内)解压缩内容(即distcp,hcatalog,hive,hive2,mapreduce-streaming,oozie,pig,spark,sqoop和sharelib。已定义的/user/${user.name}/share/lib(默认情况下)HDFS位置

但有一点需要注意,首先它会在HDFS cluster share/lib/(lib_截至时间格式)中创建一个目录,例如lib_20171230083804

并将lib的内容放在新创建的目录中,因此时间用户运行oozie作业,因为配置oozie转到oozie.service.WorkflowAppService.system.libpath定义的值,但由于上面提到的新类型目录,它没有'找到任何包含库的内容并抛出这种错误 -

ACTION[0000000-171230085230873-oozie-arif-W@sqoop-node] Error starting action [sqoop-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/arif/share/lib does not exist]

因此,您可以在这里定义oozie.service.WorkflowAppService.system.libpath /user/${user.name}/share/lib/lib_20171230083804或使用hadoop fs -mv将目录lib_20171230083804内的所有内容直接移至share/lib/,此后 - 重新开始工作,

并且可以验证sharelib列表如下 -

bin/oozie admin -oozie http://localhost:11000/oozie -shareliblist

它会解决你的问题,希望这会对某人有所帮助,谢谢。