我已安装oozie并按照此处的说明安装了共享库
http://mockus.in/forum/viewtopic.php?t=48
当我运行安装附带的示例MapReduce Workflow时,工作流程失败。在webconsole日志中,我可以看到此错误消息
2015-05-24 18:09:32,664 WARN ActionStartXCommand:544 -
SERVER[mountain] USER[mountain] GROUP[-] TOKEN[] APP[map-reduce-wf]
JOB[0000000-150524175852032-oozie-moun-W] ACTION[0000000-150524175852032-
oozie-moun-W@mr-node] Error starting action [mr-node]. ErrorType [FAILED],
ErrorCode [It should never happen], Message [File /user/mountain/share/lib
does not exist]
我可以在我的hdfs
中看到共享库mountain@mountain:~/oo/conf$ hdfs dfs -ls share/lib/*
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/distcp
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/hcatalog
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/hive
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/mapreduce-streaming
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/oozie
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/pig
-rw-r--r-- 3 mountain supergroup 1348 2015-05-24 21:47 share/lib/lib_20150524214701/sharelib.properties
drwxr-xr-x - mountain supergroup 0 2015-05-24 21:47 share/lib/lib_20150524214701/sqoop
列出共享库的命令也返回空
mountain@mountain:~/oo/conf$ oozie admin -shareliblist -oozie http://localhost:11000/oozie
[Available ShareLib]
我错过了任何配置吗?
答案 0 :(得分:0)
请注意,您的用户名是山,是oozie-site.xml的属性
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
<description>
System library path to use for workflow applications.
This path is added to workflow application if their job properties sets
the property 'oozie.use.system.libpath' to true.
</description>
</property>
说明Share lib jar将被添加到上述位置 / user / mountain / share / lib 。注意: / user / $ {user.name} 是作业的用户名,应该根据计算机的不同而不同。 确保您的共享库jar放在 / user / mountain 下。将oozie目录中的.gz(tar)文件解压缩为 oozie-sharelib-4.1.0 4.1.0是oozie的版本,并将共享文件夹上传到HDFS中,如上面的路径所述。
答案 1 :(得分:0)
除了上述回复之外,您还可以从$OOZIE_HOME/bin
oozie-setup.sh sharelib create -fs hdfs://localhost:9000
oozie-setup.sh
推送oozie-sharelib-4.x.x.tar.gz
(位于$OOZIE_HOME
dir内)解压缩内容(即distcp,hcatalog,hive,hive2,mapreduce-streaming,oozie,pig,spark,sqoop和sharelib。已定义的/user/${user.name}/share/lib
(默认情况下)HDFS
位置
但有一点需要注意,首先它会在HDFS cluster
share/lib/
(lib_截至时间格式)中创建一个目录,例如lib_20171230083804
并将lib的内容放在新创建的目录中,因此时间用户运行oozie作业,因为配置oozie转到oozie.service.WorkflowAppService.system.libpath
定义的值,但由于上面提到的新类型目录,它没有'找到任何包含库的内容并抛出这种错误 -
ACTION[0000000-171230085230873-oozie-arif-W@sqoop-node] Error starting action [sqoop-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/arif/share/lib does not exist]
因此,您可以在这里定义oozie.service.WorkflowAppService.system.libpath
/user/${user.name}/share/lib/lib_20171230083804
或使用hadoop fs -mv
将目录lib_20171230083804
内的所有内容直接移至share/lib/
,此后 - 重新开始工作,
并且可以验证sharelib列表如下 -
bin/oozie admin -oozie http://localhost:11000/oozie -shareliblist
它会解决你的问题,希望这会对某人有所帮助,谢谢。