Question

在运行hive客户端工具的数据节点上运行我的脚本正在运行。但是当我使用Oozie计划配置单元脚本时，我得到的错误如下所示。

我已将tez-site.xml中的tez.lib.uris设置为hdfs：/// apps / tez /，hdfs：/// apps / tez / lib /

我在这里缺少什么？

Hive脚本：

  USE av_raw;

  LOAD DATA INPATH '${INPUT}' INTO TABLE alarms_stg;

  INSERT INTO TABLE alarms PARTITION (year, month)
  SELECT * FROM alarms_stg WHERE job_id = '${JOBID}';

工作流程操作：

<!-- load processed data and store in hive -->
<action name="load-data">
    <hive xmlns="uri:oozie:hive-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>hive-site.xml</job-xml>
        <script>load_data.hive</script>
        <param>INPUT=${complete}</param>
        <param>JOBID=${wf:actionData('stage-data')['hadoopJobs']}</param>
    </hive>

    <ok to="end"/>

    <error to="fail"/>
</action>

错误：

 Log Type: stderr

  Log Length: 3227
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/grid/5/hadoop/yarn/local/filecache/2418/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".

  Logging initialized using configuration in file:/grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1417175595182_12259/container_1417175595182_12259_01_000002/hive-log4j.properties
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:358)
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
  at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:316)
  at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:277)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
  at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
  Caused by: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:137)
  at org.apache.tez.client.TezSession.start(TezSession.java:105)
  at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:185)
  at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:123)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:356)
  ... 19 more

Answer 1

请尝试在Oozie工作的workflow.xml中添加tez.lib.uris = hdfs：/// apps / tez /，hdfs：/// apps / tez / lib /

例如）workflow.xml

<!-- load processed data and store in hive -->
<action name="load-data">
    <hive xmlns="uri:oozie:hive-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>hive-site.xml</job-xml>
        <configuration>
            <property>
                <name>tez.lib.uris</name>
                <value>hdfs:///apps/tez/,hdfs:///apps/tez/lib/</value>
            </property>
        </configuration>
        <script>load_data.hive</script>
        <param>INPUT=${complete}</param>
        <param>JOBID=${wf:actionData('stage-data')['hadoopJobs']}</param>
    </hive>
    <ok to="end"/>
    <error to="fail"/>
</action>

Answer 2

最终，您可以尝试在“Hadoop属性”下的“工作流设置”中直接添加“tez.lib.uris”的值。

tez.lib.uris = hdfs：/// apps / tez /，hdfs：/// apps / tez / lib /

在添加之前，请在tez-site.xml中验证正确的值。

Oozie工作流程：由于Tez，Hive操作失败

2 个答案: