Question

我试图通过我在Hue创建的Oozie工作流程来运行Hive操作，但是“心脏跳动”的操作永远不会执行Hive SQL。

我已经阅读了有关心脏跳动的其他帖子，但是在解析了SQL语句之后，这个似乎发生在不同的地方。我已经检查了集群中每个节点的内存，并且我已经验证了任务计数参数是否合理。

这是hive-config.xml文件：

<configuration>

<property>   
  <name>javax.jdo.option.ConnectionURL</name>   
  <value>jdbc:hive://10.1.10.250:10000/testdb</value>   
  <description>JDBC connect string</description> 
</property>

<property>   
  <name>javax.jdo.option.ConnectionDriverName</name>   
  <value>org.apache.hadoop.hive.jdbc.HiveDriver</value>   
  <description>JDBC driver</description> 
</property>

</configuration>

我知道Hive连接正在运行，因为如果提供错误的SQL语句，错误的URL或错误的驱动程序名称，操作将失败。

这是动作标准日志：

  [...truncated]

  =================================================================

  >>> Invoking Hive command line now >>>

  4283 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
  4339 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
  4354 [main] INFO  hive.ql.parse.ParseDriver  - Parsing command: create table testdb.temp99 (col1 int)
  4665 [main] INFO  hive.ql.parse.ParseDriver  - Parse Completed
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=parse start=1418968298270 end=1418968298598 duration=328 from=org.apache.hadoop.hive.ql.Driver>
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
  4733 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Starting Semantic Analysis
  4735 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Creating table testdb.temp99 position=13
  4760 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Semantic Analysis Completed
  4775 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=semanticAnalyze start=1418968298598 end=1418968298706 duration=108 from=org.apache.hadoop.hive.ql.Driver>
  4784 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
  4784 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=compile start=1418968298215 end=1418968298715 duration=500 from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Concurrency mode is disabled, not creating a lock manager
  4785 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Starting command: create table testdb.temp99 (col1 int)
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=TimeToSubmit start=1418968298215 end=1418968298723 duration=508 from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
  4815 [main] INFO  hive.ql.exec.DDLTask  - Default to LazySimpleSerDe for table testdb.temp99
  4935 [main] INFO  org.apache.hadoop.hive.metastore.HiveMetaStore  - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  4959 [main] INFO  org.apache.hadoop.hive.metastore.ObjectStore  - ObjectStore, initialize called
  5261 [main] INFO  DataNucleus.Persistence  - Property datanucleus.cache.level2 unknown - will be ignored
  Heart beat
  Heart beat
  [...forever...]

为什么工作流程会在日志中心跳而不是继续？

附录：

与此Hive操作相关联的Oozie工作流程为：

<workflow-app name="Hive-copy" xmlns="uri:oozie:workflow:0.4">
    <start to="Hive"/>
    <action name="Hive">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <job-xml>/user/test/hive-config.xml</job-xml>
            <script>/user/test/test.sql</script>
            <file>hive-config.xml#hive-config.xml</file>
        </hive>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

Answer 1

此问题与nodeManager有关。尝试在 mapred-site.xml 文件中设置以下属性来解决问题：

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>

与此问题类似：https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/v0BHtQ0hlBg

Oozie Hive动作挂起，心脏永远跳动

1 个答案: