我有一个符合此规范的准Oozie协调员:
<coordinator-app name="my-coord" frequency="${coord:days(1)}"
start="${startDate}" end="${endDate}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.4">
<controls>
<timeout>${timeout}</timeout>
</controls>
<action>
<workflow>
<app-path>${workflow}</app-path>
</workflow>
</action>
</coordinator-app>
它在预定的 nominal 开始时间附近启动了工作流程作业。但是后来,日志显示工作流程作业进入了失败状态。要检索工作信息,我运行了:
oozie job -info 0000909-190113225141152-oozie-oozi-W
提供了有用的信息,包括以下异常跟踪:
] Launcher exception: org.apache.spark.SparkException: Application application_1547448533998_26676 finished with failed status
org.apache.oozie.action.hadoop.JavaMainException: org.apache.spark.SparkException: Application application_1547448533998_26676 finished with failed status
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.spark.SparkException: Application application_1547448533998_26676 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1169)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
... 15 more
不幸的是,这个堆栈跟踪(显然是由SparkSubmit
产生的)并没有说明为什么我的工作流作业(一个Scala程序)实际上失败了。
这似乎是一个足够普遍的情况-工作流逻辑失败并触发了自己的堆栈跟踪。
在Hadoop / Oozie / Coordinator / Workflow设置中是否还有其他地方可以找到此类堆栈跟踪?
答案 0 :(得分:1)
在Hadoop环境中,您可以从Yarn Resource Manager视图访问所有应用程序日志。它应该显示所有正在运行的应用程序及其关联日志的列表。 如果要在Azure的HDInsight群集上运行此系统,则可以在此链接https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-access-yarn-app-logs-linux中找到更多信息。在这里,可以在Ambari视图-> Yarn->资源管理器Web视图中找到日志。否则,也许您的环境中可能也有类似的东西。
答案 1 :(得分:1)
使用yarn applications -list
查看Hadoop集群上运行的作业列表。然后按照以下步骤操作:
yarn logs -applicationId <application_ID>
。结果日志应显示您的Scala程序日志以及其他非Scala程序生成的日志。如果您的Scala程序在每个日志记录指令中都嵌入唯一的前缀,则可以帮助您过滤其他程序的日志,这将很有帮助。