Question

我从源代码使用最新的hadoop 3.0.0版本。我启动并运行了我的时间轴服务，并配置了hadoop以将其用于作业历史记录。但是当我点击resoucemanager UI中的历史记录时，我得到以下错误： -

HTTP ERROR 404

Problem accessing /jobhistory/job/job_1444395439959_0001. Reason:

    NOT_FOUND

有人可以指出我在这里缺少的东西。以下是我的yarn-site.xml： -

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <description>The hostname of the Timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>0.0.0.0</value>
</property>
<property>
  <description>Address for the Timeline server to start the RPC server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10200</value>
</property>

<property>
  <description>The http address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>

<property>
  <description>The https address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:8190</value>
</property>

<property>
  <description>Handler thread count to serve the client RPC requests.</description>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>10</value>
</property>
<property>
  <description>Indicate to ResourceManager as well as clients whether
  history-service is enabled or not. If enabled, ResourceManager starts
  recording historical data that Timelien service can consume. Similarly,
  clients can redirect to the history service when applications
  finish if this is enabled.</description>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Store class name for history store, defaulting to file system
  store</description>
  <name>yarn.timeline-service.generic-application-history.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>
<property>
     <description>URI pointing to the location of the FileSystem path where the history will be persisted.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.uri</name>
     <value>/tmp/yarn/system/history</value>
</property>
<property>
     <description>T-file compression types used to compress history data.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.compression-type</name>
     <value>none</value>
</property>



 <property>
     <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

和我的mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>mapreduce.job.emit-timeline-data</name>
<value>true</value>
</property>
</configuration>

JPS输出：

6022 NameNode
27976 NodeManager
27859 ResourceManager
6139 DataNode
6310 SecondaryNameNode
28482 ApplicationHistoryServer
29230 Jps

Answer 1

如果要通过YARN RM Web UI查看日志，则需要启用日志聚合。为此，您需要在yarn-site.xml中设置以下参数：

  <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
  </property>
  <property>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
  </property>
  <property>
      <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
      <value>logs</value>
  </property>

如果您不启用日志聚合，则NM将在本地存储日志。使用上述设置，日志将在HDFS中以“/ app-logs / {username} / logs /”聚合。在此文件夹下，您可以找到目前为止运行的所有应用程序的日志。同样，日志保留由配置参数“yarn.log-aggregation.retain-seconds”确定（保留聚合日志的时间长度）。

当MapReduce应用程序运行时，您可以从YARN的Web UI访问日志。应用程序完成后，将通过“作业历史记录服务器”提供日志。

另外，在yarn-site.xml中设置以下配置参数：

<property>
  <name>yarn.log.server.url</name>
  <value>http://{job-history-hostname}:8188/jobhistory/logs</value>
</property>

YARN的工作经历没有到来

1 个答案: