Question

我的错误：

org.apache.hadoop.mapreduce.counters.LimitExceededException:
Too many **counter groups**: 51 max=50

问题：我将最大计数器组设置为500，如对类似问题的不同答案所示。

<property>
  <name>mapreduce.job.counters.limit</name>
  <value>1000</value>
</property>

<property>
  <name>mapreduce.job.counters.groups.max</name>
  <value>500</value>
</property>

（我知道.limit不应再使用了）

即使我设置了值，我仍然会收到此错误（当max为50时为51）。此外，我不知道为什么，但这个错误仅发生在最后3或4个减速器上（在完成所有32个映射作业和28个左右减少作业之后）。当我查看计数器信息时，看起来从一开始就创建了超过50个组（第一个地图作业），所以我不确定为什么错误发生在最后。

堆栈跟踪：

LimitExceededException: Too many counter groups: 51 max=50]
org.apache.hadoop.mapreduce.counters.LimitExceededException: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counter groups: 51 max=50
    at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:97)
    at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
    at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
    at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
    at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
    at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
    at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
    at org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
    at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:243)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
    at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:562)
    at org.apache.hadoop.mapreduce.Job$7.run(Job.java:761)
    at org.apache.hadoop.mapreduce.Job$7.run(Job.java:758)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)
    at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
    at org.apache.oozie.action.hadoop.LauncherMapperHelper.isMainSuccessful(LauncherMapperHelper.java:193)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1490)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
    at org.apache.oozie.command.XCommand.call(XCommand.java:287)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

**Caused by:** org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counter groups: 51 max=50
    at org.apache.hadoop.mapreduce.counters.Limits.checkGroups(Limits.java:118)
    at org.apache.hadoop.mapreduce.counters.AbstractCounters.addGroup(AbstractCounters.java:129)
    at org.apache.hadoop.mapreduce.counters.AbstractCounters.addGroup(AbstractCounters.java:143)
    at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
    ... 19 more

Answer 1

也许属性应该是 mapreduce.job.counters.max 而不是 mapreduce.job.counters.limit。我使用的是 Hadoop 版本 2.9.2。

<property>
    <name>mapreduce.job.counters.max</name>
    <value>1000</value>
</property>
<property>
    <name>mapreduce.job.counters.groups.max</name>
    <value>500</value>
</property>

我在 mapred-site.xml 文件中使用了这个配置，它对我有用。
需要重新启动每个节点才能应用配置。

太多的计数器组异常Hadoop

1 个答案: