Question

我的工作在map-reduce步骤之前有一些例外，但是工作不会被杀死。如何配置hadoop以使作业在异常后被杀死？

现在调用主类

心跳加速心跳

完成主类的调用

Oozie Launcher结束

stderr logs

org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Io exception: Unknown host specified )
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:577)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:792)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:815)
    at com.seven.crcs.export.dao.ReportDAOImpl.recreateReportEntity(ReportDAOImpl.java:151)
    at com.seven.crcs.export.dao.ReportDAOImpl.saveActiveUserCount(ReportDAOImpl.java:93)
    at com.seven.crcs.export.ReportJdbcExporter.saveActiveUserCount(ReportJdbcExporter.java:55)
    at com.seven.dataprocessor.oc.jobs.reports.export.day.ExportDailyUserReducer.exportUserCounts(ExportDailyUserReducer.java:32)
    at com.seven.dataprocessor.oc.jobs.reports.export.ExportActiveUser
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Io exception: Unknown host specified )

和

2013-02-28 06:06:46,487 INFO org.apache.hadoop.mapred.JobClient: Task Id : attempt_201302270945_0181_r_000000_0, Status : FAILED
2013-02-28 06:07:00,600 INFO org.apache.hadoop.mapred.JobClient: Task Id : attempt_201302270945_0181_r_000000_1, Status : FAILED
2013-02-28 06:07:16,650 INFO org.apache.hadoop.mapred.JobClient: Task Id : attempt_201302270945_0181_r_000000_2, Status : FAILED
2013-02-28 06:07:31,731 INFO org.apache.hadoop.mapred.JobClient: Job complete: job_201302270945_0181

但是工作完成了成功

Answer 1

您的工作实际上已经终止，但只有在任务ID显示3次失败的地图任务尝试之后：

attempt_201302270945_0181_r_000000_的 0

attempt_201302270945_0181_r_000000_的 1

attempt_201302270945_0181_r_000000_的 2

您可以通过将parameter mapred.map.max.attempts设置为1或使用JobConf#setMaxMapAttempts(int)JobConf#setMaxMapAttempts来限制每项任务的最大尝试次数。

这将导致您的地图任务在第一个异常时失败，从而更快地终止您的工作。

Hadoop工作没有被杀死

1 个答案: