LeaseExpiredException：Hadoop mapReduce中的租约不匹配|怎么解决？

时间：2013-11-13 10:56:08

标签： java hadoop mapreduce

我在stackoverflow中看到了几个与此相关的问题，但它们并没有解决我的问题。

使用90 Mb文件运行作业时，我收到LeaseExpiredException 

13/11/12 15:46:41 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/11/12 15:46:42 INFO input.FileInputFormat: Total input paths to process : 1
13/11/12 15:46:43 INFO mapred.JobClient: Running job: job_201310301645_25033
13/11/12 15:46:44 INFO mapred.JobClient:  map 0% reduce 0%
13/11/12 15:46:56 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000000_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /user/hdfs/in/map owned by DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by DFSClient_NONMAPREDUCE_-1561990512_1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)
    at org.
attempt_201310301645_25033_m_000000_0: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_0: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
13/11/12 15:47:02 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000000_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /user/hdfs/in/map owned by DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by DFSClient_NONMAPREDUCE_-1662926329_1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2262)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2175)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:299)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44954)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java
attempt_201310301645_25033_m_000000_1: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_1: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
attempt_201310301645_25033_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201310301645_25033_m_000000_1: log4j:WARN Please initialize the log4j system properly.
attempt_201310301645_25033_m_000000_1: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
13/11/12 15:47:10 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000001_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hdfs/in/map: File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1622335545_1 does not have any open files.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2452)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)

为什么会这样？ 我的映射器代码的第一部分是

public void map(Object key, Text value, Context context)
            throws IOException, InterruptedException {

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);

 Path inputfile = new Path("in/map");
 BufferedWriter getdatabuffer = new BufferedWriter(new OutputStreamWriter(fs.create(inputfile)));
   getdatabuffer.write(value.toString());
   getdatabuffer.close();
Path Attribute = new Path("in/Attribute");
        int row =0;
        BufferedReader read = new BufferedReader(new InputStreamReader(fs.open(inputfile)));
        String str = null;
        while((str = read.readLine())!=null){

         row++;                     //total row count
         StringTokenizer st =new StringTokenizer(str," ");
         col = st.countTokens();
          }
        read.close();
...........
...........
.............
............

进一步的计算基于上述“地图”文件。

为什么会这样？ 我认为它无法多次写入/ map。

我没有删除任何文件。 如何摆脱这个？

任何建议？

编辑：11月15日

当我在/ map中检查时，我的群集中没有创建。 为什么是not getting created？ 我认为这就是获得LeaseExpiredException

的原因

方案

我的输入文件为1GB。内容类似于

file1.txt
0 0 6
3 4 8
5 9 3
12 4 6
8 7 8
9 8 1
6 12 0
10 8 0
8 5 1
14 8 1

我需要找出Atranspose * A，其中A [] []是文件的输入数据。

所以我的逻辑就像：

有什么数据进入mapper我将找到Atranspose*A，在Reducer中，我将计算从每个映射器计算的所有Atran * A的总和。这样我就可以获得File1.txt的Atranspose * A.

为此我想把每个映射器数据写入一个文件，然后进入A [] []数组并找到ATrn * A.

1 个答案:

答案 0 :(得分：2)

看起来您可能有多个映射器尝试写入HDFS中的同一文件：

Path inputfile = new Path("in/map");
BufferedWriter getdatabuffer = new BufferedWriter(new OutputStreamWriter(fs.create(inputfile)));
getdatabuffer.write(value.toString());
getdatabuffer.close();

如果您执行此代码的单个地图任务不止一个，那么您将遇到所遇到的问题。

您不应该直接在映射器代码中写入HDFS，并且您拥有的逻辑将覆盖每个映射器中的文件，对于每个输入值 - 您能解释一下您要实现的目标吗？

相关问题
最新问题