Question

我刚刚运行了这样的hdfs演示：

public final class HDFSRemoveDemo {

    public static void main(String[] args) throws Exception {
        Path root = new Path("hdfs://localhost:49000/");
        FileSystem fs = root.getFileSystem(new Configuration());

        fs.create(new Path("/tmp/test"));
        fs.delete(new Path("/tmp/test"), false);
        fs.close();
    }
}

一个令人费解的例外情况如下：

org.apache.hadoop.hdfs.DFSClient closeAllFilesBeingWritt
en
SEVERE: Failed to close file /tmp/test
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.Le
aseExpiredException: No lease on /tmp/test File does not exist. Holder DFSClient
_NONMAPREDUCE_-1727094995_1 does not have any open files
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1999)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1990)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSN
amesystem.java:2045)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesyste
m.java:2033)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:805)
    at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

    at org.apache.hadoop.ipc.Client.call(Client.java:1113)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at com.sun.proxy.$Proxy1.complete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocati
onHandler.java:85)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHand
ler.java:62)
    at com.sun.proxy.$Proxy1.complete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.jav
a:4121)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:4022)
    at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:41
7)
    at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:433)
    at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.jav
a:369)

当我删除fs.close();时，效果很好。

环境是：

hadoop-core - 1.2.1
jdk - 1.6.0_21

文件系统关闭后发生了什么？有人遇到过这个问题吗？

Answer 1

通常，当您执行fs.close()时，不应致电FileSystem.get(...) FileSystem.get(...)实际上不会打开“新”FileSystem对象。当您在该FileSystem上执行close()时，您也将关闭任何上游进程。

例如，如果在映射器中关闭FileSystem，则MapReduce驱动程序在再次尝试关闭文件系统时将失败。

HDFS FileSystem关闭异常

1 个答案: