Question

我一直在尝试配置Hadoop kms以使用hdfs作为密钥提供程序。我关注Hadoop文档，并将以下字段添加到我的kms-site.xml中：

  <property> 
      <name>hadoop.kms.key.provider.uri</name>
      <value>jceks://hdfs@nn1.example.com/kms/test.jceks</value>
      <description>
      URI of the backing KeyProvider for the KMS.
      </description>
  </property>

该路由存在于hdfs中，我希望kms为其keystore创建文件test.jceks。但是，由于此错误，kms无法启动：

ERROR: Hadoop KMS could not be started

REASON: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"

Stacktrace:
---------------------------------------------------
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3220)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3240)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3291)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3259)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:470)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
    at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:132)
    at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:88)
    at org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:660)
    at org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.java:96)
    at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:187)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1080)
    at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1003)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:507)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
    at org.apache.catalina.core.StandardService.start(StandardService.java:525)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:761)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

对于我能理解的内容，似乎这个错误是因为没有为HDFS实现FileSystem。我已经查找了这个错误，但它总是指在升级时hdfs-client缺少jar，我还没有做过（这是一个全新的安装）。我正在使用Hadoop 2.7.2

感谢您的帮助！

Answer 1

我在Hadoop的Jira问题跟踪器here中问了同样的问题。正如用户Wei-Chiu Chuang指出的那样，在HDFS中拥有密钥库不是有效的用例。 KMS不能使用HDFS作为后备存储，因为每个HDFS客户端文件访问都将通过HDFS NameNode循环 - ＆gt; KMS - ＆gt; HDFS NameNode - ＆gt; KMS ....

因此，只有基于文件的KMS才能在本地文件系统上使用密钥库文件。

带有HDFS密钥库的Hadoop KMS：没有用于方案“hdfs”的FileSystem

1 个答案: