Question

我使用Spark standalone 1.6.x版本连接启用了Kerberos的hadoop 2.7.x

JavaDStream<String> status = stream.map(new Function<String, String>() { 
    public String call(String arg0) throws Exception {
        Configuration conf = new Configuration();
        FileSystem fs = null; 
        conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("dfs.namenode.kerberos.principal", "hdfs/_HOST@REALM");
        UserGroupInformation.setConfiguration(conf); 
        UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("abc","~/abc.ketyab"));
        System.out.println("Logged in successfully."); 
        fs = FileSystem.get(new URI(activeNamenodeURI), conf);
        FileStatus[] s = fs.listStatus(new Path("/"));
        for (FileStatus status : s) {
            System.out.println(status.getPath().toString());
        }
        return "success";
    }
});

但要低于异常

用户：abc @ REALM（auth：KERBEROS）引起：java.io.IOException：本地异常失败：java.io.IOException： org.apache.hadoop.security.AccessControlException：客户端不能通过验证：[TOKEN，KERBEROS];主机详细信息：本地主机是：＆＃34;主机名1 / 0.0.0.0＆＃34 ;;目标主机是：＆＃34; hostname2＆＃34;：8020;在 org.apache.hadoop.net.NetUtils.wrapException（NetUtils.java:772）at at org.apache.hadoop.ipc.Client.call（Client.java:1472）at org.apache.hadoop.ipc.Client.call（Client.java:1399）at org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke（ProtobufRpcEngine.java:232）在com.sun.proxy。$ Proxy44.create（未知来源）at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create（ClientNamenodeProtocolTranslatorPB.java:295） at sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method）at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）在 sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）在java.lang.reflect.Method.invoke（Method.java:606）at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod（RetryInvocationHandler.java:187）在 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke（RetryInvocationHandler.java:102）在com.sun.proxy。$ Proxy45.create（未知来源）at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate（DFSOutputStream.java:1725）在org.apache.hadoop.hdfs.DFSClient.create（DFSClient.java:1668）at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1593）at org.apache.hadoop.hdfs.DistributedFileSystem $ 6.doCall（DistributedFileSystem.java:397）在 org.apache.hadoop.hdfs.DistributedFileSystem $ 6.doCall（DistributedFileSystem.java:393）在 org.apache.hadoop.fs.FileSystemLinkResolver.resolve（FileSystemLinkResolver.java:81）在 org.apache.hadoop.hdfs.DistributedFileSystem.create（DistributedFileSystem.java:393）在 org.apache.hadoop.hdfs.DistributedFileSystem.create（DistributedFileSystem.java:337）在org.apache.hadoop.fs.FileSystem.create（FileSystem.java:908）at org.apache.hadoop.fs.FileSystem.create（FileSystem.java:889）at at com.abc.HDFSFileWriter.createOutputFile（HDFSFileWriter.java:354）... 21更多引起：java.io.IOException： org.apache.hadoop.security.AccessControlException：客户端不能通过身份验证：[TOKEN，KERBEROS] at org.apache.hadoop.ipc.Client $ Connection $ 1.run（Client.java:680）at at java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject.java:415）at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1628）在 org.apache.hadoop.ipc.Client $ Connection.handleSaslConnectionFailure（Client.java:643）在 org.apache.hadoop.ipc.Client $ Connection.setupIOstreams（Client.java:730）在 org.apache.hadoop.ipc.Client $ Connection.access $ 2800（Client.java:368）在org.apache.hadoop.ipc.Client.getConnection（Client.java:1521）at org.apache.hadoop.ipc.Client.call（Client.java:1438）... 43更多引起 by：org.apache.hadoop.security.AccessControlException：客户端不能通过身份验证：[TOKEN，KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient（SaslRpcClient.java:172）在 org.apache.hadoop.security.SaslRpcClient.saslConnect（SaslRpcClient.java:396）在 org.apache.hadoop.ipc.Client $ Connection.setupSaslConnection（Client.java:553）在 org.apache.hadoop.ipc.Client $ Connection.access $ 1800（Client.java:368）在org.apache.hadoop.ipc.Client $ Connection $ 2.run（Client.java:722）at at org.apache.hadoop.ipc.Client $ Connection $ 2.run（Client.java:718）at at java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject.java:415）

无法将spark standlone应用程序与kerberized hadoop

0 个答案: