Cloudera Spark无法在群集模式

时间:2018-06-15 14:35:05

标签: apache-spark hadoop hbase kerberos cloudera

我正在做Spark2提交,通常用于在客户端模式下执行相同操作,并且从未遇到过任何Kerberos身份验证问题。但是现在当我开始在集群中尝试相同的代码时,我收到的身份验证问题如下:

WARN UserGroupInformation :PriviledgedActionException as:user_name(auth:SIMPLE)原因:javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:失败)找到任何Kerberos tgt)]

WARN RpcClientImpl :连接到服务器时遇到异常:javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:无法找到任何Kerberos) TGT)]

错误RpcClientImpl :SASL身份验证失败。最可能的原因是凭据丢失或无效。考虑一下' kinit'。 javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)]         在com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)         在org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:181)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.setupSaslConnection(RpcClientImpl.java:618)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.access $ 700(RpcClientImpl.java:163)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection $ 2.run(RpcClientImpl.java:744)         在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection $ 2.run(RpcClientImpl.java:741)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:422)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.setupIOstreams(RpcClientImpl.java:741)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.writeRequest(RpcClientImpl.java:907)         at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.tracedWriteRequest(RpcClientImpl.java:874)         在org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1243)         在org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)         at org.apache.hadoop.hbase.ipc.AbstractRpcClient $ BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)         at org.apache.hadoop.hbase.protobuf.generated.ClientProtos $ ClientService $ BlockingStub.scan(ClientProtos.java:34094)         在org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:400)         在org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)         在org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas $ RetryingRPC.call(ScannerCallableWithReplicas.java:381)         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas $ RetryingRPC.call(ScannerCallableWithReplicas.java:355)         在org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)         at org.apache.hadoop.hbase.client.ResultBoundedCompletionService $ QueueingFuture.run(ResultBoundedCompletionService.java:80)         在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617)         在java.lang.Thread.run(Thread.java:745) 引起:GSSException:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)         在com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)         ......还有26个

我在shell脚本中提交spark提交,该脚本已经运行kinit命令,如下所示:

kinit -kt keytab_file.keytab user_name @ realm

另外,我已经在使用principal的spark submit中传递了keytab文件,并且我获得了成功的身份验证日志消息。当我使用keytab在代码中登录时,我面临的问题是:

hBaseConf = HBaseConfiguration.create()
hBaseConf.set("hadoop.security.authentication", "kerberos")
hBaseConf.set("hbase.security.authentication", "kerberos")
hBaseConf.set("hbase.zookeeper.quorum", NameNode)
hBaseConf.set("hbase.zookeeper.property.client.port", "2181")

//Login Using KeyTab
UserGroupInformation.setConfiguration(hBaseConf)
UserGroupInformation.loginUserFromKeytab(userName, keyTab)

对此的任何帮助都将受到高度赞赏!

0 个答案:

没有答案