Question

我正在从kafka开始从事poc spark-streaming工作。我可以对非安全的kafka 0.10群集使用相同的代码，但是当我切换到针对ssl / kerberos（hdp 2.5）设置运行时，我得到了一个例外：

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 29 more

spark会话创建正常但是当执行程序启动以消耗主题中的新内容时，我得到上述异常。

提交代码非常简单：

spark-submit \
--master yarn \
--keytab ./bilsch.keytab \
--principal bilsch@HDP.SOME.ORG \
--files kafka_client_jaas.conf,bilsch.keytab \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.0.2 \
--repositories http://repo.hortonworks.com/content/repositories/releases \
--num-executors 1 \
--class producer \
--driver-java-options "-Djava.security.auth.login.config=./kafka_client_jaas.conf -Dhdp.version=2.5.3.0-37" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_client_jaas.conf" \
streaming_0331_2.11-1.0.jar 2>&1 | tee out

如果它有助于我跟随michael-noll使用kafka教程进行火花流式传输

不确定需要传递给我尚未传递的执行程序或者它只是一个jaas配置问题？

Answer 1

尝试按照以下步骤操作： -

从spark-submit命令中删除--keytab和--principal，因为此信息已存在于JAAS配置中。
运行kinit -kt bilsch.keytab bilsch@HDP.SOME.ORG，然后尝试运行spark-submit命令。

火花流kafka kerberos

1 个答案: