livi服务器无法连接到启用了身份验证的spark群集

Question

我正在使用spark.authenticate=true运行3节点Spark 2.1.1独立群集，并且Livy服务器无法连接到它。 这是我livy.conf

的一部分

livy.spark.authenticate = true
livy.spark.master = spark://HOSTNAME:25000
livy.spark.authenticate.secret = SECRET

在日志中，我可以看到由于某种原因，安全性被声明为DISABLED，我猜这会导致此错误跟踪......

INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(vva); groups with view permissions: Set(); users  with modify permissions: Set(vva); groups with modify permissions: Set()
INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://x.x.xxx.xx:25000...
INFO TransportClientFactory: Successfully created connection to /x.x.xxx.xx:25000 after 9 ms (0 ms spent in bootstraps)
WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master x.x.xxx.xx:25000
org.apache.spark.SparkException: Exception thrown in awaitResult:
.........
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Expected SaslMessage, received something else (maybe your client does not have SASL enabled?)

PS。我在单节点环境中测试了相同的内容，似乎运行正常。

感谢。

Answer 1

发现某些火花参数无法从livy.conf传递，必须将它们添加到spark-defaults.conf。