我正面临着Zeppelin的问题 - ＆gt; Shiro - ＆gt; Livy整合。如果有人可以帮助我，那就太好了。
我目前的环境设置如下：
•1个主节点和2个从节点正在运行
•Zeppelin安装在主节点上并运行
•使用shiro.ini文件和zeppelin启用了Shiro身份验证，shiro工作正常（尚未进行LDAP身份验证）
•Livy服务器安装在主节点上并运行
<property>
<name>hadoop.proxyuser.livy.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.livy.groups</name>
<value>*</value>
</property>
*************************************************************
Also livy.conf under /livy/conf has been ocnfigurd as follows:
*************************************************************
# What port to start the server on.
livy.server.port 8998
# What spark master Livy sessions should use.
livy.spark.master yarn
livy.impersonation.enabled true
***************************************************************************
From Zeppelin UI , I have configured the %livy interpreter with below values
***************************************************************************
livy.spark.master : local[*]
zeppelin.livy.url : http://localhost:8998
My Testing:
Logged into to Zeppelin as “user1“ successfully. And to test the connectivity between the Zeppelin - -> Shiro -- > Livy Integration I am running below simple codes .
Code 1:
%livy.spark
val accum = sc.accumulator(0)
sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)
accum.value
Output Error: Cannot start spark.
Code 2:
%livy.spark
sc.version
Output Error on zeppelin ui: --> Cannot start spark.
Code3:
%livy.pyspark
print("1")
Output Error: --> Cannot start spark.
Issue:
The notebook takes a while for running and throws error message "Cannot start spark." on the zeppelin UI against the notebook output.
Further while investigating the Log file “livy-livy-server.out” under ,Log File Path: /var/log/livy/
Below error is visible on the log file.
18/04/09 12:15:25 INFO WebServer: Starting server on http://IReplcaedmyHostNameFromHere:8999
18/04/09 12:17:56 INFO InteractiveSession$: Creating Interactive session 0: [owner: null, request: [kind: spark, proxyUser: Some(user1), conf: spark.master -> local[*], heartbeatTimeoutInSecond: 0]]
18/04/09 12:17:56 INFO RpcServer: Connected to the port 10001
18/04/09 12:17:56 WARN RSCConf: Your hostname, <My Host>, resolves to a loopback address, but we couldn't find any external IP address!
18/04/09 12:17:56 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
18/04/09 12:17:56 INFO InteractiveSessionManager: Registering new session 0
18/04/09 12:17:59 INFO LineBufferedStream: stdout: 18/04/09 12:17:59 INFO RSCDriver: Connecting to: IReplcaedmyHostNameFromHere.internal:10001
.
.
.
.
18/04/09 12:18:07 INFO LineBufferedStream: stdout: ERROR: org.apache.hadoop.security.authorize.AuthorizationException: User: livy is not allowed to impersonate user1