apache-spark - 无法从hue服务器

我正在尝试从hue服务器创建一个spark会话，该服务器已配置了livy服务器以与启用kerberos的spark通信。

尝试使用查询编辑器启动spark scala会话时出现以下错误。但我可以从色调服务器提交火花罐工作，它工作正常。

堆栈跟踪：

 


     15:21:40 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@70609b0c","\nYARN Diagnostics: ","Application killed by user."]}
    [23/Apr/2018 02:52:05 +0000] decorators   ERROR    Error running create_session
    Traceback (most recent call last):
      File "/home/hue/desktop/libs/notebook/src/notebook/decorators.py", line 97, in decorator
        return func(*args, **kwargs)
      File "/home/hue/desktop/libs/notebook/src/notebook/api.py", line 85, in create_session
        response['session'] = get_api(request, session).create_session(lang=session['type'], properties=properties)
      File "/home/hue/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 227, in create_session
        raise QueryError(_('The Spark session could not be created in the cluster: %s') % info)
    QueryError

使用kerberos凭据配置了livy，并且使用curl命令与spark通信时也能正常工作。

Hue配置：



     livy_server_host=localhost
      livy_server_port=8998 
      livy_server_session_kind=yarn
      livy_impersonation_enabled=true 
      livy_server_session_timeout=3600000

Livy配置：


    livy.server.csrf_protection.enabled = false
    livy.spark.master = yarn
    livy.spark.deployMode = cluster
    livy.superusers = hue,livy
    livy.impersonation.enabled = true
    livy.server.access_control.enabled = true 
    livy.server.access_control.users = *
    livy.server.auth.kerberos.keytab livy.keytab
    livy.server.auth.kerberos.principal HTTP/livyserver@FQDN
    #livy.server.auth.type kerberos
    livy.server.launch.kerberos.keytab hue.keytab
    livy.server.launch.kerberos.principal hue/hueserver@FQDN
    livy.server.session.timeout = 5h
    livy.server.request-log-retain.days = 5

还在hadoop core-site.xml中添加了hue和livy的代理用户

无法从hue服务器

0 个答案: