我正在尝试连接RStudio的Spark。当前,我们正在使用运行Spark(2.2)的Cloudera Hadoop发行版。我从边缘节点测试了所有内容,能够创建Spark上下文并执行查询。直到昨天RStudio一切正常,突然我们收到RStudio的问题。
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (14727): Sparklyr gateway did not respond while retrieving ports information after 60 seconds
Path: /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/bin/spark-submit
Parameters: --class, sparklyr.Shell, '/usr/lib64/R/library/sparklyr/java/sparklyr-2.2-2.11.jar', 8880, 14727
Log: /tmp/RtmpoNJQEH/file151b437c0313b_spark.log
---- Output Log ----
18/11/12 13:54:50 INFO sparklyr: Session (14727) is starting under 127.0.0.1 port 8880
18/11/12 13:54:50 INFO sparklyr: Session (14727) found port 8880 is not available
18/11/12 13:54:50 INFO sparklyr: Backend (14727) found port 8884 is available
18/11/12 13:54:50 INFO sparklyr: Backend (14727) is registering session in gateway
18/11/12 13:54:50 INFO sparklyr: Backend (14727) is waiting for registration in gateway
---- Error Log ----
{{1}}
我也验证了sparklyr的版本,它是0.9.2
能不能让我知道怎么了?
答案 0 :(得分:1)
您可以尝试
library(httr)
library(sparklyr)
Sys.setenv(SPARK_HOME = '/opt/cloudera/parcels/SPARK2/lib/spark2')
Sys.setenv(YARN_CONF_DIR = '/opt/cloudera/parcels/SPARK2/lib/spark2/conf/yarn-conf/')
config <- list()
config=c(config,list("sparklyr.shell.deploy-mode"="client"))
httr::with_config(
config = httr::authenticate(user=":", password="", type="gssnegotiate"),
sc <- spark_connect(master = "yarn-client", version = "2.2.0", config = config))
sc
如果启用了SSL和Kerberos,则可能需要使用此选项
library(httr)
library(sparklyr)
set_config(config(cainfo = "/opt/cloudera/security/global_cacerts.pem"))
Sys.setenv(SPARK_HOME = '/opt/cloudera/parcels/SPARK2/lib/spark2')
Sys.setenv(YARN_CONF_DIR = '/opt/cloudera/parcels/SPARK2/lib/spark2/conf/yarn-conf/')
config <- list()
config=c(config,list("sparklyr.shell.keytab"="/PATH/PATH.keytab",
"sparklyr.shell.principal"="user@DOMAIN.COM",
"sparklyr.shell.deploy-mode"="client"
))
httr::with_config(
config = httr::authenticate(user=":", password="", type="gssnegotiate"),
sc <- spark_connect(master = "yarn-client", version = "2.2.0", config = config))
sc
注意:将cainfo替换为pem格式的Root CA路径,在sparklyr.shell.keytab中指定用户keytab,在sparklyr.shell.principal中指定UPN(用户主体名称)