我正在共享代码，日志和控制台输出。
###### this is global code.
##### No error in this. Spark is getting connected.
config <- spark_config()
config[["sparklyr.defaultPackages"]] <- NULL
config$sparklyr.gateway.port <- 8880
sc <- spark_connect(master = "local", config = config, spark_home = Sys.getenv("SPARK_HOME"))
在一种方法中使用以下代码时出错：
add_aggr_cols = function(DF=NULL,par=NULL,ZONE=FALSE,ODID=FALSE,type="RANGE"){
range<-sdf_copy_to(sc,DF0[,select_cols[grep("_OD",end_char)]])
}
我认为错误所在的火花日志。网关即将终止。
19/07/09 14:02:38 INFO sparklyr: Gateway (7119) is terminating backend
(19/07/09 14:02:38 INFO sparklyr: Gateway (7119) is shutting down with expected SocketException,java.net.SocketException: Socket closed)
19/07/09 14:02:38 INFO SparkContext: Invoking stop() from shutdown hook
19/07/09 14:02:38 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
19/07/09 14:02:38 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/09 14:02:38 INFO MemoryStore: MemoryStore cleared
19/07/09 14:02:38 INFO BlockManager: BlockManager stopped
19/07/09 14:02:38 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/09 14:02:38 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/09 14:02:38 INFO SparkContext: Successfully stopped SparkContext
19/07/09 14:02:38 INFO ShutdownHookManager: Shutdown hook called
Everything is set up. Spark home, connection
$app_name
[1] "sparklyr"
$config
$config$spark.env.SPARK_LOCAL_IP.local
[1] "127.0.0.1"
$config$sparklyr.connect.csv.embedded
[1] "^1.*"
$config$spark.sql.catalogImplementation
[1] "hive"
$config$sparklyr.connect.cores.local
[1] 12
$config$spark.sql.shuffle.partitions.local
[1] 12
$config$sparklyr.gateway.port
[1] 8880
$config$`sparklyr.shell.driver-memory`
[1] "2g"
attr(,"config")
[1] "default"
attr(,"file")
[1] "/Library/Frameworks/R.framework/Versions/3.6/Resources/library/sparklyr/conf/config-template.yml"
$state
<environment: 0x7f825acca8c8>
$extensions
$extensions$jars
character(0)
$extensions$packages
character(0)
$extensions$initializers
list()
$extensions$catalog_jars
character(0)
$spark_home
[1] "/Users/jk1000263410/spark/spark-2.4.3-bin-hadoop2.7"
$backend
A connection with
description "->localhost:8881"
class "sockconn"
mode "wb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$monitoring
A connection with
description "->localhost:8881"
class "sockconn"
mode "wb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$gateway
A connection with
description "->localhost:8880"
class "sockconn"
mode "rb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$output_file
[1] "/var/folders/45/n359jszj6zn2bd9nf_b0rhdw0000gp/T//RtmppmOxWo/file3c3b353bc548_spark.log"
$sessionId
[1] 7119
$home_version
[1] "2.4.3"
attr(,"class")
[1] "spark_connection" "spark_shell_connection" "DBIConnection"
当执行到达sdf_copy_to（）函数时，它将给出错误。到此为止，它如上所述可以正常工作。