将left_joint与sparklyr一起使用时删除了表格

Question

我正在使用一些我想要加入的表，因为我使用了sparklyr（由于表大小）和left_joint的dplyr。 这是代码示例：

query.1 <- left_join(pa11, pa12, by = c("CODIGO_HAB_D","ID_EST","ID_ME","ID_PARTE_D","ID_PAR", "ID_REP")) %>% left_join(., pa13, by = c("ID_SINI" = "ID_SINI"))

query.1 <- left_join(query.1, a14, by = "ID_REP" )
query.1 <-left_join(query.1, a16, by = c("ID_MEJ" = "ID_ME"))
query.1 <-left_join(query.1, a17, by = c("ID_EST"  = "ID_ESTE"))
query.1 <-left_join(query.1, a18, by = "ID_PARTE_D" )
query.1 <-left_join(query.1, a19, by = "CODI" )
query.1 <-left_join(query.1, a110, by = c("ID_PROF.x" = "ID_PROF" ))
query.1 <-left_join(query.1, a111, by = c("ID_COM.x" = "ID_COM" ))
query.1 <-left_join(query.1, a113, by = c("ID_GRANDES.x" = "ID_GRANDES"))

当我left_joint 5个第一张表时，一切都按预期进行。当我用更多的表重复这个时，我得到了这个错误

Error in as.vector(x, "character") : 
cannot coerce type 'environment' to vector of type 'character'

然后，当我试着看一下Spark表时，我在Rstudio中遇到错误。

Answer 1

由于其他原因，我不时会收到这些错误。

根据我的经验，增加Sparklyr内存和执行程序开销记忆有助于

    config <- spark_config()
    config$`sparklyr.shell.driver-memory` <- "8G"
    config$`sparklyr.shell.executor-memory` <- "8G"
    config$spark.yarn.executor.memoryOverhead <- "2g"