Spark_apply in sparklyr:无法运行程序" Rscript":错误= 13,权限被拒绝

时间:2017-08-07 13:46:30

标签: apache-spark rscript sparklyr

我正在尝试在the spark_apply文档上运行第一个示例。

我使用的是基于Yarn的群集,我正在使用RStudio Server Pro运行:

library(sparklyr)
Sys.setenv(SPARK_HOME="/usr/hdp/2.6.0.3-8/spark/")
sc <- spark_connect(master = "yarn-client")
sdf_len(sc, 5, repartition = 1) %>% spark_apply(function(e) I(e))

有时会过去,然后我收到此错误消息:

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, an061.e.mindstorm): java.io.IOException: Cannot run program "Rscript": error=13, Permission denied

它说Rscript被许可拒绝,但它应该具有世界rx权限:

[hahad@an007 ~]$ ls -l /ifs/opt/app/R/3.3.3-x11/bin/Rscript
-rwxr-xr-x 1 gcadmin gcadmin 22556 Jun 15 13:21 /ifs/opt/app/R/3.3.3-x11/bin/Rscript

在我获得权限被拒绝的机器上,是工作节点,是驱动程序还是RStudio服务器?

当程序显然可执行时,为什么我会收到错误?

我的猜测是,某种其他版本的R正在参与其中。工作人员将以哪个用户身份运行?

强制版本的东西:

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ggplot2_2.2.1       dplyr_0.7.2         DBI_0.7             sparklyr_0.7.0-9000 RODBC_1.3-15       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.11     dbplyr_1.1.0     plyr_1.8.4       bindr_0.1        base64enc_0.1-3  tools_3.3.3      digest_0.6.12   
 [8] gtable_0.2.0     jsonlite_1.5     tibble_1.3.3     nlme_3.1-131     lattice_0.20-35  pkgconfig_2.0.1  rlang_0.1.1     
[15] psych_1.7.5      shiny_1.0.3      rstudioapi_0.6   yaml_2.1.14      parallel_3.3.3   bindrcpp_0.2     withr_2.0.0     
[22] httr_1.2.1       stringr_1.2.0    rprojroot_1.2    grid_3.3.3       glue_1.1.0       R6_2.2.2         foreign_0.8-68  
[29] reshape2_1.4.2   tidyr_0.6.3      magrittr_1.5     scales_0.4.1     backports_1.1.0  htmltools_0.3.6  assertthat_0.2.0
[36] mnormt_1.5-5     colorspace_1.3-2 mime_0.5         xtable_1.8-2     httpuv_1.3.3     config_0.2       stringi_1.1.5   
[43] munsell_0.4.3    lazyeval_0.2.0   broom_0.4.2

0 个答案:

没有答案