我正在尝试在the spark_apply文档上运行第一个示例。
我使用的是基于Yarn的群集,我正在使用RStudio Server Pro运行:
library(sparklyr)
Sys.setenv(SPARK_HOME="/usr/hdp/2.6.0.3-8/spark/")
sc <- spark_connect(master = "yarn-client")
sdf_len(sc, 5, repartition = 1) %>% spark_apply(function(e) I(e))
有时会过去,然后我收到此错误消息:
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, an061.e.mindstorm): java.io.IOException: Cannot run program "Rscript": error=13, Permission denied
它说Rscript被许可拒绝,但它应该具有世界rx权限:
[hahad@an007 ~]$ ls -l /ifs/opt/app/R/3.3.3-x11/bin/Rscript
-rwxr-xr-x 1 gcadmin gcadmin 22556 Jun 15 13:21 /ifs/opt/app/R/3.3.3-x11/bin/Rscript
在我获得权限被拒绝的机器上,是工作节点,是驱动程序还是RStudio服务器?
当程序显然可执行时,为什么我会收到错误?
我的猜测是,某种其他版本的R正在参与其中。工作人员将以哪个用户身份运行?
强制版本的东西:
> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] ggplot2_2.2.1 dplyr_0.7.2 DBI_0.7 sparklyr_0.7.0-9000 RODBC_1.3-15
loaded via a namespace (and not attached):
[1] Rcpp_0.12.11 dbplyr_1.1.0 plyr_1.8.4 bindr_0.1 base64enc_0.1-3 tools_3.3.3 digest_0.6.12
[8] gtable_0.2.0 jsonlite_1.5 tibble_1.3.3 nlme_3.1-131 lattice_0.20-35 pkgconfig_2.0.1 rlang_0.1.1
[15] psych_1.7.5 shiny_1.0.3 rstudioapi_0.6 yaml_2.1.14 parallel_3.3.3 bindrcpp_0.2 withr_2.0.0
[22] httr_1.2.1 stringr_1.2.0 rprojroot_1.2 grid_3.3.3 glue_1.1.0 R6_2.2.2 foreign_0.8-68
[29] reshape2_1.4.2 tidyr_0.6.3 magrittr_1.5 scales_0.4.1 backports_1.1.0 htmltools_0.3.6 assertthat_0.2.0
[36] mnormt_1.5-5 colorspace_1.3-2 mime_0.5 xtable_1.8-2 httpuv_1.3.3 config_0.2 stringi_1.1.5
[43] munsell_0.4.3 lazyeval_0.2.0 broom_0.4.2