Question

我正在尝试使this R script for a speed test适用于集群。

在类型为sfInit的情况下使用makecluster和"SOCK"函数时，脚本可以在群集上成功运行，但是速度没有任何提高-与我的计算机不同：当我更改时从detectcores()到1，该脚本的运行速度明显慢于4核。

不过，我非常确定我需要将类型更改为"MPI"，以使节点之间以内存方式进行通信。

但是：如果这样做，脚本将停止并显示以下错误代码：

Loading required package: Rmpi
Error: package or namespace load failed for ‘Rmpi’:
 .onLoad failed in loadNamespace() for 'Rmpi', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/cluster/sfw/R/3.5.1-gcc73-base/lib64/R/library/Rmpi/libs/Rmpi.so':
  libmpi.so.20: cannot open shared object file: No such file or directory
Failed to load required library: Rmpi for parallel mode MPI
Fallback to sequential execution
snowfall 1.84-6.1 initialized: sequential execution, one CPU.

我认为“轻松一点”，并添加了以下几行：

install.packages('Rmpi', repos = "http://cran.us.r-project.org",
dependencies = TRUE, lib = '/personalpath') install.packages('doMPI',
repos = "http://cran.us.r-project.org", dependencies = TRUE, lib = '/personalpath') library(topicmodels, lib.loc = '/personalpath')
library(Rmpi, lib.loc = '/personalpath')

这将导致安装成功，但是：

Error in library(Rmpi, lib.loc = "/personalpath") :
there is no package called ‘Rmpi’

1。如何安装这些软件包？

2。我真的需要安装它们吗，或者这是一种完全错误的方法？

我们非常感谢您的帮助！我知道这里有几个问题（请参见this，this和this）。但是我不熟悉Linux中的调用，更重要的是，我对该集群没有任何权限。所以我需要在R中提出一个解决方案...

所以..这是我的代码：

sfInit(parallel=TRUE, cpus=detectCores(), type="MPI")

cl <- makeCluster(detectCores(), type = "MPI")
registerDoSNOW(cl) 

sfExport('dtm_stripped', 'control_LDA_Gibbs')
sfLibrary(topicmodels)

clusterEvalQ(cl, library(topicmodels))
clusterExport(cl, c("dtm_stripped", "control_LDA_Gibbs"))

BASE <- system.time(best.model.BASE <<- lapply(seq, function(d){LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)}))
PLYR_S <- system.time(best.model.PLYR_S <<- llply(seq, function(d){LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)}, .progress = "text"))

wrapper <- function (d) topicmodels:::LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)
PARLAP <- system.time(best.model.PARLAP <<- parLapply(cl, seq, wrapper))
DOPAR <- system.time(best.model.DOPAR <<- foreach(i = seq, .export = c("dtm_stripped", "control_LDA_Gibbs"), .packages = "topicmodels", .verbose = TRUE) %dopar% (LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', k=i)))
SFLAPP <- system.time(best.model.SFLAPP <<- sfLapply(seq, function(d){topicmodels:::LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)})) 
SFCLU <- system.time(best.model.SFCLU <<- sfClusterApplyLB(seq, function(d){topicmodels:::LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)})) 
PLYRP <- system.time(best.model.PLYRP <<- llply(seq, function(d){topicmodels:::LDA(dtm_stripped, control = control_LDA_Gibbs, method ='Gibbs', d)}, .parallel = TRUE))

results_speedtest <- rbind(BASE, PLYR_S, PARLAP, DOPAR, SFLAPP, SFCLU, PLYRP)
print(results_speedtest)

Answer 1

还有其他在R中并行化的方法。如第二页所述，此链接可能会有所帮助，例如socket，mpi和fork等这些群集类型的作用： https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf

否则，我也可以建议检查软件包foreach，因为语法更像是常规的for循环。请注意，某些并行化软件包并非对所有操作系统都可用。

如何使用sfInit和makeCluster类型“ MPI” /在集群中的R /并行化中传递消息

1 个答案: