在centos集群上使用doparallel和foreach的问题

时间:2019-03-18 16:18:45

标签: r foreach centos7 doparallel

我正在尝试使用foreachdoParallel来优化某些图像处理的计算。我有ParamIter这是要迭代的参数。 ImageProcessing是我的自定义函数,用于处理图像并保存已处理的数据。

library(methods)
library(fftwtools)
library(EBImage)
library(tidyverse)
library(foreach)
library(doParallel)

registerDoParallel(20)

      foreach(ParamIter = unique(AllImages$ParamIter)) %dopar% {

        AllImagesTemp = AllImages[AllImages$ParamIter == ParamIter,]

        ImageProcessing(Image = AllImagesTemp,
                        Plate = unique(AllImagesTemp$Plate),
                        TimePoint = unique(AllImagesTemp$TimePoint),
                        Marker = unique(AllImagesTemp$Marker),
                        Replicate = unique(AllImagesTemp$Replicate),
                        Well = unique(AllImagesTemp$ID),
                        Position = unique(AllImagesTemp$Position),
                        SaveDir = WellDir,
                        SaveParam = ParamIter,
                        ThrEmptyImage = .04)

        }

在具有Ubuntu 18.04和会话的机器上,一切都在本地完美运行,如下所示:

> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=de_DE.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=de_DE.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=de_DE.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] bindrcpp_0.2.2    fftwtools_0.9-8   doParallel_1.0.14 iterators_1.0.10  foreach_1.4.4     forcats_0.3.0     stringr_1.4.0     dplyr_0.7.7      
 [9] purrr_0.3.1       readr_1.1.1       tidyr_0.8.2       tibble_2.0.1      ggplot2_3.1.0     tidyverse_1.2.1   EBImage_4.24.0   

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5    locfit_1.5-9.1      haven_1.1.2         lattice_0.20-38     colorspace_1.4-0    htmltools_0.3.6     yaml_2.2.0         
 [8] rlang_0.3.1         pillar_1.3.1        withr_2.1.2         glue_1.3.0          BiocGenerics_0.28.0 modelr_0.1.2        readxl_1.1.0       
[15] jpeg_0.1-8          bindr_0.1.1         plyr_1.8.4          munsell_0.5.0       gtable_0.2.0        cellranger_1.1.0    rvest_0.3.2        
[22] htmlwidgets_1.3     codetools_0.2-16    broom_0.5.0         Rcpp_1.0.0          backports_1.1.3     scales_1.0.0        jsonlite_1.6       
[29] abind_1.4-5         hms_0.4.2           png_0.1-7           digest_0.6.18       stringi_1.3.1       tiff_0.1-5          grid_3.5.3         
[36] cli_1.0.1           tools_3.5.3         bitops_1.0-6        magrittr_1.5        RCurl_1.95-4.11     lazyeval_0.2.1      crayon_1.3.4       
[43] pkgconfig_2.0.2     xml2_1.2.0          lubridate_1.7.4     assertthat_0.2.0    httr_1.4.0          rstudioapi_0.9.0    R6_2.3.0           
[50] nlme_3.1-137        compiler_3.5.3 

但是,如果我尝试在本地群集上执行此操作,它将无法正常工作并为我提供此错误:

Error in { : 
  task 1 failed - "a call to callNextMethod() appears in a call to ‘.Method’, but the call does not seem to come from either a generic function or another 'callNextMethod'"
Calls: %dopar% -> <Anonymous>
Execution halted

我在centos群集上的会话如下:

R version 3.4.1 (2017-06-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS: /cm/shared/apps/R/64/3.4.1/lib64/R/lib/libRblas.so
LAPACK: /cm/shared/apps/R/64/3.4.1/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] doParallel_1.0.14 iterators_1.0.10  foreach_1.4.4     forcats_0.3.0    
 [5] stringr_1.3.1     dplyr_0.7.8       purrr_0.3.0       readr_1.3.1      
 [9] tidyr_0.8.2       tibble_2.0.1      ggplot2_3.1.0     tidyverse_1.2.1  
[13] EBImage_4.20.1    fftwtools_0.9-8  

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5    locfit_1.5-9.1      haven_2.0.0        
 [4] lattice_0.20-35     colorspace_1.4-0    generics_0.0.2     
 [7] htmltools_0.3.6     rlang_0.3.1         pillar_1.3.1       
[10] withr_2.1.2         glue_1.3.0          BiocGenerics_0.24.0
[13] modelr_0.1.2        readxl_1.2.0        bindrcpp_0.2.2     
[16] jpeg_0.1-8          bindr_0.1.1         plyr_1.8.4         
[19] munsell_0.5.0       gtable_0.2.0        cellranger_1.1.0   
[22] rvest_0.3.2         htmlwidgets_1.3     codetools_0.2-15   
[25] broom_0.5.1         Rcpp_1.0.0          scales_1.0.0       
[28] backports_1.1.3     jsonlite_1.6        abind_1.4-5        
[31] hms_0.4.2           png_0.1-7           digest_0.6.18      
[34] stringi_1.2.4       tiff_0.1-5          grid_3.4.1         
[37] cli_1.0.1           bitops_1.0-6        tools_3.4.1        
[40] magrittr_1.5        RCurl_1.95-4.10     lazyeval_0.2.1     
[43] crayon_1.3.4        pkgconfig_2.0.2     xml2_1.2.0         
[46] lubridate_1.7.4     assertthat_0.2.0    httr_1.4.0         
[49] rstudioapi_0.9.0    R6_2.3.0            nlme_3.1-131       
[52] compiler_3.4.1     

我不了解该错误及其对我的影响。有人对我该如何解决这个问题有想法吗?

在我的本地群集上不可能升级到R的较新版本。

干杯

Mathieu

0 个答案:

没有答案