Many rsessions mostly idle with parallelism = "future"

[matthiasgomolka]

Prework

• Read and agree to the code of conduct and contributing guidelines.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
  • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • Readable: format your code according to the tidyverse style guide.

Before trying to create a reprex, I wanted to make sure there is no obvious short answer. If this is not the case, I'll try to create a reprex.

Description

I noticed that very often, many of the rsessions which should be used for parallel processing in my drake plan, are idle (see screenshot).

I'm sure that the general setup with future::plan() before drake::make is correct, because some targets really run in parallel as I expect.

Is there a possibility that this is related to setting format = "file" for dynamic targets? Whenever I noticed that only one session is actually active, drake was working on a dynamic target with format = "file".

Desired result

All dynamic subtargets should run in parallel.

Session info

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252    LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] knitr_1.30        arrow_4.0.0       lubridate_1.7.9   haven_2.4.0       readxl_1.3.1      fs_1.5.0          anytime_0.3.9    
 [8] stringi_1.5.3     purrr_0.3.4       data.table_1.14.0 drake_7.12.7      dobby_0.1.0.111   conflicted_1.0.4 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5          txtq_0.2.3          listenv_0.8.0       visNetwork_2.0.9    prettyunits_1.1.1   ps_1.4.0           
 [7] assertthat_0.2.1    digest_0.6.27       R6_2.5.0            cellranger_1.1.0    backports_1.2.0     evaluate_0.14      
[13] pillar_1.4.7        rlang_0.4.10        settings_0.2.4      progress_1.2.2      rstudioapi_0.13     callr_3.5.1        
[19] R.utils_2.10.1      R.oo_1.24.0         DT_0.16             checkmate_2.0.0     rmarkdown_2.5       qs_0.24.1          
[25] stringr_1.4.0       htmlwidgets_1.5.2   igraph_1.2.6        bit_4.0.4           validate_0.9.3      compiler_4.0.2     
[31] xfun_0.19           pkgconfig_2.0.3     askpass_1.1         globals_0.13.1      htmltools_0.5.0     openssl_1.4.3      
[37] tidyselect_1.1.0    tibble_3.0.4        codetools_0.2-16    future_1.19.1       crayon_1.3.4        R.methodsS3_1.8.1  
[43] jsonlite_1.7.1      lifecycle_0.2.0     magrittr_2.0.1      storr_1.2.4         RcppParallel_5.0.2  cli_2.5.0          
[49] cachem_1.0.4        renv_0.13.2         ellipsis_0.3.1      filelock_1.0.2      vctrs_0.3.5         generics_0.1.0     
[55] stringfish_0.15.1   fst_0.9.4           RApiSerialize_0.1.0 tools_4.0.2         forcats_0.5.0       bit64_4.0.5        
[61] glue_1.4.2          hms_0.5.3           processx_3.4.4      parallel_4.0.2      fastmap_1.0.1       yaml_2.2.1         
[67] base64url_1.4       memoise_2.0.0      

[wlandau]

make(parallelism = "future") deliberately waits for the checksums of files to agree before moving on to subsequent targets (to compensate for network file system-related delays, usually NFS on clusters). Otherwise, drake cannot be sure if downstream targets have the latest versions of upstream targets. Seems a likely explanation in your case.

[matthiasgomolka]

I understand that drake does not start downstream targets before all subtargets of a dynamic target with format = "file" are finished. This is fine.

But in my case, not just subsequent targets are waiting. I observe this behaviour for subtargets which all belong to the same dynamic target with format = "file". Is this also expected?

[wlandau]

Checksums get checked on a subtarget-by-subtarget basis, so yes, this bottleneck is unsurprising when those dynamic files do not live on your local machine.

[matthiasgomolka]

OK, then I think it's better to create a reprex.