Draft in progress: address #71 `with_nix()` #76

philipp-baumann · 2023-09-12T12:21:27Z

@b-rodrigues not for merging yet ;-)

Merge branch 'master' into 71-with-nix # Conflicts: # .Rbuildignore

…n host R and nix-R

b-rodrigues · 2023-10-08T19:24:05Z

dev/build_envs.Rmd

+  #    into host R session
+
+  cmd <- switch(program,
+    # do 2), 3), 4) in nix-shell-R session (check how to deal with shellHook)


I guess we should run the R expression with R --vanilla?

… disk

philipp-baumann · 2023-10-15T21:54:38Z

The logic is now even simpler. Here a short example of running

with_nix(expr = function(m = mtcars) nrow(mtcars), exec_mode = "non-blocking")

In step 2 (get formals/arguments in nix-R), we now just write an R script file. This has two advantages. 1) we can save us some eval-parse round, and 2) it is way easier to run multi-line R code via Rscript, in fact it is recommended to use source() in those cases.

Here's the resulting scriptfile in the temp_folder:

philipp@philipp-private:/tmp/RtmpOPLmZu$ cat with_nix_r.R 
#!/bin/sh
temp_dir <- "/tmp/RtmpOPLmZu"
r_version_num <- paste0(R.version$major, ".", R.version$minor)
# assign `args_vec` as in c(...) form.
args_vec <- c(m = "mtcars")
# assign elements of `args_vec` individually
m <- "mtcars"
# actual deserialization step
read_args <- function (args_vec, temp_dir) 
{
    for (i in seq_len(length(args_vec))) {
        nm <- args_vec[i]
        obj <- names(args_vec)[i]
        assign(nm, readRDS(file = file.path(temp_dir, paste0(obj, ".Rds"))))
        cat(paste0("reading ", obj, ".Rds"))
    }
}
# evaluate function
read_args(args_vec, temp_dir)
ls()

philipp-baumann · 2023-10-22T22:08:56Z

f23de2d is a first step towards boilderplating code that loads extra packages which are defined in globals' environments found in the function supplied to expr. These extra packages will be later attached in the Nix-R session.

This means we can soon deal with things like:

library("dplyr")

nrow_mtcars <- function(m = mtcars) {
  a <- datasets::mtcars |> filter(cyl == 6L)
  nrow_impl(a)
}

nrow_impl <- function(a) nrow(a)

In this example, extra packages to be attached in nix-R is "dplyr". If a qualified call would have been made, no extra packages would be necessary:

nrow_mtcars <- function(m = mtcars) {
  a <- datasets::mtcars |> dplyr::filter(cyl == 6L)
  nrow_impl(a)
}

philipp-baumann · 2023-11-01T21:22:35Z

@b-rodrigues FYI, I am now testing this example. If you have further scenarios of a typical expr setting in terms of the abstract syntax tree, the environment stacking etc., would be great to hear :-)

library("dplyr")
library("data.table")

env_a <- new.env()
env_a$key <- "value"

nrow_impl <- function(a) {
  not_returned <- data.table(a = 1:3)
  cat("`not_returned`", "has", nrow(not_used), "rows.")
  nrow(a)
}

nrow_mtcars <- function(data = mtcars) {
  a <- data |> filter(cyl == 6L)
  cat("Object outside of function scope is of class: ", class(env_a))
  nrow_impl(a)
}

with_nix(
  expr = nrow_mtcars,
  exec_mode = "non-blocking",
  project_path = "inst/extdata/with_nix"
)

… objects

philipp-baumann · 2023-11-10T22:18:41Z

make R-nix code layer bullet proof for this case:

with_nix(
  expr = function(m) print(m),
  exec_mode = "non-blocking",
  project_path = "inst/extdata/with_nix"
)

Code runs, but it currently evaluates to symbol due to code use d here

https://github.com/b-rodrigues/rix/blob/ff9d67d4d91421a2afdab9cf44596e99366a4754/R/find_rev.R#L979

and here

https://github.com/b-rodrigues/rix/blob/ff9d67d4d91421a2afdab9cf44596e99366a4754/R/find_rev.R#L901-L907

which try to deal with formal args without defaults (empty symbols in formals/pairlist).

Meaning we get

> with_nix(
+   expr = function(m) print(m),
+   exec_mode = "non-blocking",
+   project_path = "inst/extdata/with_nix"
+ )
* checking code in `expr` for potential problems:
 `codetools::checkUsage(fun = expr)`

==> Running deparsed expression via `nix-shell` in non-blocking mode:

{
    cat("\n* wrote R script evaluated via `Rscript` in `nix-shell`:", "/tmp/RtmpA2hL1A/with_nix_r.R")
    temp_dir <- "/tmp/RtmpA2hL1A"
    r_version_num <- paste0(R.version$major, ".", R.version$minor)
    cat("\n* using Nix with R version", r_version_num, "\n\n")
    args_vec <- c("m")
    for (i in seq_along(args_vec)) {
        nm <- args_vec[i]
        obj <- args_vec[i]
        assign(x = nm, value = readRDS(file = file.path(temp_dir, paste0(obj, ".Rds"))))
        cat(paste0("  => reading ", obj, ".Rds", " for argument named `", obj, "`\n"))
    }
    lst <- as.list(args_vec)
    names(lst) <- args_vec
    lst <- lapply(lst, as.name)
    rnix_out <- do.call(function (m) 
    print(m), lst)
    typeof(rnix_out)
    cat("\n* called `expr` with args", args_vec, ":")
    cat("\n", deparse(function (m) 
    print(m)))
    cat("\n\n* the following objects are in the global environment:\n")
    cat(ls())
    cat("\n* `sessionInfo()` output:\n")
    cat(capture.output(sessionInfo()), sep = "\n")
}
==> Process ID (PID) is 78333.
==> Receiving stdout and stderr streams...

* wrote R script evaluated via `Rscript` in `nix-shell`: /tmp/RtmpA2hL1A/with_nix_r.R
* using Nix with R version 4.3.1 

  => reading m.Rds for argument named `m`
m

* called `expr` with args m :
 function (m)  print(m)

* the following objects are in the global environment:
args_vec i lst m nm obj r_version_num rnix_out temp_dir
* `sessionInfo()` output:
R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS/LAPACK: /nix/store/8lqs4s515p2597vl838mrzb03qgwwvag-blas-3/lib/libblas.so.3;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Zurich
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.3.1

==> `expr` succeeded!

philipp-baumann · 2023-11-15T16:39:12Z

On macOS, nix-shell is not found, because path seems not exported. This works. The same problem for base::Sys.which("nix-shell").

One way is to hard-code and have a switch for macOS

sys::exec_internal(cmd = "/nix/var/nix/profiles/default/bin/nix-shell", cmd_rnix_deparsed)

The other way I could resolve was following NixOS/nix#8831 and linked https://gist.github.com/Linerre/f11ad4a6a934dcf01ee8415c9457e7b2

It was not sufficient to add those lines to ~/.zshrc, but for ~/.zshenv it worked :-)

export PATH=$PATH:/nix/var/nix/profiles/default/bin/

This path issue I only find in RStudio, but not in the terminal outside RStudio, even when I do not set this in zsh.

This is the session info:

> sessioninfo::session_info()
─ Session info ────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.1 (2023-06-16)
 os       macOS Sonoma 14.0
 system   aarch64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/Zurich
 date     2023-11-15
 rstudio  2023.09.1+494 Desert Sunflower (desktop)
 pandoc   3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)

─ Packages ────────────────────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 attachment    0.4.0   2023-05-31 [1] CRAN (R 4.3.0)
 backports     1.4.1   2021-12-13 [1] CRAN (R 4.3.0)
 brio          1.1.3   2021-11-30 [1] CRAN (R 4.3.0)
 cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.0)
 callr         3.7.3   2022-11-02 [1] CRAN (R 4.3.0)
 checkmate     2.2.0   2023-04-27 [1] CRAN (R 4.3.0)
 cli           3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
 codetools     0.2-19  2023-02-01 [1] CRAN (R 4.3.1)
 crayon        1.5.2   2022-09-29 [1] CRAN (R 4.3.0)
 desc          1.4.2   2022-09-08 [1] CRAN (R 4.3.0)
 devtools      2.4.5   2022-10-11 [1] CRAN (R 4.3.0)
 digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.0)
 dplyr         1.1.3   2023-09-03 [1] CRAN (R 4.3.0)
 ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.3.0)
 evaluate      0.22    2023-09-29 [1] CRAN (R 4.3.1)
 fansi         1.0.5   2023-10-08 [1] CRAN (R 4.3.1)
 fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
 fs            1.6.3   2023-07-20 [1] CRAN (R 4.3.0)
 fusen         0.5.2   2023-08-17 [1] CRAN (R 4.3.0)
 generics      0.1.3   2022-07-05 [1] CRAN (R 4.3.0)
 glue          1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
 hms           1.1.3   2023-03-21 [1] CRAN (R 4.3.0)
 htmltools     0.5.6.1 2023-10-06 [1] CRAN (R 4.3.1)
 htmlwidgets   1.6.2   2023-03-17 [1] CRAN (R 4.3.0)
 httpuv        1.6.11  2023-05-11 [1] CRAN (R 4.3.0)
 httr          1.4.7   2023-08-15 [1] CRAN (R 4.3.0)
 jsonlite      1.8.7   2023-06-29 [1] CRAN (R 4.3.0)
 knitr         1.44    2023-09-11 [1] CRAN (R 4.3.0)
 later         1.3.1   2023-05-02 [1] CRAN (R 4.3.0)
 lifecycle     1.0.3   2022-10-07 [1] CRAN (R 4.3.0)
 magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
 memoise       2.0.1   2021-11-26 [1] CRAN (R 4.3.0)
 mime          0.12    2021-09-28 [1] CRAN (R 4.3.0)
 miniUI        0.1.1.1 2018-05-18 [1] CRAN (R 4.3.0)
 parsermd      0.1.2   2021-05-20 [1] CRAN (R 4.3.0)
 pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.0)
 pkgbuild      1.4.2   2023-06-26 [1] CRAN (R 4.3.0)
 pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.0)
 pkgload       1.3.3   2023-09-22 [1] CRAN (R 4.3.1)
 prettyunits   1.2.0   2023-09-24 [1] CRAN (R 4.3.1)
 processx      3.8.2   2023-06-30 [1] CRAN (R 4.3.0)
 profvis       0.3.8   2023-05-02 [1] CRAN (R 4.3.0)
 promises      1.2.1   2023-08-10 [1] CRAN (R 4.3.0)
 ps            1.7.5   2023-04-18 [1] CRAN (R 4.3.0)
 purrr         1.0.2   2023-08-10 [1] CRAN (R 4.3.0)
 R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
 Rcpp          1.0.11  2023-07-06 [1] CRAN (R 4.3.0)
 readr         2.1.4   2023-02-10 [1] CRAN (R 4.3.0)
 remotes       2.4.2.1 2023-07-18 [1] CRAN (R 4.3.0)
 rlang         1.1.1   2023-04-28 [1] CRAN (R 4.3.0)
 rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
 roxygen2      7.2.3   2022-12-08 [1] CRAN (R 4.3.0)
 rprojroot     2.0.3   2022-04-02 [1] CRAN (R 4.3.0)
 rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.0)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
 shiny         1.7.5.1 2023-10-14 [1] CRAN (R 4.3.1)
 stringi       1.7.12  2023-01-11 [1] CRAN (R 4.3.0)
 stringr       1.5.0   2022-12-02 [1] CRAN (R 4.3.0)
 sys         * 3.4.2   2023-05-23 [1] CRAN (R 4.3.0)
 testthat      3.2.0   2023-10-06 [1] CRAN (R 4.3.1)
 tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.0)
 tidyr         1.3.0   2023-01-24 [1] CRAN (R 4.3.0)
 tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.0)
 tzdb          0.4.0   2023-05-12 [1] CRAN (R 4.3.0)
 urlchecker    1.0.1   2021-11-30 [1] CRAN (R 4.3.0)
 usethis       2.2.2   2023-07-06 [1] CRAN (R 4.3.0)
 utf8          1.2.3   2023-01-31 [1] CRAN (R 4.3.0)
 vctrs         0.6.4   2023-10-12 [1] CRAN (R 4.3.1)
 withr         2.5.1   2023-09-26 [1] CRAN (R 4.3.1)
 xfun          0.40    2023-08-09 [1] CRAN (R 4.3.0)
 xml2          1.3.5   2023-07-06 [1] CRAN (R 4.3.0)
 xtable        1.8-4   2019-04-21 [1] CRAN (R 4.3.0)
 yaml          2.3.7   2023-01-23 [1] CRAN (R 4.3.0)

 [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library

philipp-baumann · 2023-12-29T20:58:53Z

Testing current with_nix() via

# test without long inflation time
fusen::load_flat_functions(flat_file = "dev/build_envs.Rmd")
fusen::load_flat_functions(flat_file = "dev/get_os.Rmd")

rix::rix(
  r_ver = "latest", 
  r_pkgs = c("dplyr", "data.table"),
  project_path = ".",
  overwrite = TRUE
)

library("dplyr")
library("data.table")

env_a <- new.env()
env_a$key <- "value"

nrow_impl <- function(a) {
  not_returned <- data.table(a = 1:3)
  cat("`not_returned`", "has", nrow(not_used), "rows.")
  df_check(a)
  nrow(a)
}

df_check <- function(x) stopifnot("x must be data.frame" = is_df(x))

# critical, not assigned
is_df <- function(x) is.data.frame(x)

nrow_mtcars <- function(m = mtcars) {
  a <- m |> filter(cyl == 6L)
  cat("Object outside of function scope is of class: ", class(env_a))
  nrow_impl(a)
  test_df(a)
}

test_df <- function(a) is_df_a(a)

is_df_a <- function(a) is.data.frame(a)

with_nix(
  expr = nrow_mtcars,
  exec_mode = "non-blocking",
  project_path = "."
)

this finds the relevant bits of code to be exported correctly, when putting a browser
after this line

globals_expr <- recurse_find_check_globals(expr, args_vec)
browser()

Browse[1]> globals_expr
$pkgs
pkgs         pkgs 
"dplyr" "data.table" 

$globalenv_fun
nrow_impl       test_df      df_check       is_df_a         is_df 
"R_GlobalEnv" "R_GlobalEnv" "R_GlobalEnv" "R_GlobalEnv" "R_GlobalEnv" 

$globalenv_other
env_a 
"R_GlobalEnv" 

$env_other
env_a 
"R_GlobalEnv" 

$env_fun
NULL

env_a occurs in two places; maybe env_other is not necessary...?

Merge branch 'master' into 71-with-nix # Conflicts: # NAMESPACE

…` in .libPaths()

Philipp Baumann added 7 commits September 11, 2023 14:36

start templating

03fc257

factor out polling; boilerplating

f83b3f2

Merge branch 'master' into 71-with-nix

74d849c

factor out polling

8213d1e

update {fusen}

eb7d118

fix poll cmds

4097856

use calls and function in expr

028c105

philipp-baumann changed the title ~~Draft in progress: 71 with nix~~ Draft in progress: address #71 with_nix() Sep 12, 2023

Philipp Baumann added 15 commits September 17, 2023 13:45

more templating

02d2785

Merge branch 'master' into 71-with-nix

0c4186a

factor out polling funs

5ad3e95

omit nix build artefacts in local checks

cc6a370

wip; start matching fun args

f7ac503

merge master

71462b9

Merge branch 'master' into 71-with-nix # Conflicts: # .Rbuildignore

Merge branch 'master' into 71-with-nix

065b16a

step 1: serialize formals in host R session

bbf20c9

come up with serialization/deserialization plan for evaluation betwee…

7391c84

…n host R and nix-R

align order in Map f args with corresponding body

a69357a

update comments with plan

e1ee1aa

lay out commented version of 2) deserialization

9477beb

inflate

1ac1ff9

better comment plan

2eec542

fix commented example fun in expr arg

fe4a5b3

b-rodrigues reviewed Oct 8, 2023

View reviewed changes

Philipp Baumann added 5 commits October 12, 2023 20:51

use --vanilla

da5367a

add deparser for deserializing formals/arguments in 2) (nix-R)

31f3f5d

do proper quoting; use sprintf()

c7cc24d

implement creation and evaluation of calls to read function args from…

b13ef8e

… disk

put strings deparsed into custom R script in tmp folder

43200ea

Philipp Baumann and others added 2 commits November 1, 2023 16:41

add WIP: recursive search for globals

bdf1b5e

avoid stats import

647f312

philipp-baumann and others added 2 commits November 1, 2023 22:24

implement proper categorization of globals at one level of globals set.

97ef135

simplify and make nix-R steps string-injection safe by using language…

ff9d67d

… objects

philipp-baumann added 3 commits December 29, 2023 19:11

factor out recurse_find_check_globals()

f2935bc

extract necessary exports correctly

49c6746

remove helper; not working/needed

9ca48a7

philipp-baumann added 3 commits January 5, 2024 21:03

update ignores

2e5a520

keep up with current master

817ac4a

Merge branch 'master' into 71-with-nix # Conflicts: # NAMESPACE

serialize and deserialize globals found by code analysis

d8241c5

philipp-baumann self-assigned this Jan 6, 2024

philipp-baumann added 11 commits January 6, 2024 15:02

first working version of with_nix() 🚀

a403477

first working version of with_nix() 🚀

0ad51f7

improve docs

6a11fb8

correct description of with_nix()

69a492f

macOS: manually remove in case there is `like "/Users/<your-user>/...…

a47d35d

…` in .libPaths()

tweak doc

2e6a46d

also remove user libs for linux

dfacb0f

fix typo

2f63062

fix glibc error on linux by pure nix-shell

963fd41

fix detection of nix-shell and stop with what to do

dd2c0b6

better formatting

57cac6b

philipp-baumann merged commit 43153c1 into master Jan 7, 2024
9 checks passed

philipp-baumann deleted the 71-with-nix branch January 7, 2024 15:48

philipp-baumann mentioned this pull request Jan 7, 2024

with_nix() or nix_run_r() and nix_run_shell() #71

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft in progress: address #71 `with_nix()` #76

Draft in progress: address #71 `with_nix()` #76

philipp-baumann commented Sep 12, 2023

b-rodrigues Oct 8, 2023

philipp-baumann commented Oct 15, 2023

philipp-baumann commented Oct 22, 2023 •

edited

Loading

philipp-baumann commented Nov 1, 2023 •

edited

Loading

philipp-baumann commented Nov 10, 2023 •

edited

Loading

philipp-baumann commented Nov 15, 2023 •

edited

Loading

philipp-baumann commented Dec 29, 2023 •

edited

Loading

Draft in progress: address #71 with_nix() #76

Draft in progress: address #71 with_nix() #76

Conversation

philipp-baumann commented Sep 12, 2023

b-rodrigues Oct 8, 2023

Choose a reason for hiding this comment

philipp-baumann commented Oct 15, 2023

philipp-baumann commented Oct 22, 2023 • edited Loading

philipp-baumann commented Nov 1, 2023 • edited Loading

philipp-baumann commented Nov 10, 2023 • edited Loading

philipp-baumann commented Nov 15, 2023 • edited Loading

philipp-baumann commented Dec 29, 2023 • edited Loading

Draft in progress: address #71 `with_nix()` #76

Draft in progress: address #71 `with_nix()` #76

philipp-baumann commented Oct 22, 2023 •

edited

Loading

philipp-baumann commented Nov 1, 2023 •

edited

Loading

philipp-baumann commented Nov 10, 2023 •

edited

Loading

philipp-baumann commented Nov 15, 2023 •

edited

Loading

philipp-baumann commented Dec 29, 2023 •

edited

Loading