## [1] 3.092## [1] 3.1We now parallelize piApprox() with batchtools: We create 10 jobs, each doing a MC simulation with \(10^5\) jobs. We use batchMap() to define the jobs (note that this does not yet start the calculation):
batchMap(fun = piApprox, n = rep(1e5, 10))## Adding 10 jobs ...If you are absolutely sure that your function works, you can take a shortcut and use batchtools in an lapply fashion using btlapply(). This function creates a temporary registry (but you may also pass one yourself), calls batchMap(), wait for the jobs to terminate with waitForJobs() and then uses reduceResultsList() to return the results.
res = btlapply(rep(1e5, 10), piApprox)
mean(unlist(res))## [1] 3.143692## [1] 3.14168If we want to peek into the R log file of a job to see more context for the error we can use showLog() which opens a pager or use getLog() to get the log as character vector:
tail(getLog(id = 9))## [1] "### [bt 2017-08-07 19:31:56]: Memory measurement disabled"
-## [2] "### [bt 2017-08-07 19:31:56]: Starting job [batchtools job.id=9]"
+## [1] "### [bt 2017-08-07 19:53:34]: Memory measurement disabled"
+## [2] "### [bt 2017-08-07 19:53:34]: Starting job [batchtools job.id=9]"
## [3] "Error in (function (value) : Ooops."
## [4] ""
-## [5] "### [bt 2017-08-07 19:31:56]: Job terminated with an exception [batchtools job.id=9]"
-## [6] "### [bt 2017-08-07 19:31:56]: Calculation finished!"
+## [5] "### [bt 2017-08-07 19:53:34]: Job terminated with an exception [batchtools job.id=9]"
+## [6] "### [bt 2017-08-07 19:53:34]: Calculation finished!"You can also grep for messages (output suppressed in this vignette for technical reasons):
grepLogs(pattern = "simple", ignore.case = TRUE)addExperiments() now warns if a design is passed as data.frame with factor columns and stringsAsFactors is TRUE.setJobNames() and getJobNames() to control the name of jobs on batch systems. Templates should be adapted to use job.name instead of job.hash for naming.flatten of getJobResources(), getJobPars and getJobTable is deprecated and will be removed. Future versions of the functions will behave like flatten is set to FALSE explicitly. Single resources/parameters must be extracted manually (or using unnest in package tidyr).flatten of getJobResources(), getJobPars() and getJobTable() is deprecated and will be removed. Future versions of the functions will behave like flatten is set to FALSE explicitly. Single resources/parameters must be extracted manually (or with tidyr::unnest()).findStarted(), findNotStarted and getStatus().findStarted(), findNotStarted() and getStatus().findExperiments() now performs an exact string match (instead of matching substrings) for patterns specified via prob.name and algo.name. For substring matching, use prob.pattern or algo.pattern, respectively.reduceResultsDataTable()
@@ -146,7 +146,7 @@ getJobTable() returned difftimes with the wrong unit (e.g., in minutes instead of seconds).clusterFunctionsDocker().ClusterFunctionsDocker.#>#>#>+#>#>#># resources are usually set in submitJobs() jc = makeJobCollection(1:3, resources = list(foo = "bar"), reg = tmp) ls(jc)#> [1] "array.jobs" "array.var" "file.dir" "job.hash" "job.name" diff --git a/docs/reference/JobExperiment.html b/docs/reference/JobExperiment.html index 372027aa..8ffab71d 100644 --- a/docs/reference/JobExperiment.html +++ b/docs/reference/JobExperiment.html @@ -148,12 +148,12 @@Value
Examples
-#>#>#>#>job = makeJob(1, reg = tmp) +#>#>#>#>job = makeJob(1, reg = tmp) print(job)#> <Job> #> Inherits from: <BaseJob> #> Public: #> external.dir: active binding -#> file.dir: /tmp/RtmpZmp9mQ/registry3616660968e4 +#> file.dir: /tmp/RtmpFgx6O2/registry4fc1318ece8 #> fun: active binding #> id: 1 #> initialize: function (file.dir, reader, id, pars, seed, resources) diff --git a/docs/reference/JobNames.html b/docs/reference/JobNames.html index 39f82d2e..89add4db 100644 --- a/docs/reference/JobNames.html +++ b/docs/reference/JobNames.html @@ -131,7 +131,7 @@Value
Examples
-#>#>#>setJobNames(ids, letters[1:nrow(ids)], reg = tmp) +#>#>#>setJobNames(ids, letters[1:nrow(ids)], reg = tmp) getJobNames(reg = tmp)#> job.id job.name #> <int> <char> #> 1: 1 a diff --git a/docs/reference/JoinTables.html b/docs/reference/JoinTables.html index cb84c888..b6e65c8a 100644 --- a/docs/reference/JoinTables.html +++ b/docs/reference/JoinTables.html @@ -149,7 +149,7 @@Value
Examples
#>#>#>#>#>#>x = getJobPars(reg = tmp) y = findJobs(x >= 2 & x <= 5, reg = tmp) y$extra.col = head(letters, nrow(y)) diff --git a/docs/reference/Tags.html b/docs/reference/Tags.html index 877f6336..bc9efc13 100644 --- a/docs/reference/Tags.html +++ b/docs/reference/Tags.html @@ -129,7 +129,7 @@Value
Examples
-#>#>#>+#>#>#>#> job.id tags diff --git a/docs/reference/addAlgorithm.html b/docs/reference/addAlgorithm.html index de71dfc4..e4e727f9 100644 --- a/docs/reference/addAlgorithm.html +++ b/docs/reference/addAlgorithm.html @@ -133,7 +133,7 @@Value
Examples
-#>#>#>#>#> [1] "p1" "p2"+diff --git a/docs/reference/addExperiments.html b/docs/reference/addExperiments.html index 20ccd785..0351572d 100644 --- a/docs/reference/addExperiments.html +++ b/docs/reference/addExperiments.html @@ -156,7 +156,7 @@#>#>#>#>#> [1] "p1" "p2"addAlgorithm("a1", fun = function(job, data, instance) instance, reg = tmp)#>getAlgorithmIds(reg = tmp)#> [1] "a1"removeAlgorithms("a1", reg = tmp)#>#>#>#>#>#>#>#>getAlgorithmIds(reg = tmp)#> character(0)See a
Examples
-#>#>+#>#># add first problem fun = function(job, data, n, mean, sd, ...) rnorm(n, mean = mean, sd = sd) addProblem("rnorm", fun = fun, reg = tmp)#>diff --git a/docs/reference/addProblem.html b/docs/reference/addProblem.html index fb17c2d8..6cb67f58 100644 --- a/docs/reference/addProblem.html +++ b/docs/reference/addProblem.html @@ -158,7 +158,7 @@Value
Examples
-#>#>addProblem("p1", fun = function(job, data) data, reg = tmp)#>addProblem("p2", fun = function(job, data) job, reg = tmp)#>getProblemIds(reg = tmp)#> [1] "p1" "p2"+diff --git a/docs/reference/batchExport.html b/docs/reference/batchExport.html index 1577f534..140f3657 100644 --- a/docs/reference/batchExport.html +++ b/docs/reference/batchExport.html @@ -125,21 +125,21 @@#>#>addProblem("p1", fun = function(job, data) data, reg = tmp)#>addProblem("p2", fun = function(job, data) job, reg = tmp)#>getProblemIds(reg = tmp)#> [1] "p1" "p2"#>#> [1] "a1"#>#>#>#>#>#>#>#>#> character(0)Value
Examples
-#>#>++#> 1: y /tmp/RtmpFgx6O2/registry4fc16c961092/exports/PE.rds#>#># list exports exports = batchExport(reg = tmp) print(exports)#> Empty data.table (0 rows) of 2 cols: name,uri#> NULL#>exports = batchExport(export = list(y = 99, z = 1), reg = tmp)#>print(exports)#> name uri #> <char> <char> -#> 1: y /tmp/RtmpZmp9mQ/registry3616555ae6ca/exports/PE.rds -#> 2: z /tmp/RtmpZmp9mQ/registry3616555ae6ca/exports/PI.rds+#> 1: y /tmp/RtmpFgx6O2/registry4fc16c961092/exports/PE.rds +#> 2: z /tmp/RtmpFgx6O2/registry4fc16c961092/exports/PI.rds#>#>#> [1] TRUEstopifnot(loadResult(1, reg = tmp) == 101) # Un-export z exports = batchExport(unexport = "z", reg = tmp)#>print(exports)#> name uri #> <char> <char> -#> 1: y /tmp/RtmpZmp9mQ/registry3616555ae6ca/exports/PE.rds