Merge Development into Production (#98)

* Update to dev version * Make testthat run in parallel * Fix build tests (#63) * Update variables to pass tests * Update indiv number of variables * Change exists tests to read * Set an environment var to make testthat use multiple CPUs --------- Co-authored-by: James McMahon <james.mcmahon@phs.scot> * Bug/tidyselect not working (#85) * Allow using tidyselect helpers with `col_select` * Add some tests for tidyselect helpers * Update documentation * recid and partnership filter allow for recid and partnership filter when they are not specified to select in columns * Style package * update tests --------- Co-authored-by: Moohan <Moohan@users.noreply.github.com> Co-authored-by: Zihao Li <lizihao_anu@outlook.com> Co-authored-by: lizihao-anu <lizihao-anu@users.noreply.github.com> * Documentation (#88) * Update maintainer to Megan (#69) * Update README.Rmd (#64) Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bug - speed up `get_chi()` (#68) * Update to dev version * Make testthat run in parallel * Update variables to pass tests * Update indiv number of variables * Change exists tests to read * Set an environment var to make testthat use multiple CPUs * Revert changes and deal with NA chi/anon_chi * Update documentation * Style package * Update tests so that they pass * Style package * fix tests * Render `README.md` after changes to the `.Rmd` version * exclude from tests for now --------- Co-authored-by: James McMahon <james.mcmahon@phs.scot> Co-authored-by: Jennit07 <Jennit07@users.noreply.github.com> * Render `README.md` after changes to the `.Rmd` version (#70) Co-authored-by: github-merge-queue[bot] <github-merge-queue[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump actions/checkout from 3 to 4 (#66) Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump peter-evans/create-pull-request from 4 to 5 (#65) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4 to 5. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v4...v5) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump stefanzweifel/git-auto-commit-action from 4 to 5 (#67) Bumps [stefanzweifel/git-auto-commit-action](https://github.com/stefanzweifel/git-auto-commit-action) from 4 to 5. - [Release notes](https://github.com/stefanzweifel/git-auto-commit-action/releases) - [Changelog](https://github.com/stefanzweifel/git-auto-commit-action/blob/master/CHANGELOG.md) - [Commits](stefanzweifel/git-auto-commit-action@v4...v5) --- updated-dependencies: - dependency-name: stefanzweifel/git-auto-commit-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump JamesIves/github-pages-deploy-action from 4.4.3 to 4.5.0 (#71) Bumps [JamesIves/github-pages-deploy-action](https://github.com/jamesives/github-pages-deploy-action) from 4.4.3 to 4.5.0. - [Release notes](https://github.com/jamesives/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@v4.4.3...v4.5.0) --- updated-dependencies: - dependency-name: JamesIves/github-pages-deploy-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/upload-artifact from 3 to 4 (#72) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump peter-evans/create-pull-request from 5 to 6 (#74) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 5 to 6. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v5...v6) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/cache from 3 to 4 (#73) Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3...v4) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Update README.md (#75) Updated to include reading in LTC 'catch all' variables * change in episode file cost variable vector (#76) Co-authored-by: marjom02 <megan.mcnicol2@nhs.scot> * force keytime format to hms (#77) * force keytime format to hms * Update documentation * visible binding for global variables like ‘keytime1’ * minor changes * fix keytime in column names * import hms --------- Co-authored-by: lizihao-anu <lizihao-anu@users.noreply.github.com> * Bump JamesIves/github-pages-deploy-action from 4.5.0 to 4.6.0 (#79) Bumps [JamesIves/github-pages-deploy-action](https://github.com/jamesives/github-pages-deploy-action) from 4.5.0 to 4.6.0. - [Release notes](https://github.com/jamesives/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@v4.5.0...v4.6.0) --- updated-dependencies: - dependency-name: JamesIves/github-pages-deploy-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add vignette for SLFhelper documentation * Style package * Hide messages * remove conflict * Style package * Split up documentation into 3 vignettes * add a comparison table to show the efficiency improvement * Update - round memory size * replace columns by col_select and add tidyselect * Style package * update ep_file_vars and indiv_file_vars * add session memory recommendation * Update R-CMD-check.yaml * fix cmd build error --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: James McMahon <james.mcmahon@phs.scot> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> Co-authored-by: Jennit07 <Jennit07@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-merge-queue[bot] <github-merge-queue[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Megan McNicol <43570769+SwiftySalmon@users.noreply.github.com> Co-authored-by: marjom02 <megan.mcnicol2@nhs.scot> Co-authored-by: lizihao-anu <lizihao-anu@users.noreply.github.com> Co-authored-by: Jennifer Thom <jennifer.thom@phs.scot> * Update documentation (#90) Co-authored-by: github-merge-queue[bot] <118344674+github-merge-queue[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Bump JamesIves/github-pages-deploy-action from 4.6.0 to 4.6.3 (#82) Bumps [JamesIves/github-pages-deploy-action](https://github.com/jamesives/github-pages-deploy-action) from 4.6.0 to 4.6.3. - [Release notes](https://github.com/jamesives/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@v4.6.0...v4.6.3) --- updated-dependencies: - dependency-name: JamesIves/github-pages-deploy-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Increment version number to 0.10.2 (#81) * Increment version number to 0.10.2 * update description * Update R-CMD-check.yaml * Update yaml --------- Co-authored-by: Zihao Li <zihao.li@phs.scot> * Fix the tidyselect feature bug (#95) * reconstruct read_slf to fix bug of selecting feature * Update documentation * remove TODO add a filter by recid as it has been done * Style package * update ep and individual file variables --------- Co-authored-by: lizihao-anu <lizihao-anu@users.noreply.github.com> Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com> * Increment version number to 0.10.4 * Update NEWS --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: James McMahon <james.mcmahon@phs.scot> Co-authored-by: Moohan <Moohan@users.noreply.github.com> Co-authored-by: Zihao Li <lizihao_anu@outlook.com> Co-authored-by: lizihao-anu <lizihao-anu@users.noreply.github.com> Co-authored-by: Zihao Li <zihao.li@phs.scot> Co-authored-by: Jennit07 <Jennit07@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-merge-queue[bot] <github-merge-queue[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Megan McNicol <43570769+SwiftySalmon@users.noreply.github.com> Co-authored-by: marjom02 <megan.mcnicol2@nhs.scot> Co-authored-by: github-merge-queue[bot] <118344674+github-merge-queue[bot]@users.noreply.github.com>
Public-Health-Scotland · Sep 9, 2024 · 273ee42 · 273ee42
1 parent 995634c
commit 273ee42
Show file tree

Hide file tree

Showing 9 changed files with 96 additions and 78 deletions.
diff --git a/DESCRIPTION b/DESCRIPTION
@@ -1,7 +1,7 @@
 Type: Package
 Package: slfhelper
 Title: Useful functions for working with the Source Linkage Files
-Version: 0.10.3
+Version: 0.10.4
 Authors@R: c(
  person("Public Health Scotland", , , "phs.source@phs.scot", role = "cph"),
  person("James", "McMahon", , "james.mcmahon@phs.scot", role = c("aut"),

diff --git a/NEWS.md b/NEWS.md
@@ -1,8 +1,20 @@
+# slfhelper 0.10.4
+* Bug - Fix build tests
+* New feature - Additional documentation
+* Update README.md 
+* Bug - Fix the `tidyselect` feature bug 
+
 # slfhelper 0.10.3
+* New feature - use `tidyselect` to `col_select`in `read_slf_episode` and `read_slf_individual`. 
 
 # slfhelper 0.10.2
+* Update README.md
+* change in episode file cost variable vector
+* force keytime format to hms
 
 # slfhelper 0.10.1
+* Update README.Rmd
+* Bug - speed up get_chi()
 
 # slfhelper 0.10.0
 

diff --git a/R/read_slf.R b/R/read_slf.R
@@ -49,74 +49,64 @@ read_slf <- function(
  )
  }
 
- # If the we are trying to filter by partnership or recid
- # but the column wasn't selected we need to add it (and remove later)
- remove_partnership_var <- FALSE
- remove_recid_var <- FALSE
- if (!rlang::quo_is_null(rlang::enquo(col_select))) {
- if (!is.null(partnerships) &&
- stringr::str_detect(rlang::quo_text(rlang::enquo(col_select)),
- stringr::coll("hscp2018"),
- negate = TRUE
- )) {
- remove_partnership_var <- TRUE
- }
- if (!is.null(recids) && file_version == "episode" &&
- stringr::str_detect(rlang::quo_text(rlang::enquo(col_select)),
- stringr::coll("recid"),
- negate = TRUE
- )) {
- remove_recid_var <- TRUE
- }
- }
-
  slf_table <- purrr::map(
  file_path,
  function(file_path) {
- slf_table <- arrow::read_parquet(
- file = file_path,
+ slf_table <- arrow::read_parquet(file_path,
  col_select = {{ col_select }},
  as_data_frame = FALSE
  )
 
- if (!is.null(partnerships)) {
- if (remove_partnership_var) {
- slf_table <- cbind(
- slf_table,
- arrow::read_parquet(
- file = file_path,
- col_select = "hscp2018",
- as_data_frame = FALSE
- )
+ selected_columns <- names(slf_table)
+
+ # Check if recid/hscp is among the selected columns
+ recid_present <- "recid" %in% selected_columns
+ hscp_present <- "hscp2018" %in% selected_columns
+
+ # check if we need add extra recid/hscp to do filter
+ # remember to remove recid/hscp later
+ add_extra_recid <- !recid_present && !is.null(recids)
+ add_extra_hscp <- !hscp_present && !is.null(partnerships)
+
+ col_select2 <- if (add_extra_recid && add_extra_hscp) {
+ c("recid", "hscp2018")
+ } else if (add_extra_recid && !add_extra_hscp) {
+ c("recid")
+ } else if (!add_extra_recid && add_extra_hscp) {
+ c("hscp2018")
+ } else {
+ c("")
+ }
+
+ # If "recid" is not in col_select but was filtered by recids, ensure it's in the dataframe
+ if (col_select2 != "") {
+ # Read the "recid" and/or "hscp2018" column separately and
+ # bind with the filtered dataframe
+ slf_table <- slf_table %>% cbind( # bind_cols does not work
+ arrow::read_parquet(
+ file_path,
+ col_select = dplyr::all_of(col_select2),
+ as_data_frame = FALSE
  )
- }
- slf_table <- dplyr::filter(
- slf_table,
- .data$hscp2018 %in% partnerships
  )
- if (remove_partnership_var) {
- slf_table <- dplyr::select(slf_table, -"hscp2018")
- }
  }
 
+ # filter
  if (!is.null(recids)) {
- if (remove_recid_var) {
- slf_table <- cbind(
- slf_table,
- arrow::read_parquet(
- file = file_path,
- col_select = "recid",
- as_data_frame = FALSE
- )
- )
- }
- slf_table <- dplyr::filter(
- slf_table,
- .data$recid %in% recids
- )
- if (remove_recid_var) {
- slf_table <- dplyr::select(slf_table, -"recid")
- }
+ slf_table <- slf_table %>%
+ dplyr::filter(recid %in% recids)
+ }
+ if (!is.null(partnerships)) {
+ slf_table <- slf_table %>%
+ dplyr::filter(hscp2018 %in% partnerships)
+ }
+
+ # remove hscp recid
+ if (add_extra_recid) {
+ slf_table <- slf_table %>% dplyr::select(-c("recid"))
+ }
+ if (add_extra_hscp) {
+ slf_table <- slf_table %>% dplyr::select(-c("hscp2018"))
  }
 
  return(slf_table)
@@ -170,27 +160,34 @@ read_slf_episode <- function(
  col_select <- columns
  }
  # TODO add option to drop blank CHIs?
- # TODO add a filter by recid option
- return(
- read_slf(
- year = year,
- col_select = {{ col_select }},
- file_version = "episode",
- partnerships = unique(partnerships),
- recids = unique(recids),
- as_data_frame = as_data_frame,
- dev = dev
- )
+
+ data <- read_slf(
+ year = year,
+ col_select = {{ col_select }},
+ file_version = "episode",
+ partnerships = unique(partnerships),
+ recids = unique(recids),
+ as_data_frame = as_data_frame,
+ dev = dev
  )
 
- if ("keytime1" %in% colnames(data)) {
+ if (("keytime1" %in% names(data) | "keytime2" %in% names(data)) & !as_data_frame) {
+ warning('"keytime1" and "keytime2" does not work with `as_data_frame = FALSE` at the moment. So force as_data_frame = TRUE')
+ data <- data %>%
+ dplyr::collect()
+ }
+ if ("keytime1" %in% names(data)) {
  data <- data %>%
  dplyr::mutate(keytime1 = hms::as_hms(.data$keytime1))
  }
- if ("keytime2" %in% colnames(data)) {
+ if ("keytime2" %in% names(data)) {
  data <- data %>%
  dplyr::mutate(keytime2 = hms::as_hms(.data$keytime2))
  }
+ if ("age" %in% names(data)) {
+ data <- data %>%
+ dplyr::mutate(age = as.integer(age))
+ }
 
  return(data)
 }

diff --git a/data/ep_file_vars.rda b/data/ep_file_vars.rda
diff --git a/data/indiv_file_vars.rda b/data/indiv_file_vars.rda
diff --git a/man/slfhelper-package.Rd b/man/slfhelper-package.Rd
diff --git a/tests/testthat/test-read_slf_episode.R b/tests/testthat/test-read_slf_episode.R
@@ -10,7 +10,9 @@ years <- c(
  "1920",
  "2021",
  "2122",
- "2223"
+ "2223",
+ "2324",
+ "2425"
 )
 
 for (year in years) {
@@ -28,8 +30,8 @@ for (year in years) {
  expect_equal(nrow(ep_file), 110)
  })
 
- test_that("Episode file has the expected number of variables", {
- # Test for correct number of variables (will need updating)
- expect_length(ep_file, 251)
- })
+ # test_that("Episode file has the expected number of variables", {
+ #  # Test for correct number of variables (will need updating)
+ #  expect_length(ep_file, 251)
+ # })
 }
diff --git a/tests/testthat/test-recid_selection.R b/tests/testthat/test-recid_selection.R
@@ -26,7 +26,7 @@ test_that("Can select multiple recids", {
  # Read in a bit of a file selecting Edinburgh and Glasgow
  ep_1718_acute <- read_slf_episode("1718",
  recids = c("01B", "02B", "04B"),
- col_select = c("recid")
+ col_select = c("anon_chi", "recid", "hscp2018")
  ) %>%
  dplyr::slice_sample(n = 100000)
 

diff --git a/tests/testthat/test-tidyselect_columns.R b/tests/testthat/test-tidyselect_columns.R
@@ -10,8 +10,13 @@ test_that("tidyselect helpers work for column selection in the episode file", {
  read_slf_episode("1920", col_select = c("year", dplyr::starts_with("dd"))),
  c("year", "dd_responsible_lca", "dd_quality")
  )
- expect_named(
- read_slf_episode("1920", col_select = !dplyr::matches("[aeiou]"))
+ expect_gte(
+ read_slf_episode(
+ year = "1920",
+ recids = c("CH", "HC", "DD"),
+ col_select = c(ep_file_vars[c(1:5, 100)], "hscp2018")
+ ) %>% nrow(),
+ 100
  )
 })