Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opencpu client dev #271

Closed
wants to merge 129 commits into from
Closed
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
f6c223c
translate list_animal_ids to use API
PietrH Dec 9, 2022
bba0b18
Add helper function to get ETN database credentials from Sys.env
PietrH Dec 12, 2022
3cbd9bf
add httr dependency for API calls
PietrH Dec 12, 2022
03b1348
comment out unused code, warn for status, add roxygen skeleton
PietrH Dec 12, 2022
7f04862
add helper for fetching temp key from api response
PietrH Dec 12, 2022
6412f91
add list_animal_ids_api to namespace
PietrH Dec 12, 2022
8071dfb
generate help files
PietrH Dec 12, 2022
8dd8d56
move helper to utils.R, generate documentation
PietrH Dec 12, 2022
6c93b24
cleanup unused code
PietrH Dec 12, 2022
294788d
stylr, lintr
PietrH Dec 12, 2022
548af2e
Delete get_val.Rd
PietrH Dec 12, 2022
a61a4bb
stop inheriting documentation
PietrH Dec 12, 2022
a53ca14
add test
Dec 12, 2022
0f09947
Merge branch 'opencpu-client' of https://github.com/inbo/etn into ope…
Dec 12, 2022
970ed50
add test for list_animals_ids_api()
PietrH Dec 12, 2022
e24e20d
Merge branch 'opencpu-client' of https://github.com/inbo/etn into ope…
PietrH Dec 12, 2022
b6cc350
remove empty lines between @params
PietrH Jan 4, 2023
e402c03
switch stringr to glue
PietrH Jan 4, 2023
f37593f
switch glue for paste
PietrH Jan 4, 2023
c6a946c
fix typo
PietrH Jan 4, 2023
420f9d5
don't use two statics, but only one
PietrH Jan 4, 2023
6c2d358
check response header for content-type
PietrH Jan 4, 2023
71855e3
stylistic fix: function definition should start on same line as it's …
PietrH Jan 4, 2023
d7d7fca
create helper to check for response content type
PietrH Jan 4, 2023
cb8592e
use glue instead of stringr::str_glue
PietrH Jan 4, 2023
35543b7
clarify documentation
PietrH Jan 4, 2023
05d090c
use glue instead of stringr::str_glue
PietrH Jan 4, 2023
2013180
update to new endpoint, rename authentification argument etnservice#9
PietrH Feb 10, 2023
bab4e23
get started on a general function port pattern
PietrH Feb 10, 2023
2ec1692
pass credentials as object, not as string
PietrH Feb 10, 2023
cc7a49d
forward R errors
PietrH Feb 10, 2023
084db7c
helper to transform R objects into JSON primitives
PietrH Feb 10, 2023
ff805be
helper to fetch parent function arguments
PietrH Feb 10, 2023
34f7321
add lifecycle to imports
PietrH Jun 7, 2023
ceacf65
demote get_acoustic_detections_api() to helper: see #280
PietrH Jun 7, 2023
ad23e46
use two step api object fetching
PietrH Jun 7, 2023
d5839e4
add roxygen skeleton for api helper
PietrH Jun 7, 2023
d13017f
don't pass api or connection arguments to helpers
PietrH Jun 7, 2023
353fa82
pass function itself, not an empty execution
PietrH Jun 7, 2023
e369a55
don't overwrite function arguments
PietrH Jun 7, 2023
236f107
two step endpoint needs to end on slash
PietrH Jun 7, 2023
e69c253
use sys.calls() instead of match.call() because of higher nesting
PietrH Jun 7, 2023
24cd2b8
use more intelligent parent function finding
PietrH Jun 7, 2023
39be8f8
increase resiliance: support multiline calls, support out of order ar…
PietrH Jun 7, 2023
d7b6e9a
add purrr import
PietrH Jun 7, 2023
1c24bfc
add retry on first request, add http failure states
PietrH Jun 7, 2023
be3fac2
convert all tests to use API, add limit, add test for passing errors …
PietrH Jun 7, 2023
d1a2b06
prompt user for missing credentials
PietrH Jun 8, 2023
908e475
add dev dep: withr + up testthat to 3e
PietrH Jun 8, 2023
083dd21
up testthat to 3e
PietrH Jun 8, 2023
785eda7
test if get_credentials() can fetch values from sys env
PietrH Jun 8, 2023
4dede99
Merge branch 'main' into opencpu-client-dev
PietrH Jun 8, 2023
b959fa1
remove default NULL connection (default should me missing) + remove c…
PietrH Jun 8, 2023
976d4c1
remove unused helpers
PietrH Jun 8, 2023
86056c0
Clean up comments
PietrH Jun 8, 2023
3efcdda
add rlang dependency
PietrH Jun 8, 2023
32c66c6
add helpers for checking OpenCPU response objects and deprecation of …
PietrH Jun 8, 2023
a06a0e9
rewrite list_animal_ids() to use api
PietrH Jun 8, 2023
d95daed
remove old list_animal_ids_api() function, now helper in list_animal_…
PietrH Jun 8, 2023
268e229
add tests for list_animal_ids()
PietrH Jun 8, 2023
8569e9e
add comments, use deprecation helper
PietrH Jun 8, 2023
380e1c7
remove `list_animal_ids_api()`
PietrH Jun 8, 2023
68b9a25
helper to get the name of the parent function
PietrH Jun 8, 2023
f64c685
devtools::document()
PietrH Jun 8, 2023
78befb3
use helper to get function identity
PietrH Jun 8, 2023
92c913a
general helper to forward function arguments to api, including functi…
PietrH Jun 8, 2023
8d9a001
use generic helper to handle api call, remove specific helpers
PietrH Jun 8, 2023
4227e96
add tests for sql query
PietrH Jun 8, 2023
5f21753
expect_is should be expect_type in testthat 3e
PietrH Jun 8, 2023
6dcdb49
rewrite list_acoustic_project_codes() to use api
PietrH Jun 8, 2023
263a8bc
rewrite list_acoustic_project_codes() to use api
PietrH Jun 8, 2023
4515193
resolve merge conflict
PietrH Jun 8, 2023
bf8c53a
rewrite `list_acoustic_tag_ids()` to use api
PietrH Jun 8, 2023
c83fe53
assert that api argument is a flag
PietrH Jun 8, 2023
0ef96bd
rewrite list_animal_project_codes() and list_cereiver_ids() to use api
PietrH Jun 8, 2023
d571322
adapt list_scientific_names() and list_station_names() to use api
PietrH Jun 8, 2023
1bbdf1a
create helper to switch between using api or not, adapt helpers to re…
PietrH Jun 13, 2023
2876812
adapt api functions to use conductor helper
PietrH Jun 13, 2023
ca4b2ad
devtools::document()
PietrH Jun 13, 2023
98c3fe8
allow deprecate_warn_connection() to fetch function_identity itself
PietrH Jun 13, 2023
575e807
remove unused argument from deprecate_warn_connection()
PietrH Jun 13, 2023
ebf37e9
remove reference to deprecated connection argument
PietrH Jun 13, 2023
8dd182e
add tests for API helpers
PietrH Jun 15, 2023
5ec65d3
set get_val() to retry, style fixes, add domain options to helpers fo…
PietrH Jun 15, 2023
feab756
rewrite get_acoustic_deployments() to use api
PietrH Jun 15, 2023
2aec8f0
update test to use api over sql
PietrH Jun 15, 2023
7c43da5
set sql helper to not use api
PietrH Jun 15, 2023
7e7f461
add test that uses sql rather than api
PietrH Jun 15, 2023
53c9a4b
devtools::document()
PietrH Jun 15, 2023
bfc8e50
adapt list_cpod_project_codes() to use api
PietrH Jun 15, 2023
b913ae2
adapt get_acoustic_projects() to use api
PietrH Jun 15, 2023
f1877cb
adapt get_acoustic_receivers() to use api
PietrH Jun 15, 2023
41740ff
adapt get_animal_projects() to use api
PietrH Jun 15, 2023
dfa8bdb
adapt list_tag_serial_numbers() to use api
PietrH Jun 15, 2023
ccafee2
adapt get_animals() to use the api
PietrH Jun 15, 2023
67e8517
adapt list_deployment_ids() to use api
PietrH Jun 15, 2023
c88a557
adapt get_cpod_projects() to use api
PietrH Jun 15, 2023
aec4376
hard deprecate `connect_to_etn()`, use lifecycle for badges. Add to i…
PietrH Jun 15, 2023
fc73cd7
adapt get_tags() to use the api
PietrH Jun 15, 2023
3e9ab7f
fix old calls to refer to api adapted arguments
PietrH Jun 16, 2023
0e1f46b
fix old calls to refer to api adapted arguments
PietrH Jun 16, 2023
fb2ecf3
adapt download_acoustic_dataset() to use api
PietrH Jun 16, 2023
f05ba25
rewrite test for download_acoustic_dataset(), use snapshots, split of…
PietrH Jun 16, 2023
69eaadd
remove now unused snapshot file
PietrH Jun 16, 2023
2beb284
add snapshots for download_acoustic_dataset()
PietrH Jun 16, 2023
548de57
devtools::document()
PietrH Jun 16, 2023
72e1010
set helper arguments to use api
PietrH Jun 16, 2023
a6fcdab
adapt snapshots with new test titles
PietrH Jun 16, 2023
04b9f4d
update helper arguments to api rewrite
PietrH Jun 16, 2023
f9cbaed
reduce test strictness, cope with running interactivly
PietrH Jun 16, 2023
2333b61
improve test coverage by writing tests for sql helper
PietrH Jun 16, 2023
1225180
add missing connection creation to sql helper
PietrH Jun 16, 2023
aab9a83
release connection when done
PietrH Jun 16, 2023
3ec17bb
close connections after using them
PietrH Jun 16, 2023
a7c351c
plug connection leak
PietrH Jun 19, 2023
6be039c
escape capture group in regex to avoid test failure
PietrH Jun 19, 2023
25af9cf
increase number of times to retry api calls
PietrH Jun 19, 2023
0b714cf
update snapshot to full error message
PietrH Jun 19, 2023
47963f5
bump version to v3 dev
PietrH Jun 19, 2023
9b08304
Remove deprecated functions from v2.1.0 from ETN
PietrH Jun 19, 2023
3d6975f
add NEWS.md
PietrH Jun 19, 2023
dedcc02
no longer split off api and sql tests
PietrH Jun 19, 2023
bc02917
plug another connection leak
PietrH Jun 19, 2023
d161e91
adapt get_val() to close connection
PietrH Jun 19, 2023
0a93b7d
expect_is should be expect_type in testthat 3e
PietrH Jun 20, 2023
b4cc8f4
limit query in time, don't use limit, so result is identical every time
PietrH Jun 20, 2023
abf89c1
limit query in time, don't use limit, so result is identical every time
PietrH Jun 20, 2023
f035d96
Merge branch 'opencpu-client-dev' of github.com:inbo/etn into opencpu…
PietrH Jun 20, 2023
af94b67
disable test awaiting #283
PietrH Jun 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Imports:
DBI,
dplyr,
glue,
httr,
jsonlite,
lubridate,
methods,
Expand Down
1 change: 1 addition & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export(get_tags)
export(list_acoustic_project_codes)
export(list_acoustic_tag_ids)
export(list_animal_ids)
export(list_animal_ids_api)
export(list_animal_project_codes)
export(list_cpod_project_codes)
export(list_deployment_ids)
Expand Down
27 changes: 27 additions & 0 deletions R/list_animal_ids_api.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#' List all available animal ids
#'
#' @param credentials Login credentials to the ETN database, as created by
#' `get_credentials()`
#'
PietrH marked this conversation as resolved.
Show resolved Hide resolved
#' @return A vector of all unique `id_pk` present in `common.animal_release`.
#'
#' @export
list_animal_ids_api <- function(credentials = get_credentials()) {
api_domain <- "https://opencpu.lifewatch.be"
PietrH marked this conversation as resolved.
Show resolved Hide resolved
function_path <- "library/etn/R/list_animal_ids"

# POST the function request to the api_domain, request json as direct return
response <-
httr::POST(stringr::str_glue(api_domain, function_path, "json", .sep = "/"),
PietrH marked this conversation as resolved.
Show resolved Hide resolved
body = list(
con = credentials
)
)
# If request was not succesful, generate a warning
PietrH marked this conversation as resolved.
Show resolved Hide resolved
httr::warn_for_status(response, "submit request to API server")
PietrH marked this conversation as resolved.
Show resolved Hide resolved

# Parse server response JSON to a vector
response %>%
httr::content(as = "text", encoding = "UTF-8") %>%
jsonlite::fromJSON(simplifyVector = TRUE)
}
80 changes: 80 additions & 0 deletions R/utils.R
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,83 @@ check_date_time <- function(date_time, date_name = "start_date") {
)
as.character(parsed)
}

#' Get the credentials from environment variables, or set them manually
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would make more succinct and refrain from using comma's in function titles:

Get login credentials

With the different credentials floating around (R Studio server, applications, DB) we should probably figure out how these can be called consistently.

#'
#' By default, it's not necessary to set any values in this function as it's
#' used in the background by other functions. However, if you wish to provide
PietrH marked this conversation as resolved.
Show resolved Hide resolved
#' your username and password on a per function basis, this function allows you
#' to do so.
#'
#' @param username ETN Data username, by default read from the environment, but
#' you can set it manually too.
#' @param password ETN Data password, by default read from the environment, but
#' you can set it manually too.
#'
#' @return A string as it is ingested by other functions that need
#' authentication
#' @family helper functions
#' @noRd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this function is referenced in the documentation of params, I would make this a public function (even if that means users can store their credentials in a variable). I notice the same is done for e.g. show_progress(), readr_threads() in https://readr.tidyverse.org/reference/read_delim.html


get_credentials <-
function(username = Sys.getenv("userid"),
PietrH marked this conversation as resolved.
Show resolved Hide resolved
password = Sys.getenv("pwd")) {
stringr::str_glue('list(username = "{username}", password = "{password}")')
PietrH marked this conversation as resolved.
Show resolved Hide resolved
}

#' Extract the OCPU temp key from a response object
#'
#' When posting a request to the opencpu api service without the json flag, a
#' response object is returned containing all the generated objects, with a
#' unique temp key in the path. To retrieve these objects in a subsequent GET
#' request, it is convenient to retrieve this temp key from the original
#' response object
#'
#' @param response The response resulting from a POST request to a opencpu api
#' service
#'
#' @return the OCPU temp key to be used as part of a GET request to an opencpu
#' api service
#' @family helper functions
#' @noRd
extract_temp_key <- function(response) {
response %>%
httr::content(as = "text") %>%
stringr::str_extract("(?<=tmp\\/).{15}(?=\\/)")
}

#' Retrieve the result of a function called to the opencpu api
#'
#' Loading the evaluated object into the current environment, to be used
PietrH marked this conversation as resolved.
Show resolved Hide resolved
#' internally in functions calling the opencpu api service to convert a response
#' object included in the response from a post request, to the corresponding
#' objects resulting from the original call.
#'
#' @param temp_key the temp key returned from the POST request to the API
#'
#' @return the uncompressed object resulting form a GET request to the API
#' @family helper functions
#' @noRd
#' @examples
#' \dontrun{
#' etn:::extract_temp_key(response) %>% get_val()
#' }
#'
#' # using the opencpu test instance
#' api_url <- "https://cloud.opencpu.org/ocpu/library/stats/R/rnorm"
#' httr::POST(api_url, body = list(n = 10, mean = 5)) %>%
#' extract_temp_key() %>%
#' get_val(api_domain = "https://cloud.opencpu.org/ocpu")
get_val <- function(temp_key, api_domain = "https://opencpu.lifewatch.be") {
httr::GET(
stringr::str_glue(
PietrH marked this conversation as resolved.
Show resolved Hide resolved
"{api_domain}",
"tmp/{temp_key}/R/.val/rds",
.sep = "/"
)
) %>%
httr::content(as = "raw") %>%
rawConnection() %>%
gzcon() %>%
readRDS()
}
18 changes: 18 additions & 0 deletions man/list_animal_ids_api.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions tests/testthat/test-list_animal_ids_api.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
test_that("list_animal_ids_api() returns unique list of values", {
PietrH marked this conversation as resolved.
Show resolved Hide resolved
result <- list_animal_ids_api()

# tests based on test-list_animal_project_codes.R
expect_is(result, "integer")
expect_false(any(duplicated(result)))
expect_true(all(!is.na(result)))
})

test_that("list_animal_ids_api() returns at least 5 known values", {
result <- list_animal_ids_api()

# a set of 5 known id_pk present in common.animal_release
known_ids <- c("56314", "8504", "7601", "4293", "58407")

testthat::expect_true(all(known_ids %in% result))
})