index.qmd

---
title: "Supporting Companies With Validation of R Packages: A Regulatory Repository"
author:
  - name: "<span>Coline Zeballos</span> <span style='font-style: italic; margin-left: 0.5em;'><a href='http://www.roche.com'>Roche</a></span>"
  - name: "<span>Yann Féat</span> <span style='font-style: italic; margin-left: 0.5em;'><a href='http://mainanalytics.de'>mainanalytics</a></span>"
institute: "<em>useR! 2024, July 9th</em>"
reference-location: margin
---

## A Universal Conundrum {.scrollable}

_There are `n` packages for `x`, which one is the best?[^1]_

![](resources/choosing_a_package.png)

[^1]: [https://stackoverflow.com/questions/28650957]()

::: {.notes}
Coline

- A typical "first analysis" question
- Immediately need to make a decision about which packages to use: base R, `dplyr` or `data.table`
- Which one is right for me?
:::

## A Universal Conundrum

### By choosing packages, we're choosing our [^2]

* Feature set
* Dependency footprint
* Integration with other packages
* Preferred lifecycle management of our tools
* Community that we can lean on for help

[^2]: [It depends: A dialog about dependencies, Jim Hester (2019)](https://www.jimhester.com/talk/2019-rsc-deps/)

## A Universal Conundrum

### Regulated Industries: Justification as a Requirement

![](resources/white_paper_guidelines.png)

## Goals

We have two objectives:

* Generation of risk-based quality indicators

Provide a community-maintained catalog of package quality indicators ("risk metrics")

* Package recommendations

Serve subsets of packages that conform to a specified risk tolerance

::: {.notes}
Coline
- Provide a community-maintained catalog of package quality indicators ("risk metrics")
- Serve quality indicators in a standard format
- Document the system used to perform quality assessment
- Demonstrate how regulatory-ready risk assessments can be provided using public quality indicators
- Serve subsets of packages that conform to a specified risk tolerance
- Improve transparency of industry R package adoption, endorsement and regulator interaction
:::

## An evolving R ecosystem

> In close communication with many beloved R projects

:::{ .columns }

:::{ .column .center .panel width="29%" }
![](resources/rc-logo.png)

### Submissions Working Group
:::

:::{ .column .center .panel width="29%" }
![](resources/rc-logo.png)

### Repositories Working Group
:::

:::{ .column .center .panel width="29%" }
![](resources/pharmaverse.png){ width="30%" }

### `pharmaverse`
:::

:::{ .column .center .panel width="29%" }
![](resources/r-hub.png){ width="50%" }

targetting `repos` integration
:::

:::{ .column .center .panel width="29%" }
![](resources/r-lib.png){ width="40%" }

### `r-lib/pak`

targetting `pak` integration
:::

:::

## Pilot Implementation

_focus on proving capabilities, quick development_

![](resources/pilot.drawio.svg){ style='width: 100%; height: 100%; margin: 2em 0;' }

# Interacting with the repo

## Packages risk filters

::: columns
::: {.column width="50%"}

- Helper package for system administrators
- Restricts packages available for installation to those fitting a policy
- Uses packages metadata in the repo
- May be used together with manual checks (e.g., read a statistical review)
:::
::: {.column width="50%"}
```{mermaid}
flowchart TD
  A[All packages] --> B{Code\n covr.\n > 95%?}
  B -- Yes --> C{Has\n doc.?}
  C -- Yes --> D(Available for safety-critical activities)
```
:::
:::

::: {.notes}
Yann
:::

## Usage

```{r}
#| echo: false
summarize_available_packages <- function(ap) {
  out <- cbind(ap, data.frame("..." = "..."))
  out <- out[c(1:4, nrow(ap)), c("Package", "...")]
  out[4,] <- c("...", "...")
  rownames(out) <- c("1", "2", "3", "...", nrow(ap))
  out
}
```

::: columns
::: {.column width="50%"}
### Unfiltered
```{r}
#| echo: true
#| eval: false
available.packages()
```
```{r}
#| echo: false
pkgs_nofltr_path <- file.path(here::here(), "resources", "pkgs_nofltr.rds")
pkgs_nofltr <- readRDS(file = pkgs_nofltr_path)
summarize_available_packages(pkgs_nofltr)
```
:::

::: {.column width="50%"}
### Filtered
```{r}
#| echo: true
#| eval: false
fltr <- risk_filter(covr_coverage > 0.95
  & has_vignettes)
options(available_packages_filters = fltr)
available.packages()
```
```{r}
#| echo: false
pkgs_fltr_path <- file.path(here::here(), "resources", "pkgs_fltr.rds")
pkgs_fltr <- readRDS(file = pkgs_fltr_path)
summarize_available_packages(pkgs_fltr)
```
:::
:::

::: {.notes}
Yann
:::

# Repository ‘back-end’

## Infrastructure setup

::: columns
::: {.column width="50%"}
- Hosts risk assessment metadata
- Links to artifacts of the R-hub check system (via `DownloadURL`)
- Integrates with `pak::pkg_install`
- Supports multiple levels of risk tolerance
:::

::: {.column width="50%"}
DCF file forked from ``r-hub/repos``
```{.yaml}
Package: bslib
Version: 0.6.1
Depends: R (>= 2.10), R (>= 4.4), R (< 4.4.99)
License: MIT + file LICENSE
DownloadURL:
         https://github.com/cran/bslib/releases/download/0.6.1/bslib_0.6.1_b4_R4.4_x86_64-pc-linux-gnu-ubuntu-22.04.tar.gz
Built: R 4.4.0; ; 2023-11-29 16:39:06 UTC; unix
RVersion: 4.4
Platform: x86_64-pc-linux-gnu-ubuntu-22.04
Imports: base64enc, cachem, grDevices, htmltools (>= 0.5.7), jquerylib (>= 0.1.3),
         jsonlite, lifecycle, memoise (>= 2.0.1), mime, rlang, sass (>= 0.4.0)
...
```
Added fields for risk-based assessment
```{.yaml}
riskmetric_run_date: 2023-06-21
riskmetric_version: 0.2.1
covr_coverage: 0.852
has_vignettes: 1
remote_checks: 0.846
...
```
:::
:::

::: {.notes}
Yann
:::

## Packages cohort validation workflow

### Risk assessment pipeline


:::: {.columns}
::: { .column .center width="32%" }
![](resources/package-inspect-svgrepo-com.svg){ width="50%" height="50%" }

Calculates package QA metadata on updated packages and their reverse dependencies
:::

::: {.column .center width="32%"}
![](resources/document-code-2-svgrepo-com.svg){ width="50%" height="50%" }

Produces logs and other reproducibility data
:::

::: {.column .center width="32%"}
![](resources/house-svgrepo-com.svg){ width="50%" height="50%" }

In the future: can run on in-house infrastructure
:::
::::

::: {.notes}
Yann
:::

## Packages cohort validation workflow

```{dot}
digraph D {
  node [shape=plaintext fontname="Sans serif" fontsize="30"];

  pkg_1 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_1</b></td></tr>
     <tr><td align="left">Version: 1.15</td></tr>
     <tr><td align="left">covr_coverage: 0.967</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];
   
  pkg_2 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_2</b></td></tr>
     <tr><td align="left">Version: 3.5</td></tr>
     <tr><td align="left">covr_coverage: 0.984</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];
   
  pkg_3 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_3</b></td></tr>
     <tr><td align="left">Version: 1.9</td></tr>
     <tr><td align="left">covr_coverage: 0.992</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];
   
  pkg_4 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_4</b></td></tr>
     <tr><td align="left">Version: 0.5</td></tr>
     <tr><td align="left">covr_coverage: 0.864</td></tr>
     <tr><td align="left">has_vignettes: 0</td></tr>
   </table>>];
   
  pkg_5 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_5</b></td></tr>
     <tr><td align="left">Version: 4.2</td></tr>
     <tr><td align="left">covr_coverage: 0.924</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];

  pkg_2 -> pkg_1;
  pkg_3 -> pkg_1;
  pkg_3 -> pkg_2;
  pkg_5 -> pkg_4;
}
```

::: {.notes}
Yann
:::

## Packages cohort validation workflow

```{dot}
digraph D {
  node [shape=plaintext fontname="Sans serif" fontsize="30"];

  pkg_1 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_1</b></td></tr>
     <tr><td align="left">Version: 1.15</td></tr>
     <tr><td align="left">covr_coverage: <b><font color="darkgreen">...</font></b>      </td></tr>
     <tr><td align="left">has_vignettes: <b><font color="darkgreen">...</font></b>      </td></tr>
   </table>>];
   
  pkg_2 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_2</b></td></tr>
     <tr><td align="left">Version: <b><font color="darkgreen">3.6</font></b></td></tr>
     <tr><td align="left">covr_coverage: <b><font color="darkgreen">...</font></b>      </td></tr>
     <tr><td align="left">has_vignettes: <b><font color="darkgreen">...</font></b>      </td></tr>
   </table>>];
   
  pkg_3 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_3</b></td></tr>
     <tr><td align="left">Version: 1.9</td></tr>
     <tr><td align="left">covr_coverage: 0.992</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];
   
  pkg_4 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_4</b></td></tr>
     <tr><td align="left">Version: 0.5</td></tr>
     <tr><td align="left">covr_coverage: 0.864</td></tr>
     <tr><td align="left">has_vignettes: 0</td></tr>
   </table>>];
   
  pkg_5 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_5</b></td></tr>
     <tr><td align="left">Version: 4.2</td></tr>
     <tr><td align="left">covr_coverage: 0.924</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];

  pkg_2 -> pkg_1;
  pkg_3 -> pkg_1;
  pkg_3 -> pkg_2;
  pkg_5 -> pkg_4;
}
```

::: {.notes}
Yann
:::

## Packages cohort validation workflow

```{dot}
digraph D {
  node [shape=plaintext fontname="Sans serif" fontsize="30"];

  pkg_1 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_1</b></td></tr>
     <tr><td align="left">Version: 1.15</td></tr>
     <tr><td align="left">covr_coverage: <b><font color="darkgreen">0.967</font></b></td></tr>
     <tr><td align="left">has_vignettes: <b><font color="darkgreen">1</font></b></td></tr>
   </table>>];
   
  pkg_2 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_2</b></td></tr>
     <tr><td align="left">Version: <b><font color="darkgreen">3.6</font></b></td></tr>
     <tr><td align="left">covr_coverage: <b><font color="darkgreen">0.987</font></b></td></tr>
     <tr><td align="left">has_vignettes: <b><font color="darkgreen">1</font></b></td></tr>
   </table>>];
   
  pkg_3 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_3</b></td></tr>
     <tr><td align="left">Version: 1.9</td></tr>
     <tr><td align="left">covr_coverage: 0.992</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];
   
  pkg_4 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_4</b></td></tr>
     <tr><td align="left">Version: 0.5</td></tr>
     <tr><td align="left">covr_coverage: 0.864</td></tr>
     <tr><td align="left">has_vignettes: 0</td></tr>
   </table>>];
   
  pkg_5 [ label=<
   <table border="1" cellborder="0" cellspacing="1">
     <tr><td align="left"><b>pkg_5</b></td></tr>
     <tr><td align="left">Version: 4.2</td></tr>
     <tr><td align="left">covr_coverage: 0.924</td></tr>
     <tr><td align="left">has_vignettes: 1</td></tr>
   </table>>];

  pkg_2 -> pkg_1;
  pkg_3 -> pkg_1;
  pkg_3 -> pkg_2;
  pkg_5 -> pkg_4;
}
```

::: {.notes}
Yann
:::

# Our roadmap

## What's next

_Automating up-to-date quality metrics to support sponsor risk assessment_

![](resources/next-steps.drawio.svg){ style='width: 100%; height: 100%; margin: 2em 0;' }

## Reference container image(s)

:::: {.columns}
::: {.column .center width="32%"}
![](resources/laptop-svgrepo-com.svg){ width="50%" height="50%" }

Should mimic environments of companies and health authority reviewers
:::

::: {.column .center width="32%"}
![](resources/settings-gear-svgrepo-com.svg){ width="50%" height="50%" }

To be used by the Regulatory R Repository for packages cohort validation
:::

::: {.column .center width="32%"}
![](resources/dialog-svgrepo-com.svg){ width="50%" height="50%" }

Main intent: start a cross-company dialogue on infrastructure
:::
::::

# Closing

## ![](resources/rc-logo.png){height="1.5em" style="padding: 0.2em; margin: 0 0.5em 0 0; border-radius: 0.5em; background: #FFF; vertical-align: middle;"} [R Consortium]{.hidden} Impact

<style>
.image-container {
  display: flex;
  flex-direction: column;
  align-items: center;
}

.image-container img {
  margin: 0.2em;
  max-width: 7em;  /* Adjust the size of the images as needed */
  vertical-align: middle;
}

.panel {
  font-size: 0.6em;
  background: rgba(0.75, 0.75, 0.75, 0.1);
  padding: 1%;
  margin: 1%;
  border-radius: 0.5em;
  border-bottom: solid 0.2em rgba(1, 1, 1, 0.1);
}

.center {
  text-align: center;
}
</style>

<div class="image-container">
  <img src="resources/r-ladies-logo.png">
  <img src="resources/latinr-logo.png">
  <img src="resources/r-medicine-logo.png">
  <img src="resources/useR-logo.png">
</div>

::: {.columns}
::: {.column width="29%" .panel .center}
### Community Grants & Sponsorships
Over USD $1.4 Million
:::

::: {.column width="29%" .panel .center}
### Organizing Large Scale Collaborative Projects
R Validation Hub, R-Ladies
:::

::: {.column width="29%" .panel .center}
### Co-Host Multidisciplinary Data Science Forums
Stanford Data Institute
:::

::: {.column width="29%" .panel .center}
### Direct Support for Key R Events
R/Medicine, R/Pharma, useR!, LatinR, and more
:::

::: {.column width="29%" .panel .center}
### Direct Worldwide Support for R User Groups
:::
:::

## Join us

### [r-consortium.org](https://www.r-consortium.org)

![](resources/rc-logo.png){ width="60%" .nostretch }

* Help guide the future direction of the R language
* Collaborate on cross industry initiatives
* Raise your leadership profile in the R Community
* Protect your investment in R while supporting the common good

## Thank you

### To our Core Team members

::: { .columns }
::: { .column style="font-size: 0.85em; width: 30%;" }
* Coline Zeballos, <em>Roche</em>
* Doug Kelkhoff, <em>Roche</em>
* Jaime Pires, <em>Roche</em>
* Yann Féat, <em>mainanalytics</em>
* Andrew Borgman, <em>Biogen</em>
* Astrid Radermacher, <em>Jumping Rivers</em>
* Colin Gillespie, <em>Jumping Rivers</em>
:::
::: { .column style="font-size: 0.85em; width: 30%;" }
* Magnus Mengelbier, <em>Limelogic</em>
* Nicoles Jones, <em>Denali Therapeutics</em>
* Ramiro Magno, <em>Pattern Institute</em>
* Stefan Doering, <em>Boehringer-Ingelheim</em>
* Kevin Kunzmann, <em>Boehringer-Ingelheim</em>
:::
::: { .column style="font-size: 0.85em; width: 30%;" }
* Matthias Trampisch, <em>Boehringer-Ingelheim</em>
* Wilmar Igl, <em>Icon Plc</em>
* Lluís Revilla, <em>IrsiCaixa AIDS Research Institute</em>
* Yoni Sidi, <em>Pinpoint Strategies</em>
* Zhenglei Gao, <em>Bayer</em>
:::
:::