-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #66 from qbicsoftware/release/1.1.0
Release 1.1.0
- Loading branch information
Showing
144 changed files
with
10,662 additions
and
8,477 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # # This workflow runs all available tests and builds the .jar files | ||
| name: Varianstore CI | ||
|
|
||
| # Trigger the workflow on push and pull request activity | ||
| on: [push, pull_request] | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3.1.0 | ||
|
|
||
| # set up JDK 11 | ||
| - name: Set up JDK 11 | ||
| uses: actions/setup-java@v3.6.0 | ||
| with: | ||
| distribution: 'corretto' | ||
| java-version: '11' | ||
| cache: maven | ||
|
|
||
| # build with maven and run tests with Postgresql | ||
| - name: Build with Maven and run tests with PostgreSQL | ||
| run: mvn --batch-mode -Dmicronaut.environments=test-postgres -Dtest='!life.qbic.db.mariadb.**' verify | ||
|
|
||
| # build with maven and run tests with MariaDB | ||
| - name: Build with Maven and run tests with MariaDB | ||
| run: mvn --batch-mode -Dmicronaut.environments=test-mariadb -Dtest='!life.qbic.db.postgres.**' verify |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # This workflow uses actions that are not certified by GitHub. | ||
| # They are provided by a third-party and are governed by | ||
| # separate terms of service, privacy policy, and support | ||
| # documentation. | ||
|
|
||
| name: Create and publish a Docker image | ||
|
|
||
| on: | ||
| push: | ||
| branches: ['release'] | ||
|
|
||
| env: | ||
| REGISTRY: ghcr.io | ||
| IMAGE_NAME: ${{ GITHUB_REPOSITORY_OWNER }}/variantstore | ||
|
|
||
| jobs: | ||
| build-and-push-image: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| packages: write | ||
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Log in to the Container registry | ||
| uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9 | ||
| with: | ||
| registry: ${{ env.REGISTRY }} | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Extract metadata (tags, labels) for Docker | ||
| id: meta | ||
| uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38 | ||
| with: | ||
| images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} | ||
|
|
||
| - name: Build and push Docker image | ||
| uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc | ||
| with: | ||
| context: . | ||
| push: true | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| labels: ${{ steps.meta.outputs.labels }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| # Contributing guide | ||
|
|
||
| Thank you for investing your time in contributing to our project! We appreciate any bug reports, feature requests or feedback in general. | ||
|
|
||
| ## Issues | ||
|
|
||
| If you find a bug or have a feature request, feel free to open an [issue](<https://github.com/qbicsoftware/variantstore-service/issues>). | ||
|
|
||
| ## Pull Requests | ||
|
|
||
| If you want to contribute with changes to the code or documentation, feel free to open a pull request (PR) against the [development branch](https://github.com/qbicsoftware/variantstore-service/tree/development). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Variantstore | ||
|
|
||
| [](https://github.com/qbicsoftware/variantstore-service/actions/workflows/ci_test_build_package.yml) | ||
| [](https://github.com/qbicsoftware/variantstore-service/releases) | ||
| [](https://opensource.org/licenses/MIT) | ||
|
|
||
| ## Introduction | ||
|
|
||
| The **Variantstore** is a Java/Groovy-based service application implemented using the [Micronaut framework](https://micronaut.io) and enables storage and access to information on genomic variants and metadata from a connected database *via* a RESTful API. | ||
|
|
||
| ## Features | ||
|
|
||
| - Import variants as Variant Call Format (VCF) files, annotated using [SnpEff](http://snpeff.sourceforge.net) or [VEP](https://www.ensembl.org/info/docs/tools/vep/index.html>) | ||
| - Import metadata in JSON together with variants (see [Usage](/docs/USAGE..md) for details) | ||
| - Import gene information (Ensembl, GFF3 files) | ||
| - Support for [MariaDB](https://mariadb.com) and [PostgreSQL](https://www.postgresql.org) backend | ||
| - Query information on variants, genes, and cases via (secured) REST endpoints | ||
| - Ask Beacon endpoint if a specific variant exists in the store | ||
| - Export variants in Variant Call Format (VCF) and [FHIR](https://www.hl7.org/fhir/) | ||
|
|
||
| ## Full Documentation | ||
|
|
||
| See [Documentation](/docs/README.md) for the full detailed installation, run and configuration instructions. | ||
|
|
||
| Detailed documentation of the RESTful API is additionally available on [SwaggerHub](https://app.swaggerhub.com/apis/christopher-mohr/variantstore/). | ||
|
|
||
| ## Database | ||
|
|
||
| In the current version, the **Variantstore** service can be used with a MariaDB and PostgreSQL database. If you want to use a different DBMS, you have to provide an implementation for the `VariantstoreStorage` interface, make sure to use the same database model and set up the datasource accordingly in the `application.yml`. | ||
|
|
||
| ## Contributions and Support | ||
|
|
||
| If you would like to contribute to this pipeline or, please see the [contributing guidelines](/CONTRIBUTING.md). | ||
|
|
||
| For further information or help, don't hesitate to get in touch with us via [mail](mailto:christopher.mohr@uni-tuebingen.de) or [Twitter](https://twitter.com/cmohr_tue). | ||
|
|
||
| ## Credits | ||
|
|
||
| The Variantstore was created within the [DIFUTURE](https://difuture.de) (Data Integration for Future Medicine) Consortium. | ||
|
|
||
| This service was designed and implemented by [Christopher Mohr](https://github.com/christopher-mohr). | ||
| For a full list of authors, please refer to the file [AUTHORS](AUTHORS). |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # Configuration | ||
|
|
||
| The configuration of the Variantstore instance is essentially done by setting the environment variables which are used in the [application.yml](../src/main/resources/application.yml). If you want/need to set other properties, please refer to the [Micronaut Documentation](https://docs.micronaut.io/latest/guide/#introduction). | ||
|
|
||
| ## Server | ||
|
|
||
| You can set the port on which the Variantstore instance is supposed to run by setting the optional environment variable `VARIANTSTORE_PORT`(default: `8080`). | ||
|
|
||
| ```yml | ||
| server: | ||
| port: ${variantstore-port:8080} | ||
| ``` | ||
| In addition, you can enable HTTPS support by setting `VARIANTSTORE_SSL_ENABLED` to `true` and enable the generation of a self-isgned certificate by Micronaut (`VARIANTSTORE_SSL_SELFSIGNED`). | ||
|
|
||
| ## Security (Authentication and Authorization) | ||
|
|
||
| Security (authentication and authorization) is enabled by default but you can deactivate it by setting the environment variable `VARIANTSTORE_SECURITY_ENABLED` to `false`. If security is enabled all protected endpoints (i.e. all endpoints except the `/beacon` endpoint) can only be accessed by authenticated users. | ||
|
|
||
| In addition to that, the Variantstore supports authentication with OAuth 2.0 servers. This includes support for the OpenID standard. Authentication using Keycloak as provider has been tested. However, other provider that support OpenID such as [Okta](https://developer.okta.com/), [AWS Cognito](https://aws.amazon.com/cognito), and [Google](https://developers.google.com/identity/protocols/OpenIDConnect) should work as well when configured properly. | ||
|
|
||
| Authentication mode is set to `idtoken` by default. The mode can be changed by setting the environment variable `VARIANTSTORE_AUTHENTICATION_MODE`. Please refer to the official Micronaut [docs](https://micronaut-projects.github.io/micronaut-security/latest/guide/#authenticationStrategy) for valid values. | ||
|
|
||
| In order to enable authentication with OAuth 2.0 servers, set `VARIANTSTORE_OAUTH2_ENABLED` to `true` and provide all necessary details using the following environment variables: | ||
|
|
||
| * `VARIANTSTORE_OAUTH2_CLIENT_ID` | ||
| * `VARIANTSTORE_OAUTH2_CLIENT_SECRET` | ||
| * `VARIANTSTORE_OAUTH2_ISSUER` | ||
| * `VARIANTSTORE_OAUTH2_AUTH_URL` | ||
| * `VARIANTSTORE_OAUTH2_TOKEN_URL` | ||
|
|
||
| > You can check out the official Micronaut security [docs](https://micronaut-projects.github.io/micronaut-security/latest/guide/#oauth) for configuration examples. | ||
|
|
||
| ## Database | ||
|
|
||
| In the current version, the **Variantstore** service can be used with a PostgreSQL or MariaDB database instance. If you would like to use a different DBMS, you have to provide an implementation for the `VariantstoreStorage` interface, make sure to use the same database model as provided and set up the data source accordingly in the `application.yml`. | ||
|
|
||
| The main database [model](/models/) expected by the **Variantstore** looks like the following for the currently supported two database systems: | ||
|
|
||
| <details> | ||
| <summary>PostgreSQL schema</summary> | ||
|
|
||
|  | ||
| </details> | ||
|
|
||
| <details> | ||
| <summary>MariaDB schema</summary> | ||
|
|
||
|  | ||
| </details> | ||
|
|
||
| \ | ||
| Additionally, a database with the following [table](/models/transaction-db.sql) is needed to track the import transactions in the Variantstore: | ||
|
|
||
|  | ||
|
|
||
| The `Variantstore` data source can be configured by setting the following environment variables: `DB_HOST` (database host address), `DB_NAME` (database name), `DB_USER` (database user) and `DB_PWD` (database password). In addition, `DB_TRANSACTION_HOST`, `DB_TRANSACTION_NAME`, `DB_TRANSACTION_USER`, and `DB_TRANSACTION_PW` have to be specified for the transaction database. | ||
|
|
||
| If you need to change or add additional properties to the JDBC database connection URL, please do so in the `application.yml`. | ||
|
|
||
| ```yml | ||
| datasources: | ||
| variantstore_postgres: | ||
| url: jdbc:postgresql://${db-host:""}/${db-name:""} | ||
| username: ${db-user:""} | ||
| password: ${db-pwd:""} | ||
| driverClassName: org.postgresql.Driver | ||
| transactions: | ||
| url: jdbc:postgresql://${db-transaction-host:""}/${db-transaction-name:""}?... | ||
| username: ${db-transaction-user:""} | ||
| password: ${db-transaction-pwd:""} | ||
| driverClassName: org.postgresql.Driver | ||
| #variantstore_mariadb: | ||
| # url: jdbc:mariadb://${db-host:""}/${db-name:""}?... | ||
| # username: ${db-user:""} | ||
| # password: ${db-pwd:""} | ||
| # driverClassName: org.mariadb.jdbc.Driver | ||
| #transactions_mariadb: | ||
| # url: jdbc:mariadb://${db-transaction-host:""}/${db-transaction-name:""}?... | ||
| # username: ${db-transaction-user:""} | ||
| # password: ${db-transaction-pwd:""} | ||
| # driverClassName: org.mariadb.jdbc.Driver | ||
| ``` | ||
|
|
||
| The data source configuration is set to PostgreSQL by default. If you want to use a MariaDB database comment out `variantstore_postgres` and `transactions` and uncomment `variantstore_mariadb` and `transactions_mariadb`. Further, `database.specifier` has to be set to `variantstore-mariadb`. Default: | ||
|
|
||
| ```yml | ||
| database: | ||
| specifier: variantstore-postgres | ||
| ``` | ||
|
|
||
| Additionally, you have to enable (`enabled: true`) Flyway migration for the MariaDB data sources and disable it for the PostgreSQL data sources (`enabled: false`) below in the Flyway block: | ||
|
|
||
| ```yml | ||
| flyway: | ||
| datasources: | ||
| ... | ||
| ``` | ||
|
|
||
| ## Import | ||
|
|
||
| The import of variants to the store happens in batches (default: 250.000). If you would like to change the default batch size, you can use the environment variable `MAX_NUMBER_VARIANTS_PER_BATCH`. | ||
|
|
||
| ## Logging | ||
|
|
||
| All requests to the Variantstore are logged. The default location is ``/tmp`` but you can specify a different location by setting the environment variable ``SERVICES_LOG_PATH``. The generated log file is called ``variantstore.log`` whereas older log files follow the following naming scheme: ``variantstore.%d{dd-MMM}.log.gz"`` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Installation | ||
|
|
||
| The following describes the minimal steps that have to be taken in order to install a Variantstore instance. You can either [build](#Build) the project from scratch or [run](#Run) the prebuilt `JAR` file. | ||
|
|
||
| ## Requirements | ||
|
|
||
| Variantstore runs on Java 11. Building Variantstore requires Maven, however note that a maven wrapper is provided under the folder [.mvn](/.mvn). | ||
|
|
||
| ### Database | ||
|
|
||
| The complete installation of the Variantstore includes setting up and configuring the databases to store the genomic information. Details can be found under [CONFIGURATION](CONFIGURATION.md). | ||
|
|
||
| ## Build | ||
|
|
||
| To compile and run the project: | ||
|
|
||
| ```bash | ||
| git clone git@github.com:qbicsoftware/variantstore-service.git | ||
| cd variantstore-service/ | ||
| mvn clean compile | ||
| mvn exec:exec | ||
| ``` | ||
|
|
||
| To build the project: | ||
|
|
||
| ```bash | ||
| git clone git@github.com:qbicsoftware/variantstore-service.git | ||
| cd variantstore-service/ | ||
| mvn clean package | ||
| ``` | ||
|
|
||
| This command will create an executable `JAR` in your current working directory under `/target`. | ||
|
|
||
| ## Run | ||
|
|
||
| To run a prebuilt Variantstore instance, download the [latest](https://github.com/qbicsoftware/variantstore-service/releases/latest/) Variantstore release and execute. | ||
|
|
||
| ```bash | ||
| java -jar variantstore-{version}.jar | ||
| ``` | ||
|
|
||
| ## Docker | ||
|
|
||
| Docker images are created for [Releases](https://github.com/qbicsoftware/variantstore-service/releases) and are available under [Packages](https://github.com/orgs/qbicsoftware/packages?repo_name=variantstore-service). |
Oops, something went wrong.