v1.0.0-beta.3

Hutch Agent v1.0.0-beta.3

The "API alignment" release.

This release primarily aligns Hutch's TRE Controller API calls (and its own endpoints that the TRE Controller interacts with) with last minute interface changes.

It also fixes some bugs / adds some minor features:

• License properties in config are supported in a limited fashion - Name and Description only, as per the 5S Crate Profile examples.
• As per the 5S Crate Profile, Hutch now supports input crates that contain an inline Workflow Crate as a Dataset entity, instead of referencing a remote Workflow Crate.
• BagIt checksum verification is now performed on incoming 5S Crates submitted to the jobs/ endpoint
• Data Access credentials are set on predefined Environment Variable keys now, using Wfexs' support for this
  • This means workflows don't need to define dummy inputs with predefined keys; instead workflows/tools can consume these details from their container environment.
• Many formerly "Information" logs are now "Debug" logs, reducing clutter in the default output from Hutch.

Assets available for download:

• Hutch Agent for Linux (x64) native
• Hutch Agent for the .NET Runtime (cross platform, requires .NET 7.x)

What's Changed

• AB#130901: ansible updates by @dcl10 in HDRUK/hutch#268
• AB#108973: Verify BagIt Checksums before execution? by @dcl10 in HDRUK/hutch#270
• AB#130909: Debug Logging updates by @beforan in HDRUK/hutch#269
• AB#132633: bag-info.txt not bagit-info.txt by @dcl10 in HDRUK/hutch#272
• AB#131799: when containers are deleted from ouputs, we retain metadata files by @beforan in HDRUK/hutch#273
• AB#130480: Set DB settings as env vars in stage file by @vpnu in HDRUK/hutch#274
• AB#126517: Update Ansible to fetch a release instead of cloning git by @dcl10 in HDRUK/hutch#271
• AB#124151: Crate relative workflow path by @vpnu in HDRUK/hutch#275
• AB#130908: License properties? seem to be broken. by @dcl10 in HDRUK/hutch#276
• AB#132881: Controller api changes by @beforan in HDRUK/hutch#277

Full Changelog: HDRUK/hutch@v1.0.0-beta.2...v1.0.0-beta.3

v1.0.0-beta.2

Hutch Agent v1.0.0-beta.2

"TRE-FX primary Hutch functionality complete" edition.

TRE-FX status

In this release Hutch is known to run through without requiring Standalone mode, though currently internet access is still required for container image fetching, and some wfexs config is still limited.

• Hutch will now behave as the TRE-FX stack expects regarding the use of OIDC tokens for TRE Controller API interactions, and MinIO Intermediary Store interactions.
• Hutch has all TRE Controller API interactions implemented, so standalone mode is not required.
• Data access credentials from job submissions are now used, but are currently set as workflow inputs with pre-configured names
  • the intent is to move to environment variables with preconfigured keys, so they're not needed in workflow definitions

Assuming the TRE Controller API signatures have lined up correctly, Hutch itself has all the functionality it needs to operate within the TRE-FX stack.

Development improvements

Hutch can now be configured more easily to work with different, sometimes more limited, development environments.

This should make it easier to work on different areas of Hutch without needing the whole stack set up and configured at a time.

Initial documentation is available around opting in or out of:

• TRE Controller API interactions (Standalone mode, or the "Dummy" Controller API)
• OIDC auth for MinIO (vs. direct credentials)
• skipping Workflow Execution (not needing Wfexs)

Continuing Roadmap

The road to 1.0.0 from here is:

• fixing practical issues with stack interaction (such as mismatches in the aforementioned API signatures)
• outstanding issue fixes
• configuration and documentation
  • ansible updates
  • clearer docs
  • more complete samples for deployment within the whole stack
• external issue testing and remedy
  • e.g. issues with Wfexs or other external services rather than Hutch itself

Known issues:

• Required Wfexs configuration is still limited, (and ansible updates are yet to be completed):
  • use the trefx-fix branch on the following fork: https://github.com/uon-drs/WfExS-backend
  • run full wfexs install
  • run singularity local install
  • install graphviz in the environment
  • local config should be configured to use singularity
  • Hutch should therefore be configured to match the container engine.
• airgapping of container images does not work at this time due to the use of singularity as the container engine. There are outstanding issues with podman and wfexs at the time of this release.

Assets available for download:

• Hutch Agent for Linux (x64) native
• Hutch Agent for the .NET Runtime (cross platform, requires .NET 7.x)

What's Changed

• AB#130479: Data access (part 1) by @beforan in HDRUK/hutch#261
• AB#130477: Default ports by @beforan in HDRUK/hutch#262
• AB#129881: Keycloak auth by @beforan in HDRUK/hutch#263
• AB#131635: Dummy controller api by @beforan in HDRUK/hutch#264
• AB#131740: Workflow Execution skipping by @beforan in HDRUK/hutch#265
• AB#131017: Terminology updates by @beforan in HDRUK/hutch#266
• AB#131714: Controller api final outcome by @beforan in HDRUK/hutch#267

Full Changelog: HDRUK/hutch@v1.0.0-beta.1...v1.0.0-beta.2

v1.0.0-beta.1

Hutch Agent v1.0.0-beta.1

The "standalone" release.

In this release, Hutch is known to run all the way through in "standalone" mode, with an internet connection, given an appropriate WfExS and Hutch configuration.

Known good configuration:

• Wfexs:
  • use the trefx-fix branch on the following fork: https://github.com/uon-drs/WfExS-backend
  • run full wfexs install
  • run singularity local install
  • install graphviz in the environment
  • local config should be configured to use singularity
• Hutch
  • configure to generate full provenance crates
    • WorkflowExecutor:GenerateFullProvenanceCrate = true
    • this setting will change in a future release (to be default apart from anything else)
  • configure to use singularity:
    • WorkflowExecutor:ContainerEngine = singularity
  • put in "standalone mode" (to skip interactions with the TRE Controller API)
    • Flags:StandaloneMode = true

Known issues:

• TRE Controller interaction doesn't work yet due to missing authentication implementation; hence "Standalone mode" only for this release.
• Data Access credentials from job submissions are not used yet: they instead must be provided directly as workflow inputs in the job submission crate at this time.
• airgapping of container images does not work at this time due to the use of singularity as the container engine. There are outstanding issues with podman and wfexs at the time of this release.

Assets available for download:

• Hutch Agent for Linux (x64) native
• Hutch Agent for the .NET Runtime (cross platform, requires .NET 7.x)

New Contributors

• @beforan made their first contribution in HDRUK/hutch#2
• @vpnu made their first contribution in HDRUK/hutch#3
• @Samad198 made their first contribution in HDRUK/hutch#17
• @Jivanam made their first contribution in HDRUK/hutch#103

Full Changelog: https://github.com/HDRUK/hutch/commits/v1.0.0-beta.1