Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support NGFF using n5-universe #328

Merged
merged 134 commits into from
Oct 10, 2024
Merged

Conversation

ksugar
Copy link
Collaborator

@ksugar ksugar commented Oct 4, 2024

Overview

This PR implements loaders for the NGFF formats using n5-universe.

Supported formats

Format Metadata Storage Confirmed converter
HDF5 N/A local only Plugins › BigDataViewer › Export Current Image as XML/HDF5 in Fiji
OME-Zarr OME-NGFF v0.4 local or remote bioformats2raw
N5 COSEM or N5 Viewer local or remote File › Save As › HDF5/N5/Zarr/OME-NGFF ... in Fiji

Usage

  1. Select new project from URL in Mastodon launcher.
image
  1. Press the browse button to open N5 dialog.
image
  1. Click the Browse button to load local dataset, or input a URL and and press the Detect datasets button.
    Please use the URLs starting with http: or https: because the s3: protocol is not fully supported at the moment.
image
  1. Select a (multiscale) dataset and press the OK button.
image
  1. Save a .xml file that contains the SpimData information.
image
  1. Press the create button.
image
  1. Visualize the data in the bdv window.
image

Work with authentication of the remote storage

If the remote storage requires authentication or fails to load data, a dialog will appear, allowing you to enter the username (or AWS_ACCESS_KEY_ID) and password (or AWS_SECRET_ACCESS_KEY).
image

Alternatively, the user can set the credentials permanently on the system using the AWS credentials provider chain.
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html

AWS credentials provider chain that looks for credentials in this order:

  • Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
  • Java System Properties - aws.accessKeyId and aws.secretKey
  • Web Identity Token credentials from the environment or container
  • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
  • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
  • Instance profile credentials delivered through the Amazon EC2 metadata service

tinevez and others added 30 commits August 8, 2023 14:58
This branch reflects an effort to rework the main class of the Mamut
app: the MamutAppModel. The main goal is to make the app model the
core of a Mastodon session, and have all of its fields final, and
have them containing everything we need to run all featuures of the
app.

This is a WIP but so far we have:
- The windowManager is now a field of the app model.
- The app model is immutable: it cannot change under the user feet.
When a user loads a project, it gets another instance, but cannot
modify an existing one.
- And to do this, the main work of this PR is the rework of the project
loading and saving, detailed a bit below.

The ProjectManager class is gone, replaced by 4 classes with only static
methods, in the package org.mastodon.mamut.io. Each one takes care
about loading, saving, importing or exporting.

The loading methods all produces a new MamutAppModel instance. This
instance receives the MamutProject it was created on (handy to save
it later). The app model is properly initiatialized ONCE with
everything. Then a consumer can create a MainWindow to pilot it.

I have updated all the classes so that they compile. The tests all
run (except for the garbage collection from Mathias, which requires
changing two deps). But I am anxious to see if we cover all the use
cases below:

- [x] open a project
- [ ] save a modified project
- [x] 'save as' a project
- [ ] test that we can reload a saved project
- [ ] test that we can reload a 'saved as' project

The importers and exporters work, as seen by the tests, but I would like
to test them and rework their UI.
Then I will rework again the app model, rename it etc...
Also make more explicit their respective role in the javadoc.
We rely on MamutAppModel for non-view related fields.
Also, do not test for appModel == null anymore.
They will generate an error in v2.0.
This commit removes mention of the deprecated SetupAssignments and
BrightnessDialog classes. The actions that brought the corresponding
dialogs are removed as well.
There is some code to still make it possible to save/load bdv settings
and use ones that are generated 'elsewhere'.
Mainly as an effort to reduce the number of fields and methods
in the window manager class. We began with just a manager for BDV
and TrackScheme styles but now we have 5 of them. For now.
Also this will make it easier to extend Mastodon with
new views in the future.
The view lists are stored in a map, indexed by their class, like for
the manager. This diminish the number of fields and methods in the
window manager class.

This also allows for treating views in a general manner without
having to change too much code when a new view will be added.
For this I shall find a way to create view instances in a generic
manner, probably with a factory pattern.
We started with just 2 views (BDV and TrackScheme) but now there are
enough of them is that it is worthwhile having some generic way of
handling them. The next commits will be about coming with a sensible
way of doing it. Ideally, we want to make it possible for 3rd parties
to extends Mastodon with new views without having to recompile it.
- Get rid of IMastodonView and IMastodonFrameView
- An interface for views that have a ViewerFrame,
that MastodonFrameView implements.
- An interface for all Mamut views (branch and core graphs) that
extend from it, and add methods for GroupHandle on onClose().
- MamutView and MamutBranchView both implement this interface.
So that now we can have one factory type for both branch and core
graph views.
Tune the bdv factory accordingly.
@ksugar ksugar changed the base branch from master to dev October 4, 2024 16:39
@tinevez
Copy link
Contributor

tinevez commented Oct 7, 2024

@ksugar Awesome! I will make basic tests then merge. All in time!
Thank you so much Ko

@tinevez tinevez merged commit 4a5a25d into mastodon-sc:dev Oct 10, 2024
1 check failed
@tinevez
Copy link
Contributor

tinevez commented Oct 10, 2024

Doc for the N5 dialog is here:
https://github.com/saalfeldlab/n5-ij?tab=readme-ov-file#open-n5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants