Add author and tools details in RO-Crate #18820

Marie59 · 2024-09-16T17:33:42Z

Here we go again !
I am launching this PR on a bit of work I did to ameliorate the RO-Crate plugin.

I updated the _add_worfklow function. I added the information on the author of the workflow:

Name
Orcid
The update checks if the workflow attribute 'creator_metadata' exist. If it exists, it writes into the ro-crate the name of the creator (if there is no name it leaves a "") same for the idenifier (by default i put the ORCID maybe it's not a good Idea and if there is no Orcid it leaves a ""). If there is no 'creator_metadata' nothing is written on the creator.

I created a function _add_tools that checks the different info in workflow.steps to add some tools details:

Id of the tool
tool's name (label if there is one or id if there is no label)
tools' version
tool's description: the update checks if the tool has an annotations attribute if it exist it is written as the description if it does not the description remains "")
url to the the toolshed (not sure this part is relevant as I did not yet solve how to have the exact url to the tool in the toolshed and I just let the global url to the toolshed)

I am not sure if I wrote that the correct way corresponding to the run crate profile (this is a bit obscure to me)

@pauldg What do you think of this ?
Thanks !!

mvdbeek

Very nice. Any chance you could also add or extend tests ? 3dbe289#diff-9f3ad661ca1701467f48508ea79951efe90aa29a6d02dec3f2998d831e755bb4 contains the existing examples.

lib/galaxy/model/store/ro_crate_utils.py

Marie59 · 2024-09-18T07:54:44Z

Thanks a lot for the review and help @mvdbeek !

stain · 2024-09-19T08:56:25Z

@OliverWoolland could you review this? I think this solves many of the issues you had about the Workflow Run Crate from Galaxy

Marie59 · 2024-09-19T09:34:38Z

TO DO in a 2nd time (maybe in a later PR but I'm just putting that here to keep it in mind):

Add xrefs of tools
Add citations of tools
Add edam ontologies of tools

bgruening · 2024-11-01T21:30:44Z

lib/galaxy/model/store/ro_crate_utils.py

+                                "name": tool_name,
+                                "version": tool_version,
+                                "description": tool_description,
+                                "url": "https://toolshed.g2.bx.psu.edu",  # URL if relevant


This is not always true. Tools could come from multiple toolsheds.

Yes I was wondering if this part was relevant or if I should remove it

Having a url for each tool would be great - but if we can't get this due to the toolbox limitations, it should be removed.

bgruening · 2024-11-01T21:42:04Z

lib/galaxy/model/store/ro_crate_utils.py

+
+        # Add tools used in the workflow
+        self._add_tools(crate)
+        self._add_steps(crate)


Workflows can have workflow inputs. Should we add them here as well? At least the types of the workflow inputs?

That would be nice to have, yes.

This can even be done per tool (though more fiddly). There is already some capturing of the inputs and outputs as as formalParameters, but it seems only to happen if data files are included in the crate. There seem to be some helper functions further down the file for this purpose:

galaxy/lib/galaxy/model/store/ro_crate_utils.py

Line 359 in 39e38c9

def _add_step_parameter_fp(self, step: WorkflowInvocationStep, crate: ROCrate):

elichad · 2024-11-12T14:27:37Z

Quick update from the RO-Crate side (with apologies from the long delay) - the metadata that's being added looks fine to me as I read the code, but I would like to look at an example RO-Crate generated using this PR to make sure the JSON-LD output looks like it should. I'm waiting for @pauldg or @OliverWoolland to provide me with such a crate, whoever gets there first

OliverWoolland · 2024-11-15T08:11:37Z

Hi all, I've been trying to test these changes on a local 24.1.3 instance I have (base commit 71f5048)

I've not been able to get the revisions to run for me, with errors being thrown relating to the use of the toolbox.

I think I've tracked the problem down to the origin of this PR being usegalaxy-eu rather than galaxyproject. The two forks seem to be handling the toolbox a little differently.

I'd be curious if others have found the same? Or if there is an obvious solution?

Apologies if I have misunderstood anything!

martenson · 2024-11-15T10:25:37Z

@OliverWoolland what kind of errors are you seeing? There should only be minor differences on .eu compared to the upstream (release_24.1...usegalaxy-eu:galaxy:release_24.1_europe)
Plus the tests are running fine in CI, and those are run against the merge of this to galaxyproject/galaxy afaik.

OliverWoolland · 2024-11-15T10:46:52Z

Thanks @martenson! This is the error I see

[2024-11-15 10:43:15,368: ERROR/main] Task galaxy.prepare_invocation_download[2d38756c-b893-4a20-8ca7-80e243608cc4] raised unexpected: AttributeError("'GalaxyManagerApplication' object has no attribute '_toolbox'")
Traceback (most recent call last):
  File "/galaxy/server/.venv/lib/python3.12/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/.venv/lib/python3.12/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/celery/__init__.py", line 184, in wrapper
    rval = app.magic_partial(func)(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/.venv/lib/python3.12/site-packages/lagom/wrapping.py", line 28, in _bound_func
    return inner_func(*bound_args, **bound_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/.venv/lib/python3.12/site-packages/lagom/wrapping.py", line 45, in _error_handling_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/celery/tasks.py", line 392, in prepare_invocation_download
    model_store_manager.prepare_invocation_download(request)
  File "/galaxy/server/lib/galaxy/managers/model_stores.py", line 156, in prepare_invocation_download
    with model.store.get_export_store_factory(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/model/store/__init__.py", line 2481, in __exit__
    self._finalize()
  File "/galaxy/server/lib/galaxy/model/store/__init__.py", line 2850, in _finalize
    ro_crate = self._init_crate()
               ^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/model/store/__init__.py", line 2518, in _init_crate
    invocation_crate_builder = WorkflowRunCrateProfileBuilder(self)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/model/store/ro_crate_utils.py", line 49, in __init__
    self.toolbox = self.model_store.app.toolbox
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/app.py", line 373, in toolbox
    return self._toolbox
           ^^^^^^^^^^^^^
AttributeError: 'GalaxyManagerApplication' object has no attribute '_toolbox'. Did you mean: 'toolbox'?

davelopez · 2024-11-15T10:50:40Z

I'm not sure if that is the issue, but note that these changes are on the dev branch not release_24.1...

OliverWoolland · 2024-11-15T11:06:14Z

Fair enough! I'll try and cross-check myself

elichad · 2024-11-21T13:53:36Z

I don't have a local development Galaxy set up already, so I cloned Galaxy dev, checked out this PR, then installed Galaxy from scratch. When I created a simple workflow, invoked it, and tried to export as RO-Crate, I got the same error as @OliverWoolland above.

[2024-11-21 13:14:58,757: ERROR/main] Task galaxy.prepare_invocation_download[a4bab0c4-8c68-422a-bff2-e2a51b5c44a6] raised unexpected: AttributeError("'GalaxyManagerApplication' object has no attribute '_toolbox'")
Traceback (most recent call last):
  File "/home/eli/galaxy/galaxy/.venv/lib/python3.12/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/.venv/lib/python3.12/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/lib/galaxy/celery/__init__.py", line 184, in wrapper
    rval = app.magic_partial(func)(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/.venv/lib/python3.12/site-packages/lagom/wrapping.py", line 28, in _bound_func
    return inner_func(*bound_args, **bound_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/.venv/lib/python3.12/site-packages/lagom/wrapping.py", line 45, in _error_handling_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/lib/galaxy/celery/tasks.py", line 390, in prepare_invocation_download
    model_store_manager.prepare_invocation_download(request)
  File "/home/eli/galaxy/galaxy/lib/galaxy/managers/model_stores.py", line 156, in prepare_invocation_download
    with model.store.get_export_store_factory(
  File "/home/eli/galaxy/galaxy/lib/galaxy/model/store/__init__.py", line 2481, in __exit__
    self._finalize()
  File "/home/eli/galaxy/galaxy/lib/galaxy/model/store/__init__.py", line 2850, in _finalize
    ro_crate = self._init_crate()
               ^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/lib/galaxy/model/store/__init__.py", line 2518, in _init_crate
    invocation_crate_builder = WorkflowRunCrateProfileBuilder(self)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/lib/galaxy/model/store/ro_crate_utils.py", line 48, in __init__
    self.toolbox = self.model_store.app.toolbox
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eli/galaxy/galaxy/lib/galaxy/app.py", line 397, in toolbox
    return self._toolbox
           ^^^^^^^^^^^^^
AttributeError: 'GalaxyManagerApplication' object has no attribute '_toolbox'. Did you mean: 'toolbox'?

I checked out galaxyproject:dev and the export works as expected.

Finally, I merged Marie59:ro-crate2 into galaxyproject:dev on a local branch and the error appeared again.

So the error appears to be somewhere in this PR, but I don't have the expertise required to understand or fix it.

It's visible in CI too - https://github.com/galaxyproject/galaxy/actions/runs/11610465172/job/32407941424?pr=18820 has some 500 errors which I think come from this problem.

mvdbeek · 2024-11-21T14:38:05Z

lib/galaxy/model/store/ro_crate_utils.py

+        Returns:
+            dict: A dictionary containing citations, xrefs, and EDAM operations for the tool.
+        """
+        tool = self.toolbox.get_tool(tool_id)


This won't work, you can't use the toolbox in this context. We can't load up the toolbox in celery workers, that would take too much time and memory. The best solution is to either get that data from a yet to be written toolshed endpoint or load up the tool on demand, neither of these are easy to do or fit into the context of this PR. Maybe you could focus on the other enhancements in this PR.

That's a shame that the toolbox can't be used in this way. But the most essential metadata about the tools is already captured without using the toolbox, so I think it's ok to remove this function and consider this as an enhancement which could be added in future.

It does mean we would have a greater need for the url on line 369, though, as without the xrefs there's no alternative links to find the tool (and the rest of its metadata). Still not absolutely mandatory to have url, but a definite best practice.

(I wondered if we could retrieve the citation/xref/EDAM metadata from something like job_attrs.txt which is already part of the crate, but it's not included there. Nor is the url.)

davelopez · 2024-11-21T14:38:15Z

I had a closer look and this is what I found:

The export process happens in the context of a Celery task - a background task that runs out of the main galaxy process. The "galaxy app" in this context is a GalaxyManagerApplication. This is a "reduced" version of the UniverseApplication that strips out some functionality and services not required outside the main Galaxy process).

Here we can see that the toolbox is set up only in the UniverseApplication (full-fledged Galaxy app):

galaxy/lib/galaxy/app.py

Line 775 in 455da5c

self._configure_toolbox()

Moving this call (along with possibly other toolbox-related calls) to the parent class GalaxyManagerApplication will probably fix it but I don't know if the toolbox was explicitly left out of the GalaxyManagerApplication for strong reasons (like performance, etc.) or if it can be safely set up at that level to make it available in the Celery context.

Maybe someone else from the @galaxyproject/wg-backend can answer this question?

Edit: lol, Marius was answering at the same time here #18820 (comment) so, that would be the answer.

Marie59 · 2024-11-22T14:38:39Z

Hello all,
Thanks for all your feedback and reviews ! So if I understand correctly, for now, I leave this PR on the side 😢 (hopefully will be useful one day)
I can then open a PR for the things added that does not rely on the toolbox.
Does this sounds good ? Did I got everything right ?

Thanks again for all your help !!

mvdbeek · 2024-11-22T15:08:04Z

Yes, or drop the things which require tool access from this PR.

elichad · 2024-11-22T16:28:47Z

@Marie59 please don't be discouraged, this is good work on the whole and I'm happy to see the RO-Crate outputs being improved :)

As I see it, only a small amount of lines need to be removed from this PR to take out the toolbox-dependent bits (namely the _get_tool_metadata function and the few lines that call it or its output), so that would be the easiest thing to do.

If you decide to make a new PR instead, so that the toolbox code is still visible here if someone wants to find it again, that's fine as well. In that case, please tag me in the new one so I can keep track of it!

Marie59 · 2024-11-23T18:45:20Z

@Marie59 please don't be discouraged, this is good work on the whole and I'm happy to see the RO-Crate outputs being improved :)

As I see it, only a small amount of lines need to be removed from this PR to take out the toolbox-dependent bits (namely the _get_tool_metadata function and the few lines that call it or its output), so that would be the easiest thing to do.

If you decide to make a new PR instead, so that the toolbox code is still visible here if someone wants to find it again, that's fine as well. In that case, please tag me in the new one so I can keep track of it!

Not discouraged no worries ! Thanks for the encouragements ! I'll remove what need to be removed here (and keep a back up).

Marie59 · 2024-11-25T14:22:47Z

Hello all !
So, normally I removed what needed to be removed. Locally I succeed to generate a Ro-crate with my galaxy test instance. I tried to add some tests (they are working too).
Let me know if something is not good and if I can improve anything.

mvdbeek · 2024-11-25T14:59:51Z

Can you resolve the conflict in test/unit/data/model/test_model_store.py ?

Marie59 · 2024-11-27T08:26:57Z

I don't understand the last errors I get ...

lib/galaxy/model/store/ro_crate_utils.py

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

Marie59 · 2024-11-27T15:32:17Z

@nsoranzo thank you for the help ! I was a bit lost !!

mvdbeek

That looks great, thanks @Marie59

test/unit/data/model/test_model_store.py

elichad

I took a look at the output JSON-LD and I've made some suggestions based on the Provenance Run Crate profile - Galaxy RO-Crates don't strictly conform to this, but it offers some best practices that are applicable here.

From a Workflow Run Crate profile perspective (the less-strict one we actually intend to conform to), the metadata added in this PR looks good.

elichad · 2024-12-09T16:53:19Z

lib/galaxy/model/store/ro_crate_utils.py

+                                crate,
+                                creator_data.get(
+                                    "url", ""
+                                ),  # Use URL as identifier if available, otherwise empty string


Could also try to use identifier here, as that is an option in the UI too when setting an organization as a creator. For example an ROR identifier could be used. (though admittedly it's more likely for an average user to just input their institute's URL in the url field)

For this point I was not sure what to do I used identifier for the ORCID of the creator. Should I replace it by ROR or is something completely different you were suggesting ?

elichad · 2024-12-09T17:01:26Z

lib/galaxy/model/store/ro_crate_utils.py

+                                "name": tool_name,
+                                "version": tool_version,
+                                "description": tool_description,
+                                "url": "https://toolshed.g2.bx.psu.edu",  # URL if relevant


Having a url for each tool would be great - but if we can't get this due to the toolbox limitations, it should be removed.

lib/galaxy/model/store/ro_crate_utils.py

elichad · 2024-12-09T18:16:33Z

lib/galaxy/model/store/ro_crate_utils.py

+
+        # Add tools used in the workflow
+        self._add_tools(crate)
+        self._add_steps(crate)


That would be nice to have, yes.

This can even be done per tool (though more fiddly). There is already some capturing of the inputs and outputs as as formalParameters, but it seems only to happen if data files are included in the crate. There seem to be some helper functions further down the file for this purpose:

galaxy/lib/galaxy/model/store/ro_crate_utils.py

Line 359 in 39e38c9

def _add_step_parameter_fp(self, step: WorkflowInvocationStep, crate: ROCrate):

Marie59 · 2024-12-23T09:00:56Z

I took a look at the output JSON-LD and I've made some suggestions based on the Provenance Run Crate profile - Galaxy RO-Crates don't strictly conform to this, but it offers some best practices that are applicable here.

From a Workflow Run Crate profile perspective (the less-strict one we actually intend to conform to), the metadata added in this PR looks good.

Thanks a lot for the review ! I tried to add as much as I could your suggestions !

I don't understand the last comment though on formalParameters what should I do ?

Is everything else good ?

github-actions bot added the area/database Galaxy's database or data access layer label Sep 16, 2024

github-actions bot added this to the 24.2 milestone Sep 16, 2024

Marie59 changed the title ~~Ro crate2~~ Add author and tools details in RO-Crate Sep 17, 2024

mvdbeek reviewed Sep 17, 2024

View reviewed changes

Marie59 closed this Oct 22, 2024

Marie59 force-pushed the ro-crate2 branch from 0762d99 to aef57db Compare October 22, 2024 13:09

try to fix branch mess

e824db7

Marie59 reopened this Oct 22, 2024

mjosse added 3 commits October 23, 2024 11:30

add cache

a05e522

add some tests

effa82e

fix lint rm useless import

468de8a

bgruening reviewed Nov 1, 2024

View reviewed changes

mvdbeek self-requested a review November 12, 2024 15:34

mvdbeek modified the milestones: 24.2, 25.0 Nov 15, 2024

mvdbeek reviewed Nov 21, 2024

View reviewed changes

mjosse added 3 commits November 24, 2024 17:18

rollback to changes without tool metadata

803a558

remove useless tests

f26662a

add some tests

a582e52

mjosse added 3 commits November 25, 2024 16:24

try to fix conflicts

5a26dc2

rm label

875e8ea

rm useless import

58219b1

This comment was marked as resolved.

Sign in to view

fix mypy errors

f930358

nsoranzo reviewed Nov 27, 2024

View reviewed changes

lib/galaxy/model/store/ro_crate_utils.py Outdated Show resolved Hide resolved

lib/galaxy/model/store/ro_crate_utils.py Outdated Show resolved Hide resolved

Marie59 and others added 2 commits November 27, 2024 16:31

Update lib/galaxy/model/store/ro_crate_utils.py

124db0b

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

Update lib/galaxy/model/store/ro_crate_utils.py

0cbc508

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

Add missing import

f585677

mvdbeek approved these changes Nov 28, 2024

View reviewed changes

test/unit/data/model/test_model_store.py Outdated Show resolved Hide resolved

Drop stray print statement

e66652e

martenson added area/backend kind/enhancement labels Dec 5, 2024

elichad reviewed Dec 9, 2024

View reviewed changes

imrpove ro-crate with eli's review

f1b404d

Add author and tools details in RO-Crate #18820

Are you sure you want to change the base?

Add author and tools details in RO-Crate #18820

Conversation

Marie59 commented Sep 16, 2024 • edited Loading

mvdbeek left a comment

Choose a reason for hiding this comment

Marie59 commented Sep 18, 2024

stain commented Sep 19, 2024

Marie59 commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elichad commented Nov 12, 2024

OliverWoolland commented Nov 15, 2024

martenson commented Nov 15, 2024 • edited Loading

OliverWoolland commented Nov 15, 2024 • edited Loading

davelopez commented Nov 15, 2024

OliverWoolland commented Nov 15, 2024

elichad commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davelopez commented Nov 21, 2024 • edited Loading

Marie59 commented Nov 22, 2024 • edited Loading

mvdbeek commented Nov 22, 2024

elichad commented Nov 22, 2024

Marie59 commented Nov 23, 2024

Marie59 commented Nov 25, 2024 • edited Loading

mvdbeek commented Nov 25, 2024

Marie59 commented Nov 27, 2024

This comment was marked as resolved.

Marie59 commented Nov 27, 2024

mvdbeek left a comment

Choose a reason for hiding this comment

elichad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marie59 Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marie59 commented Dec 23, 2024

Marie59 commented Sep 16, 2024 •

edited

Loading

martenson commented Nov 15, 2024 •

edited

Loading

OliverWoolland commented Nov 15, 2024 •

edited

Loading

davelopez commented Nov 21, 2024 •

edited

Loading

Marie59 commented Nov 22, 2024 •

edited

Loading

Marie59 commented Nov 25, 2024 •

edited

Loading

Marie59 Dec 23, 2024 •

edited

Loading