Create a TPV tutorial on the GTN

[nuwang]

Ping @hexylena
Do you have any suggestions on what it should include, that is missing, or could enhance: https://total-perspective-vortex.readthedocs.io/en/latest/topics/tpv_by_example.html?

[hexylena]

Just first off, I love the by-example, that's a great bit of reference documentation for someone upgrading an existing configuration, really great as reference.

For the GTN side I think it would be more than sufficient to just upgrade the GAT tutorials to use TPV, rather than showing the DTD or custom python rules, just go 100% all in on TPV.

• connect to compute
• job destinations

And since job destinations would essentially be an "overhaul" and replacing the existing tutorial, I think the important things to cover for administrators would be:

• setting up + using the TPV shared database
• resubmission with increased memory/cpu (but maybe this is less important with TPVSDB)
• passing results of resource selectors into a rule
• very basic access control (e.g. admin only tools)

[nuwang]

Thanks for the suggestions, I think that's a great approach. Can you clarify passing results of resource selectors into a rule?

[hexylena]

https://training.galaxyproject.org/training-material/topics/admin/tutorials/job-destinations/tutorial.html#job-resource-selectors this feature. it's most visible on Main to select which cluster a job goes to

[quacksawbones]

As someone who had trying to get TPV doing what I want in our Galaxy instance, there are a number of things I would really love to see in the TPV documentation/tutorial:

• A list of "valid" (or invalid) YML entry types (or at least how to work it out e.g. - elif: and -else: (as mentioned in Gitter)
• Details about which TPV YML entries can execute which types of Python code (whether that's evaluating a single line or a multi-line chunk, as well as including importing modules)
• How to send the value of entries within a TPV YML file out to the Galaxy logs (for debugging/troubleshooting purposes)
• How to get TPV accessing custom functions within $GALAXYROOT/lib/galaxy/jobs/rules (if that's possible)

While I acknowledge that @hexylena was who the question was directed to (and that you cannot cover EVERYTHING in a tutorial), the above are among the things I have really struggled with while trying to implement TPV in our Galaxy instance.

[nuwang]

Thanks @quacksawbones. The hardest part about writing these docs is to figure out where to make info available so all you need is a quick scan, as it's all too easy to bury the information somewhere. These suggestions are super helpful. Regarding the individual suggestions:

• A list of "valid" (or invalid) YML entry types (or at least how to work it out e.g. - elif: and -else: (as mentioned in Gitter)

Good idea, should have been done sooner!

• Details about which TPV YML entries can execute which types of Python code (whether that's evaluating a single line or a multi-line chunk, as well as including importing modules)

I think we could combine this with the above idea, and for each entry, include a column on how it will be evaluated (f-string or eval as code block). The rule of thumb is, all strings entries are evaluated as f-strings (e.g. envs, params) and everything else as code blocks. This is in the docs, but buried somewhere.

• How to send the value of entries within a TPV YML file out to the Galaxy logs (for debugging/troubleshooting purposes)
• How to get TPV accessing custom functions within $GALAXYROOT/lib/galaxy/jobs/rules (if that's possible)

These two we can add to an FAQ?

[quacksawbones]

I completely understand @nuwang. As @hexylena mentioned, having the "By Example" sections was really useful to me while getting this working. A couple of additional clarifications:

• Multiline Python code vs F-strings
  • That's really good to know, but a table I think would help clarify it
  • Some examples of code/evaluation that won't work (with an explanation) would also help. A specific example I had recently:

rules:
- if: |
      import math
      file_size = math.ceiling(jobs.<object methods>.get_file_size() *1.1)
      if file_size > <x>:
        return True

  <code when True>

In the above, the parser tries to execute the if file_size > <x>: as a YML entry, not as part of the code block. Having a few examples of known limitations would help steer people right.

• Output to Galaxy logs/executing custom functions in $GALAXYROOT/lib/galaxy/jobs/rules:
  • Adding these into the "By Example" section would be really helpful! Maybe custom functions as a subsection of "Rules" and outputting to the Galaxy log to either "Rules" (to confirm your rules are executing as intended) or "Simple Configuration" (as logging is kind of fundamental)?

Thanks again for considering my suggestions! :)

[nuwang]

Some examples of code/evaluation that won't work

Makes sense. We could add a gotchas/things that won't work page.

Regarding the specific example, the last line of every TPV code block must evaluate to a value. This is true whether it's an if condition (should eval to a boolean value) or cores/mem (should eval to float values), so that there is some final value that can be assigned to the field. This too is in the docs, but it's not obvious unless you're looking for it. A "things that won't work" section should help clarify those.

And yes, we can add the latter two to TPV by example too.

[hexylena]

We should also consider adding https://github.com/usegalaxy-eu/ansible-tpv-lint to the training playbook to help with this validation (or just running tpv lint locally)

[hexylena]

Following up on this, what's the likelihood something could be ready by April 7th or so? @nuwang

[nuwang]

I think it's doable. Will work on it.

[nuwang]

Between the tutorial here: https://training.galaxyproject.org/training-material/topics/admin/tutorials/job-destinations/tutorial.html#job-resource-selectors and the PR here: #126, we should be able to close this issue.