Added omnistat support for individual job steps and HPO trials.

[ashwinma]

This is a draft PR just to understand what code changes are needed to be made to enable omnistat on Frontier

[allaffa]

@ashwinma
This PR cannot be merged until the CI Test pass.
@jychoi-hpc
Is it better in terms of practice if we fix the formatting ourselves or if the author (Ashwin) fixes it himself?

[ashwinma]

@allaffa I was not sure if you wanted to have this merged to main or just use this PR as a reference for you. That is why I marked this PR as "WIP"

Having said that, if there is interest in merging with main, we could probably check the system (if (system == frontier)) and then call omnistat.

I can do the code-formatting on my end with black if that is what you were hinting at.

Thoughts?

[allaffa]

@allaffa I was not sure if you wanted to have this merged to main or just use this PR as a reference for you. That is why I marked this PR as "WIP"

Having said that, if there is interest in merging with main, we could probably check the system (if (system == frontier)) and then call omnistat.

I can do the code-formatting on my end with black if that is what you were hinting at.

Thoughts?

@ashwinma I like the idea of adding a (if (system == frontier))
Instead of specifying Frontier, may it be possible to make the check more general to have it work on any system with AMD GPUs?

@jychoi-hpc what do you think?