Examples for LM tweaking

[andre-merzky]

[mtitov]

• Srun (link) for WF3 at SuperMUC-NG
  • RP default: srun --exclusive --cpu-bind=none --nodes %d --ntasks %d ...
  • Asked for a run (without extra options): srun --nodes %d --ntasks %d ...
• HPC-workflows #138
  • srun -n <n> --gpus=<g> --cpus-per-task=<tc> --gpus-per-task=<tg> --distribution=arbitrary <executable>
• RP #2451
  • not to use option --nodefile due to old version of SLURM (/usr/bin/srun: unrecognized option)
  • solution -> env variable SLURM_HOSTFILE
• R. Srinivasan's study on Theta
  • ability to tweak aprun command with adding option -r 1 (p.13 https://www.alcf.anl.gov/files/UsingPEonKNL.pdf)

[andre-merzky]

• In the HPC workflows use case: how would one make sure that the user provided flags don't conflict with the flags the LM defines?
• In the nodefile case, that is not about adding a flag but about removing one. How would that work? Also, how would that enable the user to add a valid SLURM_HOSTFILE env variable (and, more importantly, create a valid hostfile at that location)?

[mtitov]

These are use cases to consider about LM command options that could be added or updated by user (second use case was about option, which could be reset or removed by user. I've added the "solution" there, since we found what would work for all SLURM versions)

In general, my thoughts on it are: 1) we should define "core" options with which we guarantee a stable work (could be implemented with variables which contain a specific options per LM); 2) provide possibility to add or reset options by user

[mtitov]

BTW with new approach of LM configurations (within env-isolation), certain options (which could be considered "extra") could be set within a corresponding LM config (that will be as one possibility to remove "extra" options if needed)

[mtitov]

This ticket is more as a discussion and srun command for supermuc was just as an example - to be able to create a launch command with default options, but also consider possibility to overwrite that default options if needed

[mtitov]

Summary

LM tweaks are allowed, but should be introduced per use case and managed by RP (since it might affect resource management and scheduling), and will be defined within LM config section in resource configuration.

For example, for aprun and option -r 1, the configuration will be the following (* parameter's name is just for example, and could be changed when implemented):

"launch_methods" : { "APRUN" : { "os_cores": 1 } },

and RP will set one core down for each node (for RP ResourceManager component) and will add a corresponding option to the launch command

p.s. this is a preliminary proposal; case with more than one LM and when configuration of one LM may interfere with other LM is not considered for now