Limiting the number of SSH Connections on a node

[FrankD412]

In my work with @dongahn @SteVwonder @Jmast we have been running nested workflows on some of the clusters at LLNL and on some the SSH connection limit is set really low. In some discussions with @dongahn, he had mentioned that a potential workaround is to set up an SSH config that will route connections through an SSH controller as follows:

cat ~/.ssh/config
Host *
  ControlMaster auto
  ControlPath /tmp/%r@%h:%p
  ControlPersist 12

We've not gotten a chance to test this out just yet, but just dropping this here as food for thought in case someone runs into the same issue and that this can be discussed and tracked here.

[garlick]

It sounds like it might not work in this particular case (?) but flux-proxy(1) is another way to share a single remote ssh:// flux connection among many local ones.

[dongahn]

@garlick: do we have an example using flux-proxy for a case like this? This can help assess this solution further, I think.

[garlick]

flux proxy ssh://... "printenv FLUX_URI && sleep inf" could get you a local URI proxy for the ssh://.. one. Multiple copies of that one-liner could be run to do this for multiple "remote jobs". Feed the printed URIs into the workflow rather than the remote one.

The flux-proxy processes would need to be managed somehow. If there is a local flux instance, maybe the flux-proxy processes could themselves be run as jobs on one core?

Is there a way for us to augment or respin flux-proxy so that it would be more generally useful? For example, what if it took a list of URIs on the command line and set FLUX_URI_0 ... FLUX_URI_<n-1> in its child script's environment?

[dongahn]

@FrankD412:

The other suggestion was for the workflow management worker to use a local URI instead of a remote URI (i.e., ssh one in your case). This can be a bit complicated as the workflow management worker can run under the top-level Flux instance or a nested instance in your case and depending where you want to further submit tasks you will have to use a corresponding URI.

But it should have access to the right local URI for either case.