Way to run clustermq on separate HPC node?

[eveyp]

I think this is similar to #216 but not quite the same.

My institution's HPC requires ssh-ing to a login node and then ssh-ing to a development node to run R. R isn't installed on the login node (but the login node can submit jobs) and I can't ssh directly to the development node.

Is there still a way to get clustermq working with these constraints?

Thanks!!

[mschubert]

Assuming your development node can submit jobs, you can set up SSH to connect to that via your login node. This would look something like this in your (local) ~/.ssh/config:

Host login_node
    Hostname login_node_address_that_resolves_from_your_workstation
    User your_user_name
    IdentityFile ~/.ssh/your_rsa

Host development_node
    Hostname development_node_name_that_resolves_from_login_node
    User your_user_name
    #IdentityFile ~/.ssh/your_rsa # key to let you connect from login to dev, if required
    ProxyCommand ssh login_node -W %h:%p

Then you can use development_node as clustermq.ssh.host.

[eveyp]

Thanks for responding! Yep, the dev node can submit jobs. With that config setup I can now connect to the dev node, but now I'm having trouble with the remote port forwarding.

I keep timing out on the SSH connection when trying to run Q from my local machine:

> Q(fx, x=1:3, n_jobs=1)
Connecting dev-node via SSH ...
Error in initialize(...) : 
  Remote R process did not respond after 10000 seconds. Check your SSH server log.

The log file gets created on the sever, but it's empty.

Going through the ssh troubleshooting steps, I think the problem is with the remote port forwarding:

ssh -R 54709:localhost:6687 dev-node R --vanilla
Warning: remote port forwarding failed for listen port 54709

Here's the verbose debugging if that helps:

debug1: Authentication succeeded (publickey).
Authenticated to dev-node (via proxy).
debug1: Remote connections from LOCALHOST:54709 forwarded to local address localhost:6687
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: remote forward failure for: listen 54709, connect localhost:6687
Warning: remote port forwarding failed for listen port 54709
debug1: All remote forwarding requests processed
debug1: Sending command: R --vanilla

Thanks again for your help!

[mschubert]

This unfortunately sounds like your HPC configuration prohibits port forwarding, which is required for clustermq. Maybe you can contact your admins to see if there is a workaround? (e.g. specific ports that are allowed)

[eveyp]

Gotcha, thanks for your help!

[eveyp]

So it seems now that the forwarding is actually working, but I'm still hanging with:

> Q(fx, x=1:3, n_jobs=1)
Connecting dev-node via SSH ...
Error in initialize(...) : 
  Remote R process did not respond after 10000 seconds. Check your SSH server log.

ssh_proxy.log gets created but is an empty file.

When I try running system("ssh -o \"ExitOnForwardFailure yes\" -f -R 54709:localhost:6687 -R 54710:localhost:6688 dev-node R --no-save --no-restore -e 'clustermq:::ssh_proxy(ctl=54709, job=54710)' > ~/ssh_proxy.log 2>&1"), I get the following in ssh_proxy.log:

> clustermq:::ssh_proxy(ctl=54709, job=54710)
master ctl listening at: tcp://127.0.0.1:54709
forwarding local network from: tcp://dev-node-url:6379
sent PROXY_UP to master ctl

[mschubert]

Two things you can try are (1) using the latest CRAN version (if you're not already), and (2) set the clustermq.ssh.timeout option to something like 60 seconds (in your local session or .Rprofile).