Investigate possibility of spawning of container with another user than the default 'jovyan'

[rkevin-arch]

From our meeting, if we want people from the HPC side to use JupyterHub and still have access to their files, we need to mount NFS volumes into the Jupyter pods. Each person from the HPC side would have a unique UID, but the pods are usually spawned with UID 1000 (jovyan), with no way to change it easily.

Some ideas:

1. Mount the NFS volume as normal, run the container as the HPC UID, and before JupyterLab starts we manually do usermod and groupmod to change jovyan's UID/GID (or maybe grab their actual username and use that).
   The problem is there are plenty of places (e.g. /srv/conda) that uses UID 1000, so we have to do a blanket find and chown every time the pod starts, which takes tons of time (test run on my computer took ~25s).
   Also, if the user switches to a different profile that does not go through this UID mapping process, their existing Jupyter home directory will not have UID 1000 and they might not be able to edit their files. This may not be a problem if we enforce "all HPC users must always use their HPC UID", but I'm not 100% sure.

2. Mount every user's HPC data directory on every host machine, and use something like https://bindfs.org/ to bind mount a copy inside the container, and mask the UID/GID somehow. The user inside the container stays UID 1000, the mount point appears as UID 1000, while the underlying NFS filesystem accesses use their HPC UID. Not sure if this is possible.

Any other suggestions are welcome.

Currently, I can add a regular NFS volume without problems in kubespawner using something like this in the c.KubeSpawner.profile_list function:

          if self.is_hpc:
              profiles.append({
                  'display_name': 'Test mount extra NFS volume',
                  'description': "Mounts 10.0.0.200:/testvol to /testvol",
                  'kubespawner_override': {
                      'volumes': self.volumes + [
                          {
                              'name': 'testvol',
                              'nfs': {
                                  'server': '10.0.0.200',
                                  'path': '/testvol',
                              },
                          },
                      ],
                      'volume_mounts': self.volume_mounts + [
                          {
                              'name': 'testvol',
                              'mountPath': '/testvol',
                          },
                      ],
                  },
              })

So mounting NFS volumes is fine, the UID/GID part is the only problem.

[rkevin-arch]

Going with option 3: Kerberos.

[feltstykket]

a good starting place will be to add krb5-user to the Dockerfile

[rfeltstykket]

Here is how to get the uid from someone given their loginid:

ldapsearch -LLL -x -h ldap.ucdavis.edu -b ou=People,dc=ucdavis,dc=edu "uid=rf" ucdPersonUUID

Here is how to get it using their email.
ldapsearch -LLL -x -h ldap.ucdavis.edu -b ou=People,dc=ucdavis,dc=edu "mail=rafeltstykket@ucdavis.edu" ucdPersonUUID

If you want to see all attributes you can see with this ldap query, you can just omit ucdPersonUUID

[TimStewartJ]

I'm not super sure of how to approach this at the moment but I found these potentially helpful resources:

https://discourse.jupyter.org/t/starting-single-user-notebook-with-our-custom-ldap-docker-image/881

jupyterhub/zero-to-jupyterhub-k8s#1292

jupyterhub/zero-to-jupyterhub-k8s#1411

[rfeltstykket]

jupyterhub/zero-to-jupyterhub-k8s/issues/1292 looks like the best approach for this. Can we test this on staging with the LDAP queries above?

[hokiegeek2]

@TimStewartJ @rfeltstykket My use case is that I need to set NB_USER and NB_UID because I am using Jupyterhub-on-Kubernetes as a Slurm client. Specifically, the uid in JH must correspond to a Slurm user uid. I can set NB_USER with the following code in my zero-to-jupyterhub -k8s values.yaml:

extraConfig:
      preSpawnHook: |
          from pip._internal import main as pipmain

          # pip install and configure the Authenticator
          pipmain(['install', 'jupyterhub-nativeauthenticator'])

          from nativeauthenticator import NativeAuthenticator

          class GBRMAuthenticator(NativeAuthenticator):
            from tornado import gen
            @gen.coroutine
            def pre_spawn_start(self, user, spawner):
                auth_state = yield user.get_auth_state()
                spawner.environment = {'NB_USER':spawner.user.name, 'NB_UID':'1000'}

          c.JupyterHub.authenticator_class = GBRMAuthenticator
          c.KubeSpawner.args = ['--allow-root']
          c.Spawner.cmd = ['start.sh','jupyterhub-singleuser','--allow-root']

But when I try to change NB_UID to something other than 1000, I get the following error:

PermissionError: [Errno 13] Permission denied: '/home/jovyan/.local/share/jupyter/runtime/nbserver-48-open.html'

[TimStewartJ]

Thanks for sharing your config! Our ultimate goal is to use Slurm with Jupyterhub as well. I'll take a look into this and see if I can make any progress on it.

Could you share further how you've set up Jupyterhub on Kubernetes and Slurm so far?

[feltstykket]

It looks like the error here is that /home/jovyan/.local has restrictive permissions that won't allow anything other than UID 1000 to access it, which is why it fails to start with any other UID: drwx------ 3 jovyan root 3 Sep 24 2019 .local If we can find a way to make /home/<username> owned by the UID in question that might get us farther along. /home/jovyan is a persistent volume claim on the k8s cluster.

…

On Mon, Nov 29, 2021 at 5:39 PM Tim ***@***.***> wrote: Thanks for sharing your config! Our ultimate goal is to use Slurm with Jupyterhub as well. I'll take a look into this and see if I can make any progress on it. Could you share further how you've set up Jupyterhub on Kubernetes and Slurm so far? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#207 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEOZ4TDDX56YT222C65YSTUOQTNJANCNFSM4ULZNSJQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[hokiegeek2]

@feltstykket yup, and what's strange (I think) is that my config enables chown which I believe changes the owner of the jovyan home dir files/dirs when they are copied over to the logging in user.

[hokiegeek2]

Thanks for sharing your config! Our ultimate goal is to use Slurm with Jupyterhub as well. I'll take a look into this and see if I can make any progress on it.

Could you share further how you've set up Jupyterhub on Kubernetes and Slurm so far?

@TimStewartJ sure thing, here's my slurm-cloud-integration github project.