use new bigdata-operator as notebook session backend

[ImpSy]

Why

Remove the need for NFS in the notebook-service

What

• Add a new API in bigdata-operator for remoteKernelSession
• Make the notebook-service use webhookKernelSessionManager w/ the bigdata-operator

Test plan and results

API server started

2024-04-19T14:33:06Z    INFO    setup    bigdata-operator    {"buildVersion": "0.4.11-429c6a8f", "buildDate": "2024-04-19T13:34:26Z"}
2024-04-19T14:33:06Z    INFO    controller-runtime.metrics    Metrics server is starting to listen    {"addr": ":8080"}
2024-04-19T14:33:06Z    INFO    spotConfigProvider    Found Ocean controller namespace "kube-system"
2024-04-19T14:33:06Z    INFO    setup    oceanControllerNamespace: "kube-system"
2024-04-19T14:33:06Z    INFO    spotConfigProvider    SPOTINST_CLUSTER_IDENTIFIER found in config map kube-system/spotinst-kubernetes-cluster-controller-config
2024-04-19T14:33:06Z    INFO    setup    controllerClusterID: "privatelink-india"
2024-04-19T14:33:06Z    INFO    spotConfigProvider    SPOTINST_BASE_URL found in env var
2024-04-19T14:33:06Z    INFO    spotConfigProvider    SPOTINST_TOKEN found in secret kube-system/spotinst-kubernetes-cluster-controller
2024-04-19T14:33:06Z    INFO    spotConfigProvider    SPOTINST_ACCOUNT found in secret kube-system/spotinst-kubernetes-cluster-controller
2024-04-19T14:33:06Z    INFO    setup    starting API server
2024-04-19T14:33:06Z    INFO    setup    starting manager
2024-04-19T14:33:06Z    INFO    Starting server    {"path": "/metrics", "kind": "metrics", "addr": ":8080"}
2024-04-19T14:33:06Z    INFO    Starting server    {"kind": "health probe", "addr": ":8082"}
I0419 14:33:06.997726       1 leaderelection.go:248] attempting to acquire leader lease spot-system/6aaa1a16.spot.io...
I0419 14:33:22.775233       1 leaderelection.go:258] successfully acquired lease spot-system/6aaa1a16.spot.io

Notebook-service using bigdata-operator as backend

Starting Jupyter Enterprise Gateway...
[D 2024-04-19 14:45:58.877 EnterpriseGatewayApp] Searching ['/venv/etc/jupyter', '/app/services/bigdata-notebook-service/.jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2024-04-19 14:45:58.878 EnterpriseGatewayApp] Looking for jupyter_config in /etc/jupyter
[D 2024-04-19 14:45:58.878 EnterpriseGatewayApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-04-19 14:45:58.878 EnterpriseGatewayApp] Looking for jupyter_config in /app/services/bigdata-notebook-service/.jupyter
[D 2024-04-19 14:45:58.879 EnterpriseGatewayApp] Looking for jupyter_config in /venv/etc/jupyter
[D 2024-04-19 14:45:58.880 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /etc/jupyter
[D 2024-04-19 14:45:58.880 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /usr/local/etc/jupyter
[D 2024-04-19 14:45:58.880 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /app/services/bigdata-notebook-service/.jupyter
[D 2024-04-19 14:45:58.881 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /venv/etc/jupyter
[I 2024-04-19 14:45:58.883 EnterpriseGatewayApp] Initializing BigdataKernelSpecManager
[I 2024-04-19 14:45:58.885 EnterpriseGatewayApp] Webhook kernel session persistence activated
[I 2024-04-19 14:45:58.894 EnterpriseGatewayApp] Jupyter Enterprise Gateway 3.2.2 is available at http://0.0.0.0:8888
[I 240419 14:46:23 web:2348] 200 GET /api (127.0.0.1) 0.85ms

{"level":"info","service":"kernel-session-manager","httpRequest":{"proto":"HTTP/1.1","remoteIP":"192.168.197.60:56762","requestID":"bigdata-operator-6f684d75bf-56g8c/BtBoDD0yso-000007","requestMethod":"GET","requestPath":"/kernelSession","requestURL":"http://bigdata-operator-api.spot-system.svc.cluster.local/kernelSession"},"httpRequest":{"header":{"accept":"*/*","accept-encoding":"gzip, deflate","connection":"keep-alive","user-agent":"python-requests/2.31.0"},"proto":"HTTP/1.1","remoteIP":"192.168.197.60:56762","requestID":"bigdata-operator-6f684d75bf-56g8c/BtBoDD0yso-000007","requestMethod":"GET","requestPath":"/kernelSession","requestURL":"http://bigdata-operator-api.spot-system.svc.cluster.local/kernelSession","scheme":"http"},"timestamp":"2024-04-19T14:46:53.8236137Z","message":"Request: GET /kernelSession"}
{"level":"info","service":"kernel-session-manager","sessions_found":0,"timestamp":"2024-04-19T14:46:53.827606353Z","message":"list kernel sessions"}
{"level":"info","service":"kernel-session-manager","httpRequest":{"proto":"HTTP/1.1","remoteIP":"192.168.197.60:56762","requestID":"bigdata-operator-6f684d75bf-56g8c/BtBoDD0yso-000007","requestMethod":"GET","requestPath":"/kernelSession","requestURL":"http://bigdata-operator-api.spot-system.svc.cluster.local/kernelSession"},"httpResponse":{"bytes":2,"elapsed":3.990853,"header":{"content-type":"application/json"},"status":200},"timestamp":"2024-04-19T14:46:53.827639366Z","message":"Response: 200 OK"}

[thorsteinnth]

Suggested change

	name: {{ include "bigdata-operator.fullname" . }}-api
	name: bigdata-operator-api

[thorsteinnth]

since this is hardcoded in the bns chart, hardcode it here as well?

[ImpSy]

didn't wanted to hardcode it in case we ever do blue green