+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ A common question is why not just NGINX instead of the custom proxy? The reason is the dynamic routing for the applications, e.g. URLs like https://jupyterlab-abcde1234.mydomain.com/some/path: each one has a lot of fairly complex requirements.
+While not impossible to leverage NGINX to move some code from the proxy, there would still need to be custom code, and NGINX would have to communicate via some mechanism to this custom code to achieve all of the above: extra HTTP or Redis requests, or maybe through a custom NGINX module. It is suspected that this will make things more complex rather than less, and increase the burden on the developer.
+We will use a custom proxy for Data Workspace, rather than simply using NGINX.
+This will decrease the burden on the developer that would have been required by custom NGINX modules, extra HTTP or Redis requests, which all would still have required custom code.
+Using the custom proxy allows for all of the complex requirements and dynamic routing of our applications over which we have absolute control.
+Initial difficulty when onboarding new team members as they will need to understand these decisions and requirements.
+There is an extra network hop compared to not having a proxy.
+The proxy fits the typical use-case of event-loop based programming: low CPU but high IO requirements, with potentially high number of connections.
+The asyncio library aiohttp provides enough low-level control over the headers and the bytes of requests and responses to work as a controllable proxy. For example, the typical HTTP request cycle can be programmed fairly explicitly.
+An incoming request begins: its headers are received.
+The library also allows for receiving and making WebSockets requests. This is done without knowledge ahead of time which path is WebSockets, and which is HTTP. This is something that doesn't seem possible with, for example, Django Channels.
+Requests and responses can be of the order of several GBs, so this streaming behaviour is a critical requirement.
+We will use the asyncio library aiohttp.
+Allows for critical requirement of streaming behaviour.
+We can stream HTTP(S) and Websockets requests in an efficient way with one cohesive Python package.
+A core bit of infrastructure will depend on a flavour of Python unknown to even experienced Python developers.
+Aiohttp is unable to proxy things that are not HTTP or Websockets, i.e. SSH. This is why GitLab isn't behind the proxy.
+This section contains a list of Architecture Decision Records (ADRs).
+As an example, from the point of view of user abcde1234, https://jupyterlab-abcde1234.mydomain.com/ is the fixed address of their private JupyterLab application. Going to https://jupyterlab-abcde1234.mydomain.com/ in a browser will:
If the application is stopped, then a visit to https://jupyterlab-abcde1234.mydomain.com/ will repeat the process. The user will never leave https://jupyterlab-abcde1234.mydomain.com/. If the user visits https://jupyterlab-abcde1234.mydomain.com/some/path, they will also remain at https://jupyterlab-abcde1234.mydomain.com/some/path to ensure, for example, bookmarks to any in-application page work even if they need to start the application to view them.
The browser will only make GET requests during the start of an application. While potentially a small abuse of HTTP, it allows the straightfoward behaviour described: no HTML form or JavaScript is required to start an application (although JavaScript is used to show a countdown to the user and to check if an application has loaded), and the GET requests are idempotent.
+The proxy however, has a more complex behaviour. On an incoming request from the browser for https://jupyterlab-abcde1234.mydomain.com/:
GET details of an application with the host jupyterlab-abcde1234 from an internal API of the main application;GET returns a 404, it will make a PUT request to the main application that initiates creation of the Fargate task;GET returns a 200, and the details contain a URL, the proxy will attempt to proxy the incoming request to it;SPAWNING application as a true error: they are effectively swallowed.GET as STOPPED, which happens on error, it will DELETE the application, and show an error to the user.The proxy itself only responds to incoming requests from the browser, and has no long-lived tasks that go beyond one HTTP request or WebSockets connection. This ensures it can be horizontally scaled.
+ + + + + + +In addition to being able to run any Docker container, not just JupyterLab, Data Workspace has some deliberate architectural features that are different to JupyterHub.
+All state is in the database, accessed by the main Django application.
+Specifically, no state is kept in the memory of the main Django application. This means it can be horizontally scaled without issue.
+The proxy is also stateless: it fetches how to route requests from the main application, which itself fetches the data from the database. This means it can also be horizontally scaled without issue, and potentially independently from the main application. This means sticky sessions are not needed, and multiple users could access the same application, which is a planned feature for user-supplied visualisation applications.
+Authentication is completely handled by the proxy. Apart from specific exceptions like the healthcheck, non-authenticated requests do not reach the main application.
+The launched containers do not make requests to the main application, and the main application does not make requests to the launched containers. This means there are fewer cyclic dependencies in terms of data flow, and that applications don't need to be customised for this environment. They just need to open a port for HTTP requests, which makes them extremely standard web-based Docker applications.
+There is a notable exception to the statelessness of the main application: the launch of an application is made of a sequence of calls to AWS, and is done in a Celery task. If this sequence is interrupted, the launch of the application will fail. This is a solvable problem: the state could be saving into the database and the sequence resumed later. However, since this sequence of calls lasts only a few seconds, and the user will be told of the error and can refresh to try to launch the application again, at this stage of the project this has been deemed unnecessary.
+ + + + + + +Data Workspace is made of a number of components. This page explains what those are and how they work together.
+To understand the components of Data Workspace's architecture, you should have familiary with:
+At the highest level, users access the Data Workspace application, which accesses a PostgreSQL database.
+graph
+ A[User] --> B[Data Workspace]
+ B --> C["PostgreSQL (Aurora)"]The architecture is heavily Docker/ECS Fargate based.
+graph
+ A[User] -->|Staff SSO| B[Amazon Quicksight];
+ B --> C["PostgreSQL (Aurora)"];
+ A --> |Staff SSO|F["'The Proxy' (aiohttp)"];
+ F --> |rstudio-9c57e86a|G[Per-user and shared tools];
+ F --> H[Shiny, Flask, Django, NGINX];
+ F --> I[Django, Data Explorer];
+ G --> C;
+ H --> C;
+ I --> C;
+
+
+Main application: + A Django application to manage datasets and permissions, launch containers, a proxy to route requests to those containers, and an NGINX instance to route to the proxy and serve static files.
+JupyterLab: + Launched by users of the main application, and populated with credentials in the environment to access certain datasets.
+rStudio: + Launched by users of the main application, and populated with credentials in the environment to access certain datasets.
+pgAdmin: + Launched by users of the main application, and populated with credentials in the environment to access certain datasets.
+File browser: + A single-page-application that offers upload and download of files to/from each user's folder in S3. The data is transferred directly between the user's browser and S3.
+metrics: + A sidecar-container for the user-launched containers that exposes metrics from the ECS task metadata endpoint in Prometheus format.
+s3sync: + A sidecar-container for the user-launched containers that syncs to and from S3 using mobius3. This is to allow file-persistance on S3 without using FUSE, which at the time of writing is not possible on Fargate.
+dns-rewrite-proxy: + The DNS server of the VPC that launched containers run in. It selectively allows only certain DNS requests through to migitate chance of data exfiltration through DNS. When this container is deployed, it changes DHCP settings in the VPC, and will most likely break aspects of user-launched containers.
+healthcheck: + Proxies through to the healthcheck endpoint of the main application, so the main application can be in a security group locked-down to certain IP addresses, but still be monitored by Pingdom.
+mirrors-sync: + Mirrors pypi, CRAN and (ana)conda repositories to S3, so user-launched JupyterLab and rStudio containers can install packages without having to contact the public internet.
+prometheus: + Collects metrics from user-launched containers and re-exposes them through federation.
+registry: + A Docker pull-through-cache to repositories in quay.io. This allows the VPC to not have public internet access but still launch containers from quay.io in Fargate.
+sentryproxy: + Proxies errors to a Sentry instance: only used by JupyterLab.
+<%?nf^_fCi02Z%<%)POc$?p-o&Xftq+fSyL+rtpLf+4g34d#m$%|JM$Ouxa
z&F!lms^jToH?&G?Uw%|1?($E}xdrF3^5xry+cuwvgb3OH?!K4r8Q1f8v~s^W&yMH;
z?vRO031RT|-Q5^i!dNb-9f(zYya%sFX~Js1^nDgV;}eRkcCJ0X3HYL>mC**9pa1wN
znxiH9DCX`^HH-}=_%X@`V^PT@fDRUJF>mpIXP5Lw3G-PvL0j)RM$R?fHf1S3LNBEA
zypsIvKU~XTHkixB$%PqJvN&p