Need some help? Reach us at:
FastAPI is Python's hottest API framework for rapid and secure API development. In this repo, we provide tutorials and walkthroughs so that you can use FastAPI to build robust multiparty computation (MPC) applications. MPC is a class of protocol that enable multiple users to collaborate without directly seeing one another's inputs. It traditionally relied on distributed encryption protocols like secret sharing or homomorphic encryption. While these are super cool technologies, they are also very low level and often extremely slow due to rounds of communication, network latency and bandwidth issues. However, over the last few years, every major cloud provider has adopted secure enclaves (sometimes called confidential compute or trusted execution environments). These are isolated VMs that prove to users what software and OS are running inside. You can use this proof to enable trust between clients connecting to a server. In this repo, we'll get you up and started with secure enclaves using Oblivious and give you many examples of practical implementations with step-by-step tutorials on designing and building them yourself.
- Enclaves for Multiparty Computation
- MPC with FastAPI
- Example Applications
- YouTube Tutorial
- Awesome MPC with FastAPI & OBLV
- Contributing & Code Structure
- Disclaimer
Secure multiparty computation is a long-standing topic in computer security. It revolves around the challenge of having multiple "parties", i.e. servers and client computers, who collaborate to perform some form of computation. Right now, you are probably reading this from the GitHub readme, so in a sense, you are collaborating with one (or multiple) of GitHub's servers. The challenge is it's not secure, at least in the sense that you've no idea how GitHub is participating in this collaboration - are they logging your IP address? Selling your details to Evil Corp? Well, probably not - but you don't have proof they aren't.
Secure multiparty computation (MPC/SMPC) is the class of cryptography that leverages advanced cryptographic protocols to guarantee exactly what each party is doing in an interaction and that your inputs are kept confidential, only used for their intended purposes.
You may have heard of "hot" topics like Homomorphic Encryption, Secret Sharing, etc. The challenge with these are fewfold, but to summarize, they currently lack standards, tend to be very slow to run, and operate at a very low level.
An alternative approach is to use secure enclaves. These are isolated virtual machines that are now supported by every major cloud provider. They have highly limited IO and the infrastructure of the cloud attests the software that runs inside.
Oblivious (OBLV) provides an easy-to-use abstraction for you to leverage secure enclaves for multiparty computation, with a focus on data science applications. It's free to use (at least you get monthly free credits for a few hours worth of computing), and you can get started in seconds by signing up with your GitHub account here.
These enclave applications can be pretty useful if you want an iron-clad history of how you've used data throughout its life cycle. Every connection to an enclave with OBLV requires client authentication and PCR hashes which prove what's running in the enclave. Allowlisting these can be used to limit how data is allowed to be processed.
You can broker trust between multiple clients by creating an enclave and proving what's running inside. Each client connecting can validate the exact processing being performed and that their data can not be used outside of that context.
When you query a regular API, what happens to your data. Maybe you are using Google Translate on some critical legal docs; what does Google do with that data? Can Google employees ever see that information? Hopefully not, but it can be near impossible to enforce. Leveraging enclaves, you can offer and connect to enclaves with a guarantee that no one will see the inputs or outputs of the queries being performed.
When you start your enclave journey, usually it involves a lot of pain, libraries suddenly don't work, you have to deal with virtual sockets, manage networks, systems, security engineering... it's a pain. That's where OBLV tries to help. It lets you write normal webservers in your favourite framework (e.g. FastAPI) and patches all the gaps so you can focus on the important stuff -> the application logic.
graph TD;
subgraph client
A[client] --- B[client proxy];
end
B -. secured end-to-end connection .- C[enclave proxy];
subgraph enclave
C --- D[enclave services];
end
We typically encourage the development of reusable (ideally open source) applications. The idea is that you build once and can deploy for many circumstances, but the PCR code (used to establish trust) can be allowlisted as they'll be the same for each application. We do this by letting you define runtime args that get after the build phase and thus don't affect the PCR codes. These include specific user details or configuration relevant to a particular deployment. So try building apps with this in mind - how can I solve my problem in a reusable and generic fashion.
- Avoid hard-coding user details
- Use runtime configuration to specify particular parameters that slightly modify behaviour or switch between cases.
When a user connects to your API, the proxy will put their user name in the X-OBLV-User-Name headed field and X-OBLV-User-Role of the request. FastAPI has a neat way to recieve these in your application like:
@app.get("/")
def home(
x_oblv_user_name: str = Header(default=None),
x_oblv_user_role: str = Header(default=None)
):
if x_oblv_user_name is None:
raise HTTPException(401, "No X-OBLV-User-Name provided.")
elif x_oblv_user_role is None:
raise HTTPException(401, "No X-OBLV-User-Role provided.")
passMore details on the FastAPI docs.
The entire environment and code are hashed when the enclave image is built. This can be inconvenient if we want the enclave image PCR hashes to remain constant with details like who may connect to the enclave or some environment variables to change for different instantiations of enclave deployments. For this reason, the user details and a runtime YAML file are passed in just after the enclave is launched but before any party can connect to it. When launching the enclave, you can paste a YAML file which will be located in /usr/runtime.yaml.
graph TD;
a[Code] --> b[Docker Build];
b -- Hashes for attestation determined --> c[Enclave Image Build];
c --> d[Encalve Launch];
d -- Runtime YAML not effecting attestation post-launch --> e[Add Runtime YAML];
e --> f[Launch Services];
For FastAPI services leveraging runtime arguments, the pydantic library's BaseSettings can be extremely useful. For more details on how to use these generally, please consult the FastAPI documentation here. In this example project, we demonstrate how the BaseSettings can load /usr/runtime.yaml and be used within your application. The source code for this is in src/settings/settings.py and is called in the example route src/routes/config.py.
All outbound calls are blocked by default (to keep the data being processed secure and safe). However, you can allowlist URL endpoints while configuring the service. Specifically, a fully-qualified domain name (FQDN) gets allowlisted, so http://example.com will allow http://example.com/test but not http://test.example.com. This is because the allowlisting is on the TCP layer, not the HTTP layer (so it doesn't break your TLS privacy).
graph TD;
a[Service] --> b[DNS Routing]
b -- Approved FQDN 1 --> c[Outbound Socket]
b -- Approved FQDN 2 --> d[Outbound Socket]
b -- Approved FQDN 3 --> e[Outbound Socket]
When unit testing in FastAPI remember to add the header fields (X-OBLV-User-Name and X-OBLV-User-Role) to simulate the behaviour of the OBLV proxies. This can easily be done like this:
from fastapi.test client import TestClient
from .main import app
client = TestClient(app)
def test_example_accept():
response = client.get("/", headers={"X-OBLV-User-Name": "Alice", "X-OBLV-User-Role": "Admin"})
assert response.status_code == 200
def test_example_fail():
response = client.get("/", headers={"X-OBLV-User-Name": "Not Alice", "X-OBLV-User-Role": "Admin"})
assert response.status_code == 401More details on the FastAPI docs.
This repository is a valid enclave service with OBLV. It uses FastAPI routes to host five simple applications which showcase some of the functionalities and hopefully inspire you to start building your own:
Example 1: Hello World
Example 2: Outbound Calls
Example 3: Runtime Arguments
Example 4: Yao's Millionaire Problem
Example 5: Private Set Intersection Cardinality
Whether you wish to run this repository, or one of your own, the following steps will help you quickstart:
To get started with the Oblivious console, you first will need to go to the Oblivious webpage and signup with your GitHub a/c (hit the little GitHub icon). Agree to the OAuth scopes and you are all set!
The source code for the application is in /src but we will run through the main parts here for conciseness. The folder structure is as follows:
.
├── app.py # the main FastAPI server (only calls paths from /routes)
├── routes
│ ├── __init__.py
│ ├── config.py # API function to return the runtime arguments of the enclave service
│ ├── hello.py # API function to return the name and role of the user connecting
│ ├── outbound.py # API function to allow you to test outbound calls with the allow/deny lists
│ ├── psi.py # API function for private set intersection cardinality
│ └── yao.py # API function for yao's millionaires' problem
├── settings
│ ├── __init__.py
│ └── settings.py # Pydantic BaseSettings gets the runtime args from /usr/runtime.yaml
└── uvicorn_serve.py # the uvicorn server that deploys the FastAPI server inside the enclaveFeel free to explore the code. We have written it endeavouring to be simple and clear, but if there is any ambiguity please let us know.
Once your code is written (or in this case, when you are happy with the above demonstrative service in this repo), you can go ahead and configure it through the console. The steps are as follow:
- log into the console
- in the side panel on the left-hand side, hit Repositories
- find and click on the repo of interest and select the branch you wish to use
- configure how you wish the service to behave, in this case we will select the following:
auth:
- auth_name: pkipsk
auth_type: signed_headers
base_image: oblv_ubuntu_18_04_proxy_nsm_api_python_3_8
build_args: []
roles:
- role_auth: pkipsk
role_cardinality: 2
role_description: This is simply an example role for demonstration purposes
role_name: admin
paths:
- access: admin
path: /hello/
short_description: Hello world example
- access: admin
path: /outbound/
short_description: Example to demonstrate allowlist urls
- access: admin
path: /psi/submit_list
short_description: Submit CSV list for private set intersection
- access: admin
path: /psi/compare
short_description: Get result for private set intersection
- access: admin
path: /yao/submit_value
short_description: Submit value list for Yao's Millionaires Problem
- access: admin
path: /yao/compare
short_description: Get result for Yao's Millionaires Problem
- access: admin
path: /config
short_description: Get's config settings
traffic:
inbound:
- name: inbound
port: 80
type: tcp
outbound:
- domain: example.com
name: example
port: 443
type: tcp
meta:
author: Team Oblivious
author_email: hello@oblivious.ai
git: https://github.com/ObliviousAI/FastAPI-Enclave-Services.git
version: 0.1.0- When saved, this will create a
.obliviousfolder in the repository and will populate it with a.oblivious/service.yamlfile - Create a Dockerfile in the
.obliviousfolder with noFROM,CMDorENTRYPOINTcommand (these will be ignored if added), it's intended only to install dependancies
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
COPY ./src/ /code/- Create as many service as you would like to run concurrently in the enclave by creating a folder for each service in
.oblivious/services/<service_name>. These follow S6 service manager structure, so in each folder the minimum requirement is to have arunexecutable (eg bash file) which will run and restart is exitted. You can addfinish,finish-timeout, etc as per the S6 specifications here.
That's it! You now have an enclave service fully configured.
For step 3, we will act as a client application endeavouring to connect to the enclave. We've packaged this into a Google Colab for your convenience and you can access it here:
- OpenDP SmartNoise Synthetic Data from Multiple Sensitive Datasets:
(More Coming Soon! 🙌)
This repo was designed to be built upon, allowing great developers like you to help others get on board the OBLV enclave train. We highly encourage contributions via pull requests.
This code is in Beta; please treat it as such.