This is a research project aims to enable binary compatibility execution of Intel SGX enclaves on AMD SEV machines. The paper is accepted to 2022 IEEE Symposium on Security and Privacy. You can download the paper here.
All implementations except for existing code bases (Linux, Intel SGX SDK, etc.) were written and debugged by NSKernel.
This project is opensourced under GPLv2. See LICENSE.
Copyright (C) 2022 NSKernel and OSU SecLab.
Other components included in this repository are also licensed under GPLv2.
- Linux SGX Driver (See
linux-sgx-driver/License.txt) arch/x86/kvm/cpuid.candarch/x86/kvm/Makefilefrom Linux (Seekvm/LICENSE.txt)
Disclaimer: vSGX IS NOT BUG FREE AND THERE IS NO WARRANTY FOR THE CODE. KERNEL CODE MODIFICATIONS IN IT MAY CAUSE KERNEL PANICS AND/OR DESTRUCTIVE CONSEQUENCES. DO NOT TEST THE CODE ON ANY MACHINE THAT YOU ARE NOT WILLING TO WIPE.
This project once was called 'EmuSGX'. Some of the naming in the code is still carrying this name.
The following assumes a Ubuntu environment. Suppose you are at ~/.
- Machine has an SEV processor
- Machine has SEV enabled in the UEFI
- You know how to build and install Linux kernel
- You know how to build and install Linux SGX SDK and build SGX software
- You have read the paper
This is a relatively complicated system with multiple components down from kernels, kernel modules up to SDK and environments. I recommend you to have proper experiences on kernel coding and SGX SDK to minimize your frustration in your adventure.
git clone https://github.com/OSUSecLab/vSGX.git
Clone the AMD SEV project
git clone https://github.com/AMDESE/AMDSEV.git
Checkout the SEV-ES branch (Even if you just have SEV. SEV-ES works on both SEV and SEV-ES)
git checkout sev-es
The reference commit we tested is 222e2942a10e0174b5ef90439785641a956c45ad.
Follow the README.md to build the SEV environment. Install everything built EXCEPT FOR THE KERNEL.
Copy the 4 files under the vSGX/kvm folder to overwrite the corresponding ones in AMDSEV/build/linux/arch/x86/kvm.
cp vSGX/kvm/* AMDSEV/build/linux/arch/x86/kvm
Build the kernel again manually by
cd AMDSEV/build/linux
make -j $(getconf _NPROCESSORS_ONLN) LOCALVERSION='-vsgx' bindeb-pkg
Install the built kernel .deb packages. You can now launch your own VMs on SEV-ES using QEMU. Create two VMs: the AVM and the EVM, both using stock Ubuntu with OVMF according to the README.md. (Note that you do not need a GUI to run vSGX)
Clone the 5.10.20 version of 5.10.y branch of Linux Stable Tree
cd ~
git clone -b "v5.10.20" --depth 1 --single-branch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
Apply the EVM patch
git am ~/vSGX/evm-kernel.patch
git am ~/vSGX/evm-kernel-fix1.patch
Create a build folder and use the given configuration evm-config to build the EVM kernel
cd ~
mkdir evm-kernel-obj
cd linux
make O=../evm-kernel-obj/ defconfig
cp ../vSGX/evm-config ../evm-kernel-obj/.config
make O=../evm-kernel-obj/ oldconfig
Build the EVM kernel
make O=../evm-kernel-obj/ -j $(getconf _NPROCESSORS_ONLN) LOCALVERSION='-vsgx-evm' bindeb-pkg
You can now launch the EVM you just created in Step 2 and install the kernels.
Copy kvm/enclave-manager into the EVM. Build the enclave manager by
cd enclave-manager
gcc -pthread asm_helpers.c manager.c
We suggest you to use a clean and plain Linux 5.10.20 on AVM. The kernel module might not work with a different version. To build a clean and plain Linux 5.10.20, checkout back to linux-5.10.y branch, build and install. You should have CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL enabled.
Copy vSGX/emusgx-guest-module, vSGX/sdk.patch and vSGX/linux-sgx-driver into the AVM. (See why and how the driver is modified in COMPATIBILITY.md)
Now in AVM, build the Linux SGX Driver we provided
cd linux-sgx-driver
make
Clone the Linux SGX SDK
cd ~
git clone https://github.com/intel/linux-sgx.git
Checkout the commit 9671c99941814c57be575cbfebc9fe64a05533a4 (We do not guarantee that there is no conflict of our code with other commits) and create a new branch out of it
git checkout -b vsgx
Apply our patch (See why and how the SDK is modified in COMPATIBILITY.md)
git am ~/sdk.patch
Build the SDK and PSW according to the README.md, install it.
In AVM, install both the emusgx-guest-module and the provided linux-sgx-driver
cd ~/emusgx-guest-module
make install
<Type your password>
cd ~/linux-sgx-driver
sudo insmod isgx.ko
To ensure everything works, you can type dmesg and you should see this
[ xx.xxxxxx] EmuSGX: Initializing guest OS module...
[ xx.xxxxxx] vSGX: Fucking kallsyms...
[ xx.xxxxxx] vSGX: kallsyms_lookup_name@0xFFFFFFFFXXXXXXXX
[ xx.xxxxxx] EmuSGX: Hooking IDT...
[ xx.xxxxxx] EmuSGX: IDT hooked. EmuSGX is now running
[ xx.xxxxxx] EmuSGX: Initializating shared page
[ xx.xxxxxx] EmuSGX: Share page initialization is done
[ xx.xxxxxx] vSGX: Worker threads successfully created
[ xx.xxxxxx] not_intel_vsgx: hey! I'm running on AMD??
[ xx.xxxxxx] not_intel_vsgx: not even having SGX??
[ xx.xxxxxx] not_intel_vsgx: you won't believe what I'm running on
[ xx.xxxxxx] intel_sgx: Not Intel SGX Driver v2.11.0
[ xx.xxxxxx] not-intel-sgx: EPC bank 0x80000000-0x95000000
[ xx.xxxxxx] not-intel-sgx: bank 0 mapped to va @ 0xFFFFXXXXXXXXXXXX
[ xx.xxxxxx] not_intel_sgx: now registering /dev/isgx
[ xx.xxxxxx] not_intel_sgx: registered
To see all the logs in AVM in real time, type sudo dmesg -n 8. The AVM side is now ready.
Now on EVM side, run the enclave manager by
cd enclave-manager
./a.out
You should see
vSGX: Switchless worker is on
vSGX: Dispatcher worker is on
To check the EVM is registered to the AVM, a dmesg on AVM side should show
[ xx.xxxxxx] EmuSGX: Registered VM at 0 with ID XXXXXXX
Now build a sample code from the Intel SGX SDK. Remember to source the environment as instructed by the SDK installer.
source ${sgx-sdk-install-path}/environment
cd linux-sgx/SampleCode/SampleEnclave
make
Run the code by typing
./app
The app should just work.
A launched EVM must be discarded. Just kill it. The AVM can be safely shutdown.
Graphene-SGX requires some more modifications to overcome the GenuineIntel check and CPUID related problems. We will provide a patch to it later so you can try running it yourself.
EINIT's signature check is bypassed right now because I tried the kernel cryptographic APIs but failed to make it work. However this is an engineering effort and a person who is familiar with the APIs can help. Your patch is welcome.- Syscall interface in EVM is left open for debug purposes. You can disable syscalls by adding
if (current->is_enclave_thread) {
panic("vSGX: Enclave accessed syscalls. Abort.\n");
// or expand the AEX for UD and drop an AEX here
}
- Minimisation of the EVM kernel. In fact, everything except for thread and memory management can be trimmed. The enclave does not rely on any conventional system service.
- This is an implementation to demonstrate the feasibilty and to illustrate the performance of the paper. There might be some bugs remain in the system.