Investigate RISC-V confidential computing solutions

[CharlyCst]

The primary goal of Miralis is to remove the firmware from the trusted computing base (TCB), so far we mostly tested that firmware and payloads works without modifications but we didn't demonstrate a compelling use case for Miralis yet.
The goal of this issue is to investigate different options to built a demonstrator.

The area where Miralis brings the most value is confidential computing, that is where we really care about small and well defined TCB. I have two confidential computing frameworks in mind that would be got fits, so we will need to explore each of those and decide which one we want to go for.

The questions we need to answer for each option are:

• Does it seems feasible to use Miralis for this use case? Where does Miralis fit into the picture.
• What would be the impact (would that have a lot of influence on academia? On the industry?)
• What are the available platforms? Can we use existing boards? Does QEMU have support for all required extensions?
• How much engineering effort would it be to port the system on top of Miralis? What building blocks are already existing, what needs to be modified, and what needs to be created from scratch?
• Is there good documentation and an open source code base?

In addition we want to collect all relevant links (docs, source code, papers, blog posts, or anything potentially useful).

Keystone

The first is Keystone. Keystone is pretty old and well established now (the paper is from 2020), it is a system that allows to create user-space enclaves on RISC-V by using a M-mode security monitor and a S-mode runtime for the enclaves.
Keystone is built on top of OpenSBI, so includes the all of it in the TCB. We could re-build the M-mode portion of Keystone on top of Miralis and push OpenSBI outside of the TCB in vM-mode to offer enclaves without firmware in the TCB.

Links:

• Paper
• Website

CoVE

CoVE is a RISC-V effort to develop an architecture extension that offers support for confidential VMs. This seems to be mostly an industry-lead effort (whereas Keystone is much more academic-focused), it is more recent than keystone and seems to have broader community support. I am not sure if CoVE is the official name, the extension is called SmMTT where MTT stands for Memory Tracking Table.
Similar to Keystone, the confidential computing capabilities need to be integrated into an existing M-mode software. I don't know if there are existing implementations (maybe in OpenSBI or a fork of it?). I am not sure what is the current state of the proposal (is it ratified, or still evolving?) and support in the ecosystem.

Links:

• White paper
• Linux Plumber Conference talk
• SmMTT spec (PDF available in the release artifacts)

[Wesdcv]

Okay, i've looked at the both solutions for a bit, and here are some of my considerations:

Keystone

1. Miralis in a big picture:
   It seems feasible to decouple SM from OpenSBI. After boot SM works as a mostly passive module, processing some traps (mostly ecalls) or timer interrupts. It has a fairly well-documented call interface, so that should not be too complicated to reproduce.

  Keystone by default                                                                      
                      ┌────────────────┐ ┌────────────────┐                             
  ┌──────────────┐  │ │┌──────────────┐│ │┌──────────────┐│                             
  │    U-Mode    │  │ ││     App      ││ ││    E-App     ││                             
  └──────────────┘  │ │└──────────────┘│ │└──────────────┘│                             
                    │ │   untrusted    │ │    enclave     │                             
  ┌──────────────┐  │ │┌──────────────┐│ │┌──────────────┐│                             
  │    S-Mode    │  │ ││      OS      ││ ││      RT      ││                             
  └──────────────┘  │ │└──────▲┌──────┘│ │└──────▲┌──────┘│                             
                    │ Ecall/──┼┼───────┘ └───────┼┼─Ecall/┘                             
  ┌──────────────┐  │  ┌──mret┴▼─────────────────┴▼─────mret                            
  │    M-Mode    │  │  │   Security Monitor | OpenSBI    │                              
  └──────────────┘  │  └─────────────────────────────────┘                              
                                                                                        
                                                                                        
                                                                                        
                                                                                        
  Keystone with Miralis                                                                 
                      ┌────────────────────────────────────┐┌────────────────┐          
  ┌──────────────┐  │ │┌──────────────┐   ┌──────────────┐ ││┌──────────────┐│          
  │    U-Mode    │  │ ││   OpenSBI    │   │      App     │ │││    E-App     ││          
  └──────────────┘  │ │└─────┬──▲─────┘   └──────────────┘ ││└──────────────┘│          
                    │ │      │  │    untrusted             ││    enclave     │          
  ┌──────────────┐  │ │    Trap&│         ┌──────────────┐ ││┌──────────────┐│          
  │    S-Mode    │  │ │      │ Emulate    │      OS      │ │││      RT      ││          
  └──────────────┘  │ │      │  │         └──────▲┌──────┘ ││└──────▲┌──────┘│          
                    │ └──────┼──┼────────────────┼┼Ecall/── └───────┼┼Ecall/─┘          
  ┌──────────────┐  │  ┌─────▼──┴────────────────┘▼────mret─────────┴▼───mret           
  │    M-Mode    │  │  │     Miralis     |           Security Monitor       │           
  └──────────────┘  │  └────────────────────────────────────────────────────┘     

2. Academic Verification and Industry Impact:
   From an academic standpoint, if we were to rebuild SM on top of Miralis, it would be advisable to verify its correctness over Miralis as well. Given that SM is approximately 2000 LoC and Miralis is approximately 3000 LoC, with the latter likely to grow, this would be a challenging undertaking. In return, we will receive approximately 5,000 LoC TCB (for two platforms and a limited number of RV64 ISA extensions). From an industry perspective, Keystone's website lists only one company as an "Industry User." It is therefore unclear what impact this will have on the industry.

3. Available Platforms:

   • Current platforms include:
     • Generic (QEMU)
     • FPGA (Ariane)
     • HiFive Unmatched
     • Microchip Polarfire (MPFS)
     • SiFive FU540

   It is not yet clear whether the existing board will work with Keystone. A PR has been submitted which is intended to add support for the VisionFive2 board. Nevertheless, Qemu is already compatible with the master, and further testing can be conducted if necessary.

4. Engineering Effort:
   A notable portion of the code in Keystone's SM exhibits functionality comparable to Miralis, or at the very least, similar utility functions. It is to be hoped that a great deal of the existing code can be reused, for example, in the case of a trap handler, CSR read and write functionality or PMP management. However, other elements, such as all ten ECall handlers for RT and OS, the crypto attestation module (which is approximately 3 to 4k LoC in length ^_^) and a small number of additional items would have to be approached from scratch. Additionally, it is probable that we will require a rewrite of the SBI calls to remove OpenSBI from TCB completely. If there are no plans to support the "H" RISC-V extension, the SM code can be simplified.

5. Documentation and Codebase:
   In my view, Keystone's (codebase)[https://github.com/keystone-enclave/keystone/tree/master] and documentation represents the lesser of two evils, given that CoVE is still under development (version 0.7 is the latest).

CoVE

With this one we have quite an interesting story. The documentation seems to be at v0.7, but there is no central repository or any cross-references to other CoVE repos. From what i've grasped from different docs, CoVE is a separate RISC-V extension (dosc), that leverages some other non-ISA RISC-V extensions like SMMTT or IOMMU to implement some of its' CVM capabilities.

1. Miralis in a big picture:
   This is a somewhat complex issue. It should first be noted that there are multiple deployment models for CoVE. Here i'll cover three official deployment models and one unofficial, which may or may not work at all. For further information on the integration of CoVE within the proposed non-ISA RISCV security extensions framework, please refer to the following link. According to RIscvSoftwareEcosystem website, the development of CoVE software and firmware components (including CoVE guest, KVM CoVE host, KVMTOOL CoVE, and others) was delegated to Rivos company.

The official v0.7 documentation dictates (the models are simplified, please refer to p.30 of specification doc for further details) 3 deployment models:

1. CoVE TEE security manager (TSM) operates at the HS/S level with an OS, and their isolation is enforced by TSM driver operating in M-mode (presumably with firmware).

 CoVE default v1                                                 
                      ┌─────────────────┐     ┌─────────────────┐
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│
│   (V)U-Mode   │  │  ││      App      ││     ││      App      ││
└───────────────┘  │  │└───────────────┘│     │└───────────────┘│
                   │  │       host      │     │   confidential  │
┌───────────────┐  │  │                 │     │┌───────────────┐│
│    VS-Mode    │  │  │                 │     ││      TVM      ││
└───────────────┘  │  │                 │     │└──────┬─▲──────┘│
                   │  │                 │     │       │ │       │
┌───────────────┐  │  │┌───────────────┐│     │┌──────▼─┴──────┐│
│    HS-Mode    │  │  ││       OS      ││     ││      TSM      ││
└───────────────┘  │  │└──────▲─┬──────┘│     │└──────┬─▲──────┘│
                   │Ecall/─COVEH-ABI────┘     └────Ecall/───────┘
┌───────────────┐  │  ┌─mret──┴─▼─────────────────────▼mret─────┐
│     M-Mode    │  │  │           Firmware + TSM Driver         │
└───────────────┘  │  └─────────────────────────────────────────┘                                                               

2. TSM is the sole executable that operates in HS mode, whereas an OS is virtualised to VS mode. The TSM driver, however, remains active and operational in M-mode.

 CoVE default v2                                                 
                      ┌─────────────────┐     ┌─────────────────┐
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│
│   (V)U-Mode   │  │  ││      App      ││     ││      App      ││
└───────────────┘  │  │└───────────────┘│     │└───────────────┘│
                   │  │       host      │     │   confidential  │
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│
│    VS-Mode    │  │  ││       OS      ││     ││      TVM      ││
└───────────────┘  │  │└──────▲─┬──────┘│     │└──────▲─┬──────┘│
                   │  └─COVEH-ABI/H-ext─┘   Ecall/─COVG-ABI─────┘
┌───────────────┐  │  ┌───────┴─▼───────────────sret──┴─▼───────┐
│    HS-Mode    │  │  │                   TSM                   │
└───────────────┘  │  └─────────────────────────────────────────┘
                   │                                             
┌───────────────┐  │  ┌─────────────────────────────────────────┐
│     M-Mode    │  │  │           Firmware + TSM Driver         │
└───────────────┘  │  └─────────────────────────────────────────┘

3. TSM is executed in M-mode, while OS operates in HS/S mode. Consequently, there is no requirement for the TSM driver, which is therefore absent. This deployment mode appears to be capable of running one host and one confidential domain without the need for the SMMTT extension.

 CoVE default v3                                                 
                      ┌─────────────────┐     ┌─────────────────┐
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│
│   (V)U-Mode   │  │  ││      App      ││     ││      App      ││
└───────────────┘  │  │└───────────────┘│     │└───────────────┘│
                   │  │       host      │     │   confidential  │
┌───────────────┐  │  │                 │     │┌───────────────┐│
│    VS-Mode    │  │  │                 │     ││      TVM      ││
└───────────────┘  │  │                 │     │└──┬─────Ecall/─┘│
                   │  │                 │     │   │         mret│
┌───────────────┐  │  │┌───────────────┐│     │   │             │
│    HS-Mode    │  │  ││       OS      ││     │ COVG-ABI        │
└───────────────┘  │  │└────────┬──────┘│     │   │             │
                   │ Ecall/─COVH-ABI────┘     │   │             │
┌───────────────┐  │  ┌──mret───▼─────────────┘   ▼             │
│     M-Mode    │  │  │             Firmware + TSM              │
└───────────────┘  │  └─────────────────────────────────────────┘

For the 3 official models i believe that adding miralis to the picture would look somewhat similar to Keystone's scheme.

And remember about the unofficial deployment model? Well, in the Rivos repo for CoVE, a model is proposed that allows for the operation of CoVE with Miralis without any modifications.:

 CoVE unofficial(trusted firmware)                                                    
                      ┌─────────────────┐     ┌─────────────────┐                     
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│                     
│   (V)U-Mode   │  │  ││      App      ││     ││      App      ││                     
└───────────────┘  │  │└───────────────┘│     │└───────────────┘│                     
                   │  │       host      │     │   confidential  │                     
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│                     
│    VS-Mode    │  │  ││       OS      ││     ││      TVM      ││                     
└───────────────┘  │  │└──────┬─▲──────┘│     │└──────┬─▲──────┘│                     
                   │  └─────COVH-SBI────┘     └─────COVG-SBI────┘                     
┌───────────────┐  │  ┌───────▼─┴─────────────────────▼─┴───────┐                     
│    HS-Mode    │  │  │                   TSM                   │                     
└───────────────┘  │  └───────────────────┬─▲───────────────────┘                     
                   │                      SBI                                         
┌───────────────┐  │  ┌───────────────────▼─┴───────────────────┐                     
│     M-Mode    │  │  │                Firmware                 │                     
└───────────────┘  │  └─────────────────────────────────────────┘                     
                                                                                      
                                                                                      
                                                                                      
                                                                                      
 CoVE unofficial with Mirais                                                          
                      ┌─────────────────┐     ┌─────────────────┐                     
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│  ┌────────────────┐ 
│   (V)U-Mode   │  │  ││      App      ││     ││      App      ││  │    Firmware    │ 
└───────────────┘  │  │└───────────────┘│     │└───────────────┘│  └──────┬─▲───────┘ 
                   │  │       host      │     │   confidential  │         │ │         
┌───────────────┐  │  │┌───────────────┐│     │┌───────────────┐│         │ │         
│    VS-Mode    │  │  ││       OS      ││     ││      TVM      ││         │ │         
└───────────────┘  │  │└──────┬─▲──────┘│     │└──────┬─▲──────┘│      Trap&│         
                   │  └─────COVH-SBI────┘     └─────COVG-SBI────┘         │Emulate    
┌───────────────┐  │  ┌───────▼─┴─────────────────────▼─┴───────┐         │ │         
│    HS-Mode    │  │  │                   TSM                   │         │ │         
└───────────────┘  │  └───────────────────┬─▲───────────────────┘         │ │         
                   │                   Ecall/                             │ │         
┌───────────────┐  │  ┌───────────────────▼mret───────────────────────────▼─┴───────┐ 
│     M-Mode    │  │  │                           Miralis                           │ 
└───────────────┘  │  └─────────────────────────────────────────────────────────────┘                                                                                    

2. Academic Verification and Industry Impact:
   TODO
3. Available Platforms:
   Only Qemu so far, according to the Rivos repo.
4. Engineering Effort:
   Yet to be determined (TODO)
   As there are 43 CoVE-SBI calls defined in the specification, it would be reasonable to anticipate that the level of implementation complexity would be higher.
   It's also worth noting that adding support for "H" extension to Miralis would be necessary for some deployment models.
   Considering it's only a possibly unfinished spec without an open-source code implementation backing it up (There is Salus project by Rivos in development, which hasn't seen any updates in a year and is unlikely to be entirely coherent with the latest documentation), it is challenging to assess the engineering effort required.
   The only viable method for running CoVE with Miralis at present is through an unofficial deployment model.
5. Documentation and Codebase:

• Non-ISA TEE Spec - the primary spec for CoVE, I suppose
• Non-ISA TEE IO Spec - supporting spec defining secure IO SBI and device management with CoVE.
• Non-ISA Security model - provides insight into the broader relationship between CoVE and other non-ISA extensions like SMMTT, IOMMU, SMSDID, and others.
• Rivos repo for CoVE documentation - should be compliant with the Salus implementation?
• Salus - the only implementation of CoVE hypervisor i could find.
• A guide on how to run CoVE(Salus) with KVM+QEMU

[Wesdcv]

Apparently, there is a RISC-V VM-based TEE from IBM, called ACE, that implements the RISC-V CoVE spec's deployment model 3. The project seems to be supported and constantly updated, unlike Salus, and aims to have a formally verifiable Security Monitor (albeit it's size is ~9000 LoC).
For now it seems like it runs only on Qemu, and Miralis's place here would be similar to the Keystone model.
I could look into this more thoroughly if that is somewhat viable.

[CharlyCst]

Ho interesting, yes indeed we should probably investigate that project too, good catch 🙂