caprevoke: further improvements

[brooksdavis]

From @nwf in the comments of #1401: the current caprevoke is probably best considered a prototype rather than something that can be merged directly. At the very least, I think the following things should happen:

• Cap-dirty tracking needs correction for the case of aliasing PTEs; see https://github.com/CTSRD-CHERI/paper-revocation3/issues/24 . This will likely matter a lot more for colocated processes. There is some architectural work here, too.
• The vm_fault.c logic for Cornucopia Reloaded should be finished; there are a few cases where I am concerned we might not scan a page for revoked pointers before exposing it to userspace.
• The support for the old "store-side" Cornucopia mechanism should be removed, leaving only the new Reloaded approach. This obviates https://github.com/CTSRD-CHERI/paper-revocation3/issues/31 .
• The page iteration logic is very MI-layer at the moment, but should be made more MD-layer. In particular, having locked the pmap to update a PTE, the pmap is in a position to tell us about the next un-visited PTE (with wrong CLG) or where there is a gap in the mapping tables that the VM should probe.
• The page visiting logic is very open right now, always calling a function pointer for the predicate on a capability under test; that should be factored out to inline the common case of only probing the fine-granularity bitmap. It's possible that this could be done for one of the read-write or read-only paths and still get most of the effect on most workloads.
• We hold vm maps busy for far too long in an effort to stave off thinking about fork()-ing during a revocation scan, but this also interlocks with gentler operations like munmap and medium-softness operations like mmap. That definitely needs fixing for use outside of benchmarking. See https://github.com/CTSRD-CHERI/paper-revocation3/issues/32
• Something better must be done about superpages. At the moment, we destroy them and try to put them back together later, but it's imperfect.
• The current changes to {kqueue, aio, ktimer} and to perform thread register file scanning are invasive and an incomplete enumeration of hoarders. See https://github.com/CTSRD-CHERI/paper-revocation3/issues/41 and https://github.com/CTSRD-CHERI/paper-revocation3/issues/40 and https://github.com/CTSRD-CHERI/paper-revocation3/issues/36
• The "whole bitmap access" thing is useful for benchmarking but is unsafe, especially in the case of coprocesses. It'd be ideal to have an integrated revoker in an allocator before we merged to dev so that we could credibly do away with that escape hatch. See https://github.com/CTSRD-CHERI/paper-revocation3/issues/17

I believe https://github.com/CTSRD-CHERI/paper-revocation3/issues/35 is fixed, but I should double-check.

It'd also be really nice to have someone pursue the "as-user CLG-bypassing load-capability" instruction so that we could replace the whole DMAP-based sweeper with scanning directly in the VA space, but that's definitely more architectural. See https://github.com/CTSRD-CHERI/paper-revocation3/issues/33 and https://github.com/CTSRD-CHERI/paper-revocation3/issues/28 .

[nwf]

Another thing for the list:

• Convert the cheri_revoke system call from blocking to default to asynchronous, backed by kernel worker(s) and adapt the existing mrs to match.

[brooksdavis]

And another:

• Make dequeuing revoked entries async. Needs a thread which is complicated in libc.