How to handle large'ish data

[sleffler]

I'm porting an application that needs an array that's ~300KiB (the app collects realtime audio data for processing). In other systems it's just declared as a static/global and lands in .data or .bss. I'm having trouble finding a way to do this in CHERIoT. The data are used by a compartment. Declaring them static/global:

#define kSamples (5 * 16000)
static int32_t samples[kSamples]

does not compile:

$ xmake
[ 53%]: cache compiling.release soundstream.cc
[ 70%]: linking compartment soundstream.compartment
[ 97%]: linking firmware build/cheriot/cheriot/release/soundstream-firmware
error: ld.lld: error: __cpp_top_matcha.c.c:(function entry(): .text+0xDA): relocation R_RISCV_CHERIOT_COMPARTMENT_SIZE out of range: 320000 is not in [0, 4095]

Allocating from the heap has been difficult; I increased the heap to .5MiB but see this (w/ --debug-allocator=true):

Allocator: Quota exhausted trying to allocate 0x4e400 bytes (remaining quota is 0xbf0)
Allocator: Not enough free space to handle 0x4e200-byte allocation, sleeping
Scheduler: Thread 0x1 waiting on futex 0x8000f248 (v:1 0x8000f248-0x8000f24c l:0x4 o:0x0 p: G RWcgm- -- ---) for 0xffffffff ticks

I've been reading through docs + code but haven't found much to help. My other thought was to declare an MMIO region in my board spec and use that directly but haven't tried that (yet) and was warned there might be addressing issues.

This seems like a common thing to do so what's the intended way in cheriot? 300K isn't that big a data structure for deeply embedded applications (especially when your system has 4M :)).

[davidchisnall]

Thanks for the question. The first approach will work in a future compiler version, but it requires a new kind of linker relaxation to support. We currently have a relocation that sets the bounds with a single csetbounds instruction, to support larger ones we need the compiler to emit 2 instructions to materialise the immediate and then a csetbounds that takes a register. This is then relaxed to either shorter instructions to materialise the integer (one for 16-bit values, two but one compressed for slightly larger ones) or a single csetbounds with an immediate. This is problematic for two reasons:

• The way linker relaxations is done upstream has changed slightly and I’d like to put it off until after our next upstream merge.
• Even with linker relaxations working perfectly, it requires allocating an extra register in the compiler and then, in the common case, eliminating all uses and definitions, which will impact code generation quality in all functions that access globals. To avoid paying this, I’d like to either require large globals to be annotated or require compilation with a larger code model. Feedback welcome on which approach would work better.

For static variables, this is somewhat simpler because the compiler knows the size. I may be able to special case it fairly easily.

The heap allocation option should work fine assuming a sufficiently large heap, but CHERIoT is a capability system. Everything that you do outside of a compartment requires some authorising capability. The C standard malloc and free functions are wrappers around the allocator APIs that take a capability that authorises allocation. By default, this capability has a [4 KiB quota]) https://github.com/microsoft/cheriot-rtos/blob/943280eacce9d461075c50741af165b5a099e76e/sdk/include/stdlib.h#L17). You can change that by defining MALLOC_QUOTA to the quota size that you want. The quota sizes are visible with cheriot-audit, so you can write policies that restrict heap usage.

There are some examples in the network stack of different default malloc quotas for compartments, including several with no ability to allocate memory on their own behalf.

[sleffler]

MALLOC_QUOTA did what I needed for now, thank you. Separately I was unable to setup a heap larger than .5M; e.g. the allocator rejected a 1M heap (w/ up-sized shadow): ``` Allocator: malloc_internal(0x200, 0x80001530 (v:1 0x80001530-0x80001538 l:0x8 o:0x0 p: - RWcgml -- ---)) Allocator: msize 0xf8 tsize 0x100000 MinChunkSize 0x10 MChunkHeader 0x8 MaxChunkSize 0x80000 ../../../../../../../cheriot/cheriot-rtos/sdk/core/allocator/main.cc:141 Assertion failure in check_gm gm should not be null, heap 0x80300000 (v:1 0x80300000-0x80400000 l:0x100000 o:0x0 p: G RWcgm- -- ---) ``` (my log messages added) It appears I need to reconfigure the allocator's internal settings to make this work. Is there an example? (the relationship between the various definitions was unclear at first glance)

…

On Mon, Mar 18, 2024 at 1:12 AM David Chisnall ***@***.***> wrote: Thanks for the question. The first approach will work in a future compiler version, but it requires a new kind of linker relaxation to support. We currently have a relocation that sets the bounds with a single csetbounds instruction, to support larger ones we need the compiler to emit 2 instructions to materialise the immediate and then a csetbounds that takes a register. This is then relaxed to either shorter instructions to materialise the integer (one for 16-bit values, two but one compressed for slightly larger ones) or a single csetbounds with an immediate. This is problematic for two reasons: - The way linker relaxations is done upstream has changed slightly and I’d like to put it off until after our next upstream merge. - Even with linker relaxations working perfectly, it requires allocating an extra register in the compiler and then, in the common case, eliminating all uses and definitions, which will impact code generation quality in all functions that access globals. To avoid paying this, I’d like to either require large globals to be annotated or require compilation with a larger code model. Feedback welcome on which approach would work better. For static variables, this is somewhat simpler because the compiler knows the size. I may be able to special case it fairly easily. The heap allocation option should work fine assuming a sufficiently large heap, but CHERIoT is a capability system. Everything that you do outside of a compartment requires some authorising capability. The C standard malloc and free functions are wrappers around the allocator APIs that take a capability that authorises allocation. By default, this capability has a [4 KiB quota]) https://github.com/microsoft/cheriot-rtos/blob/943280eacce9d461075c50741af165b5a099e76e/sdk/include/stdlib.h#L17). You can change that by defining MALLOC_QUOTA to the quota size that you want. The quota sizes are visible with cheriot-audit, so you can write policies that restrict heap usage. There are some examples in the network stack of different default malloc quotas for compartments, including several with no ability to allocate memory on their own behalf. — Reply to this email directly, view it on GitHub <#188 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRM7IIHS7HCG246KLMPBVTYY2OWXAVCNFSM6AAAAABEYVUJFKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQMRUGM4TE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[davidchisnall]

I believe that this is a limitation from requiring the ability to represent any aligned allocation. You may be able to get away with bumping MinChunkSize so that you're always allocating 32-byte aligned chunks.

We don't currently have anyone planning on building implementations with enough memory for this to be a problem (at least, not for the next year or two), but it would be great if you could open an issue to track it. The code was refactored a while ago to add the gm indirection, with a goal of eventually allowing the heap to be backed by multiple independent ranges of memory, but we've not done that yet.

Once you get to 1 MiB, you're close to the point where you may want a totally different allocator design. For a few MiBs on up, sizeclass allocators tend to perform better (and suffer less fragmentation than) range-based allocators, so it's not clear how much it makes sense to try to scale up this allocator versus implement a different allocator for larger memories. I'm not sure exactly where the transition happens. snmalloc works quite well with a 32 MiB heap, for example, but definitely doesn't scale down to our typical <100 KiB heaps.

[davidchisnall]

@sleffler, can you try this branch of the compiler? CHERIoT-Platform/llvm-project#20

It should make it possible to support large globals, as long as their size is visible in the same compilation unit as their use. This is guaranteed for statics, and for most non-statics that aren't doing exciting things with common linkage. It isn't a general solution, but it should work in the common case (and, where it doesn't, should still give a linker error).

I'd like it to see more testing before I merge it into the compiler though because there are some potential corner cases where it's an exciting footgun. If you declare an array extern in one compilation unit and define it with a different size in another then you will get the wrong bounds (if the array is more than 4095 bytes). Without this patch, we always catch that at link time (both will see the real size in the bounds), now we catch it in all of the cases we did before, but if your array is more than 4095 bytes then you can see incorrect bounds. In some cases, this is fine. If the bounds on the extern definition are smaller, then you just see a portion of the array, but if they're bigger then you will see bounds that may encompass other objects (or an untagged capability if the bounds go past the end of the globals region).

Eventually, we will have enough relocations that we can error in those case.