Multi-kernel support

[AndrewFasano]

I've temporarily disabled builds of our 6.7 kernel. But our 6.7 kernel is important and I don't want us to forget about it. In a few weeks when our dev pace slows down, I'd like to get our 4.10 changes synced with 6.7 and re-enable 6.7.

Instead of maintaining a complex custom kernel for one version (4.10), we should minimize our patches and try porting them to multiple kernel versions.

Tasks:

• Minimize kernel diff #19
• Port to newer kernel #17
• Port to old kernel #18

[AndrewFasano]

I've minimized the main_4.10 branch to drop dyndev and other unnecessary changes. I also did some code cleanup so we can dynamically enable/disable/configure some of our kernel patches - hypercalls, blocking reboot, and igloo_task_size.

https://github.com/rehosting/linux/compare/v4.10...min_4.10?expand=1

Still needs testing.

[AndrewFasano]

With these kernel changes the test suite hangs 😦

[AndrewFasano]

Tracked things down to a crash in proc_map.

Thread 53 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6361b5c640 (LWP 937)]
on_proc_vma_update (evdata=0x5557c9f246d0, udata=0x0) at /panda/panda/plugins/proc_map/proc_map.cpp:164
164     /panda/panda/plugins/proc_map/proc_map.cpp: No such file or directory.

The test suite does not handle this sort of failure well, I had to hack up the test suite to preserve the tmp directory, then run the container manually and run gdb --args python3 -m penguin.penguin_run /tmp/config.yaml before I could see any error beyond "no .run file produced"

[AndrewFasano]

@lacraig2 / @jamcleod were brainstorming about trying to support a bunch of different kernel versions (latest 2.6, etc)

[lacraig2]

@jamcleod gave me a lot of great details on limitations we might see trying to handle our rust-based dependencies. Some of that discussion might be worthwhile to add here or in another issue.

I took a stab at porting @AndrewFasano's min_4.10 branch to 2.6 here: https://github.com/rehosting/linux/tree/min_2.6

It doesn't currently compile with linux_builder as that has GCC 11 and this requires GCC 3 or 4 as far as I can tell.

[AndrewFasano]

Tracked down the hang to be caused by a bug in penguin_plugin proc map and fixed it with https://github.com/rehosting/penguin_plugins/commit/b87a34789be64d8aa2e6294ea46b3eab3665fc36. Tests are (so far) passing - will force push to 4.10_main soon if they all do

[AndrewFasano]

I renamed what was 4.10_main to old_dyndev_4.10 (in case we ever need that code again) and updated 4.10_main to be the minimized version (diff from linux v4.10). The linux_builder CI built a v2.2.0 release with.

I created a penguin PR to switch to the new kernel version.

[AndrewFasano]

Split into 3 issues, first of which is done when we merge the penguin PR. I'm un-assigning myself from the other 2

[AndrewFasano]

Open question here: what's the best way to maintain a patch series for multiple kernel versions? Should we:

• have branches for each kernel version (e.g., 4.10_main) that is an upstream release plus a series of commits that we cherry-pick/rebase across branches?
• Have a series of patches with some sort of tags for which kernel versions they apply to and then use CI/helper scripts that takes in a kernel version, grabs it from upstream (probably linux-stable over linux?), selects all the relevant patches, and applies them.

We're currently doing option 1, but kind of like option 2 (especially if we ever support fully dynamic kernel selection and JIT building) but it would definitely have more setup to make it work. We'd need to change our patch series so each patch is more stand-alone and then figure out which kernel versions they don't apply to and fix those conflicts.

[lacraig2]

I think option 1 is reasonable, but if we wanted to follow closer to option 2 we could consider building off of an existing architecture for something like this.

In particular, buildroot might be perfect for this. They maintain patches for the linux kernel, build buildroot and other packages we might want, and would likely also give us helpful test images.

[AndrewFasano]

Are there many patches in buildroot? Not sure if I'm missing something, but it seems like that one you linked to is for an exact kernel version (5.10.162) while if we wanted to do something like that I think we'd have patches that apply to ranges of versions.

[lacraig2]

Buildroot has lots of patches for lots of targets. Look in any /board/[TARGET]/patches folder and you'll see.

For example, this one for the beaglebone that adjusts something: https://github.com/buildroot/buildroot/blob/master/board/beaglebone/patches/linux/0001-keep-jtag-clock-alive-for-debugger.patch

It even has board specific extended linux configurations per board. See: https://github.com/buildroot/buildroot/blob/master/board/beaglebone/extlinux.conf

To your question on patches that apply to a range of versions: that's not functionality I can think of anyone other than us wanting so naturally it doesn't exist here, but I don't think it's that hard to build.

Buildroot builds targets based off of a config. That config can specify things like the kernel version and a config for that kernel and such. We can also adjust the patch directory that is being used.

So we could maintain a script that updated those two values and kept patches for a series of kernels in one directory and others in a different directory and choose the right set as necessary. The other thing I like about their patches is that they're typically named like commits and they have a folder of all the changes.

[AndrewFasano]

Switching to buildroot seems like a fairly heavyweight change. I'm all for avoiding reinventing the wheel, but it seems like we'd be introducing a lot of complexity and still need to implement the feature we want. Maybe there are other advantages to buildroot that I'm not tracking though.

Perhaps we just stick with option 1 for now?

Though I'm just worried that we'll improve an existing commit on a branch like 4.10 main and then have a hard time tracking that we'd need to also update the patch for other kernel versions. For example if we support kernels 2, 3, 4, 5, and 6 it will be a pain to move changes between kernel versions, even if the same commits apply cleanly. We also don't have any history when we rebase and force push a branch (but we don't really want to keep all the cruft in the linux repo as that makes it harder to port).

We currently have 12 patches which could reasonably be squashed down to 10 independent chunks of logic that we could port around (though the first 2 are dependencies for the others):

• Build fixes for old kernels with new compilers
• Add hypercalls that can be disabled
• Customizable task size
• Boot option to block halt/shutdown/reboot
• Hypercall on VMA update
• Hypercall on ioctl errors
• Hypercall on network binds
• Hypercall on all syscalls (filtered)
• Maybe: firmadyne changes for hiding kernel modules
• Maybe: firmadyne changes for blocking mac address changes on bridges

It seems like at least some of these require no changes across 2.6 and 4.10 (e.g., rehosting/linux@455368e and rehosting/linux@12347fe) while others do.

If we just had a folder of patches with a subfolder for each of these, we could mark patches as applying to ranges of kernel commits (at hashes or versions) and then we'd have the history of patch changes and could use more standard git practices.