aufs is broken

[dimkr]

5.10.x builds fail because aufs is broken (https://github.com/puppylinux-woof-CE/woof-CE/runs/8128990805), probably because of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.10.y&id=9255a42fe7ab65bc1972ed7452966199d47b7009. Will aufs get fixed? I don't know.

Once aufs loses support for <=6.0 kernels, it will be risky to keep using aufs with kernel branches it's not guaranteed to work with, or the build will just fail. Some of these are nice and stable longterm branches, and it's sad if Puppy can't use them anymore just because aufs is unavailable.

The question is what to do about this, I see 4 options:

1. Maintain aufs for old kernels ourselves and backport+test fixes from aufs for >=6.0. Pros: Puppy can keep using aufs. Cons: somebody needs to do this hard work.
2. Drop aufs completely and switch to overlay. Pros: zero problems with aufs (no build failures, no bugs). Cons: no sfs_load and other differences in how PUPMODE 12 and 13 work with overlay; forced change for users.
3. Drop aufs for kernel branches that no longer officially supported by aufs (even if it builds fine), and automatically fall back to overlay in initrd when aufs is not available. Pros: zero problems with aufs with old kernels unsupported by aufs. Cons: users have no safe upgrade path to the next bugfix release (i.e. Vanilla Dpup 9.2.x won't have another bugfix release with aufs).
4. Keep aufs for kernel branches where aufs builds fine and seems to work, and automatically fall back to overlay in initrd if aufs is not available. Pros: zero problems with aufs with old kernels, if aufs remains compatible and stable. Cons: it's a question of time until aufs breaks in a particular kernel branch (and then what?), somebody needs to test aufs to verify its stability, it's risky for users who stick with an old and unmaintained aufs branch, and aufs can introduce severe issues and lead to data loss.
5. Proposed by @peabee: "freeze" every kernel branch at the last release officially supported by aufs. Pros: Puppy can keep using aufs with zero change for users. Cons: "zero change" also means users don't get the security and stability fixes from later bugfix releases of the same longterm branch, so users are stuck forever on a specific bugfix release (for example, users stuck on an old 5.4.x release won't receive the fix for CVE-2022-2586 from 5.4.211).

I think 3 is the best option. 1 requires human resources we don't have, 2 is too aggressive (if people want to use aufs with kernel 6.0, that's safe) and 4 is risky and just a temporary solution that leads back to option 1 eventually (i.e. kernel x.y.z+1 can introduce a data loss bug in aufs even if x.y.z worked just fine). EDIT: not a fan of 5 too, because the whole point of using longterm kernels is years of security and stability updates without breaking changes.

3 is a safe compromise, but builds that stick to an old kernel branch will need to change to overlay and that can be a breaking change for users. For example, I'll need to release a Vanilla Dpup 9.3.0 identical to the latest 9.2.x bugfix release, except the change to overlay.

[peabee]

Has information on the breakage been captured?
Has an issue been raised at J. R. Okajima Github repo?

I would think another option would be to "freeze" 5.10 with aufs at the version that works pro-tem until J. R. Okajima has had a chance to make a fix.

I think his announcement was that support for 5.15 would continue but it is due to go EOL before 5.10 which is somewhat bizarre!

[dimkr]

I would think another option would be to "freeze" 5.10 with aufs at the version that works pro-tem until J. R. Okajima has had a chance to make a fix.

But what will we do if aufs breaks again for 5.10.x or 5.15.x, after aufs for these versions is EOL?

[dimkr]

Has information on the breakage been captured?

https://github.com/puppylinux-woof-CE/woof-CE/runs/8150155791

[peabee]

Freeze them at the version where aufs works.....

J. R. Okajima has not yet stopped support for 5.10 so its likely he will fix this problem if it is properly reported to him.

[dimkr]

Interesting finding: aufs is broken for 5.4, probably for a long time.

Before 171d740, our kernel builds didn't fail on failure to apply aufs patches. Our 5.4 kernels had a partially applied aufs patch, and that can be super dangerous!

It looks like aufs is broken (at least) for 5.4, 5.10 and 5.15, and 5.4 is EOL already. Even if aufs for 5.10 is fixed, 5.4 remains broken.

[peabee]

I will raise an issue with J. R. Okajima for 5.10

[peabee]

sfjro/aufs-standalone#20

[dimkr]

Opened 2 PRs to prevent this aufs situation from blocking the development of Puppy builds that use overlay:

• Automatically fall back to overlay if aufs is unsupported #3363
• Add 5.10.x and 5.15.x kernels without aufs #3365

[peabee]

Subject:	Re: [sfjro/aufs5-standalone] aufs-5.10.117 patch fails with kernel-5.10.140 (Issue #20)
Fri, 02 Sep 2022 06:50:26 -0700
sfjro notifications@github.com

Thanx for the report.

I'll address it as soon as possible.





J. R. Okajima

[dimkr]

#3365 seems to work well.

All builds pass, except:

• 5.4.x kernels with aufs (no overlay flavor)
• focal64, because it uses 5.4.x with aufs
• 5.10.x kernels with aufs (while the overlay flavor builds fine)
• bullseye, because it uses 5.10.x with aufs

(Maybe 5.15.x will break soon, for the same reason as 5.10.x.)

[peabee]

5.4
lines 48-67
of
https://github.com/sfjro/aufs5-standalone/blob/aufs5.4.3/aufs5-standalone.patch

can be deleted as kernel /fs/file_table.c now contains EXPORT_SYMBOL(__fput_sync);

Build of 5.4.211 builds, boots and "works" in BionicPup32

[peabee]

Of course ignoring the 5.4 patch error is logically equivalent to deleting the patch from aufs5-standalone.patch so 5.4 was probably building correctly.

Unfortunately aufs5.4 series are not supported since Sep 2021.
I won't fix.
J. R. Okajim

[dimkr]

Maybe it compiles, but that doesn't mean that aufs from September 2021 with latest 5.4.x behaves exactly the same as the same aufs version on top of a 5.4.x release from September 2021. A thorough look at the diff between latest 5.4.x and 5.4.x from September 2021 is needed to tell if aufs is truly safe to use on top of a 5.4.x release it doesn't support officially.

[peabee]

We should therefore revert and freeze on
linux-5.4.150.tar.xz 30-Sep-2021 08:18
for focal64 builds
( https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/ )

[dimkr]

And miss 61+ sets of stability and security fixes?

[peabee]

OK - then for whichever version of 5.4 the change occurred that broke the patch.......

4.19 is still being used but support ended on Jun 18, 2020:
https://github.com/sfjro/aufs4-standalone/tree/aufs4.19.63%2B

[dimkr]

Pushed an improved version of #3363. Now the default is aufs, as before, but the init script automatically falls back to overlay if the kernel is built without aufs support. The user can force use of overlay even if the kernel supports aufs, by specifying punionfs=overlay through the boot loader.

Objections?

[peabee]

focal64-227 with kernel 5.4.210 boots OK but has many other issues.............
checkdeps_system.txt

[dimkr]

Tested #3363 and merged into testing. UNIONFS=aufs/overlay is gone from _00build.conf, and now initrd uses aufs if available, otherwise overlay. All build configuration with UNIONFS=overlay (plus bullseye) now use a kernel without aufs support, so next time aufs breaks, dpup won't be affected. In other words, I changed the woof-CE build configurations I maintain from aufs or overlay (with unused aufs support) to overlay without aufs support.

In addition, I triggered kernel-kit run #147: it's a rebuild of 5.10.139, so people who use 5.10.x with aufs have a stopgap solution for two weeks (the build artifact retention period).

The focal64 configuration in woof-CE is unmaintained, and Fossapup 9.5 is not reproducible with today's woof-CE. AFAIK this is the only user of the 5.4.x kernel. I wouldn't be surprised if nobody volunteers to fix aufs for 5.4.x, compare it against aufs master to ensure it's safe to use with latest 5.4.x, and test it thoroughly. (Same with 4.19.x, which builds fine but probably lacks fixes from today's aufs .)

I don't have the time and the willingness to maintain aufs support for old longterm kernels, and prefer to spend my time on polishing overlay support (things like #3348 and #3326) and other things, like Wayland support and HiDPI support.

If kernels with aufs (and the woof-CE builds that use them) remain broken, I'll disable the failing build configurations at some point in the future, to avoid waste of build time.

[dimkr]

(Quick summary)

Looks like the current situation with 5.10.x is resolved.

5.15.x and 5.10.x now build OK and still officially supported, while 4.19.x and 5.4.x are unofficially unsupported, build OK and it's unknown whether or not aufs is still safe to use with these, considering all the kernel changes since official support was dropped.

With #2963, #3317, #3343, #3363, #3365 and others, woof-CE is now fully decoupled from aufs. A Puppy that uses a kernel without aufs (and now we have automated builds for that) will just use overlay, so future breakage of aufs (like the expected breakage when all pre-6.0 kernels lose official aufs support) will not impact the development of Puppy builds that don't use it.

[dimkr]

Now that #3431 is merged, I will test these kernels for a week or two and then I'll be winding down my involvement with kernels built by https://github.com/puppylinux-woof-CE/woof-CE/actions/workflows/kernel-kit.yml. aufs is about to explode and I'm jumping ship.

I'll still fix critical issues that affect the 5.10.x kernel in Vanilla Dpup 9.2.x (at least, for some time), and general issues with kernel-kit that break the dpup kernels pipeline.