Port k3s to Android 14

[bobmorane83]

K3s Versions to be Validated

# k3s --version
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

Hi all,

I'am trying to run k3s over a real Android 14 arm64 hardware.
All kernel flags have been enabled and generic Linux directories added. BTW, I already run Docker successfully (root, no SELinux. no overlayfs)

Running k3s

k3s server -o /data/rancher/config --config /data/rancher/config.yaml

I have this "INFO":

INFO[0015] Waiting to retrieve agent configuration; server is not ready: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount overlay: invalid argument 

and I get :

# kubectl get node 
NAME        STATUS     ROLES                       AGE   VERSION
localhost   NotReady   control-plane,etcd,master   28h   v1.30.1+k3s1

So it seems that the "mount" command to mount overlayfs path do not have correct parameters on Android.
Maybe this might be due to F2FS used by Android as main FS. I don't have enough knowledge for now on this FS and k3s.

Adding --snapshotter=native gives:

E1009 13:04:09.807103   17872 kubelet.go:1547] "Failed to start ContainerManager" err="failed to get rootfs info: failed to get mount point for device \"/dev/block/dm-6\": no partition info for device \"/dev/block/dm-6\""

For the record, to run k3s successfully on Cuttlefish x86, I had to change the FS to EXT4 and use --snapshotter=native.

BTW,
I can't cross re-compile k3s for arm64 on x86 PC (to add more debug logs).
My cmd are :

mkdir -p build/data && ARCH=arm64 make download && ARCH=arm64 make generate
ARCH=arm64 SKIP_VALIDATE=true make

the k3s is still for x86

Any help will be appreciated,
Tx,

[bobmorane83]

Adding fuse-overlayfs with option --snapshotter=fuse-overlayfs I still get an error :

INFO[0010] Waiting to retrieve agent configuration; server is not ready: "fuse-overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "native": fuse-overlayfs not functional, make sure running with kernel >= 4.18: failed to mount fuse-overlayfs ({Type:fuse3.fuse-overlayfs Source:overlay Target: Options:[lowerdir=/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower2:/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower1]}) on /var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/merged: mount helper [mount.fuse3 [overlay /var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/merged -o lowerdir=/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower2:/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower1 -t fuse-overlayfs]] failed: "unknown argument ignored: lazytime\nunknown argument ignored: -t\nfuse: failed to access mountpoint fuse-overlayfs: No such file or directory\nfuse-overlayfs: cannot mount: No such file or directory\n": exit status 1

Even if I can create a working overlay with something like :
fuse-overlayfs -o lowerdir=$PWD/lower,workdir=$PWD/work,upperdir=$PWD/upper $PWD/overlay

Seems arguments are not correct ...

[bobmorane83]

Using Docker (version 20.10.21, containerd v1.6.9) with option --docker lead to the same error from F2FS :

E1009 13:04:09.807103   17872 kubelet.go:1547] "Failed to start ContainerManager" err="failed to get rootfs info: failed to get mount point for device \"/dev/block/dm-6\": no partition info for device \"/dev/block/dm-6\""

[brandond]

i suspect f2fs is lacking support some of the sysctls required for this to work properly. You'd probably have to make an xfs or ext3 partition available.

[bobmorane83]

i suspect f2fs is lacking support some of the sysctls required for this to work properly. You'd probably have to make an xfs or ext3 partition available.

Yes, using an ext4 partition is Ok.
I'll investigate what is missing.

[liyimeng]

@bobmorane83 I guess that you are at least need to mount /var/lib/rancher/k3s/agent on top of a ext4/xfs partition.

Do you use Android 14 vanilla or a customize one?

[tomaszduda23]

I run k3s agent #11130

[bobmorane83]

I run k3s agent #11130

Thanks ! Very interesting. I'll investigate in this direction.
Did you tried to run k3s as server ?

From my side, I reached to run it on ext4 partition, but have the known "route default" problem as it seems that Android do not have default IP route checked by k3s (or dependancies).

BTW, do you cross compile k3s or do you compile it directly on arm64 device ?

BR.

[tomaszduda23]

I've not checked server. You don't need ext4 I run it directly on f2fs. I updated script in the other thread. It fixes "route default". I build directly on android.

[tomaszduda23]

tomaszduda23@259433c I recently update to this if anyone is interested.

[tomaszduda23]

btw for solving "default root" ndc could be used. It can control network configuration on Android. https://android.googlesource.com/platform/system/netd/+/master/server/NdcDispatcher.cpp. E.g. for docker it fix network partially

ndc network create 30
ndc network interface add 30 docker0
ndc network route add 30 docker0 172.17.0.0/16
ndc ipfwd add lo docker0