🐛 Autoreset issues on RPi

[mauromorales]

tested ubuntu v2.4.0-alpha3 on RPi4

• process seems to finish but it doesn't auto reboot neither returns to a terminal
• GRUB characters are broken afterwards

[jimmykarily]

Maybe related? #1723

[Itxaka]

I saw broken characters from the start, the issue seems to be the change between gftxterm and normal, which produces 2 alternating outputs via the same terminal.

Autoreset is correct as the kairos-reset service does not have the reboot flag or the unattended flag.

[Itxaka]

For the autoreset: kairos-io/packages#419

[Itxaka]

autoreset is now fixed by the patch above.

Grub characters indeed are broken, looks like some files are missing when doing the reset, to be copied, modules maybe as this are the only ones copied over:

https://github.com/kairos-io/kairos-agent/blob/main/pkg/utils/grub.go#L137

I saw some messages about files missing when grub loads, but its impossible to capture the moment as it gets erased...

[Itxaka]

grub> loadfont unicode
error: ../../grub-core/fs/fshelp.c:257:file `/grub2/fonts/unicode.pf2' not
found.
grub> 

we missing copying that into the fonts dir (fonts dir doesnt even exist)

[Itxaka]

Needs confirmation

[jimmykarily]

• I built an image from master (commit sha: 91b652b3)
• I run kairos-agent --debug reset and I got this error:

INFO[2023-09-14T09:37:02Z] Unmounting disk partitions                   
DEBU[2023-09-14T09:37:02Z] Not unmounting partition, /run/cos/state doesn't look like mountpoint 
DEBU[2023-09-14T09:37:02Z] Not unmounting partition, /run/cos/persistent doesn't look like mountpoint 
DEBU[2023-09-14T09:37:02Z] Not unmounting partition, /run/cos/oem doesn't look like mountpoint 
DEBU[2023-09-14T09:37:02Z] Not unmounting partition, /run/cos/efi doesn't look like mountpoint 
INFO[2023-09-14T09:37:02Z] Formatting 'state' partition                 
DEBU[2023-09-14T09:37:02Z] Running cmd: 'mkfs.ext4 -L COS_STATE /dev/mmcblk0p2' 
1 error occurred:
        * exit status 1

• I run the failed command manually and it seems that there is prompt that makes it fail:

root@cos-recovery:/home/kairos# mkfs.ext4 -L COS_STATE /dev/mmcblk0p2
mke2fs 1.47.0 (5-Feb-2023)
/dev/mmcblk0p2 contains a ext4 file system labelled 'COS_STATE'
        last mounted on /run/initramfs/cos-state on Fri Aug 18 18:15:41 2023
Proceed anyway? (y,N) 

• I answered y and let it finish
• I run kairos-agent --debug reset again and this time it went through
• I rebooted

fonts in grub looks fine.

So the font issue is gone but we have another issue with mkfs. We probably need to pass some flag to avoid prompts.

[jimmykarily]

There is the -F flag for mkfs.ext4 that skips prompts. We should probably use it.

[Itxaka]

That is kind of weird... it should affect x86 as well unless state is somehow
different on RPI....

[Itxaka]

Looks like it's mounted on s different place to what reset tries to unmount...

[jimmykarily]

output:

INFO[2023-09-14T10:07:29Z] Unmounting disk partitions                   
DEBU[2023-09-14T10:07:29Z] Not unmounting partition, /run/cos/state doesn't look like mountpoint 
DEBU[2023-09-14T10:07:29Z] Not unmounting partition, /run/cos/persistent doesn't look like mountpoint 
DEBU[2023-09-14T10:07:29Z] Not unmounting partition, /run/cos/oem doesn't look like mountpoint 
DEBU[2023-09-14T10:07:29Z] Not unmounting partition, /run/cos/efi doesn't look like mountpoint 
INFO[2023-09-14T10:07:29Z] Formatting 'state' partition                 
DEBU[2023-09-14T10:07:29Z] Running cmd: 'mkfs.ext4 -L COS_STATE /dev/mmcblk0p2' 
1 error occurred:
        * exit status 1

mounts:

kairos@cos-recovery:~$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0  2.6G  1 loop /
mmcblk0     179:0    0 59.7G  0 disk 
├─mmcblk0p1 179:1    0   96M  0 part 
├─mmcblk0p2 179:2    0  7.9G  0 part 
├─mmcblk0p3 179:3    0  5.3G  0 part /run/initramfs/cos-state
├─mmcblk0p4 179:4    0   64M  0 part /oem
└─mmcblk0p5 179:5    0   64M  0 part 

[jimmykarily]

Running sudo su before kairos-agent reset makes it work. I'll try once with just sudo . I think that one failed once but I need to re-try to be sure.

[jimmykarily]

sudo kairos-agent reset also works. It was simple after all. I just needed permissions. It would be nice if we showed a better message though. I will check if that's possible.

[Itxaka]

SO I guess this is a closed issue, yay!

[jimmykarily]

For better error messages: kairos-io/kairos-agent#145