[BUG] The latest container cannot be pulled with podman

[Iksas]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Podman fails while trying to pull the latest container version:

user@hostname:~ $ podman pull lscr.io/linuxserver/smokeping:2.8.2-r3-ls106
Trying to pull lscr.io/linuxserver/smokeping:2.8.2-r3-ls106...
Getting image source signatures
Copying blob 0e3b4ee9bf2d done  
Copying blob df25a931801a done  
Copying blob fa81d67c2a4d done  
Copying blob 0c166d159f54 done  
Copying blob b1aeddeb68d1 done  
Copying blob 0e3b4ee9bf2d done  
Copying blob df25a931801a done  
Copying blob fa81d67c2a4d done  
Copying blob 0c166d159f54 done  
Copying blob b1aeddeb68d1 done  
Copying blob 79cad1bcf2aa done  
Copying blob f9e2d38d31b2 done  
Copying blob f7f06080ab18 done  
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:f9e2d38d31b23fe73c37322e331ea9053cf786706088eca49277071f0bb6c32d": ApplyLayer stdout:  stderr: potentially insufficient UIDs or GIDs available in user namespace (requested 197609:197609 for /root/.cpanm/work/1725395868.247/PPI-1.279): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /root/.cpanm/work/1725395868.247/PPI-1.279: invalid argument exit status 1

The same error also occurs when running podman pull lscr.io/linuxserver/smokeping:latest.

Note that the first five blobs are copied twice instead of once. This is not related to the error. On local builds, each blob is only copied once, but the error occurs nonetheless.

While pulling the image, podman seems to attempt to use a subuid:subgid combination of 197609:197609, which lies outside of the ranges configured for the user:

user@hostname:~ $ cat /etc/subuid
user:100000:65536

user@hostname:~ $ cat /etc/subgid
user:100000:65536

Running podman system migrate also does not fix the issue.

Expected Behavior

When trying to pull the previous version 2.8.2-r3-ls105, podman does not fail:

user@hostname:~ $ podman pull lscr.io/linuxserver/smokeping:2.8.2-r3-ls105
Trying to pull lscr.io/linuxserver/smokeping:2.8.2-r3-ls105...
Getting image source signatures
Copying blob 3849eecba27d done  
Copying blob df25a931801a done  
Copying blob 67b4f3e802d4 done  
Copying blob 15410b37d6a0 done  
Copying blob 3b9a5ab1d346 done  
Copying blob e4b9de32bfa4 done  
Copying blob 2067ce2189be done  
Copying blob 9c048e1d13f1 done  
Copying config 2f9f949bf5 done  
Writing manifest to image destination
Storing signatures
2f9f949bf59deb2ab2f9440936b80d8e616b784f4be814e1da3f764894c9b6a5

Steps To Reproduce

• Install podman
• Run podman pull lscr.io/linuxserver/smokeping:2.8.2-r3-ls106.

Environment

- Affected system 1:

    - OS: Raspberry Pi OS Bookworm, aarch64. `uname -a` outputs `Linux hostname 6.6.31+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux`

    - podman version 4.3.1, installed through the distro's package manager.

- Affected system 2:

    - OS: Debian Bullseye, x86-64. `uname -a` outputs `Linux hostname 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64 GNU/Linux`

    - podman version 3.0.1, installed through the distro's package manager.

CPU architecture

x86-64 and aarch64

Docker creation

git clone https://github.com/linuxserver/docker-smokeping.git
cd docker-smokeping
sudo docker build --no-cache --pull --progress=plain -t lscr.io/linuxserver/smokeping:latest .
sudo docker save docker.io/linuxserver/smokeping:latest > smokeping-latest-archive
podman pull docker-archive:/home/user/docker-smokeping/smokeping-latest-archive

Container logs

The container does not run.

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[Iksas]

Any local version of smokeping I build with docker 27.2.0 fails when trying to pull it with podman. Even version 2.8.2-r3-ls105, which works when pulling it from the official server.

This looks like a bug with docker 27.2.0 to me. Here's the timeline:

• 2024-08-20: smokeping ls105 is released and works
• 2024-08-27: docker 27.2.0 is released
• 2024-09-03: smokeping ls106 is released and doesn't work
• 2024-09-05: smokeping ls105 fails to build with docker 27.2.0

I'll try to build smokeping ls106 with the older docker 27.1.2. If it works, this bug is caused by some issue with docker 27.2.0.

[Iksas]

Ok, looks like the problem is caused by PPI-1.179, which was released on 2024-08-23. It looks like the files of this package are extracted with the wrong user/group.

Older, working version:

sudo docker run -it lscr.io/linuxserver/smokeping:2.8.2-r3-ls105 bash -c 'ls -la /root/.cpanm/work/1724186067.247'
[...]
drwxrwxr-x  6 1000 users     4096 Aug 20 20:36 PPI-1.278
-rw-r--r--  1 root root    248797 Mar 11 02:22 PPI-1.278.tar.gz
[...]

Newest, broken version:

sudo docker run -it lscr.io/linuxserver/smokeping:latest bash -c 'ls -la /root/.cpanm/work/1725395849.247'
[...]
drwxrwxr-x  6 197609  197609   4096 Sep  3 20:38 PPI-1.279
-rw-r--r--  1 root   root    243582 Aug 23 14:05 PPI-1.279.tar.gz
[...]

The PPI-1.279 tarball is fetched during the build process:

sudo docker build --no-cache --pull --progress=plain -t lscr.io/linuxserver/smokeping:local-build .
[...]
#8 699.4 --> Working on PPI
#8 699.4 Fetching http://www.cpan.org/authors/id/M/MI/MITHALDU/PPI-1.279.tar.gz ... OK
#8 699.9 Configuring PPI-1.279 ... OK
[...]
#8 740.6 Building and testing PPI-1.279 ... OK
#8 970.2 Successfully installed PPI-1.279
[...]

[thespad]

I'm generally reluctant to add workarounds for weird quirks like this because they end up hanging around forever, but the simplest fix is probably just to cleanup the /root/.cpanm folder at the end of the build.

Can you give the build from #182 a try once it completes and see if it fixes your issue.

[Iksas]

Thanks, the build from #182 works fine.

I've also opened an issue in the upstream repo to address the root cause of the problem.