No podman directory created when enabling rootless configuration for non-privleged user

[rlubke]

• Oracle Linux Server release 9.5
• Linux 5.15.0-303.171.5.2.1.el9uek.x86_64 libpod create and run #2 SMP Mon Dec 23 14:43:10 PST 2024 x86_64 x86_64 x86_64 GNU/Linux
• Podman version 5.2.2

We have a set of VMs (all similarly configured with the above) configured for rootless testing of our code. Some of our jobs use docker compose and as such we point the DOCKER_HOST environment variable to the user socket /run/user/7658/podman/podman.sock. However, there are times where, after running systemctl --user enable --now podman.socket there is no podman directory for this user. podman info shows:

remoteSocket:
    exists: false
    path: /run/user/7658/podman/podman.sock

In other environments where things are working, exists is true.

Running systemctl --user status podman.socket shows:

podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: active (listening) since Tue 2025-01-07 19:22:15 UTC; 3h 12min ago
      Until: Tue 2025-01-07 19:22:15 UTC; 3h 12min ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/7658/podman/podman.sock (Stream)
     CGroup: /user.slice/user-7658.slice/user@7658.service/app.slice/podman.socket

Pulling containers, etc works fine, but we can't do anything with docker compose without being able to use the listen endpoint.

For grins, we re-imaged this machine and the first run of the job worked fine. The VM was turned off and returned to the pool. The next job run, using the same commands, fails because of the above. Looking for advice on how to troubleshoot this as it has occurred across multiple VM instances now.

One additional data point. Before pulling/running the images/containers, we run podman system reset -f prior to ensure everything is clean. Looking at the documentation, it appears this shouldn't be destructive in a way we wouldn't expect. The fact that it ran the first time seems to support this, but it was a recent change that was added and wanted to include that.

[eriksjolund]

Maybe you need to enable lingering?

sudo loginctl enable-linger <USERNAME>

See for example https://github.com/containers/podman/blob/main/troubleshooting.md#17-rootless-containers-exit-once-the-user-session-exits

[Luap99]

If systemctl --user status podman.socket works that should not matter as the systemd session is already running.

[Luap99]

Do you run something that removes the socket file? systemd will not notice if the socket file is removed and keeps reporting the socket as active.
You can try to restart the socket unit to see if systemd recreates the socket.