Suddenly "unable to connect to Podman socket" on macOS

[command-tab]

I've been successfully using podman on macOS Sequoia for some time, even as recently as a few days ago, but today I went to run an image, and I was greeted with:

$ podman pull nginx:latest
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory

I thought perhaps my podman machine got hosed, so I ran a podman machine reset, podman machine init, and podman machine start and that succeeded in firing up a fresh podman machine. And yet, same error when trying to interact with the new machine.

podman system connection list seems OK:

Name                         URI                                                          Identity                                                      Default     ReadWrite
podman-machine-default       ssh://core@127.0.0.1:50290/run/user/5406/podman/podman.sock  /Users/me/.local/share/containers/podman/machine/machine  true        true
podman-machine-default-root  ssh://root@127.0.0.1:50290/run/podman/podman.sock            /Users/me/.local/share/containers/podman/machine/machine  false       true

It's not clear to me why my normal SSH key is involved as the error claims, since the identities listed above are not that. As well, ~/.ssh/id_ed25519 does exist, and has the properly locked-down 0600 permissions, so the error is not even accurate.

Any suggestions or pointers would be appreciated! Thanks!

• podman 5.3.0 via Homebrew
• macOS 15.1 (24B83) on M2 Pro hardware

[richardxia]

I'm still on macOS Sonoma (14.7.1 (23H222)), but I'm hitting the same issue, which just started happening after upgrading from podman 5.2 to podman 5.3.

[richardxia]

I've managed to debug this a bit further, and I at least have a workaround that lets me get my job done, even if I don't understand the root cause. Podman seems to be attempting to use the wrong SSH key, and forcing it to use the correct podman-specific one lets things work properly. This can be done by setting the CONTAINER_SSHKEY environment variable:

export CONTAINER_SSHKEY=~/.local/share/containers/podman/machine/machine

Once I set that variable, all my podman commands appear to be working correctly now.

Edit: What I originally wrote here doesn't actually work, but what actually fixed the issue for me was to edit my ~/.ssh/config file to comment out the IdentityFile ~/.ssh/id_ed25519 line that I had.

---

In case it's helpful, I can write down some of my notes from debugging this.

I believe the failed to read identity error message comes from this line:

podman/vendor/github.com/containers/common/pkg/ssh/connection_golang.go

Line 256 in dd97d3e

return nil, fmt.Errorf("failed to read identity %q: %w", value, err)

. I believe the error is due to the ~ being in the file path, and I'm assuming this code expects that ~ should have already been resolved to the user's home directory, which would explain why it believes that ~/.ssh/id_ed25519 is not a path because ~ is not a real directory.

As for how the path with the ~ got there, I think it's coming from this line,

podman/pkg/bindings/connection.go

Line 212 in dd97d3e

if val := cfg.Get(alias, "IdentityFile"); val != "" {

, which appears to read the user's ~/.ssh/config file, and in my personal file, I had the following:

Host *
  IdentityFile ~/.ssh/id_ed25519

Temporarily changing my config file to have the absolute path to my SSH key resolves the "no such file or directory" error, but this isn't the correct SSH key for podman to be using in the first place, since podman machine installs its own SSH key.

Running podman --help shows the following for the --identity flag:

--identity string           path to SSH identity file, (CONTAINER_SSHKEY) (default "/Users/rxia/.local/share/containers/podman/machine/machine")

It didn't seem like it was properly defaulting to the path that it listed there, but setting it explicitly with --identity or with the CONTAINER_SSHKEY environment variable seems to get things working again.

[eriksjolund]

This recent issue #24567 also mentions the error message

Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory

Maybe related? The issue 24567 already has a fix which will be released in Podman 5.3.1.

[command-tab]

This was super helpful! Thank you!

Temporarily removing the IdentityFile line makes podman work again. (As well, same here -- changing the un-expanded ~ key path to an absolute path had the same effect.)

Curiously, passing in --identity like podman --identity /Users/me/.local/share/containers/podman/machine/machine pull nginx:latest still prompts me for my ~/.ssh/id_ed25519 key passphrase. However, adding the following to the top of ~/.ssh/config worked:

Host 127.0.0.1
  IdentityFile /Users/me/.local/share/containers/podman/machine/machine

That's probably hacky, but it's enough to let ssh keys and podman work at the same time as before.

[fpigeonjr]

can confirm the fix for #24567 worked for me as seen here

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

[zakkg3]

In my case, the SSH fingerprint changed, and '[127.0.0.1]:60221' (ED25519) was not on the list of known hosts. By accepting the new fingerprint, the problem was gone.
So to test this try to ssh into the Podman machine:

ssh core@127.0.0.1 -p 60221                                                                                                                                          125 ↵ 
The authenticity of host '[127.0.0.1]:60221 ([127.0.0.1]:60221)' can't be established.
ED25519 key fingerprint is SHA256:Qi..Hntk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:60221' (ED25519) to the list of known hosts.
Fedora CoreOS 40.20241019.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

core@192-168-127-2:~$ exit
logout
Connection to 127.0.0.1 closed.

podman works as usual now.