git immediately dies on aarch64-darwin #208951

Sciencentistguy · 2023-01-03T23:25:56Z

Describe the bug

Running git from nixpkgs master nix shell github:nixos/nixpkgs/master#git fails (it is sent signal 9 by the OS)

This appears to be a codesigning issue: (from the system log)

Termination Reason:    Namespace CODESIGNING, Code 2

Forcing nix to build git locally solves the problem: nix-shell --pure --run "git" with default.nix:

with import <nixpkgs> {}; let
  git = pkgs.git.overrideAttrs (old: {
      rebuildMe = true;
    });
in
  mkShell {
    buildInputs = [git];
  }

Steps To Reproduce

Steps to reproduce the behavior:

be on aarch64-darwin
run git
zsh: killed git

Expected behavior

git should work

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Notify maintainers

@NixOS/darwin-maintainers
@primeos
@wmertens
@globin

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.6.0, macOS 12.5.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.12.0`
 - channels(jamie): `""`
 - channels(root): `""`
 - nixpkgs: `/etc/nix/path/nixpkgs`

The text was updated successfully, but these errors were encountered:

kubukoz · 2023-01-04T01:34:54Z

Works for me:

$ nix run github:nixos/nixpkgs/master#git status                                                                                                                                                                                                                                                                 
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
$ nix run nixpkgs#nix-info -- -m                                                                                                                                                                                                                                                                                     
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.6.0, macOS 12.5.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.12.0`
 - channels(kubukoz): `"nixpkgs"`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/Users/kubukoz/.nix-defexpr/channels/nixpkgs`

tricktron · 2023-01-04T17:12:18Z

Works for me as well:

nix shell github:nixos/nixpkgs/master#git
git status
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
nix run nixpkgs#nix-info -- -m
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.6.0, macOS 12.6.2`
 - multi-user?: `yes`
 - sandbox: `relaxed`
 - version: `nix-env (Nix) 2.11.0`
 - channels(tricktron): `"darwin"`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I am using bash though instead of zsh.

mikroskeem · 2023-01-04T17:35:51Z

What git rev did your nix shell pick up? This will end up different on anyone who tests this later than you, making reproducing more complicated.

With rev a2d2f70 (where my <nixpkgs> points) I am unable to reproduce this.

mikroskeem · 2023-01-04T17:38:40Z

With rev f289c49 I'm unable to reproduce either.

 - system: `"aarch64-darwin"`
 - host os: `Darwin 22.3.0, macOS 13.2`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.11.1`
 - channels(mark): `""`
 - channels(root): `""`
 - nixpkgs: `/nix/store/kd83mlwxgj1w1a2r0rm2mn6ncylwgyf9-source`

Sciencentistguy · 2023-01-12T23:06:35Z

I first noticed it on 7ba7b45, and it still happens as of 92f9580

Sciencentistguy · 2023-01-12T23:13:57Z

Interestingly, it is fine on a2d2f70, but broken on f289c49

viraptor · 2023-01-12T23:20:01Z

I see various reports of this happening randomly to people in 10.15.4 and a few mentions of other versions with completely random apps. This may not be a nix issue specifically, but rather some mac weirdness. Does that happen on any other apps, or just git?

Sciencentistguy · 2023-01-12T23:22:49Z

@viraptor I've only noticed it with git, but that doesn't necessarily rule out it happening elsewhere.

kamalmarhubi · 2023-06-17T18:26:18Z

I've been seeing this for a while. Eg, current nixpkgs-unstable (7cc30fd):

$ nix --extra-experimental-features 'nix-command flakes' \
    run github:NixOS/nixpkgs/nixpkgs-unstable#git
Killed: 9

Metadata:

$ nix --extra-experimental-features 'nix-command flakes' \
    flake metadata github:NixOS/nixpkgs/nixpkgs-unstable
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/7cc30fd5372ddafb3373c318507d9932bd74aafe
Description:   A collection of packages for the Nix package manager
Path:          /nix/store/vv31n6dpig07j7jj4xbdllppklgp81jm-source
Revision:      7cc30fd5372ddafb3373c318507d9932bd74aafe
Last modified: 2023-06-17 01:20:35

According to codesign, the git binary is not signed:

$ codesign --verify --verbose \
    "$(nix --extra-experimental-features 'nix-command flakes' \
        path-info github:NixOS/nixpkgs/nixpkgs-unstable#git)/bin/git"
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git: code object is not signed at all
In architecture: arm64

In case relevant,

$ uname -a
Darwin kamal-FL932PQ21V 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:43 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T8112 arm64
$ sw_vers
ProductName:            macOS
ProductVersion:         13.4
BuildVersion:           22F66

and nix run github:NixOS/nixpkgs/nixpkgs-unstable#nix-info -- --markdown output:

system: "aarch64-darwin"
host os: Darwin 22.5.0, macOS 13.4
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.12.0
channels(root): "nixpkgs"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

mikroskeem · 2023-06-17T19:13:41Z

Cannot reproduce on my end right now either - git command runs fine.

Metadata (same, except last modified):

❯ nix --extra-experimental-features 'nix-command flakes' \
    flake metadata github:NixOS/nixpkgs/nixpkgs-unstable
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/7cc30fd5372ddafb3373c318507d9932bd74aafe
Description:   A collection of packages for the Nix package manager
Path:          /nix/store/vv31n6dpig07j7jj4xbdllppklgp81jm-source
Revision:      7cc30fd5372ddafb3373c318507d9932bd74aafe
Last modified: 2023-06-17 08:20:35

Code signature:

❯ codesign --verify --verbose \
    "$(nix --extra-experimental-features 'nix-command flakes' \
        path-info github:NixOS/nixpkgs/nixpkgs-unstable#git)/bin/git"
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git: valid on disk
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git: satisfies its Designated Requirement

System info:

❯ uname -a
Darwin phoebus 22.6.0 Darwin Kernel Version 22.6.0: Wed Jun  7 17:30:08 PDT 2023; root:xnu-8796.140.23.0.2~38/RELEASE_ARM64_T8103 arm64 arm Darwin
❯ sw_vers
ProductName:		macOS
ProductVersion:		13.5
BuildVersion:		22G5048d

nix-info:

system: "aarch64-darwin"
host os: Darwin 22.6.0, macOS 13.5
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.13.3
channels(mark): ""
channels(root): ""
nixpkgs: /nix/store/22m41xwqmzigqcspaf8bwjwljg3vpvpw-source

Sciencentistguy · 2023-06-17T19:35:37Z

I have not seen this issue again since i first reported it - at some point I forgot that I was running a pinned rev, and updated with no problems. I think @viraptor is right, in that it is somewhat random and it is mac weirdness.

@kamalmarhubi if you force nix to build from source (nix-build '<nixpkgs>' -A git --check) does the resulting binary work?

mikroskeem · 2023-06-17T21:05:31Z

After forcing a rebuild this time:

❯ /nix/store/viy8kdwngv0wa348mj9m44x683yj5dac-git-2.40.1/bin/git --version
zsh: killed     /nix/store/viy8kdwngv0wa348mj9m44x683yj5dac-git-2.40.1/bin/git --version
❯ codesign --verify --verbose /nix/store/viy8kdwngv0wa348mj9m44x683yj5dac-git-2.40.1/bin/git
/nix/store/viy8kdwngv0wa348mj9m44x683yj5dac-git-2.40.1/bin/git: invalid signature (code or signature have been modified)
In architecture: arm64

kamalmarhubi · 2023-06-18T15:01:49Z

@Sciencentistguy that command reports:

$ nix-build '<nixpkgs>' -A git --check
[... lots of build output ...]
error: derivation '/nix/store/cqafml8i3c03dhg58n576bq253n7vcyw-git-2.38.1.drv' may not be deterministic: output '/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1' differs

The binary at that store path works and codesign is happy with it:

$ /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git --version
git version 2.38.1
$ codesign --verify --verbose /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git
/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git: valid on disk
/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git: satisfies its Designated Requirement

I don't know if the binary at that path is the downloaded artifact or the locally built one, so I reran the command with --keep-failed:

$ nix-build '<nixpkgs>' -A git --check --keep-failed
[... lots of build output ...]
note: keeping build directory '/private/tmp/nix-build-git-2.38.1.drv-2'
error: derivation '/nix/store/cqafml8i3c03dhg58n576bq253n7vcyw-git-2.38.1.drv' may not be deterministic: output '/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1' differs from '/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1.check'

Checking those two store paths' git binaries:

$ /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git --version
git version 2.38.1
$ codesign --verify --verbose /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git
/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git: valid on disk
/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1/bin/git: satisfies its Designated Requirement

and the .check version:

$ /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1.check/bin/git --version
Killed: 9
$ codesign --verify --verbose /nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1.check/bin/git
/nix/store/2dmvb1q91cj3h528ma9mphm99bgmkyyh-git-2.38.1.check/bin/git: invalid signature (code or signature have been modified)
In architecture: arm64

I think this means the downloaded artifact is fine but the locally built one isn't?

Aside: I'm not where my <nixpkgs> is set since I only use flakes on this machine; nix-channel --list has no output. But apparently this is with a5f661b, which is from late December:

$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"23.05pre432461.a5f661b80e4"

I ran what I think is the flake equivalent command at the current nixpkgs-unstable (2c09e8e), which also complains of non-determinism:

$ nix --extra-experimental-features 'nix-command flakes' \
    build --cores 8 --rebuild --keep-failed github:NixOS/nixpkgs/nixpkgs-unstable#git
note: keeping build directory '/private/tmp/nix-build-git-2.40.1.drv-3'
error: derivation '/nix/store/l4gkv9n392bjka197ahp264khnc51z1y-git-2.40.1.drv' may not be deterministic: output '/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1' differs from '/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check'

But in this case, both outputs' git binaries are SIGKILLed:

$ /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git
Killed: 9
$ /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git
Killed: 9

Looking slightly closer, the (presumably?) downloaded one is unsigned (different from above), and the locally built one has an invalid signature (similar to above):

$ codesign --verify --verbose /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1/bin/git: code object is not signed at all
In architecture: arm64
$ codesign --verify --verbose /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git: invalid signature (code or signature have been modified)
In architecture: arm64

Not sure what to make of this? At some point artifacts in cache.nixos.org stopped being signed? Why are my local builds producing binaries with invalid signatures?

I don't know anything about how substitution works, so I can't figure out how to fetch the built artifact for an arbitrary nixpkgs-unstable and investigate more. Ideally I'd like to be able to reproduce the non-determinism and then bisect across nixpkgs-unstable for when it was introduced or something like that?

Sciencentistguy · 2023-06-18T15:31:28Z

A bisect may be useful, but I'm not certain it will tell us much - last time, it just came up with a rev that didn't touch git's closure at all.

Testing on my machine (2021 M1 Max MBP) with rev 2c09e8e gives the following:

The downloaded (substituted) binary works, and is signed:

nix build "github:nixos/nixpkgs/2c09e8e"#git && ./result/bin/git --version && codesign --verify --verbose ./result/bin/git
# <nix output cut...>
git version 2.40.1
./result/bin/git: valid on disk
./result/bin/git: satisfies its Designated Requirement

The locally built binary does not match - nix complains about determinism

nix build "github:nixos/nixpkgs/2c09e8e"#git --rebuild --keep-failed
note: keeping build directory '/private/tmp/nix-build-git-2.40.1.drv-1'
error: derivation '/nix/store/l4gkv9n392bjka197ahp264khnc51z1y-git-2.40.1.drv' may not be deterministic: output '/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1' differs from '/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check'

and is killed my macos and fails codesign:

/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git --version
zsh: killed     /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git --versio

codesign --verify /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git --verbose
/nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/bin/git: invalid signature (code or signature have been modified)
In architecture: arm64

Interestingly, I can't even strip the signature

cp /nix/store/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check /tmp
# <cp output cut...>
cd /tmp/r1f29k5x419hwmpjhysaqbv5zgjd9lmz-git-2.40.1.check/
codesign --remove-signature ./bin/git
./bin/git: internal error in Code Signing subsystem

kamalmarhubi · 2023-06-19T15:09:47Z

@Sciencentistguy that's unfortunate about the previous bisect. Do you know how I can force my system to get an artifact from cache.nixos.org? I'm still curious to know if there are situations where the cached artifact has an invalid or missing signature.

emilazy · 2023-07-29T06:57:37Z

See also: LnL7/nix-darwin#693 (in particular LnL7/nix-darwin#693 (comment), LnL7/nix-darwin#693 (comment)). This may be related to store optimization?

emilazy · 2023-07-29T06:59:00Z

I don't know if this is a bug with the Nixpkgs Git package. It seems to involve a derivation getting somehow replaced with one that is almost identical but subtly broken, perhaps a Nix bug involving substituters or store optimization (which is quite broken on macOS in general). I don't know where the broken "shadow derivation" is coming from, especially since we don't know its derivation hash, but at least in the case of the issue I linked it was fixed by repairing the store path.

Sciencentistguy · 2024-02-14T09:24:31Z

This is cropping up again for me, btw. The workaround of forcing it to rebuild still works.

@emilazy i don't seem to be able to run nix store repair <path>? It is failing with

error: operation 'repairPath' is not supported by store 'daemon'

EDIT: running nix store repair as root does the trick, and fixes this

When the linker signs a Mach-O binary, it sets a flag in the signature’s code directory indicating that the signature was generated by a linker. Tools such as `strip` and `install_name_tool` read this flag and will update ad hoc signatures after they perform their modifications. The updated l64 supports signing binaries automatically. Both the updated cctools and LLVM will check for the linker-signed flag and resign binaries they modify automatically when it’s present. Given that, use of postLinkSignHook is unnecessary and potentially harmful. In particular, if the hook is used and an unwrapped `strip` or `install_name_tool` is on the user’s path, they will not automatically update an ad hoc signature. Instead, they will issue a warning and create a binary with a broken signature. It is more robust to let the tools handled this since the only time a signature would not be linker-signed is when the user is manually invoking `codesign` (or another tool such as `sigtool` or `rcodesign`), which by nature of the invocation updates the signature to a valid one. Since `strip` no longer needs to be wrapped for code-signing, binutils-wrapper now uses the GNU strip wrapper on Darwin. Fixes NixOS#208951.

reckenrode · 2024-06-01T12:27:02Z

Sorry about the commit message spam. I believe this will be fixed by the ld64 upgrade in #307880. As part of the upgrade, the post-link signing hook is being dropped. ld64 will automatically handle code-signing, and cctools will automatically preserve linker-generated signatures (which ld64 creates).

When the linker signs a Mach-O binary, it sets a flag in the signature’s code directory indicating that the signature was generated by a linker. Tools such as `strip` and `install_name_tool` read this flag and will update ad hoc signatures after they perform their modifications. The updated l64 supports signing binaries automatically. Both the updated cctools and LLVM will check for the linker-signed flag and resign binaries they modify automatically when it’s present. Given that, use of postLinkSignHook is unnecessary and potentially harmful. In particular, if the hook is used and an unwrapped `strip` or `install_name_tool` is on the user’s path, they will not automatically update an ad hoc signature. Instead, they will issue a warning and create a binary with a broken signature. It is more robust to let the tools handled this since the only time a signature would not be linker-signed is when the user is manually invoking `codesign` (or another tool such as `sigtool` or `rcodesign`), which by nature of the invocation updates the signature to a valid one. Since `strip` no longer needs to be wrapped for code-signing, binutils-wrapper now uses the GNU strip wrapper on Darwin. Fixes NixOS#208951.

Sciencentistguy added 0.kind: bug 6.topic: darwin Running or building packages on Darwin labels Jan 3, 2023

Sciencentistguy added a commit to Sciencentistguy/nixfiles that referenced this issue Jan 12, 2023

Pin nixpkgs to avoid NixOS/nixpkgs#208951

26317d9

reckenrode mentioned this issue Jun 17, 2024

Build failure: python3Packages.curl-cffi on aarch64-darwin #320196

Closed

reckenrode mentioned this issue Jun 26, 2024

Fix Darwin build on staging-next #322388

Merged

13 tasks

vcunat closed this as completed in c922cb2 Jul 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

git immediately dies on aarch64-darwin #208951

git immediately dies on aarch64-darwin #208951

Sciencentistguy commented Jan 3, 2023 •

edited

Loading

kubukoz commented Jan 4, 2023

tricktron commented Jan 4, 2023

mikroskeem commented Jan 4, 2023 •

edited

Loading

mikroskeem commented Jan 4, 2023

Sciencentistguy commented Jan 12, 2023

Sciencentistguy commented Jan 12, 2023

viraptor commented Jan 12, 2023

Sciencentistguy commented Jan 12, 2023

kamalmarhubi commented Jun 17, 2023 •

edited

Loading

mikroskeem commented Jun 17, 2023 •

edited

Loading

Sciencentistguy commented Jun 17, 2023

mikroskeem commented Jun 17, 2023

kamalmarhubi commented Jun 18, 2023 •

edited

Loading

Sciencentistguy commented Jun 18, 2023 •

edited

Loading

kamalmarhubi commented Jun 19, 2023

emilazy commented Jul 29, 2023

emilazy commented Jul 29, 2023

Sciencentistguy commented Feb 14, 2024 •

edited

Loading

reckenrode commented Jun 1, 2024

git immediately dies on aarch64-darwin #208951

git immediately dies on aarch64-darwin #208951

Comments

Sciencentistguy commented Jan 3, 2023 • edited Loading

Describe the bug

Steps To Reproduce

Expected behavior

Screenshots

Additional context

Notify maintainers

Metadata

kubukoz commented Jan 4, 2023

tricktron commented Jan 4, 2023

mikroskeem commented Jan 4, 2023 • edited Loading

mikroskeem commented Jan 4, 2023

Sciencentistguy commented Jan 12, 2023

Sciencentistguy commented Jan 12, 2023

viraptor commented Jan 12, 2023

Sciencentistguy commented Jan 12, 2023

kamalmarhubi commented Jun 17, 2023 • edited Loading

mikroskeem commented Jun 17, 2023 • edited Loading

Sciencentistguy commented Jun 17, 2023

mikroskeem commented Jun 17, 2023

kamalmarhubi commented Jun 18, 2023 • edited Loading

Sciencentistguy commented Jun 18, 2023 • edited Loading

kamalmarhubi commented Jun 19, 2023

emilazy commented Jul 29, 2023

emilazy commented Jul 29, 2023

Sciencentistguy commented Feb 14, 2024 • edited Loading

reckenrode commented Jun 1, 2024

Sciencentistguy commented Jan 3, 2023 •

edited

Loading

mikroskeem commented Jan 4, 2023 •

edited

Loading

kamalmarhubi commented Jun 17, 2023 •

edited

Loading

mikroskeem commented Jun 17, 2023 •

edited

Loading

kamalmarhubi commented Jun 18, 2023 •

edited

Loading

Sciencentistguy commented Jun 18, 2023 •

edited

Loading

Sciencentistguy commented Feb 14, 2024 •

edited

Loading