Clang link time optimization not working on OSX

[dipinhora]

Issue description

The example for Clang's Link Time Optimization (http://llvm.org/docs/LinkTimeOptimization.html#example-of-link-time-optimization) doesn't work in nix on OSX. I tested with clang versions 3.8.0, 3.7.1, and 3.6.2. I wasn't able to test with clang 3.9.0 because of an error compiling llvm-3.9.0.

It fails with the following on OSX with Clang 3.7.1:

[nix-shell:~/tmp]$ clang --version
clang version 3.7.1 (tags/RELEASE_371/final)
Target: x86_64-apple-darwin14.5.0
Thread model: posix
[nix-shell:~/tmp]$ clang -flto a.o main.o -o main
ld: warning: ignoring file a.o, file was built for unsupported file format ( 0xDE 0xC0 0x17 0x0B 0x00 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0xFC 0x06 0x00 0x00 ) which is not the architecture being linked (x86_64): a.o
Undefined symbols for architecture x86_64:
  "_foo1", referenced from:
      _main in main.o
ld: symbol(s) not found for architecture x86_64
clang-3.7: error: linker command failed with exit code 1 (use -v to see invocation)

It fails with the following on OSX with Clang 3.8.0:

[nix-shell:~/tmp]$ clang --version
clang version 3.8.0 (tags/RELEASE_380/final)
Target: x86_64-apple-darwin14.5.0
Thread model: posix
InstalledDir: /nix/store/w440mfm2abkxyaw7j86pmhbkj680118d-clang-3.8.0/bin
[nix-shell:~/tmp]$ clang -flto a.o main.o -o main
clang-3.8: warning: libLTO.dylib relative to clang installed dir not found; using 'ld' default search path instead
ld: warning: ignoring file a.o, file was built for unsupported file format ( 0xDE 0xC0 0x17 0x0B 0x00 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x94 0x09 0x00 0x00 ) which is not the architecture being linked (x86_64): a.o
Undefined symbols for architecture x86_64:
  "_foo1", referenced from:
      _main in main.o
ld: symbol(s) not found for architecture x86_64
clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)

Steps to reproduce

On an OSX machine:
Install nix: curl https://nixos.org/nix/install | sh
Run (for clang 3.8.0): nix-shell --pure -p llvmPackages_38.clang
Run (for clang 3.7.1): nix-shell --pure -p llvmPackages_37.clang
In shell: Follow instructions at http://llvm.org/docs/LinkTimeOptimization.html#example-of-link-time-optimization to test link time optimization with the example code.

Technical details

• System: Mac OSX 10.10.5; Output of uname -a:

Darwin Dipins-MBP.home 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

• Nix version: nix-env (Nix) 1.11.4
• Nixpkgs version: "17.03pre91937.5cda2be"

[copumpkin]

I seem to remember that the cctools we're using has the LTO option turned off. I tried turning it on a while back and ran into issues. Perhaps it'll work nowadays but I can't look for a while.

[dipinhora]

@copumpkin thanks for the quick response.

I'm more than happy to try and get cctools working with LTO if you can point me in the right direction.

[copumpkin]

I think you'll want to take the cctools-port expression and make sure to pass --enable-lto-support in its configureFlags. Unfortunately modifying that expression is painful because it'll cause you to rebootstrap your entire stdenv, so I'd probably duplicate the file, import the new file under a different name, then tinker with the build flags until you get it working. Then comes the fun of trying to get it integrated with the bootstrap, which may or may not be straightforward.

Also keep in mind that both darwin and linux use cctools-port, so please make sure the linux build still works (or call out that you can't and see if you can get someone else to test it)

[dipinhora]

Thanks for the pointers. I'll see if I can make any progress. I'll test on linux also to prevent breaking it.

[Profpatsch]

(triage) fixed?

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-system-and-stdenv-hostplatform-features/9745/1

[ttuegel]

This is still a problem.

[stale]

I marked this as stale due to inactivity. → More info

[ttuegel]

This is still an issue.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fails-to-link-on-arm64-only-when-built-through-nix/23592/2

[rkitover]

This is still broken, in clang15Stdenv as well.

[tobim]

There are 2 tools that need to be adjusted to make things work:

• It seems that the default linker on darwin does not understand lto objects, this can be worked around by using the compiler driver with -fuse-ld=lld.
• The ar tool from llvm changes behavior when it is called as just ar, as it is done in nixpkgs. In this case the lto objects aren't archived correctly and won't have any symbols. We actually need to invoke it as llvm-ar to get it to work correctly.

Regarding lld, I'm not sure why it is not default anyways, maybe someone here know more?
Regarding ar, the real binary is actually named <targetprefix>-llvm-ar, but it is called through a chain of symlinks, and none of those have the llvm component in their name.

[reckenrode]

Regarding lld, I'm not sure why it is not default anyways, maybe someone here know more?

lld isn’t a drop-in replacement for ld64. It doesn’t work with xcbuild or Qt’s build system (and probably others). I tried linking the Darwin stdenv with it, but it wasn’t even able to do that. I plan to try again with the clang 18 update next year, but it doesn’t look like much has changed so far upstream.

[multivac61]

Affecting our project too. What if anything can be done to fix this issue? I am not technically apt enough in this area but would be willing to fund the effort to get this fixed (if on the order of $100).

[reckenrode]

I’m working on updates to libtapi, cctools, and ld64 to the latest versions from Apple’s OSS releases. I currently have libtapi and ld64 building with working LTO in ld64.

$ nix run .#ld64 -- -v
@(#)PROGRAM:ld PROJECT:ld64-951.9
BUILD 00:00:00 Jan  1 1980
configured to support archs: i386 x86_64 x86_64h armv6 armv7 armv7s armv7m armv7k arm64 arm64e arm64_32
LTO support using: LLVM version 16.0.6 (static support for 29, runtime is 29)
TAPI support using: TAPI version 15.0.0 (tapi-1500.0.12.3)
$ nix run .#clang -- -flto -fuse-ld=$(nix path-info .#ld64)/bin/ld test.c -o test
# exit status: 0

Note that Apple’s upstream sources do not build on Linux. I am building ld64 (and plan to build cctools the same way) with Meson instead of the Xcode projects. I’ve also been replacing private API usage with portable APIs. I’m doing that with an eye towards restoring support for Linux (and I have limited patience for dealing with xcbuild failures).

I should have a PR open against later this week. There is a clang piece needed for LTO. It’s passing the wrong path to ld64 (via -lto_library), which doesn’t fall back to its default LTO if libLTO.dylib does not exist at that path.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-updates-news/42249/10

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-updates-news/42249/16

[emilazy]

Shall we close this now that it’s in staging?

[reckenrode]

Closed via #307880.