Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stats: build from source #333546

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

stats: build from source #333546

wants to merge 1 commit into from

Conversation

DontEatOreo
Copy link
Member

Description of changes

@NixOS/darwin-maintainers Hey folks! I need a bit of helping hand, I think I almost have everything done for stats to be build from source but I'm struggling with an error that I have no idea how to fix

Build log: 
  • $ nix-build -A stats
...
Running phase: unpackPhase
unpacking source archive /nix/store/az200p1ssqh25k1flyfrvcsfxq5sgh9d-source
calling 'unpackCmd' function hook '_defaultUnpack' /nix/store/az200p1ssqh25k1flyfrvcsfxq5sgh9d-source
source root is source
calling 'postUnpack' function hook '_updateSourceDateEpochFromSourceRoot'
Running phase: patchPhase
applying patch /nix/store/1f188qaadrz743a65n5y7lkfxhms81gc-0001-Fixup-Makefile.patch
patching file Makefile
evaling implicit 'postPatch' string hook
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: configurePhase
calling 'preConfigure' function hook '_multioutConfig'
no configure script, doing nothing
Running phase: buildPhase
build flags: SHELL=/nix/store/fyjay93q3dq2hx3dhx7zhr8kyjnkh9m8-bash-5.2p26/bin/bash
Actual version is: 582
Next version is: 583
Exporting application archive...
warning: destination option not implemented
warning: object version 54 may be unsupported
warning: unhandled Project key packageReferences
warning: unhandled ShellScriptBuildPhase key alwaysOutOfDate
warning: unhandled ShellScriptBuildPhase key inputFileListPaths
warning: unhandled ShellScriptBuildPhase key outputFileListPaths
warning: unhandled ShellScriptBuildPhase key alwaysOutOfDate
warning: unhandled ShellScriptBuildPhase key inputFileListPaths
warning: unhandled ShellScriptBuildPhase key outputFileListPaths
warning: unhandled NativeTarget key packageProductDependencies
warning: unhandled NativeTarget key packageProductDependencies
warning: unhandled NativeTarget key packageProductDependencies
Build settings from the command line:
    DERIVED_DATA_DIR=.
    SDKROOT=MacOSX11.0

=== BUILD TARGET Kit OF PROJECT Stats WITH CONFIGURATION Release ===

Check dependencies
warning: unable to find Swift libraries

Write auxiliary files
/bin/mkdir -p ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit-own-target-headers.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit-all-target-headers.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit-all-non-framework-target-headers.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit-generated-files.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Kit-project-headers.hmap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/module.modulemap
/bin/mkdir -p ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit-OutputFileMap.json
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/unextended-module.modulemap
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/unextended-module-overlay.yaml
write-file ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.LinkFileList

Create product structure

CpHeader ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Products/Release/Kit.bundle/Contents/PrivateHeaders/Kit.h '/private/tmp/nix-build-stats-2.11.5.drv-0/source/Kit/Supporting Files/Kit.h'
    cd /private/tmp/nix-build-stats-2.11.5.drv-0/source
    export DEVELOPER_DIR=/nix/store/60n185lykh89xmvmjl4nd6difsfa4bi3-xcodebuild-0.1.2-pre
    export PATH=/nix/store/kdiiaab1y56f0a6q3417wvb9ypl0wv2w-Platforms/MacOSX.platform/Developer/usr/bin:/nix/store/kdiiaab1y56f0a6q3417wvb9ypl0wv2w-Platforms/MacOSX.platform/usr/local/bin:/nix/store/kdiiaab1y56f0a6q3417wvb9ypl0wv2w-Platforms/MacOSX.platform/usr/bin:/nix/store/kdiiaab1y56f0a6q3417wvb9ypl0wv2w-Platforms/MacOSX.platform/usr/local/bin:/nix/store/3lbn1q3bbyxmfa59zwxk8g8952nl132p-Toolchains/XcodeDefault.xctoolchain/usr/bin:/nix/store/3lbn1q3bbyxmfa59zwxk8g8952nl132p-Toolchains/XcodeDefault.xctoolchain/usr/libexec:/nix/store/60n185lykh89xmvmjl4nd6difsfa4bi3-xcodebuild-0.1.2-pre/usr/bin:/nix/store/60n185lykh89xmvmjl4nd6difsfa4bi3-xcodebuild-0.1.2-pre/usr/local/bin:/nix/store/60n185lykh89xmvmjl4nd6difsfa4bi3-xcodebuild-0.1.2-pre/Tools:/nix/store/ncswzgmmm16xfckcrj9k57jx1yfp326y-ditto-impure-darwin/bin:/nix/store/ajjyja98l3xa05w6rvrzi14arj23g4sd-create-dmg-1.2.2/bin:/nix/store/60n185lykh89xmvmjl4nd6difsfa4bi3-xcodebuild-0.1.2-pre/bin:/nix/store/3lbn1q3bbyxmfa59zwxk8g8952nl132p-Toolchains/XcodeDefault.xctoolchain/bin:/nix/store/wsgvg07q93xmaqp6ln58kzr75rg6jm40-clang-wrapper-16.0.6/bin:/nix/store/xdw9w5y9xdjways9xzs7r02zdmky4bim-clang-16.0.6/bin:/nix/store/7k0qi2r54imwjfs2bklg7fv0mn5jglil-coreutils-9.5/bin:/nix/store/58jqw1w9kj3xm10jfgpr7n9rkvx3p2cy-cctools-binutils-darwin-wrapper-1010.6/bin:/nix/store/5nwhbvfxfhdz9192ihsjpqbsr5vvhbsk-cctools-binutils-darwin-1010.6/bin:/nix/store/y01yv3ylb4zyadnh8cmpv3h8j0vw9slx-findutils-4.10.0/bin:/nix/store/pj9qh3y52knawg34qfnk9nq1kk12lh48-diffutils-3.10/bin:/nix/store/0x1mwb2bl2mjm658kcciwyg7n5avzvfp-gnused-4.9/bin:/nix/store/5g9xsw7dyihfh1kcf73nqqrla3zyy3r0-gnugrep-3.11/bin:/nix/store/ssprxb26mz01akbhlswdb7nbc30iyjjh-gawk-5.2.2/bin:/nix/store/lzndf97akrpmd3vjdq06xzwk8fnjc4j7-gnutar-1.35/bin:/nix/store/ynhzyabgbx6fz49sy944ws9wnskangxc-gzip-1.13/bin:/nix/store/ci1mxirfvn961zh3m4w9mr513ii7s2lz-bzip2-1.0.8-bin/bin:/nix/store/6gylp4vygmsm12rafhzvklrfkbhwwq40-gnumake-4.4.1/bin:/nix/store/fyjay93q3dq2hx3dhx7zhr8kyjnkh9m8-bash-5.2p26/bin:/nix/store/ga67f01j6g0qq7y9flgf3fzr1lbdi58y-patch-2.7.6/bin:/nix/store/6rk472mwpc5zwfhh6siyycl28y2baqrf-xz-5.6.2-bin/bin:/nix/store/apqqsrdg6xsz9icx04siw3l4z9yrxws3-file-5.45/bin
    builtin-copy -resolve-src-symlinks -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg /private/tmp/nix-build-stats-2.11.5.drv-0/source/Kit/Supporting Files/Kit.h ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Products/Release/Kit.bundle/Contents/PrivateHeaders

Ditto ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Products/Release/Kit.bundle/Contents/Modules/Kit.swiftmodule/arm64.swiftmodule ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.swiftmodule
    cd /private/tmp/nix-build-stats-2.11.5.drv-0/source
    /usr/bin/ditto -rsrc ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.swiftmodule ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Products/Release/Kit.bundle/Contents/Modules/Kit.swiftmodule/arm64.swiftmodule
ditto: Cannot get the real path for source './Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.swiftmodule'

** BUILD FAILED **

The following build commands failed:
    Ditto ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Products/Release/Kit.bundle/Contents/Modules/Kit.swiftmodule/arm64.swiftmodule ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.swiftmodule
(1 failure)
make: *** [Makefile:17: archive] Error 1

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@ofborg ofborg bot requested a review from emilytrau August 9, 2024 23:29
@DontEatOreo DontEatOreo marked this pull request as ready for review August 10, 2024 09:08
@DontEatOreo DontEatOreo requested a review from a team August 10, 2024 09:08
@matteo-pacini
Copy link
Contributor

@DontEatOreo have you tried adding swift as a nativeBuildInput?

@DontEatOreo
Copy link
Member Author

@DontEatOreo have you tried adding swift as a nativeBuildInput?

I will try it rn

@DontEatOreo
Copy link
Member Author

Unfortunately, that did not work (I got same error) but I added swift swiftPackages.swift-docc swiftPackages.Foundation to nativeBuildInputs and got a different error:

./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/DerivedSources/Kit-Swift.h ./Stats-bkfkssqxiuvzqubnwpwicsycxaul/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit-Swift.h

I also added export SWIFT_LIBRARY_PATH=${swift.swift.lib}/${swift.swiftModuleSubdir} which got rid of warning: unable to find Swift libraries

@matteo-pacini
Copy link
Contributor

@DontEatOreo You may also want to go through the Swift files in the project, and check which frameworks are required: https://github.com/search?q=repo%3Aexelban%2Fstats%20%20path%3A*.swift%20%22import%22&type=code (import statements at the top)

I can see plenty, and I have a feeling they need to be added via darwin.apple_sdk.frameworks

@DontEatOreo
Copy link
Member Author

@matteo-pacini I went and added all of them, but still same issue, weirdly enough every time I try to build it gives different error even if nothing was changed once will be

Ditto ./Stats-errkzjlgwieyrecnhkbyzyvumguw/Build/Products/Release/Kit.bundle/Contents/Modules/Kit.swiftmodule/arm64.swiftmodule ./Stats-errkzjlgwieyrecnhkbyzyvumguw/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit.swiftmodule

another time will be ./Stats-errkzjlgwieyrecnhkbyzyvumguw/Build/Intermediates/Stats.build/Release/Kit.build/DerivedSources/Kit-Swift.h ./Stats-errkzjlgwieyrecnhkbyzyvumguw/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/Kit-Swift.h

or CompileC ./Stats-errkzjlgwieyrecnhkbyzyvumguw/Build/Intermediates/Stats.build/Release/Kit.build/Objects-normal/arm64/lldb.o Kit/lldb/lldb.m normal arm64 objective-c++ com.apple.compilers.llvm.clang.1_0.compiler

These are the frameworks I added:

darwin.apple_sdk_11_0.frameworks.AppKit
darwin.apple_sdk_11_0.frameworks.Cocoa
darwin.apple_sdk_11_0.frameworks.CoreBluetooth
darwin.apple_sdk_11_0.frameworks.CoreWLAN
darwin.apple_sdk_11_0.frameworks.DeveloperToolsSupport
darwin.apple_sdk_11_0.frameworks.DiskArbitration
darwin.apple_sdk_11_0.frameworks.Foundation
darwin.apple_sdk_11_0.frameworks.IOBluetooth
darwin.apple_sdk_11_0.frameworks.IOKit
darwin.apple_sdk_11_0.frameworks.SensorKit
darwin.apple_sdk_11_0.frameworks.ServiceManagement
darwin.apple_sdk_11_0.frameworks.SwiftUI
darwin.apple_sdk_11_0.frameworks.SystemConfiguration
darwin.apple_sdk_11_0.frameworks.UserNotifications
darwin.apple_sdk_11_0.frameworks.WidgetKit

(I tried apple_sdk too no difference...)

Interestingly, even if I remove ditto from the Makefile, it still gets invoked...

@matteo-pacini
Copy link
Contributor

matteo-pacini commented Aug 12, 2024 •
edited
Loading

@DontEatOreo not sure what the problem is here, but plenty of Darwin apps I tried to compile from source failed because of xcbuild, which doesn't exactly work like Xcode's (https://github.com/facebookarchive/xcbuild, it's an archived project, last updated 5YA).

Could you please push your latest changes here so I can have a look?

It would also be great to mark this PR as a draft for the time being.

@DontEatOreo
Copy link
Member Author

DontEatOreo commented Aug 12, 2024 •
edited
Loading

@matteo-pacini Yeah, that's unfortunate. AFAIK we can't really use the Apple Xcode on Nix? I made the commands more "friendly" to Facebook's xcbuild, which doesn't have the archive feature but does have the build one. It's also more "friendly" to Nix since I got rid of Ditto.

@matteo-pacini
Copy link
Contributor

@DontEatOreo Bit of a bummer indeed, I had a similar issue here #320122, where I had to backtrack from source-based derivation to a bin-based one, as we cannot compile Metal shaders without Xcode (yet!).

Going to double check what happens later today, and come back to you!

@matteo-pacini
Copy link
Contributor

Tried the PR with a mix of changes (darwin frameworks, darwin.ditto also) - works in Xcode but fails on Nix :( perhaps someone from @NixOS/darwin-maintainers has a clue on how to fix this?

@jwiegley
Copy link
Contributor

I tried but was unable to build swift-5.8 using this commit, which is a dependency.

@DontEatOreo
Copy link
Member Author

As far as I can tell currently, it's more or less impossible to build stats from source code, so I'm going to put this PR in draft, and hopefully in the future, it would be possible to build it from source.

I want to do a final ping to @NixOS/darwin-maintainers. If anyone has experience with Xcode, I would appreciate a review. The main issue is that Ditto always fails on either Kit.swiftmodule, Kit-Swift.h, or some other file. Every rebuild is a gamble on what error you get, even with no changes made to the file.

Running the build command outside Nix with Apple's XCode produces a Stats.app ready to be used.

@DontEatOreo DontEatOreo marked this pull request as draft August 18, 2024 16:19
@n8henrie
Copy link
Contributor

I apologize that I haven't had time to look much at this (moving this month and I'm relative novice anyway), but just wanted to mention my experience adapting my beloved Notational Velocity.app to build with nix recently. I had no luck using xcbuild in spite of days of attempts, inconsistent errors similar to what you mention above. I eventually just exported the build log when building with xcode (something like xcodebuild clean; xcodebuild -configuration Deployment build |& tee xcbuild.log) and used this to find the actual clang calls that were beneath the XCode magic and eventually got this to compile.

Not sure if that is applicable given the swift and swiftmodule issues noted above, but can't hurt to mention the (painstaking) approach that eventually worked for me.

https://github.com/n8henrie/nv/blob/master/default.nix

@afh
Copy link
Member

afh commented Aug 19, 2024

Maybe this gets easier when the transition to new (darwin sdk) pattern is complete?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emilytrau emilytrau Awaiting requested review from emilytrau

Assignees
No one assigned
Labels
10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 0 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@DontEatOreo @matteo-pacini @jwiegley @n8henrie @afh