Add buildDubPackage and dub-to-nix for building dub based packages #299618

TomaSajt · 2024-03-28T00:38:02Z

Description of changes

This PR adds three main things:

the buildDubPackage helper function
the dub-to-nix helper script
the makeDubDep utility function to be used inside the files generated by dub-to-nix

The idea first came up in #288841 where I ended up implementing it. I though it would be a better idea to separate those changes into different PR, which is this one.

I also ended up migrating all 2 packages that use dub inside of nixpkgs.
Literate didn't even have anything to lock with dub-to-nix, so it has an empty dependency list.
The changes done to btdu is much more visible, though. The package declaration became much shorter, even when it only had 4 packages to lock.

I removed btdu's update script for now, maybe when I implement something like passthru.fetch-deps from buildDotnetModule for buildDubPackage I'll add a new one back.

I inited the serve-d package in this PR too to showcase that this works with many dependencies too.

Things done

Add a 👍 reaction to pull requests you find important.

Atemu

This looks like a fine first iteration.

Could you somehow make clear that you (and perhaps @jtbx too) maintain the dlang support? A comment on top of the files or even setting yourself as the codeowner.

Some smaller things and questions:

pkgs/build-support/dlang/dub-to-nix/default.nix

Atemu · 2024-03-28T03:35:10Z

pkgs/development/tools/literate-programming/Literate/default.nix


-  installPhase = "install -D bin/lit $out/bin/lit";
+  doCheck = true;


Not sure how the other derivation wrappers do this but if this is this generally desirable for dub packages, perhaps it could be enabled by default.

Atemu · 2024-03-28T03:35:40Z

pkgs/development/tools/literate-programming/Literate/default.nix

+  checkPhase = ''
+    runHook preCheck
+    dub test
+    runHook postCheck
+  '';


This would probably make sense to have in buildDubPackage.

I'll try to come up with a sane default once I get back home.

It turns out Literate didn't actually have any tests, but had a Makefile entry for testing. I'll try to build serve-d to test if the checking logic actually works.

Atemu · 2024-03-28T03:36:13Z

pkgs/development/tools/literate-programming/Literate/default.nix

+  installPhase = ''
+    runHook preInstall
+    install -Dm755 bin/lit -t $out/bin
+    runHook preInstall
+  '';


Is there no standardised install command for dub?

No, but I can file an issue asking about it. Such a feature would be useful outside of packaging anyway.

There exist an optional targetPath value inside dub.sdl or dub.json, AFAICT defaulting to ./.. It's where it puts the final binaries and the files inside copyPath.

Perhaps some jq based manipulation could be done to the file to add/update it and set it to a fixed directory, though I am not certain if this is robust enough.
(You can convert from .sdl to .json with a dub command, so jq could be used in either case)

alternatively this could be an argument to buildDubPackage

jtbx · 2024-03-28T04:45:00Z

Could you somehow make clear that you (and perhaps @jtbx too) maintain the dlang support? A comment on top of the files or even setting yourself as the codeowner.

I would be happy to :)

TomaSajt · 2024-03-28T15:42:38Z

Added a CODEOWNERS entry for D

By the way, what should we call this language internally? I am using dlangfor build-support, because IMO d is way too short.

Atemu · 2024-03-28T16:10:45Z

Ugh, I just remembered an annoying limitation of GH CODEOWNER: You two must have commit access.

I'd say add a comment in the files instead.

TomaSajt · 2024-03-28T16:12:32Z

Ugh, I just remembered an annoying limitation of GH CODEOWNER: You two must have commit access.

I'd say add a comment in the files instead.

That's a bit annoying, yes. I'll add some comments, then.

TomaSajt · 2024-03-28T16:46:18Z

I added some extra builder options for makeDubPackage. Previously it was only dubFlags, but now we have dubFlags dubBuildFlags and dubTestFlags.
I also added combined*Deps to factor out most of the string interpolation part from the script definition.

I created pkgs/build-support/dlang/README.md with a list of maintainers.

TomaSajt · 2024-03-28T17:26:03Z

Okay, so I set doCheck to false by default instead of true, because most applications don't have proper testing, and the fallback action when running dub test without proper unittests being set up is not very desirable. People will have to opt-in to testing by setting doCheck = true;.

TomaSajt · 2024-03-30T15:48:29Z

is there a way to tell dub to respect $NIX_BUILD_CORES ?

there is --build-mode=singleFile --parallel which will separate the workload onto multiple cores, however we can't specify the exact number of cores.

Edit: after testing, it doesn't really make a difference, or just makes it worse (at least for me). If people really want to use this, they can just use dubFlags

TomaSajt · 2024-03-30T18:42:15Z

Okay, I updated #288841 to the new lockfile format, here's how you could do extra dub dependencies:
https://github.com/NixOS/nixpkgs/blob/dae2ee5c84d647add18220d5dbf14a4d8784cfd3/pkgs/applications/misc/inochi2d/generic.nix#L62-L73

TomaSajt · 2024-03-30T19:32:36Z

Okay, I added both release notes and docs for the things added by this PR (mostly based off of the nim section).

Run nix-build nixos/release.nix -A manual.x86_64-linux for the release notes.
Run nix-build doc for the docs.

symphorien

thank you for the documentation! I left a few comments

doc/languages-frameworks/dlang.section.md

symphorien · 2024-03-30T19:53:22Z

doc/languages-frameworks/dlang.section.md

+* `dubTestFlags ? []`: The flags to pass to `dub test`.
+* `compiler ? ldc`: The D compiler to be used by `dub`.
+
+## Lockfiles {#dub-lockfiles}


I think this section should go before the example. There are two steps: 1/generate the lock file, 2/ write nix code like in this example

I just swapped the lockfile and the parameters part around.
I think new the comment above dubLock in the example is enough indication of what to read next.

doc/languages-frameworks/dlang.section.md

Atemu

Result of nixpkgs-review pr 299618 run on x86_64-linux 1

2 packages blacklisted:

nixos-install-tools
tests.nixos-functions.nixos-test

6 packages built:

Literate
btdu
dub-to-nix
serve-d
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference

Successfully ran btdu.

Great work! I especially like the docs :)

Atemu · 2024-04-04T06:11:47Z

pkgs/development/tools/literate-programming/Literate/dub-lock.json

+{
+  "dependencies": {}
+}


This could be inlined or even just be the default of buildDubPackage.

It could be inlined, but I don't think many packages have 0 dependencies. If we have a default, it won't show a missing attribute error, which is worse IMO.

TomaSajt · 2024-04-04T10:35:09Z

Thanks for all the help and reviews, everyone!
This has probably been my most complex contribution to date, glad to see it get merged.

I'll get back cleaning up my other PR now.

ldc: add meta.mainProgram

e726700

github-actions bot added the 6.topic: fetch label Mar 28, 2024

TomaSajt mentioned this pull request Mar 28, 2024

inochi-creator: init at 0.8.4, inochi-session: init at 0.8.3 #288841

Merged

13 tasks

TomaSajt requested a review from jtbx March 28, 2024 00:41

jtbx approved these changes Mar 28, 2024

View reviewed changes

ofborg bot added the 8.has: package (new) label Mar 28, 2024

ofborg bot requested a review from AtilaSaraiva March 28, 2024 01:02

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Mar 28, 2024

Atemu approved these changes Mar 28, 2024

View reviewed changes

wegank added the 12.approvals: 1 label Mar 28, 2024

TomaSajt force-pushed the dub-support branch from 2653075 to e3fc707 Compare March 28, 2024 10:55

wegank removed the 12.approvals: 1 label Mar 28, 2024

TomaSajt force-pushed the dub-support branch 2 times, most recently from 05593ac to 9b22fc4 Compare March 28, 2024 15:37

github-actions bot added the 6.topic: policy discussion label Mar 28, 2024

TomaSajt force-pushed the dub-support branch from 9b22fc4 to 8854636 Compare March 28, 2024 15:52

TomaSajt force-pushed the dub-support branch from 8854636 to b048f1c Compare March 28, 2024 16:42

github-actions bot removed the 6.topic: policy discussion label Mar 28, 2024

TomaSajt force-pushed the dub-support branch from b048f1c to b34fe06 Compare March 28, 2024 16:49

ofborg bot removed the 10.rebuild-darwin: 1 label Mar 28, 2024

TomaSajt force-pushed the dub-support branch from b34fe06 to 7bc4c41 Compare March 28, 2024 17:24

imrying mentioned this pull request Mar 29, 2024

dlang dstep: init at 1.0.4 #299797

Merged

13 tasks

TomaSajt force-pushed the dub-support branch from 1556c49 to 2417530 Compare March 30, 2024 15:37

github-actions bot added 8.has: documentation 6.topic: nixos 8.has: changelog labels Mar 30, 2024

symphorien reviewed Mar 30, 2024

View reviewed changes

TomaSajt force-pushed the dub-support branch 5 times, most recently from 7546833 to e521bec Compare March 30, 2024 20:52

TomaSajt added 6 commits April 3, 2024 12:32

buildDubPackage, dub-to-nix: init

92c8f64

doc: add dlang language section

87b3dc5

nixos/doc: add release notes for buildDubPackage

75b5079

Literate: use buildDubPackage

1557b33

btdu: use buildDubPackage

f0eba88

serve-d: init at 0.7.6

b7efb31

TomaSajt force-pushed the dub-support branch from e521bec to b7efb31 Compare April 3, 2024 10:32

symphorien approved these changes Apr 3, 2024

View reviewed changes

TomaSajt mentioned this pull request Apr 3, 2024

Package request: serve-d #301302

Closed

wegank added the 12.approvals: 1 label Apr 4, 2024

Atemu approved these changes Apr 4, 2024

View reviewed changes

Atemu merged commit b136700 into NixOS:master Apr 4, 2024
31 checks passed

TomaSajt mentioned this pull request Apr 4, 2024

luneta: init at 0.7.4 #301615

Merged

13 tasks

jtbx mentioned this pull request Apr 5, 2024

ldc: 1.36.0 -> 1.38.0, clean up #293417

Merged

13 tasks

TomaSajt deleted the dub-support branch April 21, 2024 20:10

Atemu mentioned this pull request May 6, 2024

lixgame: init at 0.10.9 #228934

Open

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add buildDubPackage and dub-to-nix for building dub based packages #299618

Add buildDubPackage and dub-to-nix for building dub based packages #299618

TomaSajt commented Mar 28, 2024 •

edited

Loading

Atemu left a comment

Atemu Mar 28, 2024

Atemu Mar 28, 2024

TomaSajt Mar 28, 2024

TomaSajt Mar 28, 2024

Atemu Mar 28, 2024

jtbx Mar 28, 2024 •

edited

Loading

TomaSajt Mar 28, 2024

symphorien Mar 29, 2024

jtbx commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

Atemu commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

TomaSajt commented Mar 28, 2024 •

edited

Loading

TomaSajt commented Mar 30, 2024 •

edited

Loading

TomaSajt commented Mar 30, 2024

TomaSajt commented Mar 30, 2024 •

edited

Loading

symphorien left a comment

symphorien Mar 30, 2024

TomaSajt Mar 30, 2024

Atemu left a comment

Atemu Apr 4, 2024

TomaSajt Apr 4, 2024

TomaSajt commented Apr 4, 2024


		installPhase = "install -D bin/lit $out/bin/lit";
		doCheck = true;

Add buildDubPackage and dub-to-nix for building dub based packages #299618

Add buildDubPackage and dub-to-nix for building dub based packages #299618

Conversation

TomaSajt commented Mar 28, 2024 • edited Loading

Description of changes

Things done

Atemu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtbx Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtbx commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

Atemu commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

TomaSajt commented Mar 28, 2024

TomaSajt commented Mar 28, 2024 • edited Loading

TomaSajt commented Mar 30, 2024 • edited Loading

TomaSajt commented Mar 30, 2024

TomaSajt commented Mar 30, 2024 • edited Loading

symphorien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Atemu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomaSajt commented Apr 4, 2024

TomaSajt commented Mar 28, 2024 •

edited

Loading

jtbx Mar 28, 2024 •

edited

Loading

TomaSajt commented Mar 28, 2024 •

edited

Loading

TomaSajt commented Mar 30, 2024 •

edited

Loading

TomaSajt commented Mar 30, 2024 •

edited

Loading