Build Infrastructure

[olynch]

This PR adds nix build infrastructure that:

• Builds narya in github CI
• Makes a development environment accessible to nix users via nix develop
• Produces statically-linked narya binaries via nix build that can be distributed to any linux user (not just nix users)

@mikeshulman, you might want to host tarballs of the statically-linked narya binaries on your website, that could make it a lot easier for people to run narya!

[mikeshulman]

Thanks for this! I appreciate it, but I need to think about it a bit and make sure I understand what it's doing, not just because I like to understand what's going on in my repository but because I'll need to know how it interacts with other things in the future.

• What is the benefit of using nix for building static binaries? Are they totally static or do they link dynamically with standard libraries that exist on every system?
• flake.nix is intimidating and hard for me to comprehend. Does it really have to be that long and complicated?
• It might help me if you can separate the opam changes, the nix configuration, and the github CI configuration into separate commits or pull requests. Is the opam configuration (which we should have eventually anyway, of course) required for the nix configuration for some reason?
• Why are generated files like narya.opam and (I think) flake.lock being version-controlled? Normally I expect that generated files are not version-controlled, only the "source" that generates them.
• The .envrc file doesn't seem like something that should be version-controlled either; isn't that more of an individual user preference?
• If distributing static binaries made from a development branch (which is all there is right now), I'd like them to be able to respond intelligently to a -version flag by indicating when they were built and/or from which git commit. Can that be incorporated automatically?

[mikeshulman]

• I don't really like the idea of patching the dune file every time we build a static binary; it seems brittle to future changes in that file. Is there some other way to do that? Perhaps having a separate dune build target for the static version?

[olynch]

Sorry for the delayed response, I didn't see you had responded. You raise some good points. In no particular order:

• I think that narya.opam and .envrc should be deleted; I will update this. The flake.lock file is good to keep for reproducability.
• Static binaries link with no system libraries, and thus should work modified on any linux system, including, for instance, systems which do not have gmp installed for zarith (it bundles the gmp into the binary). They also are independent of glibc version, which is a boon for binary distribution because glibc is only forward-compatible, so the traditional strategy for linux is to build your binary on the oldest version of linux that you want to support. Which is a pain.
• The complicated bits in the flake.nix should really be factored out, because I want to use the same infrastructure for statically building other ocaml packages. Or perhaps I should really commit some of the workarounds that I had to use upstream to opam-nix. I'll get back to you once I've done that.
• Incorporating the git hash into --version can definitely be done! I'll look into this.
• I will happily split out the commits!
• The patch to the dune file is indeed brittle. I'll look into whether this can be set with an environment variable.

[mikeshulman]

Thanks! Can you explain a bit more about the reproducibility benefit of version-controlling flake.lock?

(Or point me to a good explanation of it elsewhere, if this is a standard best practice.)

[olynch]

Keeping the flake lock in the repo is pretty standard, for instance see https://git.sr.ht/~jonsterling/ocaml-forester. A flake.nix file on its own doesn't guarantee reproducability because it doesn't specify how the inputs get resolved. Also see: https://www.tweag.io/blog/2020-05-25-flakes/.

[mikeshulman]

Thanks. Would it be correct to say that the lock file is not an output of compilation but rather part of the specification of a build, such as a makefile or a dune file, which we can morally think of as being created by hand rather than automatically -- it's just that it would be too tedious to actually write it by hand, and almost always when creating or updating it we want it to lock "the current version accessible to me right now", so there is an automated tool that does that for us?

I guess this adds an extra burden on the developer, to know when the lock file has to be updated and run the appropriate command. When is that?

[olynch]

Thanks. Would it be correct to say that the lock file is not an output of compilation but rather part of the specification of a build, such as a makefile or a dune file, which we can morally think of as being created by hand rather than automatically -- it's just that it would be too tedious to actually write it by hand, and almost always when creating or updating it we want it to lock "the current version accessible to me right now", so there is an automated tool that does that for us?

I guess this adds an extra burden on the developer, to know when the lock file has to be updated and run the appropriate command. When is that?

Essentially, when we wish to update dependencies! That is, you will always get the same version of the ocaml compiler/same versions of dependencies until you update the lock file. So if you are happy with that, then you never need to update the lock file. But if you want a newer compiler/newer libraries when they come out, then one must update the lock file.

[mikeshulman]

Ok. I guess building the static binary in GitHub CI will help with that, by failing if the code changes to require a newer compiler or libraries but the lock file didn't get updated.

[olynch]

Superseded by #29, #30, #31.