Notes on reproducible builds

There are a few issues that prevent reproducible builds:

• Golang includes paths of the stdlib packages used (in GOROOT)
• Golang includes paths of libraries used (in GOPATH)
• If CGO is used (which we do via miekg/pkcs11 and the stblib net package) then the host linker will generally include the paths of linked objects used to build the binary
• The host linker includes a build ID in binaries

There are two ways of going about fixing these this:

• Using a container or specific building tree to replicate the remove the differences in the paths on the host systems used to build binaries, the main issue with this is that it generally isn't realistic. Either we have to distribute a container that replicates our staging/production build host, use that container on the build host, or just use a script or something that pulls all of the stuff we need into specific places on the host system in order to do the build
• Use a combination of relatively unknown Golang features in order to strip all of the offending stuff out of the binary such as GOROOT_FINAL=go go install -ldflags="-linkmode=internal" -gcflags="-trimpath=$GOPATH/src" github.com/letsencrypt/boulder/cmd/boulder-ca:
  • GOROOT_FINAL allows rewriting the GOROOT path prefix in binaries to a static value
  • -gcflags="-trimpath=$GOPATH/src" allows stripping paths of libraries in the GOPATH down to just the import name. Unfortunately this overrides the usage of -trimpath in compile when building CGO objects in the temporary build directory
  • -ldflags="-linkmode=internal" uses the pure Golang linker instead of the host linker which doesn't include paths of the linked objects in the binary, this also doesn't include a build ID

I like the second option more but unfortunately it has one blocker: because we use CGO by using -gcflags="-trimpath=..." we get the path of temporary build directory (/tmp/go-buildXXX) included in the binary. Really -trimpath should support multiple arguments in an ideal world but it doesn't.