Improve Build Reproducibility for Core Admin Client #154

egbedo · 2024-10-15T14:03:30Z

Description of Changes
This pull request aims to address reproducibility issues identified in the core-admin-client builds for Qubes OS. The key changes involve normalizing timestamps and other environment-dependent values that impacted the checksum and metadata consistency between builds.

Modifications Made
Environment Variable for Timestamps
Added $SOURCE_DATE_EPOCH to replace direct date calls across several files:

Files Updated:
plugins/build_rpm/scripts/rpmbuildinfo
plugins/installer/Makefile
plugins/template/Makefile
Description:
Replaced all instances of direct date commands with $SOURCE_DATE_EPOCH to ensure the builds use a consistent timestamp. This prevents new timestamps from being generated each time the build runs.

RPM Metadata Consistency
Updates to RPM metadata generation scripts to ensure deterministic checksums:

Files Updated:
_plugins/source_rpm/scripts/generate-changelog
plugins/source_rpm/scripts/clamp-changelog-entry-dat_e
Description:
These changes reset the metadata timestamps to $SOURCE_DATE_EPOCH, ensuring identical metadata across builds.
File Permission Normalization

Files Updated:
plugins/build_rpm/scripts/rpmbuildinfo
Description:
Standardized file permission handling within the scripts to ensure permissions are consistent between builds.

Testing Results
After these changes, two separate builds were created and compared using diffoscope. The results confirmed:

Identical Checksums for files in the RPM packages.
Consistent Metadata without varying timestamps or permissions across builds.

marmarek · 2024-10-15T14:54:14Z

Please don't add your temporary files (backup-rpm-build dirs, html files etc)

codecov · 2024-10-16T21:05:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.62%. Comparing base (c302e71) to head (dfae88a).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #154   +/-   ##
=======================================
  Coverage   78.61%   78.62%           
=======================================
  Files          47       47           
  Lines        5382     5384    +2     
=======================================
+ Hits         4231     4233    +2     
  Misses       1151     1151

Flag	Coverage Δ
	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

marmarek

Besides the few comments below, I'll also run full test suite (default one runs only subset of tests). It seems like template and iso builds are broken by this change, but lets check.

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

new_src_header.txt

new_rpm_header.txt

.gitignore

qubesbuilder/plugins/installer/Makefile

qubesbuilder/plugins/template/Makefile

marmarek · 2024-10-17T10:50:05Z

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

+# Check if SOURCE_DATE_EPOCH is set; if not, use the current date
+if [ -z "$SOURCE_DATE_EPOCH" ]; then
+    CURRENT_DATE=$(date -u +"%Y-%m-%d %H:%M:%S")
+else
+    CURRENT_DATE=$(date -u -d @"$SOURCE_DATE_EPOCH" +"%Y-%m-%d %H:%M:%S")
+fi


This section is redundant now, you set CURRENT_DATE again below.

marmarek · 2024-10-17T10:50:21Z

.gitignore

+
+
+


Don't add empty lines either.

marmarek · 2024-10-17T10:52:21Z

I've started full tests here: https://gitlab.com/QubesOS/qubes-builderv2/-/pipelines/1500114091 it will take few hours to complete

HW42 · 2024-10-17T14:06:08Z

qubesbuilder/plugins/build_rpm/scripts/rpmbuildinfo

@@ -71,7 +71,7 @@ Version: $(rpm -qp --queryformat '%{version}-%{release}' "$SRPM")
 Architecture: $(rpm -qp --queryformat '%{arch}' "$SRPM")
 Binary: $(printf '%s' "$RPMS" | xargs rpm -qp --qf "%{name} ")
 Build-Origin: $(getos)
-Build-Date: $(date -R)
+Build-Date: $(date -d @"$SOURCE_DATE_EPOCH" -R)


The .buildinfo file is intended to document (to some reasonable extend) the build environment that was used for a specific build attempt. As currently implemented it is not limited to those parts of the build environment that we consider to be necessary to reproduce a build. As such recording a fake build date doesn't make sense.

If we want to move to only record parts that are necessary then the right action would be to drop this field completely. SOURCE_DATE_EPOCH is already recorded in the Environment section.

(I prefer the current approach to .buildinfo but other projects do things differently, so I'm open to arguments)

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

HW42 · 2024-10-17T14:54:36Z

qubesbuilder/plugins/build_deb/scripts/create-local-repo

@@ -71,7 +71,8 @@ mkdir -p "dists/$SUITE/main/binary-amd64"
 dpkg-scanpackages --multiversion . > "dists/$SUITE/main/binary-amd64/Packages"
 gzip -9c "dists/$SUITE/main/binary-amd64/Packages" > "dists/$SUITE/main/binary-amd64/Packages.gz"

-DATE=$(LC_ALL=C date -u +"%a, %d %b %Y %H:%M:%S %Z")
+DATE=$(LC_ALL=C date -u -d "@${SOURCE_DATE_EPOCH:-$(date +%s)}" +"%a, %d %b %Y %H:%M:%S %Z")
+


I think this local repo is only used internally by qubes-builderv2 so the included date shouldn't matter. If it should be made reproducible for some reason then this is not enough. For example the gzip above records the file modification time.

This is only for generating on the fly local build repository.

marmarek · 2024-10-17T23:18:07Z

Pipeline with all tests enabled: https://gitlab.com/QubesOS/qubes-builderv2/-/pipelines/1501302493

marmarek · 2024-10-21T20:16:34Z

qubesbuilder/plugins/installer/Makefile

 ISO_VERSION ?= $(shell date +%Y%m%d)
+ISO_VERSION ?= $(shell date -d @"$(SOURCE_DATE_EPOCH)" +%Y%m%d 2>/dev/null || date +%Y%m%d)


ISO_VERSION is set already on the line just above. I think you wanted to replace that one, not add another one

Same comment than template applies here too. We cannot reproduce a given ISO or template based on timestamp. We don't hold any snapshot at a given point of time.

marmarek · 2024-10-21T20:16:48Z

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

@@ -22,6 +22,9 @@

 # Clamp the topmost changelog entry date ("Build for ...") to the previous one
 # (actual meaningful entry)
+#Reverted clamp-changelog-entry-date
+
+CHANGELOG_PATH="$1"


Why? It's set to the same value just below.

marmarek · 2024-10-21T20:22:00Z

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

+# PREVIOUS_DATE=$(grep '^ --' "$CHANGELOG_PATH" | head -n 2 | tail -n 1 | grep -o '  .*')
+
+# Replace the topmost date with CURRENT_DATE
+sed -e "0,/^ --/s/^\( --.*\)\(  .*\)/\1$CURRENT_DATE/" -i "$CHANGELOG_PATH"


CURRENT_DATE is not set

I think what you need is to set CURRENT_DATE like this:

if [ -n "$SOURCE_DATE_EPOCH" ]; then CURRENT_DATE=$(date -R -u -d "@$SOURCE_DATE_EPOCH") else CURRENT_DATE=$(grep '^ --' "$CHANGELOG_PATH" | head -n 2 | tail -n 1 | grep -o ' .*') fi

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

Final commit Final commit Final commit Final commit final commit final one made changes to clamp-changelog-entry-date made more changes removed the spurious files and made appropriate changes to scripts edited script edited script retrieved .txt files edited files ammended the file added files added and removed files Removed unnecessary files and updated .gitignore edited the script edited the script Replaced date with 1727261746 for reproducibility final changes Last commit final changes shellcheck-error fixes shellcheck change fixed clamp-changelog-entry-date Reverting to fixed-clamp-change final changes final changes shellcheck-error fixes shellcheck change fixed clamp-changelog-entry-date retrieved .txt files retrieved the .txt files and gitignore shellcheck-error fixes shellcheck change fixed clamp-changelog-entry-date Reverting to fixed-clamp-change final changes final changes shellcheck-error fixes shellcheck change fixed clamp-changelog-entry-date retrieved .txt files retrieved the .txt files and gitignore Delete new_rpm_header.txt Delete new_src_header.txt Delete e -i HEAD~5 Update .gitignore Delete qubesbuilder/.gitignore Delete qubesbuilder/dependencies-debian.txt Delete qubesbuilder/dependencies-fedora-qubes-executor.txt Delete qubesbuilder/dependencies-fedora.txt Delete e -i HEAD~5 Delete qubesbuilder/dependencies-debian.txt Delete qubesbuilder/.gitignore Delete qubesbuilder/dependencies-fedora-qubes-executor.txt Delete qubesbuilder/dependencies-fedora.txt Delete new_rpm_header.txt Update modify-changelog-for-build Update clamp-changelog-entry-date Update clamp-changelog-entry-date Update clamp-changelog-entry-date Update clamp-changelog-entry-date Update clamp-changelog-entry-date Update Makefile Update clamp-changelog-entry-date Update clamp-changelog-entry-date

fepitre · 2024-10-22T13:06:24Z

qubesbuilder/plugins/template/Makefile

@@ -45,7 +45,11 @@ fix_up := $(shell TEMPLATE_NAME=$(TEMPLATE_NAME) TEMPLATE_LABEL="$(TEMPLATE_LABE
 	$(BUILDER_DIR)/scripts/builder-fix-filenames)

 TEMPLATE_NAME := $(word 1,$(fix_up))
-TEMPLATE_TIMESTAMP ?= $(shell date -u +%Y%m%d%H%MZ)
+
+TEMPLATE_TIMESTAMP ?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" +%Y%m%d%H%MZ 2>/dev/null || date -u +%Y%m%d%H%MZ)


It does not make sense here because we cannot reproduce from a given timestamp everything that follows after the build process. It's just a marker, not used at all as a reference.

I now understand that the TEMPLATE_TIMESTAMP is just a marker and not critical for reproducibility, especially since it doesn't affect the rest of the build process. Given this, I'll revert the change and keep the original date logic.

fepitre · 2024-10-22T13:09:55Z

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date

+else
+    # Get previous date from changelog
+    CURRENT_DATE=$(grep '^ --' "$CHANGELOG_PATH" | head -n 2 | tail -n 1 | grep -o '  .*')
+fi


To be honest, I don't think this is a good idea. The reference should be the date provided by the CHANGELOG which means that SOURCE_DATE_EPOCH should somehow match that on Debian side.

I see your point about using the SOURCE_DATE_EPOCH as the reference for Debian reproducibility. I've look adjusted the script so that SOURCE_DATE_EPOCH is used more effectively, ensuring consistency with Debian's side rather than relying on the previous changelog entry date.

egbedo force-pushed the reproducibility-fixes branch 2 times, most recently from e897d97 to 33552cf Compare October 16, 2024 10:23

marmarek requested changes Oct 16, 2024

View reviewed changes

marmarek requested changes Oct 17, 2024

View reviewed changes

HW42 reviewed Oct 17, 2024

View reviewed changes

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date Show resolved Hide resolved

HW42 reviewed Oct 17, 2024

View reviewed changes

egbedo force-pushed the reproducibility-fixes branch 14 times, most recently from 0bd1bb9 to 717642f Compare October 21, 2024 19:26

marmarek requested changes Oct 21, 2024

View reviewed changes

egbedo force-pushed the reproducibility-fixes branch from fbebd15 to d21c9ec Compare October 21, 2024 20:37

marmarek reviewed Oct 21, 2024

View reviewed changes

qubesbuilder/plugins/source_deb/scripts/clamp-changelog-entry-date Outdated Show resolved Hide resolved

egbedo force-pushed the reproducibility-fixes branch from d1cc15c to 936e43f Compare October 21, 2024 21:02

fepitre reviewed Oct 22, 2024

View reviewed changes

egbedo added 2 commits October 22, 2024 06:34

Updated Makefile

212c7b9

Updated clamp-changelog-entry-date

dfae88a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Build Reproducibility for Core Admin Client #154

Improve Build Reproducibility for Core Admin Client #154

egbedo commented Oct 15, 2024 •

edited

Loading

marmarek commented Oct 15, 2024

codecov bot commented Oct 16, 2024 •

edited

Loading

marmarek left a comment

marmarek Oct 17, 2024

marmarek Oct 17, 2024

marmarek commented Oct 17, 2024

HW42 Oct 17, 2024

HW42 Oct 17, 2024

fepitre Oct 22, 2024

marmarek commented Oct 17, 2024

marmarek Oct 21, 2024

fepitre Oct 22, 2024

marmarek Oct 21, 2024

marmarek Oct 21, 2024

fepitre Oct 22, 2024

egbedo Oct 22, 2024

fepitre Oct 22, 2024

egbedo Oct 22, 2024

		ISO_VERSION ?= $(shell date +%Y%m%d)
		ISO_VERSION ?= $(shell date -d @"$(SOURCE_DATE_EPOCH)" +%Y%m%d 2>/dev/null \|\| date +%Y%m%d)

Improve Build Reproducibility for Core Admin Client #154

Are you sure you want to change the base?

Improve Build Reproducibility for Core Admin Client #154

Conversation

egbedo commented Oct 15, 2024 • edited Loading

marmarek commented Oct 15, 2024

codecov bot commented Oct 16, 2024 • edited Loading

Codecov Report

marmarek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmarek commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmarek commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

egbedo commented Oct 15, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading