[Bug]: Maintenance builds don't create corresponding packages for the apt repo

[SteeManMI]

What happened?

This isn't an issue with the build framework, explicitly, but an issue with the release management process overall.

A forum user reported a problem where they can't install the correct version of kernel headers for the version of the kernel they have installed (https://forum.armbian.com/topic/42376-helios64-kernel-headers-for-this-kernel-do-not-seem-to-be-installed/). This happens every release, and isn't a new issue.

From what I witness, this is what is happening:

1. Original release goes out (24.5.1) in this case, all images are built and the apt repository is populated with the corresponding packages and everything is in sync at that time.
2. Issues need to get addressed and maintenance images are produced (24.5.2, 24.5.3, etc) These images appear to be built off of current main branch of armbian/build (not a patched version of the 24.5 branch). So instead of picking up all the same versions of referenced code (that we pinned at the time we froze the original release), it picks up the current patch release level of the kernel (and other code as well). So these images built for a maintenance release (24.5.2, 24.5.3, etc) end up including newer versions of the kernel than 24.5.1 was released with. But these new kernel packages aren't also put in the apt repository.
3. So when a user goes to install an additional package (like the kernel headers) after installing 24.5.3, the version in the apt repo doesn't match the version of the kernel they have in their image. And thus you have the problem reported by the user.

From my perspective the correct way to produce maintenance releases is that the desired changes should first be made and tested in mainline armbian/build and then backported to the release branch (24.5 in this case). And then images should be produced from that branch, otherwise you can't really control what changes you are incorporating into a maintenance release.
With the logic added this year to pin all dependencies, this should result in dependencies consistent with the original release. Of course any artifacts that change as a result of what was fixed in armbian/build (likely to be uboots, kernels, etc), should be also be versioned correctly and added to the apt repo. I don't know if the current packaging versioning logic can handle that case, (i.e. can you have a 24.5.1 and a 24.5.3 version of the same kernel i.e. 6.6.36?)

With consistent builds (coming from a mostly stable branch) and corresponding artifacts being pushed to the apt repo, a user should always be able to have access to the corresponding .debs for their image and be able to install anything needed (generally, kernel headers, but it could be for a minimal install other packages like armbian-config).

How to reproduce?

Described above.

Branch

main (main development branch)

On which host OS are you running the build script and observing this problem?

Ubuntu 24.04 Noble

Are you building on Windows WSL2?

• Yes, my Ubuntu/Debian/OtherOS is running on WSL2

Relevant log URL

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Jira ticket: AR-2417

[igorpecovnik]

is that the desired changes should first be made and tested in mainline armbian/build and then backported to the release branch (24.5 in this case)

Agree. What to do before this will be possible? We don't have one device or one arch. Testing is very complex and expensive process even if there would be only one device. I can only rely on (primitive) automated testings. Which is also not flexible enough to test other things then stable / beta. And even that is not in good shape.

corresponding artifacts being pushed to the apt repo, a user should always be able to have access to the corresponding .debs for their image and be able to install anything needed

How this problem was addressed in the past? All images that are being produced are dropping their build artifacts somewhere to the repository server. And until recently, those artifacts needed manual intervention to be included into repository ...

Procedure

Last week I have changed this repo management to this:

which means that it goes like this:

1. Person that is in a group "Release manager" generates a new image for, lets say, Helios 64 and give him automated version (next from what it is now - 24.5.4). In reality maintainer tells me to make an image and me should be removed from this process first.
2. Images build by CI are uploaded here https://rsync.armbian.com/incoming/ (not directly to the download page)
3. Release manager (myself ATM) tells me, "Hey, I made a new image, they are tested, please move it to main download area"
4. I do SSH to rsync server and move files from incoming to main download. The rest is taken care by automation - download pages index recreation, sync, redirector update, ...
5. Artifact from images creation are uploaded from CI runners to common /incoming/partial folder (exposed on https://netcup.armbian.com/partial/).
6. If we also need to push out artifacts, they need to be copyed to /incoming/debs folder
7. Once they are there, automation takes over and they are included into repo at next build
8. But. There is another problem related to apt mirror syncing. In order to avoid errors on update, we need to update files first, then update index once files are already synced around - which we don't know just yet and timeout is used. This script is an attempt to address this, but hasn't been put into production yet. There is also another script that add / remove mirrors that are (not) in sync. Which is partially accurate, but with timeout, things works well enough.

This is the status in short.

[The-going]

I don't know if the current packaging versioning logic can handle that case, (i.e. can you have a 24.5.1 and a 24.5.3 version of the same kernel i.e. 6.6.36?)

Yes, it can. That's exactly what happens.
I am building an image and ignoring all artifacts.
I.e., I am collecting the current state on the local machine. version 24.5.1.
Then I change the u-boot package two times and to distinguish three different builds,
I change the version of the u-boot package 24.5.2, 24.5.3.
I assemble an image each time for the test.
I have not changed the kernel package, but different versions of the kernel will be installed
in each image, although it is the same thing.

What's the logic here?

[igorpecovnik]

What's the logic here?

WIP documentation:
armbian/documentation#449

Need real-world testers.