What is Freshmaker?

Freshmaker is a service which:

• Rebuilds various artifacts after its dependencies changed. For example:
  • Rebuilds module when .spec file of RPM in a module is updated or the RPM is pushed to stable using Bodhi.
  • Rebuilds container image when RPM in a container is updated or pushed to stable using Bodhi.
  • ...
• Provides a REST API to retrieve information about artifact rebuilds and the reasons for these rebuilds and sends messages to the message bus about these event.
• Is based on Event-Handler architecture:
  • Events are fedmsg messages received from other services like dist-git, Koji, Bodhi, ...
  • Handlers are small classes using the high-level API provided to them by Freshmaker core to rebuild artifacts.
• Provides high-level API to track dependencies between artifacts and make rebuilding of them easy.

See the docs for additional information.

How does it work?

Freshmaker waits for new fedmsg messages about artifact being updated, this happens in consumer.py. Every fedmsg message handled by a Freshmaker is parsed by one of the parsers which can be found in the parsers directory. This parser converts the fedmsg to Event object (inherits from BaseEvent). All the events are defined in the events.py.

This Event object is then handled by one or more handlers defined in the handlers directory.

The handlers typically do following:

• Find out the list of artifacts like RPMs, Container images, ... which must be rebuilt as result of Event.
  • There are built-in classes to find these information in Lightblue.
• Plan the rebuild of artifacts and store them in Freshmaker database in tree-like structure.
  • Artifacts can depend on each other and Freshmaker ensures they are built in the right order.
• Prepare the prerequisites for a rebuild.
  • This can for example mean generating RPM repository with updated packages using ODCS.
• Start the rebuild.

Most of the common tasks for handlers are already implemented in Freshmaker including:

• Database and methods to store artifacts, their dependencies and all the information to build them there.
• Classes to get the information about artifacts from Koji, Lightblue, ...
• Classes to start the builds in Koji, ODCS, ...

The handler which rebuilds all the images that contain packages affected by CVEs in advisory when the advisory is shipped can look like this (only sample code for demonstration):

class RebuildImagesOnRPMAdvisoryChange(ContainerBuildHandler):

    name = 'RebuildImagesOnRPMAdvisoryChange'

    def can_handle(self, event):
        if not isinstance(event, ErrataAdvisoryRPMsSignedEvent):
            return False
        return True

    def handle(self, event):
        self.event = event

        db_event = Event.get_or_create_from_event(db.session, event)
        db.session.commit()

        # Check if we are allowed to build this advisory.
        if not self.event.is_allowed(self):
            db_event.transition(
                EventState.SKIPPED, f"Advisory {event.advisory.errata_id} is not allowed")
            db.session.commit()
            return []

        # Get and record all images to rebuild
        batches = self._find_images_to_rebuild(db_event.search_key)
        builds = self._record_batches(batches, event)

        self.start_to_build_images(db_event.get_image_builds_in_first_batch(db.session))
        db_event.transition(EventState.BUILDING, msg)

        return []

Initial development setup

Create and activate a Python virtual environment.

Install the dependencies with:

python3 setup.py develop

Install required rpm packages:

dnf install $(sed 's/#.*//' yum-packages-devel.txt)

Install the requirements:

pip3 install -r requirements.txt

Install the requirements useful to run the tests:

pip3 install -r test-requirements.txt

Run the tests:

pytest tests/

In order to generate the requirements you need pip-tools:

python -m pip install pip-tools

more info available at: https://github.com/jazzband/pip-tools/

Formatting code with black

Freshmaker currently uses black as formatter and style-checker. The CI workflow runs a black step to check for files that would be reformatted, and if it finds any such file, it will fail.

You can manually run the black-format step in tox while developing to make sure that your code follows the repo standard. You can run

tox -e black-format <file1> <file2> ...

to point the reformat step to secific files that you would like, or

tox -e black-format

to run it in all the necessary files of the project.

Ignoring large reformattings with git blame

The commits listed in .git-blame-ignore-revs are automatically ignored in github blame view (https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view). This feature is meant to ignore commits that just introduced reformatting, and maintain the logical contributions of each line.

You can make use of the .git-blame-ignore-revs file locally by passing it as an argument to the blame command:

git blame --ignore-revs-file .git-blame-ignore-revs ...

or by adding it to your local git configurations:

git config --local blame.ignoreRevsFile .git-blame-ignore-revs

Note that this should be done for per-project, as some projects may lack the .git-blame-ignore-revs file, and in this case git blame would return an error.