prebuilder.py

A deeply opinionated packages build system.

Why?

I am deeply dissatisfied with all the systems of building packages for distros.

• They are makefile-based and bash-based. These scripting languages are too limited to do something complex. Especially if the upstream is uncooperative.
• They often require a source archive. When it is not present they refuse to build.
• They are often buggy.
• They often have unneeded stages.

See the manifesto for more info.

Requirements

• Python

  • sh
  • gpgme
  • patch
  • AnyVer
  • pyelftools
  • File2Package.py
    • File2Package.backend.dpkg
  • CMake.py
  • GitPython](https://libraries.io/github/gitpython-developers/GitPython)

• build systems

  • GNU make
  • ninja
  • meson
  • ckati
  • CMake
  • autotools
    • automake
    • autoconf
    • m4

• CLI tools

  • firejail
  • git
  • 7z
  • aria2c;
  • alien
  • strip-nondeterminism
  • fakeroot
  • reprepro
  • dpkg-sig
  • dpkg-dev

Tutorial

Reading the source code of the essential parts is strictly needed. This tool is not a ready-made stuff. It is a framework allowing you do do the stuff easily. It does lot of things for you, but you need to understand what it does. It is not bash. It is object-oriented. It is poorly documented, the docs is the code itself. You cannot use it without mastering it. But mastering it IMHO worth. And and the stuff was designed to be easy to master. So, roll up your sleeves and prepare to get your hands dirty.

** All the stuff done by build scripts and systems is explicitly trusted. Don't build software from untrusted sources.**

• Get known the purpose of classes and modules:

  • Some misc shit you usually don't want to touch directly:

    • .tools - just wrappers around command-line and not-so-command-line tools. Some of them use subrocesses. Some of them are not command line tools, but just python bindings (and we wanna rewrite as much of them to use bindings). This subpackage is just a semantic way to organize them. If you wanna need to use some CLI tool during your build, just check that subpackage first.
    • .utils - just miscellaneous utils.
    • .webServices - interactions to services likr GitHub, GitLab, BitBucket and Launchpad.

  • The global shit you likely need to touch:

    • RunConfig - some args to the tool. Temporary dirs mostly. You likely don't need to touch it directly, the defaults are sensible.
      • .core.Person - an entity with nickname o email. core.Maintainer is a person which fields are automatically captured from environment variables.
    • .buildPipeline.BuildPipeline - A build pipeline for a single piece of software
    • .repoPipeline.RepoPipeline - A pipeline creating a single repository.

  • In order to build a software you need to construct BuildPipeline:

    • PackageMetadata - Each package must have metadata. Some of it is retrieved automatically. Some is not. Highly depends on what is provided by the build system. The metadata provided here is merged to the metadata provided by build system. You can specify different metadata for each distro.

      • PackageRef - you need to call the package somehow.
      • interpackage relations.
      • descriptions

    • BuildRecipy - describes how to build a certain piece of software.

      • IFetcher - an interface describing how to get source code (or data) for building a package. Also responsible for extraction of some metadata (more precisely, a version).

        • DiscoverDownloadVerifyUnpackFetcher - Discovers an archive with the latest release, downloads it, verifies its integrity, then unpacks it. As the name suggests, combines 4 components:
          • Discoverer - Interacts with various services.
            • GithubReleasesDiscoverer - discovers releases on GitHub.
          • Downloader - Downloads the software using aria2c.
          • Verifier - verifies integrity.
            • GPGVerifier - checks an OpenPGP signature.
            • HashesVerifier - checks hashes against the file with them. Integrity of that file must be verified with an another verifier.
          • Unpacker - unpacks archives.
        • GitRepoFetcher - Clones a git repo.

      • BuildSystem - A class abstracting a build system. You usually don't need to instantiate them. It

        • configures packages
        • builds them
        • installs them into a dedicated dir
          • Action is the stuff describing what to do in order to install a piece of package.
          • core.FilesTracker.FilesTracker tracks files, symlinks and dirs. It also generates checksums.
          • Some build systems emit subpackages and actions.

      • .systems - contains build systems. Each one has own arguments. Read the code.

        • A special Dummy build system builds nothing. Useful when repackaging already built stuff.

      • buildOptions - you may add options applied on configuration stage as a dict.

      • patches - you may want to apply some patches.

  • Once you have built the software, it is transformed into packages and packed into repos:

    • .tearers - split packages into subpackages having certain roles, we call them groups. For example, when you build a software and install it, you can get a library, a command-line interface to that library, syntax highlighting to its configs, bash autocompletion, and docs. They are usually mixed int a pile. Tearers are what automatically split this pile into different packages.

    • Distro - Each distro has a set of conventions how to split the software into multiple packages and where to install them. This object incapsulates these conventions.

    • .distros - contains descriptions of distros and distro-specific code. Such as Debian, Fedoara, Arch and pseudodistros as Android, Windows, etc... You usually don't need to interact to them directly, you only need to specify them, and the pipeline will do most of stuff itself`.

      • Each distro has an own naming conventions on how to name packages belonging to each group. For example, packages having libs are usually have names beginning with lib and packages containing python packages have names beginning from python3. These contains the code to convert names to distro-specific and back. core.NamingPolicy is a class encapsulating stuff needed for that. Some widespread naming transformations can be found in .namingPolicies.
      • Each distro has own locations where to put files belonging to different packages.
      • Each distro has own release cycle and names of releases.
      • Each distro uses some package manager and uses some tools to build packages and repositories. We abstract it too.

    • RepoPipelineMeta is a special metaclass providing syntax sugar allowing creation of RepoPipeline to look like a class.

  • .importers - sometimes you already have the metadata in the format of other tools, this submodule contains the tools to transform these metadata into the format available to prebuilder:

    • debhelper debian directory.
    • RPM spec files.
• Import everything you need
• Create a

class build(metaclass=RepoPipelineMeta):
	pass

• Add there a function for every software you need packages for. For [example]

	def comps(): # just a convenient name for us. Prfix with `_` in order to disable
		repoURI = "https://github.com/rpm-software-management/libcomps" # GitHub Repo URI
		class cfg(metaclass=ClassDictMeta): # special syntax sugar for making a dict definition look like a class definition. Here we define our metadata
			descriptionShort = "Libcomps is a pure C alternative for yum.comps library."
			descriptionLong = "And there are bindings for python2 and python3."
			section = "devel"
			homepage = repoURI
		
		buildRecipy = BuildRecipy(
			CMake, # the stuff use CMake as a build system
			GitRepoFetcher(repoURI, refspec="master"), # git the source from a git repo
			
			patches = [(thisDir / "patches" / "libcomps")], # apply patches. libcomps is a dir, so apply the patches in it.
			buildOptions = { # pass the following additional options to CMake
				"PYTHON_DESIRED": 3,
				"ENABLE_TESTS": False,
				"ENABLE_DOCS": False,
			},
			subdir="libcomps" # usually unneeded. In our case CMakeLists is not in the root of the repo.
		)
		metadata = PackageMetadata("comps", **cfg) # Create a metadata object. The first arg should be a `PackageRef` but here it is OK provide string, it will do everything needed itself. THE NAME IS WITHOUT ANY PREFIXES AND SUFFIXES! IT WILL GENERATE THEM ITSELF.
		
		# we must return a BuildPipeline
		return BuildPipeline(
			buildRecipy,
			(
				(Debian, metadata), # we need to build packages for Debian having metadata `metadata`
			)
		)