Pylightnix is a library for manipulating immutable data objects. It provides core API for checking, creating and querying such objects using files and folders as a data sotrage. One kind of applications which could benefit from this API is package managers.
We illustrate the basic concepts by designing a toy package manager able to compile and run the GNU hello program.
GNU Hello is a demo application which prints ‘Hello world!’ on its standard output. It’s purpose is to demonstrate the usage of Automake tools. We will see how Pylighnix could help us to solve this quest. We assume that the host system provides an access to the GNU Automake toolchain, that is, paths to Autoconf, Automake, gcc, etc should present in system PATH variable.
We go through the following plan of actions:
- Use built-in rules to download and unpack the GNU Hello sources.
- Define a custom rule for compiling the GNU Hello from sources.
- Run the application by querying the Pylightnix artifact storage.
Pylightnix offers functions which form a kind of domain-specific
language, embedded in Python language. Our program is a Python script,
which could be executed by a common python3 interpreter. In this demo
we will need certain standard Python functions. Later we will import
Pylightnix functions as needed.
from os.path import join
from os import system, chdir, getcwd
from shutil import copytree
from tempfile import TemporaryDirectory
from typing import Any
from subprocess import Popen, PIPEFirst things first, Pylightnix uses filesystem for tracking immutable data objects which could depend on each other. Objects reside partly in the filesystem, partly in memory as a Python objects. We initialize the filesystem part by calling fsinit on StorageSettings and then create the global Registry. The Registry and the part of filesystem described by StorageSettings are the only mutable objects that we will operate on.
from os import environ
from pylightnix import Registry, StorageSettings, mkSS, fsinit
S:StorageSettings=mkSS('/tmp/pylightnix_hello_demo')
fsinit(S,remove_existing=True)
R=Registry(S)
hello_version = '2.10'Pylightnix manages data processing operations called Derivations. The toolbox provides a generic constructor mkdrv and a set of pre-defined Stages which wraps it with problem-specific parameters.
In this tutorial we will need fetchurl2 and unpack stages. The first one knows how to download URLs from the Internet, the second one knows how to unpack archives. We import both functions, along with other Pylightnix APIs.
from pylightnix import (Registry, DRef, RRef, fetchurl2, unpack, mklens, selfref)Our first goal is to make derivations ready for realization by
registering them in the Registry R. We call fetchurl2 with the
appropriate parameters and get an unique
DRef reference back. Every
DRef value proofs to us that the Registry is aware of our new
derivation.
tarball:DRef = fetchurl2(
name='hello-src',
url=f'http://ftp.gnu.org/gnu/hello/hello-{hello_version}.tar.gz',
sha256='31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b',
out=[selfref, f'hello-{hello_version}.tar.gz'], r=R)In order to link derivations into a chain we should put the derivation reference of a prerequisite derivation somewhere into the config of a child derivaiton.
hello_src:DRef = unpack(
name='unpack-hello',
refpath=mklens(tarball,r=R).out.refpath,
aunpack_args=['-q'],
src=[selfref, f'hello-{hello_version}'],r=R)The selfref path is our promise to Pylightnix that the said path would
appear after the derivation is realized. Pylightnix checks such promises
and raises in case they are not fulfilled.
Now we are done with registrations and going to obtain our objects. We call instantiate to compute the dependency closure of the target object and pass it to realize1 which runs the show. As a result, we obtain a reference of another kind RRef
from pylightnix import instantiate, realize1
hello_rref:RRef = realize1(instantiate(hello_src, r=R))
print(hello_rref)rref:29eaa2c8e74cbc939dfdd8e43f3987eb-43323fae07b9e30f65ed0a1b6213b6f0-unpack-hello
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
25 708k 25 180k 0 0 208k 0 0:00:03 --:--:-- 0:00:03 208k
100 708k 100 708k 0 0 465k 0 0:00:01 0:00:01 --:--:-- 465k
hello-2.10.tar.gz: extracted to `hello-2.10'
RRefs could be converted to system paths by calling an mklens the swiss-army-knife data accessor of Pylightnix:
from pylightnix import current_storage
with current_storage(S):
print(mklens(hello_rref).val)
print(mklens(hello_rref).syspath)
print(mklens(hello_rref).src.syspath)rref:29eaa2c8e74cbc939dfdd8e43f3987eb-43323fae07b9e30f65ed0a1b6213b6f0-unpack-hello
/tmp/pylightnix_hello_demo/store-v0/43323fae07b9e30f65ed0a1b6213b6f0-unpack-hello/29eaa2c8e74cbc939dfdd8e43f3987eb
/tmp/pylightnix_hello_demo/store-v0/43323fae07b9e30f65ed0a1b6213b6f0-unpack-hello/29eaa2c8e74cbc939dfdd8e43f3987eb/hello-2.10
Pylightnix offers a number of other shell-like helper functions for
accessing realization, like lsref:
from pylightnix import lsref, catref
print(lsref(hello_rref, S))['__buildstop__.txt', '__buildstart__.txt', 'hello-2.10', 'context.json']
In this section we define a custom stage to build the newly obtained sources of GNU Hello application.
Defining Pylighnix stages requires us to provide Pylightnix with the following components:
- The JSON-like configuration object.
- The matcher Python function, dealing with non-determenistic builds.
- The realizer Python function which specifies the actual build process.
The matcher business is beyond the scope of this tutorial. We will use a
trivial match_only matcher which instructs Pylightnix to expect no
more than one realization of a stage in its storage.
We produce a Config object by reading local variables of a helper
function hello_config. We could have just call mkconfig on a dict.
from pylightnix import Config, mkconfig, mklens, selfref
def hello_config()->Config:
name = 'hello-bin'
src = mklens(hello_src,r=R).src.refpath
out_hello = [selfref, 'usr', 'bin', 'hello']
out_log = [selfref, 'build.log']
return mkconfig(locals())To specify Realizer we write another Python function which accepts the
Build context. We use mklens to query the parameters of the
derivation being built just as we used it for querying parameters of
completed realizations. The selfref paths is initialized to paths
inside the build temporary folder where we must put the build artifacts.
Here we produce the GNU hello binary and a build log as a side-product.
from pylightnix import (Path, Build, build_cattrs, build_outpath, build_path,
dirrw )
def hello_realize(b:Build)->None:
with TemporaryDirectory() as tmp:
copytree(mklens(b).src.syspath,join(tmp,'src'))
dirrw(Path(join(tmp,'src')))
cwd = getcwd()
try:
chdir(join(tmp,'src'))
system(f'( ./configure --prefix=/usr && '
f' make &&'
f' make install DESTDIR={mklens(b).syspath}'
f')>{mklens(b).out_log.syspath} 2>&1')
finally:
chdir(cwd)Finally, we introduce a new stage to Pylightnix by instantiating a generic mkdrv stage:
from pylightnix import mkdrv, build_wrapper, match_only
hello:DRef = mkdrv(hello_config(),match_only(),build_wrapper(hello_realize),R)
print(hello)dref:e48878b9f7760fe0972eb6863775045f-hello-bin
As before, we get a DRef promise pass it through instantiate and
realize1 pipeline:
rref:RRef=realize1(instantiate(hello,r=R))
print(rref)rref:b02efb753dbd11e37d4e1c5d77eceb52-e48878b9f7760fe0972eb6863775045f-hello-bin
Lets print the last few lines of the build log:
for line in open(mklens(rref,r=R).out_log.syspath).readlines()[-10:]:
print(line.strip())fi
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/mkdir -p '/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/info'
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/install -c -m 644 ./doc/hello.info '/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/info'
install-info --info-dir='/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/info' '/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/info/hello.info'
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/mkdir -p '/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/man/man1'
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/install -c -m 644 hello.1 '/tmp/pylightnix_hello_demo/tmp/210906-00:33:57:724316+0300_2b29fe60_35aw7mg4/usr/share/man/man1'
make[4]: Leaving directory '/run/user/1000/tmp3yo1o2pv/src'
make[3]: Leaving directory '/run/user/1000/tmp3yo1o2pv/src'
make[2]: Leaving directory '/run/user/1000/tmp3yo1o2pv/src'
make[1]: выход из каталога «/run/user/1000/tmp3yo1o2pv/src»
Finally, we convert RRef to the system path and run the GNU Hello binary.
print(Popen([mklens(rref,r=R).out_hello.syspath],
stdout=PIPE, shell=True).stdout.read()) # type:ignoreb'Hello, world!\n'