From 109491cb396ea9f5b6b8f5f7866d876bad4fc768 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Fri, 12 Jul 2024 23:08:12 -0400 Subject: [PATCH 01/58] Initialize developer environment --- .gitignore | 3 ++- README.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 66c6e29..6049381 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +/.venv/ /.lxc/ /jails/ -.DS_Store \ No newline at end of file +.DS_Store diff --git a/README.md b/README.md index 093cf54..d5c1f69 100644 --- a/README.md +++ b/README.md @@ -212,6 +212,35 @@ TODO: write comparison between systemd-nspawn (without `jailmaker`), LXC, VMs, D The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image server](https://images.linuxcontainers.org). These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the `alpine` image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine. +## Development + +After cloning the project, navigate into its working directory and create a self-contained Python [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). + + python3 -m venv --no-setuptools .venv + +> *Note: Due to [NAS-130029](https://ixsystems.atlassian.net/browse/NAS-130029), the user's safe working Python environment can only be bootstrapped from TrueNAS SCALE's [developer mode](https://www.truenas.com/docs/scale/scaletutorials/systemsettings/advanced/developermode/), or in a jail or elsewhere. If you like to take long walks with strangers, the following might suffice as an alternative to the above one-liner.* +> +> ``` +> curl -L https://github.com/brettcannon/microvenv/archive/refs/tags/v2023.5.tar.gz | tar xz --strip-components=1 microvenv-2023.5/microvenv +> python3 -m microvenv +> curl -OL https://bootstrap.pypa.io/pip/pip.pyz +> .venv/bin/python3 pip.pyz install virtualenv +> .venv/bin/virtualenv --no-setuptools .venv +> rm -rf microvenv pip.pyz +> ``` + +Activate the venv into your *current* shell session. + + source .venv/bin/activate + +Develop away. Note that when you're done, you can undo this activation and return to the system's default Python environment. Just call a function that activation has inserted into your shell session: + + deactivate + +For more information on Python standard venvs, go to [the source](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). + +*TODO: introduce the tools we use for building, testing, and stylechecking.* + ## Filing Issues and Community Support When in need of help or when you think you've found a bug in `jailmaker`, [please start with reading this](https://github.com/Jip-Hop/jailmaker/discussions/135). From 47f332bf75ae2d7851806961180fc163cdca9dc0 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 21:12:51 -0400 Subject: [PATCH 02/58] Introduce a formal build system --- .github/workflows/build.yml | 56 ++++++++++++++++++ .gitignore | 21 ++++++- README.md | 21 ++++++- pyproject.toml | 85 ++++++++++++++++++++++++++++ src/builder/app/__init__.py | 3 + src/builder/app/build_app.py | 56 ++++++++++++++++++ src/builder/app/hooks_app.py | 11 ++++ src/builder/app/pyproject.toml | 22 +++++++ src/builder/zip/__init__.py | 3 + src/builder/zip/build_zip.py | 60 ++++++++++++++++++++ src/builder/zip/hooks_zip.py | 11 ++++ src/builder/zip/pyproject.toml | 22 +++++++ src/jlmkr/__about__.py | 12 ++++ src/jlmkr/__main__.py | 13 +++++ src/jlmkr/donor/__init__.py | 13 +++++ jlmkr.py => src/jlmkr/donor/jlmkr.py | 0 16 files changed, 407 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/build.yml create mode 100644 pyproject.toml create mode 100644 src/builder/app/__init__.py create mode 100644 src/builder/app/build_app.py create mode 100644 src/builder/app/hooks_app.py create mode 100644 src/builder/app/pyproject.toml create mode 100644 src/builder/zip/__init__.py create mode 100644 src/builder/zip/build_zip.py create mode 100644 src/builder/zip/hooks_zip.py create mode 100644 src/builder/zip/pyproject.toml create mode 100644 src/jlmkr/__about__.py create mode 100644 src/jlmkr/__main__.py create mode 100644 src/jlmkr/donor/__init__.py rename jlmkr.py => src/jlmkr/donor/jlmkr.py (100%) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 0000000..f82226b --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,56 @@ +# Inspiration from + +name: Build + +on: + push: + branches: ["**"] + pull_request: + branches: ["**"] + + workflow_dispatch: + +jobs: + + build: + name: Build jlmkr tool + runs-on: ubuntu-24.04 + strategy: + matrix: + python-version: + - "3.11" # TrueNAS SCALE 24.04 Dragonfish + steps: + + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + - name: Set up pip cache + uses: actions/cache@v3 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }} + restore-keys: ${{ runner.os }}-pip- + + - name: Install Hatch + uses: pypa/hatch@install + + - name: Run unit tests + run: hatch run +py=${{ matrix.python-version }} test:test + + - name: Build distribution + run: hatch build -t zipapp -t appzip + +# - name: Upload artifacts +# uses: actions/upload-artifact@v4 +# with: +# path: +# - dist/jlmkr +# - dist/jlmkr-*.zip +# if-no-files-found: error diff --git a/.gitignore b/.gitignore index 6049381..6c6dbf4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,23 @@ -/.venv/ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +dist/ + +# jail-specific /.lxc/ /jails/ + +# Mac-specific .DS_Store +._.DS_Store + +# + +/.venv/ + +__pycache__/ +*.py[cod] + +.pytest_cache/ +.ruff_cache/ diff --git a/README.md b/README.md index d5c1f69..4d626fd 100644 --- a/README.md +++ b/README.md @@ -228,6 +228,7 @@ After cloning the project, navigate into its working directory and create a self > .venv/bin/virtualenv --no-setuptools .venv > rm -rf microvenv pip.pyz > ``` +*Note: This process and the resulting build environment will cache some items under `~/.local/share` in addition to the project directory.* Activate the venv into your *current* shell session. @@ -239,7 +240,25 @@ Develop away. Note that when you're done, you can undo this activation and retur For more information on Python standard venvs, go to [the source](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). -*TODO: introduce the tools we use for building, testing, and stylechecking.* +### Hatching a build + +While in an *active* session, install the [Hatch](https://hatch.pypa.io) project manager. This will load quite a flurry of dependencies, but will only do so into the new `.venv` directory. + + pip install hatch + +Build the "zipapp" target. This will create a `dist/jlmkr` tool which is the direct descendant of Jip-Hop's original `jlmkr.py` script. + + hatch build -t zipapp + +Now build the "appzip" target. This bundles the tool, `README.md` and `LICENSING` into a downloadable zip archive. + + hatch build -t appzip + +If you make any changes *to the embedded builder plugins* that perform the above, then you will need to clear caches between builds. Otherwise and generally, you will not need to do so. + + hatch env prune + +Hatch has oodles more features yet to be explored, such as: automated testing, code coverage, and style checking. For now, we've gotten it building. ## Filing Issues and Community Support diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..2088b9d --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +[project] +name = "jlmkr" +description = "Build and manage jails on TrueNAS SCALE" +dynamic = ["version"] +readme = "README.md" +classifiers = [ + "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", + "Private :: Do Not Upload", +] +authors = [ + {name = "Jip-Hop"}, + {name = "Lockszmith-GH"}, + {name = "jonct"}, +] +maintainers = [ + {name = "Jip-Hop"}, +] +requires-python = ">=3.8" + +[project.urls] +GitHub = "https://github.com/Jip-Hop/jailmaker" + +[build-system] +requires = [ + "hatchling", +# "hatch-zipapp @ file:src/builder/app", +# "hatch-appzip @ file:src/builder/zip", +] +build-backend = "hatchling.build" + +[project.scripts] +jlmkr = "jlmkr.donor:main" + +[tool.hatch.version] +path = "src/jlmkr/__about__.py" # or source = "vcs" + +[tool.hatch.build.zipapp] +dependencies = ["hatch-zipapp-builder @ file:src/builder/app"] + +[tool.hatch.build.appzip] +dependencies = ["hatch-appzip-builder @ file:src/builder/zip"] + +[tool.hatch.build.targets.custom] +path = "src/builder/builder.py" + +[tool.hatch.env] +requires = [ + "hatch-zipapp @ file:src/builder/app", + "hatch-appzip @ file:src/builder/zip", +] + +[[tool.hatch.envs.test.matrix]] +python = [ + "3.11", # TrueNAS SCALE 24.04 Dragonfish +] + +[tool.hatch.envs.types] +extra-dependencies = [ + "mypy>=1.0.0", +] +[tool.hatch.envs.types.scripts] +check = "mypy --install-types --non-interactive {args:src/jlmkr tests}" + +[tool.coverage.run] +source_pkgs = ["jlmkr", "tests"] +branch = true +parallel = true +omit = [ + "src/jlmkr/__about__.py", +] + +[tool.coverage.paths] +jlmkr = ["src/jlmkr"] +tests = ["src/tests"] + +[tool.coverage.report] +exclude_lines = [ + "no cov", + "if __name__ == .__main__.:", + "if TYPE_CHECKING:", +] diff --git a/src/builder/app/__init__.py b/src/builder/app/__init__.py new file mode 100644 index 0000000..4662c4f --- /dev/null +++ b/src/builder/app/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only diff --git a/src/builder/app/build_app.py b/src/builder/app/build_app.py new file mode 100644 index 0000000..22675ed --- /dev/null +++ b/src/builder/app/build_app.py @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os +from io import BytesIO +from pathlib import Path +from typing import Any, Callable, Iterable +from zipapp import create_archive + +from hatchling.builders.plugin.interface import BuilderInterface + + +class ZipAppBuilder(BuilderInterface): + PLUGIN_NAME = "zipapp" + + def get_version_api(self) -> dict[str, Callable[..., str]]: + return {"standard": self.build_standard} + + def clean(self, directory: str, versions: Iterable[str]) -> None: + try: + os.remove(Path(directory, 'jlmkr')) + except: + pass + + def build_standard(self, directory: str, **build_data: Any) -> str: + + # generate zipapp source archive + pyzbuffer = BytesIO() + create_archive('src/jlmkr', target=pyzbuffer, + interpreter='=PLACEHOLDER=', +# main='donor.jlmkr:main', + compressed=True) + zipdata = pyzbuffer.getvalue() #.removeprefix(b"#!=PLACEHOLDER=\n") + + # output with preamble + outpath = Path(directory, 'jlmkr') + with open(outpath, 'wb') as f: + f.write(preamble(self.metadata.version).encode()) + f.write(zipdata) + os.chmod(outpath, 0o755) + return os.fspath(outpath) + + +# 10 lines will conveniently match the default of head(1) +def preamble(version): return f'''#!/usr/bin/env python3 + +jlmkr {version} + +Persistent Linux 'jails' on TrueNAS SCALE to install software (k3s, docker, portainer, podman, etc.) with full access to all files via bind mounts. + +SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +SPDX-License-Identifier: LGPL-3.0-only + +-=-=-=- this is a zip file -=-=-=- what follows is binary -=-=-=- +''' diff --git a/src/builder/app/hooks_app.py b/src/builder/app/hooks_app.py new file mode 100644 index 0000000..beb4253 --- /dev/null +++ b/src/builder/app/hooks_app.py @@ -0,0 +1,11 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from hatchling.plugin import hookimpl + +from build_app import ZipAppBuilder + +@hookimpl +def hatch_register_builder(): + return ZipAppBuilder diff --git a/src/builder/app/pyproject.toml b/src/builder/app/pyproject.toml new file mode 100644 index 0000000..8af91bc --- /dev/null +++ b/src/builder/app/pyproject.toml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +[build-system] +build-backend = "hatchling.build" +requires = ["hatchling"] + +[project] +name = "hatch-zipapp" +version = "0.0.dev0" +classifiers = [ + "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", + "Private :: Do Not Upload", +] +dependencies = ["hatchling"] + +[project.entry-points.hatch] +zipapp = "hooks_app" + +[tool.hatch.build.targets.wheel] +packages = ["."] diff --git a/src/builder/zip/__init__.py b/src/builder/zip/__init__.py new file mode 100644 index 0000000..4662c4f --- /dev/null +++ b/src/builder/zip/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only diff --git a/src/builder/zip/build_zip.py b/src/builder/zip/build_zip.py new file mode 100644 index 0000000..6c982a5 --- /dev/null +++ b/src/builder/zip/build_zip.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +# hat tip: + +import os +from hatchling.builders.config import BuilderConfig +from hatchling.builders.plugin.interface import BuilderInterface +from hatchling.builders.plugin.interface import IncludedFile +from hatchling.builders.utils import normalize_relative_path +from pathlib import Path +from typing import Any, Callable, Iterable +from zipfile import ZipFile, ZIP_DEFLATED + + +class AppZipBuilderConfig(BuilderConfig): pass + +class AppZipBuilder(BuilderInterface): + PLUGIN_NAME = "appzip" + + @classmethod + def get_config_class(cls): + return AppZipBuilderConfig + + def get_version_api(self) -> dict[str, Callable[..., str]]: + return {'standard': self.build_standard} + + def clean(self, directory: str, versions: Iterable[str]) -> None: + for filename in os.listdir(directory): + if filename.startswith('jlmkr-') and filename.endswith('.zip'): + os.remove(Path(directory, filename)) + + def build_standard(self, directory: str, **build_data: Any) -> str: + outpath = Path(directory, f'jlmkr-{self.metadata.version}.zip') + with ZipFile(outpath, 'w') as zip: + zip.write(Path(directory, 'jlmkr'), 'jlmkr') + force_map = build_data['force_include'] + for included_file in self.recurse_forced_files(force_map): + zip.write( + included_file.relative_path, + included_file.distribution_path, + ZIP_DEFLATED) + return os.fspath(outpath) + + def get_default_build_data(self) -> dict[str, Any]: + build_data: dict[str, Any] = super().get_default_build_data() + + extra_files = [] + if self.metadata.core.readme_path: + extra_files.append(self.metadata.core.readme_path) + if self.metadata.core.license_files: + extra_files.extend(self.metadata.core.license_files) + + force_include = build_data.setdefault("force_include", {}) + for fn in map(normalize_relative_path, extra_files): + force_include[os.path.join(self.root, fn)] = Path(fn).name + build_data['force_include'] = force_include + + return build_data diff --git a/src/builder/zip/hooks_zip.py b/src/builder/zip/hooks_zip.py new file mode 100644 index 0000000..5bd8a60 --- /dev/null +++ b/src/builder/zip/hooks_zip.py @@ -0,0 +1,11 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from hatchling.plugin import hookimpl + +from build_zip import AppZipBuilder + +@hookimpl +def hatch_register_builder(): + return AppZipBuilder diff --git a/src/builder/zip/pyproject.toml b/src/builder/zip/pyproject.toml new file mode 100644 index 0000000..1aa6674 --- /dev/null +++ b/src/builder/zip/pyproject.toml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +[build-system] +build-backend = "hatchling.build" +requires = ["hatchling"] + +[project] +name = "hatch-appzip" +version = "0.0.dev0" +classifiers = [ + "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", + "Private :: Do Not Upload", +] +dependencies = ["hatchling"] + +[project.entry-points.hatch] +appzip = "hooks_zip" + +[tool.hatch.build.targets.wheel] +packages = ["."] diff --git a/src/jlmkr/__about__.py b/src/jlmkr/__about__.py new file mode 100644 index 0000000..8ac8fa5 --- /dev/null +++ b/src/jlmkr/__about__.py @@ -0,0 +1,12 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +__version__ = "3.0.0.dev1" + +__author__ = "Jip-Hop" +__copyright__ = "Copyright © 2023, Jip-Hop and the Jailmakers" +__license__ = "LGPL-3.0-only" + +__disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK! +IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.""" diff --git a/src/jlmkr/__main__.py b/src/jlmkr/__main__.py new file mode 100644 index 0000000..0c3eb92 --- /dev/null +++ b/src/jlmkr/__main__.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import donor +import sys + +if __name__ == "__main__": + try: + sys.exit(donor.main()) + except KeyboardInterrupt: + sys.exit(130) diff --git a/src/jlmkr/donor/__init__.py b/src/jlmkr/donor/__init__.py new file mode 100644 index 0000000..de56c40 --- /dev/null +++ b/src/jlmkr/donor/__init__.py @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from .jlmkr import main + +#### +# +# Transitional, as we split up jlmkr.py into smaller components +# +# +# +#### diff --git a/jlmkr.py b/src/jlmkr/donor/jlmkr.py similarity index 100% rename from jlmkr.py rename to src/jlmkr/donor/jlmkr.py From de5223f782ea97408ea9c604fef81da3518eb748 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 22:48:07 -0400 Subject: [PATCH 03/58] Shift script path awareness to the exterior zipapp --- src/jlmkr/donor/jlmkr.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 42ee4fe..2a16e12 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -117,10 +117,13 @@ DOWNLOAD_SCRIPT_DIGEST = ( "cfcb5d08b24187d108f2ab0d21a6cc4b73dcd7f5d7dfc80803bfd7f1642d638d" ) -SCRIPT_PATH = os.path.realpath(__file__) +ZIPAPP_PATH = os.path.realpath(__file__) +while not os.path.exists(ZIPAPP_PATH): + ZIPAPP_PATH = os.path.dirname(ZIPAPP_PATH) +SCRIPT_PATH = os.path.realpath(ZIPAPP_PATH) SCRIPT_NAME = os.path.basename(SCRIPT_PATH) SCRIPT_DIR_PATH = os.path.dirname(SCRIPT_PATH) -COMMAND_NAME = os.path.basename(__file__) +COMMAND_NAME = os.path.basename(ZIPAPP_PATH) JAILS_DIR_PATH = os.path.join(SCRIPT_DIR_PATH, "jails") JAIL_CONFIG_NAME = "config" JAIL_ROOTFS_NAME = "rootfs" From 351cef783d635fa7a6f500fd1037bb45748f4f73 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 23:06:23 -0400 Subject: [PATCH 04/58] Extract the config file parser --- src/jlmkr/donor/jlmkr.py | 132 +--------------------------- src/jlmkr/utils/KeyValueParser.py | 138 ++++++++++++++++++++++++++++++ src/jlmkr/utils/__init__.py | 3 + 3 files changed, 142 insertions(+), 131 deletions(-) create mode 100644 src/jlmkr/utils/KeyValueParser.py create mode 100644 src/jlmkr/utils/__init__.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 2a16e12..c0ee093 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -12,10 +12,8 @@ IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.""" import argparse -import configparser import contextlib import hashlib -import io import json import os import platform @@ -141,135 +139,7 @@ DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" -# Used in parser getters to indicate the default behavior when a specific -# option is not found. Created to enable `None` as a valid fallback value. -_UNSET = object() - - -class KeyValueParser(configparser.ConfigParser): - """Simple comment preserving parser based on ConfigParser. - Reads a file containing key/value pairs and/or comments. - Values can span multiple lines, as long as they are indented - deeper than the first line of the value. Comments or keys - must NOT be indented. - """ - - def __init__(self, *args, **kwargs): - # Set defaults if not specified by user - if "interpolation" not in kwargs: - kwargs["interpolation"] = None - if "allow_no_value" not in kwargs: - kwargs["allow_no_value"] = True - if "comment_prefixes" not in kwargs: - kwargs["comment_prefixes"] = "#" - - super().__init__(*args, **kwargs) - - # Backup _comment_prefixes - self._comment_prefixes_backup = self._comment_prefixes - # Unset _comment_prefixes so comments won't be skipped - self._comment_prefixes = () - # Starting point for the comment IDs - self._comment_id = 0 - # Default delimiter to use - delimiter = self._delimiters[0] - # Template to store comments as key value pair - self._comment_template = "#{0} " + delimiter + " {1}" - # Regex to match the comment prefix - self._comment_regex = re.compile( - r"^#\d+\s*" + re.escape(delimiter) + r"[^\S\n]*" - ) - # Regex to match cosmetic newlines (skips newlines in multiline values): - # consecutive whitespace from start of line followed by a line not starting with whitespace - self._cosmetic_newlines_regex = re.compile(r"^(\s+)(?=^\S)", re.MULTILINE) - # Dummy section name - self._section_name = "a" - - def _find_cosmetic_newlines(self, text): - # Indices of the lines containing cosmetic newlines - cosmetic_newline_indices = set() - for match in re.finditer(self._cosmetic_newlines_regex, text): - start_index = text.count("\n", 0, match.start()) - end_index = start_index + text.count("\n", match.start(), match.end()) - cosmetic_newline_indices.update(range(start_index, end_index)) - - return cosmetic_newline_indices - - # TODO: can I create a solution which not depends on the internal _read method? - def _read(self, fp, fpname): - lines = fp.readlines() - cosmetic_newline_indices = self._find_cosmetic_newlines("".join(lines)) - # Preprocess config file to preserve comments - for i, line in enumerate(lines): - if i in cosmetic_newline_indices or line.startswith( - self._comment_prefixes_backup - ): - # Store cosmetic newline or comment with unique key - lines[i] = self._comment_template.format(self._comment_id, line) - self._comment_id += 1 - - # Convert to in-memory file and prepend a dummy section header - lines = io.StringIO(f"[{self._section_name}]\n" + "".join(lines)) - # Feed preprocessed file to original _read method - return super()._read(lines, fpname) - - def read_default_string(self, string, source=""): - # Ignore all comments when parsing default key/values - string = "\n".join( - [ - line - for line in string.splitlines() - if not line.startswith(self._comment_prefixes_backup) - ] - ) - # Feed preprocessed file to original _read method - return super()._read(io.StringIO("[DEFAULT]\n" + string), source) - - def write(self, fp, space_around_delimiters=False): - # Write the config to an in-memory file - with io.StringIO() as sfile: - super().write(sfile, space_around_delimiters) - # Start from the beginning of sfile - sfile.seek(0) - - line = sfile.readline() - # Throw away lines until we reach the dummy section header - while line.strip() != f"[{self._section_name}]": - line = sfile.readline() - - lines = sfile.readlines() - - for i, line in enumerate(lines): - # Remove the comment id prefix - lines[i] = self._comment_regex.sub("", line, 1) - - fp.write("".join(lines).rstrip()) - - # Set value for specified option key - def my_set(self, option, value): - if isinstance(value, bool): - value = str(int(value)) - elif isinstance(value, list): - value = str("\n ".join(value)) - elif not isinstance(value, str): - value = str(value) - - super().set(self._section_name, option, value) - - # Return value for specified option key - def my_get(self, option, fallback=_UNSET): - return super().get(self._section_name, option, fallback=fallback) - - # Return value converted to boolean for specified option key - def my_getboolean(self, option, fallback=_UNSET): - return super().getboolean(self._section_name, option, fallback=fallback) - - -class ExceptionWithParser(Exception): - def __init__(self, parser, message): - self.parser = parser - self.message = message - super().__init__(message) +from utils.KeyValueParser import KeyValueParser, ExceptionWithParser # Workaround for exit_on_error=False not applying to: diff --git a/src/jlmkr/utils/KeyValueParser.py b/src/jlmkr/utils/KeyValueParser.py new file mode 100644 index 0000000..5adb62b --- /dev/null +++ b/src/jlmkr/utils/KeyValueParser.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import configparser +import io +import re + + +# Used in parser getters to indicate the default behavior when a specific +# option is not found. Created to enable `None` as a valid fallback value. +_UNSET = object() + + +class KeyValueParser(configparser.ConfigParser): + """Simple comment preserving parser based on ConfigParser. + Reads a file containing key/value pairs and/or comments. + Values can span multiple lines, as long as they are indented + deeper than the first line of the value. Comments or keys + must NOT be indented. + """ + + def __init__(self, *args, **kwargs): + # Set defaults if not specified by user + if "interpolation" not in kwargs: + kwargs["interpolation"] = None + if "allow_no_value" not in kwargs: + kwargs["allow_no_value"] = True + if "comment_prefixes" not in kwargs: + kwargs["comment_prefixes"] = "#" + + super().__init__(*args, **kwargs) + + # Backup _comment_prefixes + self._comment_prefixes_backup = self._comment_prefixes + # Unset _comment_prefixes so comments won't be skipped + self._comment_prefixes = () + # Starting point for the comment IDs + self._comment_id = 0 + # Default delimiter to use + delimiter = self._delimiters[0] + # Template to store comments as key value pair + self._comment_template = "#{0} " + delimiter + " {1}" + # Regex to match the comment prefix + self._comment_regex = re.compile( + r"^#\d+\s*" + re.escape(delimiter) + r"[^\S\n]*" + ) + # Regex to match cosmetic newlines (skips newlines in multiline values): + # consecutive whitespace from start of line followed by a line not starting with whitespace + self._cosmetic_newlines_regex = re.compile(r"^(\s+)(?=^\S)", re.MULTILINE) + # Dummy section name + self._section_name = "a" + + def _find_cosmetic_newlines(self, text): + # Indices of the lines containing cosmetic newlines + cosmetic_newline_indices = set() + for match in re.finditer(self._cosmetic_newlines_regex, text): + start_index = text.count("\n", 0, match.start()) + end_index = start_index + text.count("\n", match.start(), match.end()) + cosmetic_newline_indices.update(range(start_index, end_index)) + + return cosmetic_newline_indices + + # TODO: can I create a solution which not depends on the internal _read method? + def _read(self, fp, fpname): + lines = fp.readlines() + cosmetic_newline_indices = self._find_cosmetic_newlines("".join(lines)) + # Preprocess config file to preserve comments + for i, line in enumerate(lines): + if i in cosmetic_newline_indices or line.startswith( + self._comment_prefixes_backup + ): + # Store cosmetic newline or comment with unique key + lines[i] = self._comment_template.format(self._comment_id, line) + self._comment_id += 1 + + # Convert to in-memory file and prepend a dummy section header + lines = io.StringIO(f"[{self._section_name}]\n" + "".join(lines)) + # Feed preprocessed file to original _read method + return super()._read(lines, fpname) + + def read_default_string(self, string, source=""): + # Ignore all comments when parsing default key/values + string = "\n".join( + [ + line + for line in string.splitlines() + if not line.startswith(self._comment_prefixes_backup) + ] + ) + # Feed preprocessed file to original _read method + return super()._read(io.StringIO("[DEFAULT]\n" + string), source) + + def write(self, fp, space_around_delimiters=False): + # Write the config to an in-memory file + with io.StringIO() as sfile: + super().write(sfile, space_around_delimiters) + # Start from the beginning of sfile + sfile.seek(0) + + line = sfile.readline() + # Throw away lines until we reach the dummy section header + while line.strip() != f"[{self._section_name}]": + line = sfile.readline() + + lines = sfile.readlines() + + for i, line in enumerate(lines): + # Remove the comment id prefix + lines[i] = self._comment_regex.sub("", line, 1) + + fp.write("".join(lines).rstrip()) + + # Set value for specified option key + def my_set(self, option, value): + if isinstance(value, bool): + value = str(int(value)) + elif isinstance(value, list): + value = str("\n ".join(value)) + elif not isinstance(value, str): + value = str(value) + + super().set(self._section_name, option, value) + + # Return value for specified option key + def my_get(self, option, fallback=_UNSET): + return super().get(self._section_name, option, fallback=fallback) + + # Return value converted to boolean for specified option key + def my_getboolean(self, option, fallback=_UNSET): + return super().getboolean(self._section_name, option, fallback=fallback) + + +class ExceptionWithParser(Exception): + def __init__(self, parser, message): + self.parser = parser + self.message = message + super().__init__(message) diff --git a/src/jlmkr/utils/__init__.py b/src/jlmkr/utils/__init__.py new file mode 100644 index 0000000..4662c4f --- /dev/null +++ b/src/jlmkr/utils/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only From 34f2164715051000a89dba8ef0d54aab42c8f2a4 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 23:25:12 -0400 Subject: [PATCH 05/58] Name modules in snake_case Per PEP 8 --- src/jlmkr/donor/jlmkr.py | 2 +- src/jlmkr/utils/{KeyValueParser.py => config_parser.py} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename src/jlmkr/utils/{KeyValueParser.py => config_parser.py} (100%) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index c0ee093..a0efbde 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -139,7 +139,7 @@ DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" -from utils.KeyValueParser import KeyValueParser, ExceptionWithParser +from utils.config_parser import KeyValueParser, ExceptionWithParser # Workaround for exit_on_error=False not applying to: diff --git a/src/jlmkr/utils/KeyValueParser.py b/src/jlmkr/utils/config_parser.py similarity index 100% rename from src/jlmkr/utils/KeyValueParser.py rename to src/jlmkr/utils/config_parser.py From 18150bf9ae03d18f3dc1eac38cb97a8b1b8f2624 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 23:43:42 -0400 Subject: [PATCH 06/58] Bypass root ownership check by setting JLMKR_DEBUG --- src/jlmkr/donor/jlmkr.py | 1 + 1 file changed, 1 insertion(+) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index a0efbde..d6c54a7 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -1463,6 +1463,7 @@ def add_parser(subparser, **kwargs): def main(): if os.stat(SCRIPT_PATH).st_uid != 0: + if os.environ.get('JLMKR_DEBUG') is not None: fail( f"This script should be owned by the root user... Fix it manually with: `chown root {SCRIPT_PATH}`." ) From c155404fed780ff1c709df5a2b8eddfe93828669 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Sun, 14 Jul 2024 23:44:54 -0400 Subject: [PATCH 07/58] Run basic lint/style check during build action --- .github/workflows/build.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index f82226b..118e1cf 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -40,9 +40,12 @@ jobs: - name: Install Hatch uses: pypa/hatch@install + + - name: Run style check + run: hatch fmt --check - - name: Run unit tests - run: hatch run +py=${{ matrix.python-version }} test:test +# - name: Run unit tests +# run: hatch run +py=${{ matrix.python-version }} test:test - name: Build distribution run: hatch build -t zipapp -t appzip From b25c9642314ed8b7733876c3cca26b49cc06682c Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 00:35:31 -0400 Subject: [PATCH 08/58] Extract jail dataset utils --- src/jlmkr/donor/jlmkr.py | 29 ++----------------------- src/jlmkr/utils/jail_dataset.py | 38 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 27 deletions(-) create mode 100644 src/jlmkr/utils/jail_dataset.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index d6c54a7..6160553 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -187,16 +187,7 @@ def fail(*args, **kwargs): sys.exit(1) -def get_jail_path(jail_name): - return os.path.join(JAILS_DIR_PATH, jail_name) - - -def get_jail_config_path(jail_name): - return os.path.join(get_jail_path(jail_name), JAIL_CONFIG_NAME) - - -def get_jail_rootfs_path(jail_name): - return os.path.join(get_jail_path(jail_name), JAIL_ROOTFS_NAME) +from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path # Test intel GPU by decoding mp4 file (output is discarded) @@ -1318,23 +1309,6 @@ def get_all_jail_names(): return jail_names -def parse_os_release(new_root): - result = {} - with Chroot(new_root): - # Use chroot to correctly resolve os-release symlink (for nixos) - for candidate in ["/etc/os-release", "/usr/lib/os-release"]: - try: - with open(candidate, encoding="utf-8") as f: - # TODO: can I create a solution which not depends on the internal _parse_os_release method? - result = platform._parse_os_release(f) - break - except OSError: - # Silently ignore failing to read os release info - pass - - return result - - def list_jails(): """ List all available and running jails. @@ -1409,6 +1383,7 @@ def list_jails(): ) return 0 +from utils.jail_dataset import parse_os_release def startup_jails(): diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/jail_dataset.py new file mode 100644 index 0000000..cc71edf --- /dev/null +++ b/src/jlmkr/utils/jail_dataset.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os.path +import platform + + +from donor.jlmkr import Chroot, JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME + + +def get_jail_path(jail_name): + return os.path.join(JAILS_DIR_PATH, jail_name) + + +def get_jail_config_path(jail_name): + return os.path.join(get_jail_path(jail_name), JAIL_CONFIG_NAME) + + +def get_jail_rootfs_path(jail_name): + return os.path.join(get_jail_path(jail_name), JAIL_ROOTFS_NAME) + + +def parse_os_release(new_root): + result = {} + with Chroot(new_root): + # Use chroot to correctly resolve os-release symlink (for nixos) + for candidate in ["/etc/os-release", "/usr/lib/os-release"]: + try: + with open(candidate, encoding="utf-8") as f: + # TODO: can I create a solution which not depends on the internal _parse_os_release method? + result = platform._parse_os_release(f) + break + except OSError: + # Silently ignore failing to read os release info + pass + + return result From 2718c89613be16c74d1cb94e033f8805719b5225 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 00:44:44 -0400 Subject: [PATCH 09/58] Extract more config parser --- src/jlmkr/donor/jlmkr.py | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 6160553..8995555 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -139,7 +139,8 @@ DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" -from utils.config_parser import KeyValueParser, ExceptionWithParser +from utils.config_parser import ExceptionWithParser, KeyValueParser +from utils.config_parser import parse_config_file # Workaround for exit_on_error=False not applying to: @@ -381,20 +382,6 @@ def shell_jail(args): return subprocess.run(["machinectl", "shell"] + args).returncode -def parse_config_file(jail_config_path): - config = KeyValueParser() - # Read default config to fallback to default values - # for keys not found in the jail_config_path file - config.read_default_string(DEFAULT_CONFIG) - try: - with open(jail_config_path, "r") as fp: - config.read_file(fp) - return config - except FileNotFoundError: - eprint(f"Unable to find config file: {jail_config_path}.") - return - - def systemd_escape_path(path): """ Escape path containing spaces, while properly handling backslashes in filenames. @@ -1383,6 +1370,7 @@ def list_jails(): ) return 0 +from utils.parent_dataset import get_all_jail_names from utils.jail_dataset import parse_os_release From a6c5ebbb0c26b4dddfc152e66b65051e6d737a5d Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 00:49:53 -0400 Subject: [PATCH 10/58] Extract more config parser --- src/jlmkr/utils/config_parser.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/jlmkr/utils/config_parser.py b/src/jlmkr/utils/config_parser.py index 5adb62b..9264ab4 100644 --- a/src/jlmkr/utils/config_parser.py +++ b/src/jlmkr/utils/config_parser.py @@ -6,6 +6,8 @@ import io import re +from donor.jlmkr import DEFAULT_CONFIG + # Used in parser getters to indicate the default behavior when a specific # option is not found. Created to enable `None` as a valid fallback value. @@ -136,3 +138,17 @@ def __init__(self, parser, message): self.parser = parser self.message = message super().__init__(message) + + +def parse_config_file(jail_config_path): + config = KeyValueParser() + # Read default config to fallback to default values + # for keys not found in the jail_config_path file + config.read_default_string(DEFAULT_CONFIG) + try: + with open(jail_config_path, "r") as fp: + config.read_file(fp) + return config + except FileNotFoundError: + eprint(f"Unable to find config file: {jail_config_path}.") + return From 461cdf7cac5bc3ba83c9af3164e4d33fa6b05937 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 00:51:17 -0400 Subject: [PATCH 11/58] Extract parent dataset utils --- src/jlmkr/donor/jlmkr.py | 9 --------- src/jlmkr/utils/parent_dataset.py | 16 ++++++++++++++++ 2 files changed, 16 insertions(+), 9 deletions(-) create mode 100644 src/jlmkr/utils/parent_dataset.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 8995555..62543ec 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -1287,15 +1287,6 @@ def run_command_and_parse_json(command): return None -def get_all_jail_names(): - try: - jail_names = os.listdir(JAILS_DIR_PATH) - except FileNotFoundError: - jail_names = [] - - return jail_names - - def list_jails(): """ List all available and running jails. diff --git a/src/jlmkr/utils/parent_dataset.py b/src/jlmkr/utils/parent_dataset.py new file mode 100644 index 0000000..2dce34a --- /dev/null +++ b/src/jlmkr/utils/parent_dataset.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os + +from donor.jlmkr import JAILS_DIR_PATH + + +def get_all_jail_names(): + try: + jail_names = os.listdir(JAILS_DIR_PATH) + except FileNotFoundError: + jail_names = [] + + return jail_names From f2a020f1a848b391a265c37ebdcd2342847a3ff2 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 00:59:30 -0400 Subject: [PATCH 12/58] Extract console utils --- src/jlmkr/actions/list.py | 119 +++++++++++++++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 26 +------- src/jlmkr/utils/console.py | 31 ++++++++++ 3 files changed, 152 insertions(+), 24 deletions(-) create mode 100644 src/jlmkr/actions/list.py create mode 100644 src/jlmkr/utils/console.py diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py new file mode 100644 index 0000000..04df74c --- /dev/null +++ b/src/jlmkr/actions/list.py @@ -0,0 +1,119 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import json +import subprocess + +from collections import defaultdict +from utils.config_parser import parse_config_file +from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path, parse_os_release +from utils.parent_dataset import get_all_jail_names + + +def list_jails(): + """ + List all available and running jails. + """ + + jails = {} + empty_value_indicator = "-" + + jail_names = get_all_jail_names() + + if not jail_names: + print("No jails.") + return 0 + + # Get running jails from machinectl + running_machines = run_command_and_parse_json(["machinectl", "list", "-o", "json"]) + # Index running_machines by machine name + # We're only interested in systemd-nspawn machines + running_machines = { + item["machine"]: item + for item in running_machines + if item["service"] == "systemd-nspawn" + } + + for jail_name in jail_names: + jail_rootfs_path = get_jail_rootfs_path(jail_name) + jails[jail_name] = {"name": jail_name, "running": False} + jail = jails[jail_name] + + config = parse_config_file(get_jail_config_path(jail_name)) + if config: + jail["startup"] = config.my_getboolean("startup") + jail["gpu_intel"] = config.my_getboolean("gpu_passthrough_intel") + jail["gpu_nvidia"] = config.my_getboolean("gpu_passthrough_nvidia") + + if jail_name in running_machines: + machine = running_machines[jail_name] + # Augment the jails dict with output from machinectl + jail["running"] = True + jail["os"] = machine["os"] or None + jail["version"] = machine["version"] or None + + addresses = machine.get("addresses") + if not addresses: + jail["addresses"] = empty_value_indicator + else: + addresses = addresses.split("\n") + jail["addresses"] = addresses[0] + if len(addresses) > 1: + jail["addresses"] += "…" + else: + # Parse os-release info ourselves + jail_platform = parse_os_release(jail_rootfs_path) + jail["os"] = jail_platform.get("ID") + jail["version"] = jail_platform.get("VERSION_ID") or jail_platform.get( + "VERSION_CODENAME" + ) + + print_table( + [ + "name", + "running", + "startup", + "gpu_intel", + "gpu_nvidia", + "os", + "version", + "addresses", + ], + sorted(jails.values(), key=lambda x: x["name"]), + empty_value_indicator, + ) + + return 0 + + +def run_command_and_parse_json(command): + result = subprocess.run(command, capture_output=True, text=True) + output = result.stdout.strip() + + try: + parsed_output = json.loads(output) + return parsed_output + except json.JSONDecodeError as e: + eprint(f"Error parsing JSON: {e}") + return None + + +def print_table(header, list_of_objects, empty_value_indicator): + # Find max width for each column + widths = defaultdict(int) + for obj in list_of_objects: + for hdr in header: + value = obj.get(hdr) + if value is None: + obj[hdr] = value = empty_value_indicator + widths[hdr] = max(widths[hdr], len(str(value)), len(str(hdr))) + + # Print header + print( + UNDERLINE + " ".join(hdr.upper().ljust(widths[hdr]) for hdr in header) + NORMAL + ) + + # Print rows + for obj in list_of_objects: + print(" ".join(str(obj.get(hdr)).ljust(widths[hdr]) for hdr in header)) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 62543ec..b8c510a 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -127,15 +127,7 @@ JAIL_ROOTFS_NAME = "rootfs" SHORTNAME = "jlmkr" -# Only set a color if we have an interactive tty -if sys.stdout.isatty(): - BOLD = "\033[1m" - RED = "\033[91m" - YELLOW = "\033[93m" - UNDERLINE = "\033[4m" - NORMAL = "\033[0m" -else: - BOLD = RED = YELLOW = UNDERLINE = NORMAL = "" +from utils.console import BOLD, RED, YELLOW, UNDERLINE, NORMAL DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" @@ -173,21 +165,7 @@ def __exit__(self, exc_type, exc_value, traceback): os.chdir(self.initial_cwd) -def eprint(*args, **kwargs): - """ - Print to stderr. - """ - print(*args, file=sys.stderr, **kwargs) - - -def fail(*args, **kwargs): - """ - Print to stderr and exit. - """ - eprint(*args, **kwargs) - sys.exit(1) - - +from utils.console import eprint, fail from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path diff --git a/src/jlmkr/utils/console.py b/src/jlmkr/utils/console.py new file mode 100644 index 0000000..20018f7 --- /dev/null +++ b/src/jlmkr/utils/console.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import sys + + +# Only set a color if we have an interactive tty +if sys.stdout.isatty(): + BOLD = "\033[1m" + RED = "\033[91m" + YELLOW = "\033[93m" + UNDERLINE = "\033[4m" + NORMAL = "\033[0m" +else: + BOLD = RED = YELLOW = UNDERLINE = NORMAL = "" + + +def eprint(*args, **kwargs): + """ + Print to stderr. + """ + print(*args, file=sys.stderr, **kwargs) + + +def fail(*args, **kwargs): + """ + Print to stderr and exit. + """ + eprint(*args, **kwargs) + sys.exit(1) From 1461af8598b62f67fb4f89a5849f848696724d78 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:00:14 -0400 Subject: [PATCH 13/58] Extract list action --- src/jlmkr/actions/__init__.py | 3 + src/jlmkr/actions/list.py | 1 + src/jlmkr/donor/jlmkr.py | 108 +--------------------------------- 3 files changed, 5 insertions(+), 107 deletions(-) create mode 100644 src/jlmkr/actions/__init__.py diff --git a/src/jlmkr/actions/__init__.py b/src/jlmkr/actions/__init__.py new file mode 100644 index 0000000..4662c4f --- /dev/null +++ b/src/jlmkr/actions/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py index 04df74c..6b3ca4b 100644 --- a/src/jlmkr/actions/list.py +++ b/src/jlmkr/actions/list.py @@ -6,6 +6,7 @@ import subprocess from collections import defaultdict +from utils.console import NORMAL, UNDERLINE from utils.config_parser import parse_config_file from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path, parse_os_release from utils.parent_dataset import get_all_jail_names diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index b8c510a..88a3ccd 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -26,7 +26,6 @@ import tempfile import time import urllib.request -from collections import defaultdict from inspect import cleandoc from pathlib import Path, PurePath from textwrap import dedent @@ -1233,114 +1232,9 @@ def remove_jail(jail_name): return 1 -def print_table(header, list_of_objects, empty_value_indicator): - # Find max width for each column - widths = defaultdict(int) - for obj in list_of_objects: - for hdr in header: - value = obj.get(hdr) - if value is None: - obj[hdr] = value = empty_value_indicator - widths[hdr] = max(widths[hdr], len(str(value)), len(str(hdr))) - - # Print header - print( - UNDERLINE + " ".join(hdr.upper().ljust(widths[hdr]) for hdr in header) + NORMAL - ) - - # Print rows - for obj in list_of_objects: - print(" ".join(str(obj.get(hdr)).ljust(widths[hdr]) for hdr in header)) - - -def run_command_and_parse_json(command): - result = subprocess.run(command, capture_output=True, text=True) - output = result.stdout.strip() - - try: - parsed_output = json.loads(output) - return parsed_output - except json.JSONDecodeError as e: - eprint(f"Error parsing JSON: {e}") - return None - - -def list_jails(): - """ - List all available and running jails. - """ - - jails = {} - empty_value_indicator = "-" - - jail_names = get_all_jail_names() - - if not jail_names: - print("No jails.") - return 0 - - # Get running jails from machinectl - running_machines = run_command_and_parse_json(["machinectl", "list", "-o", "json"]) - # Index running_machines by machine name - # We're only interested in systemd-nspawn machines - running_machines = { - item["machine"]: item - for item in running_machines - if item["service"] == "systemd-nspawn" - } - - for jail_name in jail_names: - jail_rootfs_path = get_jail_rootfs_path(jail_name) - jails[jail_name] = {"name": jail_name, "running": False} - jail = jails[jail_name] - - config = parse_config_file(get_jail_config_path(jail_name)) - if config: - jail["startup"] = config.my_getboolean("startup") - jail["gpu_intel"] = config.my_getboolean("gpu_passthrough_intel") - jail["gpu_nvidia"] = config.my_getboolean("gpu_passthrough_nvidia") - - if jail_name in running_machines: - machine = running_machines[jail_name] - # Augment the jails dict with output from machinectl - jail["running"] = True - jail["os"] = machine["os"] or None - jail["version"] = machine["version"] or None - - addresses = machine.get("addresses") - if not addresses: - jail["addresses"] = empty_value_indicator - else: - addresses = addresses.split("\n") - jail["addresses"] = addresses[0] - if len(addresses) > 1: - jail["addresses"] += "…" - else: - # Parse os-release info ourselves - jail_platform = parse_os_release(jail_rootfs_path) - jail["os"] = jail_platform.get("ID") - jail["version"] = jail_platform.get("VERSION_ID") or jail_platform.get( - "VERSION_CODENAME" - ) - - print_table( - [ - "name", - "running", - "startup", - "gpu_intel", - "gpu_nvidia", - "os", - "version", - "addresses", - ], - sorted(jails.values(), key=lambda x: x["name"]), - empty_value_indicator, - ) - - return 0 from utils.parent_dataset import get_all_jail_names from utils.jail_dataset import parse_os_release +from actions.list import list_jails def startup_jails(): From 36ba561120463e33d4bbf33622774e59df1e2e4b Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:24:34 -0400 Subject: [PATCH 14/58] Extract filesystem path inference utils --- src/jlmkr/donor/jlmkr.py | 14 +++----------- src/jlmkr/utils/jail_dataset.py | 3 ++- src/jlmkr/utils/parent_dataset.py | 2 +- src/jlmkr/utils/paths.py | 22 ++++++++++++++++++++++ 4 files changed, 28 insertions(+), 13 deletions(-) create mode 100755 src/jlmkr/utils/paths.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 88a3ccd..56d98e1 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -114,18 +114,10 @@ DOWNLOAD_SCRIPT_DIGEST = ( "cfcb5d08b24187d108f2ab0d21a6cc4b73dcd7f5d7dfc80803bfd7f1642d638d" ) -ZIPAPP_PATH = os.path.realpath(__file__) -while not os.path.exists(ZIPAPP_PATH): - ZIPAPP_PATH = os.path.dirname(ZIPAPP_PATH) -SCRIPT_PATH = os.path.realpath(ZIPAPP_PATH) -SCRIPT_NAME = os.path.basename(SCRIPT_PATH) -SCRIPT_DIR_PATH = os.path.dirname(SCRIPT_PATH) -COMMAND_NAME = os.path.basename(ZIPAPP_PATH) -JAILS_DIR_PATH = os.path.join(SCRIPT_DIR_PATH, "jails") -JAIL_CONFIG_NAME = "config" -JAIL_ROOTFS_NAME = "rootfs" -SHORTNAME = "jlmkr" +from utils.paths import SCRIPT_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH +from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME +from utils.paths import COMMAND_NAME, SHORTNAME from utils.console import BOLD, RED, YELLOW, UNDERLINE, NORMAL DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/jail_dataset.py index cc71edf..5e536b4 100644 --- a/src/jlmkr/utils/jail_dataset.py +++ b/src/jlmkr/utils/jail_dataset.py @@ -6,7 +6,8 @@ import platform -from donor.jlmkr import Chroot, JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME +from donor.jlmkr import Chroot +from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME def get_jail_path(jail_name): diff --git a/src/jlmkr/utils/parent_dataset.py b/src/jlmkr/utils/parent_dataset.py index 2dce34a..fd68893 100644 --- a/src/jlmkr/utils/parent_dataset.py +++ b/src/jlmkr/utils/parent_dataset.py @@ -4,7 +4,7 @@ import os -from donor.jlmkr import JAILS_DIR_PATH +from utils.paths import JAILS_DIR_PATH def get_all_jail_names(): diff --git a/src/jlmkr/utils/paths.py b/src/jlmkr/utils/paths.py new file mode 100755 index 0000000..bfee658 --- /dev/null +++ b/src/jlmkr/utils/paths.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os.path + + +# When running as a zipapp, the script file is a parent +ZIPAPP_PATH = os.path.realpath(__file__) +while not os.path.exists(ZIPAPP_PATH): + ZIPAPP_PATH = os.path.dirname(ZIPAPP_PATH) + +SCRIPT_PATH = os.path.realpath(ZIPAPP_PATH) +SCRIPT_NAME = os.path.basename(SCRIPT_PATH) +SCRIPT_DIR_PATH = os.path.dirname(SCRIPT_PATH) +COMMAND_NAME = os.path.basename(ZIPAPP_PATH) + +JAILS_DIR_PATH = os.path.join(SCRIPT_DIR_PATH, "jails") +JAIL_CONFIG_NAME = "config" +JAIL_ROOTFS_NAME = "rootfs" + +SHORTNAME = "jlmkr" From b7a6fdd84cf2ff7b1ba43626c01446b6316f8790 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:27:27 -0400 Subject: [PATCH 15/58] Extract chroot utils --- src/jlmkr/donor/jlmkr.py | 20 +------------------- src/jlmkr/utils/chroot.py | 24 ++++++++++++++++++++++++ src/jlmkr/utils/jail_dataset.py | 2 +- 3 files changed, 26 insertions(+), 20 deletions(-) create mode 100644 src/jlmkr/utils/chroot.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 56d98e1..60a90ce 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -137,25 +137,7 @@ def error(self, message): raise ExceptionWithParser(self, message) -class Chroot: - def __init__(self, new_root): - self.new_root = new_root - self.old_root = None - self.initial_cwd = None - - def __enter__(self): - self.old_root = os.open("/", os.O_PATH) - self.initial_cwd = os.path.abspath(os.getcwd()) - os.chdir(self.new_root) - os.chroot(".") - - def __exit__(self, exc_type, exc_value, traceback): - os.chdir(self.old_root) - os.chroot(".") - os.close(self.old_root) - os.chdir(self.initial_cwd) - - +from utils.chroot import Chroot from utils.console import eprint, fail from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path diff --git a/src/jlmkr/utils/chroot.py b/src/jlmkr/utils/chroot.py new file mode 100644 index 0000000..93c9f11 --- /dev/null +++ b/src/jlmkr/utils/chroot.py @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os + + +class Chroot: + def __init__(self, new_root): + self.new_root = new_root + self.old_root = None + self.initial_cwd = None + + def __enter__(self): + self.old_root = os.open("/", os.O_PATH) + self.initial_cwd = os.path.abspath(os.getcwd()) + os.chdir(self.new_root) + os.chroot(".") + + def __exit__(self, exc_type, exc_value, traceback): + os.chdir(self.old_root) + os.chroot(".") + os.close(self.old_root) + os.chdir(self.initial_cwd) diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/jail_dataset.py index 5e536b4..0a0bcb1 100644 --- a/src/jlmkr/utils/jail_dataset.py +++ b/src/jlmkr/utils/jail_dataset.py @@ -6,7 +6,7 @@ import platform -from donor.jlmkr import Chroot +from utils.chroot import Chroot from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME From 673e77e003ad27e46c112300b83e2070ea6b5600 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:53:41 -0400 Subject: [PATCH 16/58] Extract more jail dataset utils --- src/jlmkr/donor/jlmkr.py | 10 +--------- src/jlmkr/utils/jail_dataset.py | 13 ++++++++++++- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 60a90ce..bdfba90 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -1116,15 +1116,7 @@ def create_jail(**kwargs): return 0 -def jail_is_running(jail_name): - return ( - subprocess.run( - ["machinectl", "show", jail_name], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ).returncode - == 0 - ) +from utils.jail_dataset import jail_is_running def edit_jail(jail_name): diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/jail_dataset.py index 0a0bcb1..7cad961 100644 --- a/src/jlmkr/utils/jail_dataset.py +++ b/src/jlmkr/utils/jail_dataset.py @@ -4,7 +4,7 @@ import os.path import platform - +import subprocess from utils.chroot import Chroot from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME @@ -37,3 +37,14 @@ def parse_os_release(new_root): pass return result + + +def jail_is_running(jail_name): + return ( + subprocess.run( + ["machinectl", "show", jail_name], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ).returncode + == 0 + ) From 3a833923c087ce98a11ed26ab1e48d86b1e2a2b6 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:54:39 -0400 Subject: [PATCH 17/58] Extract file utils --- src/jlmkr/donor/jlmkr.py | 7 +------ src/jlmkr/utils/files.py | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 6 deletions(-) create mode 100644 src/jlmkr/utils/files.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index bdfba90..ce3ded9 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -707,12 +707,7 @@ def run_lxc_download_script( return 0 -def stat_chmod(file_path, mode): - """ - Change mode if file doesn't already have this mode. - """ - if mode != stat.S_IMODE(os.stat(file_path).st_mode): - os.chmod(file_path, mode) +from utils.files import stat_chmod def get_mount_point(path): diff --git a/src/jlmkr/utils/files.py b/src/jlmkr/utils/files.py new file mode 100644 index 0000000..a328223 --- /dev/null +++ b/src/jlmkr/utils/files.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os +import stat + + +def stat_chmod(file_path, mode): + """ + Change mode if file doesn't already have this mode. + """ + if mode != stat.S_IMODE(os.stat(file_path).st_mode): + os.chmod(file_path, mode) From c5239749b952590a3472639d3712f3181068dd4c Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:55:15 -0400 Subject: [PATCH 18/58] Extract start action --- src/jlmkr/actions/start.py | 264 +++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 391 +------------------------------------ src/jlmkr/utils/gpu.py | 158 +++++++++++++++ 3 files changed, 423 insertions(+), 390 deletions(-) create mode 100644 src/jlmkr/actions/start.py create mode 100644 src/jlmkr/utils/gpu.py diff --git a/src/jlmkr/actions/start.py b/src/jlmkr/actions/start.py new file mode 100644 index 0000000..abe50d9 --- /dev/null +++ b/src/jlmkr/actions/start.py @@ -0,0 +1,264 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os.path +import shlex +import subprocess +import tempfile + +from pathlib import Path +from textwrap import dedent +from utils.config_parser import parse_config_file +from utils.console import eprint +from utils.files import stat_chmod +from utils.gpu import passthrough_intel, passthrough_nvidia +from utils.jail_dataset import get_jail_path, jail_is_running +from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path +from utils.paths import SHORTNAME, JAIL_ROOTFS_NAME + + +def start_jail(jail_name): + """ + Start jail with given name. + """ + skip_start_message = ( + f"Skipped starting jail {jail_name}. It appears to be running already..." + ) + + if jail_is_running(jail_name): + eprint(skip_start_message) + return 0 + + jail_path = get_jail_path(jail_name) + jail_config_path = get_jail_config_path(jail_name) + jail_rootfs_path = get_jail_rootfs_path(jail_name) + + config = parse_config_file(jail_config_path) + + if not config: + eprint("Aborting...") + return 1 + + seccomp = config.my_getboolean("seccomp") + + systemd_run_additional_args = [ + f"--unit={SHORTNAME}-{jail_name}", + f"--working-directory={jail_path}", + f"--description=My nspawn jail {jail_name} [created with jailmaker]", + ] + + systemd_nspawn_additional_args = [ + f"--machine={jail_name}", + f"--directory={JAIL_ROOTFS_NAME}", + ] + + # The systemd-nspawn manual explicitly mentions: + # Device nodes may not be created + # https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html + # This means docker images containing device nodes can't be pulled + # https://github.com/moby/moby/issues/35245 + # + # The solution is to use DevicePolicy=auto + # https://github.com/kinvolk/kube-spawn/pull/328 + # + # DevicePolicy=auto is the default for systemd-run and allows access to all devices + # as long as we don't add any --property=DeviceAllow= flags + # https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html + # + # We can now successfully run: + # mknod /dev/port c 1 4 + # Or pull docker images containing device nodes: + # docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643 + + # Add hooks to execute commands on the host before/after starting and after stopping a jail + add_hook( + jail_path, + systemd_run_additional_args, + config.my_get("pre_start_hook"), + "ExecStartPre", + ) + + add_hook( + jail_path, + systemd_run_additional_args, + config.my_get("post_start_hook"), + "ExecStartPost", + ) + + add_hook( + jail_path, + systemd_run_additional_args, + config.my_get("post_stop_hook"), + "ExecStopPost", + ) + + gpu_passthrough_intel = config.my_getboolean("gpu_passthrough_intel") + gpu_passthrough_nvidia = config.my_getboolean("gpu_passthrough_nvidia") + + passthrough_intel(gpu_passthrough_intel, systemd_nspawn_additional_args) + passthrough_nvidia( + gpu_passthrough_nvidia, systemd_nspawn_additional_args, jail_name + ) + + if seccomp is False: + # Disabling seccomp filtering by passing --setenv=SYSTEMD_SECCOMP=0 to systemd-run will improve performance + # at the expense of security: it allows syscalls which otherwise would be blocked or would have to be explicitly allowed by passing + # --system-call-filter to systemd-nspawn + # https://github.com/systemd/systemd/issues/18370 + # + # However, and additional layer of seccomp filtering may be undesirable + # For example when using docker to run containers inside the jail created with systemd-nspawn + # Even though seccomp filtering is disabled for the systemd-nspawn jail itself, docker can still use seccomp filtering + # to restrict the actions available within its containers + # + # Proof that seccomp can be used inside a jail started with --setenv=SYSTEMD_SECCOMP=0: + # Run a command in a docker container which is blocked by the default docker seccomp profile: + # docker run --rm -it debian:jessie unshare --map-root-user --user sh -c whoami + # unshare: unshare failed: Operation not permitted + # Now run unconfined to show command runs successfully: + # docker run --rm -it --security-opt seccomp=unconfined debian:jessie unshare --map-root-user --user sh -c whoami + # root + + systemd_run_additional_args += [ + "--setenv=SYSTEMD_SECCOMP=0", + ] + + initial_setup = False + + # If there's no machine-id, then this the first time the jail is started + if not os.path.exists(os.path.join(jail_rootfs_path, "etc/machine-id")) and ( + initial_setup := config.my_get("initial_setup") + ): + # initial_setup has been assigned due to := expression above + # Ensure the jail init system is ready before we start the initial_setup + systemd_nspawn_additional_args += [ + "--notify-ready=yes", + ] + + cmd = [ + "systemd-run", + *shlex.split(config.my_get("systemd_run_default_args")), + *systemd_run_additional_args, + "--", + "systemd-nspawn", + *shlex.split(config.my_get("systemd_nspawn_default_args")), + *systemd_nspawn_additional_args, + *shlex.split(config.my_get("systemd_nspawn_user_args")), + ] + + print( + dedent( + f""" + Starting jail {jail_name} with the following command: + + {shlex.join(cmd)} + """ + ) + ) + + returncode = subprocess.run(cmd).returncode + if returncode != 0: + eprint( + dedent( + f""" + Failed to start jail {jail_name}... + In case of a config error, you may fix it with: + {COMMAND_NAME} edit {jail_name} + """ + ) + ) + + return returncode + + # Handle initial setup after jail is up and running (for the first time) + if initial_setup: + if not initial_setup.startswith("#!"): + initial_setup = "#!/bin/sh\n" + initial_setup + + with tempfile.NamedTemporaryFile( + mode="w+t", + prefix="jlmkr-initial-setup.", + dir=jail_rootfs_path, + delete=False, + ) as initial_setup_file: + # Write a script file to call during initial setup + initial_setup_file.write(initial_setup) + + initial_setup_file_name = os.path.basename(initial_setup_file.name) + initial_setup_file_host_path = os.path.abspath(initial_setup_file.name) + stat_chmod(initial_setup_file_host_path, 0o700) + + print(f"About to run the initial setup script: {initial_setup_file_name}.") + print("Waiting for networking in the jail to be ready.") + print( + "Please wait (this may take 90s in case of bridge networking with STP is enabled)..." + ) + returncode = exec_jail( + jail_name, + [ + "--", + "systemd-run", + f"--unit={initial_setup_file_name}", + "--quiet", + "--pipe", + "--wait", + "--service-type=exec", + "--property=After=network-online.target", + "--property=Wants=network-online.target", + "/" + initial_setup_file_name, + ], + ) + + if returncode != 0: + eprint("Tried to run the following commands inside the jail:") + eprint(initial_setup) + eprint() + eprint(f"{RED}{BOLD}Failed to run initial setup...") + eprint( + f"You may want to manually run /{initial_setup_file_name} inside the jail for debugging purposes." + ) + eprint(f"Or stop and remove the jail and try again.{NORMAL}") + return returncode + else: + # Cleanup the initial_setup_file_host_path + Path(initial_setup_file_host_path).unlink(missing_ok=True) + print(f"Done with initial setup of jail {jail_name}!") + + return returncode + + +def add_hook(jail_path, systemd_run_additional_args, hook_command, hook_type): + if not hook_command: + return + + # Run the command directly if it doesn't start with a shebang + if not hook_command.startswith("#!"): + systemd_run_additional_args += [f"--property={hook_type}={hook_command}"] + return + + # Otherwise write a script file and call that + hook_file = os.path.abspath(os.path.join(jail_path, f".{hook_type}")) + + # Only write if contents are different + if not os.path.exists(hook_file) or Path(hook_file).read_text() != hook_command: + print(hook_command, file=open(hook_file, "w")) + + stat_chmod(hook_file, 0o700) + systemd_run_additional_args += [ + f"--property={hook_type}={systemd_escape_path(hook_file)}" + ] + + +def systemd_escape_path(path): + """ + Escape path containing spaces, while properly handling backslashes in filenames. + https://manpages.debian.org/bookworm/systemd/systemd.syntax.7.en.html#QUOTING + https://manpages.debian.org/bookworm/systemd/systemd.service.5.en.html#COMMAND_LINES + """ + return "".join( + map( + lambda char: r"\s" if char == " " else "\\\\" if char == "\\" else char, + path, + ) + ) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index ce3ded9..bb1526e 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -142,152 +142,6 @@ def error(self, message): from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path -# Test intel GPU by decoding mp4 file (output is discarded) -# Run the commands below in the jail: -# curl -o bunny.mp4 https://www.w3schools.com/html/mov_bbb.mp4 -# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i bunny.mp4 -f null - && echo 'SUCCESS!' - - -def passthrough_intel(gpu_passthrough_intel, systemd_nspawn_additional_args): - if not gpu_passthrough_intel: - return - - if not os.path.exists("/dev/dri"): - eprint( - dedent( - """ - No intel GPU seems to be present... - Skip passthrough of intel GPU.""" - ) - ) - return - - systemd_nspawn_additional_args.append("--bind=/dev/dri") - - -def passthrough_nvidia( - gpu_passthrough_nvidia, systemd_nspawn_additional_args, jail_name -): - jail_rootfs_path = get_jail_rootfs_path(jail_name) - ld_so_conf_path = Path( - os.path.join(jail_rootfs_path), f"etc/ld.so.conf.d/{SHORTNAME}-nvidia.conf" - ) - - if not gpu_passthrough_nvidia: - # Cleanup the config file we made when passthrough was enabled - ld_so_conf_path.unlink(missing_ok=True) - return - - # Load the nvidia kernel module - if subprocess.run(["modprobe", "nvidia-current-uvm"]).returncode != 0: - eprint( - dedent( - """ - Failed to load nvidia-current-uvm kernel module.""" - ) - ) - - # Run nvidia-smi to initialize the nvidia driver - # If we can't run nvidia-smi successfully, - # then nvidia-container-cli list will fail too: - # we shouldn't continue with gpu passthrough - if subprocess.run(["nvidia-smi", "-f", "/dev/null"]).returncode != 0: - eprint("Skip passthrough of nvidia GPU.") - return - - try: - # Get list of libraries - nvidia_libraries = set( - [ - x - for x in subprocess.check_output( - ["nvidia-container-cli", "list", "--libraries"] - ) - .decode() - .split("\n") - if x - ] - ) - # Get full list of files, but excluding library ones from above - nvidia_files = set( - ( - [ - x - for x in subprocess.check_output(["nvidia-container-cli", "list"]) - .decode() - .split("\n") - if x and x not in nvidia_libraries - ] - ) - ) - except Exception: - eprint( - dedent( - """ - Unable to detect which nvidia driver files to mount. - Skip passthrough of nvidia GPU.""" - ) - ) - return - - # Also make nvidia-smi available inside the path, - # while mounting the symlink will be resolved and nvidia-smi will appear as a regular file - nvidia_files.add("/usr/bin/nvidia-smi") - - nvidia_mounts = [] - - for file_path in nvidia_files: - if not os.path.exists(file_path): - # Don't try to mount files not present on the host - print(f"Skipped mounting {file_path}, it doesn't exist on the host...") - continue - - if file_path.startswith("/dev/"): - nvidia_mounts.append(f"--bind={file_path}") - else: - nvidia_mounts.append(f"--bind-ro={file_path}") - - # Check if the parent dir exists where we want to write our conf file - if ld_so_conf_path.parent.exists(): - library_folders = set(str(Path(x).parent) for x in nvidia_libraries) - # Add the library folders as mounts - for lf in library_folders: - nvidia_mounts.append(f"--bind-ro={lf}") - - # Only write if the conf file doesn't yet exist or has different contents - existing_conf_libraries = set() - if ld_so_conf_path.exists(): - existing_conf_libraries.update( - x for x in ld_so_conf_path.read_text().splitlines() if x - ) - - if library_folders != existing_conf_libraries: - print("\n".join(x for x in library_folders), file=ld_so_conf_path.open("w")) - - # Run ldconfig inside systemd-nspawn jail with nvidia mounts... - subprocess.run( - [ - "systemd-nspawn", - "--quiet", - f"--machine={jail_name}", - f"--directory={jail_rootfs_path}", - *nvidia_mounts, - "ldconfig", - ] - ) - else: - eprint( - dedent( - """ - Unable to write the ld.so.conf.d directory inside the jail (it doesn't exist). - Skipping call to ldconfig. - The nvidia drivers will probably not be detected...""" - ) - ) - - systemd_nspawn_additional_args += nvidia_mounts - - def exec_jail(jail_name, cmd): """ Execute a command in the jail with given name. @@ -333,250 +187,7 @@ def shell_jail(args): return subprocess.run(["machinectl", "shell"] + args).returncode -def systemd_escape_path(path): - """ - Escape path containing spaces, while properly handling backslashes in filenames. - https://manpages.debian.org/bookworm/systemd/systemd.syntax.7.en.html#QUOTING - https://manpages.debian.org/bookworm/systemd/systemd.service.5.en.html#COMMAND_LINES - """ - return "".join( - map( - lambda char: r"\s" if char == " " else "\\\\" if char == "\\" else char, - path, - ) - ) - - -def add_hook(jail_path, systemd_run_additional_args, hook_command, hook_type): - if not hook_command: - return - - # Run the command directly if it doesn't start with a shebang - if not hook_command.startswith("#!"): - systemd_run_additional_args += [f"--property={hook_type}={hook_command}"] - return - - # Otherwise write a script file and call that - hook_file = os.path.abspath(os.path.join(jail_path, f".{hook_type}")) - - # Only write if contents are different - if not os.path.exists(hook_file) or Path(hook_file).read_text() != hook_command: - print(hook_command, file=open(hook_file, "w")) - - stat_chmod(hook_file, 0o700) - systemd_run_additional_args += [ - f"--property={hook_type}={systemd_escape_path(hook_file)}" - ] - - -def start_jail(jail_name): - """ - Start jail with given name. - """ - skip_start_message = ( - f"Skipped starting jail {jail_name}. It appears to be running already..." - ) - - if jail_is_running(jail_name): - eprint(skip_start_message) - return 0 - - jail_path = get_jail_path(jail_name) - jail_config_path = get_jail_config_path(jail_name) - jail_rootfs_path = get_jail_rootfs_path(jail_name) - - config = parse_config_file(jail_config_path) - - if not config: - eprint("Aborting...") - return 1 - - seccomp = config.my_getboolean("seccomp") - - systemd_run_additional_args = [ - f"--unit={SHORTNAME}-{jail_name}", - f"--working-directory={jail_path}", - f"--description=My nspawn jail {jail_name} [created with jailmaker]", - ] - - systemd_nspawn_additional_args = [ - f"--machine={jail_name}", - f"--directory={JAIL_ROOTFS_NAME}", - ] - - # The systemd-nspawn manual explicitly mentions: - # Device nodes may not be created - # https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html - # This means docker images containing device nodes can't be pulled - # https://github.com/moby/moby/issues/35245 - # - # The solution is to use DevicePolicy=auto - # https://github.com/kinvolk/kube-spawn/pull/328 - # - # DevicePolicy=auto is the default for systemd-run and allows access to all devices - # as long as we don't add any --property=DeviceAllow= flags - # https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html - # - # We can now successfully run: - # mknod /dev/port c 1 4 - # Or pull docker images containing device nodes: - # docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643 - - # Add hooks to execute commands on the host before/after starting and after stopping a jail - add_hook( - jail_path, - systemd_run_additional_args, - config.my_get("pre_start_hook"), - "ExecStartPre", - ) - - add_hook( - jail_path, - systemd_run_additional_args, - config.my_get("post_start_hook"), - "ExecStartPost", - ) - - add_hook( - jail_path, - systemd_run_additional_args, - config.my_get("post_stop_hook"), - "ExecStopPost", - ) - - gpu_passthrough_intel = config.my_getboolean("gpu_passthrough_intel") - gpu_passthrough_nvidia = config.my_getboolean("gpu_passthrough_nvidia") - - passthrough_intel(gpu_passthrough_intel, systemd_nspawn_additional_args) - passthrough_nvidia( - gpu_passthrough_nvidia, systemd_nspawn_additional_args, jail_name - ) - - if seccomp is False: - # Disabling seccomp filtering by passing --setenv=SYSTEMD_SECCOMP=0 to systemd-run will improve performance - # at the expense of security: it allows syscalls which otherwise would be blocked or would have to be explicitly allowed by passing - # --system-call-filter to systemd-nspawn - # https://github.com/systemd/systemd/issues/18370 - # - # However, and additional layer of seccomp filtering may be undesirable - # For example when using docker to run containers inside the jail created with systemd-nspawn - # Even though seccomp filtering is disabled for the systemd-nspawn jail itself, docker can still use seccomp filtering - # to restrict the actions available within its containers - # - # Proof that seccomp can be used inside a jail started with --setenv=SYSTEMD_SECCOMP=0: - # Run a command in a docker container which is blocked by the default docker seccomp profile: - # docker run --rm -it debian:jessie unshare --map-root-user --user sh -c whoami - # unshare: unshare failed: Operation not permitted - # Now run unconfined to show command runs successfully: - # docker run --rm -it --security-opt seccomp=unconfined debian:jessie unshare --map-root-user --user sh -c whoami - # root - - systemd_run_additional_args += [ - "--setenv=SYSTEMD_SECCOMP=0", - ] - - initial_setup = False - - # If there's no machine-id, then this the first time the jail is started - if not os.path.exists(os.path.join(jail_rootfs_path, "etc/machine-id")) and ( - initial_setup := config.my_get("initial_setup") - ): - # initial_setup has been assigned due to := expression above - # Ensure the jail init system is ready before we start the initial_setup - systemd_nspawn_additional_args += [ - "--notify-ready=yes", - ] - - cmd = [ - "systemd-run", - *shlex.split(config.my_get("systemd_run_default_args")), - *systemd_run_additional_args, - "--", - "systemd-nspawn", - *shlex.split(config.my_get("systemd_nspawn_default_args")), - *systemd_nspawn_additional_args, - *shlex.split(config.my_get("systemd_nspawn_user_args")), - ] - - print( - dedent( - f""" - Starting jail {jail_name} with the following command: - - {shlex.join(cmd)} - """ - ) - ) - - returncode = subprocess.run(cmd).returncode - if returncode != 0: - eprint( - dedent( - f""" - Failed to start jail {jail_name}... - In case of a config error, you may fix it with: - {COMMAND_NAME} edit {jail_name} - """ - ) - ) - - return returncode - - # Handle initial setup after jail is up and running (for the first time) - if initial_setup: - if not initial_setup.startswith("#!"): - initial_setup = "#!/bin/sh\n" + initial_setup - - with tempfile.NamedTemporaryFile( - mode="w+t", - prefix="jlmkr-initial-setup.", - dir=jail_rootfs_path, - delete=False, - ) as initial_setup_file: - # Write a script file to call during initial setup - initial_setup_file.write(initial_setup) - - initial_setup_file_name = os.path.basename(initial_setup_file.name) - initial_setup_file_host_path = os.path.abspath(initial_setup_file.name) - stat_chmod(initial_setup_file_host_path, 0o700) - - print(f"About to run the initial setup script: {initial_setup_file_name}.") - print("Waiting for networking in the jail to be ready.") - print( - "Please wait (this may take 90s in case of bridge networking with STP is enabled)..." - ) - returncode = exec_jail( - jail_name, - [ - "--", - "systemd-run", - f"--unit={initial_setup_file_name}", - "--quiet", - "--pipe", - "--wait", - "--service-type=exec", - "--property=After=network-online.target", - "--property=Wants=network-online.target", - "/" + initial_setup_file_name, - ], - ) - - if returncode != 0: - eprint("Tried to run the following commands inside the jail:") - eprint(initial_setup) - eprint() - eprint(f"{RED}{BOLD}Failed to run initial setup...") - eprint( - f"You may want to manually run /{initial_setup_file_name} inside the jail for debugging purposes." - ) - eprint(f"Or stop and remove the jail and try again.{NORMAL}") - return returncode - else: - # Cleanup the initial_setup_file_host_path - Path(initial_setup_file_host_path).unlink(missing_ok=True) - print(f"Done with initial setup of jail {jail_name}!") - - return returncode +from actions.start import add_hook, start_jail def restart_jail(jail_name): diff --git a/src/jlmkr/utils/gpu.py b/src/jlmkr/utils/gpu.py new file mode 100644 index 0000000..b71c3ae --- /dev/null +++ b/src/jlmkr/utils/gpu.py @@ -0,0 +1,158 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os.path +import subprocess + +from pathlib import Path +from textwrap import dedent +from utils.console import eprint +from utils.jail_dataset import get_jail_rootfs_path +from utils.paths import SHORTNAME + + +# Test intel GPU by decoding mp4 file (output is discarded) +# Run the commands below in the jail: +# curl -o bunny.mp4 https://www.w3schools.com/html/mov_bbb.mp4 +# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i bunny.mp4 -f null - && echo 'SUCCESS!' + + +def passthrough_intel(gpu_passthrough_intel, systemd_nspawn_additional_args): + if not gpu_passthrough_intel: + return + + if not os.path.exists("/dev/dri"): + eprint( + dedent( + """ + No intel GPU seems to be present... + Skip passthrough of intel GPU.""" + ) + ) + return + + systemd_nspawn_additional_args.append("--bind=/dev/dri") + + +def passthrough_nvidia( + gpu_passthrough_nvidia, systemd_nspawn_additional_args, jail_name +): + jail_rootfs_path = get_jail_rootfs_path(jail_name) + ld_so_conf_path = Path( + os.path.join(jail_rootfs_path), f"etc/ld.so.conf.d/{SHORTNAME}-nvidia.conf" + ) + + if not gpu_passthrough_nvidia: + # Cleanup the config file we made when passthrough was enabled + ld_so_conf_path.unlink(missing_ok=True) + return + + # Load the nvidia kernel module + if subprocess.run(["modprobe", "nvidia-current-uvm"]).returncode != 0: + eprint( + dedent( + """ + Failed to load nvidia-current-uvm kernel module.""" + ) + ) + + # Run nvidia-smi to initialize the nvidia driver + # If we can't run nvidia-smi successfully, + # then nvidia-container-cli list will fail too: + # we shouldn't continue with gpu passthrough + if subprocess.run(["nvidia-smi", "-f", "/dev/null"]).returncode != 0: + eprint("Skip passthrough of nvidia GPU.") + return + + try: + # Get list of libraries + nvidia_libraries = set( + [ + x + for x in subprocess.check_output( + ["nvidia-container-cli", "list", "--libraries"] + ) + .decode() + .split("\n") + if x + ] + ) + # Get full list of files, but excluding library ones from above + nvidia_files = set( + ( + [ + x + for x in subprocess.check_output(["nvidia-container-cli", "list"]) + .decode() + .split("\n") + if x and x not in nvidia_libraries + ] + ) + ) + except Exception: + eprint( + dedent( + """ + Unable to detect which nvidia driver files to mount. + Skip passthrough of nvidia GPU.""" + ) + ) + return + + # Also make nvidia-smi available inside the path, + # while mounting the symlink will be resolved and nvidia-smi will appear as a regular file + nvidia_files.add("/usr/bin/nvidia-smi") + + nvidia_mounts = [] + + for file_path in nvidia_files: + if not os.path.exists(file_path): + # Don't try to mount files not present on the host + print(f"Skipped mounting {file_path}, it doesn't exist on the host...") + continue + + if file_path.startswith("/dev/"): + nvidia_mounts.append(f"--bind={file_path}") + else: + nvidia_mounts.append(f"--bind-ro={file_path}") + + # Check if the parent dir exists where we want to write our conf file + if ld_so_conf_path.parent.exists(): + library_folders = set(str(Path(x).parent) for x in nvidia_libraries) + # Add the library folders as mounts + for lf in library_folders: + nvidia_mounts.append(f"--bind-ro={lf}") + + # Only write if the conf file doesn't yet exist or has different contents + existing_conf_libraries = set() + if ld_so_conf_path.exists(): + existing_conf_libraries.update( + x for x in ld_so_conf_path.read_text().splitlines() if x + ) + + if library_folders != existing_conf_libraries: + print("\n".join(x for x in library_folders), file=ld_so_conf_path.open("w")) + + # Run ldconfig inside systemd-nspawn jail with nvidia mounts... + subprocess.run( + [ + "systemd-nspawn", + "--quiet", + f"--machine={jail_name}", + f"--directory={jail_rootfs_path}", + *nvidia_mounts, + "ldconfig", + ] + ) + else: + eprint( + dedent( + """ + Unable to write the ld.so.conf.d directory inside the jail (it doesn't exist). + Skipping call to ldconfig. + The nvidia drivers will probably not be detected...""" + ) + ) + + systemd_nspawn_additional_args += nvidia_mounts From ef479d7052d13942685cd20c1f2ecb5808ee230f Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 01:57:55 -0400 Subject: [PATCH 19/58] Extract stop action --- src/jlmkr/actions/stop.py | 31 +++++++++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 21 +-------------------- 2 files changed, 32 insertions(+), 20 deletions(-) create mode 100644 src/jlmkr/actions/stop.py diff --git a/src/jlmkr/actions/stop.py b/src/jlmkr/actions/stop.py new file mode 100644 index 0000000..9babdb4 --- /dev/null +++ b/src/jlmkr/actions/stop.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import subprocess +import time + +from utils.console import eprint +from utils.jail_dataset import jail_is_running + + +def stop_jail(jail_name): + """ + Stop jail with given name and wait until stopped. + """ + + if not jail_is_running(jail_name): + return 0 + + returncode = subprocess.run(["machinectl", "poweroff", jail_name]).returncode + if returncode != 0: + eprint("Error while stopping jail.") + return returncode + + print(f"Wait for {jail_name} to stop", end="", flush=True) + + while jail_is_running(jail_name): + time.sleep(1) + print(".", end="", flush=True) + + return 0 diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index bb1526e..e2a5634 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -751,26 +751,7 @@ def edit_jail(jail_name): return 0 -def stop_jail(jail_name): - """ - Stop jail with given name and wait until stopped. - """ - - if not jail_is_running(jail_name): - return 0 - - returncode = subprocess.run(["machinectl", "poweroff", jail_name]).returncode - if returncode != 0: - eprint("Error while stopping jail.") - return returncode - - print(f"Wait for {jail_name} to stop", end="", flush=True) - - while jail_is_running(jail_name): - time.sleep(1) - print(".", end="", flush=True) - - return 0 +from actions.stop import stop_jail def remove_jail(jail_name): From d26e8cc5180f80a0c66e1ba715274f037707d37a Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 02:01:35 -0400 Subject: [PATCH 20/58] Extract restart action --- src/jlmkr/actions/restart.py | 24 ++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 16 ++-------------- 2 files changed, 26 insertions(+), 14 deletions(-) create mode 100644 src/jlmkr/actions/restart.py diff --git a/src/jlmkr/actions/restart.py b/src/jlmkr/actions/restart.py new file mode 100644 index 0000000..1740230 --- /dev/null +++ b/src/jlmkr/actions/restart.py @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + + +import subprocess +import time + +from actions.start import start_jail +from actions.stop import stop_jail +from utils.console import eprint + + +def restart_jail(jail_name): + """ + Restart jail with given name. + """ + + returncode = stop_jail(jail_name) + if returncode != 0: + eprint("Abort restart.") + return returncode + + return start_jail(jail_name) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index e2a5634..b70d4be 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -187,20 +187,8 @@ def shell_jail(args): return subprocess.run(["machinectl", "shell"] + args).returncode -from actions.start import add_hook, start_jail - - -def restart_jail(jail_name): - """ - Restart jail with given name. - """ - - returncode = stop_jail(jail_name) - if returncode != 0: - eprint("Abort restart.") - return returncode - - return start_jail(jail_name) +from actions.start import start_jail +from actions.restart import restart_jail def cleanup(jail_path): From dbb3dd4e226be9c0109896be21a62bf3a0fb96ac Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 02:04:26 -0400 Subject: [PATCH 21/58] Extract exec action --- src/jlmkr/actions/exec.py | 24 ++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 19 +------------------ 2 files changed, 25 insertions(+), 18 deletions(-) create mode 100644 src/jlmkr/actions/exec.py diff --git a/src/jlmkr/actions/exec.py b/src/jlmkr/actions/exec.py new file mode 100644 index 0000000..eba8b2f --- /dev/null +++ b/src/jlmkr/actions/exec.py @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import subprocess + + +def exec_jail(jail_name, cmd): + """ + Execute a command in the jail with given name. + """ + return subprocess.run( + [ + "systemd-run", + "--machine", + jail_name, + "--quiet", + "--pipe", + "--wait", + "--collect", + "--service-type=exec", + *cmd, + ] + ).returncode diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index b70d4be..772601b 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -141,24 +141,7 @@ def error(self, message): from utils.console import eprint, fail from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path - -def exec_jail(jail_name, cmd): - """ - Execute a command in the jail with given name. - """ - return subprocess.run( - [ - "systemd-run", - "--machine", - jail_name, - "--quiet", - "--pipe", - "--wait", - "--collect", - "--service-type=exec", - *cmd, - ] - ).returncode +from actions.exec import exec_jail def status_jail(jail_name, args): From f686bbb60b04645281e9b0396bb07e2a23819c9d Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 02:05:38 -0400 Subject: [PATCH 22/58] Extract shell action --- src/jlmkr/actions/shell.py | 12 ++++++++++++ src/jlmkr/donor/jlmkr.py | 8 +------- 2 files changed, 13 insertions(+), 7 deletions(-) create mode 100644 src/jlmkr/actions/shell.py diff --git a/src/jlmkr/actions/shell.py b/src/jlmkr/actions/shell.py new file mode 100644 index 0000000..ed51dd0 --- /dev/null +++ b/src/jlmkr/actions/shell.py @@ -0,0 +1,12 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import subprocess + + +def shell_jail(args): + """ + Open a shell in the jail with given name. + """ + return subprocess.run(["machinectl", "shell"] + args).returncode diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 772601b..508dc67 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -163,13 +163,7 @@ def log_jail(jail_name, args): ).returncode -def shell_jail(args): - """ - Open a shell in the jail with given name. - """ - return subprocess.run(["machinectl", "shell"] + args).returncode - - +from actions.shell import shell_jail from actions.start import start_jail from actions.restart import restart_jail From f29e0bc075d199281f74530c92e91d142ee1905e Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 02:08:09 -0400 Subject: [PATCH 23/58] Extract status action --- src/jlmkr/actions/status.py | 17 +++++++++++++++++ src/jlmkr/donor/jlmkr.py | 11 +---------- 2 files changed, 18 insertions(+), 10 deletions(-) create mode 100644 src/jlmkr/actions/status.py diff --git a/src/jlmkr/actions/status.py b/src/jlmkr/actions/status.py new file mode 100644 index 0000000..e3fa570 --- /dev/null +++ b/src/jlmkr/actions/status.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import subprocess + +from utils.paths import SHORTNAME + + +def status_jail(jail_name, args): + """ + Show the status of the systemd service wrapping the jail with given name. + """ + # Alternatively `machinectl status jail_name` could be used + return subprocess.run( + ["systemctl", "status", f"{SHORTNAME}-{jail_name}", *args] + ).returncode diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 508dc67..9325280 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -142,16 +142,7 @@ def error(self, message): from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path from actions.exec import exec_jail - - -def status_jail(jail_name, args): - """ - Show the status of the systemd service wrapping the jail with given name. - """ - # Alternatively `machinectl status jail_name` could be used - return subprocess.run( - ["systemctl", "status", f"{SHORTNAME}-{jail_name}", *args] - ).returncode +from actions.status import status_jail def log_jail(jail_name, args): From a08149daeaa4163ac94b4f54d81e8e486cc5126e Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 02:09:38 -0400 Subject: [PATCH 24/58] Extract log action --- src/jlmkr/actions/log.py | 16 ++++++++++++++++ src/jlmkr/donor/jlmkr.py | 12 +----------- 2 files changed, 17 insertions(+), 11 deletions(-) create mode 100644 src/jlmkr/actions/log.py diff --git a/src/jlmkr/actions/log.py b/src/jlmkr/actions/log.py new file mode 100644 index 0000000..b0e7b32 --- /dev/null +++ b/src/jlmkr/actions/log.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import subprocess + +from utils.paths import SHORTNAME + + +def log_jail(jail_name, args): + """ + Show the log file of the jail with given name. + """ + return subprocess.run( + ["journalctl", "-u", f"{SHORTNAME}-{jail_name}", *args] + ).returncode diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 9325280..ca4bf44 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -143,17 +143,7 @@ def error(self, message): from actions.exec import exec_jail from actions.status import status_jail - - -def log_jail(jail_name, args): - """ - Show the log file of the jail with given name. - """ - return subprocess.run( - ["journalctl", "-u", f"{SHORTNAME}-{jail_name}", *args] - ).returncode - - +from actions.log import log_jail from actions.shell import shell_jail from actions.start import start_jail from actions.restart import restart_jail From a0c0599e7b503eef350265f0963778384f8d1480 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:20:04 -0400 Subject: [PATCH 25/58] Extract download utils --- src/jlmkr/donor/jlmkr.py | 93 -------------------------------- src/jlmkr/utils/download.py | 103 ++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 93 deletions(-) create mode 100644 src/jlmkr/utils/download.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index ca4bf44..3d942ed 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -111,10 +111,6 @@ # Always add --bind-ro=/sys/module to make lsmod happy # https://manpages.debian.org/bookworm/manpages/sysfs.5.en.html -DOWNLOAD_SCRIPT_DIGEST = ( - "cfcb5d08b24187d108f2ab0d21a6cc4b73dcd7f5d7dfc80803bfd7f1642d638d" -) - from utils.paths import SCRIPT_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME from utils.paths import COMMAND_NAME, SHORTNAME @@ -175,95 +171,6 @@ def _onerror(func, path, exc_info): shutil.rmtree(jail_path, onerror=_onerror) -def validate_sha256(file_path, digest): - """ - Validates if a file matches a sha256 digest. - """ - try: - with open(file_path, "rb") as f: - file_hash = hashlib.sha256(f.read()).hexdigest() - return file_hash == digest - except FileNotFoundError: - return False - - -def run_lxc_download_script( - jail_name=None, jail_path=None, jail_rootfs_path=None, distro=None, release=None -): - arch = "amd64" - lxc_dir = ".lxc" - lxc_cache = os.path.join(lxc_dir, "cache") - lxc_download_script = os.path.join(lxc_dir, "lxc-download.sh") - - # Create the lxc dirs if nonexistent - os.makedirs(lxc_dir, exist_ok=True) - stat_chmod(lxc_dir, 0o700) - os.makedirs(lxc_cache, exist_ok=True) - stat_chmod(lxc_cache, 0o700) - - try: - if os.stat(lxc_download_script).st_uid != 0: - os.remove(lxc_download_script) - except FileNotFoundError: - pass - - # Fetch the lxc download script if not present locally (or hash doesn't match) - if not validate_sha256(lxc_download_script, DOWNLOAD_SCRIPT_DIGEST): - urllib.request.urlretrieve( - "https://raw.githubusercontent.com/Jip-Hop/lxc/97f93be72ebf380f3966259410b70b1c966b0ff0/templates/lxc-download.in", - lxc_download_script, - ) - - if not validate_sha256(lxc_download_script, DOWNLOAD_SCRIPT_DIGEST): - eprint("Abort! Downloaded script has unexpected contents.") - return 1 - - stat_chmod(lxc_download_script, 0o700) - - if None not in [jail_name, jail_path, jail_rootfs_path, distro, release]: - cmd = [ - lxc_download_script, - f"--name={jail_name}", - f"--path={jail_path}", - f"--rootfs={jail_rootfs_path}", - f"--arch={arch}", - f"--dist={distro}", - f"--release={release}", - ] - - if rc := subprocess.run(cmd, env={"LXC_CACHE_PATH": lxc_cache}).returncode != 0: - eprint("Aborting...") - return rc - - else: - # List images - cmd = [lxc_download_script, "--list", f"--arch={arch}"] - - p1 = subprocess.Popen( - cmd, stdout=subprocess.PIPE, env={"LXC_CACHE_PATH": lxc_cache} - ) - - for line in iter(p1.stdout.readline, b""): - line = line.decode().strip() - # Filter out the known incompatible distros - if not re.match( - r"^(alpine|amazonlinux|busybox|devuan|funtoo|openwrt|plamo|voidlinux)\s", - line, - ): - # TODO: check if output matches expected output, if it does then return 0 - # Else treat this as an error and return 1 - print(line) - - rc = p1.wait() - # Currently --list will always return a non-zero exit code, even when listing the images was successful - # https://github.com/lxc/lxc/pull/4462 - # Therefore we must currently return 0, to prevent aborting the interactive create process - - # return rc - - return 0 - - from utils.files import stat_chmod diff --git a/src/jlmkr/utils/download.py b/src/jlmkr/utils/download.py new file mode 100644 index 0000000..f1fbfe0 --- /dev/null +++ b/src/jlmkr/utils/download.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import hashlib +import os +import subprocess + +from utils.files import stat_chmod + + +DOWNLOAD_SCRIPT_DIGEST = ( + "cfcb5d08b24187d108f2ab0d21a6cc4b73dcd7f5d7dfc80803bfd7f1642d638d" +) + + +def run_lxc_download_script( + jail_name=None, jail_path=None, jail_rootfs_path=None, distro=None, release=None +): + arch = "amd64" + lxc_dir = ".lxc" + lxc_cache = os.path.join(lxc_dir, "cache") + lxc_download_script = os.path.join(lxc_dir, "lxc-download.sh") + + # Create the lxc dirs if nonexistent + os.makedirs(lxc_dir, exist_ok=True) + stat_chmod(lxc_dir, 0o700) + os.makedirs(lxc_cache, exist_ok=True) + stat_chmod(lxc_cache, 0o700) + + try: + if os.stat(lxc_download_script).st_uid != 0: + os.remove(lxc_download_script) + except FileNotFoundError: + pass + + # Fetch the lxc download script if not present locally (or hash doesn't match) + if not validate_sha256(lxc_download_script, DOWNLOAD_SCRIPT_DIGEST): + urllib.request.urlretrieve( + "https://raw.githubusercontent.com/Jip-Hop/lxc/97f93be72ebf380f3966259410b70b1c966b0ff0/templates/lxc-download.in", + lxc_download_script, + ) + + if not validate_sha256(lxc_download_script, DOWNLOAD_SCRIPT_DIGEST): + eprint("Abort! Downloaded script has unexpected contents.") + return 1 + + stat_chmod(lxc_download_script, 0o700) + + if None not in [jail_name, jail_path, jail_rootfs_path, distro, release]: + cmd = [ + lxc_download_script, + f"--name={jail_name}", + f"--path={jail_path}", + f"--rootfs={jail_rootfs_path}", + f"--arch={arch}", + f"--dist={distro}", + f"--release={release}", + ] + + if rc := subprocess.run(cmd, env={"LXC_CACHE_PATH": lxc_cache}).returncode != 0: + eprint("Aborting...") + return rc + + else: + # List images + cmd = [lxc_download_script, "--list", f"--arch={arch}"] + + p1 = subprocess.Popen( + cmd, stdout=subprocess.PIPE, env={"LXC_CACHE_PATH": lxc_cache} + ) + + for line in iter(p1.stdout.readline, b""): + line = line.decode().strip() + # Filter out the known incompatible distros + if not re.match( + r"^(alpine|amazonlinux|busybox|devuan|funtoo|openwrt|plamo|voidlinux)\s", + line, + ): + # TODO: check if output matches expected output, if it does then return 0 + # Else treat this as an error and return 1 + print(line) + + rc = p1.wait() + # Currently --list will always return a non-zero exit code, even when listing the images was successful + # https://github.com/lxc/lxc/pull/4462 + # Therefore we must currently return 0, to prevent aborting the interactive create process + + # return rc + + return 0 + + +def validate_sha256(file_path, digest): + """ + Validates if a file matches a sha256 digest. + """ + try: + with open(file_path, "rb") as f: + file_hash = hashlib.sha256(f.read()).hexdigest() + return file_hash == digest + except FileNotFoundError: + return False From ff2416b63809ab495408b87fc1dca589a54c8e18 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:30:40 -0400 Subject: [PATCH 26/58] Extract images action --- src/jlmkr/actions/images.py | 5 +++++ src/jlmkr/donor/jlmkr.py | 1 + 2 files changed, 6 insertions(+) create mode 100644 src/jlmkr/actions/images.py diff --git a/src/jlmkr/actions/images.py b/src/jlmkr/actions/images.py new file mode 100644 index 0000000..645734b --- /dev/null +++ b/src/jlmkr/actions/images.py @@ -0,0 +1,5 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from utils.download import run_lxc_download_script diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 3d942ed..b63de9a 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -143,6 +143,7 @@ def error(self, message): from actions.shell import shell_jail from actions.start import start_jail from actions.restart import restart_jail +from actions.images import run_lxc_download_script def cleanup(jail_path): From fbf6e4d9a5bfb3f843a9c4a17366be9c70338c3b Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:35:00 -0400 Subject: [PATCH 27/58] Extract default configuration data --- src/jlmkr/donor/data.py | 70 ++++++++++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 65 ----------------------------- src/jlmkr/utils/config_parser.py | 2 +- 3 files changed, 71 insertions(+), 66 deletions(-) create mode 100644 src/jlmkr/donor/data.py diff --git a/src/jlmkr/donor/data.py b/src/jlmkr/donor/data.py new file mode 100644 index 0000000..f230f0d --- /dev/null +++ b/src/jlmkr/donor/data.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + + +DEFAULT_CONFIG = """startup=0 +gpu_passthrough_intel=0 +gpu_passthrough_nvidia=0 +# Turning off seccomp filtering improves performance at the expense of security +seccomp=1 + +# Below you may add additional systemd-nspawn flags behind systemd_nspawn_user_args= +# To mount host storage in the jail, you may add: --bind='/mnt/pool/dataset:/home' +# To readonly mount host storage, you may add: --bind-ro=/etc/certificates +# To use macvlan networking add: --network-macvlan=eno1 --resolv-conf=bind-host +# To use bridge networking add: --network-bridge=br1 --resolv-conf=bind-host +# Ensure to change eno1/br1 to the interface name you want to use +# To allow syscalls required by docker add: --system-call-filter='add_key keyctl bpf' +systemd_nspawn_user_args= + +# Specify command/script to run on the HOST before starting the jail +# For example to load kernel modules and config kernel settings +pre_start_hook= +# pre_start_hook=#!/usr/bin/bash +# set -euo pipefail +# echo 'PRE_START_HOOK_EXAMPLE' +# echo 1 > /proc/sys/net/ipv4/ip_forward +# modprobe br_netfilter +# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables +# echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables + +# Specify command/script to run on the HOST after starting the jail +# For example to attach to multiple bridge interfaces +# when using --network-veth-extra=ve-myjail-1:veth1 +post_start_hook= +# post_start_hook=#!/usr/bin/bash +# set -euo pipefail +# echo 'POST_START_HOOK_EXAMPLE' +# ip link set dev ve-myjail-1 master br2 +# ip link set dev ve-myjail-1 up + +# Specify a command/script to run on the HOST after stopping the jail +post_stop_hook= +# post_stop_hook=echo 'POST_STOP_HOOK_EXAMPLE' + +# Only used while creating the jail +distro=debian +release=bookworm + +# Specify command/script to run IN THE JAIL on the first start (once networking is ready in the jail) +# Useful to install packages on top of the base rootfs +initial_setup= +# initial_setup=bash -c 'apt-get update && apt-get -y upgrade' + +# Usually no need to change systemd_run_default_args +systemd_run_default_args=--collect + --property=Delegate=yes + --property=RestartForceExitStatus=133 + --property=SuccessExitStatus=133 + --property=TasksMax=infinity + --property=Type=notify + --setenv=SYSTEMD_NSPAWN_LOCK=0 + --property=KillMode=mixed + +# Usually no need to change systemd_nspawn_default_args +systemd_nspawn_default_args=--bind-ro=/sys/module + --boot + --inaccessible=/sys/module/apparmor + --quiet + --keep-unit""" diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index b63de9a..1f866dc 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -30,71 +30,6 @@ from pathlib import Path, PurePath from textwrap import dedent -DEFAULT_CONFIG = """startup=0 -gpu_passthrough_intel=0 -gpu_passthrough_nvidia=0 -# Turning off seccomp filtering improves performance at the expense of security -seccomp=1 - -# Below you may add additional systemd-nspawn flags behind systemd_nspawn_user_args= -# To mount host storage in the jail, you may add: --bind='/mnt/pool/dataset:/home' -# To readonly mount host storage, you may add: --bind-ro=/etc/certificates -# To use macvlan networking add: --network-macvlan=eno1 --resolv-conf=bind-host -# To use bridge networking add: --network-bridge=br1 --resolv-conf=bind-host -# Ensure to change eno1/br1 to the interface name you want to use -# To allow syscalls required by docker add: --system-call-filter='add_key keyctl bpf' -systemd_nspawn_user_args= - -# Specify command/script to run on the HOST before starting the jail -# For example to load kernel modules and config kernel settings -pre_start_hook= -# pre_start_hook=#!/usr/bin/bash -# set -euo pipefail -# echo 'PRE_START_HOOK_EXAMPLE' -# echo 1 > /proc/sys/net/ipv4/ip_forward -# modprobe br_netfilter -# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables -# echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables - -# Specify command/script to run on the HOST after starting the jail -# For example to attach to multiple bridge interfaces -# when using --network-veth-extra=ve-myjail-1:veth1 -post_start_hook= -# post_start_hook=#!/usr/bin/bash -# set -euo pipefail -# echo 'POST_START_HOOK_EXAMPLE' -# ip link set dev ve-myjail-1 master br2 -# ip link set dev ve-myjail-1 up - -# Specify a command/script to run on the HOST after stopping the jail -post_stop_hook= -# post_stop_hook=echo 'POST_STOP_HOOK_EXAMPLE' - -# Only used while creating the jail -distro=debian -release=bookworm - -# Specify command/script to run IN THE JAIL on the first start (once networking is ready in the jail) -# Useful to install packages on top of the base rootfs -initial_setup= -# initial_setup=bash -c 'apt-get update && apt-get -y upgrade' - -# Usually no need to change systemd_run_default_args -systemd_run_default_args=--collect - --property=Delegate=yes - --property=RestartForceExitStatus=133 - --property=SuccessExitStatus=133 - --property=TasksMax=infinity - --property=Type=notify - --setenv=SYSTEMD_NSPAWN_LOCK=0 - --property=KillMode=mixed - -# Usually no need to change systemd_nspawn_default_args -systemd_nspawn_default_args=--bind-ro=/sys/module - --boot - --inaccessible=/sys/module/apparmor - --quiet - --keep-unit""" # Use mostly default settings for systemd-nspawn but with systemd-run instead of a service file: # https://github.com/systemd/systemd/blob/main/units/systemd-nspawn%40.service.in diff --git a/src/jlmkr/utils/config_parser.py b/src/jlmkr/utils/config_parser.py index 9264ab4..720f79f 100644 --- a/src/jlmkr/utils/config_parser.py +++ b/src/jlmkr/utils/config_parser.py @@ -6,7 +6,7 @@ import io import re -from donor.jlmkr import DEFAULT_CONFIG +from donor.data import DEFAULT_CONFIG # Used in parser getters to indicate the default behavior when a specific From 1c0a7a46d4061e2895e1597959ea224ad4abe795 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:39:06 -0400 Subject: [PATCH 28/58] Extract create action --- src/jlmkr/actions/create.py | 303 +++++++++++++++++++++ src/jlmkr/actions/list.py | 3 +- src/jlmkr/donor/jlmkr.py | 423 +----------------------------- src/jlmkr/utils/files.py | 10 + src/jlmkr/utils/jail_dataset.py | 137 +++++++++- src/jlmkr/utils/parent_dataset.py | 16 -- 6 files changed, 456 insertions(+), 436 deletions(-) create mode 100644 src/jlmkr/actions/create.py delete mode 100644 src/jlmkr/utils/parent_dataset.py diff --git a/src/jlmkr/actions/create.py b/src/jlmkr/actions/create.py new file mode 100644 index 0000000..b91fe6a --- /dev/null +++ b/src/jlmkr/actions/create.py @@ -0,0 +1,303 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import contextlib +import os +import re + +from inspect import cleandoc +from pathlib import Path, PurePath +from textwrap import dedent +from donor.jlmkr import DISCLAIMER +from utils.chroot import Chroot +from utils.config_parser import KeyValueParser, DEFAULT_CONFIG +from utils.console import YELLOW, BOLD, NORMAL, eprint +from utils.download import run_lxc_download_script +from utils.files import stat_chmod, get_mount_point +from utils.jail_dataset import check_jail_name_valid, check_jail_name_available +from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path +from utils.jail_dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset, cleanup +from utils.paths import COMMAND_NAME, JAILS_DIR_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH + + +def create_jail(**kwargs): + print(DISCLAIMER) + + if os.path.basename(SCRIPT_DIR_PATH) != "jailmaker": + eprint( + dedent( + f""" + {COMMAND_NAME} needs to create files. + Currently it can not decide if it is safe to create files in: + {SCRIPT_DIR_PATH} + Please create a dedicated dataset called "jailmaker", store {SCRIPT_NAME} there and try again.""" + ) + ) + return 1 + + if not PurePath(get_mount_point(SCRIPT_DIR_PATH)).is_relative_to("/mnt"): + print( + dedent( + f""" + {YELLOW}{BOLD}WARNING: BEWARE OF DATA LOSS{NORMAL} + + {SCRIPT_NAME} should be on a dataset mounted under /mnt (it currently is not). + Storing it on the boot-pool means losing all jails when updating TrueNAS. + Jails will be stored under: + {SCRIPT_DIR_PATH} + """ + ) + ) + + jail_name = kwargs.pop("jail_name") + start_now = False + + if not check_jail_name_valid(jail_name): + return 1 + + if not check_jail_name_available(jail_name): + return 1 + + start_now = kwargs.pop("start", start_now) + jail_config_path = kwargs.pop("config") + + config = KeyValueParser() + + if jail_config_path: + # TODO: fallback to default values for e.g. distro and release if they are not in the config file + if jail_config_path == "-": + print(f"Creating jail {jail_name} from config template passed via stdin.") + config.read_string(sys.stdin.read()) + else: + print(f"Creating jail {jail_name} from config template {jail_config_path}.") + if jail_config_path not in config.read(jail_config_path): + eprint(f"Failed to read config template {jail_config_path}.") + return 1 + else: + print(f"Creating jail {jail_name} with default config.") + config.read_string(DEFAULT_CONFIG) + + for option in [ + "distro", + "gpu_passthrough_intel", + "gpu_passthrough_nvidia", + "release", + "seccomp", + "startup", + "systemd_nspawn_user_args", + ]: + value = kwargs.pop(option) + if ( + value is not None + # String, non-empty list of args or int + and (isinstance(value, int) or len(value)) + and value is not config.my_get(option, None) + ): + # TODO: this will wipe all systemd_nspawn_user_args from the template... + # Should there be an option to append them instead? + print(f"Overriding {option} config value with {value}.") + config.my_set(option, value) + + jail_path = get_jail_path(jail_name) + + distro = config.my_get("distro") + release = config.my_get("release") + + # Cleanup in except, but only once the jail_path is final + # Otherwise we may cleanup the wrong directory + try: + # Create the dir or dataset where to store the jails + if not os.path.exists(JAILS_DIR_PATH): + if get_zfs_dataset(SCRIPT_DIR_PATH): + # Creating "jails" dataset if "jailmaker" is a ZFS Dataset + create_zfs_dataset(JAILS_DIR_PATH) + else: + os.makedirs(JAILS_DIR_PATH, exist_ok=True) + stat_chmod(JAILS_DIR_PATH, 0o700) + + # Creating a dataset for the jail if the jails dir is a dataset + if get_zfs_dataset(JAILS_DIR_PATH): + create_zfs_dataset(jail_path) + + jail_config_path = get_jail_config_path(jail_name) + jail_rootfs_path = get_jail_rootfs_path(jail_name) + + # Create directory for rootfs + os.makedirs(jail_rootfs_path, exist_ok=True) + # LXC download script needs to write to this file during install + # but we don't need it so we will remove it later + open(jail_config_path, "a").close() + + if ( + returncode := run_lxc_download_script( + jail_name, jail_path, jail_rootfs_path, distro, release + ) + != 0 + ): + cleanup(jail_path) + return returncode + + # Assuming the name of your jail is "myjail" + # and "machinectl shell myjail" doesn't work + # Try: + # + # Stop the jail with: + # machinectl stop myjail + # And start a shell inside the jail without the --boot option: + # systemd-nspawn -q -D jails/myjail/rootfs /bin/sh + # Then set a root password with: + # In case of amazonlinux you may need to run: + # yum update -y && yum install -y passwd + # passwd + # exit + # Then you may login from the host via: + # machinectl login myjail + # + # You could also enable SSH inside the jail to login + # + # Or if that doesn't work (e.g. for alpine) get a shell via: + # nsenter -t $(machinectl show myjail -p Leader --value) -a /bin/sh -l + # But alpine jails made with jailmaker have other issues + # They don't shutdown cleanly via systemctl and machinectl... + + with Chroot(jail_rootfs_path): + # Use chroot to correctly resolve absolute /sbin/init symlink + init_system_name = os.path.basename(os.path.realpath("/sbin/init")) + + if ( + init_system_name != "systemd" + and parse_os_release(jail_rootfs_path).get("ID") != "nixos" + ): + print( + dedent( + f""" + {YELLOW}{BOLD}WARNING: DISTRO NOT SUPPORTED{NORMAL} + + Chosen distro appears not to use systemd... + + You probably will not get a shell with: + machinectl shell {jail_name} + + You may get a shell with this command: + nsenter -t $(machinectl show {jail_name} -p Leader --value) -a /bin/sh -l + + Read about the downsides of nsenter: + https://github.com/systemd/systemd/issues/12785#issuecomment-503019081 + + {BOLD}Using this distro with {COMMAND_NAME} is NOT recommended.{NORMAL} + """ + ) + ) + + print("Autostart has been disabled.") + print("You need to start this jail manually.") + config.my_set("startup", 0) + start_now = False + + # Remove config which systemd handles for us + with contextlib.suppress(FileNotFoundError): + os.remove(os.path.join(jail_rootfs_path, "etc/machine-id")) + with contextlib.suppress(FileNotFoundError): + os.remove(os.path.join(jail_rootfs_path, "etc/resolv.conf")) + + # https://github.com/systemd/systemd/issues/852 + print( + "\n".join([f"pts/{i}" for i in range(0, 11)]), + file=open(os.path.join(jail_rootfs_path, "etc/securetty"), "w"), + ) + + network_dir_path = os.path.join(jail_rootfs_path, "etc/systemd/network") + + # Modify default network settings, if network_dir_path exists + if os.path.isdir(network_dir_path): + default_host0_network_file = os.path.join( + jail_rootfs_path, "lib/systemd/network/80-container-host0.network" + ) + + # Check if default host0 network file exists + if os.path.isfile(default_host0_network_file): + override_network_file = os.path.join( + network_dir_path, "80-container-host0.network" + ) + + # Override the default 80-container-host0.network file (by using the same name) + # This config applies when using the --network-bridge option of systemd-nspawn + # Disable LinkLocalAddressing on IPv4, or else the container won't get IP address via DHCP + # But keep it enabled on IPv6, as SLAAC and DHCPv6 both require a local-link address to function + print( + Path(default_host0_network_file) + .read_text() + .replace("LinkLocalAddressing=yes", "LinkLocalAddressing=ipv6"), + file=open(override_network_file, "w"), + ) + + # Setup DHCP for macvlan network interfaces + # This config applies when using the --network-macvlan option of systemd-nspawn + # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui + print( + cleandoc( + """ + [Match] + Virtualization=container + Name=mv-* + + [Network] + DHCP=yes + LinkLocalAddressing=ipv6 + + [DHCPv4] + UseDNS=true + UseTimezone=true + """ + ), + file=open(os.path.join(network_dir_path, "mv-dhcp.network"), "w"), + ) + + # Setup DHCP for veth-extra network interfaces + # This config applies when using the --network-veth-extra option of systemd-nspawn + # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui + print( + cleandoc( + """ + [Match] + Virtualization=container + Name=vee-* + + [Network] + DHCP=yes + LinkLocalAddressing=ipv6 + + [DHCPv4] + UseDNS=true + UseTimezone=true + """ + ), + file=open(os.path.join(network_dir_path, "vee-dhcp.network"), "w"), + ) + + # Override preset which caused systemd-networkd to be disabled (e.g. fedora 39) + # https://www.freedesktop.org/software/systemd/man/latest/systemd.preset.html + # https://github.com/lxc/lxc-ci/blob/f632823ecd9b258ed42df40449ec54ed7ef8e77d/images/fedora.yaml#L312C5-L312C38 + + preset_path = os.path.join(jail_rootfs_path, "etc/systemd/system-preset") + os.makedirs(preset_path, exist_ok=True) + print( + "enable systemd-networkd.service", + file=open(os.path.join(preset_path, "00-jailmaker.preset"), "w"), + ) + + with open(jail_config_path, "w") as fp: + config.write(fp) + + os.chmod(jail_config_path, 0o600) + + # Cleanup on any exception and rethrow + except BaseException as error: + cleanup(jail_path) + raise error + + if start_now: + return start_jail(jail_name) + + return 0 diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py index 6b3ca4b..a0b9fad 100644 --- a/src/jlmkr/actions/list.py +++ b/src/jlmkr/actions/list.py @@ -8,8 +8,7 @@ from collections import defaultdict from utils.console import NORMAL, UNDERLINE from utils.config_parser import parse_config_file -from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path, parse_os_release -from utils.parent_dataset import get_all_jail_names +from utils.jail_dataset import get_all_jail_names, get_jail_config_path, get_jail_rootfs_path, parse_os_release def list_jails(): diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 1f866dc..7d419df 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -26,7 +26,6 @@ import tempfile import time import urllib.request -from inspect import cleandoc from pathlib import Path, PurePath from textwrap import dedent @@ -70,6 +69,7 @@ def error(self, message): from utils.chroot import Chroot from utils.console import eprint, fail +from utils.jail_dataset import check_jail_name_valid, check_jail_name_available from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path from actions.exec import exec_jail @@ -80,139 +80,10 @@ def error(self, message): from actions.restart import restart_jail from actions.images import run_lxc_download_script - -def cleanup(jail_path): - """ - Cleanup jail. - """ - - if get_zfs_dataset(jail_path): - eprint(f"Cleaning up: {jail_path}.") - remove_zfs_dataset(jail_path) - - elif os.path.isdir(jail_path): - # Workaround for https://github.com/python/cpython/issues/73885 - # Should be fixed in Python 3.13 https://stackoverflow.com/a/70549000 - def _onerror(func, path, exc_info): - exc_type, exc_value, exc_traceback = exc_info - if issubclass(exc_type, PermissionError): - # Update the file permissions with the immutable and append-only bit cleared - subprocess.run(["chattr", "-i", "-a", path]) - # Reattempt the removal - func(path) - elif not issubclass(exc_type, FileNotFoundError): - raise exc_value - - eprint(f"Cleaning up: {jail_path}.") - shutil.rmtree(jail_path, onerror=_onerror) - - +from utils.jail_dataset import cleanup, check_jail_name_valid, check_jail_name_available +from utils.download import run_lxc_download_script from utils.files import stat_chmod - - -def get_mount_point(path): - """ - Return the mount point on which the given path resides. - """ - path = os.path.abspath(path) - while not os.path.ismount(path): - path = os.path.dirname(path) - return path - - -def get_relative_path_in_jailmaker_dir(absolute_path): - return PurePath(absolute_path).relative_to(SCRIPT_DIR_PATH) - - -def get_zfs_dataset(path): - """ - Get ZFS dataset path. - """ - - def clean_field(field): - # Put back spaces which were encoded - # https://github.com/openzfs/zfs/issues/11182 - return field.replace("\\040", " ") - - path = os.path.realpath(path) - with open("/proc/mounts", "r") as f: - for line in f: - fields = line.split() - if "zfs" == fields[2] and path == clean_field(fields[1]): - return clean_field(fields[0]) - - -def get_zfs_base_path(): - """ - Get ZFS dataset path for jailmaker directory. - """ - zfs_base_path = get_zfs_dataset(SCRIPT_DIR_PATH) - if not zfs_base_path: - fail("Failed to get dataset path for jailmaker directory.") - - return zfs_base_path - - -def create_zfs_dataset(absolute_path): - """ - Create a ZFS Dataset inside the jailmaker directory at the provided absolute path. - E.g. "/mnt/mypool/jailmaker/jails" or "/mnt/mypool/jailmaker/jails/newjail"). - """ - relative_path = get_relative_path_in_jailmaker_dir(absolute_path) - dataset_to_create = os.path.join(get_zfs_base_path(), relative_path) - eprint(f"Creating ZFS Dataset {dataset_to_create}") - subprocess.run(["zfs", "create", dataset_to_create], check=True) - - -def remove_zfs_dataset(absolute_path): - """ - Remove a ZFS Dataset inside the jailmaker directory at the provided absolute path. - E.g. "/mnt/mypool/jailmaker/jails/oldjail". - """ - relative_path = get_relative_path_in_jailmaker_dir(absolute_path) - dataset_to_remove = os.path.join((get_zfs_base_path()), relative_path) - eprint(f"Removing ZFS Dataset {dataset_to_remove}") - subprocess.run(["zfs", "destroy", "-r", dataset_to_remove], check=True) - - -def check_jail_name_valid(jail_name, warn=True): - """ - Return True if jail name matches the required format. - """ - if ( - re.match(r"^[.a-zA-Z0-9-]{1,64}$", jail_name) - and not jail_name.startswith(".") - and ".." not in jail_name - ): - return True - - if warn: - eprint( - dedent( - f""" - {YELLOW}{BOLD}WARNING: INVALID NAME{NORMAL} - - A valid name consists of: - - allowed characters (alphanumeric, dash, dot) - - no leading or trailing dots - - no sequences of multiple dots - - max 64 characters""" - ) - ) - return False - - -def check_jail_name_available(jail_name, warn=True): - """ - Return True if jail name is not yet taken. - """ - if not os.path.exists(get_jail_path(jail_name)): - return True - - if warn: - print() - eprint("A jail with this name already exists.") - return False +from utils.jail_dataset import get_zfs_dataset, create_zfs_dataset, remove_zfs_dataset def get_text_editor(): @@ -229,288 +100,7 @@ def get_from_environ(key): ) -def create_jail(**kwargs): - print(DISCLAIMER) - - if os.path.basename(SCRIPT_DIR_PATH) != "jailmaker": - eprint( - dedent( - f""" - {COMMAND_NAME} needs to create files. - Currently it can not decide if it is safe to create files in: - {SCRIPT_DIR_PATH} - Please create a dedicated dataset called "jailmaker", store {SCRIPT_NAME} there and try again.""" - ) - ) - return 1 - - if not PurePath(get_mount_point(SCRIPT_DIR_PATH)).is_relative_to("/mnt"): - print( - dedent( - f""" - {YELLOW}{BOLD}WARNING: BEWARE OF DATA LOSS{NORMAL} - - {SCRIPT_NAME} should be on a dataset mounted under /mnt (it currently is not). - Storing it on the boot-pool means losing all jails when updating TrueNAS. - Jails will be stored under: - {SCRIPT_DIR_PATH} - """ - ) - ) - - jail_name = kwargs.pop("jail_name") - start_now = False - - if not check_jail_name_valid(jail_name): - return 1 - - if not check_jail_name_available(jail_name): - return 1 - - start_now = kwargs.pop("start", start_now) - jail_config_path = kwargs.pop("config") - - config = KeyValueParser() - - if jail_config_path: - # TODO: fallback to default values for e.g. distro and release if they are not in the config file - if jail_config_path == "-": - print(f"Creating jail {jail_name} from config template passed via stdin.") - config.read_string(sys.stdin.read()) - else: - print(f"Creating jail {jail_name} from config template {jail_config_path}.") - if jail_config_path not in config.read(jail_config_path): - eprint(f"Failed to read config template {jail_config_path}.") - return 1 - else: - print(f"Creating jail {jail_name} with default config.") - config.read_string(DEFAULT_CONFIG) - - for option in [ - "distro", - "gpu_passthrough_intel", - "gpu_passthrough_nvidia", - "release", - "seccomp", - "startup", - "systemd_nspawn_user_args", - ]: - value = kwargs.pop(option) - if ( - value is not None - # String, non-empty list of args or int - and (isinstance(value, int) or len(value)) - and value is not config.my_get(option, None) - ): - # TODO: this will wipe all systemd_nspawn_user_args from the template... - # Should there be an option to append them instead? - print(f"Overriding {option} config value with {value}.") - config.my_set(option, value) - - jail_path = get_jail_path(jail_name) - - distro = config.my_get("distro") - release = config.my_get("release") - - # Cleanup in except, but only once the jail_path is final - # Otherwise we may cleanup the wrong directory - try: - # Create the dir or dataset where to store the jails - if not os.path.exists(JAILS_DIR_PATH): - if get_zfs_dataset(SCRIPT_DIR_PATH): - # Creating "jails" dataset if "jailmaker" is a ZFS Dataset - create_zfs_dataset(JAILS_DIR_PATH) - else: - os.makedirs(JAILS_DIR_PATH, exist_ok=True) - stat_chmod(JAILS_DIR_PATH, 0o700) - - # Creating a dataset for the jail if the jails dir is a dataset - if get_zfs_dataset(JAILS_DIR_PATH): - create_zfs_dataset(jail_path) - - jail_config_path = get_jail_config_path(jail_name) - jail_rootfs_path = get_jail_rootfs_path(jail_name) - - # Create directory for rootfs - os.makedirs(jail_rootfs_path, exist_ok=True) - # LXC download script needs to write to this file during install - # but we don't need it so we will remove it later - open(jail_config_path, "a").close() - - if ( - returncode := run_lxc_download_script( - jail_name, jail_path, jail_rootfs_path, distro, release - ) - != 0 - ): - cleanup(jail_path) - return returncode - - # Assuming the name of your jail is "myjail" - # and "machinectl shell myjail" doesn't work - # Try: - # - # Stop the jail with: - # machinectl stop myjail - # And start a shell inside the jail without the --boot option: - # systemd-nspawn -q -D jails/myjail/rootfs /bin/sh - # Then set a root password with: - # In case of amazonlinux you may need to run: - # yum update -y && yum install -y passwd - # passwd - # exit - # Then you may login from the host via: - # machinectl login myjail - # - # You could also enable SSH inside the jail to login - # - # Or if that doesn't work (e.g. for alpine) get a shell via: - # nsenter -t $(machinectl show myjail -p Leader --value) -a /bin/sh -l - # But alpine jails made with jailmaker have other issues - # They don't shutdown cleanly via systemctl and machinectl... - - with Chroot(jail_rootfs_path): - # Use chroot to correctly resolve absolute /sbin/init symlink - init_system_name = os.path.basename(os.path.realpath("/sbin/init")) - - if ( - init_system_name != "systemd" - and parse_os_release(jail_rootfs_path).get("ID") != "nixos" - ): - print( - dedent( - f""" - {YELLOW}{BOLD}WARNING: DISTRO NOT SUPPORTED{NORMAL} - - Chosen distro appears not to use systemd... - - You probably will not get a shell with: - machinectl shell {jail_name} - - You may get a shell with this command: - nsenter -t $(machinectl show {jail_name} -p Leader --value) -a /bin/sh -l - - Read about the downsides of nsenter: - https://github.com/systemd/systemd/issues/12785#issuecomment-503019081 - - {BOLD}Using this distro with {COMMAND_NAME} is NOT recommended.{NORMAL} - """ - ) - ) - - print("Autostart has been disabled.") - print("You need to start this jail manually.") - config.my_set("startup", 0) - start_now = False - - # Remove config which systemd handles for us - with contextlib.suppress(FileNotFoundError): - os.remove(os.path.join(jail_rootfs_path, "etc/machine-id")) - with contextlib.suppress(FileNotFoundError): - os.remove(os.path.join(jail_rootfs_path, "etc/resolv.conf")) - - # https://github.com/systemd/systemd/issues/852 - print( - "\n".join([f"pts/{i}" for i in range(0, 11)]), - file=open(os.path.join(jail_rootfs_path, "etc/securetty"), "w"), - ) - - network_dir_path = os.path.join(jail_rootfs_path, "etc/systemd/network") - - # Modify default network settings, if network_dir_path exists - if os.path.isdir(network_dir_path): - default_host0_network_file = os.path.join( - jail_rootfs_path, "lib/systemd/network/80-container-host0.network" - ) - - # Check if default host0 network file exists - if os.path.isfile(default_host0_network_file): - override_network_file = os.path.join( - network_dir_path, "80-container-host0.network" - ) - - # Override the default 80-container-host0.network file (by using the same name) - # This config applies when using the --network-bridge option of systemd-nspawn - # Disable LinkLocalAddressing on IPv4, or else the container won't get IP address via DHCP - # But keep it enabled on IPv6, as SLAAC and DHCPv6 both require a local-link address to function - print( - Path(default_host0_network_file) - .read_text() - .replace("LinkLocalAddressing=yes", "LinkLocalAddressing=ipv6"), - file=open(override_network_file, "w"), - ) - - # Setup DHCP for macvlan network interfaces - # This config applies when using the --network-macvlan option of systemd-nspawn - # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui - print( - cleandoc( - """ - [Match] - Virtualization=container - Name=mv-* - - [Network] - DHCP=yes - LinkLocalAddressing=ipv6 - - [DHCPv4] - UseDNS=true - UseTimezone=true - """ - ), - file=open(os.path.join(network_dir_path, "mv-dhcp.network"), "w"), - ) - - # Setup DHCP for veth-extra network interfaces - # This config applies when using the --network-veth-extra option of systemd-nspawn - # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui - print( - cleandoc( - """ - [Match] - Virtualization=container - Name=vee-* - - [Network] - DHCP=yes - LinkLocalAddressing=ipv6 - - [DHCPv4] - UseDNS=true - UseTimezone=true - """ - ), - file=open(os.path.join(network_dir_path, "vee-dhcp.network"), "w"), - ) - - # Override preset which caused systemd-networkd to be disabled (e.g. fedora 39) - # https://www.freedesktop.org/software/systemd/man/latest/systemd.preset.html - # https://github.com/lxc/lxc-ci/blob/f632823ecd9b258ed42df40449ec54ed7ef8e77d/images/fedora.yaml#L312C5-L312C38 - - preset_path = os.path.join(jail_rootfs_path, "etc/systemd/system-preset") - os.makedirs(preset_path, exist_ok=True) - print( - "enable systemd-networkd.service", - file=open(os.path.join(preset_path, "00-jailmaker.preset"), "w"), - ) - - with open(jail_config_path, "w") as fp: - config.write(fp) - - os.chmod(jail_config_path, 0o600) - - # Cleanup on any exception and rethrow - except BaseException as error: - cleanup(jail_path) - raise error - - if start_now: - return start_jail(jail_name) - - return 0 - - +from actions.create import create_jail from utils.jail_dataset import jail_is_running @@ -574,8 +164,7 @@ def remove_jail(jail_name): return 1 -from utils.parent_dataset import get_all_jail_names -from utils.jail_dataset import parse_os_release +from utils.jail_dataset import get_all_jail_names, parse_os_release from actions.list import list_jails diff --git a/src/jlmkr/utils/files.py b/src/jlmkr/utils/files.py index a328223..1c15158 100644 --- a/src/jlmkr/utils/files.py +++ b/src/jlmkr/utils/files.py @@ -12,3 +12,13 @@ def stat_chmod(file_path, mode): """ if mode != stat.S_IMODE(os.stat(file_path).st_mode): os.chmod(file_path, mode) + + +def get_mount_point(path): + """ + Return the mount point on which the given path resides. + """ + path = os.path.abspath(path) + while not os.path.ismount(path): + path = os.path.dirname(path) + return path diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/jail_dataset.py index 7cad961..d61bbdd 100644 --- a/src/jlmkr/utils/jail_dataset.py +++ b/src/jlmkr/utils/jail_dataset.py @@ -4,10 +4,15 @@ import os.path import platform +import re +import shutil import subprocess +from pathlib import PurePath +from textwrap import dedent from utils.chroot import Chroot -from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME +from utils.console import eprint, YELLOW, BOLD, NORMAL +from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME, SCRIPT_DIR_PATH def get_jail_path(jail_name): @@ -22,6 +27,50 @@ def get_jail_rootfs_path(jail_name): return os.path.join(get_jail_path(jail_name), JAIL_ROOTFS_NAME) +def get_relative_path_in_jailmaker_dir(absolute_path): + return PurePath(absolute_path).relative_to(SCRIPT_DIR_PATH) + + +def check_jail_name_valid(jail_name, warn=True): + """ + Return True if jail name matches the required format. + """ + if ( + re.match(r"^[.a-zA-Z0-9-]{1,64}$", jail_name) + and not jail_name.startswith(".") + and ".." not in jail_name + ): + return True + + if warn: + eprint( + dedent( + f""" + {YELLOW}{BOLD}WARNING: INVALID NAME{NORMAL} + + A valid name consists of: + - allowed characters (alphanumeric, dash, dot) + - no leading or trailing dots + - no sequences of multiple dots + - max 64 characters""" + ) + ) + return False + + +def check_jail_name_available(jail_name, warn=True): + """ + Return True if jail name is not yet taken. + """ + if not os.path.exists(get_jail_path(jail_name)): + return True + + if warn: + print() + eprint("A jail with this name already exists.") + return False + + def parse_os_release(new_root): result = {} with Chroot(new_root): @@ -48,3 +97,89 @@ def jail_is_running(jail_name): ).returncode == 0 ) + + +def get_zfs_dataset(path): + """ + Get ZFS dataset path. + """ + + def clean_field(field): + # Put back spaces which were encoded + # https://github.com/openzfs/zfs/issues/11182 + return field.replace("\\040", " ") + + path = os.path.realpath(path) + with open("/proc/mounts", "r") as f: + for line in f: + fields = line.split() + if "zfs" == fields[2] and path == clean_field(fields[1]): + return clean_field(fields[0]) + + +def get_zfs_base_path(): + """ + Get ZFS dataset path for jailmaker directory. + """ + zfs_base_path = get_zfs_dataset(SCRIPT_DIR_PATH) + if not zfs_base_path: + fail("Failed to get dataset path for jailmaker directory.") + + return zfs_base_path + + +def get_all_jail_names(): + try: + jail_names = os.listdir(JAILS_DIR_PATH) + except FileNotFoundError: + jail_names = [] + + return jail_names + + +def create_zfs_dataset(absolute_path): + """ + Create a ZFS Dataset inside the jailmaker directory at the provided absolute path. + E.g. "/mnt/mypool/jailmaker/jails" or "/mnt/mypool/jailmaker/jails/newjail"). + """ + relative_path = get_relative_path_in_jailmaker_dir(absolute_path) + dataset_to_create = os.path.join(get_zfs_base_path(), relative_path) + eprint(f"Creating ZFS Dataset {dataset_to_create}") + subprocess.run(["zfs", "create", dataset_to_create], check=True) + + +def remove_zfs_dataset(absolute_path): + """ + Remove a ZFS Dataset inside the jailmaker directory at the provided absolute path. + E.g. "/mnt/mypool/jailmaker/jails/oldjail". + """ + relative_path = get_relative_path_in_jailmaker_dir(absolute_path) + dataset_to_remove = os.path.join((get_zfs_base_path()), relative_path) + eprint(f"Removing ZFS Dataset {dataset_to_remove}") + subprocess.run(["zfs", "destroy", "-r", dataset_to_remove], check=True) + + +def cleanup(jail_path): + """ + Cleanup jail. + """ + + if get_zfs_dataset(jail_path): + eprint(f"Cleaning up: {jail_path}.") + remove_zfs_dataset(jail_path) + + elif os.path.isdir(jail_path): + # Workaround for https://github.com/python/cpython/issues/73885 + # Should be fixed in Python 3.13 https://stackoverflow.com/a/70549000 + def _onerror(func, path, exc_info): + exc_type, exc_value, exc_traceback = exc_info + if issubclass(exc_type, PermissionError): + # Update the file permissions with the immutable and append-only bit cleared + subprocess.run(["chattr", "-i", "-a", path]) + # Reattempt the removal + func(path) + elif not issubclass(exc_type, FileNotFoundError): + raise exc_value + + eprint(f"Cleaning up: {jail_path}.") + shutil.rmtree(jail_path, onerror=_onerror) diff --git a/src/jlmkr/utils/parent_dataset.py b/src/jlmkr/utils/parent_dataset.py deleted file mode 100644 index fd68893..0000000 --- a/src/jlmkr/utils/parent_dataset.py +++ /dev/null @@ -1,16 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -import os - -from utils.paths import JAILS_DIR_PATH - - -def get_all_jail_names(): - try: - jail_names = os.listdir(JAILS_DIR_PATH) - except FileNotFoundError: - jail_names = [] - - return jail_names From 58e8b10553062a8ec26bd27464c1a909aeebb414 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:43:47 -0400 Subject: [PATCH 29/58] Extract remove action --- src/jlmkr/actions/remove.py | 39 +++++++++++++++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 33 +------------------------------ 2 files changed, 40 insertions(+), 32 deletions(-) create mode 100644 src/jlmkr/actions/remove.py diff --git a/src/jlmkr/actions/remove.py b/src/jlmkr/actions/remove.py new file mode 100644 index 0000000..0cf2673 --- /dev/null +++ b/src/jlmkr/actions/remove.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from actions.stop import stop_jail +from utils.jail_dataset import check_jail_name_valid, check_jail_name_available +from utils.jail_dataset import get_jail_path, cleanup +from utils.console import eprint + + +def remove_jail(jail_name): + """ + Remove jail with given name. + """ + + if not check_jail_name_valid(jail_name): + return 1 + + if check_jail_name_available(jail_name, False): + eprint(f"A jail with name {jail_name} does not exist.") + return 1 + + # TODO: print which dataset is about to be removed before the user confirmation + # TODO: print that all zfs snapshots will be removed if jail has it's own zfs dataset + check = input(f'\nCAUTION: Type "{jail_name}" to confirm jail deletion!\n\n') + + if check == jail_name: + print() + jail_path = get_jail_path(jail_name) + returncode = stop_jail(jail_name) + if returncode != 0: + return returncode + + print() + cleanup(jail_path) + return 0 + else: + eprint("Wrong name, nothing happened.") + return 1 diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 7d419df..ff6163f 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -131,38 +131,7 @@ def edit_jail(jail_name): from actions.stop import stop_jail - - -def remove_jail(jail_name): - """ - Remove jail with given name. - """ - - if not check_jail_name_valid(jail_name): - return 1 - - if check_jail_name_available(jail_name, False): - eprint(f"A jail with name {jail_name} does not exist.") - return 1 - - # TODO: print which dataset is about to be removed before the user confirmation - # TODO: print that all zfs snapshots will be removed if jail has it's own zfs dataset - check = input(f'\nCAUTION: Type "{jail_name}" to confirm jail deletion!\n\n') - - if check == jail_name: - print() - jail_path = get_jail_path(jail_name) - returncode = stop_jail(jail_name) - if returncode != 0: - return returncode - - print() - cleanup(jail_path) - return 0 - else: - eprint("Wrong name, nothing happened.") - return 1 - +from actions.remove import remove_jail from utils.jail_dataset import get_all_jail_names, parse_os_release from actions.list import list_jails From 25100ef39dbefe6c51e57f83719ae9f242ba3e33 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:51:46 -0400 Subject: [PATCH 30/58] Extract text editor utils --- src/jlmkr/donor/jlmkr.py | 16 +--------------- src/jlmkr/utils/editor.py | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 15 deletions(-) create mode 100644 src/jlmkr/utils/editor.py diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index ff6163f..a2d531a 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -85,22 +85,8 @@ def error(self, message): from utils.files import stat_chmod from utils.jail_dataset import get_zfs_dataset, create_zfs_dataset, remove_zfs_dataset - -def get_text_editor(): - def get_from_environ(key): - if editor := os.environ.get(key): - return shutil.which(editor) - - return ( - get_from_environ("VISUAL") - or get_from_environ("EDITOR") - or shutil.which("editor") - or shutil.which("/usr/bin/editor") - or "nano" - ) - - from actions.create import create_jail +from utils.editor import get_text_editor from utils.jail_dataset import jail_is_running diff --git a/src/jlmkr/utils/editor.py b/src/jlmkr/utils/editor.py new file mode 100644 index 0000000..2a652c3 --- /dev/null +++ b/src/jlmkr/utils/editor.py @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os +import shutil + + +def get_text_editor(): + def get_from_environ(key): + if editor := os.environ.get(key): + return shutil.which(editor) + + return ( + get_from_environ("VISUAL") + or get_from_environ("EDITOR") + or shutil.which("editor") + or shutil.which("/usr/bin/editor") + or "nano" + ) From 7e6754cf99e21f18264146985d3c54fc9d4eca45 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:52:07 -0400 Subject: [PATCH 31/58] Extract edit action --- src/jlmkr/actions/edit.py | 37 +++++++++++++++++++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 28 +--------------------------- 2 files changed, 38 insertions(+), 27 deletions(-) create mode 100644 src/jlmkr/actions/edit.py diff --git a/src/jlmkr/actions/edit.py b/src/jlmkr/actions/edit.py new file mode 100644 index 0000000..fd5f03c --- /dev/null +++ b/src/jlmkr/actions/edit.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +import os +import subprocess + +from utils.console import eprint +from utils.editor import get_text_editor +from utils.jail_dataset import check_jail_name_valid, check_jail_name_available +from utils.jail_dataset import get_jail_config_path, jail_is_running + + +def edit_jail(jail_name): + """ + Edit jail with given name. + """ + + if not check_jail_name_valid(jail_name): + return 1 + + if check_jail_name_available(jail_name, False): + eprint(f"A jail with name {jail_name} does not exist.") + return 1 + + jail_config_path = get_jail_config_path(jail_name) + + returncode = subprocess.run([get_text_editor(), jail_config_path]).returncode + + if returncode != 0: + eprint(f"An error occurred while editing {jail_config_path}.") + return returncode + + if jail_is_running(jail_name): + print("\nRestart the jail for edits to apply (if you made any).") + + return 0 diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index a2d531a..3a839b2 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -89,33 +89,7 @@ def error(self, message): from utils.editor import get_text_editor from utils.jail_dataset import jail_is_running - -def edit_jail(jail_name): - """ - Edit jail with given name. - """ - - if not check_jail_name_valid(jail_name): - return 1 - - if check_jail_name_available(jail_name, False): - eprint(f"A jail with name {jail_name} does not exist.") - return 1 - - jail_config_path = get_jail_config_path(jail_name) - - returncode = subprocess.run([get_text_editor(), jail_config_path]).returncode - - if returncode != 0: - eprint(f"An error occurred while editing {jail_config_path}.") - return returncode - - if jail_is_running(jail_name): - print("\nRestart the jail for edits to apply (if you made any).") - - return 0 - - +from actions.edit import edit_jail from actions.stop import stop_jail from actions.remove import remove_jail From 3125b13aa26248b1b98f15ad9eb3d8d0648ab10e Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 03:57:47 -0400 Subject: [PATCH 32/58] Rename dataset utils --- src/jlmkr/actions/create.py | 6 +++--- src/jlmkr/actions/edit.py | 4 ++-- src/jlmkr/actions/list.py | 2 +- src/jlmkr/actions/remove.py | 4 ++-- src/jlmkr/actions/start.py | 4 ++-- src/jlmkr/actions/stop.py | 2 +- src/jlmkr/donor/jlmkr.py | 12 ++++++------ src/jlmkr/utils/{jail_dataset.py => dataset.py} | 0 src/jlmkr/utils/gpu.py | 2 +- 9 files changed, 18 insertions(+), 18 deletions(-) rename src/jlmkr/utils/{jail_dataset.py => dataset.py} (100%) diff --git a/src/jlmkr/actions/create.py b/src/jlmkr/actions/create.py index b91fe6a..5109d2b 100644 --- a/src/jlmkr/actions/create.py +++ b/src/jlmkr/actions/create.py @@ -15,9 +15,9 @@ from utils.console import YELLOW, BOLD, NORMAL, eprint from utils.download import run_lxc_download_script from utils.files import stat_chmod, get_mount_point -from utils.jail_dataset import check_jail_name_valid, check_jail_name_available -from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path -from utils.jail_dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset, cleanup +from utils.dataset import check_jail_name_valid, check_jail_name_available +from utils.dataset import get_jail_config_path, get_jail_rootfs_path +from utils.dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset, cleanup from utils.paths import COMMAND_NAME, JAILS_DIR_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH diff --git a/src/jlmkr/actions/edit.py b/src/jlmkr/actions/edit.py index fd5f03c..aecd040 100644 --- a/src/jlmkr/actions/edit.py +++ b/src/jlmkr/actions/edit.py @@ -7,8 +7,8 @@ from utils.console import eprint from utils.editor import get_text_editor -from utils.jail_dataset import check_jail_name_valid, check_jail_name_available -from utils.jail_dataset import get_jail_config_path, jail_is_running +from utils.dataset import check_jail_name_valid, check_jail_name_available +from utils.dataset import get_jail_config_path, jail_is_running def edit_jail(jail_name): diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py index a0b9fad..c444cc6 100644 --- a/src/jlmkr/actions/list.py +++ b/src/jlmkr/actions/list.py @@ -8,7 +8,7 @@ from collections import defaultdict from utils.console import NORMAL, UNDERLINE from utils.config_parser import parse_config_file -from utils.jail_dataset import get_all_jail_names, get_jail_config_path, get_jail_rootfs_path, parse_os_release +from utils.dataset import get_all_jail_names, get_jail_config_path, get_jail_rootfs_path, parse_os_release def list_jails(): diff --git a/src/jlmkr/actions/remove.py b/src/jlmkr/actions/remove.py index 0cf2673..28588f9 100644 --- a/src/jlmkr/actions/remove.py +++ b/src/jlmkr/actions/remove.py @@ -3,8 +3,8 @@ # SPDX-License-Identifier: LGPL-3.0-only from actions.stop import stop_jail -from utils.jail_dataset import check_jail_name_valid, check_jail_name_available -from utils.jail_dataset import get_jail_path, cleanup +from utils.dataset import check_jail_name_valid, check_jail_name_available +from utils.dataset import get_jail_path, cleanup from utils.console import eprint diff --git a/src/jlmkr/actions/start.py b/src/jlmkr/actions/start.py index abe50d9..790f5c2 100644 --- a/src/jlmkr/actions/start.py +++ b/src/jlmkr/actions/start.py @@ -13,8 +13,8 @@ from utils.console import eprint from utils.files import stat_chmod from utils.gpu import passthrough_intel, passthrough_nvidia -from utils.jail_dataset import get_jail_path, jail_is_running -from utils.jail_dataset import get_jail_config_path, get_jail_rootfs_path +from utils.dataset import get_jail_path, jail_is_running +from utils.dataset import get_jail_config_path, get_jail_rootfs_path from utils.paths import SHORTNAME, JAIL_ROOTFS_NAME diff --git a/src/jlmkr/actions/stop.py b/src/jlmkr/actions/stop.py index 9babdb4..71e3d87 100644 --- a/src/jlmkr/actions/stop.py +++ b/src/jlmkr/actions/stop.py @@ -6,7 +6,7 @@ import time from utils.console import eprint -from utils.jail_dataset import jail_is_running +from utils.dataset import jail_is_running def stop_jail(jail_name): diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 3a839b2..8fa2a5f 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -69,8 +69,8 @@ def error(self, message): from utils.chroot import Chroot from utils.console import eprint, fail -from utils.jail_dataset import check_jail_name_valid, check_jail_name_available -from utils.jail_dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path +from utils.dataset import check_jail_name_valid, check_jail_name_available +from utils.dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path from actions.exec import exec_jail from actions.status import status_jail @@ -80,20 +80,20 @@ def error(self, message): from actions.restart import restart_jail from actions.images import run_lxc_download_script -from utils.jail_dataset import cleanup, check_jail_name_valid, check_jail_name_available +from utils.dataset import cleanup, check_jail_name_valid, check_jail_name_available from utils.download import run_lxc_download_script from utils.files import stat_chmod -from utils.jail_dataset import get_zfs_dataset, create_zfs_dataset, remove_zfs_dataset +from utils.dataset import get_zfs_dataset, create_zfs_dataset, remove_zfs_dataset from actions.create import create_jail from utils.editor import get_text_editor -from utils.jail_dataset import jail_is_running +from utils.dataset import jail_is_running from actions.edit import edit_jail from actions.stop import stop_jail from actions.remove import remove_jail -from utils.jail_dataset import get_all_jail_names, parse_os_release +from utils.dataset import get_all_jail_names, parse_os_release from actions.list import list_jails diff --git a/src/jlmkr/utils/jail_dataset.py b/src/jlmkr/utils/dataset.py similarity index 100% rename from src/jlmkr/utils/jail_dataset.py rename to src/jlmkr/utils/dataset.py diff --git a/src/jlmkr/utils/gpu.py b/src/jlmkr/utils/gpu.py index b71c3ae..8ff1151 100644 --- a/src/jlmkr/utils/gpu.py +++ b/src/jlmkr/utils/gpu.py @@ -8,7 +8,7 @@ from pathlib import Path from textwrap import dedent from utils.console import eprint -from utils.jail_dataset import get_jail_rootfs_path +from utils.dataset import get_jail_rootfs_path from utils.paths import SHORTNAME From 2feb8739f46bb8375225f9341e29077ea679032d Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 04:06:44 -0400 Subject: [PATCH 33/58] Extract startup action --- src/jlmkr/actions/startup.py | 21 +++++++++++++++++++++ src/jlmkr/donor/jlmkr.py | 15 +-------------- 2 files changed, 22 insertions(+), 14 deletions(-) create mode 100644 src/jlmkr/actions/startup.py diff --git a/src/jlmkr/actions/startup.py b/src/jlmkr/actions/startup.py new file mode 100644 index 0000000..8314f8a --- /dev/null +++ b/src/jlmkr/actions/startup.py @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +from actions.start import start_jail +from utils.dataset import get_all_jail_names, get_jail_config_path +from utils.config_parser import parse_config_file + + +def startup_jails(): + start_failure = False + for jail_name in get_all_jail_names(): + config = parse_config_file(get_jail_config_path(jail_name)) + if config and config.my_getboolean("startup"): + if start_jail(jail_name) != 0: + start_failure = True + + if start_failure: + return 1 + + return 0 diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/donor/jlmkr.py index 8fa2a5f..12110ee 100755 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/donor/jlmkr.py @@ -95,20 +95,7 @@ def error(self, message): from utils.dataset import get_all_jail_names, parse_os_release from actions.list import list_jails - - -def startup_jails(): - start_failure = False - for jail_name in get_all_jail_names(): - config = parse_config_file(get_jail_config_path(jail_name)) - if config and config.my_getboolean("startup"): - if start_jail(jail_name) != 0: - start_failure = True - - if start_failure: - return 1 - - return 0 +from actions.startup import startup_jails def split_at_string(lst, string): From 3a0fb92ef0d87af317d8f2df187d6f4bdc96be9c Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 04:35:56 -0400 Subject: [PATCH 34/58] Extract command-line dispatch --- pyproject.toml | 8 +- src/jlmkr/__about__.py | 12 -- src/jlmkr/__main__.py | 18 ++- src/jlmkr/{donor/jlmkr.py => cli.py} | 187 +++++++++------------------ src/jlmkr/{donor => }/data.py | 6 + src/jlmkr/donor/__init__.py | 13 -- src/jlmkr/{utils => }/paths.py | 0 src/jlmkr/utils/config_parser.py | 2 +- src/jlmkr/utils/dataset.py | 3 +- src/jlmkr/utils/gpu.py | 3 +- 10 files changed, 91 insertions(+), 161 deletions(-) delete mode 100644 src/jlmkr/__about__.py rename src/jlmkr/{donor/jlmkr.py => cli.py} (79%) mode change 100755 => 100644 rename src/jlmkr/{donor => }/data.py (94%) delete mode 100644 src/jlmkr/donor/__init__.py rename src/jlmkr/{utils => }/paths.py (100%) diff --git a/pyproject.toml b/pyproject.toml index 2088b9d..b38cc72 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,16 +27,16 @@ GitHub = "https://github.com/Jip-Hop/jailmaker" [build-system] requires = [ "hatchling", -# "hatch-zipapp @ file:src/builder/app", -# "hatch-appzip @ file:src/builder/zip", + "hatch-zipapp @ file:src/builder/app", + "hatch-appzip @ file:src/builder/zip", ] build-backend = "hatchling.build" [project.scripts] -jlmkr = "jlmkr.donor:main" +jlmkr = "jlmkr:__main__" [tool.hatch.version] -path = "src/jlmkr/__about__.py" # or source = "vcs" +path = "src/jlmkr/__main__.py" # or source = "vcs" [tool.hatch.build.zipapp] dependencies = ["hatch-zipapp-builder @ file:src/builder/app"] diff --git a/src/jlmkr/__about__.py b/src/jlmkr/__about__.py deleted file mode 100644 index 8ac8fa5..0000000 --- a/src/jlmkr/__about__.py +++ /dev/null @@ -1,12 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -__version__ = "3.0.0.dev1" - -__author__ = "Jip-Hop" -__copyright__ = "Copyright © 2023, Jip-Hop and the Jailmakers" -__license__ = "LGPL-3.0-only" - -__disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK! -IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.""" diff --git a/src/jlmkr/__main__.py b/src/jlmkr/__main__.py index 0c3eb92..a73c3c2 100644 --- a/src/jlmkr/__main__.py +++ b/src/jlmkr/__main__.py @@ -3,11 +3,25 @@ # # SPDX-License-Identifier: LGPL-3.0-only -import donor +"""Create persistent Linux 'jails' on TrueNAS SCALE, \ +with full access to all files via bind mounts, \ +thanks to systemd-nspawn!""" + +__version__ = "3.0.0.dev1" +__author__ = "Jip-Hop" +__copyright__ = "Copyright © 2024, Jip-Hop and the Jailmakers" +__license__ = "LGPL-3.0-only" +__disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK! +IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.""" + + import sys +from cli import main + + if __name__ == "__main__": try: - sys.exit(donor.main()) + sys.exit(main()) except KeyboardInterrupt: sys.exit(130) diff --git a/src/jlmkr/donor/jlmkr.py b/src/jlmkr/cli.py old mode 100755 new mode 100644 similarity index 79% rename from src/jlmkr/donor/jlmkr.py rename to src/jlmkr/cli.py index 12110ee..9365259 --- a/src/jlmkr/donor/jlmkr.py +++ b/src/jlmkr/cli.py @@ -1,137 +1,30 @@ -#!/usr/bin/env python3 - -"""Create persistent Linux 'jails' on TrueNAS SCALE, \ -with full access to all files via bind mounts, \ -thanks to systemd-nspawn!""" - -__version__ = "3.0.0" -__author__ = "Jip-Hop" -__copyright__ = "Copyright (C) 2023, Jip-Hop" -__license__ = "LGPL-3.0-only" -__disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK! -IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.""" +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only import argparse -import contextlib -import hashlib -import json import os -import platform -import re -import shlex -import shutil -import stat -import subprocess import sys -import tempfile -import time -import urllib.request -from pathlib import Path, PurePath -from textwrap import dedent - - -# Use mostly default settings for systemd-nspawn but with systemd-run instead of a service file: -# https://github.com/systemd/systemd/blob/main/units/systemd-nspawn%40.service.in -# Use TasksMax=infinity since this is what docker does: -# https://github.com/docker/engine/blob/master/contrib/init/systemd/docker.service - -# Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown (but why?) -# Would give "directory tree currently busy" error and I'd have to run -# `rm /run/systemd/nspawn/locks/*` and remove the .lck file from jail_path -# Disabling locking isn't a big deal as systemd-nspawn will prevent starting a container -# with the same name anyway: as long as we're starting jails using this script, -# it won't be possible to start the same jail twice - -# Always add --bind-ro=/sys/module to make lsmod happy -# https://manpages.debian.org/bookworm/manpages/sysfs.5.en.html - -from utils.paths import SCRIPT_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH -from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME -from utils.paths import COMMAND_NAME, SHORTNAME -from utils.console import BOLD, RED, YELLOW, UNDERLINE, NORMAL - -DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" - -from utils.config_parser import ExceptionWithParser, KeyValueParser -from utils.config_parser import parse_config_file - - -# Workaround for exit_on_error=False not applying to: -# "error: the following arguments are required" -# https://github.com/python/cpython/issues/103498 -class CustomSubParser(argparse.ArgumentParser): - def error(self, message): - if self.exit_on_error: - super().error(message) - else: - raise ExceptionWithParser(self, message) - - -from utils.chroot import Chroot -from utils.console import eprint, fail -from utils.dataset import check_jail_name_valid, check_jail_name_available -from utils.dataset import get_jail_path, get_jail_config_path, get_jail_rootfs_path - -from actions.exec import exec_jail -from actions.status import status_jail -from actions.log import log_jail -from actions.shell import shell_jail -from actions.start import start_jail -from actions.restart import restart_jail -from actions.images import run_lxc_download_script -from utils.dataset import cleanup, check_jail_name_valid, check_jail_name_available -from utils.download import run_lxc_download_script +from __main__ import __version__, __disclaimer__ +from data import DISCLAIMER +from paths import SCRIPT_PATH, COMMAND_NAME, SCRIPT_NAME +from utils.editor import get_text_editor from utils.files import stat_chmod -from utils.dataset import get_zfs_dataset, create_zfs_dataset, remove_zfs_dataset from actions.create import create_jail -from utils.editor import get_text_editor -from utils.dataset import jail_is_running - from actions.edit import edit_jail -from actions.stop import stop_jail -from actions.remove import remove_jail - -from utils.dataset import get_all_jail_names, parse_os_release +from actions.exec import exec_jail +from actions.images import run_lxc_download_script from actions.list import list_jails +from actions.log import log_jail +from actions.remove import remove_jail +from actions.restart import restart_jail +from actions.shell import shell_jail +from actions.start import start_jail from actions.startup import startup_jails - - -def split_at_string(lst, string): - try: - index = lst.index(string) - return lst[:index], lst[index + 1 :] - except ValueError: - return lst, [] - - -def add_parser(subparser, **kwargs): - if kwargs.get("add_help") is False: - # Don't add help if explicitly disabled - add_help = False - else: - # Never add help with the built in add_help - kwargs["add_help"] = False - add_help = True - - kwargs["epilog"] = DISCLAIMER - kwargs["formatter_class"] = argparse.RawDescriptionHelpFormatter - kwargs["exit_on_error"] = False - func = kwargs.pop("func") - parser = subparser.add_parser(**kwargs) - parser.set_defaults(func=func) - - if add_help: - parser.add_argument( - "-h", "--help", help="show this help message and exit", action="store_true" - ) - - # Setting the add_help after the parser has been created with add_parser has no effect, - # but it allows us to look up if this parser has a help message available - parser.add_help = add_help - - return parser +from actions.status import status_jail +from actions.stop import stop_jail def main(): @@ -391,8 +284,48 @@ def main(): sys.exit(func(**args)) -if __name__ == "__main__": +# Workaround for exit_on_error=False not applying to: +# "error: the following arguments are required" +# https://github.com/python/cpython/issues/103498 +class CustomSubParser(argparse.ArgumentParser): + def error(self, message): + if self.exit_on_error: + super().error(message) + else: + raise ExceptionWithParser(self, message) + + +def add_parser(subparser, **kwargs): + if kwargs.get("add_help") is False: + # Don't add help if explicitly disabled + add_help = False + else: + # Never add help with the built in add_help + kwargs["add_help"] = False + add_help = True + + kwargs["epilog"] = DISCLAIMER + kwargs["formatter_class"] = argparse.RawDescriptionHelpFormatter + kwargs["exit_on_error"] = False + func = kwargs.pop("func") + parser = subparser.add_parser(**kwargs) + parser.set_defaults(func=func) + + if add_help: + parser.add_argument( + "-h", "--help", help="show this help message and exit", action="store_true" + ) + + # Setting the add_help after the parser has been created with add_parser has no effect, + # but it allows us to look up if this parser has a help message available + parser.add_help = add_help + + return parser + + +def split_at_string(lst, string): try: - main() - except KeyboardInterrupt: - sys.exit(130) + index = lst.index(string) + return lst[:index], lst[index + 1 :] + except ValueError: + return lst, [] diff --git a/src/jlmkr/donor/data.py b/src/jlmkr/data.py similarity index 94% rename from src/jlmkr/donor/data.py rename to src/jlmkr/data.py index f230f0d..2f68e83 100644 --- a/src/jlmkr/donor/data.py +++ b/src/jlmkr/data.py @@ -2,6 +2,12 @@ # # SPDX-License-Identifier: LGPL-3.0-only +from __main__ import __disclaimer__ +from utils.console import YELLOW, BOLD, NORMAL + + +DISCLAIMER = f"""{YELLOW}{BOLD}{__disclaimer__}{NORMAL}""" + DEFAULT_CONFIG = """startup=0 gpu_passthrough_intel=0 diff --git a/src/jlmkr/donor/__init__.py b/src/jlmkr/donor/__init__.py deleted file mode 100644 index de56c40..0000000 --- a/src/jlmkr/donor/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -from .jlmkr import main - -#### -# -# Transitional, as we split up jlmkr.py into smaller components -# -# -# -#### diff --git a/src/jlmkr/utils/paths.py b/src/jlmkr/paths.py similarity index 100% rename from src/jlmkr/utils/paths.py rename to src/jlmkr/paths.py diff --git a/src/jlmkr/utils/config_parser.py b/src/jlmkr/utils/config_parser.py index 720f79f..7e491a6 100644 --- a/src/jlmkr/utils/config_parser.py +++ b/src/jlmkr/utils/config_parser.py @@ -6,7 +6,7 @@ import io import re -from donor.data import DEFAULT_CONFIG +from data import DEFAULT_CONFIG # Used in parser getters to indicate the default behavior when a specific diff --git a/src/jlmkr/utils/dataset.py b/src/jlmkr/utils/dataset.py index d61bbdd..fa13ae7 100644 --- a/src/jlmkr/utils/dataset.py +++ b/src/jlmkr/utils/dataset.py @@ -10,9 +10,10 @@ from pathlib import PurePath from textwrap import dedent + +from paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME, SCRIPT_DIR_PATH from utils.chroot import Chroot from utils.console import eprint, YELLOW, BOLD, NORMAL -from utils.paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME, SCRIPT_DIR_PATH def get_jail_path(jail_name): diff --git a/src/jlmkr/utils/gpu.py b/src/jlmkr/utils/gpu.py index 8ff1151..c05b375 100644 --- a/src/jlmkr/utils/gpu.py +++ b/src/jlmkr/utils/gpu.py @@ -7,9 +7,10 @@ from pathlib import Path from textwrap import dedent + +from paths import SHORTNAME from utils.console import eprint from utils.dataset import get_jail_rootfs_path -from utils.paths import SHORTNAME # Test intel GPU by decoding mp4 file (output is discarded) From b52e5a4cba1b018ebb0549e7daf7644a07d1cae4 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 04:36:44 -0400 Subject: [PATCH 35/58] Nip and tuck; tidy and sort --- src/jlmkr/actions/create.py | 5 +++-- src/jlmkr/actions/edit.py | 2 +- src/jlmkr/actions/list.py | 3 ++- src/jlmkr/actions/log.py | 2 +- src/jlmkr/actions/remove.py | 2 +- src/jlmkr/actions/restart.py | 1 - src/jlmkr/actions/start.py | 7 ++++--- src/jlmkr/actions/startup.py | 2 +- src/jlmkr/actions/status.py | 2 +- 9 files changed, 14 insertions(+), 12 deletions(-) diff --git a/src/jlmkr/actions/create.py b/src/jlmkr/actions/create.py index 5109d2b..af155bc 100644 --- a/src/jlmkr/actions/create.py +++ b/src/jlmkr/actions/create.py @@ -9,7 +9,9 @@ from inspect import cleandoc from pathlib import Path, PurePath from textwrap import dedent -from donor.jlmkr import DISCLAIMER + +from cli import DISCLAIMER +from paths import COMMAND_NAME, JAILS_DIR_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH from utils.chroot import Chroot from utils.config_parser import KeyValueParser, DEFAULT_CONFIG from utils.console import YELLOW, BOLD, NORMAL, eprint @@ -18,7 +20,6 @@ from utils.dataset import check_jail_name_valid, check_jail_name_available from utils.dataset import get_jail_config_path, get_jail_rootfs_path from utils.dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset, cleanup -from utils.paths import COMMAND_NAME, JAILS_DIR_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH def create_jail(**kwargs): diff --git a/src/jlmkr/actions/edit.py b/src/jlmkr/actions/edit.py index aecd040..fe0ad07 100644 --- a/src/jlmkr/actions/edit.py +++ b/src/jlmkr/actions/edit.py @@ -6,9 +6,9 @@ import subprocess from utils.console import eprint -from utils.editor import get_text_editor from utils.dataset import check_jail_name_valid, check_jail_name_available from utils.dataset import get_jail_config_path, jail_is_running +from utils.editor import get_text_editor def edit_jail(jail_name): diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py index c444cc6..fd3423d 100644 --- a/src/jlmkr/actions/list.py +++ b/src/jlmkr/actions/list.py @@ -6,8 +6,9 @@ import subprocess from collections import defaultdict -from utils.console import NORMAL, UNDERLINE + from utils.config_parser import parse_config_file +from utils.console import NORMAL, UNDERLINE from utils.dataset import get_all_jail_names, get_jail_config_path, get_jail_rootfs_path, parse_os_release diff --git a/src/jlmkr/actions/log.py b/src/jlmkr/actions/log.py index b0e7b32..7f75e06 100644 --- a/src/jlmkr/actions/log.py +++ b/src/jlmkr/actions/log.py @@ -4,7 +4,7 @@ import subprocess -from utils.paths import SHORTNAME +from paths import SHORTNAME def log_jail(jail_name, args): diff --git a/src/jlmkr/actions/remove.py b/src/jlmkr/actions/remove.py index 28588f9..3a5e691 100644 --- a/src/jlmkr/actions/remove.py +++ b/src/jlmkr/actions/remove.py @@ -3,9 +3,9 @@ # SPDX-License-Identifier: LGPL-3.0-only from actions.stop import stop_jail +from utils.console import eprint from utils.dataset import check_jail_name_valid, check_jail_name_available from utils.dataset import get_jail_path, cleanup -from utils.console import eprint def remove_jail(jail_name): diff --git a/src/jlmkr/actions/restart.py b/src/jlmkr/actions/restart.py index 1740230..c50fbdf 100644 --- a/src/jlmkr/actions/restart.py +++ b/src/jlmkr/actions/restart.py @@ -2,7 +2,6 @@ # # SPDX-License-Identifier: LGPL-3.0-only - import subprocess import time diff --git a/src/jlmkr/actions/start.py b/src/jlmkr/actions/start.py index 790f5c2..9828ce5 100644 --- a/src/jlmkr/actions/start.py +++ b/src/jlmkr/actions/start.py @@ -9,13 +9,14 @@ from pathlib import Path from textwrap import dedent + +from paths import SHORTNAME, JAIL_ROOTFS_NAME from utils.config_parser import parse_config_file from utils.console import eprint +from utils.dataset import get_jail_config_path, get_jail_rootfs_path +from utils.dataset import get_jail_path, jail_is_running from utils.files import stat_chmod from utils.gpu import passthrough_intel, passthrough_nvidia -from utils.dataset import get_jail_path, jail_is_running -from utils.dataset import get_jail_config_path, get_jail_rootfs_path -from utils.paths import SHORTNAME, JAIL_ROOTFS_NAME def start_jail(jail_name): diff --git a/src/jlmkr/actions/startup.py b/src/jlmkr/actions/startup.py index 8314f8a..5df7b3d 100644 --- a/src/jlmkr/actions/startup.py +++ b/src/jlmkr/actions/startup.py @@ -3,8 +3,8 @@ # SPDX-License-Identifier: LGPL-3.0-only from actions.start import start_jail -from utils.dataset import get_all_jail_names, get_jail_config_path from utils.config_parser import parse_config_file +from utils.dataset import get_all_jail_names, get_jail_config_path def startup_jails(): diff --git a/src/jlmkr/actions/status.py b/src/jlmkr/actions/status.py index e3fa570..9fe54da 100644 --- a/src/jlmkr/actions/status.py +++ b/src/jlmkr/actions/status.py @@ -4,7 +4,7 @@ import subprocess -from utils.paths import SHORTNAME +from paths import SHORTNAME def status_jail(jail_name, args): From 49f751fb94a55a118172fe9d230c2248df7d9279 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 16:48:31 -0400 Subject: [PATCH 36/58] Restore CLI dependencies --- src/jlmkr/cli.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/jlmkr/cli.py b/src/jlmkr/cli.py index 9365259..ed677b9 100644 --- a/src/jlmkr/cli.py +++ b/src/jlmkr/cli.py @@ -6,9 +6,11 @@ import os import sys -from __main__ import __version__, __disclaimer__ +from __main__ import __version__ from data import DISCLAIMER from paths import SCRIPT_PATH, COMMAND_NAME, SCRIPT_NAME +from utils.config_parser import ExceptionWithParser +from utils.console import fail from utils.editor import get_text_editor from utils.files import stat_chmod From 9eddfce8fc87421ba8fc1cd690ca9454ce98e4e6 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 16:49:22 -0400 Subject: [PATCH 37/58] Fix CLI non-root developer mode --- src/jlmkr/cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/jlmkr/cli.py b/src/jlmkr/cli.py index ed677b9..ac92dcd 100644 --- a/src/jlmkr/cli.py +++ b/src/jlmkr/cli.py @@ -31,7 +31,7 @@ def main(): if os.stat(SCRIPT_PATH).st_uid != 0: - if os.environ.get('JLMKR_DEBUG') is not None: + if os.environ.get('JLMKR_DEBUG') is None: fail( f"This script should be owned by the root user... Fix it manually with: `chown root {SCRIPT_PATH}`." ) From 8769d305e812a05b8799ab84f55d92394154892e Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 16:51:15 -0400 Subject: [PATCH 38/58] Restore config parser dependencies --- src/jlmkr/utils/config_parser.py | 1 + 1 file changed, 1 insertion(+) diff --git a/src/jlmkr/utils/config_parser.py b/src/jlmkr/utils/config_parser.py index 7e491a6..669be7e 100644 --- a/src/jlmkr/utils/config_parser.py +++ b/src/jlmkr/utils/config_parser.py @@ -7,6 +7,7 @@ import re from data import DEFAULT_CONFIG +from utils.console import eprint # Used in parser getters to indicate the default behavior when a specific From 22cb05a8e4aca783d1d78fd19b6e088003a85d21 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 16:53:50 -0400 Subject: [PATCH 39/58] Restore dataset util dependency --- src/jlmkr/utils/dataset.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/jlmkr/utils/dataset.py b/src/jlmkr/utils/dataset.py index fa13ae7..1506c99 100644 --- a/src/jlmkr/utils/dataset.py +++ b/src/jlmkr/utils/dataset.py @@ -13,7 +13,7 @@ from paths import JAILS_DIR_PATH, JAIL_CONFIG_NAME, JAIL_ROOTFS_NAME, SCRIPT_DIR_PATH from utils.chroot import Chroot -from utils.console import eprint, YELLOW, BOLD, NORMAL +from utils.console import eprint, fail, YELLOW, BOLD, NORMAL def get_jail_path(jail_name): From c554673ad4138bee658c811c9f4b1ee19287ca49 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:05:19 -0400 Subject: [PATCH 40/58] Restore download util dependencies --- src/jlmkr/utils/download.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/jlmkr/utils/download.py b/src/jlmkr/utils/download.py index f1fbfe0..217333b 100644 --- a/src/jlmkr/utils/download.py +++ b/src/jlmkr/utils/download.py @@ -4,8 +4,11 @@ import hashlib import os +import re import subprocess +import urllib +from utils.console import eprint from utils.files import stat_chmod From 80f6c0e7d2119ac980067f05360a8ed1d37ebb7d Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:20:39 -0400 Subject: [PATCH 41/58] Remove unused dependency from zip builder --- src/builder/zip/build_zip.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/builder/zip/build_zip.py b/src/builder/zip/build_zip.py index 6c982a5..53caf1f 100644 --- a/src/builder/zip/build_zip.py +++ b/src/builder/zip/build_zip.py @@ -7,7 +7,6 @@ import os from hatchling.builders.config import BuilderConfig from hatchling.builders.plugin.interface import BuilderInterface -from hatchling.builders.plugin.interface import IncludedFile from hatchling.builders.utils import normalize_relative_path from pathlib import Path from typing import Any, Callable, Iterable From 7e071219145c4d986fb7d0fbfe514455fe6f160f Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:21:09 -0400 Subject: [PATCH 42/58] Realign create action dependencies --- src/jlmkr/actions/create.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/jlmkr/actions/create.py b/src/jlmkr/actions/create.py index af155bc..507f9be 100644 --- a/src/jlmkr/actions/create.py +++ b/src/jlmkr/actions/create.py @@ -2,14 +2,13 @@ # # SPDX-License-Identifier: LGPL-3.0-only -import contextlib -import os -import re +import sys from inspect import cleandoc from pathlib import Path, PurePath from textwrap import dedent +from actions.start import start_jail from cli import DISCLAIMER from paths import COMMAND_NAME, JAILS_DIR_PATH, SCRIPT_NAME, SCRIPT_DIR_PATH from utils.chroot import Chroot @@ -19,7 +18,8 @@ from utils.files import stat_chmod, get_mount_point from utils.dataset import check_jail_name_valid, check_jail_name_available from utils.dataset import get_jail_config_path, get_jail_rootfs_path -from utils.dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset, cleanup +from utils.dataset import get_jail_path, get_zfs_dataset, create_zfs_dataset +from utils.dataset import parse_os_release, cleanup def create_jail(**kwargs): From a1aaea8555603e43e5a6eb8743b061b4cc045a8a Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:22:09 -0400 Subject: [PATCH 43/58] Drop unused edit action dependency --- src/jlmkr/actions/edit.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/jlmkr/actions/edit.py b/src/jlmkr/actions/edit.py index fe0ad07..b948e0e 100644 --- a/src/jlmkr/actions/edit.py +++ b/src/jlmkr/actions/edit.py @@ -2,7 +2,6 @@ # # SPDX-License-Identifier: LGPL-3.0-only -import os import subprocess from utils.console import eprint From 6d64562c6da04937696e15bd7a7acdeb7d0339c5 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:23:14 -0400 Subject: [PATCH 44/58] Restore list action dependency --- src/jlmkr/actions/list.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/jlmkr/actions/list.py b/src/jlmkr/actions/list.py index fd3423d..216d543 100644 --- a/src/jlmkr/actions/list.py +++ b/src/jlmkr/actions/list.py @@ -8,7 +8,7 @@ from collections import defaultdict from utils.config_parser import parse_config_file -from utils.console import NORMAL, UNDERLINE +from utils.console import eprint, NORMAL, UNDERLINE from utils.dataset import get_all_jail_names, get_jail_config_path, get_jail_rootfs_path, parse_os_release From 8dae934e4b6a14cbcffbd223d9a0fab2b16c9ecd Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:24:06 -0400 Subject: [PATCH 45/58] Drop unused restart action dependencies --- src/jlmkr/actions/restart.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/jlmkr/actions/restart.py b/src/jlmkr/actions/restart.py index c50fbdf..810099f 100644 --- a/src/jlmkr/actions/restart.py +++ b/src/jlmkr/actions/restart.py @@ -2,9 +2,6 @@ # # SPDX-License-Identifier: LGPL-3.0-only -import subprocess -import time - from actions.start import start_jail from actions.stop import stop_jail from utils.console import eprint From c41ac9cd83498bb66509450786d619e8ff99f381 Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 17:25:55 -0400 Subject: [PATCH 46/58] Restore start action dependencies --- src/jlmkr/actions/start.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/jlmkr/actions/start.py b/src/jlmkr/actions/start.py index 9828ce5..c3974d5 100644 --- a/src/jlmkr/actions/start.py +++ b/src/jlmkr/actions/start.py @@ -10,9 +10,10 @@ from pathlib import Path from textwrap import dedent -from paths import SHORTNAME, JAIL_ROOTFS_NAME +from actions.exec import exec_jail +from paths import COMMAND_NAME, SHORTNAME, JAIL_ROOTFS_NAME from utils.config_parser import parse_config_file -from utils.console import eprint +from utils.console import eprint, RED, BOLD, NORMAL from utils.dataset import get_jail_config_path, get_jail_rootfs_path from utils.dataset import get_jail_path, jail_is_running from utils.files import stat_chmod From a3854531da54f668520c3500e529bb65cde821ba Mon Sep 17 00:00:00 2001 From: jonct <2807816+jonct@users.noreply.github.com> Date: Mon, 15 Jul 2024 19:48:29 -0400 Subject: [PATCH 47/58] Improve the developer bootstrap docs --- README.md | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 4d626fd..5ba7c2f 100644 --- a/README.md +++ b/README.md @@ -214,20 +214,17 @@ The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image ser ## Development -After cloning the project, navigate into its working directory and create a self-contained Python [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). - - python3 -m venv --no-setuptools .venv - -> *Note: Due to [NAS-130029](https://ixsystems.atlassian.net/browse/NAS-130029), the user's safe working Python environment can only be bootstrapped from TrueNAS SCALE's [developer mode](https://www.truenas.com/docs/scale/scaletutorials/systemsettings/advanced/developermode/), or in a jail or elsewhere. If you like to take long walks with strangers, the following might suffice as an alternative to the above one-liner.* -> -> ``` -> curl -L https://github.com/brettcannon/microvenv/archive/refs/tags/v2023.5.tar.gz | tar xz --strip-components=1 microvenv-2023.5/microvenv -> python3 -m microvenv -> curl -OL https://bootstrap.pypa.io/pip/pip.pyz -> .venv/bin/python3 pip.pyz install virtualenv -> .venv/bin/virtualenv --no-setuptools .venv -> rm -rf microvenv pip.pyz -> ``` +After cloning the project, navigate into its working directory and create a self-contained Python [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). Traditionally you would do that using the following command. Unfortunately, TrueNAS SCALE has [left out](https://ixsystems.atlassian.net/browse/NAS-130029) a necessary library from the base installation and *this command will fail:* + + python3 -m venv .venv + +The following workaround should accomplish the same task in more steps. +``` +python3 -m venv --without-pip .venv +curl -OL https://bootstrap.pypa.io/pip/pip.pyz +.venv/bin/python3 pip.pyz install pip +rm pip.pyz +``` *Note: This process and the resulting build environment will cache some items under `~/.local/share` in addition to the project directory.* Activate the venv into your *current* shell session. @@ -242,7 +239,7 @@ For more information on Python standard venvs, go to [the source](https://packag ### Hatching a build -While in an *active* session, install the [Hatch](https://hatch.pypa.io) project manager. This will load quite a flurry of dependencies, but will only do so into the new `.venv` directory. +While in an *active* session, install the [Hatch](https://hatch.pypa.io) project manager. This will load quite a flurry of dependencies, but will only do so into the new `.venv` directory. *(And a bit into self-managed directories under `~/.local/share`.)* pip install hatch From d4bc1c80d8d7817a7fad0f5b29050578b5a38b7d Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 12:38:50 +0200 Subject: [PATCH 48/58] Add missing imports --- src/jlmkr/actions/create.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/jlmkr/actions/create.py b/src/jlmkr/actions/create.py index 507f9be..ef8f2e6 100644 --- a/src/jlmkr/actions/create.py +++ b/src/jlmkr/actions/create.py @@ -2,6 +2,8 @@ # # SPDX-License-Identifier: LGPL-3.0-only +import contextlib +import os import sys from inspect import cleandoc From 2266d897550cc54b07824d3a1eac91742df9b038 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 12:46:47 +0200 Subject: [PATCH 49/58] Restore jlmkr.py to make tests happy --- jlmkr.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100755 jlmkr.py diff --git a/jlmkr.py b/jlmkr.py new file mode 100755 index 0000000..3554b0d --- /dev/null +++ b/jlmkr.py @@ -0,0 +1,25 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers +# +# SPDX-License-Identifier: LGPL-3.0-only + +if __name__ == "__main__": + import subprocess + import sys + import os + + # Get the path of the currently running script + current_path = os.path.realpath(__file__) + + # Define the relative path you want to resolve + relative_path = "src/jlmkr" + + # Resolve the relative path + script_path = os.path.join(os.path.dirname(current_path), relative_path) + + # Get the arguments passed to the current script + args = sys.argv[1:] + + # Pass all arguments to the other script using subprocess + subprocess.run(["python3", script_path] + args, check=True) + From 1679d27280dd18bfc049191f918ae3e7606a25c2 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 13:45:18 +0200 Subject: [PATCH 50/58] Chown files as root --- .github/workflows/test.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index e32fff2..9418aec 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -87,6 +87,6 @@ jobs: env: PYTHONUNBUFFERED: 1 run: | - sudo chown 0:0 jlmkr.py ./test/test-jlmkr + sudo chown -R 0:0 src jlmkr.py ./test/test-jlmkr sudo bash ./test/test-jlmkr sudo ./test/test.sh From 9bdcfedd1a1f52e041dad3c29a44bea25c790347 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 13:57:58 +0200 Subject: [PATCH 51/58] Fix import --- src/jlmkr/utils/download.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/jlmkr/utils/download.py b/src/jlmkr/utils/download.py index 217333b..3cb608a 100644 --- a/src/jlmkr/utils/download.py +++ b/src/jlmkr/utils/download.py @@ -6,7 +6,7 @@ import os import re import subprocess -import urllib +import urllib.request from utils.console import eprint from utils.files import stat_chmod From 2256d31b94f05870f0a9d86e65530bb9deab1b0b Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 16:12:12 +0200 Subject: [PATCH 52/58] Run jlmkr as zipapp --- .github/workflows/test.yml | 3 ++- .gitignore | 1 + jlmkr.py | 25 ------------------------- test/test-jlmkr | 6 +++--- test/test.sh | 4 ++-- 5 files changed, 8 insertions(+), 31 deletions(-) delete mode 100755 jlmkr.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9418aec..bc605d7 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -87,6 +87,7 @@ jobs: env: PYTHONUNBUFFERED: 1 run: | - sudo chown -R 0:0 src jlmkr.py ./test/test-jlmkr + python3 -m zipapp src/jlmkr -p "/usr/bin/env python3" -o jlmkr + sudo chown 0:0 jlmkr ./test/test-jlmkr sudo bash ./test/test-jlmkr sudo ./test/test.sh diff --git a/.gitignore b/.gitignore index 6c6dbf4..0e3c632 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ dist/ # jail-specific /.lxc/ /jails/ +jlmkr # Mac-specific .DS_Store diff --git a/jlmkr.py b/jlmkr.py deleted file mode 100755 index 3554b0d..0000000 --- a/jlmkr.py +++ /dev/null @@ -1,25 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -if __name__ == "__main__": - import subprocess - import sys - import os - - # Get the path of the currently running script - current_path = os.path.realpath(__file__) - - # Define the relative path you want to resolve - relative_path = "src/jlmkr" - - # Resolve the relative path - script_path = os.path.join(os.path.dirname(current_path), relative_path) - - # Get the arguments passed to the current script - args = sys.argv[1:] - - # Pass all arguments to the other script using subprocess - subprocess.run(["python3", script_path] + args, check=True) - diff --git a/test/test-jlmkr b/test/test-jlmkr index bc87295..f4c4bb8 100755 --- a/test/test-jlmkr +++ b/test/test-jlmkr @@ -13,9 +13,9 @@ if [[ -z "$JLMKR_PATH" && -n "$SCALE_POOL_ROOT" ]]; then elif [[ -z "$JLMKR_PATH" ]]; then JLMKR_PATH=${PWD:-.} fi -if [[ ! -r "$JLMKR_PATH/jlmkr.py" ]]; then +if [[ ! -r "$JLMKR_PATH/jlmkr" ]]; then >&2 printf "%s\n" \ - "couldn't find jlmkr.py. Are you running from the jailmaker directory?" \ + "couldn't find jlmkr. Are you running from the jailmaker directory?" \ "If not, setup either JLMKR_PATH or SCALE_POOL_ROOT" \ "" @@ -57,7 +57,7 @@ WAIT_FOR_JAIL=${WAIT_FOR_JAIL:-4s} #### Functions jlmkr () { - /usr/bin/env python3 "$JLMKR_PATH/jlmkr.py" "${@:---help}" + "$JLMKR_PATH/jlmkr" "${@:---help}" } iterate () { diff --git a/test/test.sh b/test/test.sh index 04a2aef..c9b3ec9 100755 --- a/test/test.sh +++ b/test/test.sh @@ -8,5 +8,5 @@ set -euo pipefail # TODO: test jlmkr.py from inside another working directory, with a relative path to a config file to test if it uses the config file (and doesn't look for it relative to the jlmkr.py file itself) -./jlmkr.py create --start --config=./templates/docker/config test -./jlmkr.py exec test docker run hello-world +./jlmkr create --start --config=./templates/docker/config test +./jlmkr exec test docker run hello-world From 5e78f3e13e64387d073cb5d67071a1950e3131cb Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 16:17:40 +0200 Subject: [PATCH 53/58] Fail early --- .github/workflows/test.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index bc605d7..70097b1 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -30,9 +30,10 @@ jobs: # Create a network namespace in the GitHub-hosted runner VM, # simulating a primary bridge network on TrueNAS SCALE - - name: Set up networking resources + - name: Install packages and setup networking run: | sudo -s < Date: Tue, 16 Jul 2024 16:18:31 +0200 Subject: [PATCH 54/58] Add apt-get update --- .github/workflows/test.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 70097b1..86f6cbf 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -39,6 +39,7 @@ jobs: rm /etc/resolv.conf echo 'nameserver 1.1.1.1' > /etc/resolv.conf + apt-get update apt-get install -qq -y systemd-container cat </etc/systemd/network/10-br1.network From 9d108985aaa58668533691842ac8583d22aa033e Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 16:46:15 +0200 Subject: [PATCH 55/58] Use python 3.11 --- .github/workflows/test.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 86f6cbf..ecdfa91 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -24,7 +24,9 @@ jobs: steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v4 - + - uses: actions/setup-python@v5 + with: + python-version: '3.11' # TrueNAS SCALE 24.04 Dragonfish - name: Tune GitHub-hosted runner network uses: smorimoto/tune-github-hosted-runner-network@v1 From 22ea9842d41917fbf21f81767ea69fee80868667 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 16:50:45 +0200 Subject: [PATCH 56/58] Upload build artifact --- .github/workflows/test.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ecdfa91..2307c0c 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -86,13 +86,20 @@ jobs: # zpool --version # END + - name: Build + run: | + python3 -m zipapp src/jlmkr -p "/usr/bin/env python3" -o jlmkr + + - uses: actions/upload-artifact@v4 + with: + path: jlmkr + # Run multiple commands using the runners shell - name: Run the test script env: PYTHONUNBUFFERED: 1 run: | set -euo pipefail - python3 -m zipapp src/jlmkr -p "/usr/bin/env python3" -o jlmkr sudo chown 0:0 jlmkr ./test/test-jlmkr sudo bash ./test/test-jlmkr sudo ./test/test.sh From 962b238f6afde16ce552b9c631ca81e2c9ded7f6 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 17:28:16 +0200 Subject: [PATCH 57/58] Remove Hatch --- .github/workflows/build.yml | 59 ---------------------------------- src/builder/app/__init__.py | 3 -- src/builder/app/build_app.py | 56 -------------------------------- src/builder/app/hooks_app.py | 11 ------- src/builder/app/pyproject.toml | 22 ------------- src/builder/zip/__init__.py | 3 -- src/builder/zip/build_zip.py | 59 ---------------------------------- src/builder/zip/hooks_zip.py | 11 ------- src/builder/zip/pyproject.toml | 22 ------------- 9 files changed, 246 deletions(-) delete mode 100644 .github/workflows/build.yml delete mode 100644 src/builder/app/__init__.py delete mode 100644 src/builder/app/build_app.py delete mode 100644 src/builder/app/hooks_app.py delete mode 100644 src/builder/app/pyproject.toml delete mode 100644 src/builder/zip/__init__.py delete mode 100644 src/builder/zip/build_zip.py delete mode 100644 src/builder/zip/hooks_zip.py delete mode 100644 src/builder/zip/pyproject.toml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml deleted file mode 100644 index 118e1cf..0000000 --- a/.github/workflows/build.yml +++ /dev/null @@ -1,59 +0,0 @@ -# Inspiration from - -name: Build - -on: - push: - branches: ["**"] - pull_request: - branches: ["**"] - - workflow_dispatch: - -jobs: - - build: - name: Build jlmkr tool - runs-on: ubuntu-24.04 - strategy: - matrix: - python-version: - - "3.11" # TrueNAS SCALE 24.04 Dragonfish - steps: - - - name: Checkout code - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - - - name: Set up pip cache - uses: actions/cache@v3 - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }} - restore-keys: ${{ runner.os }}-pip- - - - name: Install Hatch - uses: pypa/hatch@install - - - name: Run style check - run: hatch fmt --check - -# - name: Run unit tests -# run: hatch run +py=${{ matrix.python-version }} test:test - - - name: Build distribution - run: hatch build -t zipapp -t appzip - -# - name: Upload artifacts -# uses: actions/upload-artifact@v4 -# with: -# path: -# - dist/jlmkr -# - dist/jlmkr-*.zip -# if-no-files-found: error diff --git a/src/builder/app/__init__.py b/src/builder/app/__init__.py deleted file mode 100644 index 4662c4f..0000000 --- a/src/builder/app/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only diff --git a/src/builder/app/build_app.py b/src/builder/app/build_app.py deleted file mode 100644 index 22675ed..0000000 --- a/src/builder/app/build_app.py +++ /dev/null @@ -1,56 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -import os -from io import BytesIO -from pathlib import Path -from typing import Any, Callable, Iterable -from zipapp import create_archive - -from hatchling.builders.plugin.interface import BuilderInterface - - -class ZipAppBuilder(BuilderInterface): - PLUGIN_NAME = "zipapp" - - def get_version_api(self) -> dict[str, Callable[..., str]]: - return {"standard": self.build_standard} - - def clean(self, directory: str, versions: Iterable[str]) -> None: - try: - os.remove(Path(directory, 'jlmkr')) - except: - pass - - def build_standard(self, directory: str, **build_data: Any) -> str: - - # generate zipapp source archive - pyzbuffer = BytesIO() - create_archive('src/jlmkr', target=pyzbuffer, - interpreter='=PLACEHOLDER=', -# main='donor.jlmkr:main', - compressed=True) - zipdata = pyzbuffer.getvalue() #.removeprefix(b"#!=PLACEHOLDER=\n") - - # output with preamble - outpath = Path(directory, 'jlmkr') - with open(outpath, 'wb') as f: - f.write(preamble(self.metadata.version).encode()) - f.write(zipdata) - os.chmod(outpath, 0o755) - return os.fspath(outpath) - - -# 10 lines will conveniently match the default of head(1) -def preamble(version): return f'''#!/usr/bin/env python3 - -jlmkr {version} - -Persistent Linux 'jails' on TrueNAS SCALE to install software (k3s, docker, portainer, podman, etc.) with full access to all files via bind mounts. - -SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -SPDX-License-Identifier: LGPL-3.0-only - --=-=-=- this is a zip file -=-=-=- what follows is binary -=-=-=- -''' diff --git a/src/builder/app/hooks_app.py b/src/builder/app/hooks_app.py deleted file mode 100644 index beb4253..0000000 --- a/src/builder/app/hooks_app.py +++ /dev/null @@ -1,11 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -from hatchling.plugin import hookimpl - -from build_app import ZipAppBuilder - -@hookimpl -def hatch_register_builder(): - return ZipAppBuilder diff --git a/src/builder/app/pyproject.toml b/src/builder/app/pyproject.toml deleted file mode 100644 index 8af91bc..0000000 --- a/src/builder/app/pyproject.toml +++ /dev/null @@ -1,22 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -[build-system] -build-backend = "hatchling.build" -requires = ["hatchling"] - -[project] -name = "hatch-zipapp" -version = "0.0.dev0" -classifiers = [ - "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", - "Private :: Do Not Upload", -] -dependencies = ["hatchling"] - -[project.entry-points.hatch] -zipapp = "hooks_app" - -[tool.hatch.build.targets.wheel] -packages = ["."] diff --git a/src/builder/zip/__init__.py b/src/builder/zip/__init__.py deleted file mode 100644 index 4662c4f..0000000 --- a/src/builder/zip/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only diff --git a/src/builder/zip/build_zip.py b/src/builder/zip/build_zip.py deleted file mode 100644 index 53caf1f..0000000 --- a/src/builder/zip/build_zip.py +++ /dev/null @@ -1,59 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -# hat tip: - -import os -from hatchling.builders.config import BuilderConfig -from hatchling.builders.plugin.interface import BuilderInterface -from hatchling.builders.utils import normalize_relative_path -from pathlib import Path -from typing import Any, Callable, Iterable -from zipfile import ZipFile, ZIP_DEFLATED - - -class AppZipBuilderConfig(BuilderConfig): pass - -class AppZipBuilder(BuilderInterface): - PLUGIN_NAME = "appzip" - - @classmethod - def get_config_class(cls): - return AppZipBuilderConfig - - def get_version_api(self) -> dict[str, Callable[..., str]]: - return {'standard': self.build_standard} - - def clean(self, directory: str, versions: Iterable[str]) -> None: - for filename in os.listdir(directory): - if filename.startswith('jlmkr-') and filename.endswith('.zip'): - os.remove(Path(directory, filename)) - - def build_standard(self, directory: str, **build_data: Any) -> str: - outpath = Path(directory, f'jlmkr-{self.metadata.version}.zip') - with ZipFile(outpath, 'w') as zip: - zip.write(Path(directory, 'jlmkr'), 'jlmkr') - force_map = build_data['force_include'] - for included_file in self.recurse_forced_files(force_map): - zip.write( - included_file.relative_path, - included_file.distribution_path, - ZIP_DEFLATED) - return os.fspath(outpath) - - def get_default_build_data(self) -> dict[str, Any]: - build_data: dict[str, Any] = super().get_default_build_data() - - extra_files = [] - if self.metadata.core.readme_path: - extra_files.append(self.metadata.core.readme_path) - if self.metadata.core.license_files: - extra_files.extend(self.metadata.core.license_files) - - force_include = build_data.setdefault("force_include", {}) - for fn in map(normalize_relative_path, extra_files): - force_include[os.path.join(self.root, fn)] = Path(fn).name - build_data['force_include'] = force_include - - return build_data diff --git a/src/builder/zip/hooks_zip.py b/src/builder/zip/hooks_zip.py deleted file mode 100644 index 5bd8a60..0000000 --- a/src/builder/zip/hooks_zip.py +++ /dev/null @@ -1,11 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -from hatchling.plugin import hookimpl - -from build_zip import AppZipBuilder - -@hookimpl -def hatch_register_builder(): - return AppZipBuilder diff --git a/src/builder/zip/pyproject.toml b/src/builder/zip/pyproject.toml deleted file mode 100644 index 1aa6674..0000000 --- a/src/builder/zip/pyproject.toml +++ /dev/null @@ -1,22 +0,0 @@ -# SPDX-FileCopyrightText: © 2024 Jip-Hop and the Jailmakers -# -# SPDX-License-Identifier: LGPL-3.0-only - -[build-system] -build-backend = "hatchling.build" -requires = ["hatchling"] - -[project] -name = "hatch-appzip" -version = "0.0.dev0" -classifiers = [ - "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", - "Private :: Do Not Upload", -] -dependencies = ["hatchling"] - -[project.entry-points.hatch] -appzip = "hooks_zip" - -[tool.hatch.build.targets.wheel] -packages = ["."] From 0b967234d4e103896b29bd05bc67ec506b56a225 Mon Sep 17 00:00:00 2001 From: Jip-Hop <2871973+Jip-Hop@users.noreply.github.com> Date: Tue, 16 Jul 2024 17:28:19 +0200 Subject: [PATCH 58/58] Update README.md --- README.md | 124 +++++++++++++++++++----------------------------------- 1 file changed, 44 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index 5ba7c2f..a694511 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Persistent Linux 'jails' on TrueNAS SCALE to install software (k3s, docker, port ## Summary -TrueNAS SCALE can create persistent Linux 'jails' with systemd-nspawn. This script helps with the following: +TrueNAS SCALE can create persistent Linux 'jails' with systemd-nspawn. This app helps with the following: - Setting up the jail so it won't be lost when you update SCALE - Choosing a distro (Debian 12 strongly recommended, but Ubuntu, Arch Linux or Rocky Linux seem good choices too) @@ -23,28 +23,37 @@ TrueNAS SCALE can create persistent Linux 'jails' with systemd-nspawn. This scri ## Installation -Beginning with 24.04 (Dragonfish), TrueNAS SCALE officially includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the `jlmkr.py` script file in the right place. [Instructions with screenshots](https://www.truenas.com/docs/scale/scaletutorials/apps/sandboxes/) are provided on the TrueNAS website. Start by creating a new dataset called `jailmaker` with the default settings (from TrueNAS web interface). Then login as the root user and download `jlmkr.py`. +Beginning with 24.04 (Dragonfish), TrueNAS SCALE officially includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the `jlmkr` app in the right place. [Instructions with screenshots](https://www.truenas.com/docs/scale/scaletutorials/apps/sandboxes/) are provided on the TrueNAS website. Start by creating a new dataset called `jailmaker` with the default settings (from TrueNAS web interface). Then login as the root user and download `jlmkr`. + +TODO: update install instructions. For now one may clone or download the repo and run the below commands to create the `jlmkr` zipapp. ```shell -cd /mnt/mypool/jailmaker -curl --location --remote-name https://raw.githubusercontent.com/Jip-Hop/jailmaker/main/jlmkr.py -chmod +x jlmkr.py +rm -rf /tmp/jlmkr-build +mkdir -p /tmp/jlmkr-build +cd /tmp/jlmkr-build +curl -L https://github.com/Jip-Hop/jailmaker/archive/refs/heads/v3.0.0.tar.gz | tar xvz --strip-components=1 +python3 -m zipapp src/jlmkr -p "/usr/bin/env python3" -o jlmkr +cp jlmkr /mnt/mypool/jailmaker/ ``` -The `jlmkr.py` script (and the jails + config it creates) are now stored on the `jailmaker` dataset and will survive updates of TrueNAS SCALE. If the automatically created `jails` directory is also a ZFS dataset (which is true for new users), then the `jlmkr.py` script will automatically create a new dataset for every jail created. This allows you to snapshot individual jails. For legacy users (where the `jails` directory is not a dataset) each jail will be stored in a plain directory. +Alternatively one may download and extract `jlmkr` from the build artifacts of the [GitHub Actions](https://github.com/Jip-Hop/jailmaker/actions). + +The `jlmkr` app (and the jails + config it creates) are now stored on the `jailmaker` dataset and will survive updates of TrueNAS SCALE. If the automatically created `jails` directory is also a ZFS dataset (which is true for new users), then the `jlmkr` app will automatically create a new dataset for every jail created. This allows you to snapshot individual jails. For legacy users (where the `jails` directory is not a dataset) each jail will be stored in a plain directory. ### Alias -Optionally you may create a shell alias for the currently logged in (admin) user to conveniently run `jlmkr.py` without having to change into the `jailmaker` directory or specify the full absolute path. I suggest to create the `jlmkr` alias like this: +TODO: explain how to run `jlmkr` without using the absolute path. This probably involves building and releasing the `zipapp` on GitHub, downloading it into a directory added to the `PATH`. But this also requires the `jailmaker` directory to be configurable (instead of using the directory the `jlmkr` app itself is in) by using the `JAILMAKER_DIR` env variable. -```shell -echo "alias jlmkr=\"sudo -E '/mnt/mypool/jailmaker/jlmkr.py'\"" >> ~/.bashrc +```bash +mkdir /root/bin +cd /root/bin +curl -o jlmkr --location --remote-name https://some_url +chmod +x jlmkr +cd ../ +echo 'export PATH="/root/bin:$PATH"' | tee -a .bashrc .zshrc +echo 'export JAILMAKER_DIR=/mnt/tank/path/to/desired/jailmaker/dir' | tee -a .bashrc .zshrc ``` -Please replace `/mnt/mypool/jailmaker/` with the actual path to where you stored `jlmkr.py`. If you're using zsh instead of bash, then you should replace `.bashrc` in the command above with `.zshrc`. If you've created the alias, you may use it instead of `./jlmkr.py`. - -The alias will be available the next time you load the shell, but to use the alias immediately you can `source ~/.bashrc` or `source ~/.zshrc`, as appropriate. - ## Usage ### Create Jail @@ -52,25 +61,25 @@ The alias will be available the next time you load the shell, but to use the ali Creating a jail with the default settings is as simple as: ```shell -./jlmkr.py create --start myjail +./jlmkr create --start myjail ``` You may also specify a path to a config template, for a quick and consistent jail creation process. ```shell -./jlmkr.py create --start --config /path/to/config/template myjail +./jlmkr create --start --config /path/to/config/template myjail ``` -Or you can override the default config by using flags. See `./jlmkr.py create --help` for the available options. Anything passed after the jail name will be passed to `systemd-nspawn` when starting the jail. See the `systemd-nspawn` manual for available options, specifically [Mount Options](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Mount_Options) and [Networking Options](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Networking_Options) are frequently used. +Or you can override the default config by using flags. See `./jlmkr create --help` for the available options. Anything passed after the jail name will be passed to `systemd-nspawn` when starting the jail. See the `systemd-nspawn` manual for available options, specifically [Mount Options](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Mount_Options) and [Networking Options](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Networking_Options) are frequently used. ```shell -./jlmkr.py create --start --distro=ubuntu --release=jammy myjail --bind-ro=/mnt +./jlmkr create --start --distro=ubuntu --release=jammy myjail --bind-ro=/mnt ``` If you omit the jail name, the create process is interactive. You'll be presented with questions which guide you through the process. ```shell -./jlmkr.py create +./jlmkr create ``` After answering some questions you should have created your first jail (and it should be running if you chose to start it after creating)! @@ -78,16 +87,16 @@ After answering some questions you should have created your first jail (and it s ### Startup Jails on Boot ```shell -# Call startup using the absolute path to jlmkr.py -/mnt/mypool/jailmaker/jlmkr.py startup +# Call startup using the absolute path to jlmkr +/mnt/mypool/jailmaker/jlmkr startup ``` -In order to start jails automatically after TrueNAS boots, run `/mnt/mypool/jailmaker/jlmkr.py startup` as Post Init Script with Type `Command` from the TrueNAS web interface. This will start all the jails with `startup=1` in the config file. +In order to start jails automatically after TrueNAS boots, run `/mnt/mypool/jailmaker/jlmkr startup` as Post Init Script with Type `Command` from the TrueNAS web interface. This will start all the jails with `startup=1` in the config file. ### Start Jail ```shell -./jlmkr.py start myjail +./jlmkr start myjail ``` ### List Jails @@ -95,7 +104,7 @@ In order to start jails automatically after TrueNAS boots, run `/mnt/mypool/jail See list of jails (including running, startup state, GPU passthrough, distro, and IP). ```shell -./jlmkr.py list +./jlmkr list ``` ### Execute Command in Jail @@ -103,41 +112,41 @@ See list of jails (including running, startup state, GPU passthrough, distro, an You may want to execute a command inside a jail, for example manually from the TrueNAS shell, a shell script or a CRON job. The example below executes the `env` command inside the jail. ```shell -./jlmkr.py exec myjail env +./jlmkr exec myjail env ``` This example executes bash inside the jail with a command as additional argument. ```shell -./jlmkr.py exec myjail bash -c 'echo test; echo $RANDOM;' +./jlmkr exec myjail bash -c 'echo test; echo $RANDOM;' ``` ### Edit Jail Config ```shell -./jlmkr.py edit myjail +./jlmkr edit myjail ``` -Once you've created a jail, it will exist in a directory inside the `jails` dir next to `jlmkr.py`. For example `/mnt/mypool/jailmaker/jails/myjail` if you've named your jail `myjail`. You may edit the jail configuration file using the `./jlmkr.py edit myjail` command. This opens the config file in your favorite editor, as determined by following [Debian's guidelines](https://www.debian.org/doc/debian-policy/ch-customized-programs.html#editors-and-pagers) on the matter. You'll have to stop the jail and start it again with `jlmkr` for these changes to take effect. +Once you've created a jail, it will exist in a directory inside the `jails` dir next to `jlmkr`. For example `/mnt/mypool/jailmaker/jails/myjail` if you've named your jail `myjail`. You may edit the jail configuration file using the `./jlmkr edit myjail` command. This opens the config file in your favorite editor, as determined by following [Debian's guidelines](https://www.debian.org/doc/debian-policy/ch-customized-programs.html#editors-and-pagers) on the matter. You'll have to stop the jail and start it again with `jlmkr` for these changes to take effect. ### Remove Jail Delete a jail and remove it's files (requires confirmation). ```shell -./jlmkr.py remove myjail +./jlmkr remove myjail ``` ### Stop Jail ```shell -./jlmkr.py stop myjail +./jlmkr stop myjail ``` ### Restart Jail ```shell -./jlmkr.py restart myjail +./jlmkr restart myjail ``` ### Jail Shell @@ -145,13 +154,13 @@ Delete a jail and remove it's files (requires confirmation). Switch into the jail's shell. ```shell -./jlmkr.py shell myjail +./jlmkr shell myjail ``` ### Jail Status ```shell -./jlmkr.py status myjail +./jlmkr status myjail ``` ### Jail Logs @@ -159,12 +168,12 @@ Switch into the jail's shell. View a jail's logs. ```shell -./jlmkr.py log myjail +./jlmkr log myjail ``` ### Additional Commands -Expert users may use the following additional commands to manage jails directly: `machinectl`, `systemd-nspawn`, `systemd-run`, `systemctl` and `journalctl`. The `jlmkr` script uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by `jlmkr` and not everything will work in the context of TrueNAS SCALE. +Expert users may use the following additional commands to manage jails directly: `machinectl`, `systemd-nspawn`, `systemd-run`, `systemctl` and `journalctl`. The `jlmkr` app uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by `jlmkr` and not everything will work in the context of TrueNAS SCALE. ## Security @@ -210,52 +219,7 @@ TODO: write comparison between systemd-nspawn (without `jailmaker`), LXC, VMs, D ## Incompatible Distros -The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image server](https://images.linuxcontainers.org). These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the `alpine` image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine. - -## Development - -After cloning the project, navigate into its working directory and create a self-contained Python [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). Traditionally you would do that using the following command. Unfortunately, TrueNAS SCALE has [left out](https://ixsystems.atlassian.net/browse/NAS-130029) a necessary library from the base installation and *this command will fail:* - - python3 -m venv .venv - -The following workaround should accomplish the same task in more steps. -``` -python3 -m venv --without-pip .venv -curl -OL https://bootstrap.pypa.io/pip/pip.pyz -.venv/bin/python3 pip.pyz install pip -rm pip.pyz -``` -*Note: This process and the resulting build environment will cache some items under `~/.local/share` in addition to the project directory.* - -Activate the venv into your *current* shell session. - - source .venv/bin/activate - -Develop away. Note that when you're done, you can undo this activation and return to the system's default Python environment. Just call a function that activation has inserted into your shell session: - - deactivate - -For more information on Python standard venvs, go to [the source](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments). - -### Hatching a build - -While in an *active* session, install the [Hatch](https://hatch.pypa.io) project manager. This will load quite a flurry of dependencies, but will only do so into the new `.venv` directory. *(And a bit into self-managed directories under `~/.local/share`.)* - - pip install hatch - -Build the "zipapp" target. This will create a `dist/jlmkr` tool which is the direct descendant of Jip-Hop's original `jlmkr.py` script. - - hatch build -t zipapp - -Now build the "appzip" target. This bundles the tool, `README.md` and `LICENSING` into a downloadable zip archive. - - hatch build -t appzip - -If you make any changes *to the embedded builder plugins* that perform the above, then you will need to clear caches between builds. Otherwise and generally, you will not need to do so. - - hatch env prune - -Hatch has oodles more features yet to be explored, such as: automated testing, code coverage, and style checking. For now, we've gotten it building. +The rootfs image `jlmkr` downloads comes from the [Linux Containers Image server](https://images.linuxcontainers.org). These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the `alpine` image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine. ## Filing Issues and Community Support