Mantic merge 2023-09-27

doc/intro-to-autoinstall.rst

@@ -78,24 +78,47 @@ Autoinstall on the install media
 Another option for supplying autoinstall to the Ubuntu installer is to place a
 file named :code:`autoinstall.yaml` on the install media itself.
 
-There are two potential locations for the :code:`autoinstall.yaml` file:
- * At the root of the "CD-ROM". When you write the installation ISO to a USB
-   Flash Drive, this can be done by copying the :code:`autoinstall.yaml` to the
-   partition containing the contents of the ISO - i.e.,
-   in the directory containing the ``casper`` sub-directory.
- * On the rootfs of the installation system - this option will typically
-   require modifying the installation ISO and is not suggested, but is
-   supported.
-
-Directly specifying autoinstall as a :code:`autoinstall.yaml` file does not
-require a :code:`#cloud-config` header, and does not use a top level
-``autoinstall:`` key. The autoinstall directives are placed at the top
-level. For example:
+There are two potential locations that subiquity will check for the
+:code:`autoinstall.yaml` file:
 
-.. code-block:: yaml
+* At the root of the "CD-ROM". When you write the installation ISO to a USB
+  Flash Drive, this can be done by copying the :code:`autoinstall.yaml` to the
+  partition containing the contents of the ISO - i.e.,
+  in the directory containing the ``casper`` sub-directory.
+* On the rootfs of the installation system - this option will typically
+  require modifying the installation ISO and is not suggested, but is
+  supported.
+
+Alternatively, you can pass the location of the autoinstall file on the kernel
+command line via the :code:`subiquity.autoinstallpath` parameter, where the
+path is relative to the rootfs of the installation system. For example:
+
+* :code:`subiquity.autoinstallpath=path/to/autoinstall.yaml`
+
+.. note::
+
+    Directly specifying autoinstall as a :code:`autoinstall.yaml` file does not
+    require a :code:`#cloud-config` header, and does not use a top level
+    ``autoinstall:`` key. The autoinstall directives are placed at the top
+    level. For example:
+
+    .. code-block:: yaml
+
+        version: 1
+        ....
+
+
+Order precedence of the autoinstall locations
+======================================
+
+Since there are many ways to specify the autoinstall file, it may happen that
+multiple locations are specified at once. Subiquity will look for the
+autoinstall file in the following order and pick the first existing one:
 
-    version: 1
-    ....
+1. Kernel command line
+2. Root of the installation system
+3. Cloud Config
+4. Root of the CD-ROM (ISO)
 
 
 Cloud-init and autoinstall interaction

doc/reference/autoinstall-reference.rst

@@ -96,7 +96,7 @@ locale
 
 * **type:** string
 * **default:** ``en_US.UTF-8``
-* **can be interactive:** yes, always interactive if any section is
+* **can be interactive:** yes
 
 The locale to configure for the installed system.
 
@@ -779,7 +779,7 @@ install
 * **type:** boolean or string (special value ``auto``)
 * **default:**: ``auto``
 
-Whether to install the available OEM meta-packages. The special value ``auto`` 
+Whether to install the available OEM meta-packages. The special value ``auto``
 -- which is the default -- enables the installation on ubuntu-desktop but not
 on ubuntu-server. This option has no effect on core boot classic.
 

documentation/autoinstall-reference.md

@@ -65,7 +65,7 @@ A list of shell commands to invoke as soon as the installer starts, in particula
 
 **type:** string
 **default:** `en_US.UTF-8`
-**can be interactive:** yes, always interactive if any section is
+**can be interactive:** yes
 
 The locale to configure for the installed system.
 

snapcraft.yaml

@@ -70,7 +70,7 @@ parts:
 
     source: https://git.launchpad.net/curtin
     source-type: git
-    source-commit: a7640fdcac396f9f09044dc7ca7553043ce4231c
+    source-commit: 64ea5fbe827aa98ddc63ea87de2de45689180c82
 
     override-pull: |
       craftctl default

subiquity/models/filesystem.py

@@ -1435,7 +1435,8 @@ def get_orig_model(self):
         # use on the V2 storage API.
         orig_model = FilesystemModel(self.bootloader, root=self.root)
         orig_model.target = self.target
-        orig_model.load_probe_data(self._probe_data)
+        if self._probe_data is not None:
+            orig_model.load_probe_data(self._probe_data)
         return orig_model
 
     def process_probe_data(self):

subiquity/models/tests/test_filesystem.py

@@ -328,6 +328,15 @@ def test_lv_ok_for_xxx(self):
         self.assertFalse(lv.ok_for_raid)
         self.assertFalse(lv.ok_for_lvm_vg)
 
+    def test_get_orig_model_no_probe_data(self):
+        # When v2/get_orig_data gets called early, model._probe_data is still
+        # None. Ensure get_orig_model() does not fail.
+        model = make_model()
+
+        model._probe_data = None
+        orig_model = model.get_orig_model()
+        self.assertIsNone(orig_model._probe_data)
+
 
 def fake_up_blockdata_disk(disk, **kw):
     model = disk._m

subiquity/server/controllers/filesystem.py

@@ -20,7 +20,6 @@
 import logging
 import os
 import pathlib
-import select
 import time
 from typing import Any, Callable, Dict, List, Optional, Union
 
@@ -273,6 +272,7 @@ def __init__(self, app):
         self._system_mounter: Optional[Mounter] = None
         self._role_to_device: Dict[str, _Device] = {}
         self._device_to_structure: Dict[_Device, snapdapi.OnVolume] = {}
+        self._pyudev_context: Optional[pyudev.Context] = None
         self.use_tpm: bool = False
         self.locked_probe_data: bool = False
         # If probe data come in while we are doing partitioning, store it in
@@ -299,7 +299,7 @@ async def configured(self):
         ):
             self.app.base_model.source.search_drivers = not self.is_core_boot_classic()
         await super().configured()
-        self.stop_listening_udev()
+        self.stop_monitor()
 
     async def _mount_systems_dir(self, variation_name):
         self._source_handler = self.app.controllers.Source.get_handler(variation_name)
@@ -1293,6 +1293,13 @@ async def _probe(self, *, context=None):
             finally:
                 elapsed = time.time() - start
                 log.debug(f"{short_label} probing took {elapsed:.1f} seconds")
+                # In the past, this start_monitor() equivalent was much sooner.
+                # We don't actually need the information it provides though
+                # until a probe has finished, so the start_monitor() is delayed
+                # to here. start_monitor() is allowed after a failed probe, in
+                # case of a hotplug event, perhaps to remove a problematic
+                # device.
+                self.start_monitor()
             break
 
     async def run_autoinstall_guided(self, layout):
@@ -1456,21 +1463,31 @@ def start(self):
         self._start_task = schedule_task(self._start())
 
     async def _start(self):
-        context = pyudev.Context()
-        self._monitor = pyudev.Monitor.from_netlink(context)
-        self._monitor.filter_by(subsystem="block")
-        self._monitor.enable_receiving()
-        self.start_listening_udev()
         await self._probe_task.start()
 
-    def start_listening_udev(self):
+    def start_monitor(self):
+        if self._configured:
+            return
+
+        log.debug("start_monitor")
+        if self._pyudev_context is None:
+            self._pyudev_context = pyudev.Context()
+        self._monitor = pyudev.Monitor.from_netlink(self._pyudev_context)
+        self._monitor.filter_by(subsystem="block")
+        self._monitor.start()
         loop = asyncio.get_running_loop()
         loop.add_reader(self._monitor.fileno(), self._udev_event)
 
-    def stop_listening_udev(self):
+    def stop_monitor(self):
+        if self._monitor is None:
+            return
+
+        log.debug("stop_monitor")
         loop = asyncio.get_running_loop()
         loop.remove_reader(self._monitor.fileno())
 
+        self._monitor = None
+
     def ensure_probing(self):
         try:
             self._probe_task.start_sync()
@@ -1480,21 +1497,20 @@ def ensure_probing(self):
             log.debug("Triggered Probert run on udev event")
 
     def _udev_event(self):
+        # We outright stop monitoring because we're not super concerned about
+        # the specifics of the udev event, only that one happened and that when
+        # the events settle, we want to reprobe.  This is significantly faster
+        # than keeping a monitor around and draining the event queue.
+        # LP: #2009141
+        self.stop_monitor()
+
         cp = run_command(["udevadm", "settle", "-t", "0"])
+
         if cp.returncode != 0:
             log.debug("waiting 0.1 to let udev event queue settle")
-            self.stop_listening_udev()
             loop = asyncio.get_running_loop()
-            loop.call_later(0.1, self.start_listening_udev)
+            loop.call_later(0.1, self._udev_event)
             return
-        # Drain the udev events in the queue -- if we stopped listening to
-        # allow udev to settle, it's good bet there is more than one event to
-        # process and we don't want to kick off a full block probe for each
-        # one.  It's a touch unfortunate that pyudev doesn't have a
-        # non-blocking read so we resort to select().
-        while select.select([self._monitor.fileno()], [], [], 0)[0]:
-            action, dev = self._monitor.receive_device()
-            log.debug("_udev_event %s %s", action, dev)
         self.ensure_probing()
 
     def make_autoinstall(self):

subiquity/server/controllers/oem.py

@@ -65,6 +65,7 @@ def __init__(self, app) -> None:
 
         self.load_metapkgs_task: Optional[asyncio.Task] = None
         self.kernel_configured_event = asyncio.Event()
+        self.fs_configured_event = asyncio.Event()
 
     def start(self) -> None:
         self._wait_confirmation = asyncio.Event()
@@ -76,6 +77,9 @@ def start(self) -> None:
         self.app.hub.subscribe(
             (InstallerChannels.CONFIGURED, "kernel"), self.kernel_configured_event.set
         )
+        self.app.hub.subscribe(
+            (InstallerChannels.CONFIGURED, "filesystem"), self.fs_configured_event.set
+        )
 
         async def list_and_mark_configured() -> None:
             await self.load_metapackages_list()
@@ -128,6 +132,12 @@ async def wants_oem_kernel(self, pkgname: str, *, context, overlay) -> bool:
     async def load_metapackages_list(self, context) -> None:
         with context.child("wait_confirmation"):
             await self._wait_confirmation.wait()
+        # In normal scenarios, the confirmation event comes after the
+        # storage/filesystem is configured. However, in semi automated desktop
+        # installs (especially in CI), it is possible that the events come in
+        # the reverse order. Let's be prepared for it by also waiting for the
+        # storage configured event.
+        await self.fs_configured_event.wait()
 
         # Only look for OEM meta-packages on supported variants and if we are
         # not running core boot.

subiquity/server/server.py

@@ -574,10 +574,11 @@ async def wait_for_cloudinit(self):
 
     def select_autoinstall(self):
         # precedence
-        # 1. autoinstall at root of drive
-        # 2. command line argument autoinstall
-        # 3. autoinstall supplied by cloud config
-        # 4. autoinstall baked into the iso, found at /cdrom/autoinstall.yaml
+        # 1. command line argument autoinstall
+        # 2. kernel command line argument subiquity.autoinstallpath
+        # 3. autoinstall at root of drive
+        # 4. autoinstall supplied by cloud config
+        # 5. autoinstall baked into the iso, found at /cdrom/autoinstall.yaml
 
         # if opts.autoinstall is set and empty, that means
         # autoinstall has been explicitly disabled.
@@ -588,9 +589,12 @@ def select_autoinstall(self):
         ):
             raise Exception(f"Autoinstall argument {self.opts.autoinstall} not found")
 
+        kernel_install_path = self.kernel_cmdline.get("subiquity.autoinstallpath", None)
+
         locations = (
-            self.base_relative(root_autoinstall_path),
             self.opts.autoinstall,
+            kernel_install_path,
+            self.base_relative(root_autoinstall_path),
             self.base_relative(cloud_autoinstall_path),
             self.base_relative(iso_autoinstall_path),
         )

subiquity/server/tests/test_server.py

@@ -54,39 +54,57 @@ def create(self, path, contents):
         return path
 
     def test_autoinstall_disabled(self):
+        self.server.opts.autoinstall = ""
+        self.server.kernel_cmdline = {"subiquity.autoinstallpath": "kernel"}
         self.create(root_autoinstall_path, "root")
         self.create(cloud_autoinstall_path, "cloud")
         self.create(iso_autoinstall_path, "iso")
-        self.server.opts.autoinstall = ""
         self.assertIsNone(self.server.select_autoinstall())
 
-    def test_root_wins(self):
+    def test_arg_wins(self):
+        arg = self.create(self.path("arg.autoinstall.yaml"), "arg")
+        self.server.opts.autoinstall = arg
+        kernel = self.create(self.path("kernel.autoinstall.yaml"), "kernel")
+        self.server.kernel_cmdline = {"subiquity.autoinstallpath": kernel}
         root = self.create(root_autoinstall_path, "root")
-        autoinstall = self.create(self.path("arg.autoinstall.yaml"), "arg")
-        self.server.opts.autoinstall = autoinstall
         self.create(cloud_autoinstall_path, "cloud")
         self.create(iso_autoinstall_path, "iso")
         self.assertEqual(root, self.server.select_autoinstall())
-        self.assert_contents(root, "root")
+        self.assert_contents(root, "arg")
 
-    def test_arg_wins(self):
-        root = self.path(root_autoinstall_path)
-        arg = self.create(self.path("arg.autoinstall.yaml"), "arg")
-        self.server.opts.autoinstall = arg
+    def test_kernel_wins(self):
+        self.server.opts.autoinstall = None
+        kernel = self.create(self.path("kernel.autoinstall.yaml"), "kernel")
+        self.server.kernel_cmdline = {"subiquity.autoinstallpath": kernel}
+        root = self.create(root_autoinstall_path, "root")
         self.create(cloud_autoinstall_path, "cloud")
         self.create(iso_autoinstall_path, "iso")
         self.assertEqual(root, self.server.select_autoinstall())
-        self.assert_contents(root, "arg")
+        self.assert_contents(root, "kernel")
+
+    def test_root_wins(self):
+        self.server.opts.autoinstall = None
+        self.server.kernel_cmdline = {}
+        root = self.create(root_autoinstall_path, "root")
+        self.create(cloud_autoinstall_path, "cloud")
+        self.create(iso_autoinstall_path, "iso")
+        self.assertEqual(root, self.server.select_autoinstall())
+        self.assert_contents(root, "root")
 
     def test_cloud_wins(self):
+        self.server.opts.autoinstall = None
+        self.server.kernel_cmdline = {}
         root = self.path(root_autoinstall_path)
         self.create(cloud_autoinstall_path, "cloud")
         self.create(iso_autoinstall_path, "iso")
         self.assertEqual(root, self.server.select_autoinstall())
         self.assert_contents(root, "cloud")
 
     def test_iso_wins(self):
+        self.server.opts.autoinstall = None
+        self.server.kernel_cmdline = {}
         root = self.path(root_autoinstall_path)
+        # No cloud config file
         self.create(iso_autoinstall_path, "iso")
         self.assertEqual(root, self.server.select_autoinstall())
         self.assert_contents(root, "iso")