diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index a27a195..54f96c0 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -11,11 +11,11 @@ jobs: - uses: cachix/cachix-action@v12 with: name: rose - authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' + authToken: "${{ secrets.CACHIX_AUTH_TOKEN }}" - name: Build Nix run: nix build -j8 .#devShells.x86_64-linux.default - name: Typecheck - if: success() || failure() # Means that we run all steps even if one fails. + if: success() || failure() # Means that we run all steps even if one fails. run: nix develop --command make typecheck - name: Test if: success() || failure() diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ea35c46..1e6faba 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -81,6 +81,62 @@ queries to batch the writes. The update process is also parallelizable, so we shard workloads across multiple processes. +# Virtual Filesystem + +We use the `llfuse` library for the virtual filesystem. `llfuse` is a fairly +low-level library for FUSE, which leaves us to work with system-level concepts +such as inodes. + +Though these concepts have much to do with filesystems, they have little to do +with Rosé. Thus, we have split the Virtual Filesystem code into two main layers: + +1. `RoseLogicalCore`: A logical core of Rosé's domain logic, which exposes + syscall-like functions at a higher semantic level. +2. `VirtualFS`: A wrapper around `RoseLogicalFS` that translates syscalls into + Rosé's semantics, manages inodes, caches for performance, and "lies" a + little so that user tools work transparently. + +There are a few other useful classes as well. Because the virtual filesystem +interface is object oriented, the entire virtual filesystem module is organized +following object orientation principles. We favor composition over inheritance +^.~ + +## Ghost Files + +Some of Rosé operations break the conventional expectations of a filesystem. +Most operations are still _decent_, but the following two diverge pretty +heavily: + +1. Adding a release to a collage +2. Adding a track to a playlist + +When we add a release to a collage, we take the `mkdir` call and translate that +to an `add_release_to_collage` call. Afterwards, the release is in the collage. +And the tracks of the release magically appear inside the directory afterwards, +since that directory is simply the release. + +This is problematic, because a command like `cp -r` immediately attempts to +then replicate files into a supposedly empty directory, and errors. + +Strictly speaking, nothing's wrong. Though `cp` errors, the release has been +added to the collage, and the next `ls` command shows that the release has been +successfully added to a collage. However, it is nice when basic tools work +without erroring. + +Thus: "Ghost Files." In order to keep tools like `cp` happy, we pretend that +files exist (or don't exist!) for 2-5 seconds after one of the above two +operations occur. In the case of collages, the `VirtualFS` layer pretends that +the new directory is empty for 5 seconds, and redirects any incoming writes +into the directory to `/dev/null`. + +For playlists, the problem arises in that the copied file vanishes immediately +once the file handle is released. A command like `cp --preserve=mode`, which +attempts to replicate file attributes after releasing the file, fails. + +So in the playlists case, we pretend that the written file exists for 2 seconds +after we wrote it. This way, `cp --preserve=mode` can replicate its attributes +onto a ghost file and happily exit without errors. + # Logging Logs are written to stderr and to `${XDG_STATE_HOME:-$HOME/.local/state}/rose/rose.log`. diff --git a/docs/METADATA_MANAGEMENT.md b/docs/METADATA_MANAGEMENT.md index 3a80743..5530221 100644 --- a/docs/METADATA_MANAGEMENT.md +++ b/docs/METADATA_MANAGEMENT.md @@ -250,4 +250,3 @@ from additional fields. | Track Number | `tracknumber` | | | Disc Number | `discnumber` | | | Rose ID | `roseid` | | - diff --git a/docs/PLAYLISTS_COLLAGES.md b/docs/PLAYLISTS_COLLAGES.md index 844000b..5790585 100644 --- a/docs/PLAYLISTS_COLLAGES.md +++ b/docs/PLAYLISTS_COLLAGES.md @@ -133,12 +133,6 @@ $ rose playlists add-track "Evening" "018b6514-6fb7-7cc6-9d23-8eaf0b1beee8" Virtual filesystem: -_When copying a release directory, there will be errors if `cp` is used. They -are safe to ignore. The error happens because the action of creating the -directory adds the release to the collage. After that point, all files are -already part of the release directory, yet `cp` attempts to copy the files over -too. We will try to fix this later._ - ```bash $ cd $fuse_mount_dir diff --git a/docs/README.md b/docs/README.md index b52935c..9eae9a2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,3 +8,5 @@ For more detailed documentation, please visit the following files: - [Using Playlists & Collages](./PLAYLISTS_COLLAGES.md) - [Maintaining the Cache](./CACHE_MAINTENANCE.md) - [Architecture](./ARCHITECTURE.md) + +The code is also commented somewhat decently, if you'd like to read the source. diff --git a/docs/VIRTUAL_FILESYSTEM.md b/docs/VIRTUAL_FILESYSTEM.md index 26c321a..4b7a3d5 100644 --- a/docs/VIRTUAL_FILESYSTEM.md +++ b/docs/VIRTUAL_FILESYSTEM.md @@ -158,7 +158,6 @@ _Deletion will move the release into the trashbin, following the [freedesktop spec](https://freedesktop.org/wiki/Specifications/trash-spec/). The release can be restored later if the deletion was accidental._ - Command line: ```bash diff --git a/rose/virtualfs.py b/rose/virtualfs.py index 3a8d42b..20bbd0f 100644 --- a/rose/virtualfs.py +++ b/rose/virtualfs.py @@ -370,7 +370,7 @@ def unwrap_host(self, rose_fh: int) -> int: ] -class RoseLogicalFS: +class RoseLogicalCore: def __init__(self, config: Config, fhandler: FileHandleManager): self.config = config self.fhandler = fhandler @@ -933,7 +933,7 @@ class VirtualFS(llfuse.Operations): # type: ignore def __init__(self, config: Config): self.fhandler = FileHandleManager() - self.rose = RoseLogicalFS(config, self.fhandler) + self.rose = RoseLogicalCore(config, self.fhandler) self.inodes = INodeManager(config) self.default_attrs = { # Well, this should be ok for now. I really don't want to track this... we indeed change