diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index a27a195..54f96c0 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -11,11 +11,11 @@ jobs: - uses: cachix/cachix-action@v12 with: name: rose - authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' + authToken: "${{ secrets.CACHIX_AUTH_TOKEN }}" - name: Build Nix run: nix build -j8 .#devShells.x86_64-linux.default - name: Typecheck - if: success() || failure() # Means that we run all steps even if one fails. + if: success() || failure() # Means that we run all steps even if one fails. run: nix develop --command make typecheck - name: Test if: success() || failure() diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ea35c46..af3f030 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -81,6 +81,72 @@ queries to batch the writes. The update process is also parallelizable, so we shard workloads across multiple processes. +# Virtual Filesystem + +We use the `llfuse` library for the virtual filesystem, as it's the only +reasonable choice for Python. `llfuse` is a fairly low-level library for FUSE +though, leaving us to, for example, work with inodes and all of their problems. + +Because of this, we split the Virtual Filesystem code into two main layers: + +1. `RoseLogicalFS`: A logical core of Rosé's domain logic, which exposes + syscall-like functions at a higher semantic level. +2. `VirtualFS`: A wrapper around `RoseLogicalFS` that translates syscalls into + Rosé's semantics, manages inodes, caches for performance, and "lies" a + little so that user tools work transparently. + +There are a few other useful classes as well. Because the virtual filesystem +interface is object oriented, the entire virtual filesystem module is organized +following object orientation principles. We favor composition over inheritance +^.~ + +## Ghost Files + +While Rosé converts a music library into a virtual filesystem, some of its +behaviors do not truly follow the conventions and expectations of a filesystem. +For example, filenames are very rigid. + +For the most part, Rosé is a read only hierarchy, which still aligns with the +expectations of other tools. However, there are two operations which diverge +significantly from conventional expectations: + +1. Adding a release to a collage +2. Adding a track to a playlist + +Let's start with adding a release to a collage. When we add a release to a +collage, we take the `mkdir` call and translate that to an +`add_release_to_collage` call. Afterwards, the release is in the collage. And +the tracks of the release magically appear inside the directory we just +`mkdir`-ed, since that directory is simply the release. + +But we don't usually call `mkdir`. We call `cp -r "1. Releases/x" "8. Collages/Road Trip/"`. +And so we don't stop at a `mkdir`, we continue to attempt to `create` every +single track in the release. Which is problematic, as after the `mkdir` call, +all the files already exist in the directory. + +Strictly speaking, nothing's wrong. `cp -r` errors out, but the intended effect +has been achieved. However, it's quite jarring for the conventional tools to +produce errors and warnings when undertaking a happy path action. So, how can +we silence those errors and warnings? + +We do so with "Ghost Files." We pretend that files exist (or don't exist!) for +2-5 seconds after one of the above two operations occur. In the case of +collages, the `VirtualFS` layer simply pretends that the directory is empty for +5 seconds, and redirects any incoming writes into the directory to `/dev/null`. + +And in the case of playlists, where we copy a track into a playlist directory, +we do not have a directory magically appearing. We actually store the data +written into an in-memory dictionary, and upon release, we read that data to +parse out the `roseid` tag. However, once we invoke `add_track_to_playlist`, +the filename and inode of the new track will differ from the one passed to +the `create` call. To the tool interacting with Rosé, it appears as if the file +they just wrote vanished. + +We also solve playlists with Ghost Files: We pretend that a written playlist +track exists for 2 seconds before we delete it. This way, a command like +`cp --preserve=mode` will have time to replicate file attributes after copying +(though we do nothing with them :p). + # Logging Logs are written to stderr and to `${XDG_STATE_HOME:-$HOME/.local/state}/rose/rose.log`. diff --git a/docs/METADATA_MANAGEMENT.md b/docs/METADATA_MANAGEMENT.md index 3a80743..5530221 100644 --- a/docs/METADATA_MANAGEMENT.md +++ b/docs/METADATA_MANAGEMENT.md @@ -250,4 +250,3 @@ from additional fields. | Track Number | `tracknumber` | | | Disc Number | `discnumber` | | | Rose ID | `roseid` | | - diff --git a/docs/PLAYLISTS_COLLAGES.md b/docs/PLAYLISTS_COLLAGES.md index 844000b..5790585 100644 --- a/docs/PLAYLISTS_COLLAGES.md +++ b/docs/PLAYLISTS_COLLAGES.md @@ -133,12 +133,6 @@ $ rose playlists add-track "Evening" "018b6514-6fb7-7cc6-9d23-8eaf0b1beee8" Virtual filesystem: -_When copying a release directory, there will be errors if `cp` is used. They -are safe to ignore. The error happens because the action of creating the -directory adds the release to the collage. After that point, all files are -already part of the release directory, yet `cp` attempts to copy the files over -too. We will try to fix this later._ - ```bash $ cd $fuse_mount_dir diff --git a/docs/README.md b/docs/README.md index b52935c..9eae9a2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,3 +8,5 @@ For more detailed documentation, please visit the following files: - [Using Playlists & Collages](./PLAYLISTS_COLLAGES.md) - [Maintaining the Cache](./CACHE_MAINTENANCE.md) - [Architecture](./ARCHITECTURE.md) + +The code is also commented somewhat decently, if you'd like to read the source. diff --git a/docs/VIRTUAL_FILESYSTEM.md b/docs/VIRTUAL_FILESYSTEM.md index 26c321a..4b7a3d5 100644 --- a/docs/VIRTUAL_FILESYSTEM.md +++ b/docs/VIRTUAL_FILESYSTEM.md @@ -158,7 +158,6 @@ _Deletion will move the release into the trashbin, following the [freedesktop spec](https://freedesktop.org/wiki/Specifications/trash-spec/). The release can be restored later if the deletion was accidental._ - Command line: ```bash