Skip to content

Commit

Permalink
start splitting things up
Browse files Browse the repository at this point in the history
  • Loading branch information
@azuline
azuline committed Oct 26, 2023
1 parent b7df23f commit ee046d6
Show file tree
Hide file tree
Showing 8 changed files with 264 additions and 194 deletions.
249 changes: 55 additions & 194 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,62 @@
# Rosé

_Work in Progress. See [Issue #1](https://github.com/azuline/rose/issues/1) for
the current state._
> [!IMPORTANT]
> Rosé is under active development. See [Issue #1](https://github.com/azuline/rose/issues/1)
> for progress updates.
**WARNING: Rosé modifies your audio files. If you do not want to modify your
audio files, you should not use Rosé.**
Rosé is a music manager for Unix-based systems. Rosé provides a virtual FUSE
filesystem for managing your music library and various functions for editing
and improving your music library's metadata and tags.

A virtual filesystem for music and metadata improvement tooling.
> [!NOTE]
> Rosé modifies the managed audio files. If you do not want to modify your
> audio files, for example because they are seeding in a bittorrent client, you
> should not use Rosé.
TODO: Video

## Installation

Install Rosé with Nix Flakes. If you do not have Nix Flakes, you can install it
with [this installer](https://github.com/DeterminateSystems/nix-installer).

```bash
$ nix profile install github:azuline/rose#rose
```

In the future, other packaging systems may be considered. However, I strongly
dislike Python's packaging story, hence: Nix.

## Quickstart

TODO

## Features

TODO

## Requirements

TODO

## License

Copyright 2023 blissful <blissful@sunsetglow.net>

Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of the
License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

## Contributions

TODO

# OLD

## The Virtual Filesystem

Expand Down Expand Up @@ -92,16 +142,6 @@ manager!

I have yet to write this part of the tool. Please check back later!

# Installation

Install with Nix Flakes:

```bash
$ nix profile install github:azuline/rose#rose
```

You can install Nix and Nix Flakes with
[this installer](https://github.com/DeterminateSystems/nix-installer).

# Usage

Expand All @@ -123,75 +163,6 @@ Commands:
playlists Manage playlists.
```

## Configuration

Rosé must be configured prior to use. Rosé is configured via a TOML file
located at `${XDG_CONFIG_HOME:-$HOME/.config}/rose/config.toml`.

The configuration parameters, with examples, are:

```toml
# === Required values ===

# The directory containing the music to manage. This source directory WILL be
# modified by Rosé; if you do not want the files in directory to be modified,
# use another tool!
music_source_dir = "~/.music-source"
# The directory to mount the library's virtual filesystem on. This is the
# primary "user interface" of Rosé, and the directory in which you will be able
# to browse your music library.
fuse_mount_dir = "~/music"

# === Optional values ===

# The directory to write the cache to. Defaults to
# `${XDG_CACHE_HOME:-$HOME/.cache}/rose`.
cache_dir = "~/.cache/rose"
# Maximum parallel processes that the cache updater can spawn. Defaults to
# nproc/2. The higher this number is; the more performant the cache update will
# be.
max_proc = 4
# Artist aliases: Releases belonging to an alias will also "belong" to the main
# artist. This option improves the Artist browsing view by showing the aliased
# releases in the main artist's releases list.
artist_aliases = [
{ artist = "Abakus", aliases = ["Cinnamon Chasers"] },
{ artist = "tripleS", aliases = ["EVOLution", "LOVElution", "+(KR)ystal Eyes", "Acid Angel From Asia", "Acid Eyes"] },
]
# Artists, genres, and labels to show in the virtual filesystem navigation. By
# default, all artists, genres, and labels are shown. However, these values can
# be used to filter the listed values to a specific few. This is useful e.g. if
# you only care to browse your favorite genres and labels.
fuse_artists_whitelist = [ "xxx", "yyy" ]
fuse_genres_whitelist = [ "xxx", "yyy" ]
fuse_labels_whitelist = [ "xxx", "yyy" ]
# Artists, genres, and labels to hide from the virtual filesystem navigation.
# These options remove specific entities from the default policy of listing all
# entities. These options are mutually exclusive with the fuse_*_whitelist
# options; if both are specified for a given entity type, the configuration
# will not validate.
fuse_artists_blacklist = [ "xxx" ]
fuse_genres_blacklist = [ "xxx" ]
fuse_labels_blacklist = [ "xxx" ]
```

The `--config/-c` flag overrides the config location.

## Music Source Dir

The `music_source_dir` must be a flat directory of releases, meaning all releases
must be top-level directories inside `music_source_dir`. Each release should also
be a single directory in `music_source_dir`.

Every directory should follow the format: `$music_source_dir/$release_name/**/$track.mp3`.
A release can have arbitrarily many nested subdirectories.

So for example: `$music_source_dir/BLACKPINK - 2016. SQUARE ONE/*.mp3`.

Rosé writes playlist and collage files to the `$music_source_dir/.playlists`
and `$music_source_dir/.collages` directories. Each file is a human-readable
TOML file.

## Supported Filetypes

Rosé supports `.mp3`, `.m4a`, `.ogg` (vorbis), `.opus`, and `.flac` audio files.
Expand All @@ -213,116 +184,6 @@ TODO

TODO

## Data Querying

There are several commands that print out data from the read cache in a
JSON-encoded format (e.g. `rose releases print` and `rose collages print`). The
command output can be piped into tools like `jq`, `fx`, and others.

## Tagging Conventions

Rosé is lenient in the tags it ingests, but has opinionated conventions for the
tags it writes.

### Multi-Valued Tags

Rosé supports multiple values for the artists, genres, and labels tags. Rosé
uses the `;` character as a tag delimiter. For example, `genre=Deep House;Techno`.

Rosé also preserves the artists' role in the artist tag by using specialized
delimiters. for example, `artist=Pyotr Ilyich Tchaikovsky performed by André Previn;London Symphony Orchestra feat. Barack Obama`.

The artist tag is described by the following grammar:

```
artist-tag ::= composer dj main guest remixer producer
composer ::= name ' performed by '
dj ::= name ' pres. '
main ::= name
guest ::= ' feat. ' name
remixer ::= ' remixed by ' name
producer ::= ' produced by ' name
name ::= string ';' name | string
```

## New Releases

TODO

## Artist Aliases

TODO

## The Read Cache

For performance, Rosé stores a copy of every source file's metadata in a SQLite
read cache. The read cache does not accept writes; thus it can always be fully
recreated from the source files. It can be freely deleted and recreated without
consequence.

The cache can be updated with the command `rose cache update`. By default, the
cache updater will only recheck files that have changed since the last run. To
override this behavior and always re-read file tags, run `rose cache update
--force`.

By default, the cache is updated on `rose fs mount` and when files are changed
through the virtual filesystem. However, if the `music_source_dir` is changed
directly, Rosé does not automatically update the cache, which can lead to cache
drifts.

You can solve this problem by running `rose cache watch`. This starts a watcher
that triggers a cache update whenever a source file changes. This can be useful
if you synchronize your music library between two computers, or use another
tool to directly modify the `music_source_dir`.

## Systemd Unit Files

TODO; example unit files to schedule Rosé with systemd.

## Logging

Logs are written to stderr and to `${XDG_STATE_HOME:-$HOME/.local/state}/rose/rose.log`.

# Architecture

Rosé has a simple uni-directional looping architecture.

1. The source files: audio+playlists+collages, are the single source of truth
for your music library.
2. The read cache is transient and deterministically derived from source
files. It can always be deleted and fully recreated from source files.
3. The virtual filesystem uses the read cache (for performance). Writes to the
virtual filesystem update the source files and then refresh the read cache.
4. The metadata tooling writes to the source files directly, which in turn
refreshes the read cache. The metadata tooling reads from the read cache for
performance.

```mermaid
flowchart BT
S[Source Files]
C[Read Cache]
M[Metadata Tooling]
V[Virtual Filesystem]
S -->|Populates| C
M -->|Reads| C
M -->|Writes| S
V -->|Writes| S
V -->|Reads| C
```

This architecture takes care to ensure that there is a single source of truth
and uni-directional mutations. This has a few benefits:

- Rosé and the source files always have the same metadata. If they drift, `rose
cache update` will rebuild the cache such that it fully matches the source
files. And if `rose cache watch` is running, there should not be any drift.
- Rosé is easily synchronized across machines. As long as the source
files are synchronized, Rosé will rebuild the exact same cache regardless of
machine.

Rosé writes `.rose.{uuid}.toml` files into each release's directory as a way to
preserve release-level state and keep release UUIDs consistent across full
cache rebuilds.

Tracks are uniquely identified by the `(release_uuid, discnumber, tracknumber)`
tuple, which are also consistent across rebuilds.
66 changes: 66 additions & 0 deletions docs/ARCHITECTURE.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Architecture

Rosé has a simple uni-directional looping architecture.

```mermaid
flowchart BT
S[Source Files]
C[Read Cache]
M[Metadata Tooling]
V[Virtual Filesystem]
S -->|Populates| C
M -->|Reads| C
M -->|Writes| S
V -->|Writes| S
V -->|Reads| C
```

1. The source audio files, playlist files, and collage files are single sources
of truth. All writes are made directly to the sources files.
2. The read cache is transient and deterministically derived from source
files. It can always be deleted and fully recreated from source files. It
updates in response to changes in the source files.
3. The virtual filesystem uses the read cache for performance. All writes made
via the virtual filesystem are made to the Source Files, which in turn
refreshes the read cache.
4. The metadata tooling uses the read cache for performance, but always writes
to the source files directly, which in turn refreshes the read cache.

This architecture takes care to ensure that there is a single source of truth
and uni-directional mutations. This means that Rosé and the source files always
have the same metadata. If the source files change, `rose cache update` is
guaranteed to rebuild the cache such that it fully matches the source files.
And if `rose cache watch` is running, the cache updates should happen
automatically.

This has some nice consequences:

- External tag editing tools do not disrupt Rosé. If an external tool modifies
the audio tags, Rosé's cache can always update to match the newly modified
source files.
- Rosé is easily synchronized across machines, for example with a tool like
Syncthing. As long as the source files are in-sync, Rosé's read cache will
match.
- An inconsistent state between source files and Rosé is trivially resolved.
This is different from music managers that retain a separate database,
because conflict resolution is then ambiguous. Whereas in Rosé, there are no
conflicts.

## Stable Release & Track Identifiers

Rosé assigns UUIDs to each release and track in order to identify them across
arbitrarily large metadata changes. These UUIDs are persisted to the source
files.

- Each release has a `.rose.{uuid}.toml` file, which preserves release-level
state, such as `New`. The UUID is in the filename instead of the file
contents for improved performance: we can collect the UUID via a `readdir`
call instead of an expensive file read.
- Each track has a custom `roseid` tag. This tag is written to the source audio
file. Read the `tagger.py` file for the exact field name used.

## Logging

Logs are written to stderr and to `${XDG_STATE_HOME:-$HOME/.local/state}/rose/rose.log`.
Debug logging can be turned on with the `--verbose/-v` option. Rosé is heavily
instrumented with debug logging.
21 changes: 21 additions & 0 deletions docs/CACHE_MAINTENANCE.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Maintaining the Cache

For performance, Rosé stores a copy of every source file's metadata in a SQLite
read cache. The read cache does not accept writes; thus it can always be fully
recreated from the source files. It can be freely deleted and recreated without
consequence.

The cache can be updated with the command `rose cache update`. By default, the
cache updater will only recheck files that have changed since the last run. To
override this behavior and always re-read file tags, run `rose cache update
--force`.

By default, the cache is updated on `rose fs mount` and when files are changed
through the virtual filesystem. However, if the `music_source_dir` is changed
directly, Rosé does not automatically update the cache, which can lead to cache
drifts.

You can solve this problem by running `rose cache watch`. This starts a watcher
that triggers a cache update whenever a source file changes. This can be useful
if you synchronize your music library between two computers, or use another
tool to directly modify the `music_source_dir`.
Loading

0 comments on commit ee046d6

Please sign in to comment.