Skip to content

Commit

Permalink
refactor + clean up cli interface
Browse files Browse the repository at this point in the history
  • Loading branch information
azuline committed Oct 17, 2023
1 parent d2e2d46 commit 3e0d185
Show file tree
Hide file tree
Showing 12 changed files with 121 additions and 85 deletions.
65 changes: 35 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ A virtual filesystem for music and metadata improvement tooling.

## The Virtual Filesystem

Rosé reads a source directory of albums like this:
Rosé reads a source directory of releases like this:

```tree
.
Expand All @@ -33,7 +33,7 @@ The virtual filesystem constructed from the above source directory is:

```tree
.
├── Albums
├── Releases
│   ├── BLACKPINK - 2016. SQUARE ONE - Single [K-Pop] {YG Entertainment}
│   ├── BLACKPINK - 2016. SQUARE TWO - Single [K-Pop] {YG Entertainment}
│   ├── LOOΠΔ 1_3 - 2017. Love & Evil [K-Pop] {BlockBerry Creative}
Expand Down Expand Up @@ -103,7 +103,7 @@ You can install Nix and Nix Flakes with
# Usage

```
Usage: rose [OPTIONS] COMMAND [ARGS]...
Usage: python -m rose [OPTIONS] COMMAND [ARGS]...
A virtual filesystem for music and metadata improvement tooling.
Expand All @@ -113,9 +113,10 @@ Options:
--help Show this message and exit.
Commands:
cache Manage the read cache.
fs Manage the virtual library.
print Print cached library data (JSON-encoded).
cache Manage the read cache.
collages Manage collages.
fs Manage the virtual library.
releases Manage releases.
```

## Configuration
Expand Down Expand Up @@ -148,11 +149,11 @@ The `--config/-c` flag overrides the config location.

## Music Source Dir

The `music_source_dir` must be a flat directory of albums, meaning all albums
must be top-level directories inside `music_source_dir`. Each album should also
The `music_source_dir` must be a flat directory of releases, meaning all releases
must be top-level directories inside `music_source_dir`. Each release should also
be a single directory in `music_source_dir`.

Every directory should follow the format: `$music_source_dir/$album_name/$track.mp3`.
Every directory should follow the format: `$music_source_dir/$release_name/$track.mp3`.
Additional nested directories are not currently supported.

So for example: `$music_source_dir/BLACKPINK - 2016. SQUARE ONE/*.mp3`.
Expand Down Expand Up @@ -182,10 +183,9 @@ TODO

## Data Querying

The `rose print` family of commands (e.g. `rose print albums`) prints out data
in the read cache in a JSON-encoded format. The output of this command can be
piped into tools like `jq`, `fx`, and others in order to further process the
output.
There are several commands that print out data from the read cache in a
JSON-encoded format (e.g. `rose releases print` and `rose collages print`). The
command output can be piped into tools like `jq`, `fx`, and others.

## Tagging Conventions

Expand Down Expand Up @@ -222,23 +222,25 @@ TODO

## The Read Cache

For performance, Rosé processes every audio file in the `music_source_dir` and
records all metadata in a SQLite read cache. This read cache does not accept
writes, meaning it can always be fully recreated from the source audio files.
For performance, Rosé stores a copy of every source file's metadata in a SQLite
read cache. The read cache does not accept writes; thus it can always be fully
recreated from the source files. It can be freely deleted and recreated without
consequence.

The cache can be updated with the command `rose cache update`. By default, the
cache updater will only recheck files that have changed since the last run. To
override this behavior and always re-read file tags, run `rose cache update --force`.
override this behavior and always re-read file tags, run `rose cache update
--force`.

By default, the cache is updated on `mount` and on changes made through the
virtual filesystem. However, changes made directly to the `music_source_dir`
will not trigger a cache update. This can lead to cache drift.
By default, the cache is updated on `rose fs mount` and when files are changed
through the virtual filesystem. However, if the `music_source_dir` is changed
directly, Rosé does not automatically update the cache, which can lead to cache
drifts.

You can solve this by running `rose cache watch`. This starts a background
watcher that listens to inotify and reactively updates the cache whenever a
source file changes. This can be useful if you synchronize your music library
between two computers, or use an external tool to directly modify the source
files (instead of modifying through the virtual filesystem).
You can solve this problem by running `rose cache watch`. This starts a watcher
that triggers a cache update whenever a source file changes. This can be useful
if you synchronize your music library between two computers, or use another
tool to directly modify the `music_source_dir`.

## Systemd Unit Files

Expand All @@ -252,19 +254,22 @@ Logs are written to stderr and to `${XDG_STATE_HOME:-$HOME/.local/state}/rose/ro

Rosé has a simple uni-directional looping architecture.

1. The source files: audio+playlists+collages, are the single source of truth.
1. The source files: audio+playlists+collages, are the single source of truth
for your music library.
2. The read cache is transient and deterministically derived from source
files. It can always be deleted and fully recreated from source files.
3. The virtual filesystem uses the read cache (for performance). Writes to the
virtual filesystem update the source files and then refresh the read cache.
4. The metadata manager writes to the source files directly, which in turn
refreshes the read cache.
4. The metadata tooling writes to the source files directly, which in turn
refreshes the read cache. The metadata tooling reads from the read cache for
performance.

```mermaid
flowchart BT
M[Metadata Tooling] -->|Writes| S
S[Source Files] -->|Populates| C
C[Read Cache] -->|Renders| V
C[Read Cache] -->|Improves Performance| M
V[Virtual Filesystem] -->|Updates| S
```

Expand All @@ -273,12 +278,12 @@ and uni-directional mutations. This has a few benefits:

- Rosé and the source files always have the same metadata. If they drift, `rose
cache update` will rebuild the cache such that it fully matches the source
files. And if the watchdog is running, there should not be any drift.
files. And if `rose cache watch` is running, there should not be any drift.
- Rosé is easily synchronized across machines. As long as the source
files are synchronized, Rosé will rebuild the exact same cache regardless of
machine.

Rosé writes `.rose.{uuid}.toml` files into each album's directory as a way to
Rosé writes `.rose.{uuid}.toml` files into each release's directory as a way to
preserve release-level state and keep release UUIDs consistent across full
cache rebuilds.

Expand Down
27 changes: 20 additions & 7 deletions rose/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
import click

from rose.cache import migrate_database, update_cache
from rose.collages import print_collages
from rose.config import Config
from rose.print import print_releases
from rose.releases import print_releases
from rose.virtualfs import mount_virtualfs, unmount_virtualfs


Expand Down Expand Up @@ -76,17 +77,29 @@ def unmount(ctx: Context) -> None:
unmount_virtualfs(ctx.config)


@cli.group(name="print")
def printg() -> None:
"""Print cached library data (JSON-encoded)."""
@cli.group()
def releases() -> None:
"""Manage releases."""


@printg.command()
@releases.command(name="print")
@click.pass_obj
def albums(ctx: Context) -> None:
"""Print albums."""
def print1(ctx: Context) -> None:
"""Print JSON-encoded releases."""
print_releases(ctx.config)


@cli.group()
def collages() -> None:
"""Manage collages."""


@collages.command(name="print")
@click.pass_obj
def print2(ctx: Context) -> None:
"""Print JSON-encoded collages."""
print_collages(ctx.config)


if __name__ == "__main__":
cli()
3 changes: 2 additions & 1 deletion rose/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
import uuid6

from rose.artiststr import format_artist_string
from rose.common import sanitize_filename
from rose.config import Config
from rose.sanitize import sanitize_filename
from rose.tagger import AudioFile

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -1140,6 +1140,7 @@ def get_release_id_from_virtual_dirname(c: Config, release_virtual_dirname: str)
(release_virtual_dirname,),
)
if row := cursor.fetchone():
assert isinstance(row["id"], str)
return row["id"]
return None

Expand Down
26 changes: 13 additions & 13 deletions rose/cache_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -371,10 +371,10 @@ def test_update_cache_collages_nonexistent_release_id(config: Config) -> None:

@pytest.mark.usefixtures("seeded_cache")
def test_list_releases(config: Config) -> None:
albums = list(list_releases(config))
assert albums == [
releases = list(list_releases(config))
assert releases == [
CachedRelease(
datafile_mtime=albums[0].datafile_mtime, # IGNORE THIS FIELD.
datafile_mtime=releases[0].datafile_mtime, # IGNORE THIS FIELD.
id="r1",
source_path=Path(config.music_source_dir / "r1"),
cover_image_path=None,
Expand All @@ -393,7 +393,7 @@ def test_list_releases(config: Config) -> None:
formatted_artists="Techno Man;Bass Man",
),
CachedRelease(
datafile_mtime=albums[1].datafile_mtime, # IGNORE THIS FIELD.
datafile_mtime=releases[1].datafile_mtime, # IGNORE THIS FIELD.
id="r2",
source_path=Path(config.music_source_dir / "r2"),
cover_image_path=Path(config.music_source_dir / "r2" / "cover.jpg"),
Expand All @@ -413,10 +413,10 @@ def test_list_releases(config: Config) -> None:
),
]

albums = list(list_releases(config, sanitized_artist_filter="Techno Man"))
assert albums == [
releases = list(list_releases(config, sanitized_artist_filter="Techno Man"))
assert releases == [
CachedRelease(
datafile_mtime=albums[0].datafile_mtime, # IGNORE THIS FIELD.
datafile_mtime=releases[0].datafile_mtime, # IGNORE THIS FIELD.
id="r1",
source_path=Path(config.music_source_dir / "r1"),
cover_image_path=None,
Expand All @@ -436,10 +436,10 @@ def test_list_releases(config: Config) -> None:
),
]

albums = list(list_releases(config, sanitized_genre_filter="Techno"))
assert albums == [
releases = list(list_releases(config, sanitized_genre_filter="Techno"))
assert releases == [
CachedRelease(
datafile_mtime=albums[0].datafile_mtime, # IGNORE THIS FIELD.
datafile_mtime=releases[0].datafile_mtime, # IGNORE THIS FIELD.
id="r1",
source_path=Path(config.music_source_dir / "r1"),
cover_image_path=None,
Expand All @@ -459,10 +459,10 @@ def test_list_releases(config: Config) -> None:
),
]

albums = list(list_releases(config, sanitized_label_filter="Silk Music"))
assert albums == [
releases = list(list_releases(config, sanitized_label_filter="Silk Music"))
assert releases == [
CachedRelease(
datafile_mtime=albums[0].datafile_mtime, # IGNORE THIS FIELD.
datafile_mtime=releases[0].datafile_mtime, # IGNORE THIS FIELD.
id="r1",
source_path=Path(config.music_source_dir / "r1"),
cover_image_path=None,
Expand Down
14 changes: 14 additions & 0 deletions rose/collage.py → rose/collages.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
import json
import logging
from pathlib import Path
from typing import Any

import tomli_w
import tomllib

from rose.cache import (
get_release_id_from_virtual_dirname,
list_collage_releases,
list_collages,
update_cache_evict_nonexistent_collages,
update_cache_for_collages,
)
Expand Down Expand Up @@ -64,5 +68,15 @@ def delete_collage(c: Config, collage_name: str) -> None:
update_cache_evict_nonexistent_collages(c)


def print_collages(c: Config) -> None:
out: dict[str, list[dict[str, Any]]] = {}
collage_names = list(list_collages(c))
for name in collage_names:
out[name] = []
for pos, virtual_dirname in list_collage_releases(c, name):
out[name].append({"position": pos, "release": virtual_dirname})
print(json.dumps(out))


def collage_path(c: Config, name: str) -> Path:
return c.music_source_dir / "!collages" / f"{name}.toml"
2 changes: 1 addition & 1 deletion rose/collage_test.py → rose/collages_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from rose.cache import connect, update_cache
from rose.cache_test import TEST_COLLAGE_1, TEST_RELEASE_2, TEST_RELEASE_3
from rose.collage import (
from rose.collages import (
add_release_to_collage,
create_collage,
delete_collage,
Expand Down
5 changes: 5 additions & 0 deletions rose/sanitize.py → rose/common.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
import re


class RoseError(Exception):
pass


ILLEGAL_FS_CHARS_REGEX = re.compile(r'[:\?<>\\*\|"\/]+')


Expand Down
2 changes: 1 addition & 1 deletion rose/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import tomllib

from rose.errors import RoseError
from rose.common import RoseError

XDG_CONFIG_HOME = Path(os.environ.get("XDG_CONFIG_HOME", os.environ["HOME"] + "/.config"))
CONFIG_PATH = XDG_CONFIG_HOME / "rose" / "config.toml"
Expand Down
2 changes: 0 additions & 2 deletions rose/errors.py

This file was deleted.

File renamed without changes.
2 changes: 1 addition & 1 deletion rose/tagger.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import mutagen.oggvorbis

from rose.artiststr import Artists, parse_artist_string
from rose.errors import RoseError
from rose.common import RoseError

TAG_SPLITTER_REGEX = re.compile(r" \\\\ | / |; ?| vs\. ")
YEAR_REGEX = re.compile(r"\d{4}$")
Expand Down
Loading

0 comments on commit 3e0d185

Please sign in to comment.