better global config with configurable user directory

docs/api/stream_daq.md

@@ -2,7 +2,7 @@
 This module is a data acquisition module that captures video streams from Miniscopes based on the `Miniscope-SAMD-Framework` firmware. The firmware repository will be published in future updates but is currently under development and private.
 
 ## Command
-After [installation](../guide/installation.md) and customizing [device configurations](stream-dev-config) and [runtime configuration](models/config.md) if necessary, run the command described in [CLI Usage](../cli/main.rst).
+After [installation](../guide/installation.md) and customizing [device configurations](stream-dev-config) and [runtime configuration](models/config.md) if necessary, run the command described in [CLI Usage](../cli/index).
 
 One example of this command is the following:
 ```bash

docs/cli/config.md

@@ -0,0 +1,5 @@
+# `config`
+
+```{click} miniscope_io.cli.config:config
+:prog: mio config
+```

docs/cli/main.rst → docs/cli/index.md

@@ -1,5 +1,10 @@
-CLI Usage
-=========
+# CLI Usage
+
+```{toctree}
+config
+stream
+update
+```
 
 Refer to the following page for details regarding ``stream_daq`` device config files.
 

docs/cli/stream.md

@@ -0,0 +1,5 @@
+# `stream`
+
+```{click} miniscope_io.cli.stream:stream
+:prog: mio stream
+```

docs/cli/update.md

@@ -0,0 +1,5 @@
+# `update`
+
+```{click} miniscope_io.cli.update:update
+:prog: mio update
+```

docs/conf.py

@@ -28,9 +28,11 @@
     "sphinx.ext.napoleon",
     "sphinx.ext.autodoc",
     "sphinxcontrib.autodoc_pydantic",
+    "sphinxcontrib.programoutput",
     "sphinx.ext.intersphinx",
     "sphinx.ext.todo",
     "sphinx_click",
+    "sphinx_design",
 ]
 
 templates_path = ["_templates"]

docs/guide/config.md

@@ -0,0 +1,157 @@
+# Configuration
+
+```{tip}
+See also the API docs in {mod}`miniscope_io.models.config`
+```
+
+Config in `miniscope-io` uses a combination of pydantic models and
+[`pydantic-settings`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/).
+
+Configuration takes a few forms:
+
+- **Global config:** control over basic operation of `miniscope-io` like logging,
+  location of user directories, plugins, etc.
+- **Device config:** control over the operation of specific devices and miniscopes like
+  firmware versions, ports, capture parameters, etc.
+- **Runtime/experiment config:** control over how a device behaves when it runs, like
+  plotting, data output, etc.
+
+## Global Config
+
+Global config uses the {class}`~miniscope_io.models.config.Config` class
+
+Config values can be set (in order of priority from high to low, where higher
+priorities override lower priorities)
+
+* in the arguments passed to the class constructor (not user configurable)
+* in environment variables like `export MINISCOPE_IO_LOG_DIR=~/`
+* in a `.env` file in the working directory
+* in a `mio_config.yaml` file in the working directory
+* in the `tool.miniscope_io.config` table in a `pyproject.toml` file in the working directory
+* in a user `mio_config.yaml` file in the user directory (see [below](user-directory))
+* in the global `mio_config.yaml` file in the platform-specific data directory
+  (use `mio config global path` to find its location)
+* the default values in the {class}`~miniscope_io.models.config.Config` model
+
+Parent directories are _not_ checked - `.env` files, `mio_config.yaml`, and `pyproject.toml`
+files need to be in the current working directory to be discovered.
+
+You can see your current configuration with `mio config`
+
+(user-directory)=
+### User Directory
+
+The configuration system allows project-specific configs per-directory with
+`mio_config.yaml` files in the working directory, as well as global configuration
+via `mio_config.yaml` in the system-specific config directory 
+(via [platformdirs](https://pypi.org/project/platformdirs/)). 
+By default, `miniscope-io` does not create new directories in the user's home directory
+to be polite, but the site config directory might be inconvenient or hard to reach,
+so it's possible to create a user directory in a custom location.
+
+`miniscope_io` discovers this directory from the `user_dir` setting from 
+any of the available sources, though the global `mio_config.yaml` file is the most reliable.
+
+To create a user directory, use the `mio config user create` command.
+(ignore the `--dry-run` flag, which are just used to avoid
+overwriting configs while rendering the docs ;)
+
+```{command-output} mio config user create ~/my_new_directory --dry-run
+```   
+
+You can confirm that this will be where miniscope_io discovers the user directory like
+
+```{command-output} mio config user path
+```
+
+If a directory is not supplied, the default `~/.config/miniscope_io` is used:
+
+```{command-output} mio config user create --dry-run
+```
+
+### Setting Values
+
+```{todo}
+Implement setting values from CLI.
+
+For now, please edit the configuration files directly.
+```
+
+### Keys
+
+#### Prefix
+
+Keys for environment variables (i.e. set in a shell with e.g. `export` or in a `.env` file)
+are prefixed with `MINISCOPE_IO_` to not shadow other environment variables.
+Keys in `toml` or `yaml` files are not prefixed with `MINISCOPE_IO_` .
+
+#### Nesting
+
+Keys for nested models are separated by a `__` double underscore in `.env`
+files or environment variables (eg. `MINISCOPE_IO_LOGS__LEVEL`)
+
+Keys in `toml` or `yaml` files do not have a dunder separator because
+they can represent the nesting directly (see examples below)
+
+When setting values from the cli, keys for nested models are separated with a `.`.
+
+#### Case
+
+Keys are case-insensitive, i.e. these are equivalent::
+
+    export MINISCOPE_IO_LOGS__LEVEL=INFO
+    export miniscope_io_logs__level=INFO
+
+### Examples
+
+`````{tab-set}
+````{tab-item} mio_config.yaml
+```{code-block} yaml
+user_dir: ~/.config/miniscope_io
+log_dir: ~/.config/miniscope_io/logs
+logs:
+  level_file: INFO
+  level_stream: WARNING
+  file_n: 5
+``` 
+````
+````{tab-item} env vars
+```{code-block} bash
+export MINISCOPE_IO_USER_DIR='~/.config/miniscope_io'
+export MINISCOPE_IO_LOG_DIR='~/config/miniscope_io/logs'
+export MINISCOPE_IO_LOGS__LEVEL_FILE='INFO'
+export MINISCOPE_IO_LOGS__LEVEL_STREAM='WARNING'
+export MINISCOPE_IO_LOGS__FILE_N=5
+```
+````
+````{tab-item} .env file
+```{code-block} python
+MINISCOPE_IO_USER_DIR='~/.config/miniscope_io'
+MINISCOPE_IO_LOG_DIR='~/config/miniscope_io/logs'
+MINISCOPE_IO_LOG__LEVEL_FILE='INFO'
+MINISCOPE_IO_LOG__LEVEL_STREAM='WARNING'
+MINISCOPE_IO_LOG__FILE_N=5
+```
+````
+````{tab-item} pyproject.toml
+```{code-block} toml
+[tool.miniscope_io.config]
+user_dir = "~/.config/miniscope_io"
+
+[tool.linkml.config.log]
+dir = "~/config/miniscope_io/logs"
+level_file = "INFO"
+level_stream = "WARNING"
+file_n = 5
+``` 
+````
+````{tab-item} cli
+TODO
+````
+`````
+
+## Device Configs
+
+```{todo}
+Document device configuration
+```

docs/index.md

@@ -7,7 +7,8 @@ Generic I/O interfaces for miniscopes :)
 :maxdepth: 2
 
 guide/installation
-cli/main
+guide/config
+cli/index
 ```
 
 ```{toctree}

miniscope_io/cli/config.py

@@ -0,0 +1,148 @@
+"""
+CLI commands for configuration
+"""
+
+from pathlib import Path
+
+import click
+import yaml
+
+from miniscope_io.models import config as _config
+from miniscope_io.models.config import set_user_dir
+
+
+@click.group(invoke_without_command=True)
+@click.pass_context
+def config(ctx: click.Context) -> None:
+    """
+    Command group for config
+
+    When run without arguments, displays current config from all sources
+    """
+    if ctx.invoked_subcommand is None:
+        config_str = _config.Config().to_yaml()
+        click.echo(f"miniscope-io configuration:\n-----\n{config_str}")
+
+
+@config.group("global", invoke_without_command=True)
+@click.pass_context
+def global_(ctx: click.Context) -> None:
+    """
+    Command group for global configuration directory
+
+    When run without arguments, displays contents of current global config
+    """
+    if ctx.invoked_subcommand is None:
+
+        with open(_config._global_config_path) as f:
+            config_str = f.read()
+
+        click.echo(f"Global configuration: {str(_config._global_config_path)}\n-----\n{config_str}")
+
+
+@global_.command("path")
+def global_path() -> None:
+    """Location of the global miniscope-io config"""
+    click.echo(str(_config._global_config_path))
+
+
+@config.group(invoke_without_command=True)
+@click.pass_context
+def user(ctx: click.Context) -> None:
+    """
+    Command group for the user config directory
+
+    When invoked without arguments, displays the contents of the current user directory
+    """
+    if ctx.invoked_subcommand is None:
+        config = _config.Config()
+        config_file = list(config.user_dir.glob("mio_config.*"))
+        if len(config_file) == 0:
+            click.echo(
+                f"User directory specified as {str(config.user_dir)} "
+                "but no mio_config.yaml file found"
+            )
+            return
+        else:
+            config_file = config_file[0]
+
+        with open(config_file) as f:
+            config_str = f.read()
+
+        click.echo(f"User configuration: {str(config_file)}\n-----\n{config_str}")
+
+
+@user.command("create")
+@click.argument("user_dir", type=click.Path(), required=False)
+@click.option(
+    "--force/--no-force",
+    default=False,
+    help="Overwrite existing config file if it exists",
+)
+@click.option(
+    "--clean/--dirty",
+    default=False,
+    help="Create a fresh mio_config.yaml file containing only the user_dir. "
+    "Otherwise, by default (--dirty), any other settings from .env, pyproject.toml, etc."
+    "are included in the created user config file.",
+)
+@click.option(
+    "--dry-run/--no-dry-run",
+    default=False,
+    help="Show the config that would be written and where it would go without doing anything",
+)
+def create(
+    user_dir: Path = None, force: bool = False, clean: bool = False, dry_run: bool = False
+) -> None:
+    """
+    Create a user directory,
+    setting it as the default in the global config
+
+    Args:
+        user_dir (Path): Path to the directory to create
+        force (bool): Overwrite existing config file if it exists
+    """
+    if user_dir is None:
+        user_dir = _config._default_userdir
+
+    try:
+        user_dir = Path(user_dir).expanduser().resolve()
+    except RuntimeError:
+        user_dir = Path(user_dir).resolve()
+
+    if user_dir.is_file and user_dir.suffix in (".yaml", ".yml"):
+        config_file = user_dir
+        user_dir = user_dir.parent
+    else:
+        config_file = user_dir / "mio_config.yaml"
+
+    if config_file.exists() and not force and not dry_run:
+        click.echo(f"Config file already exists at {str(config_file)}, use --force to overwrite")
+        return
+
+    if clean:
+        config = {"user_dir": str(user_dir)}
+
+        if not dry_run:
+            with open(config_file, "w") as f:
+                yaml.safe_dump(config, f)
+
+        config_str = yaml.safe_dump(config)
+    else:
+        config = _config.Config(user_dir=user_dir)
+        config_str = config.to_yaml() if dry_run else config.to_yaml(config_file)
+
+    # update global config pointer
+    if not dry_run:
+        set_user_dir(user_dir)
+
+    prefix = "DRY RUN - No files changed\n-----\nWould have created" if dry_run else "Created"
+
+    click.echo(f"{prefix} user config at {str(config_file)}:\n-----\n{config_str}")
+
+
+@user.command("path")
+def user_path() -> None:
+    """Location of the current user config"""
+    path = list(_config.Config().user_dir.glob("mio_config.*"))[0]
+    click.echo(str(path))

miniscope_io/cli/main.py

@@ -4,6 +4,7 @@
 
 import click
 
+from miniscope_io.cli.config import config
 from miniscope_io.cli.stream import stream
 from miniscope_io.cli.update import device, update
 
@@ -21,3 +22,4 @@ def cli(ctx: click.Context) -> None:
 cli.add_command(stream)
 cli.add_command(update)
 cli.add_command(device)
+cli.add_command(config)