Set configuration options via environment variables

[DominicOram]

As a user trying to use the client with stomp it is frustrating that I have to provide the config file every time I call the CLI. It would be better if I could specify this as an environment variable so that I do not need to keep specifying it. We could do either:

• export BLUEAPI.config.file=config_file - this is the same sort of file we would currently use
• export BLUEAPI.config.api.host=my_host - we can then split out all the config fields

[stan-dot]

@DominicOram what is your preference?

what if we have ~/.blueapi/config.yaml as a default location? it feels unix-like

[stan-dot]

@callumforrester what was the reasoning to require it always in the first place?

[DominicOram]

No preference, as long as it's well documented I'm not too fussed

[callumforrester]

@stan-dot the configuration file is not always required per-se, but only when you want configuration that's different to the defaults. The intention was always to provide other ways of overriding configuration than a file, which is a bit clunky for a single value. That is what this ticket is for.

For that reason, I have a preference for the second of @DominicOram's options. It's relatively common to only want to override one value, for example export BLUEAPI.config.stomp.host=localhost would automatically turn on the message bus for local dev and make #615 easier. It seems like overkill to need a whole file for this. Let alone to need to know a default location where you need to put it to make the magic work. N.b. this is where I struggle with databroker sometimes (see docs), it's nice to be able to do something simple just from the terminal.

Eventually I would like to see blueapi use the same configuration override order that @tpoliaw has introduced to GDA:

1. Default values, unless overridden by...
2. File, unless overridden by...
3. Environment variables, unless overridden by...
4. Values passed in via a CLI argument

[callumforrester]

So the long and short of it.

Acceptance Criteria

1. If I run export BLUEAPI.config.<some value>, the default configuration for that value (found in blueapi/src/config.py is overridden by <some value>
2. If I run the above and the value is already specified in the configuration file, the environment variable still overrides
3. The docs explain this
4. Tests cover this
5. The temporary parsing of environment variables for just the message bus password (see init container cannot be run when mounting configuration values from envVars without also mounting those envVars to the init container #620) is removed, this mechanism would replace it

[callumforrester]

This issue also relates to/could be done at the same time as #532

[stan-dot]

Eventually I would like to see blueapi use the same configuration override order that @tpoliaw has introduced to GDA:
Default values, unless overridden by...
File, unless overridden by...
Environment variables, unless overridden by...
Values passed in via a CLI argument

What is the rationale for this? it's more functionality exposed, and users might come to use all 4 in various combinations hard to debug.

the Zen of Python says:

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

passing a file path is bulky, a long override in the cli also feels bulky

Let alone to need to know a default location where you need to put it to make the magic work.

it's literally how most unix tools keep config, in the home directory in a hidden file, or in .config. It is a common pattern.

you say that a 'whole file is overkill', implying a spectrum of 'sizes' from a cli argument to file, where an etcd cache could be postulated as an extrapolation. I don't think the size is the relevant criterion here, but what is easy to work with.

Files are easier to debug imo than env variables. One potential difference might be if the config has any secret fields, then in deployment some extra setup would be needed.

[callumforrester]

passing a file path is bulky, a long override in the cli also feels bulky

Indeed, but the following feels like the least bulky solution:

export BLUEAPI.config.stomp.host=localhost
blueapi serve

You could easily just have this in a quickstart tutorial in the docs and it would take someone 5 seconds, as opposed to:

1. Create a file in /some/path
2. Add the following contents:

stomp:
  host: localhost

3. Run blueapi serve
4. Don't forget you might be in a devcontainer...

[stan-dot]

this could all be addressed, but I concede for time savings now

[stan-dot]

@callumforrester should the issue just be to do the whole override system with 4 layers?

[callumforrester]

No, just environment variables (see acceptance criteria above)

[stan-dot]

this issue is purely UX/DX, we could make it full from the first time.