Implement the ability to more logically share level hierarchies

[evanphx]

This implements the SyncParentLevel flag, which alters how Levels are propagated amongst sub-loggers. It's mutually exclusive with IndependentLevels.

When set, sub-loggers will take on the level of a direct parent/grandparent/etc and be unaffected by sibling sub-loggers changing their level.

This allows for more obvious hierarchies of sub-loggers to be created and used that share a common level.

[banks]

🤔 I find this behaviour surprising.

My expectation would be that if some sub-logger (in this case b) has had its level specifically overridden (e.g. by an operator enabling more detailed level on a sub-component), than a subsequent change to the global level should not override the more specific, explicitly set level. It should cascade to all sub-loggers that haven't had an explicitly set level though.

I'm thinking of a UX where a peice of software allows operators to specify a targetted log level e.g via ENV var. Something like FOO_SUBCOMPONENT_LOG=TRACE while investigating an issue.

If they subsequently choose to reconfigure the application e.g. via config file to global log at DEBUG instead of INFO I'd expect that to apply to everything except the component that has already been overridden.

In this example, unlike the comment here, the override is for more detail. There could be an argument that you might want different behaviour depending on whether the override is for more or less detail, but I feel like it's less surprising in all cases to continue to honour more specific sub-logger overrides when changing the parent/global level.

Is there a use-case I'm not seeing where this behavior seems better?

[mpalmi]

Evan and I chatted about this briefly and we contemplated the principle of least surprise. I don't feel strongly either way, but I think there are a couple of things to consider for Vault's implementation -- specifically around the distinction between /sys/loggers and your environment variable example.

In my mind, our implementation is slightly different from the environment variable case, since logger level state (at least for Vault) is entirely in program memory. With environment variables, you have state outside the scope of the program that would be honored on restart; whereas in Vault, that's not the case. I thought it would be less surprising to keep reload and restart behavior consistent and advertise the fact that /sys/loggers is really for transient maintenance tasks.

The flip side of that is the maintenance tasks might include reload, at which point this UX could become burdensome (edit: This may not actually be true. I think that log level is only sync'ed on reload if an operator changes the server's HCL, so someone would have to intentionally reset all levels.)

Like I said, I don't feel too strongly either way, but just some thoughts to consider.

[evanphx]

Funny you ask this, I asked @mpalmi the same question. Effectively the idea here is that parent logger changes win over any previous decisions the sub-loggers made. @mpalmi thoughts on this? I think this is probably the biggest behavior that we need to sort out.

[evanphx]

The idea put forth was that if, at runtime, a decision to put a hierarchy into TRACE mode is requested, everything in the hierarchy should get that change. If a specific component in hierarchy had been set to ERROR at program start, the desire to set the whole hierachry into TRACE would need to override ERROR.

[evanphx]

@banks In your specific example, the user would have to be careful to configure the toplevel first, then components down the hierarchy second. In that way, they could achieve the same results.

[banks]

I don't feel super strongly. At least to me I feel like I'd expect more specific override to take precedence and yes - for it's whole subtree. I've not thought through the actual use-cases or UX of vault /sys/loggers though.

But I think if I'd bothered to make an API call to set a logging level on a specific component, I'd not expect a general change of the logging level via some other mechanism to override that. I don't think I'd lose sleep over it, but it at least wouldn't be my expectation 🤷 .

If you thought this through Mike and this is the preferred UX then I'm fine as long as we document that clearly (both here and in our product docs etc.)

[banks]

forever detached from changes by a?

Actually that's a bit more nuanced. I guess what I'd expect would be that if the component specific override for b is removed, then b and it's whole subtree are "re-attached" to the parent and start to respect that setting again.

I have no idea if anyone cares enough to warrant implementing that though! It would probably need an API change too so that it was possible to set a sub-logger back to "follow parent" level rather than some specific level...

[evanphx]

Right, the ability to specify "hey, this is an override" and then "hey, reset this back to not an override" isn't available (and it would be probably a bit confusing).

[kisunji]

I put in a vote for "parent always changing level for children is less surprising".

It's possible to achieve the detached logging levels with IndependentLevels and some use of state + sublogger hooks (I have a related poc in Consul: hashicorp/consul#16665)

[mpalmi]

I agree, the current approach seems to be the least surprising -- especially if overrides cannot be reset.

[mpalmi]

Looks good to me! Thanks for getting to this @evanphx!