refactor(config): reduce config load and increase priority

[vvanglro]

Summary

#2440

Checklist

• I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.

[vvanglro]

All right, 😑 that doesn't work because some objects in fastapi can't be serialized, so they have to be loaded after the process starts. fastapi/fastapi#9078

Although the tests passed, they couldn't really start up.

[Kludex]

It's not only on FastAPI.