feat(sidecar): support large staking operations

[merklefruit]

Some professional node operators rely on custom setups that look quite different from solo stakers. Some of the differences are:

• validator clients connected to N EL/CL pairs, non-sticky (i.e. these can rotate as need be, and as servers go down)
• rotating validator keys between different validator clients and EL/CL pairs, non-sticky
• stateless MEV-boost MUXers used interchangeably depending on availability (or custom criteria like specific validator pubkeys)

Bolt works under the assumption that commitments rely on a single source of truth, which in our case is the bolt-sidecar.
-> It's not possible to run 2 bolt-sidecars with the same validator key as that could lead to safety faults.

This does constrain the flexibility of some of the setups described above, where for instance MUXers are stateless (e.g. get_header and get_payload calls could be made from different machines). This doesn't work for us since we need to be able to ship local blocks to beacon clients in case of locally built fallback blocks.

-> On first analysis, it looks to me like the "stickiness" of the bolt-sidecar<->beacon<->validator_key link is limited to 1 epoch.

Another point is allowing bolt-sidecar to use fallback EL/CL URLs as to have built-in redundancy for these types of setups.
This could be added as an optional flag or env variable.

[thedevbirb]

Thanks for these notes, really interesting!
Along this reasoning, I've noticed that the constraint submission logic + proof verification on MEV-Boost makes it lose it's statelessness property: the get_header call that we proxy into a get_header_with_proofs is stateful, because of the constraints that are stored in local memory and sent by the bolt sidecar.
As you suggested for scaling the bolt sidecar horizontally, a good solution might be saving those constraints in a Redis cluster inside MEV-Boost.