Add per-operation block processing metrics

[realbigsean]

Description

We have metrics for block processing but it would be good to add some granularity as we continue to add more operations to it. Relevant comment here:

#5741 (comment)

[sergerad]

@realbigsean could you please elaborate on your comment about per-operation metrics?

Are you suggesting a new metric within pub fn get_pending_balance_to_withdraw() ? A timer metric?

Thanks

[realbigsean]

Sorry for the delayed response @sergerad !

We have this method process_operations. Within it, you can see a bunch of methods all prefixed process_*. So I was thinking we add timer metrics for each of these process_* methods. There was a performance concerned related to get_pending_balance_to_withdraw which lies in process_execution_layer_withdrawal_requests. So adding timer metrics for process_* methods would also give us an idea of whether that is actually problematic in practice.

[dapplion]

Another option is do add benches, as the time per op should be predictable given the block input. Then if we see that the overall block processing time metric spikes for block X, we can run the local benches against block X and optimize