Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Register processor queue length as histogram #6012

Merged
merged 2 commits into from
Sep 9, 2024
Merged

Register processor queue length as histogram #6012

merged 2 commits into from
Sep 9, 2024

Conversation

dapplion
Copy link
Collaborator

@dapplion dapplion commented Jun 27, 2024 •
edited
Loading

Issue Addressed

Second attempt at

It would be very useful to know how often queues reach zero or very high capacity values. Today we only have sparse snapshots of the queue length, which can change very rapidly in miliseconds

Proposed Changes

Register the queue length as histogram instead of a gauge.

This PR registers the length metric for a single queue instead of for all queues. Reasons why:

  • We don't want to register the same value over and over. It would make the metric less useful and be quite wasteful
  • Using an enum ensures that we register metrics for all queues. With the current approach it's easy to forget to add metrics for new queues: this has happened when adding lightclient messages and with peerdas.

Metrics breaking changes

All metrics previously used to track queue lengths change. Also the type label for all metrics in the processor changes capitalization from gossip_block to GossipBlock. Is that okay? Does the IntoStaticStr macro support a different capitalization?

This was referenced Jun 27, 2024
Closed
Closed
@dapplion dapplion added the ready-for-review The code is ready for review label Jul 30, 2024
@michaelsproul michaelsproul added the backwards-incompat Backwards-incompatible API change label Sep 5, 2024
@realbigsean realbigsean added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 5, 2024
@michaelsproul michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 6, 2024
michaelsproul
michaelsproul reviewed Sep 6, 2024
View reviewed changes
beacon_node/beacon_processor/src/lib.rs
@@ -987,46 +1008,48 @@ impl<E: EthSpec> BeaconProcessor<E> {
.map_or(false, |event| event.drop_during_sync);

let idle_tx = idle_tx.clone();
match work_event {
let modified_queue_id = match work_event {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this variable called modified queue ID? it never gets modified

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refers to what queue has been mutated by a pop operation. Tells downstream code "the queue associated with this queue ID has been mutated, so register its length"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah I get you, makes sense

let's do it

michaelsproul
michaelsproul approved these changes Sep 6, 2024
View reviewed changes
Copy link
Member

@michaelsproul michaelsproul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo my query about variable naming

I would like to merge this ASAP so it doesn't conflict with:

@michaelsproul michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 9, 2024
@michaelsproul
Copy link
Member

@mergify queue

@mergify Mergify
Copy link

mergify bot commented Sep 9, 2024 •
edited
Loading

queue

🛑 The pull request has been removed from the queue default

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

mergify bot added a commit that referenced this pull request Sep 9, 2024
@mergify
Merge of #6012
c9bd5b0
@mergify mergify bot mentioned this pull request Sep 9, 2024
Closed
5 tasks
@jimmygchen
Copy link
Member

@mergify requeue

@mergify Mergify
Copy link

mergify bot commented Sep 9, 2024

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

@mergify Mergify
Copy link

mergify bot commented Sep 9, 2024 •
edited
Loading

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 51091a4

mergify bot added a commit that referenced this pull request Sep 9, 2024
@mergify
Merge of #6012
2b5ab17
This was referenced Sep 9, 2024
Closed
Merged
mergify bot added a commit that referenced this pull request Sep 9, 2024
@mergify
Merge of #6012
37c4eed
@mergify mergify bot mentioned this pull request Sep 9, 2024
Closed
5 tasks
@mergify mergify bot merged commit 51091a4 into sigp:unstable Sep 9, 2024
27 of 28 checks passed
@dapplion dapplion deleted the processor-queue-histogram branch September 10, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaelsproul michaelsproul michaelsproul approved these changes

@realbigsean realbigsean Awaiting requested review from realbigsean

Assignees
No one assigned
Labels
backwards-incompat Backwards-incompatible API change ready-for-merge This PR is ready to merge.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dapplion @michaelsproul @jimmygchen @realbigsean