Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aggregateWithRandomness performance issue when subscribing to all subnets #7197

Closed
twoeths opened this issue Oct 25, 2024 · 2 comments · Fixed by #7204
Closed

aggregateWithRandomness performance issue when subscribing to all subnets #7197

twoeths opened this issue Oct 25, 2024 · 2 comments · Fixed by #7204
Labels
meta-bug Issues that identify a bug and require a fix.

Comments

@twoeths
Copy link
Contributor

twoeths commented Oct 25, 2024

Describe the bug

It takes up to 3 minutes on holesky for aggregateWithRandomness when subscribing to all subnets

Screenshot 2024-10-25 at 13 55 45

considering 1 epoch = 6.4 minutes, this takes almost 50% of main thread cpu time this cause performance issue for attached validators

cpu profile shows 51.6% for this time

Screenshot 2024-10-25 at 13 59 56

holesky_subscribe_all_subnets_network.cpuprofile.zip
holesky_subscribe_all_subnets_main.cpuprofile.zip

Expected behavior

reduce this time, or consider offload this to native using async/await call
cc @wemeetagain @matthewkeil

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

unstable, v1.22.0
@twoeths twoeths added the meta-bug Issues that identify a bug and require a fix. label Oct 25, 2024
@twoeths
Copy link
Contributor Author

twoeths commented Oct 25, 2024

beacon_attestation messages per slot increased from <3k to ~45k since we turned subscribe_all_subnets flag on

Screenshot 2024-10-25 at 14 15 16

@matthewkeil matthewkeil mentioned this issue Oct 25, 2024
Merged
@ChainSafe ChainSafe deleted a comment Oct 26, 2024
@matthewkeil matthewkeil mentioned this issue Oct 29, 2024
Merged
@matthewkeil
Copy link
Member

Closed by #7204. Might be some more optimizations possible by tuning the number of libuv threads though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
meta-bug Issues that identify a bug and require a fix.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: asyncAggregateWithRandomness ChainSafe/lodestar
2 participants
@twoeths @matthewkeil