Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring is causing CPU spikes every 5 minutes #393

Open
zer777 opened this issue Oct 1, 2023 · 3 comments
Open

Monitoring is causing CPU spikes every 5 minutes #393

zer777 opened this issue Oct 1, 2023 · 3 comments

Comments

@zer777
Copy link

zer777 commented Oct 1, 2023

I experienced CPU spikes on testnet in July which were then fixed in Smartnode v1.9.8. I am now running Smartnode v1.10.2 on mainnet and experiencing almost the same CPU spikes every 5 minutes. When I disable monitoring, I do not have any CPU spikes anymore. Could we have a similar bug like the one that was fixed in 1.9.8?
The CPU spikes seem to correlate with eth1 RPC calls.

Another user reported CPU spikes every 5 minutes, but thought to be normal for block validation. I do not think that's correct, because disabling monitoring is resolving this. So I do not seem to be the only one experiencing this.

@Darkmessage
Copy link
Contributor

I was the one thinking it was validator duties. Because I never saw it differently. If this behaviour was "fixed" in 1.9.8, it never did anything for my Lighthouse + Besu combination having monitoring and Beaconcha.in monitoring on.

Screenshot_20230930-223658566 (1)

I tested without monitoring and the graph is a bit less noisy. However, I don't feel like it makes that much of a difference but it's hard to tell with s-tui command line graphs.

That's why I still believe this is normal. If it's not, then maybe it has to do with my low base frequency of 1.1 GHz. Basically, whenever there's just a little work, it's a stress which is why there's a spike roughly whenever I need to validate.

@Darkmessage
Copy link
Contributor

As per Fu Manshu on Discord:

Fyi i'm running on Besu and also noticed these CPU spikes every 5 minutes. You can solve the problem with "docker stop rocketpool_node" (apparently there is a bug in the current version of Besu RPC) and keep the monitoring on. See https://discord.com/channels/405159462932971535/405163713063288832/1125261530515132489 for reference.

Can confirm it's that for me, so I would suggest to close this issue.

@theslipp
Copy link

theslipp commented Nov 6, 2023

Seeing the same thing with Besu/Nimbus.

Stopped the spikes by following the below steps as suggested back in July.

  • nano ~/.rocketpool/user-settings.yml
  • Within root section change enableMetrics to "false"
  • Then do docker start rocketpool_node

Above changes can bee seen having been implemented at the 1930 mark. Between 1920 to 1930 the rocketpool_node docker container had been stopped.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants