[Bug] /metrics endpoint unavailable for client

[vicsn]

🐛 Bug Report

We observed the /metrics endpoint became unavailable for a client when under heavy load.

Steps to Reproduce

We cannot reproduce, but observed the following:

• Start a client on an existing network.
• Start a prometheus instance polling the /metrics endpoint
• Spam heavy transactions and solutions
  • 10,000 proof solutions are submitted per block, only four are accepted
  • Before we also had ~30 TPS.
• After syncing up to block 11223 at average speed of 1.88 seconds per block, the /metrics endpoint became unavailable.

No suspicious logs were noticed around block 11223, logs are available upon request.

From the PrometheusBuilder::install documentation: a new single-threaded Tokio runtime is created on a background thread, and the exporter is spawned there.

Your Environment

SnarkOS version: dde9e176123130355a4686cf8241f111e04094b3
Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-67-generic x86_64), 16 cores
Flags used: snarkos start --client --nocdn --nodisplay --rest <> --node <> --verbosity 1 --metrics --peers <> --rest-rps 100000