Rate Limiting for RPC Service / EVM and Metrics (backoff periods) #2270

dirvine · 2024-10-29T15:02:51Z

Rate Limiting for RPC Service

Summary

This PR introduces rate limiting to the SafeNode RPC service to prevent potential abuse and ensure service stability. It implements both global and per-IP rate limiting using the governor crate.

Changes

Added RateLimiter struct with:
- Global limit of 100 requests per minute across all clients
- Per-IP limit of 20 requests per minute
Integrated rate limiting checks into RPC endpoints
Added proper error handling for rate limit exceeded cases
Cleaned up unused metrics buffer implementation
Fixed replication manager to use proper error types and concurrency control

Technical Details

Uses governor crate for efficient rate limiting
Implements thread-safe rate limiting using Tokio's Mutex
Memory-efficient implementation that creates per-IP limiters only when needed
Proper error propagation using gRPC status codes

Testing

Test rate limiting with concurrent requests
Verify rate limit error messages
Test per-IP limits with multiple clients
Test global rate limit enforcement

Related Issues

Closes #XXXX (CPU usage issues)

Future Improvements

Make rate limits configurable via node settings
Add metrics for rate limiting events
Consider implementing more sophisticated rate limiting strategies (e.g., token bucket)

feat: print for peers connected and net size in loud mode

feat(autonomi): implement client events

BREAKING CHANGE: services configured by the node manager are no longer compatible with networks that are not EVM. The node manager `add` command now supports the following EVM arguments: * The subcommand for selecting the network type is appended. * The `evm-custom` subcommand makes the `--data-payments-address`, `--payment-token-address` and `--rpc-url` arguments available. * The `--rewards-address` argument is provided. The `CustomNetwork` type was modified to enable it to be serialized in the node manager registry.

…local feat: adapt node manager `add` command for evm network

…with_network_keys feat(node): using different storage dir for different network

The `autonomi_cli` binary is renamed to `autonomi` because the `_cli` suffix is not necessary. The crate is named `autonomi-cli` rather than `autonomi_cli` because we agreed we prefer underscores.

…_cli2 chore: rename autonomi crate and binary

uses node control flow to exit a node if we consistently detect high CPU

…t arg

… range replication

merge the results after we see split record. Use Quorum::Majority at least (instead of Quorum::One)

this indirectly affects how many close nodes returned and could stymie the GetRange effectiveness

ie, even if query times out if we have peers, we update seems like we timeout more due to high K_VALUE

feat(kad): reduce bootstrap interval

feat(autonomi): allow the uploader to work with the new apis

fix(launchpad): update status after resetting

fix(launchpad): rewards address req to start nodes

This is a utility for the user's reference. The owner name field is removed from the output. It doesn't serve much purpose without the Discord link.

fix(launchpad): throbber stops after a while

feat: print rewards address in `status --details`

chore(network): logging multiple fetched versions

Revert "feat(autonomi): allow the uploader to work with the new apis"

* Implements per-IP and global rate limiting for RPC endpoints * Adds resource exhaustion error handling * Removes unused metrics buffer fields * Fixes replication manager concurrency control

grumbach and others added 30 commits October 15, 2024 08:45

Merge pull request maidsafe#2240 from grumbach/loud_2

4648e2c

feat: print for peers connected and net size in loud mode

Merge pull request maidsafe#2241 from RolandSherwin/client_events

2a91e86

feat(autonomi): implement client events

Merge branch 'main' into DisconnectBad

fb17d79

feat(node): using different storage dir for different network

069060b

feat(autonomi): archive API in wasm

33cb6e8

Merge pull request maidsafe#2235 from jacderida/feat-evm_testnet_non_…

00ad51d

…local feat: adapt node manager `add` command for evm network

feat(autonomi): vault API calls

7fce069

refactor(autonomi): wasm refactor; vault test

91bf045

fix(autonomi): feature gate vault api

2e32497

Merge pull request maidsafe#2243 from maqi/record_store_dir_prefixed_…

147ba8e

…with_network_keys feat(node): using different storage dir for different network

chore: rename autonomi crate and binary

c338819

The `autonomi_cli` binary is renamed to `autonomi` because the `_cli` suffix is not necessary. The crate is named `autonomi-cli` rather than `autonomi_cli` because we agreed we prefer underscores.

Merge pull request maidsafe#2246 from jacderida/chore-rename_autonomi…

567200e

…_cli2 chore: rename autonomi crate and binary

feat(node): kill nodes running on host with little spare CPU

26c5eee

uses node control flow to exit a node if we consistently detect high CPU

feat(networking): initial range based gets implementation

6a0e16e

chore(networking): rename sort_peers fns to clarify the accepted limi…

2499b46

…t arg

chore(networking): remove REPLICATION_PEER_COUNT

adae8c5

fix(networking): reset raw spend get to be Majority

a615f8a

fix(networking): return true for exceeded quorum for non sensitive data

e9d77ee

fix(networking): use cloest_node_buffer_zone when selecting nodes for…

548ba92

… range replication

fix(networking): use libp2p for register retrieval

184d8bc

merge the results after we see split record. Use Quorum::Majority at least (instead of Quorum::One)

chore(protocol): move get_type_from_record to protocol

f7ceb5d

fix(networking): remove self from get_peers_excluding_self

39de958

chore: small refactors and doc tweaks

636b84f

fix(networking): remove custom REPLICATION_FACTOR

4a9fa8f

this indirectly affects how many close nodes returned and could stymie the GetRange effectiveness

ci: normalize send amount for tests

834923f

ci: fixes

05d97b8

fix(client): retry faucet loading if genesis is spent

b85d51b

fix(faucet): loop if initial faucet balance 0 but genesis spent

862cf9f

fix(networking): ensure we update getrange when we can

00b06ec

ie, even if query times out if we have peers, we update seems like we timeout more due to high K_VALUE

RolandSherwin and others added 30 commits October 25, 2024 14:19

test(autonomi): add tests for uploader

985034a

feat(uploader): error out on io error

5e5aa3f

feat(uploader): upload chunk and registers with correct cfg

a4f3da9

feat(autonomi): use the uploader for register and chunk uploads

c320816

feat: provide target arch based mpsc channel

f4cd35c

feat(uploader): feature gate registers

78c526a

feat(ci): enable autonomi unit tests

de22bb7

feat(uploader): feature gate data

e7ee280

chore(uploader): add docs and small fixes

40390bc

chore: update based on comments

a46c666

Merge pull request #2317 from joshuef/ReduceKadBoot

d0d368c

feat(kad): reduce bootstrap interval

chore(network): logging multiple fetched multiple versions

19b8471

Merge pull request maidsafe#2258 from RolandSherwin/optimize_uploads

b4010bb

feat(autonomi): allow the uploader to work with the new apis

fix(launchpad): update status after resetting

9a243fd

Merge branch 'main' into fix_update_after_reset

7ef21a3

Merge pull request #2318 from mazzi/fix_update_after_reset

c61de4a

fix(launchpad): update status after resetting

fix(launchpad): rewards address req to start nodes

793f584

Merge branch 'main' into fix_rewards_address_required

32b36aa

Merge branch 'main' into fix_rewards_address_required

7ea058b

Merge pull request #2321 from mazzi/fix_rewards_address_required

4344055

fix(launchpad): rewards address req to start nodes

feat: print rewards address in status --details

4f02f66

This is a utility for the user's reference. The owner name field is removed from the output. It doesn't serve much purpose without the Discord link.

fix(launchpad): throbbler stops after a while

dc243ad

Merge branch 'main' into fix_throbbler_stops

4919059

Merge pull request #2326 from mazzi/fix_throbbler_stops

f996df0

fix(launchpad): throbber stops after a while

Merge pull request #2325 from jacderida/feat-print_rewards_address

f6e6023

feat: print rewards address in `status --details`

Merge branch 'stable'

4751eed

Merge pull request #2319 from maqi/split_debug

cefd686

chore(network): logging multiple fetched versions

Revert "feat(autonomi): allow the uploader to work with the new apis"

f37b4d3

Merge pull request #2334 from maidsafe/revert-2258-optimize_uploads

31721be

Revert "feat(autonomi): allow the uploader to work with the new apis"

feat(node): add rate limiting to RPC service

f826336

* Implements per-IP and global rate limiting for RPC endpoints * Adds resource exhaustion error handling * Removes unused metrics buffer fields * Fixes replication manager concurrency control

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rate Limiting for RPC Service / EVM and Metrics (backoff periods) #2270

Rate Limiting for RPC Service / EVM and Metrics (backoff periods) #2270

dirvine commented Oct 29, 2024

Rate Limiting for RPC Service / EVM and Metrics (backoff periods) #2270

Are you sure you want to change the base?

Rate Limiting for RPC Service / EVM and Metrics (backoff periods) #2270

Conversation

dirvine commented Oct 29, 2024

Rate Limiting for RPC Service

Summary

Changes

Technical Details

Testing

Related Issues

Future Improvements