Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(beacon-node): network worker event latency metrics #5800

Merged
merged 42 commits into from
Oct 12, 2023
Merged

feat(beacon-node): network worker event latency metrics #5800

merged 42 commits into from
Oct 12, 2023

Conversation

matthewkeil
Copy link
Member

@matthewkeil matthewkeil commented Jul 25, 2023 •
edited
Loading

NOTE panel updates moved to PR#5827

Motivation

Add metrics for network event latency and message port latency to network worker. Similar to how we collect data for latency across bls worker message port. Also captures latency from initial emit of network event until re-emit of event on other thread event bus.

Description

Add additional metrics to help resolve #5604

@matthewkeil matthewkeil requested a review from a team as a code owner July 25, 2023 10:41
dapplion
dapplion suggested changes Jul 25, 2023
View reviewed changes
packages/beacon-node/src/network/events.ts Outdated Show resolved Hide resolved
packages/beacon-node/src/network/gossip/gossipsub.ts Outdated Show resolved Hide resolved
@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: 343c7a4 Previous: 1aa6561 Ratio
Object set x1000 24.663 ns/op 80.276 ns/op 0.31
Map set x1000 15.940 ns/op 57.346 ns/op 0.28
forkChoice updateHead vc 600000 bc 64 eq 300000 11.877 ms/op 63.321 ms/op 0.19
phase0 processBlock - 250000 vs - 7PWei normalcase 1.5242 ms/op 5.5778 ms/op 0.27
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.8950 us/op 30.016 us/op 0.26
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 31.267 us/op 118.90 us/op 0.26
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 9.4120 us/op 32.712 us/op 0.29
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.8650 us/op 33.810 us/op 0.26
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 873.02 us/op 2.6884 ms/op 0.32
Full benchmark results
Benchmark suite Current: 343c7a4 Previous: 1aa6561 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 848.05 us/op 714.36 us/op 1.19
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 50.351 us/op 88.192 us/op 0.57
BLS verify - blst-native 1.1119 ms/op 1.3477 ms/op 0.83
BLS verifyMultipleSignatures 3 - blst-native 2.4005 ms/op 2.8464 ms/op 0.84
BLS verifyMultipleSignatures 8 - blst-native 5.2009 ms/op 6.2056 ms/op 0.84
BLS verifyMultipleSignatures 32 - blst-native 19.008 ms/op 23.126 ms/op 0.82
BLS verifyMultipleSignatures 64 - blst-native 37.258 ms/op 45.174 ms/op 0.82
BLS verifyMultipleSignatures 128 - blst-native 73.907 ms/op 89.280 ms/op 0.83
BLS deserializing 10000 signatures 768.84 ms/op 913.10 ms/op 0.84
BLS deserializing 100000 signatures 7.7188 s/op 9.3535 s/op 0.83
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.1564 ms/op 1.3807 ms/op 0.84
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.2592 ms/op 1.5972 ms/op 0.79
BLS verifyMultipleSignatures - same message - 32 - blst-native 1.9791 ms/op 2.7197 ms/op 0.73
BLS verifyMultipleSignatures - same message - 64 - blst-native 2.9416 ms/op 4.5061 ms/op 0.65
BLS verifyMultipleSignatures - same message - 128 - blst-native 4.8591 ms/op 6.8206 ms/op 0.71
BLS aggregatePubkeys 32 - blst-native 22.341 us/op 27.031 us/op 0.83
BLS aggregatePubkeys 128 - blst-native 87.074 us/op 104.21 us/op 0.84
getAttestationsForBlock 27.124 ms/op 56.920 ms/op 0.48
isKnown best case - 1 super set check 290.00 ns/op 469.00 ns/op 0.62
isKnown normal case - 2 super set checks 288.00 ns/op 459.00 ns/op 0.63
isKnown worse case - 16 super set checks 287.00 ns/op 537.00 ns/op 0.53
CheckpointStateCache - add get delete 3.3830 us/op 6.3910 us/op 0.53
validate api signedAggregateAndProof - struct 2.3086 ms/op 3.0603 ms/op 0.75
validate gossip signedAggregateAndProof - struct 2.3590 ms/op 2.9941 ms/op 0.79
validate gossip attestation - vc 640000 1.1211 ms/op 1.5056 ms/op 0.74
batch validate gossip attestation - vc 640000 - chunk 32 130.97 us/op 179.83 us/op 0.73
batch validate gossip attestation - vc 640000 - chunk 64 116.75 us/op 175.27 us/op 0.67
batch validate gossip attestation - vc 640000 - chunk 128 108.34 us/op 176.23 us/op 0.61
batch validate gossip attestation - vc 640000 - chunk 256 103.30 us/op 153.22 us/op 0.67
pickEth1Vote - no votes 881.96 us/op 1.4832 ms/op 0.59
pickEth1Vote - max votes 8.1162 ms/op 14.304 ms/op 0.57
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.298 ms/op 21.561 ms/op 0.66
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 25.583 ms/op 34.248 ms/op 0.75
pickEth1Vote - Eth1Data fastSerialize value x2048 359.10 us/op 656.42 us/op 0.55
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.5780 ms/op 6.6833 ms/op 1.13
bytes32 toHexString 385.00 ns/op 707.00 ns/op 0.54
bytes32 Buffer.toString(hex) 273.00 ns/op 321.00 ns/op 0.85
bytes32 Buffer.toString(hex) from Uint8Array 387.00 ns/op 578.00 ns/op 0.67
bytes32 Buffer.toString(hex) + 0x 275.00 ns/op 349.00 ns/op 0.79
Object access 1 prop 0.17400 ns/op 0.20600 ns/op 0.84
Map access 1 prop 0.17600 ns/op 0.16600 ns/op 1.06
Object get x1000 4.5980 ns/op 8.8510 ns/op 0.52
Map get x1000 0.47600 ns/op 0.85800 ns/op 0.55
Object set x1000 24.663 ns/op 80.276 ns/op 0.31
Map set x1000 15.940 ns/op 57.346 ns/op 0.28
Return object 10000 times 0.20520 ns/op 0.34780 ns/op 0.59
Throw Error 10000 times 2.5214 us/op 4.7681 us/op 0.53
fastMsgIdFn sha256 / 200 bytes 1.8480 us/op 3.9220 us/op 0.47
fastMsgIdFn h32 xxhash / 200 bytes 261.00 ns/op 355.00 ns/op 0.74
fastMsgIdFn h64 xxhash / 200 bytes 312.00 ns/op 396.00 ns/op 0.79
fastMsgIdFn sha256 / 1000 bytes 5.7620 us/op 12.690 us/op 0.45
fastMsgIdFn h32 xxhash / 1000 bytes 376.00 ns/op 497.00 ns/op 0.76
fastMsgIdFn h64 xxhash / 1000 bytes 374.00 ns/op 472.00 ns/op 0.79
fastMsgIdFn sha256 / 10000 bytes 49.380 us/op 114.88 us/op 0.43
fastMsgIdFn h32 xxhash / 10000 bytes 1.6530 us/op 2.0720 us/op 0.80
fastMsgIdFn h64 xxhash / 10000 bytes 1.1340 us/op 1.4680 us/op 0.77
send data - 1000 256B messages 10.786 ms/op 24.115 ms/op 0.45
send data - 1000 512B messages 14.326 ms/op 34.112 ms/op 0.42
send data - 1000 1024B messages 21.710 ms/op 45.501 ms/op 0.48
send data - 1000 1200B messages 20.954 ms/op 34.828 ms/op 0.60
send data - 1000 2048B messages 22.048 ms/op 41.150 ms/op 0.54
send data - 1000 4096B messages 22.542 ms/op 34.417 ms/op 0.65
send data - 1000 16384B messages 49.277 ms/op 95.662 ms/op 0.52
send data - 1000 65536B messages 221.37 ms/op 500.19 ms/op 0.44
enrSubnets - fastDeserialize 64 bits 859.00 ns/op 1.7210 us/op 0.50
enrSubnets - ssz BitVector 64 bits 404.00 ns/op 568.00 ns/op 0.71
enrSubnets - fastDeserialize 4 bits 183.00 ns/op 239.00 ns/op 0.77
enrSubnets - ssz BitVector 4 bits 400.00 ns/op 584.00 ns/op 0.68
prioritizePeers score -10:0 att 32-0.1 sync 2-0 63.715 us/op 128.25 us/op 0.50
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 72.631 us/op 183.02 us/op 0.40
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 101.71 us/op 253.99 us/op 0.40
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 172.80 us/op 401.58 us/op 0.43
prioritizePeers score 0:0 att 64-1 sync 4-1 200.75 us/op 454.68 us/op 0.44
array of 16000 items push then shift 1.2611 us/op 1.7843 us/op 0.71
LinkedList of 16000 items push then shift 5.7840 ns/op 12.945 ns/op 0.45
array of 16000 items push then pop 53.351 ns/op 121.21 ns/op 0.44
LinkedList of 16000 items push then pop 5.5610 ns/op 10.290 ns/op 0.54
array of 24000 items push then shift 1.8567 us/op 2.7081 us/op 0.69
LinkedList of 24000 items push then shift 5.7400 ns/op 11.959 ns/op 0.48
array of 24000 items push then pop 72.896 ns/op 171.17 ns/op 0.43
LinkedList of 24000 items push then pop 5.5370 ns/op 10.182 ns/op 0.54
intersect bitArray bitLen 8 5.2420 ns/op 7.1830 ns/op 0.73
intersect array and set length 8 40.305 ns/op 65.670 ns/op 0.61
intersect bitArray bitLen 128 25.299 ns/op 34.785 ns/op 0.73
intersect array and set length 128 583.28 ns/op 938.38 ns/op 0.62
bitArray.getTrueBitIndexes() bitLen 128 1.1610 us/op 1.8060 us/op 0.64
bitArray.getTrueBitIndexes() bitLen 248 1.7960 us/op 3.0090 us/op 0.60
bitArray.getTrueBitIndexes() bitLen 512 3.2730 us/op 6.6920 us/op 0.49
Buffer.concat 32 items 863.00 ns/op 1.1040 us/op 0.78
Uint8Array.set 32 items 1.7370 us/op 2.1800 us/op 0.80
Set add up to 64 items then delete first 1.7048 us/op 4.9253 us/op 0.35
OrderedSet add up to 64 items then delete first 2.5840 us/op 6.3959 us/op 0.40
Set add up to 64 items then delete last 1.9359 us/op 5.6163 us/op 0.34
OrderedSet add up to 64 items then delete last 2.8544 us/op 7.5228 us/op 0.38
Set add up to 64 items then delete middle 1.9255 us/op 5.3409 us/op 0.36
OrderedSet add up to 64 items then delete middle 4.1326 us/op 8.3779 us/op 0.49
Set add up to 128 items then delete first 3.8631 us/op 10.697 us/op 0.36
OrderedSet add up to 128 items then delete first 5.9942 us/op 14.302 us/op 0.42
Set add up to 128 items then delete last 3.6832 us/op 10.375 us/op 0.35
OrderedSet add up to 128 items then delete last 5.5155 us/op 13.961 us/op 0.40
Set add up to 128 items then delete middle 3.7116 us/op 10.701 us/op 0.35
OrderedSet add up to 128 items then delete middle 10.340 us/op 19.100 us/op 0.54
Set add up to 256 items then delete first 7.4982 us/op 20.721 us/op 0.36
OrderedSet add up to 256 items then delete first 11.874 us/op 29.602 us/op 0.40
Set add up to 256 items then delete last 7.2758 us/op 21.718 us/op 0.34
OrderedSet add up to 256 items then delete last 11.001 us/op 28.646 us/op 0.38
Set add up to 256 items then delete middle 7.1938 us/op 21.535 us/op 0.33
OrderedSet add up to 256 items then delete middle 30.158 us/op 55.187 us/op 0.55
transfer serialized Status (84 B) 1.3940 us/op 2.5480 us/op 0.55
copy serialized Status (84 B) 1.2380 us/op 2.1990 us/op 0.56
transfer serialized SignedVoluntaryExit (112 B) 1.5200 us/op 2.3210 us/op 0.65
copy serialized SignedVoluntaryExit (112 B) 1.3100 us/op 1.9930 us/op 0.66
transfer serialized ProposerSlashing (416 B) 1.9420 us/op 2.8900 us/op 0.67
copy serialized ProposerSlashing (416 B) 1.9130 us/op 2.9990 us/op 0.64
transfer serialized Attestation (485 B) 1.8190 us/op 2.9710 us/op 0.61
copy serialized Attestation (485 B) 1.8970 us/op 2.9590 us/op 0.64
transfer serialized AttesterSlashing (33232 B) 2.1200 us/op 2.8090 us/op 0.75
copy serialized AttesterSlashing (33232 B) 3.6950 us/op 11.071 us/op 0.33
transfer serialized Small SignedBeaconBlock (128000 B) 2.1240 us/op 4.0300 us/op 0.53
copy serialized Small SignedBeaconBlock (128000 B) 7.7120 us/op 43.183 us/op 0.18
transfer serialized Avg SignedBeaconBlock (200000 B) 2.1880 us/op 4.8940 us/op 0.45
copy serialized Avg SignedBeaconBlock (200000 B) 10.815 us/op 50.015 us/op 0.22
transfer serialized BlobsSidecar (524380 B) 2.3000 us/op 6.4640 us/op 0.36
copy serialized BlobsSidecar (524380 B) 69.359 us/op 196.66 us/op 0.35
transfer serialized Big SignedBeaconBlock (1000000 B) 2.8360 us/op 9.3580 us/op 0.30
copy serialized Big SignedBeaconBlock (1000000 B) 132.93 us/op 336.19 us/op 0.40
pass gossip attestations to forkchoice per slot 2.4581 ms/op 5.4530 ms/op 0.45
forkChoice updateHead vc 100000 bc 64 eq 0 433.61 us/op 710.91 us/op 0.61
forkChoice updateHead vc 600000 bc 64 eq 0 2.7426 ms/op 8.1950 ms/op 0.33
forkChoice updateHead vc 1000000 bc 64 eq 0 4.3813 ms/op 9.2631 ms/op 0.47
forkChoice updateHead vc 600000 bc 320 eq 0 2.6499 ms/op 5.0442 ms/op 0.53
forkChoice updateHead vc 600000 bc 1200 eq 0 2.6194 ms/op 5.7430 ms/op 0.46
forkChoice updateHead vc 600000 bc 7200 eq 0 3.1818 ms/op 6.7940 ms/op 0.47
forkChoice updateHead vc 600000 bc 64 eq 1000 9.9127 ms/op 13.998 ms/op 0.71
forkChoice updateHead vc 600000 bc 64 eq 10000 9.0964 ms/op 15.769 ms/op 0.58
forkChoice updateHead vc 600000 bc 64 eq 300000 11.877 ms/op 63.321 ms/op 0.19
computeDeltas 500000 validators 300 proto nodes 2.8065 ms/op 7.7764 ms/op 0.36
computeDeltas 500000 validators 1200 proto nodes 2.7968 ms/op 7.4626 ms/op 0.37
computeDeltas 500000 validators 7200 proto nodes 2.8024 ms/op 7.0544 ms/op 0.40
computeDeltas 750000 validators 300 proto nodes 4.2461 ms/op 10.417 ms/op 0.41
computeDeltas 750000 validators 1200 proto nodes 4.2628 ms/op 10.575 ms/op 0.40
computeDeltas 750000 validators 7200 proto nodes 4.2282 ms/op 10.382 ms/op 0.41
computeDeltas 1400000 validators 300 proto nodes 8.1176 ms/op 19.317 ms/op 0.42
computeDeltas 1400000 validators 1200 proto nodes 8.1618 ms/op 19.463 ms/op 0.42
computeDeltas 1400000 validators 7200 proto nodes 8.2161 ms/op 19.085 ms/op 0.43
computeDeltas 2100000 validators 300 proto nodes 12.232 ms/op 28.160 ms/op 0.43
computeDeltas 2100000 validators 1200 proto nodes 12.387 ms/op 28.340 ms/op 0.44
computeDeltas 2100000 validators 7200 proto nodes 12.299 ms/op 32.182 ms/op 0.38
computeProposerBoostScoreFromBalances 500000 validators 2.7271 ms/op 4.2463 ms/op 0.64
computeProposerBoostScoreFromBalances 750000 validators 2.7474 ms/op 4.0646 ms/op 0.68
computeProposerBoostScoreFromBalances 1400000 validators 2.7321 ms/op 4.0478 ms/op 0.67
computeProposerBoostScoreFromBalances 2100000 validators 2.7299 ms/op 4.1755 ms/op 0.65
altair processAttestation - 250000 vs - 7PWei normalcase 1.6593 ms/op 4.4791 ms/op 0.37
altair processAttestation - 250000 vs - 7PWei worstcase 2.1734 ms/op 5.8387 ms/op 0.37
altair processAttestation - setStatus - 1/6 committees join 94.022 us/op 280.88 us/op 0.33
altair processAttestation - setStatus - 1/3 committees join 189.79 us/op 471.42 us/op 0.40
altair processAttestation - setStatus - 1/2 committees join 275.61 us/op 627.59 us/op 0.44
altair processAttestation - setStatus - 2/3 committees join 366.74 us/op 729.25 us/op 0.50
altair processAttestation - setStatus - 4/5 committees join 474.18 us/op 1.0848 ms/op 0.44
altair processAttestation - setStatus - 100% committees join 568.51 us/op 1.3829 ms/op 0.41
altair processBlock - 250000 vs - 7PWei normalcase 5.6676 ms/op 13.623 ms/op 0.42
altair processBlock - 250000 vs - 7PWei normalcase hashState 23.597 ms/op 45.015 ms/op 0.52
altair processBlock - 250000 vs - 7PWei worstcase 30.213 ms/op 53.893 ms/op 0.56
altair processBlock - 250000 vs - 7PWei worstcase hashState 62.062 ms/op 143.26 ms/op 0.43
phase0 processBlock - 250000 vs - 7PWei normalcase 1.5242 ms/op 5.5778 ms/op 0.27
phase0 processBlock - 250000 vs - 7PWei worstcase 23.591 ms/op 43.929 ms/op 0.54
altair processEth1Data - 250000 vs - 7PWei normalcase 285.27 us/op 784.86 us/op 0.36
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.8950 us/op 30.016 us/op 0.26
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 31.267 us/op 118.90 us/op 0.26
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 9.4120 us/op 32.712 us/op 0.29
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.8650 us/op 33.810 us/op 0.26
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 103.11 us/op 264.34 us/op 0.39
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 919.46 us/op 2.3591 ms/op 0.39
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.1586 ms/op 2.6897 ms/op 0.43
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 873.02 us/op 2.6884 ms/op 0.32
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.7559 ms/op 7.0213 ms/op 0.39
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.6140 ms/op 3.5215 ms/op 0.46
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.6927 ms/op 8.2806 ms/op 0.45
Tree 40 250000 create 220.24 ms/op 571.47 ms/op 0.39
Tree 40 250000 get(125000) 111.51 ns/op 253.12 ns/op 0.44
Tree 40 250000 set(125000) 694.97 ns/op 2.6365 us/op 0.26
Tree 40 250000 toArray() 9.6015 ms/op 39.932 ms/op 0.24
Tree 40 250000 iterate all - toArray() + loop 9.7993 ms/op 36.043 ms/op 0.27
Tree 40 250000 iterate all - get(i) 42.918 ms/op 93.942 ms/op 0.46
MutableVector 250000 create 7.5637 ms/op 18.903 ms/op 0.40
MutableVector 250000 get(125000) 5.5970 ns/op 8.3910 ns/op 0.67
MutableVector 250000 set(125000) 183.99 ns/op 674.67 ns/op 0.27
MutableVector 250000 toArray() 2.7061 ms/op 5.6759 ms/op 0.48
MutableVector 250000 iterate all - toArray() + loop 2.7454 ms/op 5.3933 ms/op 0.51
MutableVector 250000 iterate all - get(i) 1.3436 ms/op 1.7909 ms/op 0.75
Array 250000 create 2.2067 ms/op 5.8248 ms/op 0.38
Array 250000 clone - spread 872.71 us/op 1.4296 ms/op 0.61
Array 250000 get(125000) 0.53600 ns/op 0.83400 ns/op 0.64
Array 250000 set(125000) 0.60700 ns/op 1.2470 ns/op 0.49
Array 250000 iterate all - loop 76.592 us/op 89.878 us/op 0.85
effectiveBalanceIncrements clone Uint8Array 300000 11.546 us/op 91.443 us/op 0.13
effectiveBalanceIncrements clone MutableVector 300000 327.00 ns/op 349.00 ns/op 0.94
effectiveBalanceIncrements rw all Uint8Array 300000 167.14 us/op 202.06 us/op 0.83
effectiveBalanceIncrements rw all MutableVector 300000 61.575 ms/op 185.35 ms/op 0.33
phase0 afterProcessEpoch - 250000 vs - 7PWei 73.241 ms/op 126.82 ms/op 0.58
phase0 beforeProcessEpoch - 250000 vs - 7PWei 25.032 ms/op 69.973 ms/op 0.36
altair processEpoch - mainnet_e81889 376.43 ms/op 536.05 ms/op 0.70
mainnet_e81889 - altair beforeProcessEpoch 54.782 ms/op 81.447 ms/op 0.67
mainnet_e81889 - altair processJustificationAndFinalization 9.6970 us/op 30.166 us/op 0.32
mainnet_e81889 - altair processInactivityUpdates 4.4695 ms/op 10.650 ms/op 0.42
mainnet_e81889 - altair processRewardsAndPenalties 41.093 ms/op 88.119 ms/op 0.47
mainnet_e81889 - altair processRegistryUpdates 1.2980 us/op 5.2090 us/op 0.25
mainnet_e81889 - altair processSlashings 378.00 ns/op 1.3710 us/op 0.28
mainnet_e81889 - altair processEth1DataReset 639.00 ns/op 1.1870 us/op 0.54
mainnet_e81889 - altair processEffectiveBalanceUpdates 909.84 us/op 1.3170 ms/op 0.69
mainnet_e81889 - altair processSlashingsReset 1.9180 us/op 5.1340 us/op 0.37
mainnet_e81889 - altair processRandaoMixesReset 2.0480 us/op 8.7760 us/op 0.23
mainnet_e81889 - altair processHistoricalRootsUpdate 391.00 ns/op 710.00 ns/op 0.55
mainnet_e81889 - altair processParticipationFlagUpdates 1.2620 us/op 3.6310 us/op 0.35
mainnet_e81889 - altair processSyncCommitteeUpdates 335.00 ns/op 876.00 ns/op 0.38
mainnet_e81889 - altair afterProcessEpoch 81.078 ms/op 131.16 ms/op 0.62
capella processEpoch - mainnet_e217614 1.1948 s/op 1.8423 s/op 0.65
mainnet_e217614 - capella beforeProcessEpoch 193.50 ms/op 314.52 ms/op 0.62
mainnet_e217614 - capella processJustificationAndFinalization 5.7390 us/op 25.122 us/op 0.23
mainnet_e217614 - capella processInactivityUpdates 14.827 ms/op 25.520 ms/op 0.58
mainnet_e217614 - capella processRewardsAndPenalties 177.22 ms/op 335.12 ms/op 0.53
mainnet_e217614 - capella processRegistryUpdates 7.7800 us/op 34.203 us/op 0.23
mainnet_e217614 - capella processSlashings 428.00 ns/op 1.4200 us/op 0.30
mainnet_e217614 - capella processEth1DataReset 317.00 ns/op 1.5190 us/op 0.21
mainnet_e217614 - capella processEffectiveBalanceUpdates 3.1234 ms/op 8.0649 ms/op 0.39
mainnet_e217614 - capella processSlashingsReset 1.1210 us/op 7.6600 us/op 0.15
mainnet_e217614 - capella processRandaoMixesReset 1.5730 us/op 9.9600 us/op 0.16
mainnet_e217614 - capella processHistoricalRootsUpdate 466.00 ns/op 1.0310 us/op 0.45
mainnet_e217614 - capella processParticipationFlagUpdates 1.4310 us/op 4.5300 us/op 0.32
mainnet_e217614 - capella afterProcessEpoch 201.91 ms/op 327.21 ms/op 0.62
phase0 processEpoch - mainnet_e58758 317.63 ms/op 546.03 ms/op 0.58
mainnet_e58758 - phase0 beforeProcessEpoch 98.795 ms/op 186.78 ms/op 0.53
mainnet_e58758 - phase0 processJustificationAndFinalization 10.720 us/op 25.948 us/op 0.41
mainnet_e58758 - phase0 processRewardsAndPenalties 38.480 ms/op 61.931 ms/op 0.62
mainnet_e58758 - phase0 processRegistryUpdates 6.7350 us/op 18.204 us/op 0.37
mainnet_e58758 - phase0 processSlashings 341.00 ns/op 1.1160 us/op 0.31
mainnet_e58758 - phase0 processEth1DataReset 302.00 ns/op 1.2460 us/op 0.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 732.35 us/op 3.8441 ms/op 0.19
mainnet_e58758 - phase0 processSlashingsReset 1.8560 us/op 4.1600 us/op 0.45
mainnet_e58758 - phase0 processRandaoMixesReset 3.3480 us/op 7.1030 us/op 0.47
mainnet_e58758 - phase0 processHistoricalRootsUpdate 347.00 ns/op 1.5110 us/op 0.23
mainnet_e58758 - phase0 processParticipationRecordUpdates 2.9650 us/op 14.612 us/op 0.20
mainnet_e58758 - phase0 afterProcessEpoch 65.211 ms/op 117.37 ms/op 0.56
phase0 processEffectiveBalanceUpdates - 250000 normalcase 926.27 us/op 2.5391 ms/op 0.36
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.1143 ms/op 1.5595 ms/op 0.71
altair processInactivityUpdates - 250000 normalcase 13.153 ms/op 32.271 ms/op 0.41
altair processInactivityUpdates - 250000 worstcase 14.237 ms/op 30.210 ms/op 0.47
phase0 processRegistryUpdates - 250000 normalcase 3.4720 us/op 15.405 us/op 0.23
phase0 processRegistryUpdates - 250000 badcase_full_deposits 224.98 us/op 405.34 us/op 0.56
phase0 processRegistryUpdates - 250000 worstcase 0.5 113.00 ms/op 158.14 ms/op 0.71
altair processRewardsAndPenalties - 250000 normalcase 51.439 ms/op 72.373 ms/op 0.71
altair processRewardsAndPenalties - 250000 worstcase 48.230 ms/op 76.935 ms/op 0.63
phase0 getAttestationDeltas - 250000 normalcase 5.9828 ms/op 11.734 ms/op 0.51
phase0 getAttestationDeltas - 250000 worstcase 5.4121 ms/op 13.231 ms/op 0.41
phase0 processSlashings - 250000 worstcase 1.6005 ms/op 3.3191 ms/op 0.48
altair processSyncCommitteeUpdates - 250000 99.496 ms/op 173.69 ms/op 0.57
BeaconState.hashTreeRoot - No change 349.00 ns/op 448.00 ns/op 0.78
BeaconState.hashTreeRoot - 1 full validator 110.37 us/op 180.95 us/op 0.61
BeaconState.hashTreeRoot - 32 full validator 978.47 us/op 2.1819 ms/op 0.45
BeaconState.hashTreeRoot - 512 full validator 8.8825 ms/op 22.483 ms/op 0.40
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 104.50 us/op 189.96 us/op 0.55
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3277 ms/op 2.6305 ms/op 0.50
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.464 ms/op 36.150 ms/op 0.46
BeaconState.hashTreeRoot - 1 balances 84.587 us/op 145.15 us/op 0.58
BeaconState.hashTreeRoot - 32 balances 743.89 us/op 1.3299 ms/op 0.56
BeaconState.hashTreeRoot - 512 balances 7.2345 ms/op 14.497 ms/op 0.50
BeaconState.hashTreeRoot - 250000 balances 138.28 ms/op 246.58 ms/op 0.56
aggregationBits - 2048 els - zipIndexesInBitList 9.1440 us/op 22.680 us/op 0.40
regular array get 100000 times 30.821 us/op 46.653 us/op 0.66
wrappedArray get 100000 times 30.808 us/op 34.334 us/op 0.90
arrayWithProxy get 100000 times 9.8295 ms/op 15.777 ms/op 0.62
ssz.Root.equals 238.00 ns/op 276.00 ns/op 0.86
byteArrayEquals 231.00 ns/op 250.00 ns/op 0.92
shuffle list - 16384 els 4.5869 ms/op 7.3162 ms/op 0.63
shuffle list - 250000 els 68.406 ms/op 104.38 ms/op 0.66
processSlot - 1 slots 11.008 us/op 17.462 us/op 0.63
processSlot - 32 slots 2.3544 ms/op 4.1741 ms/op 0.56
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 44.907 ms/op 59.783 ms/op 0.75
getCommitteeAssignments - req 1 vs - 250000 vc 2.2677 ms/op 2.6488 ms/op 0.86
getCommitteeAssignments - req 100 vs - 250000 vc 3.4002 ms/op 4.1590 ms/op 0.82
getCommitteeAssignments - req 1000 vs - 250000 vc 3.7031 ms/op 4.6536 ms/op 0.80
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.8800 ns/op 6.2500 ns/op 0.78
state getBlockRootAtSlot - 250000 vs - 7PWei 540.90 ns/op 684.02 ns/op 0.79
computeProposers - vc 250000 5.9922 ms/op 9.9774 ms/op 0.60
computeEpochShuffling - vc 250000 73.400 ms/op 108.94 ms/op 0.67
getNextSyncCommittee - vc 250000 108.61 ms/op 172.79 ms/op 0.63
computeSigningRoot for AttestationData 18.259 us/op 30.558 us/op 0.60
hash AttestationData serialized data then Buffer.toString(base64) 1.2967 us/op 2.4419 us/op 0.53
toHexString serialized data 791.61 ns/op 1.4676 us/op 0.54
Buffer.toString(base64) 156.43 ns/op 307.97 ns/op 0.51

by benchmarkbot/action

@matthewkeil matthewkeil marked this pull request as draft July 27, 2023 02:28
@matthewkeil matthewkeil force-pushed the mkeil/network-worker-port-metrics branch from 94afbec to 2c3d7e1 Compare July 27, 2023 08:27
@twoeths twoeths mentioned this pull request Aug 1, 2023
Merged
@matthewkeil matthewkeil marked this pull request as ready for review August 1, 2023 08:44
nflaig
nflaig reviewed Aug 1, 2023
View reviewed changes
dashboards/lodestar_networking.json Outdated Show resolved Hide resolved
packages/beacon-node/src/metrics/metrics/lodestar.ts Outdated Show resolved Hide resolved
packages/beacon-node/src/util/workerEvents.ts Show resolved Hide resolved
packages/beacon-node/src/util/workerEvents.ts Outdated Show resolved Hide resolved
@matthewkeil matthewkeil requested review from nflaig and dapplion and removed request for dapplion August 3, 2023 02:17
@nflaig nflaig mentioned this pull request Aug 3, 2023
Merged
@matthewkeil
Copy link
Member Author

matthewkeil commented Aug 7, 2023 •
edited
Loading

@dapplion this was updated to just have the worker metrics and the dashboards were mergerd in #5827

dapplion
dapplion previously approved these changes Aug 8, 2023
View reviewed changes
nflaig
nflaig requested changes Aug 8, 2023
View reviewed changes
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should clarify in this PR if we want Sec-suffix or not before merging it.

I think we should omit that see #5800 (comment) and conversation here #5800 (comment).

packages/beacon-node/src/util/workerEvents.ts Outdated Show resolved Hide resolved
@wemeetagain wemeetagain self-requested a review October 10, 2023 14:12
@nflaig
Copy link
Member

nflaig commented Oct 10, 2023 •
edited
Loading

@nflaig point is that since all metrics are seconds it may not be able to specify so in the name. Actually we don't do that anywhere else (please check if this is true). So for consistency we should: use seconds everywhere, and don't declare that on the name. For code variables we use multiple units so in that case it makes sense to specify the unit

I agree with what @dapplion mentioned in #5800 (comment)

  • should use seconds for all metrics
  • do not declare that on the name (remove Sec-suffix)
  • For code variables (other variables in the code that are time related, not including metric variables) we should use a unit suffix

As outlined here #5800 (comment), we do not suffix metric variable names with a time unit right now, or at least not besides a few outliers.

@matthewkeil
fix: remove unused import
0c84dcb
@matthewkeil
feat(metrics): add unit to metric name
daa2640
@matthewkeil
feat(metrics): add unit to metric name
18e4561
@matthewkeil
feat: add trace log statement to network worker
d18702c
@matthewkeil
fix: change trace to debug log
6099560
@matthewkeil
feat(metrics): add eventName to network worker message metrics
5949206
@matthewkeil
bug(logger): check if trace is broken
e1ad935
@matthewkeil
feat(metrics): add worker eventDirection label
106ae07
@matthewkeil
fix(metrics): use string instead of enum for eventDirection
570162b
@matthewkeil
fix(metrics): remove eventDirection label
5f76c36
@matthewkeil
feat(dashboards): add average panel for network worker message
e76e672
@matthewkeil
fix(metrics): update naming per Nico's suggestions
f69bfbb
@matthewkeil
refactor: remove unused logger from workerEvents.ts
1b92def
@matthewkeil
fix(metrics): add network worker unit name back
6b04c02
@matthewkeil
fix(beacon-node): use bigint for hrtime in worker message metric
f7874c0
@matthewkeil
fix(dashboards): remove dashboard changes. moved to PR#5827
042b913
@matthewkeil
fix(dashboards): remove dashboard changes. moved to PR#5827
8477afb
@matthewkeil
fix(dashboards): remove dashboard changes. moved to PR#5827
e106735
@matthewkeil
fix: constant case for nano conversion
cfcb7a6
@matthewkeil
Revert "fix(beacon-node): use bigint for hrtime in worker message met…
e802437 
…ric"

This reverts commit 59f12ff.
@matthewkeil
refactor: remove Sec suffix in metric variable name
537d465
@matthewkeil matthewkeil force-pushed the mkeil/network-worker-port-metrics branch from 41b1d88 to 537d465 Compare October 12, 2023 00:43
nflaig
nflaig approved these changes Oct 12, 2023
View reviewed changes
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@wemeetagain wemeetagain merged commit dd57c96 into unstable Oct 12, 2023
15 checks passed
@wemeetagain wemeetagain deleted the mkeil/network-worker-port-metrics branch October 12, 2023 19:27
@nflaig nflaig mentioned this pull request Oct 18, 2023
Merged
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.12.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

@nflaig nflaig nflaig approved these changes

@dapplion dapplion dapplion left review comments

@twoeths twoeths Awaiting requested review from twoeths

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Network worker extremely busy / high event loop lag / high I/O lag
5 participants
@matthewkeil @nflaig @wemeetagain @twoeths @dapplion