feat: switch bls to napi rebuild #6616

matthewkeil · 2024-03-31T16:11:46Z

Motivation

Swaps out SWIG/Herumi implementation of blst with Napi implementation.

Currently deployed on feat1 group and is developing metrics. Will post below as they progress.

~~DO NOT MERGE until the package version for bls is added. Needs to be merged and published first though~~

Description

Remove references to @chainsafe/blst
Add new version of @chainsafe/bls that uses the updated @chainsafe/blst under the hood
~~Add export UV_THREADPOOL_SIZE=$(nproc) to ./lodestar entrance script and benchmark commands to ensure adequate performance~~ Set process.env.UV_THREADPOOL_SIZE = os.availableParallelism().toString() in cli/src/cmds/beacon/beaconHandler.ts
Move G2_POINT_AT_INFINITY is_inf test to @chainsafe/blst
Add a hidden chain.blsAddVerificationRandomness CLI option for applying mult randomness to same message verification. Defaults to true unless explicitly set as false via the CLI
Add implementation of mult randomness for same message verification
Flatten beacon-node/src/chain/bls directory structure by removing multithread folder
Add addVerificationRandomness to verification options of queued jobs. Not available via the external function signature and is set via the CLI options and merged with options passed to the call to IBls.verifySignatureSets
Remove support in BlsMultiThreadWorkerPool for Herumi implementation. (see notes below)
Remove bls Worker support and code (see notes below)
Remove serialization of keys and sigs and pass as deserialized objects because no worker boundary exists anymore
Renamed "./maybeBatch.js" to "./verifySets.js"
Renamed "./worker.js" to "./runBlsWorkReq.js"
Combined beacon-node/src/chain/bls/multithread/utils.ts with beacon-node/src/chain/bls/utils.ts
Fixed one unit test because deep equal of Vitest failed because Uint8Array and Buffer are non-compatible so wrapped the buffer in Uint8Arry.from

NOTES

Before this can be merged @chainsafe/blst and @chainsafe/bls need to both be published and the package version needs to be swapped in the package.json files in here. For now a sha is being used to target the head of the feature branch of @chainsafe/bls
Potentially need to tune how canAcceptWork() functions to optimize performance. Possibly should remove queue functionality and only use queue on the gossipsub side or vice versa and only use queue in bls so that there are not two queues introducing throttling
The functionality to test is_inf for G2_POINT_AT_INFINITY is not longer exported from the deps so no way to test from the Lodestar side. Potentially should export that constant from @chainsafe/bls and move the test to there so that the value and unit test for it are in the same place. Unfortunately the is_inf in not available from Herumi so it really should be in @chainsafe/blst however if that is where the test and value live it will no longer be a peerDependency which is not great Need to import @chainsafe/blst as a dependency to lodestar now and we are maintaining it as a peerDependency in @chainsafe/bls, so its possible to use the value from the blst repo now.
Removed workers and Herumi entirely. It was not possible to even access herumi after digging extensively through the code. Would require a major refactor of all imports of @chainsafe/bls to even allow for using Herumi. Removing inaccessible, unused and unnecessary option greatly simplified class structure of BlsMultiThreadWorkerPool because workers should no longer be used realistically. The only case were workers would have been beneficial to retain would be to run Herumi on-thread in a worker. It would not be ideal to run napi on-thread in workers and is better to run it via libuv scheduling
Follow-up PR could potentially improve the metrics but leaving as-is for now so we can see apples-to-apples comparisons with existing branches. Can come back as a second pass to add/remove/update now that workers are no longer present
randomBytesNonZero is exported by @chainsafe/blst now but not via @chainsafe/bls because re-exporting would cause the same peerDependency issue as re-exporting G2_POINT_AT_INFINITY. This need to be resolved but highlighting here to get the teams' feelings on the best way to handle this

… matches for toEqual

github-actions · 2024-03-31T16:55:20Z

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite	Current: `64ea2a6`	Previous: `2649a87`	Ratio
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	945.21 us/op	2.8663 ms/op	0.33

Full benchmark results

Benchmark suite	Current: `64ea2a6`	Previous: `2649a87`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	1.1793 ms/op	636.32 us/op	1.85
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	55.616 us/op	84.777 us/op	0.66
BLS verify - blst-native	1.1414 ms/op	1.3040 ms/op	0.88
BLS verifyMultipleSignatures 3 - blst-native	2.2029 ms/op	2.7369 ms/op	0.80
BLS verifyMultipleSignatures 8 - blst-native	4.7596 ms/op	6.0260 ms/op	0.79
BLS verifyMultipleSignatures 32 - blst-native	17.315 ms/op	22.172 ms/op	0.78
BLS verifyMultipleSignatures 64 - blst-native	34.093 ms/op	43.702 ms/op	0.78
BLS verifyMultipleSignatures 128 - blst-native	67.902 ms/op	86.533 ms/op	0.78
BLS deserializing 10000 signatures	820.53 ms/op	925.19 ms/op	0.89
BLS deserializing 100000 signatures	7.9583 s/op	9.3033 s/op	0.86
BLS verifyMultipleSignatures - same message - 3 - blst-native	1.1029 ms/op	1.3281 ms/op	0.83
BLS verifyMultipleSignatures - same message - 8 - blst-native	1.2567 ms/op	1.6232 ms/op	0.77
BLS verifyMultipleSignatures - same message - 32 - blst-native	1.9913 ms/op	2.3353 ms/op	0.85
BLS verifyMultipleSignatures - same message - 64 - blst-native	2.9455 ms/op	4.4814 ms/op	0.66
BLS verifyMultipleSignatures - same message - 128 - blst-native	4.9853 ms/op	5.8185 ms/op	0.86
BLS aggregatePubkeys 32 - blst-native	24.181 us/op	27.680 us/op	0.87
BLS aggregatePubkeys 128 - blst-native	93.082 us/op	100.30 us/op	0.93
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	53.053 ms/op	61.307 ms/op	0.87
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	43.214 ms/op	60.989 ms/op	0.71
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	30.850 ms/op	37.075 ms/op	0.83
getSlashingsAndExits - default max	121.85 us/op	226.71 us/op	0.54
getSlashingsAndExits - 2k	375.47 us/op	402.46 us/op	0.93
proposeBlockBody type=full, size=empty	4.5395 ms/op	5.6591 ms/op	0.80
isKnown best case - 1 super set check	523.00 ns/op	376.00 ns/op	1.39
isKnown normal case - 2 super set checks	380.00 ns/op	345.00 ns/op	1.10
isKnown worse case - 16 super set checks	486.00 ns/op	359.00 ns/op	1.35
InMemoryCheckpointStateCache - add get delete	6.8120 us/op	8.3980 us/op	0.81
validate api signedAggregateAndProof - struct	2.0753 ms/op	2.7972 ms/op	0.74
validate gossip signedAggregateAndProof - struct	2.0485 ms/op	2.7936 ms/op	0.73
validate gossip attestation - vc 640000	1.1831 ms/op	1.3835 ms/op	0.86
batch validate gossip attestation - vc 640000 - chunk 32	150.72 us/op	169.52 us/op	0.89
batch validate gossip attestation - vc 640000 - chunk 64	143.09 us/op	150.27 us/op	0.95
batch validate gossip attestation - vc 640000 - chunk 128	144.84 us/op	137.45 us/op	1.05
batch validate gossip attestation - vc 640000 - chunk 256	134.45 us/op	127.97 us/op	1.05
pickEth1Vote - no votes	1.0683 ms/op	1.1700 ms/op	0.91
pickEth1Vote - max votes	15.321 ms/op	11.084 ms/op	1.38
pickEth1Vote - Eth1Data hashTreeRoot value x2048	23.785 ms/op	23.030 ms/op	1.03
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	37.798 ms/op	25.331 ms/op	1.49
pickEth1Vote - Eth1Data fastSerialize value x2048	637.60 us/op	568.94 us/op	1.12
pickEth1Vote - Eth1Data fastSerialize tree x2048	5.6957 ms/op	6.5836 ms/op	0.87
bytes32 toHexString	674.00 ns/op	527.00 ns/op	1.28
bytes32 Buffer.toString(hex)	405.00 ns/op	313.00 ns/op	1.29
bytes32 Buffer.toString(hex) from Uint8Array	591.00 ns/op	441.00 ns/op	1.34
bytes32 Buffer.toString(hex) + 0x	543.00 ns/op	297.00 ns/op	1.83
Object access 1 prop	0.30100 ns/op	0.18800 ns/op	1.60
Map access 1 prop	0.20700 ns/op	0.15300 ns/op	1.35
Object get x1000	5.4250 ns/op	7.2170 ns/op	0.75
Map get x1000	0.95100 ns/op	0.80300 ns/op	1.18
Object set x1000	32.383 ns/op	51.313 ns/op	0.63
Map set x1000	20.102 ns/op	40.766 ns/op	0.49
Return object 10000 times	0.26530 ns/op	0.24210 ns/op	1.10
Throw Error 10000 times	3.1656 us/op	3.8667 us/op	0.82
fastMsgIdFn sha256 / 200 bytes	2.5380 us/op	3.3450 us/op	0.76
fastMsgIdFn h32 xxhash / 200 bytes	480.00 ns/op	309.00 ns/op	1.55
fastMsgIdFn h64 xxhash / 200 bytes	467.00 ns/op	359.00 ns/op	1.30
fastMsgIdFn sha256 / 1000 bytes	6.8950 us/op	11.510 us/op	0.60
fastMsgIdFn h32 xxhash / 1000 bytes	581.00 ns/op	418.00 ns/op	1.39
fastMsgIdFn h64 xxhash / 1000 bytes	656.00 ns/op	438.00 ns/op	1.50
fastMsgIdFn sha256 / 10000 bytes	56.210 us/op	101.56 us/op	0.55
fastMsgIdFn h32 xxhash / 10000 bytes	2.0430 us/op	1.8830 us/op	1.08
fastMsgIdFn h64 xxhash / 10000 bytes	1.5320 us/op	1.2910 us/op	1.19
send data - 1000 256B messages	18.784 ms/op	17.039 ms/op	1.10
send data - 1000 512B messages	24.748 ms/op	27.106 ms/op	0.91
send data - 1000 1024B messages	32.523 ms/op	37.967 ms/op	0.86
send data - 1000 1200B messages	49.305 ms/op	39.754 ms/op	1.24
send data - 1000 2048B messages	55.025 ms/op	49.087 ms/op	1.12
send data - 1000 4096B messages	35.486 ms/op	42.347 ms/op	0.84
send data - 1000 16384B messages	147.24 ms/op	118.49 ms/op	1.24
send data - 1000 65536B messages	667.54 ms/op	513.20 ms/op	1.30
enrSubnets - fastDeserialize 64 bits	2.4250 us/op	1.2560 us/op	1.93
enrSubnets - ssz BitVector 64 bits	1.0960 us/op	426.00 ns/op	2.57
enrSubnets - fastDeserialize 4 bits	339.00 ns/op	173.00 ns/op	1.96
enrSubnets - ssz BitVector 4 bits	1.1260 us/op	438.00 ns/op	2.57
prioritizePeers score -10:0 att 32-0.1 sync 2-0	374.25 us/op	221.88 us/op	1.69
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	447.81 us/op	274.25 us/op	1.63
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	532.83 us/op	324.12 us/op	1.64
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	738.98 us/op	551.62 us/op	1.34
prioritizePeers score 0:0 att 64-1 sync 4-1	858.46 us/op	668.63 us/op	1.28
array of 16000 items push then shift	1.5983 us/op	1.5434 us/op	1.04
LinkedList of 16000 items push then shift	9.8470 ns/op	8.6090 ns/op	1.14
array of 16000 items push then pop	163.98 ns/op	68.234 ns/op	2.40
LinkedList of 16000 items push then pop	11.507 ns/op	8.5840 ns/op	1.34
array of 24000 items push then shift	2.0961 us/op	2.2806 us/op	0.92
LinkedList of 24000 items push then shift	10.071 ns/op	8.5950 ns/op	1.17
array of 24000 items push then pop	228.95 ns/op	108.55 ns/op	2.11
LinkedList of 24000 items push then pop	10.107 ns/op	8.4950 ns/op	1.19
intersect bitArray bitLen 8	5.1370 ns/op	5.6550 ns/op	0.91
intersect array and set length 8	97.247 ns/op	61.013 ns/op	1.59
intersect bitArray bitLen 128	29.886 ns/op	34.368 ns/op	0.87
intersect array and set length 128	1.3011 us/op	860.02 ns/op	1.51
bitArray.getTrueBitIndexes() bitLen 128	2.7480 us/op	1.5360 us/op	1.79
bitArray.getTrueBitIndexes() bitLen 248	4.0190 us/op	2.3440 us/op	1.71
bitArray.getTrueBitIndexes() bitLen 512	8.1680 us/op	5.7720 us/op	1.42
Buffer.concat 32 items	1.3760 us/op	1.0830 us/op	1.27
Uint8Array.set 32 items	3.9640 us/op	2.2310 us/op	1.78
Set add up to 64 items then delete first	2.9115 us/op	4.2896 us/op	0.68
OrderedSet add up to 64 items then delete first	5.0076 us/op	5.4740 us/op	0.91
Set add up to 64 items then delete last	3.1114 us/op	4.5915 us/op	0.68
OrderedSet add up to 64 items then delete last	5.2422 us/op	5.6249 us/op	0.93
Set add up to 64 items then delete middle	3.1459 us/op	4.4770 us/op	0.70
OrderedSet add up to 64 items then delete middle	6.4125 us/op	6.9446 us/op	0.92
Set add up to 128 items then delete first	6.3451 us/op	9.1081 us/op	0.70
OrderedSet add up to 128 items then delete first	9.5575 us/op	12.384 us/op	0.77
Set add up to 128 items then delete last	6.4033 us/op	9.4588 us/op	0.68
OrderedSet add up to 128 items then delete last	10.342 us/op	11.844 us/op	0.87
Set add up to 128 items then delete middle	5.9799 us/op	9.1230 us/op	0.66
OrderedSet add up to 128 items then delete middle	13.122 us/op	18.325 us/op	0.72
Set add up to 256 items then delete first	8.9316 us/op	21.399 us/op	0.42
OrderedSet add up to 256 items then delete first	11.520 us/op	27.505 us/op	0.42
Set add up to 256 items then delete last	10.101 us/op	21.759 us/op	0.46
OrderedSet add up to 256 items then delete last	18.448 us/op	23.330 us/op	0.79
Set add up to 256 items then delete middle	11.065 us/op	18.308 us/op	0.60
OrderedSet add up to 256 items then delete middle	39.355 us/op	46.835 us/op	0.84
transfer serialized Status (84 B)	2.0130 us/op	1.8530 us/op	1.09
copy serialized Status (84 B)	1.3750 us/op	1.3970 us/op	0.98
transfer serialized SignedVoluntaryExit (112 B)	1.9290 us/op	1.8830 us/op	1.02
copy serialized SignedVoluntaryExit (112 B)	1.4760 us/op	1.4350 us/op	1.03
transfer serialized ProposerSlashing (416 B)	2.1580 us/op	2.7000 us/op	0.80
copy serialized ProposerSlashing (416 B)	2.2270 us/op	2.4480 us/op	0.91
transfer serialized Attestation (485 B)	2.5630 us/op	2.6350 us/op	0.97
copy serialized Attestation (485 B)	3.3420 us/op	2.3810 us/op	1.40
transfer serialized AttesterSlashing (33232 B)	4.1980 us/op	2.5690 us/op	1.63
copy serialized AttesterSlashing (33232 B)	13.145 us/op	6.5080 us/op	2.02
transfer serialized Small SignedBeaconBlock (128000 B)	4.7240 us/op	2.8170 us/op	1.68
copy serialized Small SignedBeaconBlock (128000 B)	38.905 us/op	24.166 us/op	1.61
transfer serialized Avg SignedBeaconBlock (200000 B)	5.4010 us/op	3.3370 us/op	1.62
copy serialized Avg SignedBeaconBlock (200000 B)	54.148 us/op	37.386 us/op	1.45
transfer serialized BlobsSidecar (524380 B)	6.3190 us/op	4.1410 us/op	1.53
copy serialized BlobsSidecar (524380 B)	131.34 us/op	134.69 us/op	0.98
transfer serialized Big SignedBeaconBlock (1000000 B)	4.8260 us/op	3.4780 us/op	1.39
copy serialized Big SignedBeaconBlock (1000000 B)	274.54 us/op	266.65 us/op	1.03
pass gossip attestations to forkchoice per slot	2.9301 ms/op	4.8770 ms/op	0.60
forkChoice updateHead vc 100000 bc 64 eq 0	531.33 us/op	713.88 us/op	0.74
forkChoice updateHead vc 600000 bc 64 eq 0	3.4486 ms/op	4.5982 ms/op	0.75
forkChoice updateHead vc 1000000 bc 64 eq 0	4.8778 ms/op	7.4526 ms/op	0.65
forkChoice updateHead vc 600000 bc 320 eq 0	2.7590 ms/op	4.3258 ms/op	0.64
forkChoice updateHead vc 600000 bc 1200 eq 0	3.1495 ms/op	4.4475 ms/op	0.71
forkChoice updateHead vc 600000 bc 7200 eq 0	4.1264 ms/op	5.5324 ms/op	0.75
forkChoice updateHead vc 600000 bc 64 eq 1000	10.601 ms/op	11.314 ms/op	0.94
forkChoice updateHead vc 600000 bc 64 eq 10000	10.143 ms/op	12.614 ms/op	0.80
forkChoice updateHead vc 600000 bc 64 eq 300000	37.207 ms/op	17.589 ms/op	2.12
computeDeltas 500000 validators 300 proto nodes	3.6189 ms/op	7.0477 ms/op	0.51
computeDeltas 500000 validators 1200 proto nodes	3.2911 ms/op	6.9657 ms/op	0.47
computeDeltas 500000 validators 7200 proto nodes	2.9934 ms/op	6.7845 ms/op	0.44
computeDeltas 750000 validators 300 proto nodes	4.5319 ms/op	9.9807 ms/op	0.45
computeDeltas 750000 validators 1200 proto nodes	4.4379 ms/op	10.211 ms/op	0.43
computeDeltas 750000 validators 7200 proto nodes	4.3508 ms/op	9.6837 ms/op	0.45
computeDeltas 1400000 validators 300 proto nodes	8.4624 ms/op	19.120 ms/op	0.44
computeDeltas 1400000 validators 1200 proto nodes	8.6495 ms/op	18.891 ms/op	0.46
computeDeltas 1400000 validators 7200 proto nodes	8.6819 ms/op	18.831 ms/op	0.46
computeDeltas 2100000 validators 300 proto nodes	12.886 ms/op	27.609 ms/op	0.47
computeDeltas 2100000 validators 1200 proto nodes	12.405 ms/op	29.178 ms/op	0.43
computeDeltas 2100000 validators 7200 proto nodes	12.746 ms/op	28.522 ms/op	0.45
altair processAttestation - 250000 vs - 7PWei normalcase	1.5685 ms/op	2.7003 ms/op	0.58
altair processAttestation - 250000 vs - 7PWei worstcase	2.2472 ms/op	3.9959 ms/op	0.56
altair processAttestation - setStatus - 1/6 committees join	74.079 us/op	162.14 us/op	0.46
altair processAttestation - setStatus - 1/3 committees join	147.56 us/op	313.15 us/op	0.47
altair processAttestation - setStatus - 1/2 committees join	204.19 us/op	394.80 us/op	0.52
altair processAttestation - setStatus - 2/3 committees join	270.66 us/op	506.12 us/op	0.53
altair processAttestation - setStatus - 4/5 committees join	379.45 us/op	683.69 us/op	0.56
altair processAttestation - setStatus - 100% committees join	485.10 us/op	814.94 us/op	0.60
altair processBlock - 250000 vs - 7PWei normalcase	6.4861 ms/op	8.6462 ms/op	0.75
altair processBlock - 250000 vs - 7PWei normalcase hashState	24.736 ms/op	36.135 ms/op	0.68
altair processBlock - 250000 vs - 7PWei worstcase	29.734 ms/op	44.700 ms/op	0.67
altair processBlock - 250000 vs - 7PWei worstcase hashState	75.489 ms/op	95.075 ms/op	0.79
phase0 processBlock - 250000 vs - 7PWei normalcase	2.2413 ms/op	2.8443 ms/op	0.79
phase0 processBlock - 250000 vs - 7PWei worstcase	25.001 ms/op	31.875 ms/op	0.78
altair processEth1Data - 250000 vs - 7PWei normalcase	267.25 us/op	581.53 us/op	0.46
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	11.182 us/op	20.499 us/op	0.55
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	51.567 us/op	85.485 us/op	0.60
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	14.816 us/op	20.470 us/op	0.72
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	12.767 us/op	9.8950 us/op	1.29
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	178.63 us/op	204.64 us/op	0.87
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	666.65 us/op	1.3329 ms/op	0.50
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	917.45 us/op	1.8906 ms/op	0.49
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	945.21 us/op	2.8663 ms/op	0.33
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	2.4634 ms/op	3.8840 ms/op	0.63
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.7348 ms/op	2.4073 ms/op	0.72
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.7144 ms/op	4.6979 ms/op	0.79
Tree 40 250000 create	291.91 ms/op	357.84 ms/op	0.82
Tree 40 250000 get(125000)	112.02 ns/op	193.95 ns/op	0.58
Tree 40 250000 set(125000)	751.19 ns/op	1.0165 us/op	0.74
Tree 40 250000 toArray()	9.2622 ms/op	19.851 ms/op	0.47
Tree 40 250000 iterate all - toArray() + loop	9.3640 ms/op	19.722 ms/op	0.47
Tree 40 250000 iterate all - get(i)	42.492 ms/op	72.656 ms/op	0.58
MutableVector 250000 create	13.808 ms/op	20.241 ms/op	0.68
MutableVector 250000 get(125000)	5.7230 ns/op	6.5480 ns/op	0.87
MutableVector 250000 set(125000)	196.82 ns/op	266.52 ns/op	0.74
MutableVector 250000 toArray()	2.5259 ms/op	3.3467 ms/op	0.75
MutableVector 250000 iterate all - toArray() + loop	2.6788 ms/op	3.4308 ms/op	0.78
MutableVector 250000 iterate all - get(i)	1.2761 ms/op	1.5437 ms/op	0.83
Array 250000 create	2.3234 ms/op	2.8759 ms/op	0.81
Array 250000 clone - spread	1.1232 ms/op	1.2529 ms/op	0.90
Array 250000 get(125000)	1.0280 ns/op	1.0580 ns/op	0.97
Array 250000 set(125000)	1.2260 ns/op	4.1560 ns/op	0.29
Array 250000 iterate all - loop	150.13 us/op	166.54 us/op	0.90
effectiveBalanceIncrements clone Uint8Array 300000	23.937 us/op	28.882 us/op	0.83
effectiveBalanceIncrements clone MutableVector 300000	419.00 ns/op	383.00 ns/op	1.09
effectiveBalanceIncrements rw all Uint8Array 300000	177.22 us/op	207.45 us/op	0.85
effectiveBalanceIncrements rw all MutableVector 300000	65.827 ms/op	84.325 ms/op	0.78
phase0 afterProcessEpoch - 250000 vs - 7PWei	77.237 ms/op	113.95 ms/op	0.68
phase0 beforeProcessEpoch - 250000 vs - 7PWei	44.097 ms/op	59.553 ms/op	0.74
altair processEpoch - mainnet_e81889	411.41 ms/op	533.84 ms/op	0.77
mainnet_e81889 - altair beforeProcessEpoch	67.744 ms/op	84.158 ms/op	0.80
mainnet_e81889 - altair processJustificationAndFinalization	10.152 us/op	20.637 us/op	0.49
mainnet_e81889 - altair processInactivityUpdates	6.5388 ms/op	6.8851 ms/op	0.95
mainnet_e81889 - altair processRewardsAndPenalties	58.207 ms/op	91.783 ms/op	0.63
mainnet_e81889 - altair processRegistryUpdates	1.8220 us/op	3.0980 us/op	0.59
mainnet_e81889 - altair processSlashings	537.00 ns/op	799.00 ns/op	0.67
mainnet_e81889 - altair processEth1DataReset	870.00 ns/op	850.00 ns/op	1.02
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.0272 ms/op	2.5950 ms/op	0.40
mainnet_e81889 - altair processSlashingsReset	2.9560 us/op	5.8920 us/op	0.50
mainnet_e81889 - altair processRandaoMixesReset	3.5410 us/op	6.7730 us/op	0.52
mainnet_e81889 - altair processHistoricalRootsUpdate	716.00 ns/op	967.00 ns/op	0.74
mainnet_e81889 - altair processParticipationFlagUpdates	1.3150 us/op	3.5530 us/op	0.37
mainnet_e81889 - altair processSyncCommitteeUpdates	635.00 ns/op	713.00 ns/op	0.89
mainnet_e81889 - altair afterProcessEpoch	82.559 ms/op	119.11 ms/op	0.69
capella processEpoch - mainnet_e217614	1.9002 s/op	2.1339 s/op	0.89
mainnet_e217614 - capella beforeProcessEpoch	402.99 ms/op	489.14 ms/op	0.82
mainnet_e217614 - capella processJustificationAndFinalization	8.9150 us/op	17.466 us/op	0.51
mainnet_e217614 - capella processInactivityUpdates	19.060 ms/op	20.665 ms/op	0.92
mainnet_e217614 - capella processRewardsAndPenalties	455.08 ms/op	567.11 ms/op	0.80
mainnet_e217614 - capella processRegistryUpdates	18.231 us/op	18.097 us/op	1.01
mainnet_e217614 - capella processSlashings	556.00 ns/op	532.00 ns/op	1.05
mainnet_e217614 - capella processEth1DataReset	534.00 ns/op	515.00 ns/op	1.04
mainnet_e217614 - capella processEffectiveBalanceUpdates	3.4573 ms/op	4.4113 ms/op	0.78
mainnet_e217614 - capella processSlashingsReset	2.0770 us/op	4.7630 us/op	0.44
mainnet_e217614 - capella processRandaoMixesReset	2.7520 us/op	5.6880 us/op	0.48
mainnet_e217614 - capella processHistoricalRootsUpdate	609.00 ns/op	899.00 ns/op	0.68
mainnet_e217614 - capella processParticipationFlagUpdates	1.3370 us/op	1.9600 us/op	0.68
mainnet_e217614 - capella afterProcessEpoch	204.95 ms/op	314.26 ms/op	0.65
phase0 processEpoch - mainnet_e58758	446.41 ms/op	517.43 ms/op	0.86
mainnet_e58758 - phase0 beforeProcessEpoch	122.10 ms/op	157.55 ms/op	0.77
mainnet_e58758 - phase0 processJustificationAndFinalization	7.6320 us/op	22.472 us/op	0.34
mainnet_e58758 - phase0 processRewardsAndPenalties	64.227 ms/op	58.509 ms/op	1.10
mainnet_e58758 - phase0 processRegistryUpdates	7.2250 us/op	18.418 us/op	0.39
mainnet_e58758 - phase0 processSlashings	636.00 ns/op	1.3940 us/op	0.46
mainnet_e58758 - phase0 processEth1DataReset	532.00 ns/op	1.0490 us/op	0.51
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.3569 ms/op	1.2612 ms/op	1.08
mainnet_e58758 - phase0 processSlashingsReset	2.1550 us/op	3.7270 us/op	0.58
mainnet_e58758 - phase0 processRandaoMixesReset	3.0540 us/op	6.0100 us/op	0.51
mainnet_e58758 - phase0 processHistoricalRootsUpdate	536.00 ns/op	553.00 ns/op	0.97
mainnet_e58758 - phase0 processParticipationRecordUpdates	2.9570 us/op	8.6270 us/op	0.34
mainnet_e58758 - phase0 afterProcessEpoch	65.884 ms/op	106.43 ms/op	0.62
phase0 processEffectiveBalanceUpdates - 250000 normalcase	984.59 us/op	1.9017 ms/op	0.52
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.1360 ms/op	1.5183 ms/op	0.75
altair processInactivityUpdates - 250000 normalcase	21.552 ms/op	30.700 ms/op	0.70
altair processInactivityUpdates - 250000 worstcase	25.108 ms/op	40.756 ms/op	0.62
phase0 processRegistryUpdates - 250000 normalcase	6.6850 us/op	17.907 us/op	0.37
phase0 processRegistryUpdates - 250000 badcase_full_deposits	405.06 us/op	574.86 us/op	0.70
phase0 processRegistryUpdates - 250000 worstcase 0.5	97.878 ms/op	188.88 ms/op	0.52
altair processRewardsAndPenalties - 250000 normalcase	70.823 ms/op	88.707 ms/op	0.80
altair processRewardsAndPenalties - 250000 worstcase	53.473 ms/op	73.811 ms/op	0.72
phase0 getAttestationDeltas - 250000 normalcase	5.6450 ms/op	11.072 ms/op	0.51
phase0 getAttestationDeltas - 250000 worstcase	5.7926 ms/op	10.068 ms/op	0.58
phase0 processSlashings - 250000 worstcase	46.667 us/op	135.45 us/op	0.34
altair processSyncCommitteeUpdates - 250000	101.74 ms/op	168.96 ms/op	0.60
BeaconState.hashTreeRoot - No change	414.00 ns/op	679.00 ns/op	0.61
BeaconState.hashTreeRoot - 1 full validator	104.72 us/op	182.31 us/op	0.57
BeaconState.hashTreeRoot - 32 full validator	1.1568 ms/op	2.0479 ms/op	0.56
BeaconState.hashTreeRoot - 512 full validator	13.223 ms/op	23.489 ms/op	0.56
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	115.32 us/op	252.84 us/op	0.46
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.8845 ms/op	3.1474 ms/op	0.60
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	32.287 ms/op	36.328 ms/op	0.89
BeaconState.hashTreeRoot - 1 balances	146.53 us/op	171.48 us/op	0.85
BeaconState.hashTreeRoot - 32 balances	1.4047 ms/op	1.6792 ms/op	0.84
BeaconState.hashTreeRoot - 512 balances	13.767 ms/op	17.492 ms/op	0.79
BeaconState.hashTreeRoot - 250000 balances	195.52 ms/op	243.22 ms/op	0.80
aggregationBits - 2048 els - zipIndexesInBitList	19.352 us/op	39.610 us/op	0.49
byteArrayEquals 32	65.459 ns/op	79.775 ns/op	0.82
Buffer.compare 32	38.180 ns/op	57.434 ns/op	0.66
byteArrayEquals 1024	1.7839 us/op	2.2474 us/op	0.79
Buffer.compare 1024	46.495 ns/op	73.111 ns/op	0.64
byteArrayEquals 16384	28.316 us/op	36.478 us/op	0.78
Buffer.compare 16384	234.88 ns/op	273.37 ns/op	0.86
byteArrayEquals 123687377	197.81 ms/op	273.15 ms/op	0.72
Buffer.compare 123687377	5.2271 ms/op	9.8119 ms/op	0.53
byteArrayEquals 32 - diff last byte	57.238 ns/op	91.843 ns/op	0.62
Buffer.compare 32 - diff last byte	34.142 ns/op	73.230 ns/op	0.47
byteArrayEquals 1024 - diff last byte	1.6099 us/op	2.1916 us/op	0.73
Buffer.compare 1024 - diff last byte	39.540 ns/op	76.234 ns/op	0.52
byteArrayEquals 16384 - diff last byte	25.276 us/op	34.985 us/op	0.72
Buffer.compare 16384 - diff last byte	180.71 ns/op	267.50 ns/op	0.68
byteArrayEquals 123687377 - diff last byte	195.82 ms/op	256.60 ms/op	0.76
Buffer.compare 123687377 - diff last byte	5.1256 ms/op	8.7266 ms/op	0.59
byteArrayEquals 32 - random bytes	4.7890 ns/op	5.8890 ns/op	0.81
Buffer.compare 32 - random bytes	40.283 ns/op	63.447 ns/op	0.63
byteArrayEquals 1024 - random bytes	4.5790 ns/op	5.5080 ns/op	0.83
Buffer.compare 1024 - random bytes	39.062 ns/op	60.960 ns/op	0.64
byteArrayEquals 16384 - random bytes	4.5650 ns/op	5.2360 ns/op	0.87
Buffer.compare 16384 - random bytes	39.145 ns/op	61.787 ns/op	0.63
byteArrayEquals 123687377 - random bytes	8.2300 ns/op	8.6500 ns/op	0.95
Buffer.compare 123687377 - random bytes	43.080 ns/op	64.250 ns/op	0.67
regular array get 100000 times	41.891 us/op	45.596 us/op	0.92
wrappedArray get 100000 times	42.451 us/op	46.005 us/op	0.92
arrayWithProxy get 100000 times	10.067 ms/op	14.750 ms/op	0.68
ssz.Root.equals	55.978 ns/op	54.514 ns/op	1.03
byteArrayEquals	55.614 ns/op	53.728 ns/op	1.04
Buffer.compare	9.5120 ns/op	11.081 ns/op	0.86
shuffle list - 16384 els	5.8837 ms/op	8.8014 ms/op	0.67
shuffle list - 250000 els	86.172 ms/op	133.10 ms/op	0.65
processSlot - 1 slots	15.824 us/op	22.011 us/op	0.72
processSlot - 32 slots	3.0807 ms/op	3.6393 ms/op	0.85
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	52.418 ms/op	63.205 ms/op	0.83
getCommitteeAssignments - req 1 vs - 250000 vc	2.4661 ms/op	2.7351 ms/op	0.90
getCommitteeAssignments - req 100 vs - 250000 vc	3.6256 ms/op	3.8954 ms/op	0.93
getCommitteeAssignments - req 1000 vs - 250000 vc	3.7302 ms/op	4.2608 ms/op	0.88
findModifiedValidators - 10000 modified validators	298.23 ms/op	410.87 ms/op	0.73
findModifiedValidators - 1000 modified validators	154.87 ms/op	235.58 ms/op	0.66
findModifiedValidators - 100 modified validators	147.06 ms/op	190.57 ms/op	0.77
findModifiedValidators - 10 modified validators	148.74 ms/op	193.01 ms/op	0.77
findModifiedValidators - 1 modified validators	135.88 ms/op	231.17 ms/op	0.59
findModifiedValidators - no difference	182.29 ms/op	215.84 ms/op	0.84
compare ViewDUs	3.9814 s/op	5.3908 s/op	0.74
compare each validator Uint8Array	893.41 ms/op	1.5551 s/op	0.57
compare ViewDU to Uint8Array	903.16 ms/op	1.4050 s/op	0.64
migrate state 1000000 validators, 24 modified, 0 new	729.28 ms/op	821.19 ms/op	0.89
migrate state 1000000 validators, 1700 modified, 1000 new	1.0853 s/op	1.0797 s/op	1.01
migrate state 1000000 validators, 3400 modified, 2000 new	1.5013 s/op	1.2682 s/op	1.18
migrate state 1500000 validators, 24 modified, 0 new	759.73 ms/op	752.75 ms/op	1.01
migrate state 1500000 validators, 1700 modified, 1000 new	1.0405 s/op	1.0783 s/op	0.96
migrate state 1500000 validators, 3400 modified, 2000 new	1.2600 s/op	1.3108 s/op	0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	5.2100 ns/op	4.4000 ns/op	1.18
state getBlockRootAtSlot - 250000 vs - 7PWei	960.66 ns/op	696.46 ns/op	1.38
computeProposers - vc 250000	6.6109 ms/op	9.0030 ms/op	0.73
computeEpochShuffling - vc 250000	87.521 ms/op	124.82 ms/op	0.70
getNextSyncCommittee - vc 250000	101.53 ms/op	148.58 ms/op	0.68
computeSigningRoot for AttestationData	23.986 us/op	28.096 us/op	0.85
hash AttestationData serialized data then Buffer.toString(base64)	1.2240 us/op	2.2610 us/op	0.54
toHexString serialized data	813.79 ns/op	1.0287 us/op	0.79
Buffer.toString(base64)	153.53 ns/op	213.90 ns/op	0.72

by benchmarkbot/action

packages/beacon-node/src/chain/bls/jobItem.ts

packages/beacon-node/src/chain/bls/utils.ts

packages/beacon-node/src/chain/options.ts

codecov · 2024-04-01T04:22:53Z

Codecov Report

Merging #6616 (0b8b4e8) into unstable (fd2fb16) will increase coverage by 0.00%.
Report is 1 commits behind head on unstable.
The diff coverage is 87.50%.

❗ Current head 0b8b4e8 differs from pull request most recent head b720eab. Consider uploading reports for the commit b720eab to get more accurate results

Additional details and impacted files

@@            Coverage Diff            @@
##           unstable    #6616   +/-   ##
=========================================
  Coverage     61.87%   61.87%           
=========================================
  Files           557      557           
  Lines         59201    59209    +8     
  Branches       1915     1915           
=========================================
+ Hits          36628    36635    +7     
- Misses        22530    22531    +1     
  Partials         43       43

lodestar

packages/beacon-node/src/chain/bls/jobItem.ts

packages/beacon-node/src/chain/bls/runBlsWorkReq.ts

packages/beacon-node/src/chain/bls/verifySets.ts

packages/beacon-node/src/chain/bls/jobItem.ts

twoeths

this looks great, looking forward to a release of bls to move this forward

packages/cli/src/preInitialization.ts

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

nflaig

LGTM

twoeths · 2024-04-24T02:04:23Z

@matthewkeil sim tests failed consistently, looks good to me other than that

matthewkeil · 2024-04-24T09:10:02Z

@matthewkeil sim tests failed consistently, looks good to me other than that

Finding lots and lots of tcp errors in the sim test logs....

[node-1-beacon-lodestar] [60618]: 2024-04-24T09:00:53.776Z libp2p:tcp:socket:error encrypting inbound connection to Error: Error occurred during XX handshake: read ECONNRESET failed
[node-1-beacon-lodestar] [60618]: 2024-04-24T09:00:53.776Z libp2p:tcp:socket:error failed to upgrade inbound connection CodeError: Error occurred during XX handshake: read ECONNRESET
    at DefaultUpgrader._encryptInbound (file:///Users/matthewkeil/Documents/dev/chainsafe/lodestar/node_modules/libp2p/dist/src/upgrader.js:492:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async DefaultUpgrader.upgradeInbound (file:///Users/matthewkeil/Documents/dev/chainsafe/lodestar/node_modules/libp2p/dist/src/upgrader.js:113:25) {
  code: 'ERR_ENCRYPTION_FAILED',
  props: {}
}

packages/beacon-node/src/chain/bls/multiThread.ts

matthewkeil · 2024-04-24T09:58:47Z

@nflaig your wish is my command. Moved the logging of libuv size back to the worker. Wasnt getting logged during tests and that needs to happen so its back "home" now

matthewkeil · 2024-04-24T10:08:53Z

@matthewkeil sim tests failed consistently, looks good to me other than that

@tuyennhv I merged in unstable again to get the old version of libp2p-tcp ~~and set the UV_THREADPOOL_SIZE=4 for sim tests~~

Fixed the issues I was seeing in the logs

wemeetagain

Behind this PR is hard work that's been honed and polished for months.
Great work @matthewkeil !

matthewkeil · 2024-04-24T13:22:45Z

Culmination of:
18,394 additions
34,709 deletions

(not including any intermediate branches or PR's, abandoned attempts or unused implementations)

This reverts commit 97d9aa8.

* Revert "feat: use @chainsafe/blst directly (#6706)" This reverts commit 66fe753. * Revert "feat: switch bls to napi rebuild (#6616)" This reverts commit 97d9aa8.

* chore: n-historical state - track state persist count per epoch (#6699) * feat: add lodestar binaries to release (#6666) * feat: added binaries creation support * feat: added sanity checks * chore: define reusable workflow * chore: fix sanity check,; disable arm64 * chore: added workflow_dispatch * fix: needs cleanup * chore: update old space size Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments * chore: address comments * chore: fixed typo * chore: typo Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments * chore: wording Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: use frozen lock file Co-authored-by: Nico Flaig <nflaig@protonmail.com> --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * feat: switch bls to napi rebuild (#6616) * chore: update package.json deps and run yarn * feat: remove unused blst config from vitest.browser.*.config.ts * test: move G2@INF test to base blst repo. No longer exported. * fix: add Uint8Array.from for Buffer return value in unit test so type matches for toEqual * refactor: update imports of CoordType to use bls instead of blst * feat: add blsAddVerificationRandomness cli option * feat: add randomBytesNonZero to utils * feat: implement verification randomness for same message * feat: implement napi blst and switch to libuv worker pool * chore: lint * fix: update comments for availableParallelism * feat: set UV_THREADPOOL_SIZE for perf tests * feat: set UV_THREADPOOL_SIZE=$(nproc) in lodestar entrance script * feat: remove case for serialized PublicKey in deserializeSet * feat: add back workBusy count logic to canAcceptWork * feat: automatically set threadpool size in beaconHandler * refactor: rename flag to disableSameMessageVerificationRandomness * fix: remove UV_THREADPOOL_SIZE from ./lodestar * docs: add TODO about updating metrics * fix: make benchmark multi threading cross-compatible * chore: add bls@8.0.0 * feat: default to added randomness * feat: add warning for no same message randomness * fix: passing to updated bls functions * feat: add blst peerDep * fix: default to same message randomness * chore: update yarn.lock * chore: remove unused function to clear lint * feat: break out setThreadPoolSize * docs: update packages/cli/src/cmds/beacon/setThreadPoolSize.ts Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: update location for blst dep * chore: run lint again * fix: remove peerDep from light-client * test: fix browser based tests * fix: bad relative imports * docs: add note to readme about switchable bls * docs: typo * chore: lint readme * fix: light-client unit tests * fix: light-client e2e test * feat: remove disableSameMessageVerificationRandomness * refactor: remove unused type * docs: update wordlist * fix: e2e tests * Lint package.json * Revert "fix: e2e tests" This reverts commit f71acd7. * Revert "fix: light-client e2e test" This reverts commit 23c11f0. * Revert "fix: light-client unit tests" This reverts commit fa7893e. * Revert "test: fix browser based tests" This reverts commit 98d04ab. * docs: update readme to remove switchable bls argument * fix: replace optimizeDeps in vitest.browser.config * chore: updated to bls 8.1.0 * feat: move setting threadpool size and warnings to applyPreset.ts * fix: remove setThreadpoolSize from beacon handler * feat: move libuv size logging into handler to use logger * refactor: rename applyPreset to preInitialization * refactor: logging about pool size * Update packages/cli/src/options/globalOptions.ts Co-authored-by: Nico Flaig <nflaig@protonmail.com> * refactor: rename prover applyPreset * docs: change comment in cli/index * fix: update logging for * fix: throw error for invalid uvThreadpoolSize * Update packages/cli/src/preInitialization.ts Co-authored-by: Nico Flaig <nflaig@protonmail.com> * Fix lint issues * Remove unused import * refactor: move bls pool logging back to multithread.ts * fix: log of threadcount --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: update references to consensus spec to v1.4.0 (#6704) * chore: update references to consensus spec to v1.4.0 * Remove unintended formatting changes * chore: update consensus spec version in README (#6705) * docs: correct plausible url (#6709) fix: correct plausible url * docs: incorrect scripts location (#6710) fix: incorrect scripts location * docs: no more broken images (#6708) * fix: no more broken images * fix: incorrect paths * feat: check aggregationBits length of `SignedAggregateAndProof` in gossip validation (#6700) * Add unit test * Update packages/beacon-node/src/chain/validation/aggregateAndProof.ts Co-authored-by: Nico Flaig <nflaig@protonmail.com> --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * feat: add and track blob source for metrics (#6628) * feat: add and track blob source for metrics * adding blobsSource for blobsPromise * address feedback * apply feedback * updates * only allow non blobspromise blocks to be imported * feat: add support for browser bundle for lightclient (#6673) * Reorganize the code so it is accessible from one package * Add support for browser build for lightclient * Update the build config * Improve the bls vite plugin * Restructure the vite and vitest scripts * Simplify vite config * Remove unused polyfill * Fix the doc lint error * Add support for bundle test * Update the package json files * Add dist build to default build task * Fix spelling in the docs * Fix the lint error * Fix type error * Disable eslint errors * Increase the timeout for bundle test * Fix eslint bundle * Fix lint warning * Remove the unused config * Add the default export to bundle * Enable compression on th build * Update packages/light-client/README.md Co-authored-by: Nico Flaig <nflaig@protonmail.com> * Increase timeout for one test * Optimize package build task * Update the readme --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * docs: fix plausible domain (#6712) * ci: add paths-ignore to workflows (#6713) * chore: added paths-ignore * chore: address comments * feat: add arm64 binary support (#6707) * feat: add arm64 binary support * chore: workaround yarn * chore: add comment * chore: workaround runner limitations * ci: publish binaries for release candidates (#6714) * feat: add types support to esm bundle for lightclient (#6685) * Add vite dts plugin * Restructure types export * Update vite build config * Remove unused file * Update the build process * Fix doc formatting * Build the dist for type tests * chore: update vitest (#6718) Update vitest * chore: do not build all binaries atomically (#6719) * feat: update libp2p deps (#6711) * feat: update libp2p deps * chore: fix linter/type errors * chore(deps): bump ejs from 3.1.9 to 3.1.10 (#6721) Bumps [ejs](https://github.com/mde/ejs) from 3.1.9 to 3.1.10. - [Release notes](https://github.com/mde/ejs/releases) - [Commits](mde/ejs@v3.1.9...v3.1.10) --- updated-dependencies: - dependency-name: ejs dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: revert "ci: add paths-ignore to workflows (#6713)" (#6730) Revert "ci: add paths-ignore to workflows (#6713)" This reverts commit e460bb1. * docs: restore docs pages lost during docusaurus migration (#6725) * ci: fix incorrect build caching (#6731) fix: incorrect build caching * fix: build error log during esm bundle build (#6737) Fix build error * chore: update fastify to v4.27.0 (#6740) * deps: upgrade vitest minor version (#6745) Upgrade vitest * chore: warn when heap limit is too low (#6722) * chore: warn when heap limit is too low * chore: units Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments * chore: added faq element * chore: address comments Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * feat: disable flood publish (#6755) * feat: disable flood publish * feat: add network.disableFloodPublish cli flag * chore: address PR comments * test: allow compact diff for test failures (#6783) * Allow the compact diff for tests * Update the doc * Update docs/pages/contribution/testing/index.md Co-authored-by: Julien <jeluard@users.noreply.github.com> * Fix linting on docs * Fix linting --------- Co-authored-by: Julien <jeluard@users.noreply.github.com> * chore: upgrade classic-level (#6793) * chore: upgrade classic-level * chore: address comments * feat(prover): support non-mutated verification provider in prover (#6727) * Restructure the assertions * Add an inspector to run the logic to detect providers * Update web3 provdier logic to use inspetor * Fix the types for proxy * Make the default type for mutation * Rename elrpc to elrpcprovider * Apply suggestions from code review Co-authored-by: Julien <jeluard@users.noreply.github.com> * Fix build error * Update the readme doc * Apply suggestions from code review Co-authored-by: Julien <jeluard@users.noreply.github.com> * Fix the docs linting * Add missing words --------- Co-authored-by: Julien <jeluard@users.noreply.github.com> * docs: added debugging section (#6743) * docs: added debugging section * chore: spell checks * chore: added extra docs * chore: address comments * chore: added extra configuration * chore: updated docs * chore: added extra configuration * chore: fix lint * chore: fix typos * chore: .gitignore * chore: address comments * chore: address comments * chore: review debugging section PR (#6807) * chore: update the naming conventions used in sim tests (#6805) * Rename simulation test to crucible * Rename SimulationEnvironment to Simulation * Use consistent function names * Update readme * Rename interfaces for consistent pattern * Fix lint error * feat: recognize grandine from agentVersion (#6808) * Add grandine * add grandine to wordlist * chore: refactor event loop interactions (#6806) * chore: refactor sleep(0) usage * chore: refactor setTimeout * chore: address comments * chore: cleanup * Apply suggestions from code review --------- Co-authored-by: Cayman <caymannava@gmail.com> * chore: fix broken docker (#6813) * chore: log newPayload engine api (#6810) * chore: add `get_proposer_head` check in fork choice spec test (#6814) * Check get_proposer_head * Fix enum typo * lint * chore: consolidate classic-level usage (#6795) * chore: do not rely on leveldown * chore: replace level with classic-level * chore: added docker support for osx (#6696) * chore: added docker support for osx * chore: address comments * chore: address comments * Update docker-compose.yml Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: address comments --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: fix vitest performance with spec tests artifacts in place (#6817) Fix vitest performance with spec tests in place * chore: clean up yarn lockfile (#6816) * feat: disable flood publish by default (#6815) * feat: use @chainsafe/blst directly (#6706) * feat: use @chainsafe/blst directly * chore: update to blst@1.0.1 * refactor: remove randomBytesNonZero and user blst exported version * chore: update blst references * test: catch invalid deserialization in spec tests and return false * feat: create signatureFromBytes and signatureFromBytesNoCheck in utils package * feat: implement signatureFromBytes from utils package * feat: implement signatureFromBytes everywhere * fix: light-client empty module for blst --------- Co-authored-by: matthewkeil <me@matthewkeil.com> * fix: avoid Buffer.from copies (#6723) * fix: avoid Buffer.from copies * chore: simplify shuffling * fix: use subarray instead of slice in shuffling * chore: remove unnecessary devDependencies * chore: rely on fastify 4.x behavior * chore: avoid copy in verifyMerkleBranch * use toBase64 * relax assertions in shuffle function * Update packages/state-transition/src/util/shuffle.ts Co-authored-by: twoeths <tuyen@chainsafe.io> --------- Co-authored-by: twoeths <tuyen@chainsafe.io> * chore: update nodejs to latest 22 (#6729) * Upgrade node js version to 22 * Update node types * Revert action config * Add package from git hash * Fix the build error * Update the docs for node-22 * Update docker version for Nodejs to 22 * Update package.json * Update package.json Co-authored-by: Nico Flaig <nflaig@protonmail.com> * Update readme docs * Add word in dictionary * Add word in dictionary --------- Co-authored-by: Cayman <caymannava@gmail.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com> * deps: update systeminformation to 5.22.9 (#6823) * chore: add distutils to build phase of Dockerfile (#6845) * test: increase timeout of keystore tests (#6846) * test: increase timeout of keystore cache tests * Increase hook timeout * Consistent number formatting * chore: only log warning if media type is not supported (415) (#6847) * fix: revert napi bls (#6853) * Revert "feat: use @chainsafe/blst directly (#6706)" This reverts commit 66fe753. * Revert "feat: switch bls to napi rebuild (#6616)" This reverts commit 97d9aa8. * chore: bump package versions to 1.19.0 * chore: bump @chainsafe/blst to v0.2.11 (#6856) * fix: arm64 binaries creation issue (#6858) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: tuyennhv <tuyen@chainsafe.io> Co-authored-by: Julien <jeluard@users.noreply.github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com> Co-authored-by: Matthew Keil <me@matthewkeil.com> Co-authored-by: NC <adrninistrator1@protonmail.com> Co-authored-by: g11tech <develop@g11tech.io> Co-authored-by: Nazar Hussain <nazarhussain@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

wemeetagain · 2024-06-07T11:56:56Z

🎉 This PR is included in v1.19.0 🎉

matthewkeil added 13 commits March 31, 2024 13:58

chore: update package.json deps and run yarn

a74b8f5

feat: remove unused blst config from vitest.browser.*.config.ts

f692d1b

test: move G2@INF test to base blst repo. No longer exported.

e2dfc26

fix: add Uint8Array.from for Buffer return value in unit test so type…

923daad

… matches for toEqual

refactor: update imports of CoordType to use bls instead of blst

f53a496

feat: add blsAddVerificationRandomness cli option

03be472

feat: add randomBytesNonZero to utils

bbd3436

feat: implement verification randomness for same message

0b363ae

feat: implement napi blst and switch to libuv worker pool

23f263c

chore: lint

d152dfc

fix: update comments for availableParallelism

8dbc8cc

feat: set UV_THREADPOOL_SIZE for perf tests

c2158f2

feat: set UV_THREADPOOL_SIZE=$(nproc) in lodestar entrance script

01cfbac

matthewkeil requested a review from a team as a code owner March 31, 2024 16:11

twoeths reviewed Apr 1, 2024

View reviewed changes

packages/beacon-node/src/chain/bls/jobItem.ts Outdated Show resolved Hide resolved

feat: remove case for serialized PublicKey in deserializeSet

de1e3c5

twoeths reviewed Apr 1, 2024

View reviewed changes

packages/beacon-node/src/chain/bls/utils.ts Outdated Show resolved Hide resolved

twoeths reviewed Apr 1, 2024

View reviewed changes

packages/beacon-node/src/chain/options.ts Outdated Show resolved Hide resolved

nflaig reviewed Apr 1, 2024

View reviewed changes

lodestar Outdated Show resolved Hide resolved

matthewkeil marked this pull request as draft April 1, 2024 10:29

matthewkeil added 2 commits April 3, 2024 11:07

feat: add back workBusy count logic to canAcceptWork

efede8c

feat: automatically set threadpool size in beaconHandler

9757f43

matthewkeil marked this pull request as ready for review April 4, 2024 16:16