-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement PeerDAS #14129
base: develop
Are you sure you want to change the base?
Implement PeerDAS #14129
Commits on Oct 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 5e55834 - Browse repository at this point
Copy the full SHA 5e55834View commit details -
Configuration menu - View commit details
-
Copy full SHA for aef68db - Browse repository at this point
Copy the full SHA aef68dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for ac905f6 - Browse repository at this point
Copy the full SHA ac905f6View commit details -
Add Support For Discovery Of Column Subnets (#13883)
* Add Support For Discovery Of Column Subnets * Lint for SubnetsPerNode * Manu's Review * Change to a better name
Configuration menu - View commit details
-
Copy full SHA for dee92e5 - Browse repository at this point
Copy the full SHA dee92e5View commit details -
Add Data Column Gossip Handlers (#13894)
* Add Data Column Subscriber * Add Data Column Vaidator * Wire all Handlers In * Fix Build * Fix Test * Fix IP in Test * Fix IP in Test
Configuration menu - View commit details
-
Copy full SHA for ab917ba - Browse repository at this point
Copy the full SHA ab917baView commit details -
Add Request And Response RPC Methods For Data Columns (#13909)
* Add RPC Handler * Add Column Requests * Update beacon-chain/db/filesystem/blob.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Update beacon-chain/p2p/rpc_topic_mappings.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Manu's Review * Manu's Review * Interface Fixes * mock manager --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for ee2d0f4 - Browse repository at this point
Copy the full SHA ee2d0f4View commit details -
* Bump `c-kzg-4844` lib to the `das` branch. * Implement `MerkleProofKZGCommitments`. * Implement `das-core.md`. * Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`. * `CustodyColumnSubnets`: Include `i` in the for loop. * Remove `computeSubscribedColumnSubnet`. * Remove `peerdas.CustodyColumns` out of the for loop.
Configuration menu - View commit details
-
Copy full SHA for bb70e1a - Browse repository at this point
Copy the full SHA bb70e1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 325a5b0 - Browse repository at this point
Copy the full SHA 325a5b0View commit details
Commits on Oct 24, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b891d58 - Browse repository at this point
Copy the full SHA b891d58View commit details -
Implement peer DAS proposer RPC (#13922)
* Remove capital letter from error messages. * `[4]byte` => `[fieldparams.VersionLength]byte`. * Prometheus: Remove extra `committee`. They are probably due to a bad copy/paste. Note: The name of the probe itself is remaining, to ensure backward compatibility. * Implement Proposer RPC for data columns. * Fix TestProposer_ProposeBlock_OK test. * Remove default peerDAS activation. * `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
Configuration menu - View commit details
-
Copy full SHA for a574b87 - Browse repository at this point
Copy the full SHA a574b87View commit details -
Add DA Check For Data Columns (#13938)
* Add new DA check * Exit early in the event no commitments exist. * Gazelle * Fix Mock Broadcaster * Fix Test Setup * Update beacon-chain/blockchain/process_block.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Manu's Review * Fix Build --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for 4b2d7bb - Browse repository at this point
Copy the full SHA 4b2d7bbView commit details -
* Update `consensus_spec_version` to `v1.5.0-alpha.1`. * `CustodyColumns`: Fix and implement spec tests. * Make deepsource happy. * `^uint64(0)` => `math.MaxUint64`. * Fix `TestLoadConfigFile` test.
Configuration menu - View commit details
-
Copy full SHA for 004f913 - Browse repository at this point
Copy the full SHA 004f913View commit details -
SendDataColumnSidecarByRoot
: ReturnRODataColumn
instead of `ROBl……ob`. (#13957) * `SendDataColumnSidecarByRoot`: Return `RODataColumn` instead of `ROBlob`. * Make deepsource happier.
Configuration menu - View commit details
-
Copy full SHA for a50d7aa - Browse repository at this point
Copy the full SHA a50d7aaView commit details -
[PeerDAS] Upgrade c-kzg-4844 package (#13967)
* Upgrade c-kzg-4844 package * Upgrade bazel deps
Configuration menu - View commit details
-
Copy full SHA for 9802e50 - Browse repository at this point
Copy the full SHA 9802e50View commit details -
Enable E2E For PeerDAS (#13945)
* Enable E2E And Add Fixes * Register Same Topic For Data Columns * Initialize Capacity Of Slice * Fix Initialization of Data Column Receiver * Remove Mix In From Merkle Proof * E2E: Subscribe to all subnets. * Remove Index Check * Remaining Bug Fixes to Get It Working * Change Evaluator to Allow Test to Finish * Fix Build * Add Data Column Verification * Fix LoopVar Bug * Do Not Allocate Memory * Update beacon-chain/blockchain/process_block.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Update beacon-chain/core/peerdas/helpers.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Update beacon-chain/core/peerdas/helpers.go Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com> * Gofmt * Fix It Again * Fix Test Setup * Fix Build * Fix Trusted Setup panic * Fix Trusted Setup panic * Use New Test --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for b622e22 - Browse repository at this point
Copy the full SHA b622e22View commit details -
Implement Data Columns By Range Request And Response Methods (#13972)
* Add Data Structure for New Request Type * Add Data Column By Range Handler * Add Data Column Request Methods * Add new validation for columns by range requests * Fix Build * Allow Prysm Node To Fetch Data Columns * Allow Prysm Node To Fetch Data Columns And Sync * Bug Fixes For Interop * GoFmt * Use different var * Manu's Review
Configuration menu - View commit details
-
Copy full SHA for 8bf9481 - Browse repository at this point
Copy the full SHA 8bf9481View commit details -
Sample from peers some data columns. (#13980)
* PeerDAS: Implement sampling. * `TestNewRateLimiter`: Fix with the new number of expected registered topics.
Configuration menu - View commit details
-
Copy full SHA for cf42d6c - Browse repository at this point
Copy the full SHA cf42d6cView commit details -
Set Custody Count Correctly (#14004)
* Set Custody Count Correctly * Fix Discovery Count
Configuration menu - View commit details
-
Copy full SHA for f5acce7 - Browse repository at this point
Copy the full SHA f5acce7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5072f2d - Browse repository at this point
Copy the full SHA 5072f2dView commit details -
Fix
CustodyColumns
to comply with alpha-2 spectests. (#14008)* Adding error wrapping * Fix `CustodyColumnSubnets` tests.
Configuration menu - View commit details
-
Copy full SHA for 3ddcbe6 - Browse repository at this point
Copy the full SHA 3ddcbe6View commit details -
Request Data Columns When Fetching Pending Blocks (#14007)
* Support Data Columns For By Root Requests * Revert Config Changes * Fix Panic * Fix Process Block * Fix Flags * Lint * Support Checkpoint Sync * Manu's Review * Add Support For Columns in Remaining Methods * Unmarshal Uncorrectly
Configuration menu - View commit details
-
Copy full SHA for 6fd443a - Browse repository at this point
Copy the full SHA 6fd443aView commit details -
Disable Evaluators For E2E (#14019)
* Hack E2E * Fix it For Real * Gofmt * Remove
Configuration menu - View commit details
-
Copy full SHA for 904b782 - Browse repository at this point
Copy the full SHA 904b782View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f80111 - Browse repository at this point
Copy the full SHA 4f80111View commit details -
PeerDAS: Implement reconstruction. (#14036)
* Wrap errors, add logs. * `missingColumnRequest`: Fix blobs <-> data columns mix. * `ColumnIndices`: Return `map[uint64]bool` instead of `[fieldparams.NumberOfColumns]bool`. * `DataColumnSidecars`: `interfaces.SignedBeaconBlock` ==> `interfaces.ReadOnlySignedBeaconBlock`. We don't need any of the non read-only methods. * Fix comments. * `handleUnblidedBlock` ==> `handleUnblindedBlock`. * `SaveDataColumn`: Move log from debug to trace. If we attempt to save an already existing data column sidecar, a debug log was printed. This case could be quite common now with the data column reconstruction enabled. * `sampling_data_columns.go` --> `data_columns_sampling.go`. * Reconstruct data columns.
Configuration menu - View commit details
-
Copy full SHA for 5615f09 - Browse repository at this point
Copy the full SHA 5615f09View commit details -
PeerDAS: Implement / use data column feed from database. (#14062)
* Remove some `_` identifiers. * Blob storage: Implement a notifier system for data columns. * `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`. * Address Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for e7f4807 - Browse repository at this point
Copy the full SHA e7f4807View commit details -
PeerDAS: Withhold data on purpose. (#14076)
* Introduce hidden flag `data-columns-withhold-count`. * Address Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for d72ce70 - Browse repository at this point
Copy the full SHA d72ce70View commit details -
recoverBlobs
: Cover the `0 < blobsCount < fieldparams.MaxBlobsPerBl……ock` case. (#14066) * `recoverBlobs`: Cover the `0 < blobsCount < fieldparams.MaxBlobsPerBlock` case. * Fix Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for b431acd - Browse repository at this point
Copy the full SHA b431acdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 87152c0 - Browse repository at this point
Copy the full SHA 87152c0View commit details -
PeerDAS: Gossip the reconstructed columns (#14079)
* PeerDAS: Broadcast not seen via gossip but reconstructed data columns. * Address Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for 96f57c3 - Browse repository at this point
Copy the full SHA 96f57c3View commit details -
PeerDAS: Stop generating new P2P private key at start. (#14099)
* `privKey`: Improve logs. * peerDAS: Move functions in file. Add documentation. * PeerDAS: Remove unused `ComputeExtendedMatrix` and `RecoverMatrix` functions. * PeerDAS: Stop generating new P2P private key at start. * Fix sammy' comment.
Configuration menu - View commit details
-
Copy full SHA for 5ae7885 - Browse repository at this point
Copy the full SHA 5ae7885View commit details -
[PeerDAS] rework ENR custody_subnet_count and add tests (#14077)
* [PeerDAS] rework ENR custody_subnet_count related code * update according to proposed spec change * Run gazelle
Configuration menu - View commit details
-
Copy full SHA for bae92a7 - Browse repository at this point
Copy the full SHA bae92a7View commit details -
[PeerDAS] fixes and tests for gossiping out data columns (#14102)
* [PeerDAS] Minor fixes and tests for gossiping out data columns * Fix metrics
Configuration menu - View commit details
-
Copy full SHA for 68e25dd - Browse repository at this point
Copy the full SHA 68e25ddView commit details -
[PeerDAS] implement DataColumnSidecarsByRootReq and fix related bugs (#…
…14103) * [PeerDAS] add data column related protos and fix data column by root bug * Add more tests
Configuration menu - View commit details
-
Copy full SHA for cf90fae - Browse repository at this point
Copy the full SHA cf90faeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9cf8cfc - Browse repository at this point
Copy the full SHA 9cf8cfcView commit details -
[PeerDAS] rework ENR custody_subnet_count and add tests (#14077)
* [PeerDAS] rework ENR custody_subnet_count related code * update according to proposed spec change * Run gazelle
Configuration menu - View commit details
-
Copy full SHA for b5c798b - Browse repository at this point
Copy the full SHA b5c798bView commit details -
Configuration menu - View commit details
-
Copy full SHA for bbf8abe - Browse repository at this point
Copy the full SHA bbf8abeView commit details -
PeerDAS: Implement IncrementalDAS (#14109)
* `ConvertPeerIDToNodeID`: Add tests. * Remove `extractNodeID` and uses `ConvertPeerIDToNodeID` instead. * Implement IncrementalDAS. * `DataColumnSamplingLoop` ==> `DataColumnSamplingRoutine`. * HypergeomCDF: Add test. * `GetValidCustodyPeers`: Optimize and add tests. * Remove blank identifiers. * Implement `CustodyCountFromRecord`. * Implement `TestP2P.CustodyCountFromRemotePeer`. * `NewTestP2P`: Add `swarmt.Option` parameters. * `incrementalDAS`: Rework and add tests. * Remove useless warning.
Configuration menu - View commit details
-
Copy full SHA for 66a37a6 - Browse repository at this point
Copy the full SHA 66a37a6View commit details -
chore: Encapsulate all kzg functionality for PeerDAS into the kzg pac…
…kage (#14136) * chore: move all ckzg related functionality into kzg package * refactor code to match * run: bazel run //:gazelle -- fix * chore: add some docs and stop copying large objects when converting between types * fixes * manually add kzg.go dep to Build.Hazel * move kzg methods to kzg.go * chore: add RecoverCellsAndProofs method * bazel run //:gazelle -- fix * use BytesPerBlob constant * chore: fix some deepsource issues * one declaration for commans and blobs
Configuration menu - View commit details
-
Copy full SHA for eb5c5e0 - Browse repository at this point
Copy the full SHA eb5c5e0View commit details -
chore!: Refactor
RecoverBlob
toRecoverCellsAndProofs
(#14160)* change recoverBlobs to recoverCellsAndProofs * modify code to take in the cells and proofs for a particular blob instead of the blob itself * add CellsAndProofs structure * modify recoverCellsAndProofs to return `cellsAndProofs` structure * modify `DataColumnSidecarsForReconstruct` to accept the `cellsAndKZGProofs` structure * bazel run //:gazelle -- fix * use kzg abstraction for kzg method * move CellsAndProofs to kzg.go
Configuration menu - View commit details
-
Copy full SHA for 6b18fb8 - Browse repository at this point
Copy the full SHA 6b18fb8View commit details -
Activate PeerDAS with the EIP7594 Fork Epoch (#14184)
* Save All the Current Changes * Add check for data sampling * Fix Test * Gazelle * Manu's Review * Fix Test
Configuration menu - View commit details
-
Copy full SHA for d20e441 - Browse repository at this point
Copy the full SHA d20e441View commit details -
Move log from error to debug. (#14194)
Reason: If a peer does not exposes its `csc` field into it's ENR, then there is nothing we can do.
Configuration menu - View commit details
-
Copy full SHA for 9100fe7 - Browse repository at this point
Copy the full SHA 9100fe7View commit details -
chore!: Make Cell be a flat sequence of bytes (#14159)
* chore: move all ckzg related functionality into kzg package * refactor code to match * run: bazel run //:gazelle -- fix * chore: add some docs and stop copying large objects when converting between types * fixes * manually add kzg.go dep to Build.Hazel * move kzg methods to kzg.go * chore: add RecoverCellsAndProofs method * bazel run //:gazelle -- fix * make Cells be flattened sequence of bytes * chore: add test for flattening roundtrip * chore: remove code that was doing the flattening outside of the kzg package * fix merge * fix * remove now un-needed conversion * use pointers for Cell parameters * linter * rename cell conversion methods (this only applies to old version of c-kzg)
Configuration menu - View commit details
-
Copy full SHA for 1cd0555 - Browse repository at this point
Copy the full SHA 1cd0555View commit details -
PeerDAS: Add KZG verification when sampling (#14187)
* `validateDataColumn`: Add comments and remove debug computation. * `sampleDataColumnsFromPeer`: Add KZG verification * `VerifyKZGInclusionProofColumn`: Add unit test. * Make deepsource happy. * Address Nishant's comment. * Address Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for 8a218d7 - Browse repository at this point
Copy the full SHA 8a218d7View commit details -
Trigger PeerDAS At Deneb For E2E (#14193)
* Trigger At Deneb * Fix Rate Limits
Configuration menu - View commit details
-
Copy full SHA for a11636a - Browse repository at this point
Copy the full SHA a11636aView commit details -
chore!: Use
RecoverCellsAndKZGProofs
instead ofRecoverAllCells
……-> `CellsToBlob` -> `ComputeCellsAndKZGProofs` (#14183) * use recoverCellsAndKZGProofs * make recoverAllCells and CellsToBlob private * chore: all methods now return CellsAndProof struct * chore: update code
Configuration menu - View commit details
-
Copy full SHA for 2ef0725 - Browse repository at this point
Copy the full SHA 2ef0725View commit details -
[PeerDAS] Parallelize data column sampling (#14105)
* PeerDAS: parallelizing sample queries * PeerDAS: select sample from non custodied columns * Finish rebase * Add more test cases
Configuration menu - View commit details
-
Copy full SHA for 1d83f33 - Browse repository at this point
Copy the full SHA 1d83f33View commit details -
Configuration menu - View commit details
-
Copy full SHA for e59d059 - Browse repository at this point
Copy the full SHA e59d059View commit details -
Configuration menu - View commit details
-
Copy full SHA for f58c309 - Browse repository at this point
Copy the full SHA f58c309View commit details -
Update ckzg4844 to latest version of das branch (#14223)
* Update ckzg4844 to latest version * Run go mod tidy * Remove unnecessary tests & run goimports * Remove fieldparams from blockchain/kzg * Add back blank line * Avoid large copies * Run gazelle * Use trusted setup from the specs & fix issue with struct * Run goimports * Fix mistake in makeCellsAndProofs --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for 18c1f8f - Browse repository at this point
Copy the full SHA 18c1f8fView commit details -
PeerDAS: Run reconstruction in parallel. (#14236)
* PeerDAS: Run reconstruction in parallel. * `isDataAvailableDataColumns` --> `isDataColumnsAvailable` * `isDataColumnsAvailable`: Return `nil` as soon as half of the columns are received. * Make deepsource happy.
Configuration menu - View commit details
-
Copy full SHA for aa658e0 - Browse repository at this point
Copy the full SHA aa658e0View commit details -
Make deepsource happy (#14237)
* DeepSource: Pass heavy objects by pointers. * `removeBlockFromQueue`: Remove redundant error checking. * `fetchBlobsFromPeer`: Use same variable for `append`. * Remove unused arguments. * Combine types. * `Persist`: Add documentation. * Remove unused receiver * Remove duplicated import. * Stop using both pointer and value receiver at the same time. * `verifyAndPopulateColumns`: Remove unused parameter * Stop using mpty slice literal used to declare a variable.
Configuration menu - View commit details
-
Copy full SHA for bf0342a - Browse repository at this point
Copy the full SHA bf0342aView commit details -
PeerDAS: Fix initial sync (#14208)
* `SendDataColumnsByRangeRequest`: Add some new fields in logs. * `BlobStorageSummary`: Implement `HasDataColumnIndex` and `AllDataColumnsAvailable`. * Implement `fetchDataColumnsFromPeers`. * `fetchBlobsFromPeer`: Return only one error.
Configuration menu - View commit details
-
Copy full SHA for 68721e8 - Browse repository at this point
Copy the full SHA 68721e8View commit details -
Fix data columns sampling (#14263)
* Fix the obvious... * Data columns sampling: Modify logging. * `waitForChainStart`: Set it threadsafe - Do only wait once. * Sampling: Wait for chain start before running the sampling. Reason: `newDataColumnSampler1D` needs `s.ctxMap`. `s.ctxMap` is only set when chain is started. Previously `waitForChainStart` was only called in `s.registerHandlers`, it self called in a go-routine. ==> We had a race condition here: Sometimes `newDataColumnSampler1D` were called once `s.ctxMap` were set, sometimes not. * Adresse Nishant's comments. * Sampling: Improve logging. * `waitForChainStart`: Remove `chainIsStarted` check.
Configuration menu - View commit details
-
Copy full SHA for 9ae8492 - Browse repository at this point
Copy the full SHA 9ae8492View commit details -
PeerDAS: Add
MetadataV3
withcustody_subnet_count
(#14274)* `sendPingRequest`: Add some comments. * `sendPingRequest`: Replace `stream.Conn().RemotePeer()` by `peerID`. * `pingHandler`: Add comments. * `sendMetaDataRequest`: Add comments and implement an unique test. * Gather `SchemaVersion`s in the same `const` definition. * Define `SchemaVersionV3`. * `MetaDataV1`: Fix comment. * Proto: Define `MetaDataV2`. * `MetaDataV2`: Generate SSZ. * `newColumnSubnetIDs`: Use smaller lines. * `metaDataHandler` and `sendMetaDataRequest`: Manage `MetaDataV2`. * `RefreshPersistentSubnets`: Refactor tests (no functional change). * `RefreshPersistentSubnets`: Refactor and add comments (no functional change). * `RefreshPersistentSubnets`: Compare cache with both ENR & metadata. * `RefreshPersistentSubnets`: Manage peerDAS. * `registerRPCHandlersPeerDAS`: Register `RPCMetaDataTopicV3`. * `CustodyCountFromRemotePeer`: Retrieve the count from metadata. Then default to ENR, then default to the default value. * Update beacon-chain/sync/rpc_metadata.go Co-authored-by: Nishant Das <nishdas93@gmail.com> * Fix duplicate case. * Remove version testing. * `debug.proto`: Stop breaking ordering. --------- Co-authored-by: Nishant Das <nishdas93@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 8c5dac1 - Browse repository at this point
Copy the full SHA 8c5dac1View commit details -
Add Data Column Verification (#14287)
* Persist All Changes * Fix All Tests * Fix Build * Fix Build * Fix Build * Fix Test Again * Add missing verification * Add Test Cases for Data Column Validation * Fix comments for methods * Fix comments for methods * Fix Test * Manu's Review
Configuration menu - View commit details
-
Copy full SHA for 75f06f4 - Browse repository at this point
Copy the full SHA 75f06f4View commit details -
Implement
/eth/v1/beacon/blob_sidecars/{block_id}
for peerDAS. (#14312) * `parseIndices`: `O(n**2)` ==> `O(n)`. * PeerDAS: Implement `/eth/v1/beacon/blob_sidecars/{block_id}`. * Update beacon-chain/core/peerdas/helpers.go Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com> * Rename some functions. * `Blobs`: Fix empty slice. * `recoverCellsAndProofs` --> Move function in `beacon-chain/core/peerdas`. * peerDAS helpers: Add missing tests. * Implement `CustodyColumnCount`. * `RecoverCellsAndProofs`: Remove useless argument `columnsCount`. * Tests: Add cleanups. * `blobsFromStoredDataColumns`: Reconstruct if needed. * Make deepsource happy. * Beacon API: Use provided indices. * Make deepsource happier. --------- Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for afc69af - Browse repository at this point
Copy the full SHA afc69afView commit details -
* Update go.yml * Disable mnd * Update .golangci.yml * Update go.yml * Update go.yml * Update .golangci.yml * Update go.yml * Fix Lint Issues * Remove comment * Update .golangci.yml
Configuration menu - View commit details
-
Copy full SHA for 1304eb8 - Browse repository at this point
Copy the full SHA 1304eb8View commit details -
Update Config To Latest Value (#14352)
* Update values * Update Spec To v1.5.0-alpha.5 * Fix Discovery Tests * Hardcode Subnet Count For Tests * Fix All Initial Sync Tests * Gazelle * Less Chaotic Service Initialization * Gazelle
Configuration menu - View commit details
-
Copy full SHA for edae704 - Browse repository at this point
Copy the full SHA edae704View commit details -
Use Data Column Validation Across Prysm (#14377)
* Use Data Column Validation Everywhere * Fix Build * Fix Lint * Fix Clock Synchronizer * Fix Panic
Configuration menu - View commit details
-
Copy full SHA for b14e269 - Browse repository at this point
Copy the full SHA b14e269View commit details -
Change Custody Count to Uint8 (#14386)
* Add Changes for Uint8 Csc * Fix Build * Fix Build for Sync * Fix Discovery Test
Configuration menu - View commit details
-
Copy full SHA for b242f40 - Browse repository at this point
Copy the full SHA b242f40View commit details -
Fix Bugs in PeerDAS Testing (#14396)
* Fix Various Bugs in PeerDAS * Remove Log * Remove useless copy var. --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for a7ee0e7 - Browse repository at this point
Copy the full SHA a7ee0e7View commit details -
PeerDAS: Improve logging and reduce the number of needed goroutines f…
…or reconstruction (#14397) * `broadcastAndReceiveDataColumns`: Use real `sidecar.ColumnIndex` instead of position in the slice. And improve logging as well. * `isDataColumnsAvailable`: Improve logging. * `validateDataColumn`: Print `Accepted data column sidecar gossip` really at the end. * Subscriber: Improve logging. * `sendAndSaveDataColumnSidecars`: Use common used function for logging. * `dataColumnSidecarByRootRPCHandler`: Logging - Pring `all` instead of all the columns for a super node. * Verification: Improve logging. * `DataColumnsWithholdCount`: Set as `uint64` instead `int`. * `DataColumnFields`: Improve logging. * Logging: Remove now useless private `columnFields`function. * Avoid useless goroutines blocking for reconstruction. * Update beacon-chain/sync/subscriber.go Co-authored-by: Nishant Das <nishdas93@gmail.com> * Address Nishant's comment. * Improve logging. --------- Co-authored-by: Nishant Das <nishdas93@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 155ac62 - Browse repository at this point
Copy the full SHA 155ac62View commit details -
Configuration menu - View commit details
-
Copy full SHA for bbac508 - Browse repository at this point
Copy the full SHA bbac508View commit details -
Add Data Column Computation Metrics (#14400)
* Add Data Column Metrics * Shift it All To Peerdas Package
Configuration menu - View commit details
-
Copy full SHA for f1d49a9 - Browse repository at this point
Copy the full SHA f1d49a9View commit details -
Fix Initial Sync with 128 data columns subnets (#14403)
* `pingPeers`: Add log with new ENR when modified. * `p2p Start`: Use idiomatic go error syntax. * P2P `start`: Fix error message. * Use not bootnodes at all if the `--chain-config-file` flag is used and no `--bootstrap-node` flag is used. Before this commit, if the `--chain-config-file` flag is used and no `--bootstrap-node` flag is used, then bootnodes are (incorrectly) defaulted on `mainnet` ones. * `validPeersExist`: Centralize logs. * `AddConnectionHandler`: Improve logging. "Peer connected" does not really reflect the fact that a new peer is actually connected. --> "New peer connection" is more clear. Also, instead of writing `0`, `1`or `2` for direction, now it's writted "Unknown", "Inbound", "Outbound". * Logging: Add 2 decimals for timestamt in text and JSON logs. * Improve "no valid peers" logging. * Improve "Some columns have no peers responsible for custody" logging. * `pubsubSubscriptionRequestLimit`: Increase to be consistent with data columns. * `sendPingRequest`: Improve logging. * `FindPeersWithSubnet`: Regularly recheck in our current set of peers if we have enough peers for this topic. Before this commit, new peers HAD to be found, even if current peers are eventually acceptable. For very small network, it used to lead to infinite search. * `subscribeDynamicWithSyncSubnets`: Use exactly the same subscription function initially and every slot. * Make deepsource happier. * Nishant's commend: Change peer disconnected log. * NIshant's comment: Change `Too many incoming subscription` log from error to debug. * `FindPeersWithSubnet`: Address Nishant's comment. * `batchSize`: Address Nishant's comment. * `pingPeers` ==> `pingPeersAndLogEnr`. * Update beacon-chain/sync/subscriber.go Co-authored-by: Nishant Das <nishdas93@gmail.com> --------- Co-authored-by: Nishant Das <nishdas93@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for d4b3a65 - Browse repository at this point
Copy the full SHA d4b3a65View commit details -
Configuration menu - View commit details
-
Copy full SHA for db424df - Browse repository at this point
Copy the full SHA db424dfView commit details -
Fix CPU usage in small devnets (#14446)
* `CustodyCountFromRemotePeer`: Set happy path in the outer scope. * `FindPeersWithSubnet`: Improve logging. * `listenForNewNodes`: Avoid infinite loop in a small subnet. * Address Nishant's comment. * FIx Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for 4bf2c27 - Browse repository at this point
Copy the full SHA 4bf2c27View commit details -
Peerdas: Full subnet sampling and
sendBatchRootRequest
fix. (#14452)* `sendBatchRootRequest`: Refactor and add comments. * `sendBatchRootRequest`: Do send requests to peers that custodies a superset of our columns. Before this commit, we sent "data columns by root requests" for data columns peers do not custody. * Data columns: Use subnet sampling only. (Instead of peer sampling.) aaa * `areDataColumnsAvailable`: Improve logs. * `GetBeaconBlock`: Improve logs. Rationale: A `begin` log should always be followed by a `success` log or a `failure` log.
Configuration menu - View commit details
-
Copy full SHA for 6b1b04e - Browse repository at this point
Copy the full SHA 6b1b04eView commit details -
Peerdas: Several updates (#14459)
* `validateDataColumn`: Refactor logging. * `dataColumnSidecarByRootRPCHandler`: Improve logging. * `isDataAvailable`: Improve logging. * Add hidden debug flag: `--data-columns-reject-slot-multiple`. * Add more logs about peer disconnection. * `validPeersExist` --> `enoughPeersAreConnected` * `beaconBlocksByRangeRPCHandler`: Add remote Peer ID in logs. * Stop calling twice `writeErrorResponseToStream` in case of rate limit.
Configuration menu - View commit details
-
Copy full SHA for 5ca87d5 - Browse repository at this point
Copy the full SHA 5ca87d5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4675f9e - Browse repository at this point
Copy the full SHA 4675f9eView commit details -
PeerDAS: Multiple improvements (#14467)
* `scheduleReconstructedDataColumnsBroadcast`: Really minor refactor. * `receivedDataColumnsFromRootLock` -> `dataColumnsFromRootLock` * `reconstructDataColumns`: Stop looking into the DB to know if we have some columns. Before this commit: Each time we receive a column, we look into the filesystem for all columns we store. ==> For 128 columns, it looks for 1 + 2 + 3 + ... + 128 = 128(128+1)/2 = 8256 files look. Also, as soon as a column is saved into the file system, then if, right after, we look at the filesystem again, we assume the column will be available (strict consistency). It happens not to be always true. ==> Sometimes, we can reconstruct and reseed columns more than once, because of this lack of filesystem strict consistency. After this commit: We use a (strictly consistent) cache to determine if we received a column or not. ==> No more consistency issue, and less stress for the filesystem. * `dataColumnSidecarByRootRPCHandler`: Improve logging. Before this commit, logged values assumed that all requested columns correspond to the same block root, which is not always the case. After this commit, we know which columns are requested for which root. * Add a log when broadcasting a data column. This is useful to debug "lost data columns" in devnet. * Address Nishant's comment
Configuration menu - View commit details
-
Copy full SHA for d9a1f96 - Browse repository at this point
Copy the full SHA d9a1f96View commit details -
PeerDAS: Re-enable full node joining the main fork (#14475)
* `columnErrBuilder`: Uses `Wrap` instead of `Join`. Reason: `Join` makes a carriage return. The log is quite unreadable. * `validateDataColumn`: Improve log. * `areDataColumnsAvailable`: Improve log. * `SendDataColumnSidecarByRoot` ==> `SendDataColumnSidecarsByRootRequest`. * `handleDA`: Refactor error message. * `sendRecentBeaconBlocksRequest` ==> `sendBeaconBlocksRequest`. Reason: There is no notion at all of "recent" in the function. If the caller decides to call this function only with "recent" blocks, that's fine. However, the function itself will know nothing about the "recentness" of these blocks. * `sendBatchRootRequest`: Improve comments. * `sendBeaconBlocksRequest`: Avoid `else` usage and use map of bool instead of `struct{}`. * `wrapAndReportValidation`: Remove `agent` from log. Reason: This prevent the log to hold on one line, and it is not really useful to debug. * `validateAggregateAndProof`: Add comments. * `GetValidCustodyPeers`: Fix typo. * `GetValidCustodyPeers` ==> `DataColumnsAdmissibleCustodyPeers`. * `CustodyHandler` ==> `DataColumnsHandler`. * `CustodyCountFromRemotePeer` ==> `DataColumnsCustodyCountFromRemotePeer`. * Implement `DataColumnsAdmissibleSubnetSamplingPeers`. * Use `SubnetSamplingSize` instead of `CustodySubnetCount` where needed. * Revert "`wrapAndReportValidation`: Remove `agent` from log." This reverts commit 55db351.
Configuration menu - View commit details
-
Copy full SHA for 4d3e389 - Browse repository at this point
Copy the full SHA 4d3e389View commit details -
PeerDAS: Misc improvements (#14482)
* `retrieveMissingDataColumnsFromPeers`: Improve logging. * `dataColumnSidecarByRootRPCHandler`: Stop decreasing peer's score if asking for a column we do not custody. * `dataColumnSidecarByRootRPCHandler`: If a data column is unavailable, stop waiting for it. This behaviour was useful for peer sampling. Now, just return the data column if we store it. If we don't, skip. * Dirty code comment. * `retrieveMissingDataColumnsFromPeers`: Improve logs. * `SendDataColumnsByRangeRequest`: Improve logs. * `dataColumnSidecarsByRangeRPCHandler`: Improve logs.
Configuration menu - View commit details
-
Copy full SHA for a62d616 - Browse repository at this point
Copy the full SHA a62d616View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d9c41f - Browse repository at this point
Copy the full SHA 1d9c41fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 679737b - Browse repository at this point
Copy the full SHA 679737bView commit details -
PeerDAS: Fix initial sync (#14494)
* `BestFinalized`: Refactor (no functional change). * `BestNonFinalized`: Refactor (no functional change). * `beaconBlocksByRangeRPCHandler`: Remove useless log. The same is already printed at the start of the function. * `calculateHeadAndTargetEpochs`: Avoid `else`. * `ConvertPeerIDToNodeID`: Improve error. * Stop printing noisy "peer should be banned" logs. * Initial sync: Request data columns from peers which: - custody a superset of columns we need, and - have a head slot >= our target slot. * `requestDataColumnsFromPeers`: Shuffle peers before requesting. Before this commit, we always requests peers in the same order, until one responds something. Without shuffling, we always requests data columns from the same peer. * `requestDataColumnsFromPeers`: If error from a peer, just log the error and skip the peer. * Improve logging. * Fix tests.
Configuration menu - View commit details
-
Copy full SHA for 190a27a - Browse repository at this point
Copy the full SHA 190a27aView commit details -
PeerDAS: Fix initial sync with super nodes (#14495)
* Improve logging. * `retrieveMissingDataColumnsFromPeers`: Limit to `512` items per request. * `retrieveMissingDataColumnsFromPeers`: Allow `nil` peers. Before this commit: If, when this funcion is called, we are not yet connected to enough peers, then `peers` will be possibly not be satisfaying, and, if new peers are connected, we will never see them. After this commit: If `peers` is `nil`, then we regularly check for all connected peers. If `peers` is not `nil`, then we use them.
Configuration menu - View commit details
-
Copy full SHA for f61938c - Browse repository at this point
Copy the full SHA f61938cView commit details -
PeerDAS: Improve initial sync logs (#14496)
* `retrieveMissingDataColumnsFromPeers`: Search only for needed peers. * Improve logging.
Configuration menu - View commit details
-
Copy full SHA for 6535ace - Browse repository at this point
Copy the full SHA 6535aceView commit details -
Fix Commitments Check (#14493)
* Fix Commitments Check * `highestFinalizedEpoch`: Refactor (no functional change). * `retrieveMissingDataColumnsFromPeers`: Fix logs. * `VerifyDataColumnSidecarKZGProofs`: Optimise with capacity. * Save data columns when initial syncing. * `dataColumnSidecarsByRangeRPCHandler`: Add logs when a request enters. * Improve logging. * Improve logging. * `peersWithDataColumns: Do not filter any more on peer head slot. * Fix Nishant's comment. --------- Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Configuration menu - View commit details
-
Copy full SHA for 9ecf085 - Browse repository at this point
Copy the full SHA 9ecf085View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9c67a71 - Browse repository at this point
Copy the full SHA 9c67a71View commit details -
PeerDAS: Fix major bug in
dataColumnSidecarsByRangeRPCHandler
and a……llow syncing from full nodes. (#14532) * `validateDataColumnsByRange`: `current` ==> `currentSlot`. * `validateRequest`: Extract `remotePeer` variable. * `dataColumnSidecarsByRangeRPCHandler`: Small non functional refactor. * `streamDataColumnBatch`: Fix major bug. Before this commit, the node was unable to respond with a data column index higher than the count of stored data columns. For example, if there is 8 data columns stored for a given block, the node was able to respond for data columns indices 1, 3, and 5, but not for 10, 16 or 127. The issue was visible only for full nodes, since super nodes always store 128 data columns. * Initial sync: Fetch data columns from all peers. (Not only from supernodes.) * Nishant's comment: Fix `lastSlot` and `endSlot` duplication. * Address Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for dfdb265 - Browse repository at this point
Copy the full SHA dfdb265View commit details -
streamDataColumnBatch
: Sort columns by index. (#14542)https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/p2p-interface.md#datacolumnsidecarsbyrange-v1 The following data column sidecars, where they exist, MUST be sent in (slot, column_index) order.
Configuration menu - View commit details
-
Copy full SHA for 083f59b - Browse repository at this point
Copy the full SHA 083f59bView commit details -
PeerDAS: Batch columns verifications (#14559)
* `ColumnAlignsWithBlock`: Split lines. * Data columns verifications: Batch * Remove completely `DataColumnBatchVerifier`. Only `DataColumnsVerifier` (with `s`) on columns remains. It is the responsability of the function which receive the data column (either by gossip, by range request or by root request) to verify the data column wrt. corresponding checks. * Fix Nishant's comment.
Configuration menu - View commit details
-
Copy full SHA for 2ab8c3b - Browse repository at this point
Copy the full SHA 2ab8c3bView commit details