Optimise mutations in single-pass epoch processing #4573

michaelsproul · 2023-08-07T07:19:47Z

Issue Addressed

Fix a tree-states performance regression introduced in #4483.

Proposed Changes

Refactor single-pass epoch processing so that it avoids unnecessary mutations of the validators, balances and inactivity_scores. The initial version of single-pass epoch processing elected to mutate these values for every validator, even if no actual changes were required. Under the hood this caused the milhouse lists to allocate new nodes for all of the "updated" values, which wasted CPU time and memory. I think this could explain some of the memory issues observed in #4521.

The fix is to use milhouse::Cow (as we did previously), and only upgrade the cow to a mutable reference when a mutation needs to be made.

Benchmarking lcli transition-blocks on an epoch boundary block showed a stark reduction in the time required to tree-hash the post-block state, from 210ms+ on my machine, down to 70ms (a 3x reduction).

Additional Info

Waiting on sigp/milhouse#22

michaelsproul · 2023-08-07T07:22:54Z

Benchmarks

Before:

[2023-08-07T05:15:04Z INFO  lcli::transition_blocks] Using mainnet spec
[2023-08-07T05:15:04Z INFO  lcli::transition_blocks] Doing 1 runs
[2023-08-07T05:15:04Z INFO  lcli::transition_blocks] Config { no_signature_verification: false, exclude_cache_builds: false, exclude_post_block_thc: false }
[2023-08-07T05:15:04Z INFO  lcli::transition_blocks] Block path: "/home/michael/Programming/lighthouse/block_6300864.ssz"
[2023-08-07T05:15:04Z INFO  lcli::transition_blocks] Pre-state path: "/home/michael/Programming/lighthouse/state_6300863.ssz"
[2023-08-07T05:15:04Z DEBUG lcli::transition_blocks] SSZ decoding /home/michael/Programming/lighthouse/state_6300863.ssz: 165.765329ms
[2023-08-07T05:15:04Z DEBUG lcli::transition_blocks] SSZ decoding /home/michael/Programming/lighthouse/block_6300864.ssz: 2.370627ms
[2023-08-07T05:15:04Z DEBUG lcli::transition_blocks] Building pubkey cache (might take some time)
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Build caches: 781.380927ms
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Initial tree hash: 46.345649ms
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Slot processing: 291.169759ms
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Build all caches (again): 870ns
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Batch verify block signatures: 9.287444ms
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Process block: 27.094788ms
[2023-08-07T05:15:34Z DEBUG lcli::transition_blocks] Post-block tree hash: 227.625198ms
[2023-08-07T05:15:34Z INFO  lcli::transition_blocks] Run 0: 1.383012308s

After:

[2023-08-07T05:58:04Z INFO  lcli::transition_blocks] Using mainnet spec
[2023-08-07T05:58:04Z INFO  lcli::transition_blocks] Doing 1 runs
[2023-08-07T05:58:04Z INFO  lcli::transition_blocks] Config { no_signature_verification: false, exclude_cache_builds: false, exclude_post_block_thc: false }
[2023-08-07T05:58:04Z INFO  lcli::transition_blocks] Block path: "/home/michael/Programming/lighthouse/block_6300864.ssz"
[2023-08-07T05:58:04Z INFO  lcli::transition_blocks] Pre-state path: "/home/michael/Programming/lighthouse/state_6300863.ssz"
[2023-08-07T05:58:04Z DEBUG lcli::transition_blocks] SSZ decoding /home/michael/Programming/lighthouse/state_6300863.ssz: 163.370615ms
[2023-08-07T05:58:04Z DEBUG lcli::transition_blocks] SSZ decoding /home/michael/Programming/lighthouse/block_6300864.ssz: 2.309347ms
[2023-08-07T05:58:04Z DEBUG lcli::transition_blocks] Building pubkey cache (might take some time)
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Build caches: 758.590088ms
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Initial tree hash: 44.091696ms
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Slot processing: 255.939054ms
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Build all caches (again): 950ns
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Batch verify block signatures: 8.519052ms
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Process block: 27.302409ms
[2023-08-07T05:58:33Z DEBUG lcli::transition_blocks] Post-block tree hash: 70.534357ms
[2023-08-07T05:58:33Z INFO  lcli::transition_blocks] Run 0: 1.165083305s

Michael Sproul <michael@sigmaprime.io>

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

commit 8e2f7c5 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Feb 8 09:20:28 2024 +1100 Delete unused epoch processing code (sigp#5170) * Delete unused epoch processing code * Compare total deltas * Remove unnecessary apply_pending * cargo fmt * Remove newline commit ae6620e Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Feb 2 17:47:14 2024 +1100 Delete `lighthouse db diff` (sigp#5171) * Delete `lighthouse db diff` * Fix help text commit 25bcd2a Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Jan 31 10:13:18 2024 +1100 Tree states v4.6.222-exp (sigp#5147) commit 68a9a2e Merge: 8e68926 0f345c7 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jan 30 17:13:57 2024 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit 8e68926 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jan 30 17:08:43 2024 +1100 `fsync` during backfill to prevent DB corruption (sigp#5144) commit 7862c71 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jan 30 15:56:48 2024 +1100 Fix tree-states sub-epoch diffs (sigp#5097) commit 11461d8 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jan 30 09:59:25 2024 +1100 Fix new CLI tests for tree-states (sigp#5132) commit 262e5f2 Merge: 6262be7 1be5253 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Jan 25 15:10:19 2024 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit 6262be7 Author: Michael Sproul <michael@sigmaprime.io> Date: Sun Jan 14 09:41:42 2024 +1100 Don't error on inactive indices in att. rewards commit 9cd9243 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Jan 12 10:50:34 2024 +1100 Tree states release v4.6.111-exp commit 664a778 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Jan 11 17:13:43 2024 +1100 Add cache for parallel HTTP requests (sigp#4879) commit 8db17da Merge: c8dc082 2e8e160 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Jan 11 13:15:06 2024 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit c8dc082 Merge: 4741bf1 a3a3703 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Dec 14 09:59:43 2023 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit 4741bf1 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Dec 5 09:23:24 2023 +1100 Remove stray println commit cefe9fd Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Dec 4 17:15:25 2023 +1100 Restore crash safety for database pruning (sigp#4975) * Add some DB sanity checks * Restore crash safety for database pruning commit 66d30bc Merge: e880d9d 44aaf13 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Dec 1 12:02:21 2023 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit e880d9d Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Dec 1 11:06:27 2023 +1100 Fix cache initialization in block rewards API (sigp#4960) commit 9cdc4b9 Merge: d36ebba 051c3e8 Author: Michael Sproul <michael@sigmaprime.io> Date: Sun Nov 12 22:19:07 2023 +0300 Merge remote-tracking branch 'origin/unstable' into tree-states commit d36ebba Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Oct 26 18:09:28 2023 +1100 Handle out-of-order forks in epoch cache (sigp#4881) commit 1aca484 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Oct 20 12:31:41 2023 +1100 Tree states release v4.5.444-exp - Update xdelta3 to remove dodgy build steps - Fix asset paths for draft release commit 6b4f154 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Oct 19 15:57:11 2023 +1100 Tree states release v4.5.333-exp commit 0cb8fdf Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Oct 19 14:59:29 2023 +1100 Various small tree-states fixes (sigp#4861) * Fix block backfill with genesis skip slots * Fix freezer upper limit * Fix: write post state in lcli skip-slots (sigp#4843) * Added CARGO_USE_GIT_CLI to the Dockerfile (sigp#4828) * chore: replace deprecated hub with gh for releases (sigp#4839) * Put schema version back to 24 (ignore Deneb) * Minimise diff --------- Co-authored-by: Joe Clapis <jclapis@outlook.com> Co-authored-by: Dustin Brickwood <dustinbrickwood204@gmail.com> commit 72d8c38 Merge: bb6675e 98cac2b Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Oct 19 12:07:35 2023 +1100 Merge remote-tracking branch 'origin/unstable' into tree-states commit bb6675e Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Oct 13 16:45:56 2023 +1100 Clean up progressive balance slashings further (sigp#4834) * Clean up progressive balance slashings further * Fix Rayon deadlock in test utils * Fix cargo-fmt commit b121e69 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Oct 13 16:45:35 2023 +1100 Fix cache logic for epoch boundary skips (sigp#4833) commit b77de69 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 14:37:46 2023 +1100 Re-enable ARM builds commit dfa3b43 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 14:03:11 2023 +1100 Fix Clippy for 1.73 commit 6ae4c22 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 11:57:39 2023 +1100 Fix merge snafu in tests commit e63d02e Merge: d4f87ef 4ad7e15 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 11:52:39 2023 +1100 Merge remote-tracking branch 'origin/deneb-free-blobs' into tree-states-deneb commit d4f87ef Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 11:18:22 2023 +1100 Fix three consensus bugs! commit ca1abfe Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Oct 11 10:43:49 2023 +1100 Support iterables in compare_fields commit e2a60a6 Merge: 9446fc8 203ac65 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Oct 6 11:11:36 2023 +1100 Merge remote-tracking branch 'origin/deneb-free-blobs' into tree-states-deneb commit 9446fc8 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Oct 3 16:07:25 2023 +1100 Fix semantic Deneb <> tree-states conflicts commit 109c4a5 Merge: f1f76f2 57edc0f Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 29 16:34:29 2023 +1000 Merge remote-tracking branch 'origin/deneb-free-blobs' into tree-states commit f1f76f2 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Sep 26 12:23:28 2023 +1000 Tree states release v4.5.222-exp (sigp#4782) commit cae73a4 Merge: 364074d 441fc16 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Sep 26 11:21:44 2023 +1000 Merge tag 'v4.5.0' into tree-states v4.5.0 commit 364074d Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 22 15:52:23 2023 +1000 Tree states release v4.5.111-exp (sigp#4769) commit d24875f Merge: cd23c89 69c39ad Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 22 15:11:42 2023 +1000 Merge remote-tracking branch 'origin/unstable' into tree-states commit cd23c89 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 22 14:49:15 2023 +1000 Improve state cache eviction and reduce mem usage (sigp#4762) * Improve state cache eviction and reduce mem usage * Fix epochs_per_state_diff tests commit 1b4bc88 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Sep 14 10:07:26 2023 +1000 Release v4.4.111-exp (sigp#4729) commit 5cb2ed3 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 14:43:02 2023 +1000 Restore custom image for Cross commit f7c6b7d Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 14:00:28 2023 +1000 Bump schema version to v24 commit 68f80cc Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 13:56:53 2023 +1000 Change default epochs-per-state-diff to 16 This should make replaying diffs during non-finality a bit quicker. commit 838e104 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 13:54:03 2023 +1000 Attempt to fix flaky test commit d961d2c Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 12:51:20 2023 +1000 Disable ARM docker builds commit b8e04ce Merge: 1e4ee7a 35f47f4 Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Sep 13 11:25:18 2023 +1000 Merge remote-tracking branch 'origin/unstable' into tree-states commit 1e4ee7a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Sep 11 10:19:40 2023 +1000 Tree states to support per-slot state diffs (sigp#4652) * Support per slot state diffs * Store HierarchyConfig on disk. Support storing hdiffs at per slot level. * Revert HierachyConfig change for testing. * Add validity check for the hierarchy config when opening the DB. * Update HDiff tests. * Fix `get_cold_state` panic when the diff for the slot isn't stored. * Use slots instead of epochs for storing snapshots in freezer DB. * Add snapshot buffer to `diff_buffer_cache` instead of loading it from db every time. * Add `hierarchy-exponents` cli flag to beacon node. * Add test for `StorageStrategy::ReplayFrom` and ignore a flaky test. * Drop hierarchy_config in tests for more frequent snapshot and fix an issue where hdiff wasn't stored unless it's a epoch boundary slot. commit e373e9a Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Aug 9 19:42:14 2023 +1000 Fix genesis state storage for genesis sync (sigp#4589) commit bba1526 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Aug 8 13:57:05 2023 +1000 Fix deadlock in finalization migration (sigp#4576) commit 18e64e6 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Aug 8 11:25:26 2023 +1000 Optimise mutations in single-pass epoch processing (sigp#4573) * Optimise mutations in single-pass epoch processing * Use safer Cow::make_mut * Update to upstream milhouse commit 8423e9f Merge: 5d2063d fc7f1ba Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Jul 19 11:23:52 2023 +1000 Merge remote-tracking branch 'origin/unstable' into tree-states commit 5d2063d Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jul 18 16:59:55 2023 +1000 Single-pass epoch processing (sigp#4483) commit 079cd67 Merge: 0291998 835fa70 Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Jul 3 15:03:54 2023 +1000 Merge remote-tracking branch 'origin/tree-states' into tree-states commit 0291998 Merge: b414c32 46be05f Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Jul 3 15:01:21 2023 +1000 Merge remote-tracking branch 'origin/unstable' into tree-states commit b414c32 Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Jul 3 12:03:14 2023 +1000 Implement activation queue cache commit 835fa70 Author: Michael Sproul <michael@sigmaprime.io> Date: Sat Jul 1 09:53:06 2023 +1000 Fix EpochCache handling in ef-tests (sigp#4454) commit f631b51 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Jun 30 22:57:36 2023 +1000 Fix EpochCache handling in ef-tests commit 2df714e Author: Jimmy Chen <jchen.tc@gmail.com> Date: Fri Jun 30 11:25:51 2023 +1000 Tree states optimization using `EpochCache` (sigp#4429) * Relocate epoch cache to BeaconState * Optimize per block processing by pulling previous epoch & current epoch calculation up. * Revert `get_cow` change (no performance improvement) * Initialize `EpochCache` in epoch processing and load it from state when getting base rewards. * Initialize `EpochCache` at start of block processing if required. * Initialize `EpochCache` in `transition_blocks` if `exclude_cache_builds` is enabled * Fix epoch cache initialization logic * Remove FIXME comment. * Cache previous & current epochs in `consensus_context.rs`. * Move `get_base_rewards` from `ConsensusContext` to `BeaconState`. * Update Milhouse version commit 160bbde Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Jun 30 10:29:34 2023 +1000 Fix db-migration-period default (sigp#4441) * Fix db-migration-period default * Fix version regex commit 6954de6 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 17:34:41 2023 +1000 Tree states alpha release v4.2.990-exp commit 8dc374e Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 17:33:25 2023 +1000 Temporarily disable ARM builds commit af5fb20 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 15:10:52 2023 +1000 Tree states alpha release v4.2.99-exp commit 56c7a52 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 16:52:24 2023 +1000 Install Clang 5 in Cross builder image commit 7c2eb96 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 15:06:43 2023 +1000 Set epochs per migration to 1 Workaround for sigp#4236 commit 88e30b6 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 27 15:04:39 2023 +1000 Fix failing tests (sigp#4423) * Get tests passing * Get benchmarks compiling * Fix EF withdrawals test * Remove unused deps * Fix tree_hash panic in tests * Fix slasher compilation * Fix ssz_generic test * Get more tests passing * Fix EF tests for real * Fix local testnet scripts commit ca412ab Author: Michael Sproul <michael@sigmaprime.io> Date: Wed Jun 21 11:05:09 2023 +1000 Use rebasing to minimise BeaconState mem usage (sigp#4416) * Use "rebasing" to minimise BeaconState mem usage * Update metastruct * Use upstream milhouse, update cargo lock * Rebase caches for extra memory savings commit 6eb1513 Author: Michael Sproul <michael@sigmaprime.io> Date: Tue Jun 20 19:10:05 2023 +1000 Configurable diff buffer cache size (sigp#4420) commit d56cec8 Author: Paul Hauner <paul@paulhauner.com> Date: Tue Jun 20 11:47:52 2023 +1000 Address clippy lints in `tree-states` (sigp#4414) * Address some clippy lints * Box errors to fix error size lint * Add Default impl for Validator * Address more clippy lints * Re-implement `check_state_diff` * Fix misc test compile errors commit 23db089 Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Jun 19 10:14:47 2023 +1000 Implement tree states & hierarchical state DB

* Single-pass epoch processing (#4483, #4573) Co-authored-by: Michael Sproul <michael@sigmaprime.io> * Delete unused epoch processing code (#5170) * Delete unused epoch processing code * Compare total deltas * Remove unnecessary apply_pending * cargo fmt * Remove newline * Use epoch cache in block packing (#5223) * Remove progressive balances mode (#5224) * inline inactivity_penalty_quotient_for_state * drop previous_epoch_total_active_balance * fc lint * spec compliant process_sync_aggregate (#15) * spec compliant process_sync_aggregate * Update consensus/state_processing/src/per_block_processing/altair/sync_committee.rs Co-authored-by: Michael Sproul <micsproul@gmail.com> --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Delete the participation cache (#16) * update help * Fix op_pool tests * Fix fork choice tests * Merge remote-tracking branch 'sigp/unstable' into epoch-single-pass * Simplify exit cache (#5280) * Fix clippy on exit cache * Clean up single-pass a bit (#5282) * Address Mark's review of single-pass (#5386) * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * Address Sean's review comments (#5414) * Address most of Sean's review comments * Simplify total balance cache building * Clean up unused junk * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * More self-review * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * Merge branch 'unstable' into epoch-single-pass * Fix imports for beta compiler * Fix tests, probably

* Add attestation simulator, blobs info and some updates to Lighthouse Book (#5364) * Apply suggestions from code review * Revise attestation simulator doc * Revise blobs.md * Summary * Add blobs * Simulator docs * Revise attestation simulator * minor formatting * Revise vm node * Update faq * Update faq * Add link to v4.6.0 * Remove minification in the docs * Update Goerli to Holesky * Add a note on moved vm validator monitor * Update Rpi 4 note * Revise attestation simulator doc * Add docs for attestation simulator * update database table * Update faq on resources used * Fix and update table * Dedup parent blob requests (#5432) * de dup parent blob requests * add new line * Fix Rust beta compiler errors 1.78.0-beta.1 (#5439) * remove redundant imports * fix test * contains key * fmt * Merge branch 'unstable' into fix-beta-compiler * Improvements and Fixes in Documentation, Including Corrected Command Usage (#4998) * Fix typo: change 'periodical' to 'periodic' in progress updates description * Fix wrong command in Usage section * fix typo in Development Environment section * Fix typo: change 'Explictly' to 'Explicitly' * Fix typos in Lighthouse UI and Contributing sections * Fix typo: replace 'confirms' with 'conforms' in Beacon Node API description * fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message * Fix spelling error in Detailed Guide section * Revert "Fix typo: change 'Explictly' to 'Explicitly'" This reverts commit 6b07816. * Revert "fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message" This reverts commit a4904a0. * compiled * Revert "compiled" This reverts commit 425a553. * Revert "Revert "compiled"" This reverts commit b1f871c. * Empty commit to trigger CI. * Fix peer count metrics (#5404) * Set the peers_per_client metrics directly, rather than using increment/decrement * Move PEERS_CONNECTED related update to the same place * Move PEERS_CONNECTED_MULTI related update to the same place * Rename * Remove unused variables * fix NAT `nat_open` metrics report (#5427) * fix nat reporting * Fix gossip verification of duplicate attester slashings (#5385) * Fix gossip verification of duplicate attester slashings * disable libp2p upnp (#5449) * disable libp2p upnp when passing --disable-upnp cli flag * Release v5.1.2 (#5453) * Release v5.1.2 * Improve parent lookup logging (#5451) * upgrade parent lookup result processing logs to debug, use display instead of debug for BlockError in case a blob parent unknown error is hit, add block root to BlockIsAlreadyKnown * fix compile * fix compile * fix compile * fix: tail command typo (#5456) * fix: tail command typo * Merge branch 'unstable' of https://github.com/sigp/lighthouse into patch-1 * chore: remove stale comment (#5440) * rm stale comment * Merge branch 'unstable' of https://github.com/sigp/lighthouse into rm_irelevant_comment * chore: reduce scope of commitment (#5426) * reduce scope of commitment * avoid clone for last reference * Merge branch 'unstable' of https://github.com/sigp/lighthouse into mod_merge_single_blob * Fix one and hide all beacon-processor flags (#5397) * Fix `beacon-processor-work-queue-len` * Hide beacon-processor flags * Delete unused incomplete_processing_components (#5418) * Delete unused incomplete_processing_components * lint * Make sure all geth processes are killed when stopping a local testnet (#5383) * Fix geth processes not being killed when stopping a local testnet * Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix_stop_testnet * Verify whether validators really are unknown during sync committee duty API request (#5174) * Verify whether validators really are unknown during sync committee duty API request * Merge branch 'unstable' into fix-4717 * Merge branch 'unstable' into fix-4717 * Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix-4717 * Built-in documentation text width in Lighthouse book (#5394) * Adjust width * Commit changes * Trigger Build * Delete PRE_CAPELLA_ENGINE_CAPABILITIES (#5406) * Adjust width * Commit changes * Delete PRE_CAPELLA_ENGINE_CAPABILITIES * Revert "Adjust width" This reverts commit 6fea81b. * Revert "Commit changes" This reverts commit d00859a. * Simplify * Merge branch 'delete-pre-capella' of https://github.com/chong-he/lighthouse into delete-pre-capella * Add Capella & Deneb light client support (#4946) * rebase and add comment * conditional test * test * optimistic chould be working now * finality should be working now * try again * try again * clippy fix * add lc bootstrap beacon api * add lc optimistic/finality update to events * fmt * That error isn't occuring on my computer but I think this should fix it * Merge branch 'unstable' into light_client_beacon_api_1 # Conflicts: # beacon_node/beacon_chain/src/events.rs # beacon_node/http_api/src/lib.rs # beacon_node/http_api/src/test_utils.rs # beacon_node/http_api/tests/main.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/network/src/beacon_processor/worker/rpc_methods.rs # beacon_node/tests/test.rs # common/eth2/src/types.rs # lighthouse/src/main.rs * Add missing test file * Update light client types to comply with Altair light client spec. * Fix test compilation * Merge branch 'unstable' into light_client_beacon_api_1 * Support deserializing light client structures for the Bellatrix fork * Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`. * Misc fixes. - log cleanup - move http_api config mutation to `config::get_config` for consistency - fix light client API responses * Add light client bootstrap API test and fix existing ones. * Merge branch 'unstable' into light_client_beacon_api_1 * Fix test for `light-client-server` http api config. * Appease clippy * Add Altair light client SSZ tests * Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1 * updates to light client header * light client header from signed beacon block * using options * implement helper functions * placeholder conversion from vec hash256 to exec branch * add deneb * using fixed vector * remove unwraps * by epoch * compute merkle proof * merkle proof * update comments * resolve merge conflicts * linting * Merge branch 'unstable' into light-client-ssz-tests # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # consensus/types/src/light_client_bootstrap.rs # consensus/types/src/light_client_header.rs * superstruct attempt * superstruct changes * lint * altair * update * update * changes to light_client_optimistic_ and finality * merge unstable * refactor * resolved merge conflicts * Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types * block_to_light_client_header fork aware * fmt * comment fix * comment fix * include merge fork, update deserialize_by_fork, refactor * fmt * pass by ref to prevent clone * rename merkle proof fn * add FIXME * LightClientHeader TestRandom * fix comments * fork version deserialize * merge unstable * move fn arguments, fork name calc * use task executor * remove unneeded fns * remove dead code * add manual ssz decoding/encoding and add ssz_tests_by_fork macro * merge deneb types with tests * merge ssz tests, revert code deletion, cleanup * move chainspec * update ssz tests * fmt * light client ssz tests * change to superstruct * changes from feedback * linting * Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types * test fix * cleanup * Remove unused `derive`. * Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types * beta compiler fix * merge * Fix double counted metrics (#5476) * Fix double counted metrics * Move gossipsub into a separate crate (#5401) * move gossipsub into a separate crate * Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub * address review 2 * clippy beta * update logging to log gossipsub logs * Run fork choice after RPC blob import (#5475) * Run fork choice after RPC blob import * Single lookup improvements (#5488) * Fix unexpected `UnrequestedBlobId` and `ExtraBlocksReturned` errors due to race conditions. * Continue chain segment processing and skip any blocks that are already known, rather than returning an error. * more de-dup checking * ensure we don't reset `requested_ids` during rpc download * better fix * Merge branch 'unstable' of https://github.com/sigp/lighthouse into more-dup-lookup-fixes * remove chain hash check * Merge branch 'fix-block-lookup-race' of https://github.com/jimmygchen/lighthouse into sean-test-lookups * remove block check * add back tests * Log and CI fixes * undue extra check * Merge branch 'sean-test-lookups' of https://github.com/realbigsean/lighthouse into sean-test-lookups * log improvements * Improve logging * Lookup log improvements (#5491) * log improvements * Bump duplicate cache time (#5493) * Bump seen_ttl for gossipsub duplicate cache * Release v5.1.3 (#5497) * Release v5.1.3 * Add Electra fork boilerplate (#5122) * Add Electra fork boilerplate * Remove electra from spec tests * Fix tests * Remove sneaky log file * Fix more tests * Fix even more tests and add suggestions * Remove unrelated lcli addition * Update more tests * Merge branch 'unstable' into electra * Add comment for test-suite lcli override * Merge branch 'unstable' into electra * Cleanup * Merge branch 'unstable' into electra * Apply suggestions * Merge branch 'unstable' into electra * Merge sigp/unstable into electra * Merge branch 'unstable' into electra * Use `E` for `EthSpec` globally (#5264) * Use `E` for `EthSpec` globally * Fix tests * Merge branch 'unstable' into e-ethspec * Merge branch 'unstable' into e-ethspec # Conflicts: # beacon_node/execution_layer/src/engine_api.rs # beacon_node/execution_layer/src/engine_api/http.rs # beacon_node/execution_layer/src/engine_api/json_structures.rs # beacon_node/execution_layer/src/test_utils/handle_rpc.rs # beacon_node/store/src/partial_beacon_state.rs # consensus/types/src/beacon_block.rs # consensus/types/src/beacon_block_body.rs # consensus/types/src/beacon_state.rs # consensus/types/src/config_and_preset.rs # consensus/types/src/execution_payload.rs # consensus/types/src/execution_payload_header.rs # consensus/types/src/light_client_optimistic_update.rs # consensus/types/src/payload.rs # lcli/src/parse_ssz.rs * Bump `h2` for RUSTSEC-2024-0332 (#5514) * Bump `h2` for RUSTSEC-2024-0332 * Return `not synced` errors for endpoints that require syncing (#5136) * add not synced filter into then blocks * refactor * Single-pass epoch processing and optimised block processing (#5279) * Single-pass epoch processing (#4483, #4573) Co-authored-by: Michael Sproul <michael@sigmaprime.io> * Delete unused epoch processing code (#5170) * Delete unused epoch processing code * Compare total deltas * Remove unnecessary apply_pending * cargo fmt * Remove newline * Use epoch cache in block packing (#5223) * Remove progressive balances mode (#5224) * inline inactivity_penalty_quotient_for_state * drop previous_epoch_total_active_balance * fc lint * spec compliant process_sync_aggregate (#15) * spec compliant process_sync_aggregate * Update consensus/state_processing/src/per_block_processing/altair/sync_committee.rs Co-authored-by: Michael Sproul <micsproul@gmail.com> --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Delete the participation cache (#16) * update help * Fix op_pool tests * Fix fork choice tests * Merge remote-tracking branch 'sigp/unstable' into epoch-single-pass * Simplify exit cache (#5280) * Fix clippy on exit cache * Clean up single-pass a bit (#5282) * Address Mark's review of single-pass (#5386) * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * Address Sean's review comments (#5414) * Address most of Sean's review comments * Simplify total balance cache building * Clean up unused junk * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * More self-review * Merge remote-tracking branch 'origin/unstable' into epoch-single-pass * Merge branch 'unstable' into epoch-single-pass * Fix imports for beta compiler * Fix tests, probably * Remove DataAvailabilityView trait from ChildComponents (#5421) * Remove DataAvailabilityView trait from ChildComponents * PR reviews * Update beacon_node/network/src/sync/block_lookups/common.rs Co-authored-by: realbigsean <seananderson33@GMAIL.com> * Merge branch 'unstable' of https://github.com/sigp/lighthouse into child_components_independent * Add `is_parent_strong` proposer re-org check (#5417) * initial fork choice additions * add helper fns * add is_parent_strong * Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check * disabling proposer reorg should set parent_threshold to u64 max * add new flag, is_parent_strong check in override fcu params * cherry-pick changes * Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check * cleanup * fmt * Minor review tweaks * Add missing header to `eth/v1/builder/blinded_blocks` (#5407) * add missing header * read header in mock builder * Merge branch 'unstable' into builder-blinded-blocks-missing-header * Delete `ParticipationCache` (#5525) * Delete `ParticipationCache` * Use hashset to filter validators ids in http_api (#5468) * Use hashset to filter validators ids in http_api * Update beacon_node/http_api/src/validators.rs * Support `LightClientFinalityUpdate` and `LightClientOptimisticUpdate` rpcs (#3849) * add light client optimistic and finality update rpc * Arc the updates in the response * add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate * alter display for inboundrequest light client optimistic and finality updates * remove LightClientOptimistic/FinalityReuest struct and some minor fixes * rebase * failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient * minor change * added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable * almost there but ran into issue where needed to implement BaseOutboundRequest. * failing but running with a light client service of sorts * small test change * changed Protocol::LightClientBootstrap response limit * deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here * deleted purging expired inbound/outbound streams code * deleted drive inbound streams that need to be processed * removed unused imports * made things private again * deleted inject_fully_negotiated_inbound * made more things private again * more * turned the logger off in the test * added failing test for new rpc * add rate limit for new rpcs * change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate * rebasing fix * add LightClientUpdate to handle_rpc functions * added context bytes * fmt * use correct unsed_tcp4_port function * fix for recent config changes and adding context_bytes for the light client protocols * fix clippy complaint * Merge branch 'unstable' into lc-reqresp # Conflicts: # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/peer_manager/mod.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/mod.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/rpc/self_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/tests/common/mod.rs # beacon_node/lighthouse_network/tests/rpc_tests.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/router.rs * Error handling updates and various cleanups. * Moar minor clean ups. * Do not ban peer for rate limiting light client requests * Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940). # Conflicts: # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/mod.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/tests/common/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/router.rs # consensus/types/src/light_client_bootstrap.rs # consensus/types/src/light_client_finality_update.rs # consensus/types/src/light_client_optimistic_update.rs * Remove unnecessary changes * Add missing light client queue handling. * Merge branch 'unstable' into lc-reqresp * Merge branch 'unstable' into lc-reqresp # Conflicts: # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/service/api_types.rs # consensus/types/src/light_client_finality_update.rs # consensus/types/src/light_client_optimistic_update.rs * Add context bytes for light client RPC responses. * Add RPC limits for light client object. * Fix lint * Fix incorrect light client max size computation. * Merge branch 'unstable' into lc-reqresp # Conflicts: # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/service/api_types.rs * Remove unwanted local changes. * Merge branch 'unstable' into lc-reqresp * Replace `unimplemented` electra code path with deneb values. --------- Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com> Co-authored-by: realbigsean <sean@sigmaprime.io> Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu> Co-authored-by: joao <22820692+joaolago1113@users.noreply.github.com> Co-authored-by: Akihito Nakano <sora.akatsuki@gmail.com> Co-authored-by: João Oliveira <hello@jxs.pt> Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: Afanti <127061691+threewebcode@users.noreply.github.com> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Co-authored-by: Daniel Ramírez-Chiquillo <hi@danielrachi.com> Co-authored-by: dknopik <107140945+dknopik@users.noreply.github.com> Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> Co-authored-by: Mac L <mjladson@pm.me> Co-authored-by: 0xalex88 <113263502+0xalex88@users.noreply.github.com> Co-authored-by: GeemoCandama <104614073+GeemoCandama@users.noreply.github.com>

Optimise mutations in single-pass epoch processing

62a64b0

michaelsproul added work-in-progress PR is a work-in-progress tree-states Upcoming state and database overhaul labels Aug 7, 2023

michaelsproul mentioned this pull request Aug 7, 2023

Beacon chain lags behind frequently when running the latest tree-states branch #4521

Closed

michaelsproul added 2 commits August 8, 2023 09:27

Use safer Cow::make_mut

690c366

Update to upstream milhouse

66f0160

michaelsproul merged commit 18e64e6 into sigp:tree-states Aug 8, 2023
6 checks passed

michaelsproul deleted the tree-states-opt-sp branch August 8, 2023 01:25

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 7, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

4019244

Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 7, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

128bc22

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 7, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

51fcc68

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 7, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

23e10c6

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 7, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

3b615e4

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 8, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

3682028

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

dapplion added a commit to dapplion/lighthouse that referenced this pull request Feb 8, 2024

Single-pass epoch processing (sigp#4483, sigp#4573)

8cb05de

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimise mutations in single-pass epoch processing #4573

Optimise mutations in single-pass epoch processing #4573

michaelsproul commented Aug 7, 2023

michaelsproul commented Aug 7, 2023

Optimise mutations in single-pass epoch processing #4573

Optimise mutations in single-pass epoch processing #4573

Conversation

michaelsproul commented Aug 7, 2023

Issue Addressed

Proposed Changes

Additional Info

michaelsproul commented Aug 7, 2023