Merge unstable into deneb-free-blobs
#4781
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Issue Addressed sigp#4402 ## Proposed Changes This PR adds QUIC support to Lighthouse. As this is not officially spec'd this will only work between lighthouse <-> lighthouse connections. We attempt a QUIC connection (if the node advertises it) and if it fails we fallback to TCP. This should be a backwards compatible modification. We want to test this functionality on live networks to observe any improvements in bandwidth/latency. NOTE: This also removes the websockets transport as I believe no one is really using it. It should be mentioned in our release however. Co-authored-by: João Oliveira <hello@jxs.pt>
Web3Signer now requires Java runtime v17, see [v23.8.0 release](https://github.com/Consensys/web3signer/releases/tag/23.8.0). We have some Web3Signer tests that requires a compatible Java runtime to be installed on dev machines. This PR updates `setup` documentation in Lighthouse book, and also fixes a small typo.
## Issue Addressed On a new network a user might require importing validators before waiting until genesis has occurred. ## Proposed Changes Starts the validator client http api before waiting for genesis ## Additional Info cc @antondlr
## Issue Addressed CI is plagued by `AddrAlreadyInUse` failures, which are caused by race conditions in allocating free ports. This PR removes all usages of the `unused_port` crate for Lighthouse's HTTP API, in favour of passing `:0` as the listen address. As a result, the listen address isn't known ahead of time and must be read from the listening socket after it binds. This requires tying some self-referential knots, which is a little disruptive, but hopefully doesn't clash too much with Deneb 🤞 There are still a few usages of `unused_tcp4_port` left in cases where we start external processes, like the `watch` Postgres DB, Anvil, Geth, Nethermind, etc. Removing these usages is non-trivial because it's hard to read the port back from an external process after starting it with `--port 0`. We might be able to do something on Linux where we read from `/proc/`, but I'll leave that for future work.
…igp#4688) ## Issue Addressed I went through the code base and look for places where we acquire fork choice locks (after the deadlock bug was found and fixed in sigp#4687), and discovered an instance where we re-acquire a lock immediately after dropping it. This shouldn't cause deadlock like the other issue, but is slightly less efficient.
## Proposed Changes This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. ## Additional Info - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).
There is an issue with the file `scripts/local_testnet/start_local_testnet.sh` - when we use a non-default `$SPEC-PRESET` in `vars.env` it runs into an error: ``` executing: ./setup.sh >> /home/ck/.lighthouse/local-testnet/testnet/setup.log parse error: Invalid numeric literal at line 1, column 7 ``` @jimmygchen found the issue and the updated script includes the flag `--spec $SPEC-PRESET`
Fix local testnet to generate keys in the correct folders when `BN_COUNT` and `VC_COUNT` don't match. The current script place the generated validator keys in validator folders based on the `BN_COUNT` config, e.g. `node_1/validators`, `node_2/validators`..etc. We should be using `VC_COUNT` here instead, otherwise the number of validator clients may not match the number of directories generated, and would result in either: 1. a VC not having any keys (when `BN_COUNT` < `VC_COUNT`) 2. a validator key directory not being used (when `BN_COUNT` > `VC_COUNT`).
## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch sigp#3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <seananderson33@gmail.com> Co-authored-by: realbigsean <sean@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net>
## Issue Addressed sigp#4531 ## Proposed Changes add SSZ support to the following block production endpoints: GET /eth/v2/validator/blocks/{slot} GET /eth/v1/validator/blinded_blocks/{slot} ## Additional Info i updated a few existing tests to use ssz instead of writing completely new tests
## Issue Addressed Closes sigp#4751 ## Proposed Changes Prevent `state_root_at_slot` and `block_root_at_slot` from erroring out due to a call to `self.slot()?` that fails before genesis. This fixes pre-genesis queries for: - block at slot 0 - block by genesis block root - state at slot 0 - state by genesis state root - state at `finalized` tag - state at `justified` tag
## Issue Addressed update boot ENR for Holesky relaunch
## Issue Addressed NA ## Proposed Changes Bumps `quinn-proto` to address a QUIC-related vulnerability: https://rustsec.org/advisories/RUSTSEC-2023-0063 Fixes a `cargo audit` failure. ## Additional Info NA
## Issue Addressed following discussion on sigp#4639 (comment) this PR makes the `http` and `metrics` sub-flags to require those main flags enabled
…cification (sigp#4679) ## Issue Addressed sigp#4635 ## Proposed Changes Wrap the `SignedVoluntaryExit` object in a `GenericResponse` container, adding an additional `data` layer, to ensure compliance with the key manager API specification. The new response would look like this: ```json {"data":{"message":{"epoch":"196868","validator_index":"505597"},"signature":"0xhexsig"}} ``` This is a backward incompatible change and will affect Siren as well.
Adds new Teku bootnodes Co-authored-by: Paul Hauner <paul@paulhauner.com>
## Issue Addressed Build releases on self-hosted hardware to speed the process up
## Issue Addressed Synchronize dependencies and edition on the workspace `Cargo.toml` ## Proposed Changes with rust-lang/cargo#8415 merged it's now possible to synchronize details on the workspace `Cargo.toml` like the metadata and dependencies. By only having dependencies that are shared between multiple crates aligned on the workspace `Cargo.toml` it's easier to not miss duplicate versions of the same dependency and therefore ease on the compile times. ## Additional Info this PR also removes the no longer required direct dependency of the `serde_derive` crate. should be reviewed after sigp#4639 get's merged. closes sigp#4651 Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: Michael Sproul <micsproul@gmail.com>
## Issue Addressed NA ## Proposed Changes Disables some commands for self-hosted runners to prevent failures. ## Additional Info NA
## Issue Addressed Fixes breaking change introduced on sigp#4674 that doesn't allow multiple `http_enabled` `ArgGroup` flags
## Issue Addressed NA ## Proposed Changes Bump versions from v4.4.1 to v4.5.0. ## Additional Info NA
…reshold` field in `ApiTesterConfig`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Merge latest
unstablebranch intodeneb-free-blobsas of 2023.09.26.Changes
BlockContentsSSZ decoding, fixed in 7a3cb13