Skip to content

Commit

Permalink
feat: PeerDAS
Browse files Browse the repository at this point in the history
  • Loading branch information
samcm committed Jul 25, 2024
1 parent 3f79c2d commit 094e08a
Show file tree
Hide file tree
Showing 19 changed files with 737 additions and 98 deletions.
17 changes: 16 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![GoDoc](https://pkg.go.dev/badge/github.com/probe-lab/hermes)](https://pkg.go.dev/github.com/probe-lab/hermes)

Hermes is a GossipSub listener and tracer. It subscribes to all relevant pubsub topics
and traces all protocol interactions. As of `2024-03-27`, Hermes supports the Ethereum
and traces all protocol interactions. As of `2024-05-21`, Hermes supports the Ethereum
network.

## Table of Contents
Expand All @@ -21,6 +21,7 @@ network.
- [Telemetry](#telemetry)
- [Metrics](#metrics)
- [Tracing](#tracing)
- [Differences with other tools](#differences-with-other-tools)
- [Maintainers](#maintainers)
- [Contributing](#contributing)
- [License](#license)
Expand Down Expand Up @@ -199,6 +200,20 @@ You can find the UI at [`http://localhost:16686`](http://localhost:16686). Port

Run Hermes with the `--tracing` flag. To change the address of the trace collector, you can also specify `--tracing.addr` and `--tracing.port`.

## Differences with other tools
Hermes jumps to the web3/blockchain/libp2p ecosystem despite a large variety of tools around it, such as the many existing network crawlers or light clients for most mature networks. Although at first sight it might look like a competitor to those, there was still a large incentive to develop it. Here, we describe the gap that Hermes fills as well as the use-cases that it is suitable for.
Hermes was designed to behave as a light node in each supported network, where, in addition to being an honest participant in the network and supporting all the protocols and RPC endpoints, it also allows streaming of custom internal events (mostly libp2p-related).

Hermes avoids being based on a custom fork of existing full/light clients, which would come with non-negligible maintenance baggage and would complicate having control of events.

Currently available similar tools:

### Armiarma Crawler from MigaLabs vs Hermes from ProbeLab
Although both Hermes and Armiarma seem to be focusing on the same goals at first sight, they have significant differences in their use cases and their targets in data collection and metrics.

[Armiarma](https://github.com/migalabs/armiarma) is a network crawler that relies on running discv5 peer discovery service and a libp2p host 24/7 to establish connections. However, significant modifications have been made to connect to as many peers as possible (custom peering module). This way, it tries to identify as many peers as possible in the network periodically. Thus, its focus is mainly on opening and identifying as many peers as possible, rather than maintaining stable connections to other peers in the network.

On the other hand, although Hermes also relies on a continuously running discv5 and libp2p host, it uses the libp2p connection manager for a different purpose to Armiarma (which is to connect to as many peers as possible). In the case of Hermes, the connection manager is used to decide who it connects to (i.e., simulating the behaviour of a standard node). Furthermore, it backs up some of the RPC, which requires keeping the chain db calls on a trusted node. This way, it behaves like a light node to the network, which is honest and beneficial for the rest of the network, allowing us to track all desired networking events from stable connections, while at the same time having a highly customizable tracing system.

## Maintainers

Expand Down
85 changes: 73 additions & 12 deletions cmd/hermes/cmd_eth.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ var ethConfig = &struct {
DialConcurrency int
DialTimeout time.Duration
MaxPeers int
GenesisSSZURL string
ConfigURL string
BootnodesURL string
DepositContractBlockURL string
}{
PrivateKeyStr: "", // unset means it'll be generated
Chain: params.MainnetName,
Expand All @@ -48,6 +52,10 @@ var ethConfig = &struct {
DialConcurrency: 16,
DialTimeout: 5 * time.Second,
MaxPeers: 30, // arbitrary
GenesisSSZURL: "",
ConfigURL: "",
BootnodesURL: "",
DepositContractBlockURL: "",
}

var cmdEth = &cli.Command{
Expand Down Expand Up @@ -167,6 +175,34 @@ var cmdEthFlags = []cli.Flag{
Value: ethConfig.MaxPeers,
Destination: &ethConfig.MaxPeers,
},
&cli.StringFlag{
Name: "genesis.ssz.url",
EnvVars: []string{"HERMES_ETH_GENESIS_SSZ_URL"},
Usage: "The .ssz URL from which to fetch the genesis data, requires 'chain=devnet'",
Value: ethConfig.GenesisSSZURL,
Destination: &ethConfig.GenesisSSZURL,
},
&cli.StringFlag{
Name: "config.yaml.url",
EnvVars: []string{"HERMES_ETH_CONFIG_URL"},
Usage: "The .yaml URL from which to fetch the beacon chain config, requires 'chain=devnet'",
Value: ethConfig.ConfigURL,
Destination: &ethConfig.ConfigURL,
},
&cli.StringFlag{
Name: "bootnodes.yaml.url",
EnvVars: []string{"HERMES_ETH_BOOTNODES_URL"},
Usage: "The .yaml URL from which to fetch the bootnode ENRs, requires 'chain=devnet'",
Value: ethConfig.BootnodesURL,
Destination: &ethConfig.BootnodesURL,
},
&cli.StringFlag{
Name: "deposit-contract-block.txt.url",
EnvVars: []string{"HERMES_ETH_DEPOSIT_CONTRACT_BLOCK_URL"},
Usage: "The .txt URL from which to fetch the deposit contract block. Requires 'chain=devnet'",
Value: ethConfig.DepositContractBlockURL,
Destination: &ethConfig.DepositContractBlockURL,
},
}

func cmdEthAction(c *cli.Context) error {
Expand All @@ -176,20 +212,45 @@ func cmdEthAction(c *cli.Context) error {
// Print hermes configuration for debugging purposes
printEthConfig()

// Extract chain configuration parameters based on the given chain name
genConfig, netConfig, beaConfig, err := eth.GetConfigsByNetworkName(ethConfig.Chain)
if err != nil {
return fmt.Errorf("get config for %s: %w", ethConfig.Chain, err)
var config *eth.NetworkConfig
// Derive network configuration
if ethConfig.Chain != params.DevnetName {
slog.Info("Deriving known network config:", "chain", ethConfig.Chain)

c, err := eth.DeriveKnownNetworkConfig(c.Context, ethConfig.Chain)
if err != nil {
return fmt.Errorf("derive network config: %w", err)
}

config = c
} else {
slog.Info("Deriving devnet network config")

c, err := eth.DeriveDevnetConfig(c.Context, eth.DevnetOptions{
ConfigURL: ethConfig.ConfigURL,
BootnodesURL: ethConfig.BootnodesURL,
DepositContractBlockURL: ethConfig.DepositContractBlockURL,
GenesisSSZURL: ethConfig.GenesisSSZURL,
})
if err != nil {
return fmt.Errorf("failed to derive devnet network config: %w", err)
}
config = c
}

genesisRoot := genConfig.GenesisValidatorRoot
genesisTime := genConfig.GenesisTime
// Overriding configuration so that functions like ComputForkDigest take the
// correct input data from the global configuration.
params.OverrideBeaconConfig(config.Beacon)
params.OverrideBeaconNetworkConfig(config.Network)

genesisRoot := config.Genesis.GenesisValidatorRoot
genesisTime := config.Genesis.GenesisTime

// compute fork version and fork digest
currentSlot := slots.Since(genesisTime)
currentEpoch := slots.ToEpoch(currentSlot)

currentForkVersion, err := eth.GetCurrentForkVersion(currentEpoch, beaConfig)
currentForkVersion, err := eth.GetCurrentForkVersion(currentEpoch, config.Beacon)
if err != nil {
return fmt.Errorf("compute fork version for epoch %d: %w", currentEpoch, err)
}
Expand All @@ -201,13 +262,13 @@ func cmdEthAction(c *cli.Context) error {

// Overriding configuration so that functions like ComputForkDigest take the
// correct input data from the global configuration.
params.OverrideBeaconConfig(beaConfig)
params.OverrideBeaconNetworkConfig(netConfig)
params.OverrideBeaconConfig(config.Beacon)
params.OverrideBeaconNetworkConfig(config.Network)

cfg := &eth.NodeConfig{
GenesisConfig: genConfig,
NetworkConfig: netConfig,
BeaconConfig: beaConfig,
GenesisConfig: config.Genesis,
NetworkConfig: config.Network,
BeaconConfig: config.Beacon,
ForkDigest: forkDigest,
ForkVersion: currentForkVersion,
PrivateKeyStr: ethConfig.PrivateKeyStr,
Expand Down
19 changes: 9 additions & 10 deletions cmd/hermes/cmd_eth_chains.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,28 +27,27 @@ func cmdEthChainsAction(c *cli.Context) error {

slog.Info("Supported chains:")
for _, chain := range chains {

genConfig, _, beaConfig, err := eth.GetConfigsByNetworkName(chain)
config, err := eth.DeriveKnownNetworkConfig(c.Context, chain)
if err != nil {
return fmt.Errorf("get config for %s: %w", chain, err)
}
slog.Info(chain)

forkVersions := [][]byte{
beaConfig.GenesisForkVersion,
beaConfig.AltairForkVersion,
beaConfig.BellatrixForkVersion,
beaConfig.CapellaForkVersion,
beaConfig.DenebForkVersion,
config.Beacon.GenesisForkVersion,
config.Beacon.AltairForkVersion,
config.Beacon.BellatrixForkVersion,
config.Beacon.CapellaForkVersion,
config.Beacon.DenebForkVersion,
}

for _, forkVersion := range forkVersions {
epoch, found := beaConfig.ForkVersionSchedule[[4]byte(forkVersion)]
epoch, found := config.Beacon.ForkVersionSchedule[[4]byte(forkVersion)]
if !found {
return fmt.Errorf("fork version schedule not found for %x", forkVersion)
}

forkName, found := beaConfig.ForkVersionNames[[4]byte(forkVersion)]
forkName, found := config.Beacon.ForkVersionNames[[4]byte(forkVersion)]
if !found {
return fmt.Errorf("fork version name not found for %x", forkVersion)
}
Expand All @@ -57,7 +56,7 @@ func cmdEthChainsAction(c *cli.Context) error {
continue
}

digest, err := signing.ComputeForkDigest(forkVersion, genConfig.GenesisValidatorRoot)
digest, err := signing.ComputeForkDigest(forkVersion, config.Genesis.GenesisValidatorRoot)
if err != nil {
return err
}
Expand Down
121 changes: 121 additions & 0 deletions eth/fetch.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
package eth

import (
"context"
"encoding/binary"
"io"
"net/http"

"github.com/prysmaticlabs/prysm/v5/config/params"
"gopkg.in/yaml.v3"
)

// FetchConfigFromURL fetches the beacon chain config from a given URL.
func FetchConfigFromURL(ctx context.Context, url string) (*params.BeaconChainConfig, error) {
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, err
}

response, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer response.Body.Close()

data, err := io.ReadAll(response.Body)
if err != nil {
return nil, err
}

config := params.MainnetConfig().Copy()

out, err := params.UnmarshalConfig(data, config)
if err != nil {
return nil, err
}

return out, nil
}

// FetchBootnodeENRsFromURL fetches the bootnode ENRs from a given URL.
func FetchBootnodeENRsFromURL(ctx context.Context, url string) ([]string, error) {
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, err
}

response, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer response.Body.Close()

data, err := io.ReadAll(response.Body)
if err != nil {
return nil, err
}

var enrs []string
err = yaml.Unmarshal(data, &enrs)
if err != nil {
return nil, err
}

return enrs, nil
}

// FetchDepositContractBlockFromURL fetches the deposit contract block from a given URL.
func FetchDepositContractBlockFromURL(ctx context.Context, url string) (uint64, error) {
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return 0, err
}

response, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer response.Body.Close()

data, err := io.ReadAll(response.Body)
if err != nil {
return 0, err
}

var block uint64

err = yaml.Unmarshal(data, &block)
if err != nil {
return 0, err
}

return block, nil
}

// FetchGenesisDetailsFromURL fetches the genesis time and validators root from a given URL.
func FetchGenesisDetailsFromURL(ctx context.Context, url string) (uint64, [32]byte, error) {
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return 0, [32]byte{}, err
}

response, err := http.DefaultClient.Do(req)
if err != nil {
return 0, [32]byte{}, err
}
defer response.Body.Close()

// Read only the first 40 bytes (8 bytes for GenesisTime + 32 bytes for GenesisValidatorsRoot)
data := make([]byte, 40)
_, err = io.ReadFull(response.Body, data)
if err != nil {
return 0, [32]byte{}, err
}

genesisTime := binary.LittleEndian.Uint64(data[:8])
var genesisValidatorsRoot [32]byte
copy(genesisValidatorsRoot[:], data[8:])

return genesisTime, genesisValidatorsRoot, nil
}
15 changes: 0 additions & 15 deletions eth/genesis.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,21 +44,6 @@ type GenesisConfig struct {
GenesisTime time.Time // Time at Genesis
}

// GetConfigsByNetworkName returns the GenesisConfig, NetworkConfig,
// BeaconChainConfig and any error based on the input network name
func GetConfigsByNetworkName(net string) (*GenesisConfig, *params.NetworkConfig, *params.BeaconChainConfig, error) {
switch net {
case params.MainnetName:
return GenesisConfigs[net], params.BeaconNetworkConfig(), params.MainnetConfig(), nil
case params.SepoliaName:
return GenesisConfigs[net], params.BeaconNetworkConfig(), params.SepoliaConfig(), nil
case params.HoleskyName:
return GenesisConfigs[net], params.BeaconNetworkConfig(), params.HoleskyConfig(), nil
default:
return nil, nil, nil, fmt.Errorf("network %s not found", net)
}
}

var GenesisConfigs = map[string]*GenesisConfig{
params.MainnetName: {
GenesisValidatorRoot: hexToBytes("4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95"),
Expand Down
Loading

0 comments on commit 094e08a

Please sign in to comment.