Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(docs): cannon #203

Merged
merged 1 commit into from
Sep 22, 2023
Merged

feat(docs): cannon #203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ Xatu can run in multiple modes. Each mode can be run independently. The followin
```
┌───────────┐
│ CONSENSUS │
│ CLIENT │
└─────▲─────┘
│
│
┌───▼────┐ ┌──────────┐ ┌───────────┐
│ XATU │ │ XATU │ │ XATU │
│ SENTRY │ │ MIMICRY │ │ DISCOVERY │
└───┬────┘ └─────┬────┘ └─────┬─────┘
│ │ │
│ │ │
│ ┌────▼─────┐ │
└───────► ◄───────┘
│ CLIENT ◄─────┐
└─────▲─────┘ │
│ │
│ │
┌───▼────┐ ┌────▼─────┐ ┌───────────┐ ┌───────────┐
│ XATU │ │ XATU │ │ XATU │ │ XATU │
│ SENTRY │ │ CANNON │ │ DISCOVERY │ │ DISCOVERY │
└───┬────┘ └─────┬────┘ └─────┬─────┘ └─────┬─────┘
│ │ │ │
│ │ │ │
│ ┌────▼─────┐ │ │
└───────► ◄───────┘─────────────┘
│ XATU │
│ SERVER │ ┌─────────────┐
│ ◄────► PERSISTENCE │
Expand All @@ -40,6 +40,7 @@ Follow the links for more information on each mode.
- [**Sentry**](./docs/sentry.md) - Client that runs along side a [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this projects sentry will connect to it via the consensus client's http server.
- [**Discovery**](./docs/discovery.md) - Client that uses the [Node Discovery Protocol v5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md) and [Node Discovery Protocol v4](https://github.com/ethereum/devp2p/blob/master/discv4.md) to discovery nodes on the network. Also attempts to connect to execution layer nodes and collect meta data from them.
- [**Mimicry**](./docs/mimicry.md) - Client that collects data from the execution layer P2P network.
- [**Cannon**](./docs/sentry.md) - Client that runs along side a [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects canonical finalized data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this projects cannon client will connect to it via the consensus client's http server.

## Getting Started

Expand Down
205 changes: 205 additions & 0 deletions docs/cannon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Cannon

Client that is run along side an [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects canonical data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this cannon will connect to it via the consensus client's http server.

This cannon can output events to various sinks and it is **not** a hard requirement to run the [Xatu server](./server.md).

## Table of contents

- [Usage](#usage)
- [Requirements](#requirements)
- [Configuration](#configuration)
- [`xatu` output](#output-xatu-configuration)
- [`http` output](#output-http-configuration)
- [Simple example](#simple-example)
- [Xatu server output example](#xatu-server-output-example)
- [HTTP server output example](#http-server-output-example)
- [Complex example with multiple outputs example](#complex-example-with-multiple-outputs-example)
- [Running locally](#running-locally)

## Usage

Cannon requires a [config file](#configuration).

```bash
Usage:
xatu cannon [flags]

Flags:
--config string config file (default is cannon.yaml) (default "cannon.yaml")
-h, --help help for cannon
```

## Requirements

- [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) with exposed [http server](https://ethereum.github.io/beacon-APIs/).
- [Server](./server.md) running with the [Coordinator](./server.md#coordinator) service enabled.

## Configuration

Cannon requires a single `yaml` config file. An example file can be found [here](../example_cannon.yaml)

| Name| Type | Default | Description |
| --- | --- | --- | --- |
| logging | string | `warn` | Log level (`panic`, `fatal`, `warn`, `info`, `debug`, `trace`) |
| metricsAddr | string | `:9090` | The address the metrics server will listen on |
| pprofAddr | string | | The address the [pprof](https://github.com/google/pprof) server will listen on. When ommited, the pprof server will not be started |
| name | string | | Unique name of the cannon |
| labels | object | | A key value map of labels to append to every cannon event |
| ethereum.beaconNodeAddress | string | | [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) http server endpoint |
| ethereum.beaconNodeAddress | object | | A key value map of headers |
| ethereum.overrideNetworkName | string | | Override the network name |
| ethereum.blockCacheSize | int | `1000` | The maximum number of blocks to cache |
| ethereum.blockCacheTtl | string | `1h` | The maximum duration to cache blocks |
| ethereum.blockPreloadWorkers | int | `5` | The number of workers to use for preloading blocks |
| ethereum.blockPreloadQueueSize | int | `5000` | The maximum number of blocks to queue for preloading |
| coordinator.address | string | | The address of the [Xatu server](./server.md) |
| coordinator.tls | bool | | Server requires TLS |
| coordinator.headers | object | | A key value map of headers to append to requests |
| derivers.attesterSlashing.enabled | bool | `true` | Enable the attester slashing deriver |
| derivers.attesterSlashing.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.blsToExecutionChange.enabled | bool | `true` | Enable the BLS to execution change deriver |
| derivers.blsToExecutionChange.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.deposit.enabled | bool | `true` | Enable the deposit deriver |
| derivers.deposit.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.withdrawal.enabled | bool | `true` | Enable the withdrawal deriver |
| derivers.withdrawal.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.executionTransaction.enabled | bool | `true` | Enable the execution transaction deriver |
| derivers.executionTransaction.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.proposerSlashing.enabled | bool | `true` | Enable the proposer slashing deriver |
| derivers.proposerSlashing.headSlotLag | int | `5` | The number of slots to lag behind the head |
| derivers.voluntaryExit.enabled | bool | `true` | Enable the voluntary exit deriver |
| derivers.voluntaryExit.headSlotLag | int | `5` | The number of slots to lag behind the head |
| ntpServer | string | `pool.ntp.org` | NTP server to calculate clock drift for events |
| outputs | array<object> | | List of outputs for the cannon to send data to |
| outputs[].name | string | | Name of the output |
| outputs[].type | string | | Type of output (`xatu`, `http`, `stdout`) |
| outputs[].config | object | | Output type configuration [`xatu`](#output-xatu-configuration)/[`http`](#output-http-configuration) |

### Output `xatu` configuration

Output configuration to send cannon events to a [Xatu server](./server.md).

| Name| Type | Default | Description |
| --- | --- | --- | --- |
| outputs[].config.address | string | | The address of the server receiving events |
| outputs[].config.tls | bool | | Server requires TLS |
| outputs[].config.headers | object | | A key value map of headers to append to requests |
| outputs[].config.maxQueueSize | int | `51200` | The maximum queue size to buffer events for delayed processing. If the queue gets full it drops the events |
| outputs[].config.batchTimeout | string | `5s` | The maximum duration for constructing a batch. Processor forcefully sends available events when timeout is reached |
| outputs[].config.exportTimeout | string | `30s` | The maximum duration for exporting events. If the timeout is reached, the export will be cancelled |
| outputs[].config.maxExportBatchSize | int | `512` | MaxExportBatchSize is the maximum number of events to process in a single batch. If there are more than one batch worth of events then it processes multiple batches of events one batch after the other without any delay |

### Output `http` configuration

Output configuration to send cannon events to a http server.

| Name| Type | Default | Description |
| --- | --- | --- | --- |
| outputs[].config.address | string | | The address of the server receiving events |
| outputs[].config.headers | object | | A key value map of headers to append to requests |
| outputs[].config.maxQueueSize | int | `51200` | The maximum queue size to buffer events for delayed processing. If the queue gets full it drops the events |
| outputs[].config.batchTimeout | string | `5s` | The maximum duration for constructing a batch. Processor forcefully sends available events when timeout is reached |
| outputs[].config.exportTimeout | string | `30s` | The maximum duration for exporting events. If the timeout is reached, the export will be cancelled |
| outputs[].config.maxExportBatchSize | int | `512` | MaxExportBatchSize is the maximum number of events to process in a single batch. If there are more than one batch worth of events then it processes multiple batches of events one batch after the other without any delay |

### Simple example

```yaml
name: xatu-cannon

coordinator:
address: http://localhost:8080

ethereum:
beaconNodeAddress: http://localhost:5052

outputs:
- name: standard-out
type: stdout
```

### Xatu server output example

```yaml
name: xatu-cannon

coordinator:
address: http://localhost:8080

ethereum:
beaconNodeAddress: http://localhost:5052

outputs:
- name: xatu-output
type: xatu
config:
address: localhost:8080
```

### http server output example

```yaml
name: xatu-cannon

coordinator:
address: http://localhost:8080

ethereum:
beaconNodeAddress: http://localhost:5052

outputs:
- name: http-basic-auth
type: http
config:
address: http://localhost:8080
headers:
authorization: "Basic Someb64Value"
```

### Complex example with multiple outputs example

```yaml
logging: "debug"
metricsAddr: ":9090"
pprofAddr: ":6060"

name: xatu-cannon

labels:
ethpandaops: rocks

ntpServer: time.google.com

coordinator:
address: http://localhost:8080

ethereum:
beaconNodeAddress: http://localhost:5052

outputs:
- name: log
type: stdout
- name: xatu-server
type: xatu
config:
address: localhost:8080
headers:
authorization: Someb64Value
maxQueueSize: 51200
batchTimeout: 5s
exportTimeout: 30s
maxExportBatchSize: 512
```

## Running locally

```bash
# docker
docker run -d --name xatu-cannon -v $HOST_DIR_CHANGE_ME/config.yaml:/opt/xatu/config.yaml -p 9090:9090 -it ethpandaops/xatu:latest cannon --config /opt/xatu/config.yaml
# build
go build -o dist/xatu main.go
./dist/xatu cannon --config cannon.yaml
# dev
go run main.go cannon --config cannon.yaml
```
110 changes: 0 additions & 110 deletions docs/mimicry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,6 @@ Client that collects data from the execution layer P2P network.
- [HTTP server output example](#http-server-output-example)
- [Complex example with multiple outputs example](#complex-example-with-multiple-outputs-example)
- [Running locally](#running-locally)
- [Events](#events)
- [Event field](#event-field)
- [Meta field](#meta-field)
- [Beacon API - Event Stream](#beacon-api---event-stream)
- [Head](#head)
- [Block](#Block)
- [Attestation](#Attestation)
- [Voluntary exit](#voluntary-exit)
- [Finalized checkpoint](#finalized-checkpoint)
- [Chain reorg](#chain-reorg)
- [Contribution and proof](#contribution-and-proof)

## Usage

Expand Down Expand Up @@ -242,102 +231,3 @@ go build -o dist/xatu main.go
# dev
go run main.go mimicry --config mimicry.yaml
```

## Events

All events output from a **mimicry** contain the following fields;

- `event` - name and date time of the event
- `meta` - metadata of the event, including specific event details as well as mimicry/server computed info
- `data` - raw data collected for the event. This could be the Beacon API payload or a transaction.

An example event payload of a Beacon API event stream for the block topic;

```json
{
"event": {
"name": "MEMPOOL_TRANSACTION",
"date_time": "2022-01-01T10:12:10.050Z"
},
"meta": {
"client": {
"name": "some-client-001",
"version": "v1.0.1",
"id": "0697583c-3c65-4f9a-bcd0-b57ef919dc6c",
"os": "amiga4000-68040",
"clock_drift": 51,
"labels": {
"network-class": "tincan"
},
"ethereum": {
"network": {
"id": 1,
"name": "mainnet"
},
"execution": {
"fork_id": {
"hash": "0xf0afd0e3",
"next": "0"
}
}
},
"additional_data": {
"call_data_size":"0",
"from":"0x1d9e028b88A1638Dfd60D0d8d1476433D9307d8E",
"gas":"116900",
"gas_price":"35062538307",
"hash":"0x26d92491babaf51d5604347fb0dfdad323218bd11f762e34223dddd18db9adcd",
"size":"111",
"to":"0x1d9e028b44A1638Dfd60D0a8a1476433D9307e8E",
"value":"0"
}
}
},
"data": "0x02f86d01808420d85580850829e3e0438301d8a4941d9e028b88a1638dfd60d0d8d1476433d9307d8e8080d001a0d8ff2f83dd91fa1d0535a8a0d113ede5301065d9b0aa7b77f9b479412deda893a0700a3f95b0456e6ff2e0794e242a058daad1a5774fef9d078dfae3fb61ffa955"
}
```

### Event field

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| event.name | string | `required` | Event name |
| event.date_time | string | `required` | When the event occured |

### Meta field

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| meta.client | object | `required` | Meta data appended from the mimicry client |
| meta.client.name | string | `required` | Mimicry name |
| meta.client.version | string | `required` | Mimicry version |
| meta.client.id | string | `required` | Mimicry ID generated on service start |
| meta.client.implementation | string | `required` | Mimicry implementation eg. Xatu |
| meta.client.os | string | `optional` | Mimicry operating system |
| meta.client.clock_drift | int | `optional` | NTP calculated clock drift of the mimicry |
| meta.client.labels | object | | A key value map of labels |
| meta.client.ethereum.network.name | string | `required` | Ethereum network name eg. `mainnet`, `sepolia` |
| meta.client.ethereum.execution.fork_id | object | `optional` | ForkID [EIP-2124](https://eips.ethereum.org/EIPS/eip-2124) |
| meta.client.ethereum.execution.fork_id.hash | string | `optional` | IEEE CRC32 checksum of the genesis hash and fork blocks numbers that already passed |
| meta.client.ethereum.execution.fork_id.next | string | `optional` | Block number of the next upcoming fork, or 0 if no next fork is known |
| meta.client.additional_data | object | `optional` | Computed additional data to compliment the events upstream raw data eg. calculating slot start date time |

### Execution layer

When the mimicry is connected to execution layer peer over the [Ethereum Wire Protocol](https://github.com/ethereum/devp2p/blob/master/caps/eth.md), the following events are emitted.

#### Transaction

`meta.event.name` is `MEMPOOL_TRANSACTION`

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| data| string | `required` | Raw transaction data |
| meta.additional_data.size | string | `required` | Size of the transaction in bytes |
| meta.additional_data.call_data_size | string | `required` | Size of the transaction call data in bytes |
| meta.additional_data.hash | string | `required` | Transaction hash |
| meta.additional_data.from | string | `required` | Transaction sender address |
| meta.additional_data.nonce | string | `required` | Transaction nonce |
| meta.additional_data.gas_price | string | `required` | Transaction gas price |
| meta.additional_data.gas | string | `required` | Transaction gas limit |
| meta.additional_data.value | string | `required` | Transaction value |
Loading
Loading