From 390c0e6e37655afd57c232d505b3ffb7a0176bfb Mon Sep 17 00:00:00 2001 From: Andrew Davis <1709934+Savid@users.noreply.github.com> Date: Fri, 22 Sep 2023 16:50:16 +1000 Subject: [PATCH] feat(docs): cannon --- README.md | 25 +++--- docs/cannon.md | 205 ++++++++++++++++++++++++++++++++++++++++++ docs/mimicry.md | 110 ----------------------- docs/sentry.md | 211 -------------------------------------------- docs/server.md | 126 -------------------------- example_cannon.yaml | 83 +++++++++++++++++ 6 files changed, 301 insertions(+), 459 deletions(-) create mode 100644 docs/cannon.md create mode 100644 example_cannon.yaml diff --git a/README.md b/README.md index 45be46b0..aecd8afd 100644 --- a/README.md +++ b/README.md @@ -9,18 +9,18 @@ Xatu can run in multiple modes. Each mode can be run independently. The followin ``` ┌───────────┐ │ CONSENSUS │ -│ CLIENT │ -└─────▲─────┘ - │ - │ - ┌───▼────┐ ┌──────────┐ ┌───────────┐ - │ XATU │ │ XATU │ │ XATU │ - │ SENTRY │ │ MIMICRY │ │ DISCOVERY │ - └───┬────┘ └─────┬────┘ └─────┬─────┘ - │ │ │ - │ │ │ - │ ┌────▼─────┐ │ - └───────► ◄───────┘ +│ CLIENT ◄─────┐ +└─────▲─────┘ │ + │ │ + │ │ + ┌───▼────┐ ┌────▼─────┐ ┌───────────┐ ┌───────────┐ + │ XATU │ │ XATU │ │ XATU │ │ XATU │ + │ SENTRY │ │ CANNON │ │ DISCOVERY │ │ DISCOVERY │ + └───┬────┘ └─────┬────┘ └─────┬─────┘ └─────┬─────┘ + │ │ │ │ + │ │ │ │ + │ ┌────▼─────┐ │ │ + └───────► ◄───────┘─────────────┘ │ XATU │ │ SERVER │ ┌─────────────┐ │ ◄────► PERSISTENCE │ @@ -40,6 +40,7 @@ Follow the links for more information on each mode. - [**Sentry**](./docs/sentry.md) - Client that runs along side a [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this projects sentry will connect to it via the consensus client's http server. - [**Discovery**](./docs/discovery.md) - Client that uses the [Node Discovery Protocol v5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md) and [Node Discovery Protocol v4](https://github.com/ethereum/devp2p/blob/master/discv4.md) to discovery nodes on the network. Also attempts to connect to execution layer nodes and collect meta data from them. - [**Mimicry**](./docs/mimicry.md) - Client that collects data from the execution layer P2P network. +- [**Cannon**](./docs/sentry.md) - Client that runs along side a [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects canonical finalized data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this projects cannon client will connect to it via the consensus client's http server. ## Getting Started diff --git a/docs/cannon.md b/docs/cannon.md new file mode 100644 index 00000000..1618868b --- /dev/null +++ b/docs/cannon.md @@ -0,0 +1,205 @@ +# Cannon + +Client that is run along side an [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) and collects canonical data via the consensus client's [Beacon API](https://ethereum.github.io/beacon-APIs/). *You must run your own consensus client* and this cannon will connect to it via the consensus client's http server. + +This cannon can output events to various sinks and it is **not** a hard requirement to run the [Xatu server](./server.md). + +## Table of contents + +- [Usage](#usage) +- [Requirements](#requirements) +- [Configuration](#configuration) + - [`xatu` output](#output-xatu-configuration) + - [`http` output](#output-http-configuration) + - [Simple example](#simple-example) + - [Xatu server output example](#xatu-server-output-example) + - [HTTP server output example](#http-server-output-example) + - [Complex example with multiple outputs example](#complex-example-with-multiple-outputs-example) +- [Running locally](#running-locally) + +## Usage + +Cannon requires a [config file](#configuration). + +```bash +Usage: + xatu cannon [flags] + +Flags: + --config string config file (default is cannon.yaml) (default "cannon.yaml") + -h, --help help for cannon +``` + +## Requirements + +- [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) with exposed [http server](https://ethereum.github.io/beacon-APIs/). +- [Server](./server.md) running with the [Coordinator](./server.md#coordinator) service enabled. + +## Configuration + +Cannon requires a single `yaml` config file. An example file can be found [here](../example_cannon.yaml) + +| Name| Type | Default | Description | +| --- | --- | --- | --- | +| logging | string | `warn` | Log level (`panic`, `fatal`, `warn`, `info`, `debug`, `trace`) | +| metricsAddr | string | `:9090` | The address the metrics server will listen on | +| pprofAddr | string | | The address the [pprof](https://github.com/google/pprof) server will listen on. When ommited, the pprof server will not be started | +| name | string | | Unique name of the cannon | +| labels | object | | A key value map of labels to append to every cannon event | +| ethereum.beaconNodeAddress | string | | [Ethereum consensus client](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients) http server endpoint | +| ethereum.beaconNodeAddress | object | | A key value map of headers | +| ethereum.overrideNetworkName | string | | Override the network name | +| ethereum.blockCacheSize | int | `1000` | The maximum number of blocks to cache | +| ethereum.blockCacheTtl | string | `1h` | The maximum duration to cache blocks | +| ethereum.blockPreloadWorkers | int | `5` | The number of workers to use for preloading blocks | +| ethereum.blockPreloadQueueSize | int | `5000` | The maximum number of blocks to queue for preloading | +| coordinator.address | string | | The address of the [Xatu server](./server.md) | +| coordinator.tls | bool | | Server requires TLS | +| coordinator.headers | object | | A key value map of headers to append to requests | +| derivers.attesterSlashing.enabled | bool | `true` | Enable the attester slashing deriver | +| derivers.attesterSlashing.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.blsToExecutionChange.enabled | bool | `true` | Enable the BLS to execution change deriver | +| derivers.blsToExecutionChange.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.deposit.enabled | bool | `true` | Enable the deposit deriver | +| derivers.deposit.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.withdrawal.enabled | bool | `true` | Enable the withdrawal deriver | +| derivers.withdrawal.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.executionTransaction.enabled | bool | `true` | Enable the execution transaction deriver | +| derivers.executionTransaction.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.proposerSlashing.enabled | bool | `true` | Enable the proposer slashing deriver | +| derivers.proposerSlashing.headSlotLag | int | `5` | The number of slots to lag behind the head | +| derivers.voluntaryExit.enabled | bool | `true` | Enable the voluntary exit deriver | +| derivers.voluntaryExit.headSlotLag | int | `5` | The number of slots to lag behind the head | +| ntpServer | string | `pool.ntp.org` | NTP server to calculate clock drift for events | +| outputs | array | | List of outputs for the cannon to send data to | +| outputs[].name | string | | Name of the output | +| outputs[].type | string | | Type of output (`xatu`, `http`, `stdout`) | +| outputs[].config | object | | Output type configuration [`xatu`](#output-xatu-configuration)/[`http`](#output-http-configuration) | + +### Output `xatu` configuration + +Output configuration to send cannon events to a [Xatu server](./server.md). + +| Name| Type | Default | Description | +| --- | --- | --- | --- | +| outputs[].config.address | string | | The address of the server receiving events | +| outputs[].config.tls | bool | | Server requires TLS | +| outputs[].config.headers | object | | A key value map of headers to append to requests | +| outputs[].config.maxQueueSize | int | `51200` | The maximum queue size to buffer events for delayed processing. If the queue gets full it drops the events | +| outputs[].config.batchTimeout | string | `5s` | The maximum duration for constructing a batch. Processor forcefully sends available events when timeout is reached | +| outputs[].config.exportTimeout | string | `30s` | The maximum duration for exporting events. If the timeout is reached, the export will be cancelled | +| outputs[].config.maxExportBatchSize | int | `512` | MaxExportBatchSize is the maximum number of events to process in a single batch. If there are more than one batch worth of events then it processes multiple batches of events one batch after the other without any delay | + +### Output `http` configuration + +Output configuration to send cannon events to a http server. + +| Name| Type | Default | Description | +| --- | --- | --- | --- | +| outputs[].config.address | string | | The address of the server receiving events | +| outputs[].config.headers | object | | A key value map of headers to append to requests | +| outputs[].config.maxQueueSize | int | `51200` | The maximum queue size to buffer events for delayed processing. If the queue gets full it drops the events | +| outputs[].config.batchTimeout | string | `5s` | The maximum duration for constructing a batch. Processor forcefully sends available events when timeout is reached | +| outputs[].config.exportTimeout | string | `30s` | The maximum duration for exporting events. If the timeout is reached, the export will be cancelled | +| outputs[].config.maxExportBatchSize | int | `512` | MaxExportBatchSize is the maximum number of events to process in a single batch. If there are more than one batch worth of events then it processes multiple batches of events one batch after the other without any delay | + +### Simple example + +```yaml +name: xatu-cannon + +coordinator: + address: http://localhost:8080 + +ethereum: + beaconNodeAddress: http://localhost:5052 + +outputs: +- name: standard-out + type: stdout +``` + +### Xatu server output example + +```yaml +name: xatu-cannon + +coordinator: + address: http://localhost:8080 + +ethereum: + beaconNodeAddress: http://localhost:5052 + +outputs: +- name: xatu-output + type: xatu + config: + address: localhost:8080 +``` + +### http server output example + +```yaml +name: xatu-cannon + +coordinator: + address: http://localhost:8080 + +ethereum: + beaconNodeAddress: http://localhost:5052 + +outputs: +- name: http-basic-auth + type: http + config: + address: http://localhost:8080 + headers: + authorization: "Basic Someb64Value" +``` + +### Complex example with multiple outputs example + +```yaml +logging: "debug" +metricsAddr: ":9090" +pprofAddr: ":6060" + +name: xatu-cannon + +labels: + ethpandaops: rocks + +ntpServer: time.google.com + +coordinator: + address: http://localhost:8080 + +ethereum: + beaconNodeAddress: http://localhost:5052 + +outputs: +- name: log + type: stdout +- name: xatu-server + type: xatu + config: + address: localhost:8080 + headers: + authorization: Someb64Value + maxQueueSize: 51200 + batchTimeout: 5s + exportTimeout: 30s + maxExportBatchSize: 512 +``` + +## Running locally + +```bash +# docker +docker run -d --name xatu-cannon -v $HOST_DIR_CHANGE_ME/config.yaml:/opt/xatu/config.yaml -p 9090:9090 -it ethpandaops/xatu:latest cannon --config /opt/xatu/config.yaml +# build +go build -o dist/xatu main.go +./dist/xatu cannon --config cannon.yaml +# dev +go run main.go cannon --config cannon.yaml +``` diff --git a/docs/mimicry.md b/docs/mimicry.md index fb89b1ea..8e6d52ac 100644 --- a/docs/mimicry.md +++ b/docs/mimicry.md @@ -14,17 +14,6 @@ Client that collects data from the execution layer P2P network. - [HTTP server output example](#http-server-output-example) - [Complex example with multiple outputs example](#complex-example-with-multiple-outputs-example) - [Running locally](#running-locally) -- [Events](#events) - - [Event field](#event-field) - - [Meta field](#meta-field) - - [Beacon API - Event Stream](#beacon-api---event-stream) - - [Head](#head) - - [Block](#Block) - - [Attestation](#Attestation) - - [Voluntary exit](#voluntary-exit) - - [Finalized checkpoint](#finalized-checkpoint) - - [Chain reorg](#chain-reorg) - - [Contribution and proof](#contribution-and-proof) ## Usage @@ -242,102 +231,3 @@ go build -o dist/xatu main.go # dev go run main.go mimicry --config mimicry.yaml ``` - -## Events - -All events output from a **mimicry** contain the following fields; - -- `event` - name and date time of the event -- `meta` - metadata of the event, including specific event details as well as mimicry/server computed info -- `data` - raw data collected for the event. This could be the Beacon API payload or a transaction. - -An example event payload of a Beacon API event stream for the block topic; - -```json -{ - "event": { - "name": "MEMPOOL_TRANSACTION", - "date_time": "2022-01-01T10:12:10.050Z" - }, - "meta": { - "client": { - "name": "some-client-001", - "version": "v1.0.1", - "id": "0697583c-3c65-4f9a-bcd0-b57ef919dc6c", - "os": "amiga4000-68040", - "clock_drift": 51, - "labels": { - "network-class": "tincan" - }, - "ethereum": { - "network": { - "id": 1, - "name": "mainnet" - }, - "execution": { - "fork_id": { - "hash": "0xf0afd0e3", - "next": "0" - } - } - }, - "additional_data": { - "call_data_size":"0", - "from":"0x1d9e028b88A1638Dfd60D0d8d1476433D9307d8E", - "gas":"116900", - "gas_price":"35062538307", - "hash":"0x26d92491babaf51d5604347fb0dfdad323218bd11f762e34223dddd18db9adcd", - "size":"111", - "to":"0x1d9e028b44A1638Dfd60D0a8a1476433D9307e8E", - "value":"0" - } - } - }, - "data": "0x02f86d01808420d85580850829e3e0438301d8a4941d9e028b88a1638dfd60d0d8d1476433d9307d8e8080d001a0d8ff2f83dd91fa1d0535a8a0d113ede5301065d9b0aa7b77f9b479412deda893a0700a3f95b0456e6ff2e0794e242a058daad1a5774fef9d078dfae3fb61ffa955" -} -``` - -### Event field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| event.name | string | `required` | Event name | -| event.date_time | string | `required` | When the event occured | - -### Meta field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.client | object | `required` | Meta data appended from the mimicry client | -| meta.client.name | string | `required` | Mimicry name | -| meta.client.version | string | `required` | Mimicry version | -| meta.client.id | string | `required` | Mimicry ID generated on service start | -| meta.client.implementation | string | `required` | Mimicry implementation eg. Xatu | -| meta.client.os | string | `optional` | Mimicry operating system | -| meta.client.clock_drift | int | `optional` | NTP calculated clock drift of the mimicry | -| meta.client.labels | object | | A key value map of labels | -| meta.client.ethereum.network.name | string | `required` | Ethereum network name eg. `mainnet`, `sepolia` | -| meta.client.ethereum.execution.fork_id | object | `optional` | ForkID [EIP-2124](https://eips.ethereum.org/EIPS/eip-2124) | -| meta.client.ethereum.execution.fork_id.hash | string | `optional` | IEEE CRC32 checksum of the genesis hash and fork blocks numbers that already passed | -| meta.client.ethereum.execution.fork_id.next | string | `optional` | Block number of the next upcoming fork, or 0 if no next fork is known | -| meta.client.additional_data | object | `optional` | Computed additional data to compliment the events upstream raw data eg. calculating slot start date time | - -### Execution layer - -When the mimicry is connected to execution layer peer over the [Ethereum Wire Protocol](https://github.com/ethereum/devp2p/blob/master/caps/eth.md), the following events are emitted. - -#### Transaction - -`meta.event.name` is `MEMPOOL_TRANSACTION` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| data| string | `required` | Raw transaction data | -| meta.additional_data.size | string | `required` | Size of the transaction in bytes | -| meta.additional_data.call_data_size | string | `required` | Size of the transaction call data in bytes | -| meta.additional_data.hash | string | `required` | Transaction hash | -| meta.additional_data.from | string | `required` | Transaction sender address | -| meta.additional_data.nonce | string | `required` | Transaction nonce | -| meta.additional_data.gas_price | string | `required` | Transaction gas price | -| meta.additional_data.gas | string | `required` | Transaction gas limit | -| meta.additional_data.value | string | `required` | Transaction value | diff --git a/docs/sentry.md b/docs/sentry.md index 541ad0fc..7f09e67a 100644 --- a/docs/sentry.md +++ b/docs/sentry.md @@ -16,17 +16,6 @@ This sentry can output events to various sinks and it is **not** a hard requirem - [HTTP server output example](#http-server-output-example) - [Complex example with multiple outputs example](#complex-example-with-multiple-outputs-example) - [Running locally](#running-locally) -- [Events](#events) - - [Event field](#event-field) - - [Meta field](#meta-field) - - [Beacon API - Event Stream](#beacon-api---event-stream) - - [Head](#head) - - [Block](#Block) - - [Attestation](#Attestation) - - [Voluntary exit](#voluntary-exit) - - [Finalized checkpoint](#finalized-checkpoint) - - [Chain reorg](#chain-reorg) - - [Contribution and proof](#contribution-and-proof) ## Usage @@ -179,203 +168,3 @@ go build -o dist/xatu main.go # dev go run main.go sentry --config sentry.yaml ``` - -## Events - -All events output from a **sentry** contain the following fields; - -- `event` - name and date time of the event -- `meta` - metadata of the event, including specific event details as well as sentry/server computed info -- `data` - raw data collected for the event. This could be the Beacon API payload or a transaction. - -An example event payload of a Beacon API event stream for the block topic; - -```json -{ - "event": { - "name": "BEACON_API_ETH_V1_EVENTS_BLOCK", - "date_time": "2022-01-01T10:12:10.050Z" - }, - "meta": { - "client": { - "name": "some-client-001", - "version": "v1.0.1", - "id": "0697583c-3c65-4f9a-bcd0-b57ef919dc6c", - "os": "amiga4000-68040", - "clock_drift": 51, - "labels": { - "network-class": "tincan" - }, - "ethereum": { - "network": { - "id": 1, - "name": "mainnet" - }, - "consensus": { - "implementation": "lighthouse", - "version": "v1.0.0-abc" - } - }, - "additional_data": { - "slot": { - "start_date_time": "2022-01-01T10:12:10.000Z" - }, - "epoch": { - "number": 1 - "start_date_time": "2022-01-01T10:10:10.000Z" - }, - "propagation": { - "slot_start_diff": 50 - } - } - } - }, - "data": { - "slot":"10", - "block":"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf", - "execution_optimistic": false - } -} -``` - -### Event field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| event.name | string | `required` | Event name | -| event.date_time | string | `required` | When the event occured | - -### Meta field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.client | object | `required` | Meta data appended from the sentry client | -| meta.client.name | string | `required` | Sentry name | -| meta.client.version | string | `required` | Sentry version | -| meta.client.id | string | `required` | Sentry ID generated on service start | -| meta.client.implementation | string | `required` | Sentry implementation eg. Xatu | -| meta.client.os | string | `optional` | Sentry operating system | -| meta.client.clock_drift | int | `optional` | NTP calculated clock drift of the sentry | -| meta.client.labels | object | | A key value map of labels | -| meta.client.ethereum.network.name | string | `required` | Ethereum network name eg. `mainnet`, `sepolia` | -| meta.client.ethereum.consensus.implementation | string | `optional` | Ethereum consensus client name upstream from this sentry | -| meta.client.ethereum.consensus.version | string | `optional` | Ethereum consensus client version upstream from this sentry | -| meta.client.additional_data | object | `optional` | Computed additional data to compliment the events upstream raw data eg. calculating slot start date time | - -### Beacon API - Event Stream - -When the sentry is connected to an upstream Ethereum consensus client, the Beacon API [Event stream](https://ethereum.github.io/beacon-APIs/#/Events/eventstream) will be subcribed to and events will be generated for each topic. - -#### Head - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_HEAD` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.slot.start_date_time | string | `required` | Slot start datetime computed from `data.slot` | -| meta.additional_data.propagation.slot_start_diff | string | `required` | Difference in milliseconds between the `meta.event.date_time` and `meta.additional_data.slot.start_date_time` | -| meta.additional_data.epoch.number | string | `required` | Epoch number computed from `data.slot` | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `meta.additional_data.epoch.number` | -| data.slot | string | `required` | | -| data.block | string | `required` | | -| data.state | string | `required` | | -| data.epoch_transition | string | `required` | | -| data.previous_duty_dependent_root | string | `required` | | -| data.current_duty_dependent_root | string | `required` | | - -#### Block - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_BLOCK` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.slot.start_date_time | string | `required` | Slot start datetime computed from `data.slot` | -| meta.additional_data.propagation.slot_start_diff | string | `required` | Difference in milliseconds between the `meta.event.date_time` and `meta.additional_data.slot.start_date_time` | -| meta.additional_data.epoch.number | string | `required` | Epoch number computed from `data.slot` | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `meta.additional_data.epoch.number` | -| data.slot | string | `required` | | -| data.block | string | `required` | | -| data.execution_optimistic | bool | `required` | | - -#### Attestation - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_ATTESTATION` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.slot.start_date_time | string | `required` | Slot start datetime computed from `data.data.slot` | -| meta.additional_data.propagation.slot_start_diff | string | `required` | Difference in milliseconds between the `meta.event.date_time` and `meta.additional_data.slot.start_date_time` | -| meta.additional_data.epoch.number | string | `required` | Epoch number computed from `data.data.slot` | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `meta.additional_data.epoch.number` | -| meta.additional_data.source.epoch.start_date_time | string | `required` | Epoch start datetime computed from `data.data.source.epoch` | -| meta.additexecutablesional_data.target.epoch.start_date_time | string | `required` | Epoch start datetime computed from `data.data.target.epoch` | -| data.aggregation_bits | string | `required` | | -| data.data.beacon_block_root | string | `required` | | -| data.data.index | string | `required` | | -| data.data.slot | string | `required` | | -| data.data.source.epoch | string | `required` | | -| data.data.source.root | string | `required` | | -| data.data.target.epoch | string | `required` | | -| data.data.target.root | string | `required` | | -| data.signature | string | `required` | | - -#### Voluntary exit - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_VOLUNTARY_EXIT` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `data.message.epoch` | -| data.message.epoch | string | `required` | | -| data.message.validator_index | string | `required` | | -| data.signature | string | `required` | | - -#### Finalized checkpoint - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_FINALIZED_CHECKPOINT` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `data.epoch` | -| data.block | string | `required` | | -| data.epoch | string | `required` | | -| data.execution_optimistic | bool | `required` | | -| data.state | string | `required` | | - -#### Chain reorg - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_CHAIN_REORG` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.slot.start_date_time | string | `required` | Slot start datetime computed from `data.slot` | -| meta.additional_data.propagation.slot_start_diff | string | `required` | Difference in milliseconds between the `meta.event.date_time` and `meta.additional_data.slot.start_date_time` | -| meta.additional_data.epoch.number | string | `required` | Epoch number computed from `data.slot` | -| meta.additional_data.epoch.start_date_time | string | `required` | Epoch start datetime computed from `meta.additional_data.epoch.number` | -| data.depth | string | `required` | | -| data.epoch | string | `required` | | -| data.execution_optimistic | bool | `required` | | -| data.new_head_block | string | `required` | | -| data.new_head_state | string | `required` | | -| data.old_head_block | string | `required` | | -| data.old_head_state | string | `required` | | -| data.slot | string | `required` | | - -#### Contribution and proof - -`meta.event.name` is `BEACON_API_ETH_V1_EVENTS_CONTRIBUTION_AND_PROOF` - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.additional_data.contribution.slot.start_date_time | string | `required` | Slot start datetime computed from `data.message.contribution.slot` | -| meta.additional_data.contribution.propagation.slot_start_diff | string | `required` | Difference in milliseconds between the `meta.event.date_time` and `meta.additional_data.slot.start_date_time` | -| meta.additional_data.contribution.epoch.number | string | `required` | Epoch number computed from `data.message.contribution.slot` | -| meta.additional_data.contribution.epoch.start_date_time | string | `required` | Epoch start datetime computed from `meta.additional_data.epoch.number` | -| data.message.aggregator_index | string | `required` | | -| data.message.contribution.aggregation_bits | string | `required` | | -| data.message.contribution.beacon_block_root | string | `required` | | -| data.message.contribution.signature | string | `required` | | -| data.message.contribution.slot | string | `required` | | -| data.message.contribution.subcommittee_index | string | `required` | | -| data.message.selection_proof | string | `required` | | -| data.signature | string | `required` | | diff --git a/docs/server.md b/docs/server.md index 905465ce..3a47d89a 100644 --- a/docs/server.md +++ b/docs/server.md @@ -16,10 +16,6 @@ A centralized server running configurable services collecting events from client - [Services](#services) - [Coordinator](#coordinator) - [Event Ingester](#event-ingester) - - [Events](#events) - - [Event field](#event-field) - - [Meta field](#meta-field) - - [Data field](#data-field) - [Persistence](#persistence) - [Migrations](#migrations) - [Running locally](#running-locally) @@ -207,128 +203,6 @@ The coordinator service is responsible for; The event ingester service is responsible for receiving events from clients (sentries), validating and then forwarding them to sinks. -#### Events - -All events output from the ingester contain the following fields; - -- `event` - name and date time of the event -- `meta` - metadata of the event, including specific event details as well as sentry/server computed info -- `data` - raw data collected for the event. This could be the Beacon API payload or a transaction. - -The following is a sample event output is from a [Xatu sentry block](./sentry.md#block) event; - -```json -{ - "event": { - "name": "BEACON_API_ETH_V1_EVENTS_BLOCK", - "date_time": "2022-01-01T10:12:10.050Z" - }, - "meta": { - "client": { - "name": "some-client-001", - "version": "v1.0.1", - "id": "0697583c-3c65-4f9a-bcd0-b57ef919dc6c", - "os": "amiga4000-68040", - "clock_drift": 51, - "labels": { - "network-class": "tincan" - }, - "ethereum": { - "network": { - "id": 1, - "name": "mainnet" - }, - "consensus": { - "implementation": "lighthouse", - "version": "v1.0.0-abc" - } - }, - "additional_data": { - "slot": { - "start_date_time": "2022-01-01T10:12:10.000Z" - }, - "epoch": { - "number": 1 - "start_date_time": "2022-01-01T10:10:10.000Z" - }, - "propagation": { - "slot_start_diff": 50 - } - } - }, - "server": { - "event": { - "received_date_time": "2022-01-01T10:12:10.050Z" - }, - "client": { - "ip": "1.1.1.1", - "geo": { - "city": "Brisbane", - "country": "Australia", - "country_code": "AU", - "continent_code": "OC", - "latitude": -27.4698, - "longitude": 153.0251, - "autonomous_system_number": 1221, - "autonomous_system_organization": "Telstra Corporation Ltd", - "isp": "Telstra Internet", - "organization": "Telstra Corporation Ltd" - } - } - } - }, - "data": { - "slot":"10", - "block":"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf", - "execution_optimistic": false - } -} -``` - -##### Event field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| event.name | string | `required` | Event name | -| event.date_time | string | `required` | When the event occured | - -##### Meta field - -| Name | Type | Required | Description | -| --- | --- | --- | --- | -| meta.client | object | `required` | Meta data appended from the sentry client | -| meta.client.name | string | `required` | Sentry name | -| meta.client.version | string | `required` | Sentry version | -| meta.client.id | string | `required` | Sentry ID generated on service start | -| meta.client.implementation | string | `required` | Sentry implementation eg. Xatu | -| meta.client.os | string | `optional` | Sentry operating system | -| meta.client.clock_drift | int | `optional` | NTP calculated clock drift of the sentry | -| meta.client.labels | object | | A key value map of labels | -| meta.client.ethereum.network.name | string | `required` | Ethereum network name eg. `mainnet`, `sepolia` | -| meta.client.ethereum.consensus.implementation | string | `optional` | Ethereum consensus client name upstream from this sentry | -| meta.client.ethereum.consensus.version | string | `optional` | Ethereum consensus client version upstream from this sentry | -| meta.client.additional_data | object | `optional` | Computed additional data to compliment the events upstream raw data eg. calculating slot start date time | -| meta.server | object | `required` | Meta data appended from the server | -| meta.server.event.received_date_time | string | `required` | When the event was received by the server | -| meta.server.client.ip | string | `optional` | IP address of the client | -| meta.server.client.geo | object | `optional` | Geo data of the client | -| meta.server.client.geo.city | string | `optional` | City of the client | -| meta.server.client.geo.country | string | `optional` | Country of the client | -| meta.server.client.geo.country_code | string | `optional` | Country code of the client | -| meta.server.client.geo.continent_code | string | `optional` | Continent code of the client | -| meta.server.client.geo.latitude | float | `optional` | Latitude of the client | -| meta.server.client.geo.longitude | float | `optional` | Longitude of the client | -| meta.server.client.geo.autonomous_system_number | int | `optional` | Autonomous system number of the client | -| meta.server.client.geo.autonomous_system_organization | string | `optional` | Autonomous system organization of the client | -| meta.server.client.geo.isp | string | `optional` | ISP of the client | -| meta.server.client.geo.organization | string | `optional` | Organization of the client | - -##### Data field - -Depending on the origin of the event the data field will contain different data; -- [Xatu sentry](./sentry.md#events) events -- [Xatu mimicry](./mimicry.md#events) events - ## Persistence The server uses persistence layer to store data from various services. Currently only [PostgreSQL](https://www.postgresql.org/) is supported. diff --git a/example_cannon.yaml b/example_cannon.yaml new file mode 100644 index 00000000..706de0f9 --- /dev/null +++ b/example_cannon.yaml @@ -0,0 +1,83 @@ +logging: "debug" # panic,fatal,warn,info,debug,trace +metricsAddr: ":9090" +# pprofAddr: ":6060" # optional. if supplied it enables pprof server + +name: example-instance + +labels: + ethpandaops: rocks + +# Better to use a NTP server close eg. +# time.aws.com - AWS +# time.windows.com - Azure +# time.google.com - GCP +# pool.ntp.org - https://www.pool.ntp.org/zone/@ +ntpServer: time.google.com + +coordinator: + address: localhost:8080 + # tls: false + # headers: + # authorization: Someb64Value + +ethereum: + beaconNodeAddress: http://localhost:5052 + # beaconNodeHeaders: + # authorization: Someb64Value + # overrideNetworkName: mainnet + # blockCacheSize: 1000 + # blockCacheTtl: 1h + # blockPreloadWorkers: 5 + # blockPreloadQueueSize: 5000 + +# derivers: +# attesterSlashing: +# enabled: true +# headSlotLag: 5 +# blsToExecutionChange: +# enabled: true +# headSlotLag: 5 +# deposit: +# enabled: true +# headSlotLag: 5 +# withdrawal: +# enabled: true +# headSlotLag: 5 +# executionTransaction: +# enabled: true +# headSlotLag: 5 +# proposerSlashing: +# enabled: true +# headSlotLag: 5 +# voluntaryExit: +# enabled: true +# headSlotLag: 5 + +outputs: +- name: http-sink + type: http + # filter: + # eventNames: + # - BEACON_API_ETH_V1_EVENTS_BLOCK_DEPOSIT + config: + address: http://localhost:8080 + headers: + authorization: Someb64Value + maxQueueSize: 51200 + batchTimeout: 5s + exportTimeout: 30s + maxExportBatchSize: 512 +- name: xatu-server + type: xatu + # filter: + # eventNames: + # - BEACON_API_ETH_V1_EVENTS_BLOCK_DEPOSIT + config: + address: localhost:8080 + tls: false + headers: + authorization: Someb64Value + maxQueueSize: 51200 + batchTimeout: 5s + exportTimeout: 30s + maxExportBatchSize: 512