Skip to content
/ refiner Public

Concurrent block specimen data processing and transformation framework for the Covalent Network.

www.covalenthq.com/network/

License

Apache-2.0 license
9 stars 13 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

covalenthq/refiner

BranchesTags

Repository files navigation

Refiner

Introduction

Layers

Refiner is the primary Block Specimen data processing framework, the purpose of which is data transformation and various block data representations.

Generally, the Refiner can perform arbitrary transformations over any binary block specimen file concurrently. This enables simultaneous data indexing, with any consumer of the data slicing and dicing the data as they see fit. Such concurrent execution of Ethereum blocks (via block specimens) makes it possible to trace, enrich or analyze blockchain data at an unprecedented rate with no sequential bottlenecks (provided each block specimen is its own independent entity and available at a decentralized content address!).

The Refiner Whitepaper goes into detail on its purpose and mission.

Among many of the Refiner's outputs feasible, the Block Result is one. The block result is a one-to-one representation of block data returned from an RPC call to a blockchain node along with the artifacts of block and tx execution like transaction receipts. The source of the block result, the block specimen, captures a few extra fields like the State Specimen and senders etc. This full specification and its requirement are described well in the BSP whitepaper.

Running the Refiner stack involves running three main pieces of CQT Network OS infrastructure, refiner, ipfs-pinner and evm-server that coordinate and execute a transformation pipeline per Block Specimen.

Here refiner serves as the primary orchestrator and supervisor for all transformation pipeline processes that locates a source CQT Network data object to apply a tracing/execution/transformational rule to and outputs a new object generated from using such a rule.

Running these nodes are not as disk I/O (or cpu/memory) intense as running bsp-geth and bsp-agent on local machines, however they do require sufficient bandwidth for access to distributed store resources and sending transaction proofs using ethereum clients - connecting to our EVM public blockchain partner - moonbeam. We shall setup all of these step-by-step in this guide in two main ways:

Raison d'être

Phase2

As it stands, the Block Specimen captures a state snapshot. That is, all of the state read and transaction information. However, it doesn't capture the side effects of executing the block, the information you would get from a trace, transaction re-execution artifacts like receipts, what contracts were interacted with during a transaction, etc.

In theory, the Block Specimen could contain this information. However, the Block Specimen only includes the minimum information required to re-execute a block and all its transactions, making the network and decentralized storage more efficient. Extracting and storing block specimens allows us to move away from relying on blockchain nodes (executing blocks sequentially) to re-execute the block and draw detailed insights from underlying captured data. This process can be made concurrent, as block specimens can be executed independently as currently done with the erigon/covalent t8n tool, and does not need any state constructed from previous block specimens.

At a very high level, the Refiner locates a source to apply a transformational rule to and outputs an object generated from using such a rule. The source and output are available through a decentralized storage service such as a wrapped IPFS node. A transformation-proof transaction is emitted, confirming that it has done this work along with the output content ids (ipfs) access URL. To define what these three components are:

  • Source: The Block Specimen that serves as an input to the Refiner. Proof transactions made earlier to a smart contract with the respective cids are where the source is checked.

  • Rule: A transformation plugin (or server) that can act on the Block Specimen (source). These can be compared to blueprints that have been shown to produce the data objects needed. Furthermore, anyone can create these rules to get a desired data object. Rule (or server) versions thus need to exist, tied to the decoded block specimen versions they are applied on.

  • Target: The output generated from running the rule over the object that came from the source that is the block result.

Architecture

Refiner Pipeline

The happy path for refiner application in the CQT Network is made up of actor processes spawned through many Gen Servers processes that are loosely coupled, here some maintain state, and some don't. The children processes can be called upon to fulfill responsibilities at different sections in the refinement/transformation process pipeline - under one umbrella Dynamic Supervisor, that can bring them back up in case of a failure to continue a given pipeline operation. Read more about the components and their operations in the FULL ARCHITECTURE document.

Resources

Production of Block Results forms the core of the cQT network’s functional data objects specs. These result objects are created using six main pieces of open-source software published by Covalent for the CQT Network’s decentralized blockchain data ETL stack.

Resources

  1. Block Specimen Producer (Produce) - Operator run & deployed. Open source granular bulk blockchain data production and export of block specimens.

  2. BSP Agent (Extract & Prove) - Operator run & deployed. Open source packaging, encoding, proving (and storing specification) of block specimens.

  3. BSP Staking (Consensus) - Covalent foundation operated & pre-deployed. Open source CQT staking and proof aggregating smart contracts deployed on moonbeam used for consensus and rewards distribution for cQT network operators.

  4. IPFS Pinner (Store) - Operator run & deployed. Open source decentralized storage layer for cQT network block specimen and transformations (results).

  5. BSP Finalizer (Reward) - Covalent foundation operated & pre-deployed. Open source (anyone can call the rewards function on BSP Staking contracts) rewards distributer for cQT network operators.

  6. T8n Server (Transform) - Operator run & deployed. Open source Ethereum virtual machine binary (stateless transition tool - t8n) plugin/http server for the refiner.

  7. Refiner (Refine & Prove) - Operator run & deployed. Open Source specialized transformation process and framework for block specimens to block results (Open sourced in Q2 2023).

Additional Resources

Requirements

Minimum

  • 2 vCPUs (cores)
  • 3GB RAM
  • 200GB HDD free storage (mostly storing artifacts on local ipfs; can be pruned periodically)
  • 8 MBit/sec download Internet service

Recommended

  • 4 vCPUs
  • 8 GB+ RAM
  • SSD with >= 500GB storage space
  • 25+ MBit/sec download Internet service

Software Requirements (docker setup)

  • 64-bit Linux, Mac OS 13+
  • SSL certificates
  • docker, docker-compose, direnv

Run With Docker Compose

Install Docker

Follow the instructions for your platform/architecture: https://docs.docker.com/engine/install.

README instructions will be based on Ubuntu 22.04 LTS x86_64/amd64: https://docs.docker.com/engine/install/ubuntu/.

Environment

Install direnv

sudo apt update
sudo apt get direnv

# bash users - add the following line to your ~/.bashrc
eval "$(direnv hook bash)"
source ~/.bashrc

# zsh users - add the following line to your ~/.zshrc
eval "$(direnv hook zsh)"
source ~/.zshrc

Create envrc.local file and add the following env vars.

Note: When passing the private key into the env vars as above please remove the 0x prefix so the private key env var has exactly 64 characters. Note: You can get the value for W3_AGENT_KEY by asking on discord. It has to be issued to you from Covalent.

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="BRP-OPERATOR-PK-WITHOUT-0x-PREFIX"
export NODE_ETHEREUM_MAINNET="<<ASK-ON-DISCORD>>"
export IPFS_PINNER_URL="http://ipfs-pinner:3001"
export EVM_SERVER_URL="http://evm-server:3002"
export W3_AGENT_KEY="<<W3_AGENT_KEY>>"

Load the env vars.

direnv allow .

That will lead to the corresponding logs:

direnv: loading ~/refiner/.envrc
direnv: loading ~/refiner/.envrc.local
direnv: export +BLOCK_RESULT_OPERATOR_PRIVATE_KEY +ERIGON_NODE +EVM_SERVER_URL +IPFS_PINNER_URL +NODE_ETHEREUM_MAINNET +W3_AGENT_KEY

This shows that the shell is loaded correctly. You can check if they're what you expect.

echo $IPFS_PINNER_URL
http://ipfs-pinner:3001

Copy over the delegation proof file to ~/.ipfs repo. You should have gotten this from Covalent in addition to the W3_AGENT_KEY value.

mv path_to_delegation_file ~/.ipfs/proof.out

Pull

Run all services including refiner with docker compose with the following.

Note: the env file is not necessary if env vars are already loaded.

For moonbase.

docker compose --env-file ".env" -f "docker-compose-mbase.yml" up --remove-orphans

For moonbeam.

docker compose --env-file ".env" -f "docker-compose-mbeam.yml" up --remove-orphans

NOTE: On a system where an ipfs-pinner instance is already running, check the instruction in the Appendix to run refiner docker alongside.

Running this will pull all the images and services that are ready to run.

This will lead to the corresponding logs:

Started refiner compose.
  refiner Pulling
  ipfs-pinner Pulling
  evm-server Pulling
  4f4fb700ef54 Downloading [==================================================>]      32B/32B
  4f4fb700ef54 Verifying Checksum
  4f4fb700ef54 Download complete
  ipfs-pinner Pulled
  0b5445b067f6 Extracting [=>                                                 ]  393.2kB/11.45MB
  0b5445b067f6 Extracting [==========>                                        ]  2.359MB/11.45MB
  1fd45119e007 Downloading [==============>                                    ]  4.317MB/14.94MB
  evm-server Pulled
  1fd45119e007 Extracting [>                                                  ]  163.8kB/14.94MB
  1fd45119e007 Extracting [=>                                                 ]  491.5kB/14.94MB
  1fd45119e007 Extracting [==============>                                    ]   4.26MB/14.94MB
  1fd45119e007 Extracting [=========================>                         ]  7.537MB/14.94MB
  1fd45119e007 Extracting [====================================>              ]  10.81MB/14.94MB
  1fd45119e007 Extracting [================================================>  ]  14.42MB/14.94MB
  1fd45119e007 Extracting [==================================================>]  14.94MB/14.94MB
  1fd45119e007 Pull complete
  refiner Pulled
  Container evm-server  Recreate
  Container ipfs-pinner  Created
  Container evm-server  Recreated
  Container refiner  Recreated
 Attaching to evm-server, ipfs-pinner, refiner

Following this step a refiner release is auto compiled within the docker container and executed.

Note: The below example is for the dev env (_build/dev/rel/refiner/bin/refiner) binary referred to as moonbase.

For production the path to the binary would be to the prod env (_build/prod/rel/refiner/bin/refiner). This distinction is important since in elixir the static env vars (such as proof-chain contract address) are packaged with dynamic env vars (such as rpc to moonbeam/moonbase) along with the application binary.

Hence there is a single binary per "Environment". To understand more about this take a look at the hex docs.

 ipfs-pinner  | generating 2048-bit RSA keypair...done
 ipfs-pinner  | peer identity: QmeP85RqKmrTwbW6PTQoX3NGwkgLX8DhGa1mkdy67sHZMZ
 evm-server   | [INFO] [04-19|16:53:31.113] Listening                                port=3002
 ipfs-pinner  |
 ipfs-pinner  | Computing default go-libp2p Resource Manager limits based on:
 ipfs-pinner  |     - 'Swarm.ResourceMgr.MaxMemory': "4.2 GB"
 ipfs-pinner  |     - 'Swarm.ResourceMgr.MaxFileDescriptors': 524288
 ipfs-pinner  |
 ipfs-pinner  | Applying any user-supplied overrides on top.
 ipfs-pinner  | Run 'ipfs swarm limit all' to see the resulting limits.
 ipfs-pinner  |
 ipfs-pinner  | 2023/04/19 16:53:31 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
 refiner       | moonbase-node: https://moonbeam-alphanet.web3.covalenthq.com/alphanet/direct-rpc
 refiner       | brp-operator: ecf0b636233c6580f60f50ee1d809336c3a76640dbd77f7cdd054a82c6fc0a31
 refiner       | evm-server: http://evm-server:3002
 refiner       | ipfs-node: http://ipfs-pinner:3001
 ipfs-pinner  | 2023/04/19 16:53:31 Listening...
 refiner       | ==> nimble_options
 refiner       | Compiling 3 files (.ex)
 refiner       | Generated nimble_options app
 refiner       | ===> Analyzing applications...
 refiner       | ===> Compiling parse_trans
 refiner       | ==> logger_file_backend
 refiner       | Compiling 1 file (.ex)
 refiner       | Generated refiner app
 refiner       | * assembling refiner-0.2.2 on MIX_ENV=dev
 refiner       | * skipping runtime configuration (config/runtime.exs not found)
 refiner       | * skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
 refiner       | * skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)
 refiner       |
 refiner       | Release created at _build/dev/rel/refiner
 refiner       |
 refiner       |     # To start your system
 refiner       |     _build/dev/rel/refiner/bin/refiner start
 refiner       |
 refiner       | Once the release is running:
 refiner       |
 refiner       |     # To connect to it remotely
 refiner       |     _build/dev/rel/refiner/bin/refiner remote
 refiner       |
 refiner       |     # To stop it gracefully (you may also send SIGINT/SIGTERM)
 refiner       |     _build/dev/rel/refiner/bin/refiner stop
 refiner       |
 refiner       | To list all commands:
 refiner       |
 refiner       |     _build/dev/rel/refiner/bin/refiner
 refiner       |
 refiner       | https://hexdocs.pm/telemetry/telemetry.html#attach/4

Docker Run

Once the binary is compiled. Refiner can start to process block specimens into block results by starting the event listener.

 refiner       | [info] starting event listener
 refiner       | [info] getting ids with status=discover
 refiner       | [info] found 1 bsps to process
 ipfs-pinner  | 2023/04/19 16:57:32 unixfsApi.Get: getting the cid: bafybeifkn67rc4lzoabvaglsifjitkhrnshhpwavutdhzeohzkxih25jpi
 ipfs-pinner  | 2023/04/19 16:57:32 trying out https://w3s.link/ipfs/bafybeifkn67rc4lzoabvaglsifjitkhrnshhpwavutdhzeohzkxih25jpi
 refiner       | [info] Counter for ipfs_metrics - [fetch: 1]
 refiner       | [info] LastValue for ipfs_metrics - [fetch_last_exec_time: 0.0015149999999999999]
 refiner       | [info] Sum for ipfs_metrics - [fetch_total_exec_time: 0.0015149999999999999]
 refiner       | [info] Summary for ipfs_metrics  - {0.0015149999999999999, 0.0015149999999999999}
 refiner       | [debug] reading schema `block-ethereum` from the file /app/priv/schemas/block-ethereum.avsc
 refiner       | [info] Counter for bsp_metrics - [decode: 1]
 refiner       | [info] LastValue for bsp_metrics - [decode_last_exec_time: 0.0]
 refiner       | [info] Sum for bsp_metrics - [decode_total_exec_time: 0.0]
 refiner       | [info] Summary for bsp_metrics  - {0.0, 0.0}
 refiner       | [info] submitting 17081820 to evm http server...
 evm-server   | [INFO] [04-19|16:57:33.824] input file at                            loc=/tmp/23851799
 evm-server   | [INFO] [04-19|16:57:33.828] output file at:                          loc=/tmp/1143694015
 evm-server   | [INFO] [04-19|16:57:34.153] Wrote file                               file=/tmp/1143694015
 refiner       | [info] writing block result into "/tmp/briefly-1681/briefly-576460651588718236-AE8SrEl8GLI9jKhCKPk"
 refiner       | [info] Counter for bsp_metrics - [execute: 1]
 refiner       | [info] LastValue for bsp_metrics - [execute_last_exec_time: 3.9e-4]
 refiner       | [info] Sum for bsp_metrics - [execute_total_exec_time: 3.9e-4]
 refiner       | [info] Summary for bsp_metrics  - {3.9e-4, 3.9e-4}
 ipfs-pinner  | 2023/04/19 16:57:34 generated dag has root cid: bafybeifd6gz6wofk3bwb5uai7zdmmr23q3nz3zt7edfujgj4kjg2es7eee
 ipfs-pinner  | 2023/04/19 16:57:34 car file location: /tmp/1543170755.car
 ipfs-pinner  | 2023/04/19 16:57:35 uploaded file has root cid: bafybeifd6gz6wofk3bwb5uai7zdmmr23q3nz3zt7edfujgj4kjg2es7eee
 refiner       | [info] Counter for ipfs_metrics - [pin: 1]
 refiner       | [info] LastValue for ipfs_metrics - [pin_last_exec_time: 0.001132]
 refiner       | [info] Sum for ipfs_metrics - [pin_total_exec_time: 0.001132]
 refiner       | [info] Summary for ipfs_metrics  - {0.001132, 0.001132}
 refiner       | [info] 17081820:556753def2ff689c6312241a1ca182d58467319b7c2dca250ca50ed6acb31a5d has been successfully uploaded at ipfs://bafybeifd6gz6wofk3bwb5uai7zdmmr23q3nz3zt7edfujgj4kjg2es7eee
 refiner       | [info] 17081820:556753def2ff689c6312241a1ca182d58467319b7c2dca250ca50ed6acb31a5d proof submitting
 refiner       | [info] Counter for brp_metrics - [proof: 1]
 refiner       | [info] LastValue for brp_metrics - [proof_last_exec_time: 3.03e-4]
 refiner       | [info] Sum for brp_metrics - [proof_total_exec_time: 3.03e-4]
 refiner       | [info] Summary for brp_metrics  - {3.03e-4, 3.03e-4}
 refiner       | [info] 17081820 txid is 0x01557912a0f7e083cbf6d34a2af21d99d129af386b95edc16162202862c60f8d
 refiner       | [info] Counter for brp_metrics - [upload_success: 1]
 refiner       | [info] LastValue for brp_metrics - [upload_success_last_exec_time: 0.0014579999999999999]
 refiner       | [info] Sum for brp_metrics - [upload_success_total_exec_time: 0.0014579999999999999]
 refiner       | [info] Summary for brp_metrics  - {0.0014579999999999999, 0.0014579999999999999}
 refiner       | [info] Counter for refiner_metrics - [pipeline_success: 1]
 refiner       | [info] LastValue for refiner_metrics - [pipeline_success_last_exec_time: 0.0035489999999999996]
 refiner       | [info] Sum for refiner_metrics - [pipeline_success_total_exec_time: 0.0035489999999999996]
 refiner       | [info] Summary for refiner_metrics  - {0.0035489999999999996, 0.0035489999999999996}
 refiner       | [info] curr_block: 4180658 and latest_block_num:4180657

Monitor

refiner already captures the most relevant performance metrics and execution times for various processes in the pipeline and exports all of it using Prometheus.

See the full document on how to setup Prometheus and Grafana for refiner metrics collection, monitoring, reporting and alerting

Build From Source

Installation Time: 35-40 mins depending on your machine and network.

Install git, go, asdf, erlang, elixir, direnv, go-ipfs.

  • Git is the source code version control manager across all our repositories.
  • Go is the programming language used to develop on go-ethereum, bsp-agent, erigon (EVM plugin), all of which are entirely written in go.
  • Asdf is a CLI tool that can manage multiple language runtime versions per-project.
  • Erlang is a programming language used to build massively scalable soft real-time systems with requirements of high availability.
  • Elixir is a programming language that runs on the Erlang VM, known for creating low-latency, distributed, high concurrency fault-tolerant systems.
  • IPFS as the InterPlanetary File System (IPFS) is a protocol, hypermedia and file sharing peer-to-peer network for storing and sharing data in a distributed file system.
  • Direnv is used for secret management and control since all the necessary sensitive parameters to the agent cannot be passed into a command line flag. Direnv allows for safe and easy management of secrets like Ethereum private keys for the operator accounts on the CQT network and redis instance access passwords etc. As these applications are exposed to the internet on http ports, it’s essential not to have the information be logged anywhere. To enable “direnv” on your machine, add these to your ~./bash_profile or ~./zshrc depending on which you use as your default shell after installing it using brew.

Linux x86_64 (Ubuntu 22.04 LTS) Install dependencies

Install dependencies for refiner.

sudo apt update
sudo apt install build-essential coreutils libssl-dev automake autoconf libncurses5-dev curl git wget direnv unzip

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.11.2
echo ". $HOME/.asdf/asdf.sh" >> ~/.bashrc
echo ". $HOME/.asdf/completions/asdf.bash" >> ~/.bashrc
source ~/.bashrc

Install required asdf version manager plugins for erlang, elixir and golang.

asdf plugin add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf plugin add golang https://github.com/kennyp/asdf-golang.git

Add to your shell for asdf (check for different shells - https://asdf-vm.com/guide/getting-started.html).

`echo -e "\n. \"$(brew --prefix asdf)/etc/bash_completion.d/asdf.bash\"" >> ~/.bash_profile`

Install Erlang, Elixir and Golang using the plugins.

asdf install erlang 25.0.3
asdf install elixir 1.14.3
asdf install golang 1.18.4

Set the versions.

asdf global erlang 25.0.3
asdf global elixir 1.14.3
asdf global golang 1.18.4

This will create a .tool-versions file in your home directory. ASDF will use these versions whenever a project doesn't specify versions of its own.

Enable direnv for shell bash/powershell(zsh).

# bash users - add the following line to your ~/.bashrc
eval "$(direnv hook bash)"

# zsh users - add the following line to your ~/.zshrc
eval "$(direnv hook zsh)"

After adding this line do not forget to source your bash/powershell config with the following, by running it in your terminal.

source ~/.zshrc
source ~/.bashrc

Install go-ipfs/kubo v0.13.0 and initialize go-ipfs.

wget https://dist.ipfs.tech/go-ipfs/v0.13.0/go-ipfs_v0.13.0_linux-amd64.tar.gz
tar -xzvf go-ipfs_v0.13.0_darwin-arm64.tar.gz
cd go-ipfs
bash install.sh
ipfs init

Note: To avoid permissions and netscan issues execute the following against ipfs binary home directory application

sudo chmod -R 700 ~/.ipfs
ipfs config profile apply server

Env Vars

Refer to the above existing environment var setup for refiner docker compose.

Note: When passing the private key into the env vars as above, please remove the 0x prefix so the private key env var has exactly 64 characters.

Source Run

The EVM-Server is a stateless EVM block execution tool. It's stateless because in Ethereum nodes like geth, it doesn't need to maintain database of blocks to do a block execution or re-execution. The evm-server service transforms Block Specimens to Block Results entirely on the input of underlying capture of Block Specimen data.

Clone covalenthq/erigon that contains the fork for this particular stateless execution function, checkout the covalent branch, build the evm-server binary and run it.

git checkout covalent
make evm

Updating git submodules
Building evm
go: downloading github.com/ledgerwatch/erigon-lib v0.0.0-20230328191829-416af23d9dcd...

Run the generated binary

./build/bin/evm t8n --server.mode --server.port 3002

INFO[04-17|11:03:07.516] Listening                                port=3002

The IPFS-Pinner is an interface to the storage layer of the CQT network using a decentralized storage network underneath. Primarily it's a custom IPFS (Inter Planetary File System) node with pinning service components for Web3.storage and Pinata; content archive manipulation etc. Additionally there's support for fetching using dweb.link. It is meant for uploading/fetching artifacts (like Block Specimens and uploading Block Results or any other processed/transformed data asset) of the Covalent Decentralized Network.

Clone covalenthq/ipfs-pinner and build the pinner server binary

git clone https://github.com/covalenthq/ipfs-pinner.git --depth 1
cd ipfs-pinner
make server-dbg

You should get the "w3 agent key" and "delegation proof file" from Covalent. After that, start the ipfs-pinner server.

./build/bin/server -w3-agent-key $W3_AGENT_KEY -w3-delegation-file $W3_DELEGATION_FILE

generating 2048-bit RSA keypair...done
peer identity: QmZQSGUEVKQuCmChKqTBGdavEKGheCQgKgo2rQpPiQp7F8

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "17 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 61440

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

2023/04/20 12:47:49 Listening...

Clone the refiner repo

git clone https://github.com/covalenthq/refiner.git
cd refiner
git checkout main

Get your BLOCK_RESULT_OPERATOR_PRIVATE_KEY that has DEV tokens for Moonbase Alpha and is already whitelisted as Block Result Producer operator. Set the following environment variables for the local refiner by creating an .envrc.local file

touch .envrc.local

Copy paste the environment variables into this file

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="BRP-OPERATOR-PK-WITHOUT-0x-PREFIX"
export NODE_ETHEREUM_MAINNET="<<ASK-ON-DISCORD>>"
export IPFS_PINNER_URL="http://127.0.0.1:3001"
export EVM_SERVER_URL="http://127.0.0.1:3002"

Call to load .envrc.local + .envrc files with the command below and observe the following output, make sure the environment variables are loaded into the shell.

direnv allow .

direnv: loading ~/Covalent/refiner/.envrc
direnv: loading ~/Covalent/refiner/.envrc.local
direnv: export +BLOCK_RESULT_OPERATOR_PRIVATE_KEY +ERIGON_NODE +IPFS_PINNER_URL +NODE_ETHEREUM_MAINNET

Once the env vars are passed into the .envrc.local file and loaded in the shell with direnv allow ., build the refiner application for the prod env i.e moonbeam mainnet or dev env for moonbase alpha as discussed before.

Get all the required dependencies and build the refiner app for the dev environment (this points to Moonbase Alpha contracts). Note: Windows is currently not supported.

mix local.hex --force && mix local.rebar --force && mix deps.get

For moonbeam mainnet.

MIX_ENV=prod mix release

For moonbase alpha.

MIX_ENV=dev mix release
.
..
....
evm-server: http://127.0.0.1:3002
ipfs-node: http://127.0.0.1:3001
* assembling refiner-0.2.12 on MIX_ENV=dev
* skipping runtime configuration (config/runtime.exs not found)
* skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
* skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)
Release created at _build/dev/rel/refiner
    # To start your system
    _build/dev/rel/refiner/bin/refiner start
Once the release is running:
    # To connect to it remotely
    _build/dev/rel/refiner/bin/refiner remote
    # To stop it gracefully (you may also send SIGINT/SIGTERM)
    _build/dev/rel/refiner/bin/refiner stop
To list all commands:
   _build/dev/rel/refiner/bin/refiner

Start the refiner application and execute the proof-chain block specimen listener call which should run the Refiner pipeline pulling Block Specimens from IPFS using the cids read from recent proof-chain finalized transactions, decoding them, and uploading and proofing Block Results while keeping a track of failed ones and continuing (soft real-time) in case of failure. The erlang concurrent fault tolerance allows each pipeline to be an independent worker that can fail (for any given Block Specimen) without crashing the entire pipeline application. Multiple pipeline worker children threads continue their work in the synchronous queue of Block Specimen AVRO binary files running the stateless EVM binary (evm-server) re-execution tool.

For moonbeam.

MIX_ENV=prod mix run --no-halt --eval 'Refiner.ProofChain.BlockSpecimenEventListener.start()';

For moonbase.

MIX_ENV=dev mix run --no-halt --eval 'Refiner.ProofChain.BlockSpecimenEventListener.start()';
..
...
refiner       | [info] found 1 bsps to process
ipfs-pinner  | 2023/06/29 20:28:30 unixfsApi.Get: getting the cid: bafybeiaxl44nbafdmydaojz7krve6lcggvtysk6r3jaotrdhib3wpdb3di
ipfs-pinner  | 2023/06/29 20:28:30 trying out https://w3s.link/ipfs/bafybeiaxl44nbafdmydaojz7krve6lcggvtysk6r3jaotrdhib3wpdb3di
ipfs-pinner  | 2023/06/29 20:28:31 got the content!
refiner       | [info] Counter for ipfs_metrics - [fetch: 1]
refiner       | [info] LastValue for ipfs_metrics - [fetch_last_exec_time: 0.001604]
refiner       | [info] Sum for ipfs_metrics - [fetch_total_exec_time: 0.001604]
refiner       | [info] Summary for ipfs_metrics  - {0.001604, 0.001604}
refiner       | [debug] reading schema `block-ethereum` from the file /app/priv/schemas/block-ethereum.avsc
refiner       | [info] Counter for bsp_metrics - [decode: 1]
refiner       | [info] LastValue for bsp_metrics - [decode_last_exec_time: 0.0]
refiner       | [info] Sum for bsp_metrics - [decode_total_exec_time: 0.0]
refiner       | [info] Summary for bsp_metrics  - {0.0, 0.0}
refiner       | [info] submitting 17586995 to evm http server...
evm-server   | [INFO] [06-29|20:28:31.859] input file at                            loc=/tmp/3082854681
evm-server   | [INFO] [06-29|20:28:31.862] output file at:                          loc=/tmp/1454174090
evm-server   | [INFO] [06-29|20:28:32.112] Wrote file                               file=/tmp/1454174090
refiner       | [info] writing block result into "/tmp/briefly-1688/briefly-576460747542186916-YRw0mRjfExGMk4M672"
refiner       | [info] Counter for bsp_metrics - [execute: 1]
refiner       | [info] LastValue for bsp_metrics - [execute_last_exec_time: 3.14e-4]
refiner       | [info] Sum for bsp_metrics - [execute_total_exec_time: 3.14e-4]
refiner       | [info] Summary for bsp_metrics  - {3.14e-4, 3.14e-4}
ipfs-pinner  | 2023/06/29 20:28:32 generated dag has root cid: bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
ipfs-pinner  | 2023/06/29 20:28:32 car file location: /tmp/249116437.car
[119B blob data]
ipfs-pinner  | 2023/06/29 20:28:34 Received /health request: source= 127.0.0.1:34980 status= OK
ipfs-pinner  | 2023/06/29 20:28:34 uploaded file has root cid: bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
refiner       | [info] Counter for ipfs_metrics - [pin: 1]
refiner       | [info] LastValue for ipfs_metrics - [pin_last_exec_time: 0.002728]
refiner       | [info] Sum for ipfs_metrics - [pin_total_exec_time: 0.002728]
refiner       | [info] Summary for ipfs_metrics  - {0.002728, 0.002728}
refiner       | [info] 17586995:48f1e992d1ac800baed282e12ef4f2200820061b5b8f01ca0a9ed9a7d6b5ddb3 has been successfully uploaded at ipfs://bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
refiner       | [info] 17586995:48f1e992d1ac800baed282e12ef4f2200820061b5b8f01ca0a9ed9a7d6b5ddb3 proof submitting
refiner       | [info] Counter for brp_metrics - [proof: 1]
refiner       | [info] LastValue for brp_metrics - [proof_last_exec_time: 3.6399999999999996e-4]
refiner       | [info] Sum for brp_metrics - [proof_total_exec_time: 3.6399999999999996e-4]
refiner       | [info] Summary for brp_metrics  - {3.6399999999999996e-4, 3.6399999999999996e-4}
refiner       | [info] 17586995 txid is 0xd8a8ea410240bb0324433bc26fdc79d496ad0c8bfd18b60314a05e3a0de4fb06
refiner       | [info] Counter for brp_metrics - [upload_success: 1]
refiner       | [info] LastValue for brp_metrics - [upload_success_last_exec_time: 0.0031149999999999997]
refiner       | [info] Sum for brp_metrics - [upload_success_total_exec_time: 0.0031149999999999997]
refiner       | [info] Summary for brp_metrics  - {0.0031149999999999997, 0.0031149999999999997}
refiner       | [info] Counter for refiner_metrics - [pipeline_success: 1]
refiner       | [info] LastValue for refiner_metrics - [pipeline_success_last_exec_time: 0.0052]
refiner       | [info] Sum for refiner_metrics - [pipeline_success_total_exec_time: 0.0052]
refiner       | [info] Summary for refiner_metrics  - {0.0052, 0.0052}

Check logs for any errors in the pipeline process and note the performance metrics in line with execution. Checkout the documentation on what is being measured and why here.

tail -f logs/log.log
..
...
refiner       | [info] Counter for refiner_metrics - [pipeline_success: 1]
refiner       | [info] LastValue for refiner_metrics - [pipeline_success_last_exec_time: 0.0052]
refiner       | [info] Sum for refiner_metrics - [pipeline_success_total_exec_time: 0.0052]
refiner       | [info] Summary for refiner_metrics  - {0.0052, 0.0052}

Alternatively - check proof-chain logs for correct block result proof submissions and transactions made by your block result producer.

For moonbeam.

For moonbase.

Note:For any issues associated with building and re-compiling execute the following commands, that cleans, downloads and re-compiles the dependencies for refiner.

rm -rf _build deps && mix clean && mix deps.get && mix deps.compile

If you got everything working so far. Congratulations! You're now a Refiner operator on the CQT Network. Set up Grafana monitoring and alerting from links in the additional resources section.

Troubleshooting

To avoid permission errors with ~/.ipfs folder execute the following in your home directory.

sudo chmod -R 770 ~/.ipfs

To avoid netscan issue execute the following against ipfs binary application.

sudo chmod -R 700 ~/.ipfs
ipfs config profile apply server

To avoid issues with ipfs-pinner v0.1.9, a small repo migration for the local ~/.ipfs directory may be needed from - https://github.com/ipfs/fs-repo-migrations/blob/master/run.md.

For linux systems follow the below steps.

wget https://dist.ipfs.tech/fs-repo-migrations/v2.0.2/fs-repo-migrations_v2.0.2_linux-amd64.tar.gz
tar -xvf fs-repo-migrations_v2.0.2_linux-amd64.tar.gz
cd fs-repo-migrations
chmod +x ./fs-repo-migrations
./fs-repo-migrations

Bugs Reporting Contributions

Please follow the guide in docs contribution guidelines for bug reporting and contributions.

Scripts

In order to run the Refiner docker compose services as a service unit. The example service unit file in docs should suffice. After adding the env vars in their respective fields in the service unit file, enable the service and start it.

sudo systemctl enable refiner-compose.service
sudo systemctl start refiner-compose.service

Note:: To run docker compose as a non-root user for the above shown service unit, you need to create a docker group (if it doesn't exist) and add the user “blockchain” to the docker group.

sudo groupadd docker
sudo usermod -aG docker blockchain
sudo su - blockchain
docker run hello-world

Appendix

Run With Existing IPFS-Pinner Service

On a system where an ipfs-pinner instance is already running, use this modified .envrc.local and docker-compose-mbase.yml

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="BRP-OPERATOR-PK-WITHOUT-0x-PREFIX"
export NODE_ETHEREUM_MAINNET="<<ASK-ON-DISCORD>>"
export IPFS_PINNER_URL="http://host.docker.internal:3001"
export EVM_SERVER_URL="http://evm-server:3002"

Note: When passing the private key into the env vars as above please remove the 0x prefix so the private key env var has exactly 64 characters.

version: '3'
# runs the entire refiner pipeline with all supporting services (including refiner) in docker
# set .env such that all services in docker are talking to each other only; ipfs-pinnern is assumed
# to be hosted on the host machine. It's accessed through http://host.docker.internal:3001/ url from
# inside refiner docker container.
services:
  evm-server:
    image: "us-docker.pkg.dev/covalent-project/network/evm-server:stable"
    container_name: evm-server
    restart: always
    labels:
      "autoheal": "true"
    expose:
      - "3002:3002"
    networks:
      - cqt-net
    ports:
      - "3002:3002"

  refiner:
    image: "us-docker.pkg.dev/covalent-project/network/refiner:stable"
    container_name: refiner
    links:
      - "evm-server:evm-server"
    restart: always
    depends_on:
      evm-server:
        condition: service_healthy
    entrypoint: >
      /bin/bash -l -c "
        echo "moonbeam-node:" $NODE_ETHEREUM_MAINNET;
        echo "evm-server:" $EVM_SERVER_URL;
        echo "ipfs-pinner:" $IPFS_PINNER;
        cd /app;
        MIX_ENV=prod mix release --overwrite;
        MIX_ENV=prod mix run --no-halt --eval 'Refiner.ProofChain.BlockSpecimenEventListener.start()';"
    environment:
      - NODE_ETHEREUM_MAINNET=${NODE_ETHEREUM_MAINNET}
      - BLOCK_RESULT_OPERATOR_PRIVATE_KEY=${BLOCK_RESULT_OPERATOR_PRIVATE_KEY}
      - EVM_SERVER_URL=${EVM_SERVER_URL}
      - IPFS_PINNER_URL=${IPFS_PINNER_URL}
    networks:
      - cqt-net
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "9568:9568"

  autoheal:
    image: willfarrell/autoheal
    container_name: autoheal
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
    environment:
      - AUTOHEAL_INTERVAL=10
      - CURL_TIMEOUT=30

networks:
  cqt-net:

and start the refiner and evm-server services:

$ docker compose -f "docker-compose-mbeam.yml" up --remove-orphans

[+] Running 3/3
 ⠿ Network refiner_cqt-net  Created                                   0.0s
 ⠿ Container evm-server     Started                                   0.7s
 ⠿ Container refiner         Started                                   1.5s

About

Concurrent block specimen data processing and transformation framework for the Covalent Network.

www.covalenthq.com/network/

Topics

actors ethereum tasks supervisor refiner elixir-lang dataflow-programming covalent-network decentralized-etl

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

9 stars

Watchers

5 watching

Forks

13 forks
Report repository

Releases 31

v0.5.4 Latest
Aug 6, 2024
+ 30 releases

Packages

No packages published

Contributors 6

Languages