Skip to content

Latest commit

 

History

History
240 lines (179 loc) · 10.1 KB

README.md

File metadata and controls

240 lines (179 loc) · 10.1 KB
Raw

Anton

This project is an open-source tool that extracts and organizes data from the TON blockchain, efficiently storing it in PostgreSQL and ClickHouse databases.

Overview

Before you start, take a look at the official docs.

Consider an arbitrary contract. It has a state that is updated with any transaction on the contract's account. Each state has the contract code and data. The contract data can be complex, but developers typically provide get-methods in the contract, which can be executed to retrieve the necessary data. The TON has standard contracts (such as TEP-62, TEP-74), and they have predefined get-method names. Therefore, you can attempt to match accounts found in the network to these standards by checking the presence of the get-methods. Contract standards also specify TL-B constructor tags (or operation ids) for each acceptable message to contract, defined as the first 32 bits of the parsed message payload cell. So you if you know standard of a given contract, you can determine the type of message to it (for example, NFT item transfer) by parsing the first 32 bits of message body.

Anton allows you to define the contract interface in just one JSON schema. Format of every schema is described in detail in abi/README.md. Every schema comprises contract get-methods, as well as incoming and outgoing message schemas for the contract. Once contract interfaces are defined and stored in the database, Anton begins scanning new blocks on the network. The tool stores every account state, transaction, and message in the database. For get-methods without arguments in the contract interface, Anton emulates these methods and saves the returned values to the database. When a message is sent to a known contract interface, Anton attempts to match the message to a known schema by comparing the parsed operation ID. If the message is successfully parsed using the identified schema, Anton also stores the parsed data.

To explore contract interfaces known to this project, visit the abi/known directory. This will provide you with an understanding of the various contract interfaces already supported and serve as examples for adding your own.

Currently, Anton offers a REST API for retrieving filtered and aggregated data from the databases. To see example queries, refer to the API.md file.

To explore how Anton stores data, visit the migrations' directory.

Project structure

Folder Description
abi get-methods and tlb cell parsing
abi/known contract interfaces known to this project
api/http JSON API Swagger documentation
docs only API query examples for now
config custom postgresql configuration
migrations database migrations
internal database repositories and services implementation

Internal directory structure

Folder Description
core contains project domain
core/rndm generates random domain structures
core/filter describes filters
core/aggregate describes aggregation metrics
core/repository implements database repositories with filters and aggregation
app contains all services interfaces and their configs
app/parser service determines contract interfaces, parse contract data and message payloads
app/fetcher service concurrently fetches data from blockchain
app/indexer service scans blocks and save parsed data to databases
app/query service aggregates database repositories
api/http implements the REST API

Starting it up

Cloning repository

git clone https://github.com/tonindexer/anton
cd anton

Running tests

Run tests on abi package:

go test -p 1 $(go list ./... | grep /abi) -covermode=count

Run repositories tests:

# start databases up
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d postgres clickhouse

go test -p 1 $(go list ./... | grep /internal/core) -covermode=count

Running linter

Firstly, install golangci-lint.

golangci-lint run

Configuration

Installation requires some environment variables.

cp .env.example .env
nano .env
Name Description Default Example
DB_NAME Database name idx
DB_USERNAME Database username user
DB_PASSWORD Database password pass
DB_CH_URL Clickhouse URL to connect to clickhouse://clickhouse:9000/db_name?sslmode=disable
DB_PG_URL PostgreSQL URL to connect to postgres://username:password@postgres:5432/db_name?sslmode=disable
FROM_BLOCK Master chain seq_no to start from 1 23532000
WORKERS Number of indexer workers 4 8
LITESERVERS Lite servers to connect to 135.181.177.59:53312 aF91CuUHuuOv9rm2W5+O/4h38M3sRm40DtSdRxQhmtQ=
DEBUG_LOGS Debug logs enabled false true

Building

# building it locally
go build -o anton .

# build local docker container via docker cli
docker build -t anton:latest .
# or via compose
docker compose -f docker-compose.yml -f docker-compose.dev.yml build

# pull public images
docker compose pull

Running

We have several options for compose run via override files:

  • base (docker-compose.yml) - allows to run services with near default configuration;
  • dev (docker-compose.dev.yml) - allows to rebuild Anton image locally and exposes databases ports;
  • prod (docker-compose.prod.yml) - allows to configure and backup databases, requires at least 128GB RAM.

You can combine it by your own. Also, there are optional profiles:

  • migrate - runs optional migrations service.

Take a look at the following run examples:

# run base compose
docker compose up -d

# run dev compose (build docker image locally)
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d

# run prod compose
# WARNING: requires at least 128GB RAM
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

To run Anton, you need at least one defined contract interface. There are some known interfaces in the abi/known directory. You can add them through this command:

docker compose exec web sh -c "anton contract /var/anton/known/*.json"

Schema migration

# run migrations service on running compose
docker compose run migrations

Reading logs

docker compose logs -f

Taking a backup

# starting up databases and API service
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d postgres clickhouse web

# stop indexer
docker compose stop indexer

# create backup directories
mkdir backups backups/pg backups/ch

# backing up postgres
docker compose exec postgres pg_dump -U user db_name | gzip > backups/pg/1.pg.backup.gz

# backing up clickhouse (available only with docker-compose.prod.yml)
## connect to the clickhouse
docker compose exec clickhouse clickhouse-client
## execute backup command
# :) BACKUP DATABASE default TO File('/backups/1/');

# execute migrations through API service
docker compose exec web anton migrate up

# start up indexer
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d indexer

Using

Show archive nodes from global config

docker run tonindexer/anton archive [--testnet]

Insert contract interface

# add from stdin
cat abi/known/tep81_dns.json | docker compose exec -T web anton contract --stdin
# add from file
docker compose exec web anton contract "/var/anton/known/tep81_dns.json"

Delete contract interface

docker compose exec web anton contract delete "dns_nft_item"

Add address label

docker compose exec web anton label "EQDj5AA8mQvM5wJEQsFFFof79y3ZsuX6wowktWQFhz_Anton" "anton.tools"

# known tonscan labels
docker compose exec web anton label --tonscan