README.md

Monitoring

This document summarizes the various details regarding to monitoring Amaru. As a pre-requisite, it's important to note that Amaru leverages OpenTelemetry to emit traces & metrics. A compatible observability backend such as Jaeger, Grafana Tempo and/or Prometheus is therefore needed to collect and visualise telemetry.

We provide example configurations using different compositions of tools:

• Jaeger + Prometheus (simple)
• Grafana + Tempo + Prometheus (more advanced)

Filtering traces

Any event (trace, span or metric) can be filtered by target and severity using two environment variables:

• AMARU_LOG: for any event emitted by the OpenTelemetry layer;
• AMARU_DEV_LOG: for any event emitted to stdout;

Tip

Both environment variable are optional.

• When omitted, AMARU_LOG defaults to all targets above the info level;
• When omitted, AMARU_DEV_LOG defaults to AMARU_LOG.

By target

A target is a ::-separated path of identifiers such as amaru::ledger::state. One can filter by providing either a full target, or a sub-path prefix. For example, the target amaru::ledger will match the following:

• amaru::ledger::state
• amaru::ledger::state::forward
• amaru::ledger::store

But it will not match any of the following:

• amaru::sync
• amaru::consensus

Refer to the tables below for the list of available targets.

By severity

It is also possible to filter events by severity: error, warn, info, debug, trace, off. Severity can be specified either globally (in which case it applies to all events) or for a specific target by specifying the severity after the target using =. For example, amaru::ledger::state=error will filter out any events below the error severity for the amaru::ledger::state target.

Refer to the tables below for an overview of the various severities.

Combining filters

Filters can be provided as a sequence of ,-separated values. Right-most filters take precedence. A usual pattern is to first define a global filter and override it with specific target. For example, error,amaru::ledger::store=debug will exclude any event below the error severity except those targetting amaru::ledger::store which will show up to the debug severity.

Traces & spans

target: amaru::ledger

Traces

ø

Spans

name	severity	description
forward	info	Wraps the processing of a block forward
backward	info	Wraps the processing of a rollback

span: `forward`

field	description
header.height	Absolute block height
header.slot	Absolute slot number
header.hash	Block header hash
stable.epoch	Current epoch of the most recent stable point
tip.epoch	Current epoch of the most recent volatile point
tip.relative_slot	Relative slot within the epoch of the most recent volatile point

span: `backward`

field	description
point.slot	Absolute slot number of the target rollback point
point.hash	Block header hash of the target rollback point

target: amaru::ledger::state

Traces

name	severity	description
volatile.warming_up	info	Emitted on start-up while the ledger volatile database is filling up

trace: `volatile.warming_up`

field	description
size	Current size of the volatile db, up to k.

Spans

name	severity	description
block.body.validate	info	Wraps the block body validation & processing
snapshot	info	Wraps the creation of a new epoch-boundary snapshot
save	info	Wraps the persistence on-disk of the next now-stable ledger delta
tick.pool	info	Wraps the update of pool parameters and enactment of retirements at an epoch-boundary
apply.transaction	info	Wraps the validation & processing of a single transaction

span: `block.body.validate`

field	description
block.transactions.total	Total number of transactions in the block
block.transactions.failed	Total number of failed transactions in the block
block.transactions.success	Total number of successful transactions in the block

span: `snapshot`

field	description
epoch	The epoch being snapshot, typically the immediately previous epoch

span: `apply.transaction`

field	description
transaction.id	The transaction identifier/hash
transaction.certificates	The number of certificates within the transaction
transaction.inputs	The number of (collateral) inputs within the transaction
transaction.outputs	The number of (collateral) outputs within the transaction

target: amaru::ledger::state::apply::transaction

Spans

ø

Traces

name	severity	description
certificate.stake.registration	debug	A new stake credential registration was processed
certificate.stake.delegation	debug	A new stake delegation was processed
certificate.stake.deregistration	debug	A new stake credential de-registration was processed
certificate.pool.registration	debug	A new stake pool registration was processed
certificate.pool.retirement	debug	A new stake pool retirement was processed

trace: `certificate.stake.registration`

field	description
credential	Stake credential being registered

trace: `certificate.stake.delegation`

field	description
credential	Stake credential being delegated
pool	Stake pool delegate

trace: `certificate.stake.deregistration`

field	description
credential	Stake credential being deregistered

trace: `certificate.stake.registration`

field	description
pool	Stake pool identifier
params	New or initial stake pool parameters

trace: `certificate.stake.retirement`

field	description
pool	Stake pool identifier
epoch	Retirement epoch

target: amaru::ledger::store

Spans

ø

Traces

name	severity	description
new.unexpected_file	warn	An unexpected file is present in the database folder, which may cause an issue later
save.point_already_known	info	Skipping saving ledger delta because it has already happened
new.found_snapshot	debug	A previous database snapshot has been found in the database folder

trace: `new.unexpected_file`

field	description
filename	The relative path to the unexpected file or directory

trace: `save.point_already_known`

field	description
point	The now-stable point already known and persisted

trace: `new.found_snapshot`

field	description
epoch	The snapshot number / epoch it is relevant to

target: amaru::ledger::store::accounts

Spans

ø

Traces

name	severity	description
add.register_no_deposit	error	Attempting (and failing) to register a brand-new account without deposit; signaling a bug in the ledger

trace: `add.register_no_deposit`

field	description
credential	Stake credential of the faulty account

target: amaru::ledger::store::pools

Traces

name	severity	description
tick.retiring	debug	Now-retiring (i.e. removing from stable storage) a pool
tick.updating	debug	Updating a pool's parameters

trace: `tick.retiring`

field	description
pool	The identifier of the now-retired stake pool

trace: `tick.updating`

field	description
pool	The identifier of the updated stake pool
new_params	The new/now-effective parameters for the pool

Spans

ø

Metrics

Coming soon.

Note

The plan so far is to maximise compatibility with the existing Haskell node Prometheus metrics such that tools like gLiveView and nview keep working out-of-the-box.

We are planning, however, to add more metrics to Amaru.