-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add bootstrap guide for Mumak Postgres data using Oura (#39)
- Loading branch information
Showing
1 changed file
with
148 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,148 @@ | ||
# Bootstrapping Mumak Postgres Data Using Oura | ||
|
||
## Introduction | ||
|
||
To bootstrap **Mumak** data, we will be utilizing a tool called [Oura](https://github.com/txpipe/oura) by TxPipe. | ||
|
||
> **Oura** is a lightweight, Rust-native pipeline designed to observe the Cardano blockchain in real-time. It connects to a Cardano node using the Ouroboros mini-protocols, filters events based on customizable patterns, and forwards them to pluggable "sinks" for further processing. Oura is resource-efficient and can run alongside a Cardano node even in constrained environments. It supports various use cases, such as watching live transactions, integrating with data-streaming systems, triggering custom actions upon specific blockchain events, and serving as a library for building custom data pipelines. | ||
This guide will walk you through the steps required to set up Oura for indexing Cardano blockchain data into a PostgreSQL database, which can then be utilized by Mumak. | ||
|
||
## Prerequisites | ||
|
||
- **Running Cardano Node**: Ensure you have a Cardano node running and accessible. | ||
- **PostgreSQL Database**: A running PostgreSQL instance where the data will be stored. | ||
- **Oura Installed**: Oura should be installed on your system. | ||
|
||
## Setup Guide | ||
|
||
### 1. Install Oura | ||
|
||
Please refer to the [Oura Installation Guide](https://txpipe.github.io/oura) for detailed instructions on how to install Oura. | ||
|
||
### 2. Configure Oura | ||
|
||
Oura needs to be configured to connect to your Cardano node and PostgreSQL database. Below are sample configurations for indexing transaction and block data. | ||
|
||
**Note**: Replace placeholders like `${network}` with your actual network (e.g., `mainnet`, `preview`, `preprod`) and ensure paths and connection strings match your environment. | ||
|
||
#### Transaction Data Indexing Configuration | ||
|
||
Create a configuration file named `oura_tx.toml` with the following content: | ||
|
||
```toml | ||
[source] | ||
type = "N2C" | ||
socket_path = "/ipc/node.socket" | ||
|
||
[intersect] | ||
type = "Origin" | ||
|
||
[chain] | ||
type = "${network}" | ||
|
||
[[filters]] | ||
type = "SplitBlock" | ||
|
||
[sink] | ||
type = "SqlDb" | ||
connection = "postgres://postgres:example@localhost:5432/postgres" | ||
apply_template = "INSERT INTO txs (slot, cbor) VALUES ('{{ point.slot }}', decode('{{ record.hex }}', 'hex'));" | ||
undo_template = "DELETE FROM txs WHERE slot = {{ point.slot }}" | ||
reset_template = "DELETE FROM txs WHERE slot > {{#if point.slot}}{{ point.slot }}{{/if}};" | ||
|
||
[cursor] | ||
type = "File" | ||
path = "/data/cursor_tx" | ||
|
||
[policy] | ||
missing_data = "Skip" | ||
cbor_errors = "Warn" | ||
|
||
[metrics] | ||
address = "0.0.0.0:9186" | ||
endpoint = "/metrics" | ||
``` | ||
|
||
#### Block Data Indexing Configuration | ||
|
||
Create another configuration file named `oura_block.toml` with the following content: | ||
|
||
```toml | ||
[source] | ||
type = "N2C" | ||
socket_path = "/ipc/node.socket" | ||
|
||
[intersect] | ||
type = "Origin" | ||
|
||
[chain] | ||
type = "${network}" | ||
|
||
[sink] | ||
type = "SqlDb" | ||
connection = "postgres://postgres:example@localhost:5432/postgres" | ||
apply_template = "INSERT INTO blocks (slot, cbor) VALUES ('{{ point.slot }}', decode('{{ record.hex }}', 'hex'));" | ||
undo_template = "DELETE FROM blocks WHERE slot = {{ point.slot }}" | ||
reset_template = "DELETE FROM blocks WHERE slot > {{#if point.slot}}{{ point.slot }}{{/if}};" | ||
|
||
[cursor] | ||
type = "File" | ||
path = "/data/cursor_block" | ||
|
||
[policy] | ||
missing_data = "Skip" | ||
cbor_errors = "Warn" | ||
|
||
[metrics] | ||
address = "0.0.0.0:9186" | ||
endpoint = "/metrics" | ||
``` | ||
|
||
### 3. Start Oura | ||
|
||
Once Oura is configured, start the Oura instances to begin indexing data into your PostgreSQL database. | ||
|
||
**For transaction data**, run: | ||
|
||
<code> | ||
oura daemon --config oura_tx.toml | ||
</code> | ||
|
||
**For block data**, run: | ||
|
||
<code> | ||
oura daemon --config oura_block.toml | ||
</code> | ||
|
||
Oura will start processing the Cardano blockchain data and store it in your PostgreSQL database. | ||
|
||
--- | ||
|
||
**Note**: Ensure that the PostgreSQL database has the necessary tables (`txs` and `blocks`) created with appropriate schemas to store the data inserted by Oura. | ||
|
||
```sql | ||
-- Table for storing CBOR blocks | ||
CREATE TABLE blocks ( | ||
slot INTEGER NOT NULL, | ||
cbor BYTEA | ||
); | ||
|
||
-- Index for the blocks table | ||
CREATE INDEX idx_blocks_slot ON blocks(slot); | ||
|
||
-- Table for storing CBOR transactions | ||
CREATE TABLE txs ( | ||
slot INTEGER NOT NULL, | ||
cbor BYTEA | ||
); | ||
|
||
-- Index for the txs table | ||
CREATE INDEX idx_txs_slot ON txs(slot); | ||
``` | ||
|
||
## Conclusion | ||
|
||
By following these steps, you have set up Oura to bootstrap Mumak data by indexing Cardano blockchain data into your PostgreSQL database. This setup enables Mumak to utilize the blockchain data for its operations. | ||
|
||
For further customization and advanced configurations, refer to the [Oura Documentation](https://txpipe.github.io/oura). |