TLDR;
Deephaven Community doesn't provide a built-in persistent storage layer (as of Spring 2024), so let's use
Clickhouse to create one.
This repo shows how to
- leverage Cryptofeed to subscribe to 24/7 real-time Crypto market data
- push the data onto Kafka to create live streams
- persist the streams via the Clickhouse Kafka Table Engine
- access real-time streams and historical data from the DH UI, plus some magic to stitch them together
- build a Deephaven app mode around it
Everything should "just work", simply run this and wait until all 5 containers start up:
docker compose build --no-cache
docker compose up -d
- Deephaven UI is running at http://localhost:10000/ide/
- ClickHouse Play is running at http://localhost:8123/play
- CLICKHOUSE_USER: default
- CLICKHOUSE_PASSWORD: password
- Redpanda Console is running at http://localhost:8080/overview
- Data is stored locally under the
/data/[clickhouse|deephaven|redpanda]folders which are mounted into the docker images
├── data <- Project data
│ ├── clickhouse <- volume mount for the 'clickhouse' container
│ ├── deephaven <- volume mount for the 'deephaven' container
│ │ ├── layouts <- contains .json file for Deephaven app layout
│ │ └── notebooks <- Deephaven File Explorer, all DH python scripts are stored here
│ └── redpanda <- volume mount for the 'redpanda' container
│
├── docker <- anything needed for Docker builds
│ ├── clickhouse <- build files for ClickHouse
│ ├── cryptofeed <- build files for Cryptofeed
│ ├── redpanda <- build files for Redpanda (e.g. Kafka retention of 1min)
│ └── deephaven <- build files for Deephaven
│
├── .gitignore <- List of files ignored by git
├── docker-compose.yaml <- docker-compose file to start up everything
├── requirements.txt <- File for installing python dependencies
├── setup.py <- File for installing project as a package
└── README.md
-
ClickHouse + Kafka: https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka
- Kafka to ClickHouse
- ClickHouse to Kafka [doable but we don't use it yet]
-
ClickHouse + Redpanda: https://redpanda.com/blog/real-time-olap-database-clickhouse-redpanda
-
Older version using QuestDB: https://github.com/kzk2000/deephaven-questdb