Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Architecture Overhaul, and Performance Improvements #55

Merged
merged 96 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
cf4a9a8
feat: do not proceed with init script if snapshotter is not active
anomit Sep 1, 2023
ee1e4ac
feat: snapshotter active status check at runtime
anomit Sep 1, 2023
813412b
chore: update example config for full pairs
xadahiya Sep 2, 2023
275f234
build image for testnet_pretask
SwaroopH Sep 2, 2023
ca73f3e
trying to use larger runner
SwaroopH Sep 2, 2023
2f0a60d
chore: increase RPC limits for pretest
xadahiya Sep 2, 2023
c98edda
Merge remote-tracking branch 'refs/remotes/origin/testnet_pretask' in…
xadahiya Sep 2, 2023
e432437
chore: skipping calculate from scratch from pretask
xadahiya Sep 2, 2023
114097c
chore: update events to work with testnet pretask protocol state
anomit Sep 2, 2023
823b916
chore: ping reporting service every 30 secs
anomit Sep 2, 2023
afd500a
chore: Update protocol state contract ABI
xadahiya Sep 2, 2023
d86d681
chore: add web3 middleware caching strategy
anomit Sep 2, 2023
4a69cba
chore: update function name for get all snapshotters
xadahiya Sep 4, 2023
2c4b1da
feat+chore: won't process further epochs is snapshotters is disabled,…
xadahiya Sep 4, 2023
82af846
Merge pull request #45 from PowerLoom/feat/testnet_pretask_snapshotte…
xadahiya Sep 4, 2023
42777ef
chore: made PROTOCOL_STATE_CONTRACT and PROST_RPC URL mandatory, remo…
xadahiya Sep 4, 2023
272fc4b
chore: remove middleware (was breaking due to version differences)
xadahiya Sep 4, 2023
1a79100
chore: update config files to snapshot only chosen 5 pairs
xadahiya Sep 8, 2023
5b46a58
chore: add testnet_5_pairs branch to build workflow
xadahiya Sep 8, 2023
2f0f4f0
doc: add sections on epoch state transition and related internal APIs
anomit Sep 8, 2023
ce19e4d
Merge branch 'testnet_pretask' into testnet_5_pairs
xadahiya Sep 8, 2023
5bc5fed
fix: checksum instance ID EVM address for ID check on protocol
anomit Sep 8, 2023
1ce6cd7
chore: replace CAW pair with Matic
xadahiya Sep 9, 2023
4b61df1
Merge remote-tracking branch 'refs/remotes/origin/testnet_5_pairs' in…
xadahiya Sep 9, 2023
74863a9
fix: project config
xadahiya Sep 11, 2023
99f607e
hotfix: parsing of cached epoch processing dataset
anomit Sep 14, 2023
c1b4f09
chore: script to report on processhub core workers' running status
anomit Sep 15, 2023
b577df8
hotfix: parsing of cached epoch processing dataset
anomit Sep 14, 2023
6cdfaf4
chore: script to report on processhub core workers' running status
anomit Sep 15, 2023
db0653b
chore: mv worker reporting script to required dir
anomit Sep 18, 2023
25e06ca
chore: mv worker reporting script to required dir
anomit Sep 18, 2023
8b759a7
fix: remove data source checksum when project is added from ProjectUp…
xadahiya Sep 20, 2023
f344e2f
fix: remove data source checksum when project is added from ProjectUp…
xadahiya Sep 20, 2023
61894ca
Merge remote-tracking branch 'refs/remotes/origin/testnet_5_pairs' in…
xadahiya Sep 20, 2023
81417f0
feat(WIP): added periodic collection of health of processing status f…
anomit Sep 20, 2023
1b74b56
feat: clean respawn of all children worker processes
anomit Sep 30, 2023
da5093a
chore: fix log message
anomit Sep 30, 2023
6f52005
chore+fix: Update CLI, add processReport, remove outdated pidStatus, …
xadahiya Oct 1, 2023
e641173
chore: remove bulk aggregation for testnet lite moide participants
anomit Oct 2, 2023
26bbc2c
Merge branch 'testnet_5_pairs' into feat/health-recovery
anomit Oct 2, 2023
984ae27
chore: revert redundant script
anomit Oct 2, 2023
1eeb2c8
chore: adding back pretask network specific config
anomit Oct 2, 2023
c400eda
Merge pull request #50 from PowerLoom/feat/health-recovery-pretask
xadahiya Oct 2, 2023
416f340
fix function name
getjiggy Oct 17, 2023
40d95b6
latest features from testnet nodes merged
anomit Oct 29, 2023
5f79e42
feat: IPFS and web3 upload moved within pooler
anomit Oct 29, 2023
f7a500e
chore+fix: proper initialization of IPFS clients
anomit Nov 2, 2023
2fb83f3
fix: add web3 storage API token in requests
anomit Nov 2, 2023
f1b26c3
fix: upload snapshot contents to web3 storage with auth
anomit Nov 3, 2023
13cd60c
chore: upload standardized JSON dumps to IPFS and web3 storage
anomit Nov 3, 2023
dcebcf0
chore: clean up upload helpers
anomit Nov 3, 2023
c22f3f5
fix: ensure respawn attempts are not made every epoch. respawn intern…
anomit Nov 3, 2023
bd80331
fix: cleaner health check conditions. minor fixes in error tracing an…
anomit Nov 3, 2023
74c9778
chore: further trimming of large exception trace logs
anomit Nov 3, 2023
a62c17c
chore: cleanup excessive exception traces in nested rpc calls
anomit Nov 3, 2023
7633ce6
fix: IPFS upload serialization
xadahiya Nov 3, 2023
13a17cb
chore: Address review comments (only uploading to web3storage if conf…
xadahiya Nov 7, 2023
86529eb
Merge branch 'feat/payload-commit-snapshotter-integration' into fix-s…
xadahiya Nov 7, 2023
0c6d2ab
Merge pull request #51 from PowerLoom/fix-spelling
xadahiya Nov 7, 2023
337c561
Merge pull request #53 from PowerLoom/feat/payload-commit-snapshotter…
xadahiya Nov 7, 2023
d886d5a
chore: update deps, add remote pinning config defaults
xadahiya Nov 8, 2023
d6762e8
chore: using settings model from ipfs_client library
xadahiya Nov 8, 2023
28e135e
chore: port improvements and todos from phase 2, minor cleanup
xadahiya Nov 10, 2023
15701f3
chore: remove project_id_generator script, improve snapshot_worker, u…
xadahiya Nov 13, 2023
18093e7
fix: snapshot_worker process_task cleanup
xadahiya Nov 14, 2023
06923c4
chore: unify processor_distributor, cleanup extra files
xadahiya Nov 16, 2023
4226ca5
chore: cleanup and initial docstring update
xadahiya Nov 17, 2023
7839a5c
chore: update snapshotter_autofill to include web3_storage token and …
xadahiya Nov 18, 2023
869504a
fix: settings path in snapshotter_autofill for web3storage token
xadahiya Nov 18, 2023
2a3cae8
chore: change prefetch count to 1 in workers, update deps
xadahiya Nov 20, 2023
5842531
Merge branch 'fix/health-recovery-respawn' into feat/single_snapshotter
xadahiya Nov 21, 2023
f970ee2
feat: cleaner epoch processing health check decoupled from process hu…
anomit Nov 24, 2023
4950fea
fix: import formatting
xadahiya Nov 24, 2023
5f5573f
Merge branch 'feat/health_recovery_port' into feat/single_snapshotter
xadahiya Nov 24, 2023
e520170
chore: cleanup imports, update dependencies
xadahiya Nov 24, 2023
49d207c
Fix Redis connection issue and handle reporter
anomit Nov 24, 2023
d5dc8b4
fix: init fixes in generic worker
xadahiya Nov 24, 2023
48328d0
fix: redis keys require integers as expire value
anomit Nov 24, 2023
cebbd9d
fix: bug fixes in processor distributor and aggregation worker
xadahiya Nov 24, 2023
087c609
chore: cleanup aiohttp library from rpc, fix duplicate logs
xadahiya Nov 24, 2023
34b3b2b
Merge branch 'chore/docs_update' into feat/single_snapshotter
xadahiya Nov 25, 2023
71b15cf
chore: init submodules
xadahiya Nov 25, 2023
a0d1c45
chore: enable submodules in docker build, add zkevm_quests branch for…
xadahiya Nov 25, 2023
06c98ed
chore: improve health check and respawning
xadahiya Nov 25, 2023
7a5ab4c
chore: improve computes naming
xadahiya Nov 25, 2023
2df1b9a
chore: udpate compute modules
xadahiya Nov 25, 2023
4a43a68
chore: update configs, add branch to docker build workflow
xadahiya Nov 25, 2023
36afc38
chore: Initial README updates
xadahiya Nov 25, 2023
6110d65
chore: update overall architecture diagram
xadahiya Nov 25, 2023
d8b7be3
chore: update epoch detection and snapshot process unhealthy interval…
xadahiya Nov 25, 2023
ac8eaf5
Merge branch 'zkevm_quests' into eth_uniswapv2
xadahiya Nov 25, 2023
a2d38c2
chore: udpate README.md
xadahiya Nov 26, 2023
b9df26b
chore: addressed review comments (add submodule references and link t…
xadahiya Nov 27, 2023
33f838a
chore: add missing README sections
xadahiya Nov 28, 2023
6346108
chore: cleanup Data Signalling section (removed wallet-boost references)
xadahiya Nov 29, 2023
db969eb
fix: typo AggreagateTradeVolumeProcessor -> AggregateTradeVolumeProce…
xadahiya Nov 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
release:
types: ['published']
push:
branches: [ "dockerify", "testnet_pretask" ]
branches: [ "dockerify", "testnet_pretask", "testnet_5_pairs", "zkevm_quests", "eth_uniswapv2"]
# Publish semver tags as releases.
tags: [ 'v*.*.*' ]
pull_request:
Expand Down Expand Up @@ -37,6 +37,8 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
with:
submodules: recursive

# Install the cosign tool except on PR
# https://github.com/sigstore/cosign-installer
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ uniswapTokenData
**/*aggregator.json
config/*settings.json
**/*.backup
.vscode/*
6 changes: 6 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[submodule "config"]
path = config
url = https://github.com/PowerLoom/snapshotter-configs/
[submodule "snapshotter/modules/computes"]
path = snapshotter/modules/computes
url = https://github.com/PowerLoom/snapshotter-computes/
258 changes: 198 additions & 60 deletions README.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions config
Submodule config added at fcf9b8
114 changes: 0 additions & 114 deletions config/aggregator.example.json

This file was deleted.

12 changes: 0 additions & 12 deletions config/auth_settings.example.json

This file was deleted.

27 changes: 0 additions & 27 deletions config/preloader.json

This file was deleted.

43 changes: 0 additions & 43 deletions config/projects.example.json

This file was deleted.

131 changes: 0 additions & 131 deletions config/settings.example.json

This file was deleted.

Binary file removed out/snapshotter/modules/pooler/flow/flow.png
Binary file not shown.
Loading
Loading