Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Architecture Overhaul, and Performance Improvements #55

Merged
merged 96 commits into from
Nov 30, 2023
Merged

Architecture Overhaul, and Performance Improvements #55

merged 96 commits into from
Nov 30, 2023

Commits on Sep 1, 2023

  1. feat: do not proceed with init script if snapshotter is not active

    @anomit
    anomit committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    cf4a9a8 View commit details
    Browse the repository at this point in the history

  2. feat: snapshotter active status check at runtime

    @anomit
    anomit committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    ee1e4ac View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2023

  1. chore: update example config for full pairs

    @xadahiya
    xadahiya committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    813412b View commit details
    Browse the repository at this point in the history

  2. build image for testnet_pretask

    @SwaroopH
    SwaroopH committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    275f234 View commit details
    Browse the repository at this point in the history

  3. trying to use larger runner

    @SwaroopH
    SwaroopH committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    ca73f3e View commit details
    Browse the repository at this point in the history

  4. chore: increase RPC limits for pretest

    @xadahiya
    xadahiya committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    2f0a60d View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'refs/remotes/origin/testnet_pretask' in… 

    …to testnet_pretask
    @xadahiya
    xadahiya committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    c98edda View commit details
    Browse the repository at this point in the history

  6. chore: skipping calculate from scratch from pretask

    @xadahiya
    xadahiya committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    e432437 View commit details
    Browse the repository at this point in the history

  7. chore: update events to work with testnet pretask protocol state

    @anomit
    anomit committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    114097c View commit details
    Browse the repository at this point in the history

  8. chore: ping reporting service every 30 secs

    @anomit
    anomit committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    823b916 View commit details
    Browse the repository at this point in the history

  9. chore: Update protocol state contract ABI

    @xadahiya
    xadahiya committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    afd500a View commit details
    Browse the repository at this point in the history

  10. chore: add web3 middleware caching strategy

    @anomit
    anomit committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    d86d681 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. chore: update function name for get all snapshotters

    @xadahiya
    xadahiya committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    4a69cba View commit details
    Browse the repository at this point in the history

  2. feat+chore: won't process further epochs is snapshotters is disabled,… 

    … reduce default rpc polling interval to 10 seconds
    @xadahiya
    xadahiya committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    2c4b1da View commit details
    Browse the repository at this point in the history

  3. Merge pull request #45 from PowerLoom/feat/testnet_pretask_snapshotte… 

    …r_id_check

Testnet pretask: snapshotter id check
    @xadahiya
    xadahiya authored Sep 4, 2023
    Configuration menu
    Copy the full SHA
    82af846 View commit details
    Browse the repository at this point in the history

  4. chore: made PROTOCOL_STATE_CONTRACT and PROST_RPC URL mandatory, remo… 

    …ve default values
    @xadahiya
    xadahiya committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    42777ef View commit details
    Browse the repository at this point in the history

  5. chore: remove middleware (was breaking due to version differences)

    @xadahiya
    xadahiya committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    272fc4b View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. chore: update config files to snapshot only chosen 5 pairs

    @xadahiya
    xadahiya committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    1a79100 View commit details
    Browse the repository at this point in the history

  2. chore: add testnet_5_pairs branch to build workflow

    @xadahiya
    xadahiya committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    5b46a58 View commit details
    Browse the repository at this point in the history

  3. doc: add sections on epoch state transition and related internal APIs

    @anomit
    anomit committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    2f0f4f0 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'testnet_pretask' into testnet_5_pairs

    @xadahiya
    xadahiya committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    ce19e4d View commit details
    Browse the repository at this point in the history

  5. fix: checksum instance ID EVM address for ID check on protocol

    @anomit
    anomit committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    5bc5fed View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2023

  1. chore: replace CAW pair with Matic

    @xadahiya
    xadahiya committed Sep 9, 2023
    Configuration menu
    Copy the full SHA
    1ce6cd7 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'refs/remotes/origin/testnet_5_pairs' in… 

    …to testnet_5_pairs
    @xadahiya
    xadahiya committed Sep 9, 2023
    Configuration menu
    Copy the full SHA
    4b61df1 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. fix: project config

    @xadahiya
    xadahiya committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    74863a9 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. hotfix: parsing of cached epoch processing dataset

    @anomit
    anomit committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    99f607e View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. chore: script to report on processhub core workers' running status

    @anomit
    anomit committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    c1b4f09 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2023

  1. hotfix: parsing of cached epoch processing dataset

    @anomit
    anomit committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    b577df8 View commit details
    Browse the repository at this point in the history

  2. chore: script to report on processhub core workers' running status

    @anomit
    anomit committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    6cdfaf4 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. chore: mv worker reporting script to required dir

    @anomit
    anomit committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    db0653b View commit details
    Browse the repository at this point in the history

  2. chore: mv worker reporting script to required dir

    @anomit
    anomit committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    25e06ca View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. fix: remove data source checksum when project is added from ProjectUp… 

    …dated event
    @xadahiya
    xadahiya committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    8b759a7 View commit details
    Browse the repository at this point in the history

  2. fix: remove data source checksum when project is added from ProjectUp… 

    …dated event
    @xadahiya
    xadahiya committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    f344e2f View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'refs/remotes/origin/testnet_5_pairs' in… 

    …to testnet_5_pairs
    @xadahiya
    xadahiya committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    61894ca View commit details
    Browse the repository at this point in the history

  4. feat(WIP): added periodic collection of health of processing status f… 

    …or last 3 epochs
    @anomit
    anomit committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    81417f0 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2023

  1. feat: clean respawn of all children worker processes

    @anomit
    anomit committed Sep 30, 2023
    Configuration menu
    Copy the full SHA
    1b74b56 View commit details
    Browse the repository at this point in the history

  2. chore: fix log message

    @anomit
    anomit committed Sep 30, 2023
    Configuration menu
    Copy the full SHA
    da5093a View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2023

  1. chore+fix: Update CLI, add processReport, remove outdated pidStatus, … 

    …fix data model (auto respawning working now)
    @xadahiya
    xadahiya committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    6f52005 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. chore: remove bulk aggregation for testnet lite moide participants

    @anomit
    anomit committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    e641173 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'testnet_5_pairs' into feat/health-recovery

    @anomit
    anomit authored Oct 2, 2023
    Configuration menu
    Copy the full SHA
    26bbc2c View commit details
    Browse the repository at this point in the history

  3. chore: revert redundant script

    @anomit
    anomit committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    984ae27 View commit details
    Browse the repository at this point in the history

  4. chore: adding back pretask network specific config

    @anomit
    anomit committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    1eeb2c8 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #50 from PowerLoom/feat/health-recovery-pretask 

    Feat/health recovery pretask
    @xadahiya
    xadahiya authored Oct 2, 2023
    Configuration menu
    Copy the full SHA
    c400eda View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. fix function name

    @getjiggy
    getjiggy committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    416f340 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2023

  1. latest features from testnet nodes merged

    @anomit
    anomit committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    40d95b6 View commit details
    Browse the repository at this point in the history

  2. feat: IPFS and web3 upload moved within pooler

    @anomit
    anomit committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    5f79e42 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. chore+fix: proper initialization of IPFS clients

    @anomit
    anomit committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    f7a500e View commit details
    Browse the repository at this point in the history

  2. fix: add web3 storage API token in requests

    @anomit
    anomit committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    2fb83f3 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. fix: upload snapshot contents to web3 storage with auth

    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    f1b26c3 View commit details
    Browse the repository at this point in the history

  2. chore: upload standardized JSON dumps to IPFS and web3 storage

    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    13cd60c View commit details
    Browse the repository at this point in the history

  3. chore: clean up upload helpers

    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    dcebcf0 View commit details
    Browse the repository at this point in the history

  4. fix: ensure respawn attempts are not made every epoch. respawn intern… 

    …al status reporter thread in case of unexpected failure
    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    c22f3f5 View commit details
    Browse the repository at this point in the history

  5. fix: cleaner health check conditions. minor fixes in error tracing an… 

    …d logger binds
    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    bd80331 View commit details
    Browse the repository at this point in the history

  6. chore: further trimming of large exception trace logs

    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    74c9778 View commit details
    Browse the repository at this point in the history

  7. chore: cleanup excessive exception traces in nested rpc calls

    @anomit
    anomit committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    a62c17c View commit details
    Browse the repository at this point in the history

  8. fix: IPFS upload serialization

    @xadahiya
    xadahiya committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    7633ce6 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. chore: Address review comments (only uploading to web3storage if conf… 

    …igured, retry on IPFS upload failure, and remove 24h aggregation skipping code)
    @xadahiya
    xadahiya committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    13a17cb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/payload-commit-snapshotter-integration' into fix-s… 

    …pelling
    @xadahiya
    xadahiya authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    86529eb View commit details
    Browse the repository at this point in the history

  3. Merge pull request #51 from PowerLoom/fix-spelling 

    fix function name
    @xadahiya
    xadahiya authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    0c6d2ab View commit details
    Browse the repository at this point in the history

  4. Merge pull request #53 from PowerLoom/feat/payload-commit-snapshotter… 

    …-integration

Feat/payload commit snapshotter integration
    @xadahiya
    xadahiya authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    337c561 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. chore: update deps, add remote pinning config defaults

    @xadahiya
    xadahiya committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    d886d5a View commit details
    Browse the repository at this point in the history

  2. chore: using settings model from ipfs_client library

    @xadahiya
    xadahiya committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    d6762e8 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. chore: port improvements and todos from phase 2, minor cleanup

    @xadahiya
    xadahiya committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    28e135e View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. chore: remove project_id_generator script, improve snapshot_worker, u… 

    …pdate Project Config model, variable naming cleanup
    @xadahiya
    xadahiya committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    15701f3 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. fix: snapshot_worker process_task cleanup

    @xadahiya
    xadahiya committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    18093e7 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. chore: unify processor_distributor, cleanup extra files

    @xadahiya
    xadahiya committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    06923c4 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. chore: cleanup and initial docstring update

    @xadahiya
    xadahiya committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4226ca5 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. chore: update snapshotter_autofill to include web3_storage token and … 

    …dynamic namespace
    @xadahiya
    xadahiya committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    7839a5c View commit details
    Browse the repository at this point in the history

  2. fix: settings path in snapshotter_autofill for web3storage token

    @xadahiya
    xadahiya committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    869504a View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. chore: change prefetch count to 1 in workers, update deps

    @xadahiya
    xadahiya committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    2a3cae8 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Merge branch 'fix/health-recovery-respawn' into feat/single_snapshotter

    @xadahiya
    xadahiya committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    5842531 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. feat: cleaner epoch processing health check decoupled from process hu… 

    …b core
    @anomit
    anomit committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    f970ee2 View commit details
    Browse the repository at this point in the history

  2. fix: import formatting

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    4950fea View commit details
    Browse the repository at this point in the history

  3. Merge branch 'feat/health_recovery_port' into feat/single_snapshotter

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    5f5573f View commit details
    Browse the repository at this point in the history

  4. chore: cleanup imports, update dependencies

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    e520170 View commit details
    Browse the repository at this point in the history

  5. Fix Redis connection issue and handle reporter 

    thread crashes
    @anomit
    anomit committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    49d207c View commit details
    Browse the repository at this point in the history

  6. fix: init fixes in generic worker

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    d5dc8b4 View commit details
    Browse the repository at this point in the history

  7. fix: redis keys require integers as expire value

    @anomit
    anomit committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    48328d0 View commit details
    Browse the repository at this point in the history

  8. fix: bug fixes in processor distributor and aggregation worker

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    cebbd9d View commit details
    Browse the repository at this point in the history

  9. chore: cleanup aiohttp library from rpc, fix duplicate logs

    @xadahiya
    xadahiya committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    087c609 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2023

  1. Merge branch 'chore/docs_update' into feat/single_snapshotter

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    34b3b2b View commit details
    Browse the repository at this point in the history

  2. chore: init submodules

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    71b15cf View commit details
    Browse the repository at this point in the history

  3. chore: enable submodules in docker build, add zkevm_quests branch for… 

    … build
    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    a0d1c45 View commit details
    Browse the repository at this point in the history

  4. chore: improve health check and respawning

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    06c98ed View commit details
    Browse the repository at this point in the history

  5. chore: improve computes naming

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    7a5ab4c View commit details
    Browse the repository at this point in the history

  6. chore: udpate compute modules

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    2df1b9a View commit details
    Browse the repository at this point in the history

  7. chore: update configs, add branch to docker build workflow

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    4a43a68 View commit details
    Browse the repository at this point in the history

  8. chore: Initial README updates

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    36afc38 View commit details
    Browse the repository at this point in the history

  9. chore: update overall architecture diagram

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    6110d65 View commit details
    Browse the repository at this point in the history

  10. chore: update epoch detection and snapshot process unhealthy interval… 

    … to 30 epochs
    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    d8b7be3 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'zkevm_quests' into eth_uniswapv2

    @xadahiya
    xadahiya committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    ac8eaf5 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2023

  1. chore: udpate README.md

    @xadahiya
    xadahiya committed Nov 26, 2023
    Configuration menu
    Copy the full SHA
    a2d38c2 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. chore: addressed review comments (add submodule references and link t… 

    …o blog)
    @xadahiya
    xadahiya committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    b9df26b View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. chore: add missing README sections

    @xadahiya
    xadahiya committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    33f838a View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. chore: cleanup Data Signalling section (removed wallet-boost references)

    @xadahiya
    xadahiya committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    6346108 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. fix: typo AggreagateTradeVolumeProcessor -> AggregateTradeVolumeProce… 

    …ssor
    @xadahiya
    xadahiya committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    db969eb View commit details
    Browse the repository at this point in the history