Architecture Overhaul, and Performance Improvements

[xadahiya]

Summary

Addresses and closes issue #47 while introducing significant architectural enhancements and performance optimizations.

Checklist

• Ensured my branch is up-to-date with the upstream/develop branch.
• Verified functionality and conducted thorough testing for Python versions 3.8.0 and above.
• Ran pre-commit checks against my changes.

Changes

Incorporates fixes from PRs #45, #50, and #53.

New Architecture

This revamped architecture transforms the snapshotter into a more modular and highly configurable system, fostering easy customization and seamless integration. The architecture comprises three core components:

1. Main Snapshotter Codebase:

   • Serves as the foundational element, defining crucial interfaces and handling a diverse set of tasks, from monitoring epoch release events to task distribution and snapshot management.

2. Configuration Files:

   • Located in the /config directory, these files play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.

3. Compute Modules:

   • Housed in the snapshotter/modules directory, these modules constitute the system's core, defining the computation logic for each project type and steering the snapshot generation process.

This architecture promotes the seamless interchange of configurations and modules. By maintaining these components in separate Git repositories, integrated through Git Submodules, adapting the system to different use cases becomes as straightforward as changing a Git branch. This approach offers unparalleled flexibility and versatility.

Learn more about using Git Submodules in the Git Submodules Documentation.

Other Changes and Improvements

• Added Docstrings for all major interfaces
• Introduced support for IPFS remote pinning
• Enhanced health check and respawning mechanisms
• Unified Phase 1 and Phase 2 architecture into a single snapshotter node
• Redesigned preloading architecture for improved performance reliability
• Moved IPFS uploads to be handled from the snapshotter node instead of Audit protocol to keep RabbitMQ messages lightweight
• Implemented bug fixes and logging improvements
• Updated README for clarity and completeness.

[anomit]

LGTM! 🚀