Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QoS Oracle V2] Processing pipeline for the Oracle #4

Open
juanmardefago opened this issue Jun 5, 2024 · 1 comment
Open

[QoS Oracle V2] Processing pipeline for the Oracle #4

juanmardefago opened this issue Jun 5, 2024 · 1 comment
Assignees
Labels
meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) p1 High priority size:large Large type:feature New or enhanced functionality

Comments

@juanmardefago
Copy link
Collaborator

juanmardefago commented Jun 5, 2024

Depending on the implementation of the Kafka consumer, we might need to either have some more complex aggregation/processing pipeline or not, but we need to make sure our time buckets are properly processed to be published to IPFS.

This means we have to make sure the data points received from Kafka are aggregated properly per time bucket (5m), thus ensuring all relevant data points are included into the aggregation for that bucket, and afterwards, for that IPFS file.
As an example, if we have multiple data points that denote different indexers QoS data that all share the same bucket, we must ensure that not only the aggregations for each of those indexers data points are done correctly, but that all of those indexer-wide aggregations are included in the IPFS file for that particular bucket.

@juanmardefago juanmardefago self-assigned this Jun 5, 2024
@juanmardefago juanmardefago added size:large Large p1 High priority type:feature New or enhanced functionality meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) labels Jun 10, 2024
@juanmardefago
Copy link
Collaborator Author

PR here: #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) p1 High priority size:large Large type:feature New or enhanced functionality
Projects
None yet
Development

No branches or pull requests

1 participant