Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexed view using DATA_STRUCTURE and fetch #8845

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kavilla
Copy link
Member

@kavilla kavilla commented Nov 12, 2024

Description

Very rough implementation of indexed view using the DATA_STRUCTURE generic object can handle this concept of indexed view.

I didn't do any clean up and it has mocked data of S3 . So i can summarize the basic changes.
Here I'm defining DATASET object. If the data type can supported Indexed Views, it will call a fetch with that current dataset. It can then know it is time to fetch my indexed views. If the user selects an index view then it will update the dataset with the indexed view but store a ref to the original data.

This way we are leveraging existing features since we should have everything we need at the point of time of selection.

Benefits

  • Follows the existing pattern with supports vs checking if a service is defined on the config
  • Re-uses existing functionality already covered by unit tests reducing how big the config is getting
  • Re-purposes a function that is generic enough to handle it while also using a generic data structure which leaves open the door for other data types to re-use this functionality

Issues Resolved

#8851

Screenshot

  • Able to query S3 table directly
    Screenshot 2024-11-13 at 4 40 27 AM
  • Selecting an that same table but with an indexed view (using an existing index)
    Screenshot 2024-11-13 at 4 40 48 AM
  • Queries the selected indexed view
    Screenshot 2024-11-13 at 4 41 03 AM

Testing the changes

Changelog

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

Copy link

codecov bot commented Nov 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.39%. Comparing base (76cf823) to head (80d028b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8845      +/-   ##
==========================================
- Coverage   60.86%   56.39%   -4.48%     
==========================================
  Files        3800     1239    -2561     
  Lines       90818    25913   -64905     
  Branches    14307     4437    -9870     
==========================================
- Hits        55276    14613   -40663     
+ Misses      32028    10559   -21469     
+ Partials     3514      741    -2773     
Flag Coverage Δ
Linux_1 ?
Linux_2 56.39% <ø> (ø)
Linux_3 ?
Linux_4 ?
Windows_1 ?
Windows_2 ?
Windows_3 ?
Windows_4 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

the `DATA_STRUCTURE` generic object can handle this concept of indexed
view. Here I'm defining DATASET object. If the data type
wants to handled the DATASET type and return an indexed view then
it can.

Added a mock of S3

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>
@kavilla kavilla changed the title WIP failing to create the dataset type Indexed view using DATA_STRUCTURE and fetch Nov 13, 2024
Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

1 similar comment
Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

@kavilla kavilla mentioned this pull request Nov 13, 2024
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant