Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request: CreateLookupVindex for multiple lookup tables #14017

Open
Tracked by #13960
maxenglander opened this issue Sep 19, 2023 · 1 comment
Open
Tracked by #13960

Feature Request: CreateLookupVindex for multiple lookup tables #14017

maxenglander opened this issue Sep 19, 2023 · 1 comment
Labels
Component: VReplication Type: Feature

Comments

@maxenglander
Copy link
Collaborator

Feature Description

Currently CreateLookupVindex only lets you backfill a single lookup vindex at a time. Enhance it to backfill multiple lookup vindexes at a time.

Use Case(s)

CreateLookupVindex will create a stream from each shard to every other shard in a keyspace. This is pretty expensive to begin with, and if you try to run multiple CreateLookupVindex workflows at the same time, it is very easy to OOM the tablets.

If CreateLookupVindex behaved more like MoveTables, each shard-shard connection would move through Copying phase for each requested lookup Vindex in sequence, periodically switch to catch-up mode, and use a single BLS to keep all tables up-to-date in Running phase.
@maxenglander maxenglander added Type: Feature Needs Triage This issue needs to be correctly labelled and triaged labels Sep 19, 2023
@mattlord mattlord added Component: VReplication and removed Needs Triage This issue needs to be correctly labelled and triaged labels Oct 26, 2023
@mattlord
Copy link
Contributor

mattlord commented Apr 3, 2024

Related to: #13960

@mattlord mattlord mentioned this issue May 20, 2024
Open
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: VReplication Type: Feature
Projects
VReplication
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maxenglander @mattlord