ref(ai-autofix): Better design for document chunk models #276
Conversation
|
|
||
| repo_info = RepositoryInfo.from_db(db_repo_info) | ||
|
|
||
| db_chunks = [chunk.to_db_model() for chunk in embedded_chunks] | ||
| session.add_all(db_chunks) |
There was a problem hiding this comment.
i'm wondering if this should be batched? not sure how large of an operation it has to be before we care about that, but i can imagine this potentially being a very large transaction in the case of a large repo
There was a problem hiding this comment.
@corps today said it shouldn't be an issue at all; he worked with millions of inserts before. And it's actually good in the case that 1 fails the whole transaction fails rather than a single batch. We don't want repos with partially missing data
There was a problem hiding this comment.
FWIW i'm pretty sure there is a way to batch the operation while still treating it as an atomic transaction (e.g. if it fails halfway through, everything is rolled back). At the very least it might be worth adding a TODO here to revisit this (also worth noting that each row here is very large relative to a typical pg operation, so it may take significantly fewer rows than usual before its problematic)
Attempts to fix TIMESERIES-ANALYSIS-SERVICE-2S
Refactors the document chunk models to not need a
repo_idin the base model (now renamedBaseDocumentChunk) while ones with embeddings are nowEmbeddedDocumentChunkand the one that has been retrieved from db isStoredDocumentChunk.This way it is clearer that you're working with
StoredDocumentChunkwhen retrieving from the database and the former two models are transitional and only used when processing chunks.