How to best store large documents

[aguadoenzo]

As part of a RAG project, I want to store the output of each step of the ingestion pipeline to avoid redoing a lot of work if I want to swap a component.
The pipeline is straightforward: Raw PDF -> Text -> Chunks -> Embeddings.

PDFs are in S3, and chunks & embeddings are in a Milvus collection. But I'm not sure where I should store the Text representation.

I only need to do the following with the text:

• Retrieve the text based on a document ID
• Check if a text has no related chunks / embeddings (so I know to generate them)

Could I store the full text in a field in my main collection ? Or should I create a new collection ? Or should I store it outside of Milvus altogether ?
I know the varchar limit is 65,535 chars, but maybe there's another feature I don't know about ?

[yhmo]

The max length of each text is 65535(bytes) for milvus Varchar field, currently milvus doesn't have the ability to store super long-length strings or binaries. You can use some other db to store the long-length strings, such as mongodb.