Query by metadata

[ahuang11]

Supersedes #883 based on:

I guess the obvious reason is that the metadata is currently only used for filtering and the embeddings are only computed for the contents? In which case I guess my question is whether we should automatically include the embeddings in the metadata somehow.

I suppose we could have another database column containing a joined text of everything, and have a toggle to query_metadata = True/False

Please augment your response with the following context if relevant:
- Receipt Invoice number 1234 Receipt number Date paid 1234 December 15, 2024 Payment method American Express - 1234 OpenAI, LLC 548 Market Street PMB 97273 San Francisco, California 1234 United States ar@openai.com Bill to Me Address United State $22.07 paid on 2024 ... (Relevance: 0.2 - Metadata: {'text': 'Filename: Receipt-2813-2096 '}

[codecov]

Codecov Report

Attention: Patch coverage is 77.63158% with 17 lines in your changes missing coverage. Please review.

Project coverage is 58.46%. Comparing base (9efa066) to head (512c229).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
lumen/ai/tools.py	0.00%	7 Missing ⚠️
lumen/ai/controls.py	0.00%	5 Missing ⚠️
lumen/ai/vector_store.py	91.37%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #911      +/-   ##
==========================================
+ Coverage   58.44%   58.46%   +0.01%     
==========================================
  Files         109      109              
  Lines       13868    13884      +16     
==========================================
+ Hits         8105     8117      +12     
- Misses       5763     5767       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[philippjfr]

Sorry I wasn't clear before. I wasn't saying I prefer this approach, I just wanted to weigh the pros and cons of both approaches.

[philippjfr]

Well that isn't quite true, I definitely prefer treating the filename as metadata rather than having a distinct row for it, but as we are discovering it does have drawbacks.

[ahuang11]

I personally prefer this approach.

Perhaps it could be more targeted with query_with_metadata = ["<key>", "<key2>"] so like query_with_metadata = ["filename"]

[ahuang11]

Alternatively, could go back to the old approach, but store metadata as a separate table, then joined on the main table.

[ahuang11]

Looking at ChromaDB, they have explicit filters for metadata filtering
https://docs.trychroma.com/docs/querying-collections/metadata-filtering

[philippjfr]

My main question is whether it's really necessary to store a text and metadata column and compute the additional embeddings. Does always including the metadata in the embedding result in any appreciable performance degredation?

[ahuang11]

Does always including the metadata in the embedding result in any appreciable performance degredation?

Do you mean rather than text | metadata | text_and_metadata, simply keep text_and_metadata | metadata, and then do postprocessing on text_and_metadata on retrieval, e.g. split by some delimiter like metadata tags ||||>

In this PR, we only look up by text_and_metadata, but return only text

[philippjfr]

So it's just metadata, not text_and_metadata?

[ahuang11]

Seems to work decently now! I also bumped up the chunk size for better context.

Edit: there seems to be duplication; let me see if I can fix

[ahuang11]

Okay this is now ready.

I also added functionality where if user uploads the same file (perhaps with modified contents), it'll overwrite the existing document.

[philippjfr]

Looks good!