Query by filename

[ahuang11]

Closes #873

For context, I uploaded two completely separate documents:

1. a receipt
2. an org chart

When I asked what's in the receipt, it referenced contents from the org chart instead of the receipt because the receipt doesn't mention it's a receipt in its contents.

This PR adds some weights to the relevancy based on filename

[amaloney]

lgtm

[philippjfr]

Would like some discussion around the decision to insert the filename as a separate kind of entry vs. simply adding it as a metadata field (which seems conceptually cleaner to me).

[philippjfr]

I guess the obvious reason is that the metadata is currently only used for filtering and the embeddings are only computed for the contents? In which case I guess my question is whether we should automatically include the embeddings in the metadata somehow.

[ahuang11]

If it's added only to the metadata, I believe embeddings can't query it, they can only perform an exact match case filter vs entry, it can do fuzzy search.

[ahuang11]

In which case I guess my question is whether we should automatically include the embeddings in the metadata somehow.

I suppose we could have another database column containing a joined text of everything, and have a toggle to query_metadata = True/False

text | metadata | both

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 22 lines in your changes missing coverage. Please review.

Project coverage is 51.59%. Comparing base (3b580de) to head (e08afca).
Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
lumen/ai/tools.py	0.00%	21 Missing ⚠️
lumen/ai/controls.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #883      +/-   ##
==========================================
- Coverage   51.66%   51.59%   -0.07%     
==========================================
  Files         108      108              
  Lines       13697    13714      +17     
==========================================
  Hits         7076     7076              
- Misses       6621     6638      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ahuang11]

Superseded by #911