How to use RAG with citations

[mfgering]

[Edit: Added screen caps of current UI]

I love this package and django, but I need some advice for how to use RAG more effectively in my app.

I have overriden get_retriever(), and it is working ok. But, I'd like to know which docs the llm used in its answer. My current approach is to prepend a string to each split, i.e., "idx: <rest of chunk", and
to have instructions for identifying these at the end of the answer.

Here is an example system_message with rag context:

You are an assistant blah blah
At the end of your answer, list the idx values for each of the context documents in this format: contexts: <idx x>, <idx 2>, etc. If there are no idx values, answer None. 
Example 1 list of idx values: contexts: 42, 312, 4
Example 2 list of idx values: contexts: None

---START OF CONTEXT---
idx: 16 Fact: blah blah

idx: 15 Fact: blah blah

idx: 32 Rule 13: blah blah

idx: 180 Bylaws Section 5.12: blah blah---END OF CONTEXT---

Here is an example response:

Blah blah

contexts: 15, 32

Several questions:

1. Is there a better way to find the context docs used in the response?
2. I have metadata for each doc, but I don't see that it can be used for the query. Can it?
3. What is a good way to get the response from an AIAssistant, massage it (like parsing the 'contexts')?

Many thanks!

Screen cap of an answer:

[fjsj]

Hi @mfgering, nice to know you're happy with Django AI Assistant!

To assist you properly, could you please provide some clarification on what you're trying to achieve? Are you looking to use the "contexts" IDs for debugging? Or should the end user see that?

For debugging, the best is to use Langsmith. To configure it, all you need is setting LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY as env vars in your local project.

For the end user, you're looking to add "citations" to the response. It's doable with Langchain, but may require some changes to the retriever step of the underlying Langgraph. Unfortunately the library doesn't allow changing that easily, but I can adjust that if you need.

About "massaging the response", it would require change to record_response.

Let me know what you're looking for and I can change the lib to put more extension points for you!

[mfgering]

Thanks, @fjsj! I've made good progress since this post, so my current approach is a little different. Anyway...

I'm building an app that assists our owners and renters for our condo community. It has legal documents for bylaws, rules, etc. And we have other data that is official, but not legal, e.g. parking space assignments. The assistant will answer questions and provide citations, e.g. Bylaw Section 8.6 with text.

To implement this, my current solution is as follows:

1. Parse the bylaws, etc., into smaller documents with metadata like {"doc": "bylaws", "id": "Section 8.6", "title": "Animals"}.
2. Make chunks from the document. Each chunk has a prefix with some of the metadata because this package does RAG by using only the doc source (not the metadata) when composing the llm question. Add to each chunk an "idx: at the front. This is important for the llm to use in identifying relevant citations. This means every retrieved chunk is of the form: "idx:
3. Use TFIDFRetriever to retrieve these docs based on the user query.
4. Tailor the system prompt to produce a list of citations at the end of the answer, using the idx values in the context.
5. In the view code for showing thread messages, process the messages, looking for "ai" messages and especially the list of "idx" citations. Retrieve the idx value to retrieve the relevant doc from above. Change the message.content to be a dict with "answer" and "citations" keys.
6. The "citations" value is a list of citation dicts. Each citation dict has info for displaying, especially the "source" of the citation.
7. The templates for showing a message handle ai messages where the content is the above dict.

Main points are:

1. Use RAG
2. Make chunks meaningful for later processing
3. Use chuck "source" to encode meaning data, e.g. the "idx" value
4. Make the llm respond with a list of idx values
5. Process the response in the view and templates to extract the idx values and produce nice UI

I hope this helps.

[fjsj]

The architecture overall makes sense to me. Worth noting you can use agentic RAG (create a tool from the retriever you have with create_retriever_tool) instead of regular RAG. If you need specific extension points in Django AI Assistants, like methods to override, etc., let me know and we can adjust the lib.