⛓ Langchain: Querying Documents and Evaluating Prompts

We use the langchain library for:

• Querying documents - retrieving information from files of text data.
• Evaluating prompts - looking at a prompt and judging whether it is malicious or not.

Querying Documents

As part of the process of chatting with the chatbot, the chatbot might decide to reference some documents that we have created and given it access to (see backend/resources/documents), mimicking how in real life a chatbot might be integrated into a company's system. See OpenAI to see how the main chat LLM initiates a query.

There are a few steps to querying the documents.

Initialising Document Vectors

Code found in these files: document.ts

As part of starting up the backend, we load up the documents (backend/resources/documents) and store them in a format that is able to be consumed by langchain's RetrievalQAChain. The type in particular is MemoryVectorStore. We recommend this blog post to learn about vector representations and embeddings.

The levels have different documents associated with them since we want the user to uncover unique information for each one. However we also have a set of documents (resources/documents/common) that are common to all levels. When we wish to query documents, we must pass the LLM exactly one VectorMemoryStore at a time. Therefore for each level we must create a VectorMemoryStore that includes both the level-specific documents and the common documents in the same embedding, You can see this happening in initDocumentVectors.

Beginning a Query

Code found in these files: openai.ts

A query begins when the main chatbot llm decides it wants to make a tool call of type askQuestion, which we check in chatGptCallFunction. If so we work out what question it wants to ask, then move to queryDocuments.

Initialising the QA Chain

Code found in these files: langchain.ts

Here is where langchain comes to shine. The library contains a number of "off the shelf" chains, one of which is the RetrievalQAChain, which combines the two processes of retrieving information from the document vectors according to a question, and generating an answer based on that information.

Initialisation happens in initQAModel, and a new instance of the chain is initialised each time we want to call the documents. There are three important things we need in order to initialise this RetrievalQAChain:

• The document vectors for the given level. See Initialising document vectors.
• A wrapper for the model which will be used in the chain. Langchain provides a bunch of wrapper classes for different chat models. We use ChatOpenAI. Do not confuse this with classes from the openai library! We initialise ChatOpenAI with our own openAI api key.
• The prompt template. We use a partial prompt template which allows us to make a prompt with some missing values that can be filled in at time of calling the chain. See makePromptTemplate.

Calling the QA Chain

Code found in these files: langchain.ts

Compared to initialisation, calling our chain is simple:

qaChain.call({
	query: question,
})

Where qaChain is our initialised instance of RetrievalQAChain. Here, the object represents the values that will be passed into the prompt template. The library handles the rest!

Processing the QA Chain Output

The output of the QA chain will be an answer to the given question, as a human language string, informed by the documents and it is returned from queryDocuments. The reply is passed up the call stack until it gets back to performToolCalls where it is appended to the chat history, so that we can get a further response from the main chat LLM based on the outcome (getFinalReplyAfterAllToolCalls).

---

Evaluating Prompts

Evaluating documents is much simpler than querying documents. There're not so many moving parts.

If we chat with the chatbot on level 3 or sandbox, then we apply active defences. One of the defences that can block a user message is the prompt evaluation LLM, which asks a LLM to judge whether a user's message is malicious or not.

Evaluation happens in evaluatePrompt and follows the same approach as queryDocuments.

Initialising the Prompt Evaluation Chain

Code found in these files: langchain.ts

Initialisation happens in initPromptEvaluationModel. Now we use another "off-the-shelf" chain provided by Langchain: LLMChain. Much like RetrievalQAChain, it relies on some things:

• A wrapper for the model which will be used in the chain. We use OpenAI. Again, do not confuse this with classes from the openai library! We initialise OpenAI with our own openAI api key.
• The prompt template. We use a partial prompt template which allows us to make a prompt with some missing values that can be filled in at time of calling the chain. See makePromptTemplate.

Calling the Prompt Evaluation Chain

Code found in these files: langchain.ts

With all the initialisation out of the way, langchain makes calling our chain simple:

const response = (await promptEvaluationChain.call({
	prompt: input,
})) as PromptEvaluationChainReply;

Now, the response is simply going to be the written output from the LLM, as a string. If it has followed the innstructions, it should come back simply with "yes." or "no.", indicating if the prompt was malicious or not. We can easily translate to a boolean value and return it from evaluatePrompt.