💬 Chatting with the Chatbot
The chat lies at the core of the app, and constitutes the most complex chain of logic which the following page will outline. Each message that the user sends leads to one exchange, where we collect the user's message and generate exactly one reply from the bot and show it to the user. In the process of generating the reply, we might query documents or send an email, which are actions that the bot might decide to take.
Code found in these files: ChatBox.tsx, ChatBoxInput.tsx
In the frontend, the main chat is rendered in the ChatBox component (ChatBox.tsx), which is responsible for showing the message feed (history), the input and some buttons, as shown in the above picture. When the user types their message then hits send, we run sendChatMessage, which begins our chain of logic by sending a POST request to the backend /openai/chat (via sendMessage, a method defined in chatService.ts).
Code found in these files: router.ts, chatController.ts
In the backend, the request is captured by express and routed (router.ts) to handleChatToGPT (chatController.ts). The request gets validated and we gather some further important information from the backend's session (if deployed, from a database): The model that we will chat to (e.g. GPT3-turbo or GPT4), the defence configuration for the current level and the chat history for the current level.
What happens next depends on which level the user is on. If they are on level 1 or 2, then we move on to getting the bot's reply to the user's message. However if they are on level 3 or sandbox, then we apply defences (asynchronously) before continuing. See handleChatWithoutDefenceDetection and handleChatWithDefenceDetection.
Code found in these files: chatController.ts
The defences defend the chatbot system from malicious users. There are two types of defence: Transformation Defences and Blocking Defences. Transformation defences will alter a user's message to make it safer for an LLM to consume (e.g. random sequence enclosure). A Blocking Defence will prevent the user from getting any reply from the chatbot if the message could be malicious (e.g. filtering).
So how do we apply these? For Transformation Defences we alter the user's message to make a transformed version, and it is this new version of the message that we pass further into our system, and that the LLM responds to. For Blocking Defences, we determine if the user's message should be blocked according to the rules of the defence and the outcomes of all the Blocking Defences are kept in a defenceReport. We continue getting the reply from the chatbot, but if the user's message is blocked, then we simply do not show the user the reply (This seems wasteful, but since the Blocking Defence detection happens in parallel with getting the bot reply, it should save time).
Code found in these files: chatController.ts, openai.ts
Finally we come to getting the reply out of the LLM. This works by appending the user's message (or transformed message) to the end of the chat history and asking the LLM to reply according to the whole history, which is how the bot "has memory" and replies according to the context of the conversation. The details of how we interfact with the LLM are written on the OpenAi page.
The following logic is encapsulated within the method getFinalReplyAfterAllToolCalls (openai.ts). First of all, we give the LLM our chatHistory (complete with new user message), and get an initial reply as described above. Now, the LLM is also aware of a number of tools (chatGptTools) that we have provided it with, which allows it to invoke methods in backend (namely querying the documents or sending an email). It might, as part of its initial reply, decide to make a tool call. If so, we execute the method that it is trying to invoke, adding the tool call and the outcome as separate items in the chat history.
Then we get a fresh reply from the LLM given the new history (which now includes the tool call). But this fresh reply could also include a tool call! And so we keep looping round until there are no more tool calls, and at that point we have our final reply.
Code found in these files: chatController.ts
We have one defence that applies to the bot's reply, which is output filtering. Like before when we applied defences, we generate a defenceReport which contains the outcome.
Code found in these files: chatController.ts
If we're in level 3 or sandbox and a message was blocked by the defences, then we roll back the chat history, removing all the new function call entries, so that the user's new message is the latest entry. We append a new message to the history saying why it was blocked and we send a response containing that defenceReport that we generated and no reply.
Else we add the chatbot reply to the history and send a response containing the reply.
Back in the frontend's sendChatMessage (ChatBox.tsx) which kicked off this whole shebang, we receive the API response. From here we call processChatResponse which, depending on the response, adds the reply and/or any generated information messages or blocked or error messages to the frontend's version of the chatHistory, when then get rendered in the ChatBoxFeed.
