🤖 OpenAI

OpenAI is obviously a company that develops LLMs. They also provide a handy dandy API to interface with models which they host on their own servers. An to add to the confusion, OpenAI is the name of a library provided by the company, which lets developers use the API without making explicit API calls (like with axios, for instance). It abstracts all that away into nice methods, which makes the API easier to work with.

This page is about how we use the OpenAI library in our project.

The readmes in the source GitHub repository act as documentation for the library, but here we will outline just the parts of it that we use in our project.

Getting an Answer to a Message

Code found in these files: openai.ts

See Chatting with the Chatbot to get the full story, from the user sending a message to seeing the final response. Here, we cover the small bit of logic that gets the reply using the library.

Actually, the logic is all captured in exactly one method call:

const chat_completion = await getOpenAI().chat.completions.create({
	model: chatModel.id,
	temperature: chatModel.configuration.temperature,
	top_p: chatModel.configuration.topP,
	frequency_penalty: chatModel.configuration.frequencyPenalty,
	presence_penalty: chatModel.configuration.presencePenalty,
	messages: getChatCompletionsInLimitedContextWindow(
		updatedChatHistory,
		chatModel.id
	),
	tools: chatGptTools,
});

This method wraps around the POST https://api.openai.com/v1/chat/completions endpoint, and reading the documentation for that endpoint will help you understand the method. There's a fair bit of complexity to unpack here, so let's do it bit by bit.

Initialising an OpenAI Object

First of we must initialise an instance of the OpenAI class. We must have an API key which you can get from the OpenAI API website. The api key sits in the environment variables, and so we have a helper method getOpenAIKey to fetch it.

function getOpenAI() {
	return new OpenAI({ apiKey: getOpenAIKey() });
}

this object contains all the methods necessary to interface with the OpenAI API. In particular we use the .chat.completions.create method, which "Creates a model response for the given chat conversation" (documentation).

What Do We Pass In?

.chat.completions.create takes one object with a bunch of properties:

• model is the string id of which OpenAI model ("gpt-4", "gpt-3.5-turbo-1106" etc) we want to use. We use gpt-3.5-turbo by default, but in the sandbox the user can choose a different model.
• temperature, top_p, frequency_penalty and presence_penalty are all number values that tweak the model's behaviour. Better descriptions for each are in the documentation. Our values for these are kept in the model configuration part of the session. They take default values unless it's sandbox, where the user can choose to alter them.
• messages is an array that contains the history of the conversation as well as the newest message sent from the user. This allows the model to respond to the latest message with all the context of what came before. It allows the bot (if you'll excuse the anthropomorphism) to have memory. Now, a conversation with the bot can become quite long. It costs us more to use the OpenAI API the more tokens we pass in as part of our response. Tokens are how the model splits up a natural language sentence to be interpreted. The upshot is that we only pass the recent conversation history to the bot so that we don't invoke too many costs. The method that does this is getChatCompletionsInLimitedContextWindow.
• tools is an array of tools that we allow the model to use. In SpyLogic, all these tools are for function calls. Then in response to a user's message, it could ask for one of these methods to be invoked with arguments of its choice. We are basically giving the model knowledge about specific functions in our codebase.

What Do We Get Out?

.chat.completions.create returns a chat completion object. From this we extract the message:

const chat_completion = await getOpenAI().chat.completions.create({...})
const message = chat_completion.choices[0].message

The purpose of the choices array is to facilitate alternate options of response from the LLM given the same input. For example We could pass n: 5 as an argument to .chat.completions.create, and then receive 5 different responses from the LLM. In our app, though, we always just get one choice.

Now, this message object contains the text of the bot's response as well as any tools that it wants to invoke.

Callin' Tools

As mentioned before, when we call .chat.completions.create we pass in an array of tools that we allow the model to use, and in our case all those tools are functions. Then in response to a user's message, it could ask for one of these functions to be invoked with arguments of its choice. We are basically giving the model knowledge about functions in our codebase.

In our app, we let it access two functions:

• askQuestion which lets us query information in the documents using a separate LLM.
• sendEmail which simulates sending an email in the game.

The reply from the bot will include the tools it wants to use, like

const chat_completion = await getOpenAI().chat.completions.create({...})
const tool_calls = chat_completion.choices[0].message.completion.tool_calls

Now we pass the tool_calls array to performToolCalls, which will check that it recognizes the tool calls and then invoke the correct method.

Valid Models

We are mindful of the fact that different people that clone and run this project might choose not to pay for their OpenAI key. In that case they only get access to a limited number of models. So we have included the method getValidModelsFromOpenAI, which finds out what models are available to a particular API key.