ChatGPT is a text-based AI assistant by OpenAI. This is an analysis of ChatGPT.
If we send the message [{"role": "user", "content": "13+37="}] to the model, it returns the following chat completion response:
| Model | Completion |
|---|---|
| gpt-3.5-turbo-1106 | Link |
| gpt-4-0125-preview | Link |
The ChatGPT models employ a distinct vocabulary compared to their predecessors. The models use a cl100k_base vocabulary with 100,000 tokens, where each token encodes an average of 3.7 characters in English. For a detailed analysis, see vocab.ipynb.
Additionally, these models employ the Chat Markup Language.
The number of prompt tokens and completion tokens are computed as follows (see OpenAI-Cookbook for details):
prompt_tokens = ['<|im_start|>', 'user', '\n', '13', '+', '37', '=', '<|im_end|>', '<|im_start|>', 'assistant', '<|message|>']
# len(tokens) is 11completion_tokens = ['50']
# len(tokens) is 1We have evaluated the ChatGPT model gpt-4-0125-preview with the HumanEval dataset. Out of 164 programming problems, the model can solve 83.54%.
| Model name | Pass@1 | Date | Comment | Results | Prompt |
|---|---|---|---|---|---|
| gpt-4-0125-preview | 83.54% | 2024-01-27 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the completed Python code, including the necessary imports, in markdown quotes:\n{code} |
| gpt-4-1106-preview | 87.20% | 2023-11-08 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the completed Python code, including the necessary imports, in markdown quotes:\n{code} |
| gpt-3.5-turbo-1106 | 71.95% | 2023-11-07 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the completed Python code, including the necessary imports, in markdown quotes:\n{code} |
| gpt-4-0613 | 86.59% | 2023-06-13 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the completed Python code in markdown quotes:\n{code} |
| gpt-3.5-turbo-0613 | 71.34% | 2023-06-13 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the completed Python code in markdown quotes:\n{code} |
| gpt-4-0314 | 78.66% | 2023-03-17 | https://platform.openai.com/ | Link | Complete the following code:\n{code} |
| gpt-3.5-turbo-0301 | 74.39% | 2023-04-24 | https://platform.openai.com/ | Link | Complete the following code. Use ```python to put the Python code in markdown quotes:\n{code} |
| text-davinci-002-render-sha | 70.12% | 2023-02-19 | https://chat.openai.com/ | ||
| text-davinci-002-render | 56.10% | 2022-12-03 | https://chat.openai.com/ | ||
| cushman-ml | 56.10% | 2022-10-23 | Copilot | ||
| code-davinci-002 | 46.95% | 2022-10-23 | https://platform.openai.com/ | ||
| code-cushman-001 | 32.93% | 2022-10-23 | https://platform.openai.com/ |