Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support Structured Outputs #813

Merged
merged 5 commits into from
Aug 7, 2024
Merged

feat: Support Structured Outputs #813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
56ef2a3
feat: Support Structured Outputs
eiixy Aug 7, 2024
1c0b108
feat: Support Structured Outputs
eiixy Aug 7, 2024
3ef0820
update imports
eiixy Aug 7, 2024
8d673cc
add integration test
eiixy Aug 7, 2024
68ee228
update JSON schema comments
eiixy Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions api_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ package openai_test

import (
"context"
"encoding/json"
"errors"
"io"
"os"
Expand Down Expand Up @@ -178,3 +179,63 @@ func TestAPIError(t *testing.T) {
t.Fatal("Empty error message occurred")
}
}

func TestChatCompletionResponseFormat_JSONSchema(t *testing.T) {
apiToken := os.Getenv("OPENAI_TOKEN")
if apiToken == "" {
t.Skip("Skipping testing against production OpenAI API. Set OPENAI_TOKEN environment variable to enable it.")
}

var err error
c := openai.NewClient(apiToken)
ctx := context.Background()

resp, err := c.CreateChatCompletion(
ctx,
openai.ChatCompletionRequest{
Model: openai.GPT4oMini,
Messages: []openai.ChatCompletionMessage{
{
Role: openai.ChatMessageRoleSystem,
Content: "Please enter a string, and we will convert it into the following naming conventions:" +
"1. PascalCase: Each word starts with an uppercase letter, with no spaces or separators." +
"2. CamelCase: The first word starts with a lowercase letter, " +
"and subsequent words start with an uppercase letter, with no spaces or separators." +
"3. KebabCase: All letters are lowercase, with words separated by hyphens `-`." +
"4. SnakeCase: All letters are lowercase, with words separated by underscores `_`.",
},
{
Role: openai.ChatMessageRoleUser,
Content: "Hello World",
},
},
ResponseFormat: &openai.ChatCompletionResponseFormat{
Type: openai.ChatCompletionResponseFormatTypeJSONSchema,
JSONSchema: openai.ChatCompletionResponseFormatJSONSchema{
Name: "cases",
Schema: jsonschema.Definition{
Type: jsonschema.Object,
Properties: map[string]jsonschema.Definition{
"PascalCase": jsonschema.Definition{Type: jsonschema.String},
"CamelCase": jsonschema.Definition{Type: jsonschema.String},
"KebabCase": jsonschema.Definition{Type: jsonschema.String},
"SnakeCase": jsonschema.Definition{Type: jsonschema.String},
},
Required: []string{"PascalCase", "CamelCase", "KebabCase", "SnakeCase"},
AdditionalProperties: false,
},
Strict: true,
},
},
},
)
checks.NoError(t, err, "CreateChatCompletion (use json_schema response) returned error")
var result = make(map[string]string)
err = json.Unmarshal([]byte(resp.Choices[0].Message.Content), &result)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be super cool if we could do that automatically based on JSON schema in the future!

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it is not possible to construct a Go type based on json schema description in the current form.

What we could do is to use struct tags like https://github.com/invopop/jsonschema is doing, and than be able to both generate json schema from Go struct and also be able to automatically unmarshal structured responses

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this to our v2 roadmap as might require breaking changes #801

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implement it like this?

package jsonschema

func Unmarshal(schema Definition, data []byte,v any) error {
	// TODO
}

Copy link
Owner

@sashabaranov sashabaranov Aug 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eiixy Yes, pretty much! The only problem is that you need a definition and Go type. In this test the definition is

openai.ChatCompletionResponseFormatJSONSchema{
	Name: "cases",
	Schema: jsonschema.Definition{
		Type: jsonschema.Object,
		Properties: map[string]jsonschema.Definition{
			"PascalCase": jsonschema.Definition{Type: jsonschema.String},
			"CamelCase":  jsonschema.Definition{Type: jsonschema.String},
			"KebabCase":  jsonschema.Definition{Type: jsonschema.String},
			"SnakeCase":  jsonschema.Definition{Type: jsonschema.String},
		},
		Required:             []string{"PascalCase", "CamelCase", "KebabCase", "SnakeCase"},
		AdditionalProperties: false,
	},
	Strict: true,
},

and Go type is map[string]string.

Ideally, we would like to

type MyStructuredResponse struct {
	PascalCase string
	CamelCase string
	KebabCase string
	SnakeCase string
}

So that could be automatically used as a schema definition (or converted to it) and unmarshalling of structured response. Potentially we can have a function that creates jsonschema.Definition from a given struct

EDIT: types in the last code sample

Copy link

@gspeicher gspeicher Aug 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together so quickly. I already have a plaintext JSON schema that I have been sending to the API in the system prompt, and already have a Go struct to hold the unmarshalled response. I was hoping to reuse both with the new Structured Outputs feature without having to manually convert the schema to a jsonschema.Definition but I don't see any way around that.

Could you possibly change the accepted schema type from jsonschema.Definition to json.Marshaler from the standard Go encoding/json package? The jsonschema.Definition godoc itself states "It is fairly limited, and you may have better luck using a third-party library." but by defining Schema as a jsonschema.Definition we are precluded from using any third-party library to construct the schema. This would be a backward compatible change since jsonschema.Definition already implements MarshalJSON.

Thanks for your consideration.

Copy link
Contributor Author

@eiixy eiixy Aug 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sashabaranov Could you review the feasibility of this implementation plan? #819

type MyStructuredResponse struct {
    PascalCase string `json:"pascal_case" required:"true" description:"PascalCase"`
    CamelCase  string `json:"camel_case" required:"true" description:"CamelCase"`
    KebabCase  string `json:"kebab_case" required:"true" description:"KebabCase"`
    SnakeCase  string `json:"snake_case" required:"true" description:"SnakeCase"`
}
schema := jsonschema.Warp(MyStructuredResponse{})
resp, err := c.CreateChatCompletion(
    ctx,
    openai.ChatCompletionRequest{
        Model: openai.GPT4oMini,
        Messages: []openai.ChatCompletionMessage{
            {
                Role: openai.ChatMessageRoleSystem,
                Content: "Please enter a string, and we will convert it into the following naming conventions:" +
                    "1. PascalCase: Each word starts with an uppercase letter, with no spaces or separators." +
                    "2. CamelCase: The first word starts with a lowercase letter, " +
                    "and subsequent words start with an uppercase letter, with no spaces or separators." +
                    "3. KebabCase: All letters are lowercase, with words separated by hyphens `-`." +
                    "4. SnakeCase: All letters are lowercase, with words separated by underscores `_`.",
            },
            {
                Role:    openai.ChatMessageRoleUser,
                Content: "Hello World",
            },
        },
        ResponseFormat: &openai.ChatCompletionResponseFormat{
            Type: openai.ChatCompletionResponseFormatTypeJSONSchema,
            JSONSchema: &openai.ChatCompletionResponseFormatJSONSchema{
                Name:   "cases",
                Schema: schema,
                Strict: true,
            },
        },
    },
)
checks.NoError(t, err, "CreateChatCompletion (use json_schema response) returned error")
if err == nil {
    _, err = schema.Unmarshal(resp.Choices[0].Message.Content)
    checks.NoError(t, err, "CreateChatCompletion (use json_schema response) unmarshal error")
}

checks.NoError(t, err, "CreateChatCompletion (use json_schema response) unmarshal error")
for _, key := range []string{"PascalCase", "CamelCase", "KebabCase", "SnakeCase"} {
if _, ok := result[key]; !ok {
t.Errorf("key:%s does not exist.", key)
}
}
}
13 changes: 12 additions & 1 deletion chat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"net/http"

"github.com/sashabaranov/go-openai/jsonschema"
)

// Chat message role defined by the OpenAI API.
Expand Down Expand Up @@ -175,11 +177,20 @@ type ChatCompletionResponseFormatType string

const (
ChatCompletionResponseFormatTypeJSONObject ChatCompletionResponseFormatType = "json_object"
ChatCompletionResponseFormatTypeJSONSchema ChatCompletionResponseFormatType = "json_schema"
ChatCompletionResponseFormatTypeText ChatCompletionResponseFormatType = "text"
)

type ChatCompletionResponseFormat struct {
Type ChatCompletionResponseFormatType `json:"type,omitempty"`
Type ChatCompletionResponseFormatType `json:"type,omitempty"`
JSONSchema ChatCompletionResponseFormatJSONSchema `json:"json_schema,omitempty"`
}

type ChatCompletionResponseFormatJSONSchema struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Schema jsonschema.Definition `json:"schema"`
Strict bool `json:"strict"`
}

// ChatCompletionRequest represents a request structure for chat completion API.
Expand Down
8 changes: 7 additions & 1 deletion jsonschema/json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,17 @@ type Definition struct {
// one element, where each element is unique. You will probably only use this with strings.
Enum []string `json:"enum,omitempty"`
// Properties describes the properties of an object, if the schema type is Object.
Properties map[string]Definition `json:"properties"`
Properties map[string]Definition `json:"properties,omitempty"`
// Required specifies which properties are required, if the schema type is Object.
Required []string `json:"required,omitempty"`
// Items specifies which data type an array contains, if the schema type is Array.
Items *Definition `json:"items,omitempty"`
// AdditionalProperties is used to control the handling of properties in an object
// that are not explicitly defined in the properties section of the schema. example:
// additionalProperties: true
// additionalProperties: false
// additionalProperties: jsonschema.Definition{Type: jsonschema.String}
AdditionalProperties any `json:"additionalProperties,omitempty"`
}

func (d Definition) MarshalJSON() ([]byte, error) {
Expand Down
Loading