sketch - Generate JSON (De)serializable Object From Go Schema

sketch is tool to generate Go structs and utility methods from a schema defined in Go code.

It is aimed to provide the basic implementation for those structs that represents something that is given through a relatively free-form definition format (such as JSON and YAML), while providing basic type-safety along with the ability to work with user-defined fields in an uniform manner.

The goal of this project is to provide a foundation for projects like github.com/lestrrat-go/jwx that need to express in Go objects that are represented as JSON.

WARNING: THIS PROJECT IS EXPERIMENTAL. It does work, but you will most likely have to implement the missing pieces if it does not do what you want it to do.

Use-Case / Assumptions

sketch should be used if you are dealing with objects that need to represent a piece of that datat that matches the following criteria:

1. The data has a JSON/YAML/etc based schema.
2. The data is defined with certain pre-defined fields. Fields may have been assigned a type, and you would like to handle them in a type-safe manner.
3. The data needs to be serialized from / deserialized from their respective formats.
4. The data may contain arbitrary user-defined fields, of which you do not know the schema before hand. These need to be serialized / deserialized the same way as the pre-defined fields.

To sum this all up: if you have a schema that defines an object, but the user is free to extend the schema, and you need to serialize/deserialize from JSON/YAML/etc, this may be the right tool for you.

How sketch works

sketch provides the schema package for you to describe an object definition.

Once you have these objects, you can use sketch command to generate the actual code to be used.

First install the sketch tool. Note that the version of your sketch tool must match the version of the github.com/lestrrat-go/sketch library you are using.

# TODO: change @latest to a released version
go get github.com/lestrrat-go/sketch/cmd/sketch@latest

Then declare a module that contains your schema.

We will assume that you want to implement a module named myproject.com/mymodule, hosted in your ~/go directory:

~/go/myproject.com/mymodule

We also assume that the above directory is where you want to generate the files into, but you want to "declare" the definitions under ~/go/myproject.com/mymodule/schema.

Given the above assumptions, create the directory ~/go/myproject.com/mymodule and initialize a new go module if you have not already done so:

mkdir ~/go/myproject.com/mymodule
cd ~/go/myproject.com/mymodule

go mod init myproject.com/mymodule

It is important to setup a proper module, as we need to compile a small intermediate program to generate the final code, and your code must be referencable from this small program.

Note that your module does NOT need to be available at an external URL, as sketch will take care of this resolution by using the replace directive in the generate go.mod file.

Next, create the directory to store your schema.

mkdir schema

Inside this directory, you can have any number of Go files, but sketch will pick up only those structs that are declared as embedding schema.Base types:

// mymodule/schema/schema.go
package schema

import (
  "github.com/lestrrat-go/sketch/schema"
)

type Thing struct {
  schema.Base
}

This will declare that you want sketch to generate code for an object Thing. The name of this package must be uppercased so that sketch can see it.

You may opt to "rename" the object. By default the name of the schema object will be used as-is, but by declaring the method Name() on the schema object, you can alter the generated object name. If below code is included, sketch will generate a struct named fooBarBaz:

func (Thing) Name() string {
  return "fooBarBaz"
}

Finally, you ou will want to declare the list of fields in this object. This is done by declaring a method named Fields() on the schema object, which returns a list of schema.FieldSpec objects.

func (Thing) Fields() []schema.FieldSpec {
  return []schema.FieldSpec{
    schema.String("Foo").
      JSON("foo-field"),
  }
}

In the above sample, the use of schema.String() implies that a field of string type is to be declared, with an exported name of Foo. It also specifies that the value of this field will be stored in a field named foo-field when serialized to JSON.

Then run the sketch command line utility. It is assumed that your schema above resides under /path/to/schema, and that you want to generate code to /path/to/dst

sketch -d /path/to/dst /path/to/schema

If successful, you should see several files created in the /path/to/dst directory.

Generated Code

Just by providing a simple schema, sketch utility generates a whole slew of methods and utilities. These include the main object, JSON serialization/deserialization, getters and setters, methods to query the presence of field values, Builders to initialize the objects, etc etc.

These are all generated by default, but you can control which ones get generated. If you want to tweak some of them yourself, or if you simply do not need them, You can use the internal names (show below) to specify that certain exclude parts should not be generated via --exclude.

Method/Struct	Internal Name	Description
Main Object	N/A	The main struct defnition. Will have the name provided by your schema
(Object).Set	object.method.Set	Method to set the value of an arbitrary field by its JSON field name
(Object).Get	object.method.Get	Method to retrieve the value of an arbitrary field by its JSON field name
(Object).Has	object.method.Has	Method to query the presence of a value of an arbitrary field by its JSON field name
(Object).HasXXXXX	object.method.HasXXXXX	Method to query the presence of a value of field XXXXX
(Object).XXXXX	object.method.XXXXX	Method to retrieve the value of field XXXXX. Unlike Get, these methods are appropriately typed
(Object).Remove	object.method.Remove	Method to remove the value of an arbitrary field by its JSON field name
(Object).Keys	object.method.Keys	Method to retrieve the JSON key names that are present in the object
(Object).MarshalJSON	object.method.MarshalJSON	Method to serialize the object into JSON
(Object).UnmarshalJSON	object.method.UnmarshalJSON	Method to deserialize the object from JSON
(Object).Clone	object.method.Clone	Method to clone an object
Builder Object	builder.struct	The builder struct definition. Will have the name of your object plus "Builder"
(Builder).XXXXX	builder.method.XXXX	Method to initialize the value of field XXXXX via the Builder. Unlike (Object).Set, these methods are appropriately typed
`(Builder).SetField	Method to set an arbitrary field in the object, presumably not on the pre-defined list of attributes
(Builder).Build	builder.method.Build	Method to build and return the object from the Builder
(Builder).MustBuild	builder.method.MustBuild	Method to build and return the object from the Builder

Templates

Syntax

Templates in sketch are all written using text/template.

Functions

sketch provides several functions that can be used from within the template

Name	Signature	Description
comment	comment (string, any)	Formats the comment. The first argument can be a text/template style template. The second argument is the variable passed to the template.
hasTemplate	hasTemplate (string) bool	Returns true if the template specified in the argument exists
runTemplate	runTemplate (string, any)	Executes the named template with the second argument as the template variables

Variables

The only available template variable is the current schema object (the one you declared using schema.Base) that is being used to generate code.

This will be set to $ globally, and is available as the default context variable for each template block

{{ define "ext/object/header" }}
{{ .Name }}{{- /* will print schema object name */ -}}
{{ $.Name }}{{- /* same as above */ -}}
{{ end }}

Extra Templates / Overriding Templates

Users can specify their own templates to be processed along side with the system templates that come with this tool.

Assuming you have your templates located under /path/to/templates, you can specify sketch to use templates in this directory by using the --tmpl-dir command line option:

sketch --tmpl-dir=/path/to/templates

Core Templates

Core Templates are templates that come with sketch tool itself. Normally you do not need to do anything with these, but if you want to fundamentally change the way sketch generates code, you can override them from your extra templates.

To do this, simply define template blocks with the same name as the template blocks provided by the core templates.

For example, to override the main template that generates object headers, you can declare a template block named "object/header" in your template:

{{ define "object/header" }}
// Your custom header code goes here
{{ end }}

Note that this does not ADD to the core template, but completely replaces it.

There are two types of templates: ones that generate files, and others that are used as components that is called from within other templates. Any template whose name starts with files/ will generate files according to some heuristics.

If the path component following files/ is named per-object, the template is applied for each of the object schema found. If the value is per-run, then the template is evaluated only once.

Unless otherwise stated, all file names will be transformed so that it ends with _gen. If the template name contains a suffix, the filename will end with _gen.$suffix.

Name	Description
files/per-object/object.go	Template for the main object generation. The filename generated by this emplate is special -- the entire file name (the portion for object.go) is replaced with the name of the object
files/per-run/sketch.go	Template for common code between all generate objects

Name	Description
object/builder	Template for the biulder part of the object
object/header	Template for the header part of the object, including the top comment, package name, imports
object/footer	Template for the footer part of the object
object/struct	Template for the struct definition of the object

Optional Templates

Optional templates are only rendered if the user provides them (you can think of them as hooks).

If you would like to augment the generated template, choose an appropriate optional template, and declare a template block by that name.

The following is the list of template block names that you may provide.

Name	Description
ext/object/header	User specified template to insert code at the beginning of the object.go code
ext/object/footer	User specified template to insert code at the end of the object.go code

Tips and Tricks

Managing JSON and Go Representations

You may find yourself in a situation where the JSON representation of a vield and the Go representation do not quite line up. For example, the JSON representation may be using an epoch time (i.e. int), but you want your Go consumers to work with time.Time objects.

In such cases, consider creating a type definition that implements the GetValue and AcceptValue methods:

type EpochTime struct {
  time.Time
}

func (t *EpochTime) AcceptValue(v interface{}) error {
  // Note: in reality you may have to work with float32, json.Number, etc
  // this code assumes that json.Unmarshal(...) of the piece of data
  // results in an `int`
  switch v := v.(type) {
  case int:
    t.Time = time.Unix(v, 0)
  default:
    return fmt.Errorf(`expected int (got %T)`, v)
  }
}

func (t *EpochTime) GetValue() interface{} {
  // Note: in reality you may have to handle cases where t == nil
  return t.Time
}

Then use this time as your field type:

func (Schema) Fields() []*schema.FieldSpec {
  // Note: if you can import the type in the schema file, then
  // AcceptValue/GetValue will be computed automatically.
  // i.e. schema.Type(EpochTime{})
  epochtype := schema.TypeName(`EpochTime`).
    AcceptValue(true).
    GetValue(true))
  return []*schema.FieldSpec{
    schema.NewField(`epoch`, epochtype),
    ...
  }
}

Command Line

Name	Description
--dst-dir=DIR	Specify the directory to write the generate files to
--exclude=PATTERN	Specify a pattern to exclude. value may be a RE2 compatible regular expression. May be specified multiple times
--tmpl-dir=DIR	Specify a template directory provided by the user. May be specified multiple times
--var=NAME=VALUE / --var=NAME=VALUE:TYPE	Specify a variable to be passed to the template as key/value pair. May optionally be followed by a type name: e.g. --var=foo=true:bool would store the value for foo as a Go bool instead of a string. Currently only supports string, bool, and int
--verbose	Enable verbose logging