json

Yet another json library. It's created to process unstructured json in a convenient and efficient way.

There is also some set of jq filters implemented on top of json.Decoder.

json usage

Decoder is stateless. Most of the methods take source buffer and index where to start parsing and return a result and index where they stopped parsing.

None of methods make a copy or allocate except these which take destination buffer in arguments.

The code is from examples.

// Parsing single object.

var d json.Decoder
data := []byte(`{"key": "value", "another": 1234}`)

i := 0 // initial position
i, err := d.Enter(data, i, json.Object)
if err != nil {
	// not an object
}

var key []byte // to not to shadow i and err in a loop

// extracted values
var value, another []byte

for d.ForMore(data, &i, json.Object, &err) {
	key, i, err = d.Key(data, i) // key decodes a string but don't decode '\n', '\"', '\xXX' and others
	if err != nil {
		// ...
	}

	switch string(key) {
	case "key":
		value, i, err = d.DecodeString(data, i, value[:0]) // reuse value buffer if we are in a loop or something
	case "another":
		another, i, err = d.Raw(data, i)
	default: // skip additional keys
		i, err = d.Skip(data, i)
	}

	// check error for all switch cases
	if err != nil {
		// ...
	}
}
if err != nil {
	// ForMore error
}

// Parsing jsonl: newline (or space, or comma) delimited values.

var err error // to not to shadow i in a loop
var d json.Decoder
data := []byte(`"a", 2 3
["array"]
`)

for i := d.SkipSpaces(data, 0); i < len(data); i = d.SkipSpaces(data, i) { // eat spaces and not try to read the value from string "\n"
	i, err = processOneObject(data, i) // do not use := here as it shadows i and loop will restart from the same index
	if err != nil {
		// ...
	}
}

jq usage

Deprecated in favour of (nikand.dev/go/jq)[https://pkg.go.dev/nikand.dev/go/jq].

The advantage of this implementation is that filters are stateless so they can be used by multiple goroutines at once. The rest are disadvantages: more complicated code -> less reliable, supports only json, less efficient, fewer filters implemented.

jq package is a set of Filters that take data from one buffer, process it, and append result to another buffer.

Also there is a state taken and returned. It's used by filters to return multiple values one by one. The caller must provide nil on the first iteration and returned state on the rest of iterations. Iteration must stop when returned state is nil. Filter may or may not add a value to dst buffer. Empty filter for example adds no value and returns nil state.

Destination buffer is returned even in case of error. This is mostly done for avoiding allocs in case the buffer was grown but error happened.

The code is from examples.

// Extract some inside value.

data := []byte(`{"key0":"skip it", "key1": {"next_key": ["array", null, {"obj":"val"}, "trailing element"]}}  "next"`)

f := jq.Query{"key1", "next_key", 2} // string keys and int array indexes are supported

var res []byte // reusable buffer
var i int      // start index

// Most filters only parse single value and return index where the value ended.
// Use jq.ApplyToAll(f, res[:0], data, 0, []byte("\n")) to process all values in a buffer.
res, i, _, err := f.Next(res[:0], data, i, nil)
if err != nil {
	// i is an index in a source buffer where the error occurred.
}

fmt.Printf("value: %s\n", res)
fmt.Printf("final position: %d of %d\n", i, len(data)) // filter only parsed first value in the buffer
_ = i < len(data)                                      // and stopped immideately after it

// Output:
// value: {"obj":"val"}
// final position: 92 of 100

This is especially convenient if you need to extract a value from json inside base64 inside json. Yes, I've seen such cases and this is how this library came to life.

// generated by command
// jq -nc '{key3: "value"} | {key2: (. | tojson)} | @base64 | {key1: .}'
data := []byte(`{"key1":"eyJrZXkyIjoie1wia2V5M1wiOlwidmFsdWVcIn0ifQ=="}`)

f := jq.NewPipe(
	jq.Key("key1"),
	&jq.Base64d{
		Encoding: base64.StdEncoding,
	},
	&jq.JSONDecoder{},
	jq.Key("key2"),
	&jq.JSONDecoder{},
	jq.Key("key3"),
)

res, _, _, err := f.Next(nil, data, 0, nil)
if err != nil {
	panic(err)
}

// res is []byte(`"value"`)