Support other encodings, like cl100k_base

[leafduo]

I'm making an application which calls ChatGPT API in Go, and I need to estimate how many tokens I'm using. ChatGPT uses cl100k_base encoding, while go-gpt-3-encoder only supports (I think) p50k_base.

Is there any plan to support cl100k_base?

I don't really know how BPE works, but according to this, different encodings have different vocabulary, pat regex and special tokens.

[j178]

I made a Go binding for Rust implemetation of tiktoken, maybe you're interested: https://github.com/j178/tiktoken-go