v0.4.5

What's Changed

• fix: unicode splitting and add unicode test cases by @zurawiki in #24

Full Changelog: v0.4.4...v0.4.5

v0.4.4

What's Changed

• Add README.md to package by @fiji-flo in #22

New Contributors

• @fiji-flo made their first contribution in #22

Full Changelog: v0.4.3...v0.4.4

v0.4.3

New Features

Support for gpt-3.5-turbo-16k models

What's Changed

• chore(deps): update pyo3 requirement from 0.18.3 to 0.19.0 by @dependabot in #19
• chore(deps): update async-openai requirement from 0.10.3 to 0.11.0 by @dependabot in #21
• Add benchmark support by @zurawiki in #20

Full Changelog: v0.4.2...v0.4.3

v0.4.2

What's Changed

• Improved interface for split_by_token by @jackbackes in #18
• Update dependencies

Full Changelog: v0.4.1...v0.4.2

v0.4.1

What's Changed

• Add ability to tokenize a string and return the decoded tokens using the correct BPE model by @thejohnbackes in #17

New Contributors

• @thejohnbackes made their first contribution in #17

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• fix: expose num_tokens_from_messages and ensure its implementation is consistent with OpenAI by @j178 in #14
• feat: make async-openai optional by @j178 in #15

New Contributors

• @j178 made their first contribution in #14

Full Changelog: v0.3.3...v0.4.0

v0.3.3

What's Changed

• chore(deps): update async-openai requirement from 0.9.5 to 0.10.0 by @dependabot in #11
• chore: Update dependencies and versions by @zurawiki in #12

New Contributors

• @dependabot made their first contribution in #11

Full Changelog: v0.3.2...v0.3.3

v0.3.2

Full Changelog: v0.3.1...v0.3.2

v0.3.1

Full Changelog: v0.3.0...v0.3.1

v0.3.0

What's Changed

• feat: Add gpt-4 models mapping and tests. by @zurawiki in #9

Full Changelog: v0.2.2...v0.3.0