fix(deps): update dependency tiktoken to v1.0.16

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
tiktoken	1.0.15 -> 1.0.16

---

Release Notes

dqbd/tiktoken (tiktoken)

v1.0.16

Compare Source

Patch Changes

• 4796f27: Add support for gpt-4o, gpt-4o-mini, text-embedding-3-small and text-embedding-3-large

---

Configuration

📅 Schedule: Branch creation - "* 0-4 * * 3" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[github-actions]

bedrock debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This PR updates the tiktoken library to add support for new GPT-4 and text embedding models. It includes changes to both the JavaScript and WebAssembly implementations, as well as updates to version numbers and changelogs.

Changes

Changes

1. js/CHANGELOG.md:

   • Added entry for version 1.0.13, noting support for new models.

2. js/package.json:

   • Updated version to 1.0.13.

3. js/src/core.ts:

   • Added support for new models in the getEncodingNameForModel function:
     • "text-embedding-3-small" and "text-embedding-3-large" use "cl100k_base" encoding.
     • "gpt-4o-2024-08-06", "gpt-4o-mini-2024-07-18", and "gpt-4o-mini" use "o200k_base" encoding.

4. tiktoken/model_to_encoding.json:

   • Added entries for new models and their corresponding encodings.

5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md:

   • Added entries for version 1.0.16, noting support for new models.

6. wasm/alias/package.json and wasm/package.json:

   • Updated version to 1.0.16.

7. wasm/src/lib.rs:

   • Updated the TiktokenModel type to include new model names.
   • Added support for new models in the encoding_for_model function.
   • Removed the "davinci" model case, which was likely outdated.

These changes ensure that the tiktoken library can properly handle tokenization for the newly added GPT-4 and text embedding models, maintaining compatibility with the latest OpenAI offerings.

[github-actions]

openai debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This pull request adds support for multiple new models, specifically gpt-4o, gpt-4o-mini, text-embedding-3-small, and text-embedding-3-large. The PR updates the version numbers accordingly across different package manifests and adds the necessary mappings in both JSON and TypeScript files to handle these new models. The changes primarily aim to enhance the codebase's capability to support these newly introduced models.

Security Hotspots

None

Changes

Changes

• js/CHANGELOG.md

  • Added a new entry for version 1.0.13 noting the addition of new models.

• js/package.json

  • Bumped version from 1.0.12 to 1.0.13.

• js/src/core.ts

  • Added support for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.

• tiktoken/model_to_encoding.json

  • Added encoding mappings for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.

• wasm/CHANGELOG.md

  • Added a new entry for version 1.0.16 noting the addition of new models.

• wasm/alias/CHANGELOG.md

  • Added a new entry for version 1.0.16 noting the addition of new models.

• wasm/alias/package.json

  • Bumped version from 1.0.15 to 1.0.16.

• wasm/package.json

  • Bumped version from 1.0.15 to 1.0.16.

• wasm/src/lib.rs

  • Added support for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.

[github-actions]

anthropic debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This PR updates the tiktoken library to support new OpenAI models: gpt-4o, gpt-4o-mini, text-embedding-3-small, and text-embedding-3-large. It includes changes to the JavaScript, TypeScript, and Rust implementations, as well as updates to version numbers and changelogs.

Changes

Changes

1. js/CHANGELOG.md:

   • Added entry for version 1.0.13, noting support for new models.

2. js/package.json:

   • Bumped version from 1.0.12 to 1.0.13.

3. js/src/core.ts:

   • Updated getEncodingNameForModel function to include new models:
     • Added text-embedding-3-small and text-embedding-3-large to use cl100k_base encoding.
     • Added gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18, and gpt-4o-mini to use o200k_base encoding.

4. tiktoken/model_to_encoding.json:

   • Added entries for new models and their corresponding encodings.

5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md:

   • Added entries for version 1.0.16, noting support for new models.

6. wasm/alias/package.json and wasm/package.json:

   • Bumped version from 1.0.15 to 1.0.16.

7. wasm/src/lib.rs:

   • Updated TiktokenModel type to include new model names.
   • Modified encoding_for_model function to support new models:
     • Added cases for text-embedding-3-small and text-embedding-3-large using cl100k_base encoding.
     • Added cases for gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18, and gpt-4o-mini using o200k_base encoding.
   • Removed the case for "davinci" model, which was redundant as it used the same encoding as other existing models.

These changes ensure that the tiktoken library can properly handle tokenization for the newly supported OpenAI models.