Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(deps): update dependency tiktoken to v1.0.16 #170

Merged
merged 1 commit into from
Aug 21, 2024

Conversation

renovate[bot]
Copy link
Contributor

@renovate renovate bot commented Aug 21, 2024

Mend Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
tiktoken 1.0.15 -> 1.0.16 age adoption passing confidence

Release Notes

dqbd/tiktoken (tiktoken)

v1.0.16

Compare Source

Patch Changes
  • 4796f27: Add support for gpt-4o, gpt-4o-mini, text-embedding-3-small and text-embedding-3-large

Configuration

📅 Schedule: Branch creation - "* 0-4 * * 3" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

Copy link

bedrock debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This PR updates the tiktoken library to add support for new GPT-4 and text embedding models. It includes changes to both the JavaScript and WebAssembly implementations, as well as updates to version numbers and changelogs.

Changes

Changes

  1. js/CHANGELOG.md:

    • Added entry for version 1.0.13, noting support for new models.
  2. js/package.json:

    • Updated version to 1.0.13.
  3. js/src/core.ts:

    • Added support for new models in the getEncodingNameForModel function:
      • "text-embedding-3-small" and "text-embedding-3-large" use "cl100k_base" encoding.
      • "gpt-4o-2024-08-06", "gpt-4o-mini-2024-07-18", and "gpt-4o-mini" use "o200k_base" encoding.
  4. tiktoken/model_to_encoding.json:

    • Added entries for new models and their corresponding encodings.
  5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md:

    • Added entries for version 1.0.16, noting support for new models.
  6. wasm/alias/package.json and wasm/package.json:

    • Updated version to 1.0.16.
  7. wasm/src/lib.rs:

    • Updated the TiktokenModel type to include new model names.
    • Added support for new models in the encoding_for_model function.
    • Removed the "davinci" model case, which was likely outdated.

These changes ensure that the tiktoken library can properly handle tokenization for the newly added GPT-4 and text embedding models, maintaining compatibility with the latest OpenAI offerings.

Copy link

openai debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This pull request adds support for multiple new models, specifically gpt-4o, gpt-4o-mini, text-embedding-3-small, and text-embedding-3-large. The PR updates the version numbers accordingly across different package manifests and adds the necessary mappings in both JSON and TypeScript files to handle these new models. The changes primarily aim to enhance the codebase's capability to support these newly introduced models.

Security Hotspots

None

Changes

Changes

  • js/CHANGELOG.md

    • Added a new entry for version 1.0.13 noting the addition of new models.
  • js/package.json

    • Bumped version from 1.0.12 to 1.0.13.
  • js/src/core.ts

    • Added support for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.
  • tiktoken/model_to_encoding.json

    • Added encoding mappings for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.
  • wasm/CHANGELOG.md

    • Added a new entry for version 1.0.16 noting the addition of new models.
  • wasm/alias/CHANGELOG.md

    • Added a new entry for version 1.0.16 noting the addition of new models.
  • wasm/alias/package.json

    • Bumped version from 1.0.15 to 1.0.16.
  • wasm/package.json

    • Bumped version from 1.0.15 to 1.0.16.
  • wasm/src/lib.rs

    • Added support for text-embedding-3-small, text-embedding-3-large, gpt-4o, gpt-4o-2024-08-06, gpt-4o-mini, and gpt-4o-mini-2024-07-18.

Copy link

anthropic debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/tiktoken@1.0.15..@dqbd/tiktoken@1.0.16

Description

This PR updates the tiktoken library to support new OpenAI models: gpt-4o, gpt-4o-mini, text-embedding-3-small, and text-embedding-3-large. It includes changes to the JavaScript, TypeScript, and Rust implementations, as well as updates to version numbers and changelogs.

Changes

Changes

  1. js/CHANGELOG.md:

    • Added entry for version 1.0.13, noting support for new models.
  2. js/package.json:

    • Bumped version from 1.0.12 to 1.0.13.
  3. js/src/core.ts:

    • Updated getEncodingNameForModel function to include new models:
      • Added text-embedding-3-small and text-embedding-3-large to use cl100k_base encoding.
      • Added gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18, and gpt-4o-mini to use o200k_base encoding.
  4. tiktoken/model_to_encoding.json:

    • Added entries for new models and their corresponding encodings.
  5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md:

    • Added entries for version 1.0.16, noting support for new models.
  6. wasm/alias/package.json and wasm/package.json:

    • Bumped version from 1.0.15 to 1.0.16.
  7. wasm/src/lib.rs:

    • Updated TiktokenModel type to include new model names.
    • Modified encoding_for_model function to support new models:
      • Added cases for text-embedding-3-small and text-embedding-3-large using cl100k_base encoding.
      • Added cases for gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18, and gpt-4o-mini using o200k_base encoding.
    • Removed the case for "davinci" model, which was redundant as it used the same encoding as other existing models.

These changes ensure that the tiktoken library can properly handle tokenization for the newly supported OpenAI models.

@thypon thypon merged commit cbfca6b into main Aug 21, 2024
8 checks passed
@thypon thypon deleted the renovate/tiktoken-1.x-lockfile branch August 21, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant