Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(chat): fix tokenizer issue (Issue #2753) #2763

Merged
merged 11 commits into from
Dec 11, 2024
Merged

Conversation

Gimir
Copy link
Contributor

@Gimir Gimir commented Dec 9, 2024

Description:

  • add js-tiktoken package (deduplication made)
  • replace getEncoding method which was using wasm with browser supported method from js-tiktoken

Issues:

Checklist:

  • the pull request name complies with Conventional Commits
  • the pull request name starts with fix(<scope>):, feat(<scope>):, feature(<scope>):, chore(<scope>):, hotfix(<scope>): or e2e(<scope>):. If contains breaking changes then the pull request name must start with fix(<scope>)!:, feat(<scope>)!:, feature(<scope>)!:, chore(<scope>)!:, hotfix(<scope>)!: or e2e(<scope>)!: where <scope> is name of affected project: chat, chat-e2e, overlay, shared, sandbox-overlay, etc.
  • the pull request name ends with (Issue #<TICKET_ID>) (comma-separated list of issues)
  • I confirm that do not share any confidential information like API keys or any other secrets and private URLs

@Gimir Gimir added bug Something isn't working _priority_1 labels Dec 9, 2024
@Gimir Gimir self-assigned this Dec 9, 2024
@Gimir
Copy link
Contributor Author

Gimir commented Dec 9, 2024

/deploy-review

Environment URL: https://chat-ai-dial-chat-pr-2763.nightly-test.deltixhub.io
E2E tests status: success

IlyaBondar
IlyaBondar previously approved these changes Dec 11, 2024
@IlyaBondar
Copy link
Collaborator

IlyaBondar commented Dec 11, 2024

/deploy-review

Environment URL: https://chat-ai-dial-chat-pr-2763.nightly-test.deltixhub.io
E2E tests status: failed

@Gimir
Copy link
Contributor Author

Gimir commented Dec 11, 2024

/deploy-review

Environment URL: https://chat-ai-dial-chat-pr-2763.nightly-test.deltixhub.io
E2E tests status: success

@Gimir Gimir enabled auto-merge (squash) December 11, 2024 13:23
@Gimir Gimir merged commit 673129d into development Dec 11, 2024
8 checks passed
@Gimir Gimir deleted the fix/2753-error-reloads branch December 11, 2024 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working _priority_1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants