Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update tokenizer dependency to new minor and chain changes #62

Merged
merged 3 commits into from
Oct 2, 2024
Merged

Update tokenizer dependency to new minor and chain changes #62

merged 3 commits into from
Oct 2, 2024

Conversation

Virviil
Copy link
Contributor

@Virviil Virviil commented Oct 2, 2024 •
edited
Loading

Changelog:

  • Update tokenizers from .19 to .20
    • with_... functions in Rust braking change for tokenizer changed, without affecting Elixir's API
    • Adding new ByteLevel normalizer
  • Bumping all Rust and Elixir dependencies - without visible changes except:
    • Refactor deprecated resource initialization flow in rustler`
  • Sync rustler's native lib version with elixir's mix version

@Virviil
Copy link
Contributor Author

Virviil commented Oct 2, 2024

@jonatanklosko @josevalim

Location of alphabet function is discussible.

Please review

jonatanklosko
jonatanklosko reviewed Oct 2, 2024
View reviewed changes
mix.exs Outdated Show resolved Hide resolved
@jonatanklosko
Copy link
Member

Thanks :)

@josevalim josevalim merged commit b3cd84b into elixir-nx:main Oct 2, 2024
3 checks passed
@josevalim
Copy link
Contributor

💚 💙 💜 💛 ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonatanklosko jonatanklosko jonatanklosko approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Virviil @jonatanklosko @josevalim