Update tokenizers to 0.15.0 #55
Conversation
cigrainger
force-pushed
the
cg/upgrade-tokenizers
branch
from
a56bd16
to
e228f7d
Compare
|
Okay @jonatanklosko exposed the option <3 |
|
|
||
| let byte_fallback = match options | ||
| .iter() | ||
| .find(|opt| matches!(opt, UnigramOption::ByteFallback(_))) |
There was a problem hiding this comment.
iter . find on each opt is
Probably it's better to find another solution.
There was a problem hiding this comment.
With two options it's fine, but we could also do the same as here:
tokenizers/native/ex_tokenizers/src/models.rs
Lines 116 to 125 in 170ceac
There was a problem hiding this comment.
Yeah I mean we're talking about an incredibly negligible bit of performance, but yeah mutating an opts struct is probably more performant :).
| _ => None, | ||
| }; | ||
|
|
||
| let byte_fallback = match options |
There was a problem hiding this comment.
Are we matching the same 2 times (here and inside matches!)? Is it compiler optimised?
There was a problem hiding this comment.
Nah, the outer match is on the found inner match, which is on each element.
| @@ -125,7 +125,7 @@ fn apply_load_options(mut tokenizer: ExTokenizerImpl, options: Vec<LoadOption>) | |||
| } | |||
|
|
|||
| if opts.disable_truncation { | |||
| tokenizer.with_truncation(None); | |||
| tokenizer.with_truncation(None).unwrap(); | |||
There was a problem hiding this comment.
Is API changed to return error? Why? I'm not sure unwrap handles it properly (mb yes)
As on the tin. This gets us various improvements and bugfixes as detailed in the release notes.