-
Notifications
You must be signed in to change notification settings - Fork 648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NFD form combining characters not picked up as part of word #1099
Comments
hey, good catch! Yeah, I agree that compromise should not tokenize these inline unicode forms. happy to add a guard for this, in the next release. |
hey, just double-checking something, your example Can you generate an example where the NFD character is more word-like? I agree it rubs-up against the javascript normalize feature, and maybe our supporting it would just complicate things. |
It's just the Combining Acute Accent:
shows
it's easy enough to normalize the input before passing it into tokenization, but that would then be a design constraint, and as mentioned, there are combining characters that have no single-char NFC form. |
logs
normalizing to NFC does work, but not every combining char combination has an NFC form (eg
'Poincare\u0301 E\u0300\u0304'.normalize('NFC')
)The text was updated successfully, but these errors were encountered: