(Beginner) Importing never_split_tokens #189
-
|
When I run this code: So how to import it? |
Beta Was this translation helpful? Give feedback.
Answered by
WissamAntoun
Dec 22, 2022
Replies: 1 comment
-
|
this is because you are using a very old version of the training scripts , this has now been updated to from arabert.preprocess import NEVER_SPLIT_TOKENSYou also shouldn't need to import any of these tokens if you are doing fine-tuning with arabert |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
YomnaEid
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
this is because you are using a very old version of the training scripts , this has now been updated to
You also shouldn't need to import any of these tokens if you are doing fine-tuning with arabert