Support more languages

[SSamDav]

Do you plan to support more languages?

Also it seems some of the optimizations where made with the western languages in mind. Do you plan to explore how this tokenizer works with non western languages?

[alasdairforsythe]

TokenMonster is not language dependent. It will work with any language that uses a space as a word boundary (non-standard spaces are also fine). It'll also work with languages that don't use space as word boundaries, but you'll need to use unfiltered optimization mode in those cases.

[Sovenok-Hacker]

I trained a model for English-Russian, it can be downloaded from IPFS: https://ipfs.io/ipfs/QmPhxHrNyogBnzxY5onAnkvvgg78RP26R1XXrv3Ka6Qc9J?filename=russian.vocab

[NotSpooky]

Could you please give me a small introduction on how to train it for another dataset?

[Sovenok-Hacker]

Here is the guide how to do it: https://github.com/alasdairforsythe/tokenmonster/tree/main/training
You need 100MB+ text in one .txt file and around 36 hours on a modern pc (I trained it on Intel Core i3-10100 for ~24 hours)