How to train in a language which doesn't have a tokenizer?

[Adamz0ne]

I'm very new to training a tts model, sorry in advanced if I'm asking dumb questions.

Currently I'm working on a language in Taiwan called Taigi.
This is a language only used in a small region, so I don't have a lot of resourse to work with.

The language looks like this:
Tsit-tah ū tsi̍t ê tuā siong-tiûnn
And I run some script to normalize it to be this:
tsit1 tah4 u7 tsit8 e5 tua7 siong1 tiunn5

And I've prepared the json just fine, but the yaml config file needs a tokenizer which Nemo doesn't support this language.
How do I prepare my own tokenizer or other way to make this work.

Btw I did come across a post,
but the link provided does not very helpful.

[titu1994]

You can create your own tokenizer - see the tutorial https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb

[titu1994]

Plus i suggest dont normalize your text like that when computing the tokenizer. Let the tokenizer and model handle it, as much as they can and then if accuracy is bad, you can consider normalization.