Example for using control_symbols (sentencepiece) for ASR? #7205
Replies: 2 comments 2 replies
-
|
Nemo ASR does not currently support user defined symbols. We could add support for it similar to how it's done for NeMo NLP, but that will take time to support. |
Beta Was this translation helpful? Give feedback.
-
|
Basically we have something called Sentencepiece legacy mode which separately tracks a set of user defined tokens but ASR does not allow that because it makes export of models difficult. |
Beta Was this translation helpful? Give feedback.
-
|
Oh really? I had just modified process_asr_text_tokenizer.py to add user defined tokens and it works fine. I'm just not sure if I can add control symbols. |
Beta Was this translation helpful? Give feedback.
-
|
Yep that can be done at construction time of Tokenizers, but not afterwords. |
Beta Was this translation helpful? Give feedback.
-
I understood that user defined symbols can be encoded by text, but what do we need to do to use control symbols? How do we add them manually if not directly in the text?
I'm trying to represent some noise tokens in the text, ideally I want it to be learned in training, but not be outputted, like in Kaldi.
Beta Was this translation helpful? Give feedback.
All reactions