Nipah Tokenizer

Installing

dotnet add package NipahTokenizer

Using

First setup the tokenizer options as needed, you can use the default as follows:

var tokenizerOptions = TokenizerOptions.Default;

Then just create a new instance of tokenizer and call "Tokenize" passing tokenizer options as arg:

var tokenizer = new Tokenizer();
var tokens = tokenizer.Tokenize("any-text-here", tokenizerOptions);

You can now just iterate over tokens as needed.