feat: implement asynchronous token counting in GPT2Tokenizer#12239

Merged

laipz8200 merged 2 commits intomainfrom feat/implement-asynchronous-token-counting-in-GPT2Tokenizer

Dec 31, 2024

+6-2

Commits on Dec 30, 2024

feat: implement asynchronous token counting in GPT2Tokenizer
laipz8200
committed

Commits on Dec 31, 2024

feat: replace ProcessPoolExecutor with gevent threadpool for token counting
laipz8200
committed