Count tokens with standalone tiktoken library

[PizBernina]

Would it yield in rather accurate results if I just use the tikoken library to calculate tokens for llama 3.1 & .2 ?

• Or should I use the the following tokenizer and adapt it? `https://github.com/meta-llama/llama3/blob/11817d47e1ba7a4959b025eb1ca308572e0e3963/llama/tokenizer.py#L20C1-L21C1
• Alternatively, I could also use `AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B") I guess?