You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think this is pretty easy to do: given a tokenizer, you can compute the bits-per-token as len(tok.encode(utf-8)) then you can get the num tokens per example then compute bpb as - sum(loss) / sum(bits-per-token x token) / ln(2)
The text was updated successfully, but these errors were encountered:
I think this is pretty easy to do: given a tokenizer, you can compute the bits-per-token as len(tok.encode(utf-8)) then you can get the num tokens per example then compute bpb as
- sum(loss) / sum(bits-per-token x token) / ln(2)
The text was updated successfully, but these errors were encountered: