Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add bpb metric #550

Open
dlwh opened this issue Apr 10, 2024 · 0 comments
Open

add bpb metric #550

dlwh opened this issue Apr 10, 2024 · 0 comments
Labels

Comments

@dlwh
Copy link
Member

dlwh commented Apr 10, 2024

I think this is pretty easy to do: given a tokenizer, you can compute the bits-per-token as len(tok.encode(utf-8)) then you can get the num tokens per example then compute bpb as - sum(loss) / sum(bits-per-token x token) / ln(2)

@dlwh dlwh added the markweb label Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant