Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quest: Add Benchmarks and Pretokenized JSON for training #222

Open
gkielian opened this issue Aug 9, 2024 · 0 comments
Open

Quest: Add Benchmarks and Pretokenized JSON for training #222

gkielian opened this issue Aug 9, 2024 · 0 comments
Labels
epic Epic level contrib

Comments

@gkielian
Copy link
Collaborator

gkielian commented Aug 9, 2024

Benchmarks

This is going to be a huge contribution.

Currently the get_batch function currently performs fast inference and tests for validation loss.

Create a new get_batch specifically for benchmarking.

This would utilize a json file (pretokenized), and has "question:" "answer:" for fields.

Most of the time we'd suspect just a single multiple choice answer (e.g. MMLU Pro), or a set of correct answers, and we can see 1 or 0 if the network has the correct answer from top 1 logit top 2 logits , etc.

@gkielian gkielian added the epic Epic level contrib label Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic Epic level contrib
Projects
None yet
Development

No branches or pull requests

1 participant