-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use llm-foundry for ICL metrics #287
Conversation
thanks for the PR! Any reason for removing the InContextLearningQAAccuracy and InContextLearningCodeEvalAccuracy in this PR? |
those metrics aren't in foundry, could add to the foundry wrapper, or fix composer to a version prior to deprecation |
ah interesting, thanks. it looks like InContextLearningQAAccuracy was renamed (mosaicml/llm-foundry#936) to InContextLearningGenerationExactMatchAccuracy. The code eval was deprecated mosaicml/llm-foundry@b301869. Could you update this to replace InContextLearningQAAccuracy with InContextLearningGenerationExactMatchAccuracy, and remove the commented line for code evals? Thanks! |
…ExactMatchAccuracy, and removed the commented line for code evals
updated! |
composer deprecated support for ICL metrics, should now be using llm-foundry instead