Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logged spec now includes overridden args #1460

Merged
merged 1 commit into from
Jan 26, 2024

Conversation

ojaffe
Copy link
Contributor

@ojaffe ojaffe commented Jan 17, 2024

Using --extra_eval_params will override args of the same name specified in the eval .yaml, but the updated values will not be logged in the spec, the original values will be logged instead. This PR fixes this problem; we just update eval_spec.args with the new values.

e.g. running oaieval dummy make-me-pay --extra_eval_params turn_cap=1 previously lead to "turn_cap": 5 being logged in the spec, since this is the default value.
In this branch, running the same command leads to "turn_cap": 1 being logged in the spec.

Copy link
Collaborator

@JunShern JunShern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ojaffe! Agree that this is more intuitive behavior for users; have checked that this works as advertised.

@JunShern JunShern merged commit 3040d6f into openai:main Jan 26, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants