Skip to content

Commit

Permalink
Allow for evals with no args (#1517)
Browse files Browse the repository at this point in the history
As raised in #1515, the `args` field of `EvalSpec` is optional.
Therefore it is possible for evals with no args to exist. Here `args` is
`None`.

However, currently our [arg overriding
code](https://github.com/openai/evals/blame/main/evals/cli/oaieval.py#L158)
mistakingly does not support this API, since it assumes `args` is not
`None`.

This PR addresses the issue with an if statement. Fixes #1515
  • Loading branch information
thesofakillers authored Apr 5, 2024
1 parent 20de8c5 commit 4ed2f6f
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion evals/cli/oaieval.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,11 @@ def to_number(x: str) -> Union[int, float, str]:
return {k: to_number(v) for k, v in str_dict.items()}

extra_eval_params = parse_extra_eval_params(args.extra_eval_params)
eval_spec.args.update(extra_eval_params)

if eval_spec.args is None:
eval_spec.args = extra_eval_params
else:
eval_spec.args.update(extra_eval_params)

# If the user provided an argument to --completion_args, parse it into a dict here, to be passed to the completion_fn creation **kwargs
completion_args = args.completion_args.split(",")
Expand Down

0 comments on commit 4ed2f6f

Please sign in to comment.