Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve evaluation model params #990

Merged
merged 3 commits into from
Nov 1, 2024

Conversation

lu-ohai
Copy link
Member

@lu-ohai lu-ohai commented Oct 31, 2024

Improve evaluation model params

  • Made all entities in evaluation as pydantic
  • Extended AquaEvalParams to accept all model parameters from customer
  • Reverted PR to remove this logic to service container side.

Notebook

  • Create evaluation with model parameters
Screenshot 2024-10-31 at 1 02 08 PM Screenshot 2024-10-31 at 12 49 06 PM Screenshot 2024-10-31 at 12 49 18 PM

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Oct 31, 2024
Copy link

📌 Cov diff with main:

Coverage-93%

📌 Overall coverage:

Coverage-58.61%

@@ -164,7 +161,7 @@ def create(
raise AquaValueError(
"Invalid create evaluation parameters. "
"Allowable parameters are: "
f"{', '.join([field.name for field in fields(CreateAquaEvaluationDetails)])}."
f"{', '.join([field for field in CreateAquaEvaluationDetails.model_fields])}."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pydantic by itself can give meaningful message if something wrong with the input params. We probably can have something like

except Exception as ex:
                raise AquaValueError(
                    "Invalid create evaluation parameters. "
                    f"Details: {ex}"

Can you check this?

In the evaluation framework i did this:

except ValidationError as err:
        custom_errors = {
            ".".join(map(str, e["loc"])): e["msg"] for e in json.loads(err.json())
        }
        raise InvalidEvaluationConfigValidationError(
            evaluation_config=config, errors=custom_errors
        )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just updated. Below are the results:

  • Missing parameters
Screenshot 2024-10-31 at 3 04 13 PM
  • Parameters with wrong type
Screenshot 2024-10-31 at 3 05 10 PM

Copy link

📌 Cov diff with main:

Coverage-91%

📌 Overall coverage:

Coverage-58.61%

@lu-ohai lu-ohai merged commit 4607e07 into main Nov 1, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants