Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterize and configure tests using YAML #23

Merged
merged 11 commits into from
Aug 29, 2023
Merged

Conversation

fivegrant
Copy link
Collaborator

This PR parameterizes test cases based off of different specified scenarios. For each scenario, the user must opt-in to each test using a config.yaml. Each test expects certain files to exist, therefore, a tests/resources.yaml is checked against for each enabled test for a scenario and raises an exception when a resource is missing.

@codecov
Copy link

codecov bot commented Aug 28, 2023

Codecov Report

Merging #23 (853e628) into main (2c32c25) will increase coverage by 1.84%.
Report is 6 commits behind head on main.
The diff coverage is 97.87%.

@@            Coverage Diff             @@
##             main      #23      +/-   ##
==========================================
+ Coverage   81.78%   83.62%   +1.84%     
==========================================
  Files           5        6       +1     
  Lines         505      519      +14     
==========================================
+ Hits          413      434      +21     
+ Misses         92       85       -7     
Files Changed Coverage Δ
api/utils.py 79.03% <85.71%> (+4.42%) ⬆️
api/server.py 95.45% <100.00%> (+4.68%) ⬆️
lib/settings.py 100.00% <100.00%> (ø)
worker/operations.py 75.21% <100.00%> (ø)
worker/utils.py 91.22% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@brandomr
Copy link
Contributor

brandomr commented Aug 28, 2023

Things to add:

  1. add AMR validation checking to code to amr and equation to amr
  2. update all amr.json examples to include the new header format
  3. update test_utils.py AMR class to pull schema information from the header
  4. add name and description to config.yaml so that we can add them to a test dashboard later
  5. basic documentation on how to add a scenario

Example code for implementing AMR schema validation utility:

result_amr = results.get("result", {}).get("job_result", {}).get("amr", None)
amr_instance = AMR(result_amr)
assert (
         amr_instance.is_valid()
        ), f"AMR failed to validate to its provided schema: {amr_instance.get_validation_error()}"

Up next (new PR I think):

  1. enhance documentation on how to add a scenario and run tests
  2. write out test results per test case and scenario in a structured manner such that we can populate a dashboard
  3. add qualitative test for AMR quality based on number of compartments and transitions in the produced AMR

e.g. (super notional)....

target.yaml

---
amr:
  - compartments: 5
  - transitions: 12

@fivegrant
Copy link
Collaborator Author

@brandomr Resolves 1, 2, 3, and 4. Let me know if 5 is already satisfied

@brandomr brandomr merged commit f38285b into main Aug 29, 2023
1 check passed
@brandomr brandomr deleted the feat/parameterize-tests branch August 29, 2023 21:31
@brandomr
Copy link
Contributor

This includes #25 so that PR can now be closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants