Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Acceptance testing framework #139

Merged
merged 9 commits into from
Nov 15, 2024
Merged

Acceptance testing framework #139

merged 9 commits into from
Nov 15, 2024

Conversation

joanlopez
Copy link
Member

@joanlopez joanlopez commented May 12, 2023

TL;DR - It sets up a basic acceptance testing framework for Killgrave.

Here's some further information:

Requirements/assumptions

  • It requires/uses the acceptance Go build flag to avoid being executed with other tests (e.g. unit or integration tests).
  • Assumes the binary killgrave exists in the local bin/ directory at testing time.
    • So, you need to compile the app before running these tests, and run them as a black box
      • To simplify that, I've added an acceptance make target that does both.
  • Relies on txtar files to simplify test cases definition.
    • In the future, we might explore the idea of reusing those, with an in-memory file system, adequately injected into the app.Run() function, so we can also use the same framework and test cases for more like integration tests.

The main testing algorithm is:

  • For every directory under acceptance/tests:
    • Creates a temporary directory with the files defined at config.txtar (imposters, schemas, responses, etc)
    • Runs Killgrave, at any available, randomly assigned port, specifying the former as the imposters directory
    • For every .txtar file on http directory:
      • It performs the request defined in req.http
      • Asserts the response against the file defined in res.http.

So, it's pretty easy to define multiple acceptance/end-2-end tests for different imposters.


Once merged, I'd suggest to consider adding one extra test to this suite as part of the definition of done (DoD) for all the remaining PRs, and for the new ones, unless completely unrelated.

Also, in the near future, we could extend this kind of framework to also assert the Killgrave's output.


Thanks!


PS: I merged #173 because it was a requirement to have ✅ checks here, and the only suggestion I left unresolved was adding more coverage with tests, which is indirectly covered by the changes present here.

@joanlopez joanlopez added enhancement New feature or request improvement labels May 12, 2023
@joanlopez joanlopez requested a review from aperezg May 12, 2023 01:30
@joanlopez joanlopez self-assigned this May 12, 2023
@joanlopez joanlopez mentioned this pull request Nov 10, 2024
@joanlopez joanlopez changed the title Baseline for acceptance tests Acceptance test framework Nov 10, 2024
@joanlopez joanlopez changed the title Acceptance test framework Acceptance testing framework Nov 10, 2024
@joanlopez joanlopez mentioned this pull request Nov 13, 2024
@joanlopez joanlopez merged commit d41b4b1 into main Nov 15, 2024
4 checks passed
@joanlopez joanlopez deleted the acceptance branch November 15, 2024 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants