You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
we had a nasty issue that the tests were not running correctly because the run_until function does not automatic cleanup.
My question is: Why?
Is this behaviour somewhere needed? IT seems very risky to have no automatic cleanup as the metric providers not being properly closed will cascade into other tests and makes this very hard to find.
@davidkopp Tagged you fyi. If you write a new test please use the cleanup guard clause for the moment as done in PR #616 . As you can see we are discussing internally if we fix that test suite design, as it is very unintuitive that a specifically written test case can destroy other test cases
The text was updated successfully, but these errors were encountered:
There is a specific reason for this, though I am very much open to feedback on better ways to do it as I agree that it is easy to break.
The point of the Tests.run_until funciton is that we want to instrument the GMT up until a certain point, then effectively pause it, and run some checks. Sometimes these checks need the containers to still be up and available for inspection, so cleanup cannot be run yet.
In order to be able to do this, I have split the instrumentation into two functions in TestFunctions - run_until and cleanup. You are not meant to use run_until without calling cleanup afterwards as well (I realize now this should be documented).
Because the implementation of what checks you want to make depends on the tests, I don't see a way to automatically trigger this cleanup.
I agree this is fragile and quite easy to break, especially if the Tests.cleanup functionality gets changed from the runner.cleanup functionality (the same is true for the run_until functionality though). I'm open to suggestions.
Hey @dan-mm
we had a nasty issue that the tests were not running correctly because the
run_until
function does not automatic cleanup.My question is: Why?
Is this behaviour somewhere needed? IT seems very risky to have no automatic cleanup as the metric providers not being properly closed will cascade into other tests and makes this very hard to find.
@davidkopp Tagged you fyi. If you write a new test please use the cleanup guard clause for the moment as done in PR #616 . As you can see we are discussing internally if we fix that test suite design, as it is very unintuitive that a specifically written test case can destroy other test cases
The text was updated successfully, but these errors were encountered: