Regression Testing Expectations

[JacksonBurns]

Regression Testing

As of right now, the regression tests are failing by default. See this comment for both the failure, and an initial reaction:

          > Not related to this PR but this is the first PR where the regression tests have had a fair chance to run. The tests failed because one edge reaction was different. From the failed edge comparison:

That result suggests a bug to me...one that may be quite complicated, but probably shouldn't be discussed here in this PR.

Originally posted by @mjohnson541 in #2474 (comment)

Let's discuss what suggests bugs, what should constitute a failure, and how much reproducibility we expect from RMG runs.

[mjohnson541]

When any react generates CC[CH]OO when constructing the reaction the radical decay framework should convert CC[CH]OO to CCC=O + [OH] since it matches the template for a radical decay recipe.

[JacksonBurns]

Is it possible then that, when CC[CH]OO was generated nothing was moved to the core that iteration, so the simulation stopped before the next reaction could be added?

[mjohnson541]

I don't understand, isn't the text indicating it was in the model (which I assume means the core)?

It's possible that the check only occurs when adding to the core (I don't think that's the case though), but regardless of that you would expect consistency in the reaction whether both were in the core or edge.

[JacksonBurns]

Notes from a developer meeting - the regression tests should 'pass' (just in the sense that we get a green checkmark from GitHub actions) if they successfully execute. The results of the actual regression comparisons should be commented onto the PR by a bot in a dropdown so that reviewers can look at the results.