Enhance regression tests.

[tobiasraabe]

• Date you used respy: 19.11.2019
• respy version used, if any: 2.0.0
• Python version, if any: any
• Operating System: any

Just collecting. No immediate discussion necessary.

Description

Over time, we collected some notes on how to improve regression tests in the future. This issue collects all the notes for an discussion.

[tobiasraabe]

@janosg in #274 (comment)

I think we could improve regression tests in general. I have the following suggestions:

• Evaluate several parameter vectors for each criterion function. This way we can save setup costs which are currently probably the main cost of regression tests with more periods
• Save some intermediate values. At least the dataset, but possibly more. We should also have asserts on those intermediate steps. This makes debugging much easier.
• Start to write regression tests for individual components. For example, there should be regression tests just for the state space creation. When testing such small components it's easier to reason about edge cases and reach a large coverage with few well designed cases.