System tests with a real AreaDetector IOC

[callumforrester]

This issue captures a discussion with @jwlodek @mrakitin @evalott100 @prjemian and others.

Despite our best efforts, updates to ophyd-async are still reasonably likely to break for someone somewhere, primarily because IOCs are fiddly, idiosyncratic, have no defined schemas and can vary per facility. @coretl actually identified this as a major risk to the project in a review a few weeks ago. This brittleness is particularly evident with AreaDetectors, so if we're going to put any work into mitigating this problem, that would be the low-hanging fruit to start with.

@jwlodek's proposal was a set of system tests that run in CI against a containerised simulated areadetector. @gilesknap could probably comment on how easy such a thing would be to set up. The main flaw is that it would only pick up problems with ADCore, and not any specific detectors, but it would be a start.

Comment if I've missed anything :)

[gilesknap]

If you would like to run a set of tests against a SimDetector then the container already exists here:
ghcr.io/epics-containers/ioc-adsimdetector-runtime:2024.8.2

Source for the container is at:
https://github.com/epics-containers/ioc-adsimdetector

We could easily spin up an instance of this in GitHub actions and run some tests against it. I could help with getting that working.

[coretl]

Discussed this with @DominicOram in relation to an Odin system test. To summarize:

• We like the idea of separating out tests/ and system-tests/ so the existing test suite isn't burdensome to run
• Anything in system-tests/ will have dependent services like the EPICS containerised IOC
• These services should be run up in the same way as @garryod does for his fake databases, can't remember if it was docker compose or a GitHub actions construct

[DominicOram]

Yep, Garry pointed me at the repos. I will do some of this as part of DiamondLightSource/dodal#700 but will probably spin out a new issue for the CI

[jwlodek]

I like the idea of a separate system_tests that have some more rigorous "real" tests with external service dependencies. Something we can run before releases to better catch issues without compromising the simple/fast current suite of tests.

[callumforrester]

@jwlodek agreed but I think running it just around releases is not often enough. Given containerised IOCs etc. there's no reason not to also run it in CI for at least every PR.