[Discussion] How to avoid regression with real services? (integration tests but not only)

[stefanofornari]

Introduction

This discussion topic comes from a recent case where despite all tests passed in developing a PR. and despite unit test coverage is good, the library broken on a real test with a real AWS service (running the sample applications). What was the root cause amd wow could this be avoided or prevented?

Discussion

The root cause is fundamentally the need of mocking the S3 client in order to run the unit tests at build time. Doing so, we basically replace real production code behaviour of the client library and the real behaviour of the service. It may therefore happen that the behaviour modelled by the mock logic hides a misbehaviour of the code using the mock.

As far as I know, I see two approaches to address the issue:

1. Integration tests: we can add to the build pipeline and to the maven/gradle build additional tests that connect to real services and perform a number of high level integration tests; execution of integration tests may be also triggered by a proper switch so that we can avoid to run them at every build; we could also enforce that both unit and integration tests pass in order to accept a PR
2. Do not use client library mocks, but the real code; in this case though we need a way to simulate S3 exchanges (this can still be considered a mock, but let's park this distinction for the sake of this thread).

Note that bullet 1 above is probably unavoidable in the long run. There are however several cons with this approach:

• we need the endpoints up and running, possibly by several providers
• the behaviour of the online service may change without control
• test suite is going to be slow
• doable for the main high level paths, it may be impossible to cover all the different cases we may encounter in the real world (think about even just authentication, maintaining the test cases can easily be a remarkable challenge)
• not easily integrated into a TDD workflow

For these reasons, even if maybe unavoidable, using integration tests is feasible for some level of smoke tests.
We can instead start exploring option n2. How does it work? I see two main options:

1. Simulate the S3 service endpoint: this means having an HTTP server that simulates how a S3 service would behave. It is a kind of mocking technique, but in this case some coding is required to program the simulator. There are libraries/tools out there that help, like WireMock, which I am using in a few projects and I am pretty happy about.
2. Simulate the S3 service even before accessing the network (i.e. it works even if you do not have internet); this is a bit trickier but the basic concept is to replace the low level HTTP client implementation with one that can be programmatically controlled. Think about it like you hack the URLConnection object in JDK8 (see this to get the concept) or you stub JDK11HttpClient. To do so we need to know the HTTP client used by the S3Client library and see if it can be hacked (it may be not possible/too hard).

PROS and CONS of options 1

PROS

• there are already good implementations like WireMock
• it works with any HTTP client
• all test cases are under our control and we can simulate any S3 nuance

CONS

• it requires networking
• it runs a server during testing and makes HTTP communication (with potential port conflicts)
• poor performance
• would it work in GitHub CI pipeline?

PROS and CONS of options 2

PROS

• it does not require networking
• all test cases are under our control and we can simulate any S3 nuance
• best performance
• no server running
• works for sure in GitHub CI/CD

CONS

• HTTP client dependency
• no framework available as far as I know
• tricky

I would also note that both could be a separate project with the goal of simulate a AWS S3 service (maybe Amazon has something similar already)?

All the above is subject to discussion and I am very interest to hear your feedback and to learn if any other approach is available out there.

[markjschreiber]

Should we use GitHub discussions for discussions rather than issues? It's not something I have used before but would be interested to know what others think.

[stefanofornari]

You can also enable the discussion feature (from project settings, go down to features).

[markjschreiber]

Done

[guicamest]

Hey there, nice discussion!
I think option 1 is a good one, maybe even with localstack (using containers) so that requests don't need to be mocked/recorded/restored. I have worked with this kind of setup before and it was powerful (all s3 commands), simple (start/stop containers), "fast" (compared to going to AWS) and reliable (no surprises with network conditions communicating with AWS).

Tests are going to take a couple of minutes longer but in exchange there would be better coverage with integration tests, flexibility in creating test scenarios and simpler maintenance (compared to mocking). It is also important to consider that the build/releases are done in a greater period of time than just minutes.

Let me know if you are interested and I can raise a PR in a short time. Maybe it can help make a decision or raise other questions 👍

[markjschreiber]

I am interested in seeing a PR around this. I think that we should try to ensure the following when making an integration testing system.

1. It doesn't require a developer to have an AWS account or S3 bucket (for development and testing)
2. It doesn't cost anything for a developer or a user to run tests
3. Any third party framework is compatible with our open source license (e.g. doesn't require a separate paid license or invalidate our current license).
4. The framework is usable on OSX, Linux and Windows
5. The framework can be used with GitHub actions (compatible with our CI process).

[guicamest]

Alright, I'll try to get a POC soon with the technologies I mentioned.

I have not dealt with LICENCEs much, but from what I've found, they look compatible. Please let me know if there are non-code things to be done in order to comply.

aws-java-nio-spi-for-s3 Apache-2.0 license

testcontainers-java MIT license

localstack Apache-2.0 license

[stefanofornari]

What is not yet clear to me is what would be the target for those tests and what the containers should run. If I had to give a priority, I would focus first on AWS S3 integration tests. I guess in this case any container would not help and how can we address bullet 1 and 2 above?
For 3rd party service, which ones should we run? Who is supposed to set them up and maintain?

[guicamest]

For the POC, I thought of having a set of tests that resemble the way a user would use the library. This should help a lot with bug fixing if the users share their scenario. It could also help to avoid regressions with refactorings and code rewrite, while improving code coverage for scenarios that need actual networking

[markjschreiber]

MIT and Apache 2 are both fine. When you add your PR can you also include an update to the THIRD_PARTY file to indicate these dependencies and their respective licenses.

[guicamest]

PR for the POC is up 🎉 Please let me know what you think, what can be improved, documented, simplified, etc.