Testing frameworks and assertion libraries

[joaopapereira]

We have been discussing the possibility of using a testing framework as well as an assertion/matching library to make our tests more readable.
The objective of this discussion is to try to converge into a solution that could be implemented in all the carvel suite of tools.

Testing framework options:

• Golang Testing Framework (we are using it in our tools)
• Spec
• Ginkgo

Assertion/matching library options:

• Golang testing.T and build your own (we are using it in our tools)
• Testify
• Gomega
• ytt specific matches

Above there is an incomplete list of options that we should check out to see if any of them are a good match for our needs.

In order for us to make a decision, we should test out some of these libraries and see some code.
Please feel free to create your own snippet as an example of the usage of any of these libraries and share it here with some pros and cons of each of the libraries.

[joaopapereira]

I am going to start us off by creating an example of spec+testify.

Snippet: carvel-dev/imgpkg@f327f9b

I created a new file called copy_from_bundle_image_with_spec_test.go that is a copy of copy_from_bundle_image_test.go but using both spec and testify

spec is a thin wrapper around testing.T with some known terms like When, It. This framework, because it uses directly testing.T, runs by default in parallel. github.com/sclevine/spec/

Plus:

• thin wrapper around testing.T
• Add well-known terms like When and It
• Allows focusing a particular test
• Allow skipping a particular test
• As no extra dependencies
• test names can now be full sentences to describe the behavior that we expect

Minus:

• Test nesting can be hard to follow sometimes

testify is an assertion library that provides the most common assertions like calculating length or equality

Plus:

• better error output
  error messaging prebuilt in to testify for common cases
• easier to read the purpose of the test, by reading the code
• Prebuilt assertion
• Allow the developer to group assertions and report on all of them
  together

Minus:

• Takes some time to convert tests
• Some third-party dependencies

[pivotaljohn]

https://www.jetbrains.com/lp/devecosystem-2019/go/

[pivotaljohn]

We've had a lot of conversations, explored some examples...

Coming back to the Go Standard Library Testing package.

I submit that there are features present that we have yet to fully avail ourselves that would satisfy our desire for more maintainable test systems and retain the simplicity this framework brings.

They are:

• sub-tests
  • better organized tests (if done well) with no additional overhead
  • more natural language test names
  • GoLand's test runner is sub-test aware, allowing you to easily focus.
• T.Cleanup
  • is a proper TearDown() equivalent that's parallel test-aware
• T.Skip make it easy to skip particular tests
• with go test -run (regex), one can focus to any set of tests that can be expressed in a regular expression.

There's some idioms to hatch for cases where we want common per-test-case setup/teardown, but the ingredients to do so (I believe) are present.