Testing Tekton Tasks and Pipelines

[afrittoli]

Today we use tasks and pipelines in the tests of some of Tekton's project to provide end of end verification of Tekton's features.
The catalog hosts reusable tasks and pipeline, and the in includes two kind of tests: linting and e2e. In the e2e tests we verify that the task and/or pipeline runs to successful completion.

We could consider producing documentation of testing strategies and/or tools to allow for deeper testing of tasks and pipelines:

• verify success with different input parameters
• verify success with or without optional components (e.g. params, workspaces)
• test the presence and correctness of results
• test different execution flows (conditionals, finally)
• verify the behaviour (was the image pushed to the registry, did the deployment happen, etc).

Advantages of this work would be:

• testing helps task and pipeline authors verify the behaviour earlier in the lifecycle of their workflows
• testing fosters reusability, since it makes it easier to ensure stable interfaces for tasks and pipelines
• testing requirements could result in best practices on how to write testable tasks, and ultimately more reusable ones

[bobcatfish]

There is some prior work in this TEP proposal that I didn't see all the way through: #170

In the PR and the linked design docs there is some requirements gathering and also some exploration of different alternatives.

Ultimately it would be great to express these tests using Tekton types.

[coyote240]

Having read through your TEP, I realize we're talking about maybe two different use cases. One the one hand, and I think this is where you're coming from, we want catalog tasks to have associated tests, for developer and user assurance. On the other, and this is what's on my mind, I would like to have a good path forward for users to have tests associated with their own pipelines.

[coyote240]

This is timely for my team as we are taking early steps to test our pipelines using helm test. While I realize helm may not be ubiquitous, the capability has potential.

@afrittoli may find my example familiar as he helped me with a workspace issue in this example

One obvious flaw, however, is that a task may fail while exiting successfully, so the feedback isn't quite right, we're left watching the console for output.