Code coverage with different test suites

[madurangasiriwardena]

We have started integrating with codecov to improve the unit test coverage. This will provide the unit test code coverage and make sure the unit test coverage is improved over time.

However, there is been little to no effort to calculate the code coverage from the integration, e2e and conformance test suites. While the scenario coverage is the main measurement of the above mentioned test suites, code coverage can provide valuable insights about the scenario coverage of these test suites.

So this improvement suggestion is to calculate the code coverage for integration, e2e and conformance test suites we have for the product.

[madurangasiriwardena]

To implement this,

• We need to retrieve the code coverage report from the integration test suite. While this is already calculated during the product build, there was no way to get this from the Jenkins builds of the product. So we need to add this to the archived artifacts list in the build job.
• Conformance test suite doesn't calculate the code coverage so far. Jacoco can be used to do so and conformance test suite GitHub actions needs to be updated to calculate the code coverage report and archive it as a build artifact. This is done with Generate jacoco report in conformance suite #21102 and Improvements to the jacoco report generation #21108
• Based on the generated code coverage reports, we can calculate the code coverage at the each repo level with Codecov. Flags can be used to differentiate between different test suites and this will help us to see the historical data of the code coverage. Draft PR Add git action to generate coverage for integration tests wso2-extensions/identity-inbound-auth-oauth#2571 creates a git action that can push the coverage reports to Codecov.
• Since the coverage reports are generated in the product build level and the coverage analysis needs to be done the the component level, we need to identify how can we integrate this to the feature development lifecycle. Current proposal is to run the report weekly and identify the coverage trends. Based on this, component owners needs to take an action.

Below screenshots demonstrates how the codecov will visualize the details.