Making code coverage metrics and CI checks smarter

[mattkwiecien]

Today we discussed edge cases where the total test coverage metric may decrease when either no new code is added or when code quality is increased. In these instances the CI would reject the PR even if the PR contained improvements. For these reasons we want to investigate better test coverage metrics for firecrown.

We discussed two features that would be nice to have from code coverage:

1. All new lines of code added in the PR should be covered by unit tests
2. Any changed lines of code in the PR should be covered by unit tests

It doesn't appear that coveralls supports either of these checks and only supports a total code coverage metric/check for CI. There are other tools which do support this. I've found the two most popular appear to be diff-cover and Codecov, though diff-cover seems more finnicky and has less documentation.

Codecov appears much more widely used with extensive documentation, and easier to set up, but would involve moving away completely from coveralls. With Codecov, we could implement both of the above features using Codecov/patch and Codecov/project (see github check details here https://docs.codecov.com/docs/github-checks). Is this something we would consider?

[mattkwiecien]

We agreed that Codecov is worth pursuing as a coverage tool.