Tracking: [CLI] testing posture improvements | Phase 1

[mrgrain]

Overview

As part of the build and release process, we are running a set of CLI integration tests if any of the CLI code has changed.
These tests currently take 50+ min each and are frequently unstable. We are also missing unit tests where we could have them. In general our branches coverage percentages are comparatively low (68% as of 2024-10-29).

Implementation

We are addressing the shortfall through a number of improvements. See issues for details.

Issue list

• cli: code coverage must never decrease rule #31931
• Test environment reservation service #31938
• cli: automatic cross account integration tests #32085

[iliapolo]

Some additional thoughts on this one:

• Add coverage reports based on integration tests.
• Run all integ tests in a cross account scenario as well.
  • Run the CLI with credentials from account A where the target account trusts account A via cdk bootstrap --trust. This would have caught ❗NOTICE: Cross-account deployments are failing in v2.163.0 with error 'Need to perform AWS calls for account' #31845
  • Deploy stacks to an account different than the bootstrap account (centralized staging bucket scenario).
• Run all integ tests in a multi environment scenario as well. E.g deploy the same stack to multiple envs within a single CLI execution. This will catch any code that incorrectly assumes a single environment (bootstrap version checks for example).
• IAM regression testing. Ensure commands don't introduce new role assumptions or new actions because users may have configured locked down permissions. This would have prevented ❗ NOTICE: cdk diff tries to assume 'deploy' role and fails to authorize. #29483. We should recognize though that sometimes we would like to rollout changes like these so we need a bypass mechanism as well. Catching these in regression gives us a chance to properly communicate these changes even though we sometimes bypass the validation.
  • We should be able to automatically capture all roles and actions performed by CLI and write them as a snapshot.