app-staging-synthesizer: get S3 access denied error on stack deployment

[tmokmss]

Describe the bug

Also see the comment: #27434 (comment)

Hi, in the article about AppStagingSynthesizer recently published, we can see the following description about IAM roles to deploy stacks.

Reduced Bootstrapping Complexity: As the only shared resources required are global Roles, the company now only needs to bootstrap every account in one Region instead of bootstrapping every Region. This simplifies the bootstrapping process, making it easier to manage with CloudFormation StackSets.
https://aws.amazon.com/blogs/devops/enhancing-resource-isolation-in-aws-cdk-with-the-app-staging-synthesizer/

So I expect that a cdk app using AppStagingSynthesizer can be deployed to any region if we've done cdk bootstrapping in one region. However, currently it seems that we need to bootstrap in each region we're deploying to.

I'd like to make sure if it is expected or not. Maybe a documentation error? Thanks.

Expected Behavior

A cdk app using AppStagingSynthesizer can be deployed to any region if we've done cdk bootstrapping in one region.

Current Behavior

We need to bootstrap in each region we're deploying to.

Reproduction Steps

Bootstrap a region e.g. us-east-1.

Then, when I try to deploy a cdk app using AppStagingSynthesizer to another region without bootstrapping, I get the following error:

 ❌ Deployment failed: Error [ValidationError]: Role arn:aws:iam::123456789012:role/cdk-hnb659fds-cfn-exec-role-123456789012-eu-west-2 is invalid or cannot be assumed

The role name contains region eu-west-2, so it seems we have to bootstrap eu-west-2 first.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.98.0

Framework Version

2.98.0

Node.js Version

v18.13.0

OS

macOS

Language

Typescript

Language Version

No response

Other information

No response

[indrora]

Can you please provide how you're setting the region for deployment? A minimal working example would be best.

[tmokmss]

Hi @indrora, I noticed that I forget to set deploymentIdentities prop, and if I set this, the error is gone.
However, I'm getting another error about permission:

 ❌ Deployment failed: Error [ValidationError]: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

The reproduction code is here:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { AppStagingSynthesizer, DeploymentIdentities } from '@aws-cdk/app-staging-synthesizer-alpha';

const app = new cdk.App({
  defaultStackSynthesizer: AppStagingSynthesizer.defaultResources({
    appId: 'my-app-id',
    deploymentIdentities: DeploymentIdentities.defaultBootstrapRoles({ bootstrapRegion: 'us-east-1' }),
  }),
});

new cdk.Stack(app, 'StagingSynthesizerStack');

And run the following command:

AWS_REGION=us-east-1 npx cdk bootstrap
AWS_REGION=us-west-2 npx cdk deploy StagingSynthesizerStack

Can you reproduce this? Not quite sure if I'm using this feature correctly though.

[sakurai-ryo]

In v2.147.3, the same code seems to cause an error about the original cfn-exec-role error instead of the S3 validation error.
❌ Deployment failed: Error [ValidationError]: Role arn:aws:iam::111111111111:role/cdk-hnb659fds-cfn-exec-role-111111111111-ap-southeast-2 is invalid or cannot be assumed.

I am not sure about the S3 validation error, but as for the error about cfn-exec-role error, the BootstraplessSynthesizer used to synthesize StagingStack is probably the problem.

aws-cdk/packages/@aws-cdk/app-staging-synthesizer-alpha/lib/default-staging-stack.ts

Line 262 in 358cead

synthesizer: new BootstraplessSynthesizer(),

Since the arn of the CFn Execution Role is not passed as an argument when initializing the BootstraplessSynthesizer, the Synthesizer will use the CFn Execution Role of the region where the StagingStack is deployed.
However, even though the IAM Role does not exist in that region, it is used as a service role in the CFn deployment process, resulting in an error.

aws-cdk/packages/aws-cdk-lib/core/lib/stack-synthesizers/default-synthesizer.ts

Line 366 in 358cead

this._cloudFormationExecutionRoleArn = spec.specialize(this.props.cloudFormationExecutionRole ?? DefaultStackSynthesizer.DEFAULT_CLOUDFORMATION_ROLE_ARN);

aws-cdk/packages/aws-cdk-lib/core/lib/stack-synthesizers/default-synthesizer.ts

Line 240 in 358cead

public static readonly DEFAULT_CLOUDFORMATION_ROLE_ARN = 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region}';

So, I think it is necessary to correctly initialize the BootstraplessSynthesizer used to deploy the StagingStack.

Maybe relates #28195.