Bug: sam deploy started failing

[danfraticiu]

Description:

sam deploy stopped working for all users on my organization's account, with the error message Error: Failed to create changeset for the stack: hello-world-node, An error occurred (ValidationError) when calling the CreateChangeSet operation: S3 error: Access Denied

This issue was observed only a day ago, don't know who longs it started happening.

Steps to reproduce:

Running any sam deploy command that was previously working yields same error, tested with different users. Was able to reproduce the issue when trying to deploy a brand new applications (using this the sample project from git@github.com:serverless-projects/aws-sam-examples.git (namely the samples_1/hello-world/node).

All users have the arn:aws:iam::aws:policy/AmazonS3FullAccess policy.

Also worth nothing the .template files is uploaded to S3 successfully and AFAICT CreateChangeSet does not produce S3 objects, so I have no idea why it would result in this error.

Observed result:

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Not found

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: hello-world-node
        AWS Region [us-east-1]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: Y
        #Preserves the state of previously provisioned resources when an operation fails
        Disable rollback [y/N]: y
        Save arguments to configuration file [Y/n]: y
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]: 

        Looking for resources needed for deployment:

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1uxe4rz6ixxw1
        A different default S3 bucket can be set in samconfig.toml and auto resolution of buckets turned off by setting resolve_s3=False

        Saved arguments to config file
        Running 'sam deploy' for future deployments will use the parameters saved above.
        The above parameters can be changed by modifying samconfig.toml
        Learn more about samconfig.toml syntax at 
        https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

        Uploading to hello-world-node/493045ecd69ba51d995e0a9eea3cf12f  1631 / 1631  (100.00%)

        Deploying with following values
        ===============================
        Stack name                   : hello-world-node
        Region                       : us-east-1
        Confirm changeset            : True
        Disable rollback             : True
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-1uxe4rz6ixxw1
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}

Initiating deployment
=====================

        Uploading to hello-world-node/80f11a5efa41b51bd7e7d97a6314d886.template  439 / 439  (100.00%)
Error: Failed to create changeset for the stack: hello-world-node, An error occurred (ValidationError) when calling the CreateChangeSet operation: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Expected result:

A succesfull deploy.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS 14.5
2. sam --version: 1.123.0
3. AWS region: us-east-1

{
  "version": "1.123.0",
  "system": {
    "python": "3.12.6",
    "os": "macOS-14.5-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "27.1.1",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[danfraticiu]

Turned out this had nothing to do with sam there was a new policy added to the account that was not aware of, the policy denied all actions where MFA was not present.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.