aws-rds: Cannot add a region to Aurora Global Database

[sadiqinz]

Describe the bug

When creating an Global Aurora Database, you cannot add another Region to it.
We get an error message Resource handler returned message: "Invalid master password

Expected Behavior

You should be able to add additional Regions to Aurora Global Database.
Once a Global Database is created, you can create a secondary Cluster and add globalClusterIdentifier . Even though there is not L2 construct for adding a cluster to Global Database, L1 construct CfnDBCluster should allow you to specify globalClusterIdentifier.

Current Behavior

When you try and create the secondary cluster using CfnDBCluster construct, following error is thrown by CloudFormation Resource handler returned message: "Invalid master password . Even though based on the documentation, if you specify globalClusterIdentifier , you can't then give master password as it uses the one from source DB cluster.

Reproduction Steps

1. Create a standard Aurora DB Cluster using DatabaseCluster
2. Create a Global Database using CfnGlobalCluster and add source DB Cluster as the one created in step one.
3. Create a secondary DB Cluster using CfnDBCluster as following

const cfnDBCluster = new rds.CfnDBCluster(this, 'AuroraCluster', {
        globalClusterIdentifier: `${props.auroraglobalcluster}-${props.envName}`,
        dbClusterParameterGroupName: 'default.aurora-mysql8.0',
        enableGlobalWriteForwarding: true,
        availabilityZones: vpc.availabilityZones,
        dbSubnetGroupName: dbSubnetGroup.dbSubnetGroupName,
        engine: 'aurora-mysql',
        engineVersion: '8.0.mysql_aurora.3.04.0',
        kmsKeyId: 'alias/aws/rds'
      })
for (const az in vpc.availabilityZones) {
        const cfnDBInstance = new rds.CfnDBInstance(this, `AuroraInstance-${props.envName}-${az}`, {
          dbClusterIdentifier: cfnDBCluster.dbClusterIdentifier,
          engine: 'aurora-mysql',
          dbInstanceClass: 'db.r6g.large',
          publiclyAccessible: false,
          availabilityZone: vpc.availabilityZones[az]    
        })        
      }

This would produce an error message by CloudFormation.

Possible Solution

Either fix the bug in CfnDBCluster construct to support GlobalDatabases or create add GlobalDatabase support in L2 construct DatabaseCluster

Additional Information/Context

No response

CDK CLI Version

2.131.0

Framework Version

No response

Node.js Version

v18.16.0

OS

macOS 14.3.1

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

According the doc here:

GlobalClusterIdentifier
If you are configuring an Aurora global database cluster and want your Aurora DB cluster to be a secondary member in the global database cluster, specify the global cluster ID of the global database cluster. To define the primary database cluster of the global cluster, use the AWS::RDS::GlobalCluster resource.

If you aren't configuring a global database cluster, don't specify this property.

You mentioned:

Even though based on the documentation, if you specify globalClusterIdentifier , you can't then give master password as it uses the one from source DB cluster.

It's still unclear for me how to provision that from CFN's perspective. Can you share the doc link for this description you mentioned above to help us clarify?

[sadiqinz]

Here is information from CFN (doc) around specifying MasterUserPassword

MasterUserPassword
The master password for the DB instance.

Note
If you specify the SourceDBClusterIdentifier, SnapshotIdentifier, or GlobalClusterIdentifier property, don't specify this property. The value is inherited from the source DB cluster, the snapshot, or the primary DB cluster for the global database cluster, respectively.

There is a CFN example on this page for setting up a GlobalDatabase. You would notice that there is not MaterUserPassword specified when adding a DBCluster to GlobalCluster

[sadiqinz]

Hi Team, Is there any update on this?

[pahud]

Hi

This doesn't seem to be a CDK bug and we need to look into this issue and report to relevant team.

Are you able to provide your code snippet for Step 1 and 2 below so we can verify it from our account?

1. Create a standard Aurora DB Cluster using DatabaseCluster
2. Create a Global Database using CfnGlobalCluster and add source DB Cluster as the one created in step one.
3. Create a secondary DB Cluster using CfnDBCluster as following

[sadiqinz]

Hi Sure,

Please see below.
Step 1 - Creating a standard Aurora DB Cluster using DatabaseCluseter

const auroraCluster = new rds.DatabaseCluster(this, "AuroraCluster", {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_04_0 }),
      credentials: rds.Credentials.fromSecret(secret),
      defaultDatabaseName: "stocks",
      writer: rds.ClusterInstance.provisioned('writer', {
        enablePerformanceInsights: true,
        instanceType: ec2.InstanceType.of(ec2.InstanceClass.R6G, ec2.InstanceSize.LARGE),
      }),        
      storageEncrypted: true,
      vpc: vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
      },
      securityGroups: [dbsg],
})

for (const az in vpc.availabilityZones) {
  const azLetter = vpc.availabilityZones[az].slice(-1);
  const cfnDBInstance = new rds.CfnDBInstance(this, `AuroraInstance-${az}`, {
    dbClusterIdentifier: auroraCluster.clusterIdentifier,
    dbInstanceIdentifier: `${azLetter}`,
    engine: 'aurora-mysql',
    dbInstanceClass: 'db.r6g.large',
    publiclyAccessible: false,
    enablePerformanceInsights: true,
    availabilityZone: vpc.availabilityZones[az]          
  })
  cfnDBInstance.node.addDependency(auroraCluster)
}    

Step 2 - Creating a Global Cluster using CfnGlobalCluster

const cfnGlobalCluster = new rds.CfnGlobalCluster(this, 'AuroraGlobalCluster', {
        deletionProtection: false,
        globalClusterIdentifier: `${props.auroraglobalcluster}`,
        sourceDbClusterIdentifier: auroraCluster.clusterIdentifier
 })

[sadiqinz]

Hi Team, Is there any update on this?

[pahud]

Hi

If I deploy like this:

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);


    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: true }); 

    // step 1 - Create a standard Aurora DB Cluster using DatabaseCluster
    const auroraCluster = new rds.DatabaseCluster(this, "AuroraCluster", {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_04_0 }),
      // credentials: rds.Credentials.fromSecret(secret),
      defaultDatabaseName: "stocks",
      writer: rds.ClusterInstance.provisioned('writer', {
        enablePerformanceInsights: true,
        instanceType: ec2.InstanceType.of(ec2.InstanceClass.R6G, ec2.InstanceSize.LARGE),
      }),        
      storageEncrypted: true,
      vpc: vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
      },
      // securityGroups: [dbsg],
    })

    for (const az in vpc.availabilityZones) {
      const azLetter = vpc.availabilityZones[az].slice(-1);
      const cfnDBInstance = new rds.CfnDBInstance(this, `AuroraInstance-${az}`, {
        dbClusterIdentifier: auroraCluster.clusterIdentifier,
        dbInstanceIdentifier: `${azLetter}`,
        engine: 'aurora-mysql',
        dbInstanceClass: 'db.r6g.large',
        publiclyAccessible: false,
        enablePerformanceInsights: true,
        availabilityZone: vpc.availabilityZones[az]          
      })
      cfnDBInstance.node.addDependency(auroraCluster)
    }   
    
    // step 2 - Create a Global Database using CfnGlobalCluster and add source DB Cluster as the one created in step one.
    const cfnGlobalCluster = new rds.CfnGlobalCluster(this, 'AuroraGlobalCluster', {
      deletionProtection: false,
      globalClusterIdentifier: 'dummy-global-cluster',
      sourceDbClusterIdentifier: auroraCluster.clusterIdentifier
    })

    // step 3 - Create a secondary DB Cluster using CfnDBCluster
    const cfnDBCluster = new rds.CfnDBCluster(this, 'AuroraCluster2', {
      globalClusterIdentifier: 'dummy-global-cluster',
      dbClusterParameterGroupName: 'default.aurora-mysql8.0',
      enableGlobalWriteForwarding: true,
      availabilityZones: vpc.availabilityZones,
      // dbSubnetGroupName: dbSubnetGroup.dbSubnetGroupName,
      engine: 'aurora-mysql',
      engineVersion: '8.0.mysql_aurora.3.04.0',
      // kmsKeyId: 'alias/aws/rds'
    })
  }
} 

It would fail at step 3 with the error message:

11:31:32 PM | CREATE_FAILED | AWS::RDS::DBCluster | AuroraCluster2
Resource handler returned message: "Replication from cluster in same region is not supported (Service: Rds, Status Code: 400, Requ
est ID: aa441cae-3903-4c06-b598-99202db5b485)" (RequestToken: 062ee76b-4933-a14e-81bf-c67ccc5fe644, HandlerErrorCode: InvalidRequest)

Obviously, you cannot create an Aurora DB cluster as a read replica of another Aurora DB cluster within the same AWS region. Amazon RDS does not support creating read replicas of Aurora clusters in the same region. [1]

To work it around, you need to separate them into 2 stacks in different regions:

stack.ts

export class PrimaryStack extends Stack {
  readonly globalClusterIdentifier: string;
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);


    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: true });
    this.globalClusterIdentifier = 'dummy-global-cluster'

    // step 1 - Create a standard Aurora DB Cluster using DatabaseCluster
    const auroraCluster = new rds.DatabaseCluster(this, "AuroraCluster", {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_04_0 }),
      // credentials: rds.Credentials.fromSecret(secret),
      defaultDatabaseName: "stocks",
      writer: rds.ClusterInstance.provisioned('writer', {
        enablePerformanceInsights: true,
        instanceType: ec2.InstanceType.of(ec2.InstanceClass.R6G, ec2.InstanceSize.LARGE),
      }),        
      storageEncrypted: true,
      vpc: vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
      },
      // securityGroups: [dbsg],
    })

    for (const az in vpc.availabilityZones.slice(0, 3)) {
      const azLetter = vpc.availabilityZones[az].slice(-1);
      const cfnDBInstance = new rds.CfnDBInstance(this, `AuroraInstance-${az}`, {
        dbClusterIdentifier: auroraCluster.clusterIdentifier,
        dbInstanceIdentifier: `${azLetter}`,
        engine: 'aurora-mysql',
        dbInstanceClass: 'db.r6g.large',
        publiclyAccessible: false,
        enablePerformanceInsights: true,
        availabilityZone: vpc.availabilityZones[az]          
      })
      cfnDBInstance.node.addDependency(auroraCluster)
    }   
    
    // step 2 - Create a Global Database using CfnGlobalCluster and add source DB Cluster as the one created in step one.
    const cfnGlobalCluster = new rds.CfnGlobalCluster(this, 'AuroraGlobalCluster', {
      deletionProtection: false,
      globalClusterIdentifier: this.globalClusterIdentifier,
      sourceDbClusterIdentifier: auroraCluster.clusterIdentifier
    })
  }
} 

export interface ReplicaStackProps extends StackProps {
  readonly globalClusterIdentifier: string;
}
export class ReplicaStack extends Stack {
  constructor(scope: Construct, id: string, props: ReplicaStackProps) {
    super(scope, id, props);

    const globalClusterIdentifier = props.globalClusterIdentifier

    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: true });

    // step 3 - Create a secondary DB Cluster using CfnDBCluster
    const subnetGroup = new rds.SubnetGroup(this, 'SubnetGroup', {
      description: 'Subnet group for Aurora cluster',
      vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        onePerAz: true,
        // select first 3 AZs from the list
        availabilityZones: vpc.availabilityZones.slice(0,3),
      },
    });
    const cfnDBCluster = new rds.CfnDBCluster(this, 'AuroraCluster2', {
      globalClusterIdentifier: globalClusterIdentifier,
      dbClusterParameterGroupName: 'default.aurora-mysql8.0',
      enableGlobalWriteForwarding: true,
      availabilityZones: vpc.availabilityZones.slice(0,3),
      dbSubnetGroupName: subnetGroup.subnetGroupName,
      engine: 'aurora-mysql',
      engineVersion: '8.0.mysql_aurora.3.04.0',
      kmsKeyId: 'alias/aws/rds',
    })
    
    for (const az in vpc.availabilityZones.slice(0, 3)) {
      new rds.CfnDBInstance(this, `AuroraInstance-${az}`, {
        dbClusterIdentifier: cfnDBCluster.ref,
        engine: 'aurora-mysql',
        dbInstanceClass: 'db.r6g.large',
        dbSubnetGroupName: subnetGroup.subnetGroupName,
        publiclyAccessible: false,
        availabilityZone: vpc.availabilityZones[az],
      })        
    }
  }
}

app.ts

// create the primary stack in us-east-1
const primaryStack = new PrimaryStack(app, 'primary-stack', { env: { region: 'us-east-1', account: process.env.CDK_DEFAULT_ACCOUNT } });

// create the replica stack in us-west-2
new ReplicaStack(app, 'replica-stack', { 
  env: { region: 'us-west-2', account: process.env.CDK_DEFAULT_ACCOUNT },
  globalClusterIdentifier: primaryStack.globalClusterIdentifier,
});

It deploys for me with no error.

Let me know if it works for you.

[sadiqinz]

Thanks, yes I am aware we need to pass in different regions to implement this.
In our implementation we only had one stack i.e. one file and were passing in a parameter of "Primary" or "Secondary". Based on that we would create primary cluster and replica. Value of "Global Cluster Identifier" was hard coded.

In your example I don't see any db instances being added to replica in secondary region. And I think that is the issue. Can you try and add DB instances to that secondary replica?

[pahud]

Yes I would see this error when adding instances in the replica cluster using globalClusterIdentifier:

We need to verify

1. Does Aurora Global Database support this use case?
2. If yes, what makes CFN error and how to make it deploy with CFN?

As long as we figure out how to do that with CFN, we would know how to do this using CDK.

I am raising this internally with relevant teams for clarifying.

internal tracking: P139881819

[pahud]

Hi @sadiqinz

I have updated my sample code above and it deploys now.

it looks like this in the RDS console:

Let me know if it works for you.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[sadiqinz]

Thanks for all the help @pahud

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[sadiqinz]

Great help from the team to find a resolution for this issue

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.