(opensearchservice): must configure zone awareness settings even when i am not enabling zone awareness #29346

orshemtov · 2024-03-03T15:18:55Z

Describe the bug

When creating an opensearch domain using AWS CDK, I am getting the following error:

Invalid request provided: You must configure zone awareness settings if you turn on zone awareness

My CDK code is as follows:

import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as opensearch from "aws-cdk-lib/aws-opensearchservice";
import { Construct } from "constructs";

export interface OpenSearchProps {
  vpc: ec2.IVpc;
  subnets: ec2.ISubnet[];
}

export class OpenSearch extends Construct {
  constructor(scope: Construct, id: string, props: OpenSearchProps) {
    super(scope, id);

    ...

    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        {
          subnets,
        },
      ],
      securityGroups: [securityGroup],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      zoneAwareness: {
        enabled: false,
      },
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });
  }
}

I've also tried with omitting zoneAwareness altogether

Expected Behavior

The cluster should be deployed in a single AZ

Current Behavior

There is an error stating that zone awareness must be configured, even tho my zone awareness is set to false

Reproduction Steps

Create a CDK stack

export class MyStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

     // VPC
    const vpcId = this.node.tryGetContext("vpcId");
    const vpc = ec2.Vpc.fromLookup(this, "Vpc", { vpcId });
    const subnets = vpc.privateSubnets;
    const subnetIds = vpc.privateSubnets.map((subnet) => subnet.subnetId);

    // OpenSearch
    const opensearch = new OpenSearch(this, "OpenSearch", {
      vpc,
      subnets,
    });

  }
}

import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as opensearch from "aws-cdk-lib/aws-opensearchservice";
import { Construct } from "constructs";

export interface OpenSearchProps {
  vpc: ec2.IVpc;
  subnets: ec2.ISubnet[];
}

export class OpenSearch extends Construct {
  constructor(scope: Construct, id: string, props: OpenSearchProps) {
    super(scope, id);

    const { vpc, subnets } = props;

    const securityGroup = new ec2.SecurityGroup(this, "SecurityGroup", {
      vpc,
      allowAllOutbound: true,
    });

    securityGroup.addIngressRule(
      ec2.Peer.ipv4(vpc.vpcCidrBlock),
      ec2.Port.tcp(9200)
    );

    securityGroup.addIngressRule(
      ec2.Peer.ipv4(vpc.vpcCidrBlock),
      ec2.Port.tcp(9300)
    );

    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        {
          subnets,
        },
      ],
      securityGroups: [securityGroup],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      zoneAwareness: {
        enabled: false,
      },
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });

    new cdk.CfnOutput(this, "Endpoint", {
      value: opensearchDomain.domainEndpoint,
    });
  }
}

Possible Solution

Downgrading to CDK version 2.85.0 seems to fix the problem

Additional Information/Context

No response

CDK CLI Version

2.117.0 (build 59d9b23)

Framework Version

2.127.0

Node.js Version

v21.6.2

OS

macos

Language

TypeScript

Language Version

5.3.3

Other information

No response

The text was updated successfully, but these errors were encountered:

msambol · 2024-03-04T01:37:32Z

I was able to reproduce this and confirmed the template contains the following:

"ZoneAwarenessEnabled": false

This feels like a CFN bug, as the error is from CFN not the CDK. cc/ @pahud

pahud · 2024-03-04T17:38:57Z

I see this from the doc:

If you specify more than one subnet, you must also configure ZoneAwarenessEnabled and ZoneAwarenessConfig within ClusterConfig, otherwise you'll see the error "You must specify exactly one subnet" during template creation.

And I got this error when I deploy across 3 subnets/AZs

12:22:07 PM | CREATE_FAILED | AWS::OpenSearchService::Domain | Domain66AC69E0
Resource handler returned message: "Invalid request provided: You must specify exactly one subnet. (Service: OpenSearch, Status Code: 400, Request
ID: ebf77162-b821-49c2-b061-bc635d708913)" (RequestToken: dcb066e2-658e-35bb-d913-d0b3640afe9b, HandlerErrorCode: InvalidRequest)

Looks like when ZoneAwarenessEnabled is disabled, only one subnet for the domain is allowed. However, if we specify the vpcSubnets like this, multiple subnets would be selected:

    vpcSubnets: [
        { subnetType: SubnetType.PRIVATE_WITH_EGRESS },
      ],

My workaround is:

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const vpc = getDefaultVpc(this);
    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        { subnetType: SubnetType.PRIVATE_WITH_EGRESS },
      ],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: RemovalPolicy.DESTROY,
      zoneAwareness: {
        enabled: false,
      },
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });
    
    const cfndomain = opensearchDomain.node.tryFindChild('Resource') as opensearch.CfnDomain

    const selectedSubnetIds = vpc.selectSubnets({ subnetType: SubnetType.PRIVATE_WITH_EGRESS }).subnetIds
    cfndomain.addPropertyOverride('VPCOptions.SubnetIds', [ selectedSubnetIds[0] ] )

  }
}

I will create an internal ticket to clarify if only 1 subnet is allowed when zoneAawreness is disabled. Meanwhile, can you share your use case that you need multiple AZs with zoneAawreness disabled?

orshemtov · 2024-03-04T17:44:56Z

My use case is that I was actually trying to create a domain with no multiple AZs, only a single AZ

When I try to do that, I receive the above error

Invalid request provided: You must configure zone awareness settings if you turn on zone awareness

At no point did I enable zone awareness explicitly, in fact I've explicitly turned it off.

Could be because I'm supplying multiple subnets tho, as mentioned above (supplying the private ones)

But I also tried doing

vpcSubnets: [
        {
          subnets: [subnets[0]],
        },
      ],

And still got the same error

pahud · 2024-03-04T17:46:43Z

@orshemtov check out my workaround in my last comment

pahud · 2024-03-04T17:46:48Z

internal tracking: V1282499345

orshemtov · 2024-03-04T17:59:39Z

@pahud So I've changed my CDK construct to what you've suggested:

import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as opensearch from "aws-cdk-lib/aws-opensearchservice";
import { Construct } from "constructs";

export interface OpenSearchProps {
  vpc: ec2.IVpc;
  subnets: ec2.ISubnet[];
}

export class OpenSearch extends Construct {
  constructor(scope: Construct, id: string, props: OpenSearchProps) {
    super(scope, id);

    const { vpc, subnets } = props;

    const securityGroup = new ec2.SecurityGroup(this, "SecurityGroup", {
      vpc,
      allowAllOutbound: true,
    });

    securityGroup.addIngressRule(
      ec2.Peer.ipv4(vpc.vpcCidrBlock),
      ec2.Port.tcp(9200)
    );

    securityGroup.addIngressRule(
      ec2.Peer.ipv4(vpc.vpcCidrBlock),
      ec2.Port.tcp(9300)
    );

    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        {
          subnets,
        },
      ],
      securityGroups: [securityGroup],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });

    const cfnDomain = opensearchDomain.node
      .defaultChild as opensearch.CfnDomain;
    const selectedSubnetIds = vpc.selectSubnets({
      subnets,
    }).subnetIds;
    cfnDomain.addPropertyOverride("VPCOptions.SubnetIds", [
      selectedSubnetIds[0],
    ]);

    new cdk.CfnOutput(this, "Endpoint", {
      value: opensearchDomain.domainEndpoint,
    });
  }
}

And I'm still getting the same error after doing cdk deploy

7:57:11 PM | CREATE_FAILED        | AWS::OpenSearchService::Domain              | OpenSea
rch/Domain
Resource handler returned message: "Invalid request provided: You must configure zone awa
reness settings if you turn on zone awareness. (Service: OpenSearch, Status Code: 400, Re
quest ID: ff6cbc56-0f51-415c-be42-080583f952e8)" (RequestToken: 241b96ab-035d-bda4-ddaf-7
0b348e25eba, HandlerErrorCode: InvalidRequest)

pahud · 2024-03-04T18:28:52Z

And I'm still getting the same error after doing cdk deploy

I didn't see you turn on zone awareness from your code snippet above and it should be "ZoneAwarenessEnabled": false

Can you check your cdk synth and verify that?

pahud · 2024-03-04T18:30:27Z

I am trying to deploy this now.

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const vpc = getDefaultVpc(this);
    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        { subnetType: SubnetType.PRIVATE_WITH_EGRESS },
      ],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: RemovalPolicy.DESTROY,
      // zoneAwareness: {
      //   enabled: false,
      // },
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });
    
    const cfndomain = opensearchDomain.node.tryFindChild('Resource') as opensearch.CfnDomain

    const selectedSubnetIds = vpc.selectSubnets({ subnetType: SubnetType.PRIVATE_WITH_EGRESS }).subnetIds
    cfndomain.addPropertyOverride('VPCOptions.SubnetIds', [ selectedSubnetIds[0] ] )
  }
}

And the synth like

 "Domain66AC69E0": {
   "Type": "AWS::OpenSearchService::Domain",
   "Properties": {
    "ClusterConfig": {
     "DedicatedMasterEnabled": false,
     "InstanceCount": 1,
     "InstanceType": "t3.small.search",
     "ZoneAwarenessEnabled": false
    },
    "DomainEndpointOptions": {
     "EnforceHTTPS": false,
     "TLSSecurityPolicy": "Policy-Min-TLS-1-2-2019-07"
    },
    "EBSOptions": {
     "EBSEnabled": true,
     "VolumeSize": 10,
     "VolumeType": "gp2"
    },
    "EncryptionAtRestOptions": {
     "Enabled": false
    },
    "EngineVersion": "OpenSearch_2.5",
    "LogPublishingOptions": {},
    "NodeToNodeEncryptionOptions": {
     "Enabled": false
    },
    "VPCOptions": {
     "SecurityGroupIds": [
      {
       "Fn::GetAtt": [
        "DomainSecurityGroup48AA5FD6",
        "GroupId"
       ]
      }
     ],
     "SubnetIds": [
      "subnet-071c85610846aa9c0"
     ]
    }
   },

It could take a while but I didn't see any error for now.

orshemtov · 2024-03-05T08:47:54Z

im getting this synth

OpenSearchDomain099259C2:
    Type: AWS::OpenSearchService::Domain
    Properties:
      ClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 1
        InstanceType: t3.small.search
        MultiAZWithStandbyEnabled: true
        ZoneAwarenessEnabled: false
      DomainEndpointOptions:
        EnforceHTTPS: false
        TLSSecurityPolicy: Policy-Min-TLS-1-2-2019-07
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: gp2
      EncryptionAtRestOptions:
        Enabled: false
      EngineVersion: OpenSearch_2.5
      LogPublishingOptions: {}
      NodeToNodeEncryptionOptions:
        Enabled: false
      Tags:
        - Key: app
          Value: vita-llms
        - Key: env
          Value: dev
      VPCOptions:
        SecurityGroupIds:
          - Fn::GetAtt:
              - OpenSearchSecurityGroup70E5053B
              - GroupId
        SubnetIds:
          - subnet-01da729e6394035cc
    UpdatePolicy:
      EnableVersionUpgrade: true
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: VitaLlmsStack/OpenSearch/Domain/Resource

original cdk code

  const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        {
          subnets,
        },
      ],
      securityGroups: [securityGroup],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
      },
    });

    const cfnDomain = opensearchDomain.node
      .defaultChild as opensearch.CfnDomain;

    const selectedSubnetIds = vpc.selectSubnets({
      subnets,
    }).subnetIds;

    cfnDomain.addPropertyOverride("VPCOptions.SubnetIds", [
      selectedSubnetIds[0],
    ]);

for some reason this comes out 'true', im not sure if thats intended:

MultiAZWithStandbyEnabled

orshemtov · 2024-03-05T08:55:09Z

it turns out this flag can be disabled by setting

capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
        multiAzWithStandbyEnabled: false,
      },

tho, i think this is a bug because the doc for this flag states the default should be false:

/**
     * Indicates whether Multi-AZ with Standby deployment option is enabled.
     * For more information, see [Multi-AZ with Standby]
     * (https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-multiaz.html#managedomains-za-standby)
     *
     * @default - no multi-az with standby
     */
    readonly multiAzWithStandbyEnabled?: boolean;

now my deployment still hasnt failed for a few minutes

orshemtov · 2024-03-05T10:34:33Z

final code that worked:

    const opensearchDomain = new opensearch.Domain(this, "Domain", {
      vpc,
      vpcSubnets: [
        {
          subnets: [subnets[0]],
        },
      ],
      securityGroups: [securityGroup],
      version: opensearch.EngineVersion.OPENSEARCH_2_5,
      tlsSecurityPolicy: opensearch.TLSSecurityPolicy.TLS_1_2,
      enableVersionUpgrade: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      capacity: {
        dataNodeInstanceType: "t3.small.search",
        dataNodes: 1,
        multiAzWithStandbyEnabled: false,
      },
    });

pahud · 2024-06-03T19:54:17Z

Thank you @orshemtov

levilugato · 2024-09-27T13:45:53Z

Thanks, I'm facing exactly same issue

orshemtov added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Mar 3, 2024

github-actions bot added the @aws-cdk/aws-opensearch Related to the @aws-cdk/aws-opensearchservice package label Mar 3, 2024

pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Mar 4, 2024

pahud self-assigned this Mar 4, 2024

github-actions bot mentioned this issue Apr 1, 2024

Monthly issue metrics report #29667

Closed

pahud removed their assignment Jun 3, 2024

pahud added p3 and removed p2 labels Jun 11, 2024

khushail mentioned this issue Sep 27, 2024

(opensearchservice): example in the doc doesn't work, error received-You must configure zone awareness settings if you turn on zone awareness. #31583

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(opensearchservice): must configure zone awareness settings even when i am not enabling zone awareness #29346

(opensearchservice): must configure zone awareness settings even when i am not enabling zone awareness #29346

orshemtov commented Mar 3, 2024 •

edited

Loading

msambol commented Mar 4, 2024 •

edited

Loading

pahud commented Mar 4, 2024

orshemtov commented Mar 4, 2024 •

edited

Loading

pahud commented Mar 4, 2024

pahud commented Mar 4, 2024

orshemtov commented Mar 4, 2024

pahud commented Mar 4, 2024

pahud commented Mar 4, 2024 •

edited

Loading

orshemtov commented Mar 5, 2024 •

edited

Loading

orshemtov commented Mar 5, 2024

orshemtov commented Mar 5, 2024

pahud commented Jun 3, 2024

levilugato commented Sep 27, 2024

(opensearchservice): must configure zone awareness settings even when i am not enabling zone awareness #29346

(opensearchservice): must configure zone awareness settings even when i am not enabling zone awareness #29346

Comments

orshemtov commented Mar 3, 2024 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

msambol commented Mar 4, 2024 • edited Loading

pahud commented Mar 4, 2024

orshemtov commented Mar 4, 2024 • edited Loading

pahud commented Mar 4, 2024

pahud commented Mar 4, 2024

orshemtov commented Mar 4, 2024

pahud commented Mar 4, 2024

pahud commented Mar 4, 2024 • edited Loading

orshemtov commented Mar 5, 2024 • edited Loading

orshemtov commented Mar 5, 2024

orshemtov commented Mar 5, 2024

pahud commented Jun 3, 2024

levilugato commented Sep 27, 2024

orshemtov commented Mar 3, 2024 •

edited

Loading

msambol commented Mar 4, 2024 •

edited

Loading

orshemtov commented Mar 4, 2024 •

edited

Loading

pahud commented Mar 4, 2024 •

edited

Loading

orshemtov commented Mar 5, 2024 •

edited

Loading