[BUG] ISM indices creation do not respect a single-node cluster setting

[jazzl0ver]

Context:
https://opensearch.slack.com/archives/C051JEH8MNU/p1719393439224069
opensearch-project/security#3130

In a single-node cluster all system indices should be created with no replicas

[dblock]

@jazzl0ver The Slack channel doesn't keep data for more than 30 days or so, care to port the description of the issue in detail here, please?

[Catch All Triage - 1, 2, 3, 4]

[jazzl0ver]

The issue description from the Slack channel:
there's a one-node installation of Opensearch 2.10. is there a way to automatically set a number of replicas to 0 for .opendistro-ism-managed-index-history* indices? i tried to create an ISM policy like:

{
    "id": "no-replicas",
    "seqNo": 101713724,
    "primaryTerm": 124,
    "policy": {
        "policy_id": "no-replicas",
        "description": "No replicas",
        "last_updated_time": 1719215357789,
        "schema_version": 19,
        "error_notification": {
            "channel": {
                "id": "vj38eYYB46ulspQUUC0C"
            },
            "message_template": {
                "source": "",
                "lang": "mustache"
            }
        },
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 0
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    ".opendistro-ism-managed-index-history-*",
                    "security-*"
                ],
                "priority": 1,
                "last_updated_time": 1719049665979
            }
        ]
    }
}

along with permitting to modify system indices:

# grep system_ind /etc/wazuh-indexer/opensearch.yml
plugins.security.system_indices.enabled: false
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opendistro-ism-managed-index-history*"]

However, the policy is not applied to that indices for some reason.

The long story short - this setting did the job:

plugins.index_state_management.history.number_of_replicas = 0

[jazzl0ver]

Now the same issue has happened with the other index:

.opendistro-alerting-alert-history-2024.10.04-000025    0 r UNASSIGNED INDEX_CREATED

[jazzl0ver]

Looks like this is a duplicate of opensearch-project/security#4739