[NEW] CLUSTER CONFIG-SET command to set config to the entire cluster.

[Shivshankar-Reddy]

The problem/use-case that the feature addresses

Implement a CLUSTER CONFIG-SET command that allows for consistent configuration across all nodes in the cluster. Currently, configurations must be set individually on each node. There is a chance of some configs could break the cluster so this command should be have list of configs which should not be propagated.

Description of the feature

Implement a command as follows which allows the clients to set config entire cluster consistently.

CLUSTER CONFIG-SET <option> <value> ..

As mentioned above there is a chance setting some configs like port or bind config would break the cluster so the command should be smart enough to detect the config to set the entire cluster. This command would help where the cases like: setting maxmemory config to be consistent in every node and it would be helpfull if we have a command which set all the node in cluster where the config value to be consistent.

Alternatives you've considered

Implement a script which can set the config each node on the cluster and set independently and verify later. or use any config monitor tools to do the same.

[madolson]

I think this is useful. I think it should also include ACLs as well. This has been a discussion point for cluster v2 for awhile, we should decide if legacy cluster should get it as well.

Some questions:

1. How is this data persisted? Is it stored in the config file, nodes.conf, or the rdb file?
2. When new nodes are added, how do we know which ones to propagate to these new nodes. Will we keep track of which nodes have have had data set with the cluster config?

[hpatro]

I was also thinking about a API to set cluster-wide configs and the mechanism to build this. In the existing state of things, how do we ensure which node has the latest value of the config and how other node fetch and apply it?

@dmitrypol I think you're all interested in this.

[nandihalli]

1. How is the data persisted - the config file feels like the place to store these configurations? Given that nodes.conf stores the cluster topology and attributes of each node? Whenever a node learns via gossip about the new config, it can also persist the same in the config file?
2. When new nodes are added, how do we know which ones to propagate - One way to do this would be to add versioning to these config values. As part of PINGs, we could have only the version numbers per config sent and when a node receives a PING from a node with a lower version config, it can conditionally generate the config payload with higher version and PONG it so the other node updates it. This prevents the actual config payload to be always included in the PING/PONG payload and just the versions and allows new nodes joining a cluster to learn about the latest configuration.

(2) relies on a external entity which can coordinate and determine the version to be used while updating a config value. This entity also can set this config on a subset of nodes (instead of just one node) for better reliability. If there is a need for this config to be quickly propagated, a node can also broadcast this new config when an admin sets the config payload with a newer version. WDYT?

[Shivshankar-Reddy]

Thank you everyone for the review and response!

How is this data persisted? Is it stored in the config file, nodes.conf, or the rdb file?

The config file would be better place as similar to normal config set and rewrite commands, with this approach I think we will not create another overhead for maintainance. To propagate the configs: I think that we take reference of how sentinel re-writes the replicaof config when the switch-over takes place, similarly we can send the specific config to every node in the cluster as part of gossip. (Other possible approach is that we can create a fake client and send the config).

When new nodes are added, how do we know which ones to propagate to these new nodes. Will we keep track of which nodes have have had data set with the cluster config?

we can add server.stat_starttime to nodes.conf so that every node will have each other's uptime, based on the age reference the new node could send propagate request. In other way to find oldest node: we can have the rank/score of the node as a new attribute in nodes.conf. With the reference of the node rank/score, newly added node can decide which is more suitable to request to propagate the configs. or As @nandihalli mentioned we could have the versioning and propagate based on version.

how do we ensure which node has the latest value of the config and how other node fetch and apply it?

Case1: If the config is needed to be updated during run time with config set in cluster mode and admin thinks that the config should be consistent across the nodes then it would be propagated entire cluster by using the cluster config-set command instead of regular config set.
Case2: To fetch entire configuration from oldest node we could try above things.

@madolson @hpatro @nandihalli This is just the beginning of the idea, and we need to implement the feature. Please feel free to share any other approaches or suggestions you might have.

[wuranxx]

Maybe we can execute config set command in all other nodes to implement CLUSTER CONFIG-SET .

Generally, the ACL configuration of all nodes in the cluster is the same. We can use the ACL configuration of this node to access other nodes and execute the command.

[wuranxx]

In fact, I believe this is an issue of information synchronization within the cluster, including but not limited to configuration, ACL, slot distribution, and cluster roles. In Cluster v2, there should be a central configuration storage that maintains global consistency by having each node either pull or receive updates via push notifications.

[Shivshankar-Reddy]

Hi @wuranxx , Thank you for the suggestions and I will consider these points for the implementation.