RFE: [multicluster] make service discovery equal for all clusters by eliminating the extendedServiceReferences attribute #3494
Comments
|
Kostas is going to add a better solution |
|
Created [CONTCNTR-4809] for internal tracking. |
|
Kostas is on PTO, pasting his comments next: He also thinks that the configuration should be more coherent But he didn´t like my approach because when pushing configs with a CI/CD tools configurations are pushed equally to all clusters and with my approach the service names need to be different. I agree with this. Instead he recommends an approach as follows Where clusterName could be also "any": But I think that "all" is a better keyword as "any" might suggest that only one cluster is chosen |
|
I like the Kostas approach. It's simpler and straight forward for application owner to understand which cluster and service the load balancer is using. Please also implement this in transport server as well. |
|
Another iteration:
The effective pool ratio for each service/cluster combination is calculated as: where the
And
One more comment about the weight calculation: Even if no service exists and the cluster matches a service/clusterX entry or "any" the weight is added to it. Just as if a service existed but didn´t have any backend or the backends were marked down by the BIG-IP monitor. In which case a re-balancing should be done. Details/handling exceptions:
Syntax for |
SUMMARY
EXAMPLE: a manifest is worth thousands wordsAn example meant to show all the syntax of the proposal, rather than a real world scenario. Note that "weights" have a total of 100 each, just to make calculations easy. Likewise the clusterWeights for each backend also have a total of 100. Using this example and assuming 4 clusters defined in CIS:
Overall the traffic sent to each service backend cluster would be: where the
if the "clusterWeight" attribute is not used it means the percentage for the backend is split across all the clusters defined in CIS. TransportServerSyntax for An example for TransportServer without the A/B deployment strategy would be as follows: Facilitating transition from single cluster to multi-clusterNote wrt section for backend-b2 above: could be also be written as: Where all clusters of backend-b2 would receive the same weight, regardless of the value indicated That is: a customer with many manifests defined only using the weight attribute would seamlessly start using multi-cluster without modification. CIS configuration & Service discoveryIt is important that unlike the current behaviour, the proposal above should behave no matter if CIS is configured standalone or in HA mode or in which cluster is deployed. It is also important to remark that the proposal above, for a given backend, CIS does service discovery in all clusters unless clusterWeight is set to 0 for a cluster. |
|
@alonsocamaro , This is how we will be supporting this feature: Controlling the service discovery in MultiClusterCase-1: Let's customer want to enable the discovery in specified clusters only without weights. Virtual Server: Transport server: Case-2: Let's say you want to discover service in all the clusters, you define the resources as follows. Virtual Server: Transport server: Case-3: Without extendedServiceReference, In this case we will follow the existing behaviour and do the service discovery in primary and secondary cluster only. Cluster Traffic Distribution UseCasesCase-4: Let's say you want to distribute the traffic to cluster1 and cluster2 at service level, this is how you define the resources. Virtual Server: Transport server: Case-5: Let's say customer wants to use alternate backend in multi-cluster scenario, you define the resources as follows. Virtual Server: Transport server: Case-6: Let's say customer have forgot to put weights for some service. Virtual Server: Transport server: In this case, we will consider default weight as 100 and traffic will be calculated accordingly. so that cluster-1 will receive 100/120 % traffic and cluster-2 will receive 20/120% traffic. Note:-
|
|
If I compare the current two proposals: The proposal by me and @skenderidis
On the other hand, the alternate proposal by @vklohiya :
|
|
@alonsocamaro , It will support the different namespaces with alternate backend as well. Here is the example for same: Virtual Server: Transport server: Note:- We will still be supporting the properties like servicePort, service, namespace under the extendedServiceReferences. But it will not be the mandatory parameter any more. we are converting them to optional parameters, which will give much flexibility. If these are not defined we will inherit it from the pool level. It will also avoid the confusion regarding alternateBackend weight distribution and will simplify overall weight distribution under extendedServiceReferences for multi-cluster CIS. And with this approach no new api version is needed for CRDs, which overall reduces the burden of maintaining two api versions for CRDs. |
|
Here is my take Starting with ConfigMap: Here is current configMap configuration Here is new proposal: Option 1: I noticed split brain behavior when CIS having issues communicating with each other which needs to be addressed. This will be a separate discussion. Option 2: In this options instead of using the Kubernetes API to check the status. When deploying Primary CIS deploy a service and load balancer that points to Primary CIS and use it for other CIS to connect to Primary. Now moving on with CRD's: In below example: Here is the new proposal: If it is same cluster multiple services: Active-Active: Active-Passive: Ratio: Standalone: Similar for transport server as well. |
|
@avinashchundu9 , Thanks for providing the detailed input. Here are my take on it:
|
|
@alonsocamaro , Here is the final proposal from engineering team after the internal discussions. Controlling the service discovery in MultiClusterCase-1: Let's say you want to discover service in all the clusters, you define the resources as follows. Here service discovery is implicit for all cluster. Virtual Server: Transport server: Case-2: Let's customer want to enable the discovery in specified clusters only without weights. Virtual Server: Transport server: Cluster Traffic Distribution UseCasesCase-3: Let's say you want to distribute the traffic to cluster1 and cluster2 at service level, this is how you define the resources. Virtual Server: Transport server: Case-4: Let's say customer wants to use alternate backend in multi-cluster scenario, you define the resources as follows. Virtual Server: Transport server: Case-5: Let's say customer have forgot to put weights for some service. Virtual Server: Transport server: In this case, we will consider default weight as 100 and traffic will be calculated accordingly. so that cluster-1 will receive 100/120 % traffic and cluster-2 will receive 20/120% traffic. Case-6 Traffic distribution with different namespaces with alternate backend as well. Here is the example for same: Virtual Server: Transport server: Note:- We will still be supporting the properties like servicePort, service, namespace under the multiClusterReferences. But it will not be the mandatory parameter any more. we are converting them to optional parameters, which will give much flexibility. If these are not defined we will inherit it from the pool level.
|
Title
RFE: make service discovery equal for all clusters by eliminating the extendedServiceReferences attribute
Description
At present a service definition behaviour depends on the global configmap, for example using the next:
the service discovery in the non-extendedServiceReferences section depends whether the customer is using a CIS HA or Standalone configuration. Furthermore, it is required for DevOps team to know in which clusters CIS is deployed. Also, It is mandatory to specify the cluster name in the extendedServiceReferences section.
This makes service discovery:
Instead, the cluster configuration should be kept to the global CM. The cluster configuration should be transparent to the application config by default. From a Service perspective it is irrelevant if its deployed in a cluster which has CIS or not.
Service discovery should be performed in all clusters configured in the global CM, treating all clusters equally. Hopefully this would simplify the code as well.
Actual Problem
extendedServiceReferences doesn´t add any benefit, just makes CIS less intuitive and less user friendly.
CIS placement and Services placement are independent concepts and using extendedServiceReferences we bind these without any advantage to CIS or to the customer.
Solution Proposed
The customer would just specify the following for an A/B use case amongst 3 clusters:
The above example shows that if customers want specific cluster placement this can be done even more easily not using extendedServiceReferences by using different service names. It is optional to know the clusters, depending if the customer wants to do specific placement.
Service discovery would search for these services in all clusters. It is up to the DevOps team where the services are actually deployed.
Additional context
Take into account existing #3493 when considering this
The text was updated successfully, but these errors were encountered: