Multi-cluster Support for Elastic Agents #5538

GaneshSPatil · 2018-12-10T09:35:12Z

GaneshSPatil · 2018-12-10T09:56:26Z

Current Problems with EA plugins:

No Multi-Cluster support
Overriding plugin-setting fields at each individual profile causes a lot of duplication and makes it harder to maintain the profiles, as, change in the field value will require changing it at all profiles.

GaneshSPatil · 2018-12-10T10:02:30Z

Proposed solution:

Allow users to define named-clusters at the plugin-settings level and then refer to these clusters from elastic profiles level. Users can define any number of named clusters.
Even if just a small field needs to be overridden, users should create a new cluster.
Eg: On kubernetes, if a user wants to use two namespaces of the same cluster. A user needs to define two different clusters with same cluster URL but different namespaces.

bdpiprava · 2018-12-10T11:54:31Z

Approach 1:
- A new endpoint for the cluster configuration. This allows a user to provide cluster configuration globally(credentials, region etc..)
- Refer the cluster settings from an elastic profile.
- Plugin settings will contain only global configuration no credential related.
- Create agent, validate-profile and should assign work call will get profile and cluster credential in the request
Approach 2:
- Modify the existing plugin settings an take a culster settings as JSON, YML or Groovy
- Validate configuration on save of the settings for uniqueness of cluster config name and mandatory fields
- Rest of the stuff will remain the same. This is simple but implementation may vary from plugin to plugin

arvindsv · 2018-12-13T21:08:15Z

@bdpiparva My preference would be approach 2. Approach 1, beyond being more work, introduces a new concept and the settings will in three places: plugin-settings, cluster-settings and profile. Users have to (mostly manually) put it all together in their head.

Approach 2 puts the onus of the settings and cluster support on the plugins. That could cause it to be inconsistent for sure, but given that endpoint provides enough flexibility to implement this, they should do it themselves. That way, if "cluster" is the wrong terminology for this in some plugin, then a user won't have to map that concept into "cluster" to understand what is happening.

For instance: Consider the case mentioned by @GaneshSPatil above, where the cluster is the same, but the namespace is different. If we ask the k8s plugin to implement it, then it might use a term more appropriate than "cluster" and lead to less confusion.

arvindsv · 2018-12-17T12:25:17Z

@bdpiparva @GaneshSPatil One other aspect to consider: Today, it's really hard for users to understand that the settings are broken up over profiles and plugin settings. They're not easily reachable from one another.

The main reason is permissions/authorization. Only super admins can see plugin settings, since it can contain credentials, etc.

If we provide authorization at the level of the settings, then we can bring the settings into the elastic profile screen itself, in case a user has permissions to see both. That will improve the user's experience and will allow them to change it in one place. We can continue to have two "Save" buttons, since they are separate sections and make separate calls. However, it'll be visible in one place, which I think is better.

GaneshSPatil · 2019-02-28T06:04:27Z

@arvindsv -- Regardless of permissions,
I think we can show the "non-credential" fields in read-only mode on the elastic-profiles page.

Showing the cluster URL, the username used and basic configurations such as VPC, EC2 config should be OK?

GaneshSPatil · 2019-02-28T12:15:41Z

Elastic Agents Support for multi cluster

Currently, the plugin settings allows users to define only one cluster configuration.
Today (plugin-setting == cluster-configuration)

To allow defining multiple clusters, we are thinking of allowing users to define multiple plugin settings.

An admin user will define multiple named plugin settings and each profile will be associated with one of the pre-defined plugin settings.

Resolution of the associated plugin-settings with the elastic profile will be handled by GoCD core.

Implementation Details:

GoCD will start supporting (only Elastic agent plugin) settings as a list of plugin settings.
Each setting will be known as cluster setting having a unique name.
User will associate each elastic agent with the named cluster settings.
For the create agent request, GoCD server will pick the appropriate plugin settings associated with the elastic agent profile and will send both to the plugin for agent creation.
The plugin settings view will be a tabular view. Each tab will represent the unique named cluster settings.
The Plugin Status report page view will be tabular view. Each tab will represent the status report of the cluster.

Few points:

But it still has duplication?

Yes, Allowing users to define multiple named clusters will result into some duplication of common properties based on the usecase.
This duplication will be lesser and manageble than the allowing overrides at individual profile level.
GoCD Server URL is always the same across all cluster settings?

Yes. GoCD Server URL is the only common field which value will be same across all the named cluster settings.
Ideally, Plugins shouldn't be asking for this information from the users. GoCD Server and plugin internally should figure out the correct GoCD server URL.
Should we allow overridding cluster properties at elastic profile?
- No. If a user wants to change the cluster settings, they should create a new setting.
- Allowing users to override properties at profile level will lead to lot of duplications across multiple profiles.
- Also, profiles defining cluster configuration causes computation of status report pages very difficult.
Why not Cluster as a GoCD entity?
- Defining a new entity for Elastic Agents brings in more complexities as the configurations will be defined at 3 places (cluster, plugin settings and profile) and the heirarchy of overridden stuff is hard to understand.
- Migration of existing settings and profiles is difficult.
Why not cluster as a concept in plugin settings?
- This will make multi-cluster support a problem of plugin authors. Not all the plugins will support this capability.
- The implementation to support multi-cluster will differ based on how plugin authors choose to implement it.
Plugin Settings Migration?

The current plugin settings will be named "default" and will be associated with all the existing profiles.
Is cluster the right term?

May be not. In the context of kubernetes it seems right, but for Openstack/Azure, the settings are just a bunch of configurations where the VMs are brought up to run the agent.

WHats the UX change?

Plugin Settings View
- Plugin Settings view changes to a tabular view with the existing settings migrated to the default cluster settings.
- (Maybe) Show the list of profiles consuming the current cluster settings.
Elastic Profiles
- Existing elastic profiles will be associated with the default cluster settings.
- The read only view of associated cluster settings will be displayed inline for the users to know which cluster the current elastic agent will be run on.
- Link from read only cluster settings view to go to plugin settings for making any changes.
  Do no allow inline edit of cluster settings from profile, as the same cluster will be used at other places, modifying the cluster inline will affect other profiles.
Plugin Status Report Page

Plugin Status report page changes to a tabular view showing the status reports of each cluster.
Agent Status Report Page

Agent Status report page remains unchanged.

Unknowns?

Changing plugin settings call only for elastic agents.
Plugin settings call is common across all the plugins. It needs to change to support multiple plugin settings only for EA extension.
Existinh Plugin settings and Elastic Profile Migration to support multi-cluster model.

\cc: @gocd/committers @arvindsv @Smarker @varshavaradarajan

ketan · 2019-02-28T14:50:19Z

* GoCD Server URL is always the same across all cluster settings?

I don't think this is necessarily the case. It should be possible to access the same host via different hostnames, in between different clusters. We ourselves do this. We use build.gocd.io for hosts inside AWS and build.gocd.org for elastic hosts outside of AWS.

bdpiprava · 2019-03-01T06:59:49Z

To allow defining multiple clusters, we are thinking of allowing users to define multiple plugin settings.

I think this should be a plugins capability whether the plugin will support the multiple settings or not. GoCD still does the migration even if the plugin doesn't support multiple configurations and associate it with all the profiles. Only the creation of the second config won't be allowed.

The Plugin Status report page view will be tabular view. Each tab will represent the status report of the cluster.

Leave this to the plugin author. We should provide all settings and let them figure out the view. @GaneshSPatil WDYT?

GoCD Server URL is always the same across all cluster settings?

No need to handle this as there is server info processor for that. If plugin author really needs the server URL other then what server info is returning then that can be defined as part of the plugin settings.

GaneshSPatil · 2019-03-01T08:59:35Z

I think this should be a plugins capability whether the plugin will support the multiple settings or not. GoCD still does the migration even if the plugin doesn't support multiple configurations and associate it with all the profiles. Only the creation of the second config won't be allowed.

Yeah, make sense. As the settings fields are completely in control of the plugin, the plugin can decide if they want to support mutli-cluster via multiple plugin-settings or some other mechanism.

maheshp · 2019-03-04T04:06:55Z

Irrespective of where the cluster settings are stored I would prefer not going to multiple places to configure and view the cluster settings and the profile.

Ideally if I am configuring an elastic agent plugin, I would prefer an option to configure a cluster and create multiple profiles for the same. The cluster configuration and the related profiles to should be grouped together which would make viewing the entire cluster + profile easier.

GaneshSPatil · 2019-03-04T07:05:48Z

EA Support for Multi Cluster using Cluster Configurations

GoCD introduces cluster as a new concept in GoCD core. All the elastic agent plugins will define a named cluster as part of cluster configurations instead of defining it using plugin settings. The named cluster config will be associated with the elastic agent profile. The users can define as many named clusters as they want.

The newer elastic agent extension will stop supporting defining and consuming plugin settings.

Tasks:

Define Elastic Agent Extension v5 (Define Elastic Agent Extension v5 #5937)
Migrate existing Elastic Agent plugins to implement v5 extension
Support for cluster config entity in GoCD (Migration + CRUD) (Support for cluster config entity in GoCD #5938)
Bump Up Elastic Agent Profiles API to support clusterProfileId
Migrate Existing EA profiles to point to default cluster config
Fix Elastic Agent Profiles UI to work with cluster config.
Remove ability to configure plugin settings for plugins implementing EA extension v5

\cc: @gocd/committers @sheroy

GaneshSPatil · 2019-03-04T12:39:43Z

@arvindsv @maheshp @bdpiparva @ketan -- How about calling the cluster "Deployment Profile"?

Deployment profile defines where the Elastic Agents will be deployed/run.
A deployment profile can either be a cluster configuration or a set of properties that define where the agent will run (eg: VM).

varshavaradarajan · 2019-03-04T18:13:00Z

@GaneshSPatil - this is what I understood from the above comments:

For each EA plugin, configure plugin settings (agent autoregister timeout?, properties common to all clusters, if any)
For each EA plugin, configure clusters or "Deployment profiles", which is stored in the db along with plugin settings?
For each EA profile, refer to the already defined cluster-config by id and configure profile level details like agent image, memory, env vars
For each job, use the EA profile id

Is that right?

I find that for EA, the amount of setup that is to be done seems to increase a lot. In order to avoid duplication of cluster related information, a new entity must now be configured. Its good that the cluster info that is currently in plugin settings will be migrated to the cluster-config global entity. But this still becomes a complex workflow, I think. Because it involves at least 3 levels of referencing. I may have misunderstood.

This is just an idea. Maybe I haven't thought it out fully, please feel free to ignore it:

Get rid of plugin settings. All the info in the plugin settings can be migrated over to the profile?
Change EA profile settings from form to a json (similar to what you have for k8s pod yaml). Let the id and plugin id be the only mandatory data to be entered. Let all of the config be template-able. Meaning, allow for each value to be something that can be overridden (like in pod yaml, there is the {{ POD_POSTFIX }} that users should change) at the job level. Those who are okay with some setup info duplication can clone existing profiles, change only the values they want and store as a new EA profile config.
In the job settings, when a profile is referenced, display the json for view. When the profile is referenced, check that the plugin's required fields are actual values ( not templated like {{ POD_POSTFIX }} ). If some values are to be overridden, show the fields that the plugin deemed mandatory for users to enter the values. You can also allow for users specifying additional key values, but they can be ignored if the key isn't present in the json? Do not perform any validation beyond that. An error can be displayed on the job console if the agent can't be brought up for the job.
So, for pipeline config, along with elastic agent id, sometimes, extra overridden info needs to be stored. This involves changing the xsl to have an additional optional element (let's call it EAProfileOverride, ask Duck for naming) when elastic agent profile id is specified. It also involves changing the pipeline config API, config-repos.
Because this is at a job level, templates will be involved. But, I think that pipeline parameters should be disallowed for now. That is, the EA profile values that are going to be overridden at the job level should not accept pipeline parameters. This involves changing the template config API to include EAProfileOverride.

May be this is too complicated as well, I don't know. Just a suggestion.

GaneshSPatil · 2019-03-06T05:36:03Z

@varshavaradarajan -- Extremely sorry for the above response.
I think I wasn't clear enough with the decided approach/intention.

We are NOT planning to keep 3 entities (plugin-settings, clusters, and profiles) to specify configurations of Elastic agents. We will get rid of plugin-settings in favor of cluster configuration.

So, Users can define multiple clusters and associate a cluster with the profile.
A cluster configuration defines the environment where the elastic agent will run. And the elastic profiles define which elastic agent and how to run.

All the info in the plugin settings can be migrated over to the profile?

Defining the cluster configurations inline with the profile might cause followin issues:

Lot of configuration duplication at the each profile.
This was the main reason for allowing users to configure cluster at plugin settings.
More complexity at the plugin implementation as the on each request from the request information we need to create 2 entities cluster and the profile to deal with the deployment environment and the elastic agent application to deploy.
Status reports are a mess to compute with inline profiles.

Change EA profile settings from form to a json

This depends on the plugin as the view and the configuration properties are defined by the plugin. We can define this as a guideline but can't make it a restriction.
The other implementation for this could be, don't ask for view from the plugin and use the JSON/YAML template view and only rely on the fields from the plugin. But that seems too much of work for GoCD.

Those who are okay with some setup info duplication can clone existing profiles, change only the values they want and store as a new EA profile config.
and
In the job settings, when a profile is referenced, display the json for view. When the profile is referenced, check that the plugin's required fields are actual values ( not templated like {{ POD_POSTFIX }} ). If some values are to be overridden, show the fields that the plugin deemed mandatory for users to enter the values. You can also allow for users specifying additional key values, but they can be ignored if the key isn't present in the json? Do not perform any validation beyond that. An error can be displayed on the job console if the agent can't be brought up for the job.

The real reason for introducing a new entity was to support multi-cluster support. The suggested approach would let users configure multiple clusters using field overrides, but this is still possible to do at the profile level (without a change in core).
Allowing overrides at profile or at the job-config will result into alot of duplication and unnecessary complexities at multiple places.

\cc: @arvindsv @sheroy

sheroy · 2019-03-07T23:00:37Z

Sorry I'm late to the party.

@varshavaradarajan I think your idea of a template based EA profile configuration yaml at a job level is interesting. I think it's a fresh take on configuring EAs, and I agree it reduces the number of levels of configuration.

I feel that introducing the concept of named clusters as a separate concept is still valuable for the following reasons:

I would rather see a list of named cluster configurations in one place rather than spread across job definitions, so I can get a view of the set of configured clusters.
It makes the status report page simpler to grok and manage, as @GaneshSPatil mentioned.
I think cluster configuration and EA profile configuration are separate concerns and don't really need to mixed. They could be configured by different individuals with different levels of authorization.

\cc: @arvindsv

sheroy · 2019-03-07T23:04:48Z

@arvindsv @maheshp @bdpiparva @ketan -- How about calling the cluster "Deployment Profile"?

I like "Deployment Profile" or "Deployment Target"

arvindsv · 2019-03-12T19:01:10Z

@sheroy I think deployment profile/target can be confusing, given the fact that "deployment" means something else to most people. They'll be thinking about their own apps' deployment.

Cluster profile is also somewhat ambiguous in some cases. But, my preference is still that.

sheroy · 2019-03-12T19:07:38Z

@arvindsv, I agree, let's go with "Cluster Profile".

gocd#5538) * Add clusterProfileId attribute on 'profile' tag * Define a default clusterProfile under elastic/clusterProfiles

* Mandate specifying cluster_profile_id field as part of Elastic Agent Profiles API v2 * Validate existence of referenced cluster profile. * Validate plugin id matches for elastic profile and referenced cluster profile. * Move API validation to entity validations for consistency.

* Always Show cluster profile id dropdown for elastic agent profiles modal. * Set first cluster profile id on model while creation/update of profile. * Fix elastic agent profiles usage

Change following plugin api call: * 'get-profile-metadata' => 'get-elastic-agent-profile-metadata' * 'get-profile-view' => 'get-elastic-agent-profile-view' * 'validate-profile' => 'validate-elastic-agent-profile'

…ocd#5538) * Perform a full config save validations to check: - Valid cluster profiles. - Valid elastic agent profiles. - Valid elastic agent profile reference from job config. * Mark plugin invalid in case if it fails to migrate the existing config.

…5538) * Perform a full config save validations to check: - Valid cluster profiles. - Valid elastic agent profiles. - Valid elastic agent profile reference from job config. * Mark plugin invalid in case if it fails to migrate the existing config.

* Config saves were failing will reason null when we have following setup: - Install a v4 extension based elastic agent plugin - Define a cluster profile with some properties belonging to that v4 elastic agent plugin. * In case of full config save, GoCD will try to encrypt secure value of plugin specified profiles using 'PluginProfile.encryptSecureConfigurations'. As the v4 EA plugin did not have the capability to read the cluster profile configurations, it was failing with null pointer exception. Fix: * Verify Configurations exists on cluster profiles, as v4 plugins will not support cluster profile configurations.

…erver ping request (gocd#5538) * Do not send all available cluster profiles from different plugins.

…erver ping request (#5538) * Do not send all available cluster profiles from different plugins.

…lugin status report (gocd#5538) * Do not send all available cluster profiles from different plugins.

…status report (gocd#5538) * Clicking on agent hostname used to render agent status report page for elastic agents. * With elastic agent extension v5, cluster profile properties being mandatory for agent status report request and there is no way to find the cluster profile properties for an elastic agent, render normal job run history page for all the agents. * This change will always cause all the hostname links to render job run history page regardless of agent being elastic or normal.

…status report (#5538) * Clicking on agent hostname used to render agent status report page for elastic agents. * With elastic agent extension v5, cluster profile properties being mandatory for agent status report request and there is no way to find the cluster profile properties for an elastic agent, render normal job run history page for all the agents. * This change will always cause all the hostname links to render job run history page regardless of agent being elastic or normal.

rajiesh · 2019-04-26T04:43:53Z

Verified on 19.3.0 (8959-259d1f79e361875995072e5c1fcc9cbda7ccc600). Enhancing server health message to show deprecation warnings will be tracked by separate issue

GaneshSPatil added enhancement elastic-agents labels Dec 10, 2018

GaneshSPatil added this to the Release 19.1.0 milestone Dec 10, 2018

rajiesh modified the milestones: Release 19.1.0, Release 19.2.0 Jan 22, 2019

maheshp modified the milestones: Release 19.2.0, Release 19.3.0 Feb 26, 2019

GaneshSPatil added the elastic-agent-multi-cluster-support label Mar 4, 2019

bdpiprava added the epic label Mar 6, 2019

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 15, 2019

Remove elastic agent profiles API v1 (gocd#5538)

bc95085

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 16, 2019

Pass along all cluster-profiles belonging to that plugin as part of s…

5cb7795

…erver ping request (gocd#5538) * Do not send all available cluster profiles from different plugins.

GaneshSPatil added a commit that referenced this issue Apr 16, 2019

Pass along all cluster-profiles belonging to that plugin as part of s…

0580a33

…erver ping request (#5538) * Do not send all available cluster profiles from different plugins.

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 16, 2019

Pass along all cluster-profiles belonging to that plugin as part of p…

1d2f3ac

…lugin status report (gocd#5538) * Do not send all available cluster profiles from different plugins.

GaneshSPatil mentioned this issue Apr 17, 2019

Fix agent status reports links #6155

Merged

GaneshSPatil added a commit that referenced this issue Apr 18, 2019

Remove elastic agent profiles API v1 (#5538)

0c5c846

bdpiprava mentioned this issue Apr 19, 2019

Support for multiple clusters gocd/kubernetes-elastic-agents#43

Closed

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 22, 2019

Remove plugin id from elastic agent profiles API (gocd#5538)

ddba073

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 22, 2019

Remove plugin id from elastic agent profiles API (gocd#5538)

9053d22

GaneshSPatil added a commit that referenced this issue Apr 22, 2019

Remove plugin id from elastic agent profiles API (#5538)

65f401c

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 22, 2019

Rename profiles to agentProfiles (gocd#6170)(gocd#5538)

75a7f35

GaneshSPatil added a commit to GaneshSPatil/gocd that referenced this issue Apr 22, 2019

Rename profiles to agentProfiles (gocd#6170)(gocd#5538)

c5b6b22

GaneshSPatil added a commit that referenced this issue Apr 22, 2019

Rename profiles to agentProfiles (#6170)(#5538)

2a9eb86

GaneshSPatil mentioned this issue Apr 22, 2019

Secure plugin properties are stored in plain text on server restart for EA plugins #6184

Closed

rajiesh closed this as completed Apr 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-cluster Support for Elastic Agents #5538

Multi-cluster Support for Elastic Agents #5538

GaneshSPatil commented Dec 10, 2018 •

edited

Loading

GaneshSPatil commented Dec 10, 2018

GaneshSPatil commented Dec 10, 2018

bdpiprava commented Dec 10, 2018

arvindsv commented Dec 13, 2018

arvindsv commented Dec 17, 2018

GaneshSPatil commented Feb 28, 2019

GaneshSPatil commented Feb 28, 2019

But it still has duplication?

GoCD Server URL is always the same across all cluster settings?

Should we allow overridding cluster properties at elastic profile?

Why not Cluster as a GoCD entity?

Why not cluster as a concept in plugin settings?

Plugin Settings Migration?

Is cluster the right term?

Plugin Settings View

Elastic Profiles

Plugin Status Report Page

Agent Status Report Page

ketan commented Feb 28, 2019

bdpiprava commented Mar 1, 2019

GaneshSPatil commented Mar 1, 2019

maheshp commented Mar 4, 2019

GaneshSPatil commented Mar 4, 2019 •

edited

Loading

GaneshSPatil commented Mar 4, 2019

varshavaradarajan commented Mar 4, 2019 •

edited

Loading

GaneshSPatil commented Mar 6, 2019

sheroy commented Mar 7, 2019

sheroy commented Mar 7, 2019

arvindsv commented Mar 12, 2019

sheroy commented Mar 12, 2019

rajiesh commented Apr 26, 2019

Multi-cluster Support for Elastic Agents #5538

Multi-cluster Support for Elastic Agents #5538

Comments

GaneshSPatil commented Dec 10, 2018 • edited Loading

Issue Type

Tasks:

GaneshSPatil commented Dec 10, 2018

GaneshSPatil commented Dec 10, 2018

bdpiprava commented Dec 10, 2018

arvindsv commented Dec 13, 2018

arvindsv commented Dec 17, 2018

GaneshSPatil commented Feb 28, 2019

GaneshSPatil commented Feb 28, 2019

Elastic Agents Support for multi cluster

Implementation Details:

Few points:

But it still has duplication?

GoCD Server URL is always the same across all cluster settings?

Should we allow overridding cluster properties at elastic profile?

Why not Cluster as a GoCD entity?

Why not cluster as a concept in plugin settings?

Plugin Settings Migration?

Is cluster the right term?

WHats the UX change?

Plugin Settings View

Elastic Profiles

Plugin Status Report Page

Agent Status Report Page

Unknowns?

ketan commented Feb 28, 2019

bdpiprava commented Mar 1, 2019

GaneshSPatil commented Mar 1, 2019

maheshp commented Mar 4, 2019

GaneshSPatil commented Mar 4, 2019 • edited Loading

EA Support for Multi Cluster using Cluster Configurations

Tasks:

GaneshSPatil commented Mar 4, 2019

varshavaradarajan commented Mar 4, 2019 • edited Loading

GaneshSPatil commented Mar 6, 2019

sheroy commented Mar 7, 2019

sheroy commented Mar 7, 2019

arvindsv commented Mar 12, 2019

sheroy commented Mar 12, 2019

rajiesh commented Apr 26, 2019

GaneshSPatil commented Dec 10, 2018 •

edited

Loading

GaneshSPatil commented Mar 4, 2019 •

edited

Loading

varshavaradarajan commented Mar 4, 2019 •

edited

Loading