Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add datadog.cluster_agent.leader_election.is_leader #19308

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add datadog.cluster_agent.leader_election.is_leader #19308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c4ea08e
add datadog.cluster_agent.leader_election.is_leader
keisku Dec 25, 2024
fc74e60
changelog
keisku Dec 25, 2024
c2c5bcd
fix metadata
keisku Dec 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions datadog_cluster_agent/changelog.d/19308.added
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add `datadog.cluster_agent.leader_election.is_leader`.
1 change: 1 addition & 0 deletions datadog_cluster_agent/datadog_checks/datadog_cluster_agent/check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
'kubernetes_apiserver_kube_events': 'kubernetes_apiserver.kube_events',
'language_detection_dca_handler_processed_requests': 'language_detection_dca_handler.processed_requests',
'language_detection_patcher_patches': 'language_detection_patcher.patches',
'leader_election_is_leader': 'leader_election.is_leader',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this metric wasn't designed to be exposed directly.
Instead it was added to be use as "label join", see:

integrations-core/datadog_cluster_agent/datadog_checks/datadog_cluster_agent/check.py

Lines 100 to 105 in c2c5bcd

'label_joins': {
'leader_election_is_leader': {
'labels_to_match': ['*'],
'labels_to_get': ['is_leader'],
}
},

IMO, if we want to make a metric that exposes the leader status we should use the metric value (1 if leader, 0 if follower) instead of the using a label.
Maybe it can be done in this check thanks to a metric transformer function that will convert the is_leader label's value to a metric value.

you can find some transformer example in other openmetrics base check like

integrations-core/kube_controller_manager/datadog_checks/kube_controller_manager/kube_controller_manager.py

Line 231 in 642b2f9

def workqueue_transformer(self, metric, scraper_config):

Copy link
Contributor Author

@keisku keisku Dec 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, if we want to make a metric that exposes the leader status we should use the metric value (1 if leader, 0 if follower) instead of the using a label.

What do you think if I achieve that in Cluster Agent side? DataDog/datadog-agent#32511

We could keep datadog cluster check(python) simple with approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it would be nice to add the metric in the OOTB Dashboard too.

Will do!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember if the join works if the value is not 1, there might a special handling of these series (always 1 with only tags)

'rate_limit_queries_limit': 'datadog.rate_limit_queries.limit',
'rate_limit_queries_period': 'datadog.rate_limit_queries.period',
'rate_limit_queries_remaining': 'datadog.rate_limit_queries.remaining',
Expand Down
1 change: 1 addition & 0 deletions datadog_cluster_agent/metadata.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ datadog.cluster_agent.kubernetes_apiserver.emitted_events,count,,,,"Datadog even
datadog.cluster_agent.kubernetes_apiserver.kube_events,count,,,,"Kubernetes events processed by the kubernetes_apiserver check",0,datadog_cluster_agent,apiserver events,
datadog.cluster_agent.language_detection_dca_handler.processed_requests,count,,,,"The number of process language detection requests processed by the handler",0,datadog_cluster_agent,language detection processed requests,
datadog.cluster_agent.language_detection_patcher.patches,count,,,,"The number of patch requests sent by the patcher to the kube api server",0,datadog_cluster_agent,language detection patches,
datadog.cluster_agent.leader_election.is_leader,gauge,,,,"The label is_leader is true if the reporting pod is leader, equals false otherwise",0,datadog_cluster_agent,leader election is leader,
datadog.cluster_agent.secret_backend.elapsed,gauge,,millisecond,,The elapsed time of secret backend invocation,0,datadog_cluster_agent,secret backend elapsed time duration,
datadog.cluster_agent.tagger.stored_entities,gauge,,,,Number of entities stored in the tagger,0,datadog_cluster_agent,tagger stored entities,
datadog.cluster_agent.tagger.updated_entities,count,,,,Number of updates made to entities in the tagger,0,datadog_cluster_agent,tagger updated entities,
Expand Down
1 change: 1 addition & 0 deletions datadog_cluster_agent/tests/test_datadog_cluster_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
'kubernetes_apiserver.kube_events',
'language_detection_dca_handler.processed_requests',
'language_detection_patcher.patches',
'leader_election.is_leader',
'secret_backend.elapsed',
'tagger.stored_entities',
'tagger.updated_entities',
Expand Down
Loading