design/impl: control plane metrics monitoring #1955

Xunzhuo · 2023-10-11T13:51:03Z

What type of PR is this?

What this PR does / why we need it:

design: control plane metrics monitoring

Which issue(s) this PR fixes:

Fixes #700

https://gateway.xunzhuo.cafe/latest/design/eg-metrics

codecov · 2023-10-11T13:56:06Z

Codecov Report

Merging #1955 (bf0b87a) into main (fa55804) will decrease coverage by 1.10%.
The diff coverage is 36.75%.

@@            Coverage Diff             @@
##             main    #1955      +/-   ##
==========================================
- Coverage   65.53%   64.43%   -1.10%     
==========================================
  Files          92      101       +9     
  Lines       13528    13965     +437     
==========================================
+ Hits         8865     8999     +134     
- Misses       4114     4407     +293     
- Partials      549      559      +10

Files	Coverage Δ
api/v1alpha1/validation/validate.go	`74.30% <100.00%> (ø)`
...ternal/infrastructure/kubernetes/infra_resource.go	`81.65% <100.00%> (+2.26%)`	⬆️
internal/provider/kubernetes/predicates.go	`54.78% <100.00%> (ø)`
internal/provider/kubernetes/routes.go	`56.00% <100.00%> (ø)`
internal/xds/translator/runner/runner.go	`69.81% <100.00%> (+0.58%)`	⬆️
internal/admin/server.go	`56.75% <66.66%> (ø)`
internal/provider/kubernetes/filters.go	`52.30% <66.66%> (ø)`
internal/xds/translator/extension.go	`74.66% <93.75%> (+2.27%)`	⬆️
internal/cmd/server.go	`17.68% <0.00%> (-0.37%)`	⬇️
internal/gatewayapi/runner/runner.go	`22.60% <25.00%> (-0.41%)`	⬇️
... and 13 more

... and 1 file with indirect coverage changes

Xunzhuo · 2023-10-11T14:11:40Z

I have set up metrics pkg basement which can be easily used in different component, and we support push/pull metrics.
But I want to know what/where metrics we should build in each components:

Provider ( already have controller-runtime metrics )
GWAPI Translator
Infra Manager
xDS Translator
Extension Manager

I would like to hear from @envoyproxy/gateway-maintainers .

arkodg · 2023-10-11T23:40:46Z

thanks for kickstarting this @Xunzhuo, hoping this lands in v0.6.0

sharing my thoughts on what we should be measuring

Input Resources from K8s
- kubernetes_provider_reconcile_duration_seconds - gauge that measures how much time was spent in the reconciler , note this may already be provided by controller-runtime (haven't looked into this yet)
Translate Resources
- <runner_duration_seconds - gauge that measures how much time was spent in every runner during a subscribe event e.g. in here, with a unique label for every runner
Output Resources to Envoy
- xds_push_duration_seconds - gauge that measures time taken to push xds from xds server to envoy
Write Status to API Server
- <resource_status_write_kubernetes_duration_seconds> - gauge that measures how much time it took to write status back to k8s server (this func) with a unique label for every resource type

request to add a design doc that highlights a few things, which are already answered by your PR

why this is needed
what are we exactly measuring
how we are observing the metrics

arkodg · 2023-10-12T19:55:09Z

site/content/en/latest/design/eg-metrics.md

@@ -0,0 +1,634 @@
+---
+date: 2023-10-10


ptal @envoyproxy/gateway-reviewers @envoyproxy/gateway-maintainers

site/content/en/latest/design/eg-metrics.md

site/content/en/latest/design/pprof.md

Xunzhuo · 2023-10-16T03:00:14Z

required: #1967

arkodg · 2023-10-16T19:12:53Z

api/v1alpha1/envoygateway_types.go

@@ -69,6 +69,12 @@ type EnvoyGatewaySpec struct {
 	// +optional
 	Debug *EnvoyGatewayDebug `json:"debug,omitempty"`

+	// Telemetry defines the desired control plane telemetry related abilities.
+	// If unspecified, the telemetry is used with default configuration.


what is the default configuration ? is it opt in ?

arkodg · 2023-10-16T19:14:12Z

api/v1alpha1/envoygateway_helpers.go

+		}
+		return e.Telemetry
+	}
+	e.Telemetry = DefaultEnvoyGatewayTelemetry()


why are we enabling prometheus metrics by default ?

I think most people in Kubernetes are/will use prometheus, it should always enable by default.

arkodg · 2023-10-16T19:15:34Z

api/v1alpha1/envoygateway_metric_types.go

+	//
+	// +optional
+	// +kubebuilder:default=true
+	Enable bool `json:"enable,omitempty"`


why do we need an explicit enable flag, isn't the instantiation of EnvoyGatewayPrometheusProvider an implicit enable ?

arkodg · 2023-10-16T19:32:53Z

api/v1alpha1/envoygateway_metric_types.go

+	// +kubebuilder:default=OpenTelemetry
+	Type MetricSinkType `json:"type"`
+	// Host define the sink service hostname.
+	Host string `json:"host"`


should we start using urls here instead of using 3 fields ?
cc @zirain

arkodg · 2023-10-16T19:55:53Z

internal/message/metrics.go

+	)
+
+	// metrics label definitions
+	// component is which component the update belong to.


you cannot map a single component to watchable, because it involves a subscriber component and a publisher component.
Also we internally call components runners

arkodg · 2023-10-16T19:57:17Z

internal/message/metrics.go

+
+import "github.com/envoyproxy/gateway/internal/metrics"
+
+var (


these metrics need to be split up in terms of subscribe and publish

arkodg · 2023-10-16T20:01:03Z

internal/metrics/api.go

@@ -0,0 +1,73 @@
+// Copyright Envoy Gateway Authors


can we take this library out as a separate PR ?

Sure, I would like to separate it with 3 PR:

design + api

library

metrics instrumentation

Signed-off-by: bitliu <bitliu@tencent.com>

Xunzhuo · 2023-10-17T08:42:45Z

Close in favor of PR separations

Xunzhuo force-pushed the design-cp-obs-metrics branch 2 times, most recently from 2eea36e to 0b9f85e Compare October 11, 2023 13:55

Xunzhuo force-pushed the design-cp-obs-metrics branch 2 times, most recently from eb8d7cf to 5ca340d Compare October 11, 2023 14:07

Xunzhuo force-pushed the design-cp-obs-metrics branch from 5ca340d to 542d663 Compare October 11, 2023 14:13

Xunzhuo force-pushed the design-cp-obs-metrics branch 13 times, most recently from 29ccc76 to 83d8d2e Compare October 12, 2023 07:07

Xunzhuo added the area/observability Observability related issues label Oct 12, 2023

Xunzhuo added this to the 0.6.0-rc1 milestone Oct 12, 2023

Xunzhuo force-pushed the design-cp-obs-metrics branch 2 times, most recently from c398366 to aff1fa3 Compare October 12, 2023 09:21

arkodg reviewed Oct 12, 2023

View reviewed changes

site/content/en/latest/design/eg-metrics.md Outdated Show resolved Hide resolved

Xunzhuo force-pushed the design-cp-obs-metrics branch 2 times, most recently from a13322e to 02ad089 Compare October 13, 2023 02:43

zirain reviewed Oct 13, 2023

View reviewed changes

site/content/en/latest/design/pprof.md Outdated Show resolved Hide resolved

Xunzhuo force-pushed the design-cp-obs-metrics branch 5 times, most recently from 19ebd3a to 2b1963b Compare October 13, 2023 10:02

Xunzhuo changed the title ~~design: control plane metrics monitoring~~ design/impl: control plane metrics monitoring Oct 13, 2023

Xunzhuo marked this pull request as ready for review October 13, 2023 10:29

Xunzhuo requested a review from a team as a code owner October 13, 2023 10:29

Xunzhuo force-pushed the design-cp-obs-metrics branch 2 times, most recently from af6b271 to 4f90b09 Compare October 16, 2023 02:36

Xunzhuo force-pushed the design-cp-obs-metrics branch 5 times, most recently from 480123f to 643eeff Compare October 16, 2023 12:23

arkodg reviewed Oct 16, 2023

View reviewed changes

Xunzhuo force-pushed the design-cp-obs-metrics branch 3 times, most recently from 82ae3fb to aec4cb8 Compare October 17, 2023 07:54

design/impl: control plane metrics monitoring

bf0b87a

Signed-off-by: bitliu <bitliu@tencent.com>

Xunzhuo force-pushed the design-cp-obs-metrics branch from aec4cb8 to bf0b87a Compare October 17, 2023 08:00

Xunzhuo closed this Oct 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

design/impl: control plane metrics monitoring #1955

design/impl: control plane metrics monitoring #1955

Xunzhuo commented Oct 11, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading

Xunzhuo commented Oct 11, 2023

arkodg commented Oct 11, 2023

arkodg Oct 12, 2023

Xunzhuo commented Oct 16, 2023

arkodg Oct 16, 2023

arkodg Oct 16, 2023

zirain Oct 17, 2023

arkodg Oct 16, 2023

arkodg Oct 16, 2023

arkodg Oct 16, 2023

arkodg Oct 16, 2023

arkodg Oct 16, 2023

Xunzhuo Oct 17, 2023 •

edited

Loading

Xunzhuo commented Oct 17, 2023 •

edited

Loading


		import "github.com/envoyproxy/gateway/internal/metrics"

		var (

design/impl: control plane metrics monitoring #1955

design/impl: control plane metrics monitoring #1955

Conversation

Xunzhuo commented Oct 11, 2023 • edited Loading

codecov bot commented Oct 11, 2023 • edited Loading

Codecov Report

Xunzhuo commented Oct 11, 2023

arkodg commented Oct 11, 2023

Choose a reason for hiding this comment

Xunzhuo commented Oct 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xunzhuo Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

Xunzhuo commented Oct 17, 2023 • edited Loading

Xunzhuo commented Oct 11, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading

Xunzhuo Oct 17, 2023 •

edited

Loading

Xunzhuo commented Oct 17, 2023 •

edited

Loading