Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support HTTP compression on stats/prometheus #3201

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions api/v1alpha1/envoyproxy_metric_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,7 @@ type ProxyOpenTelemetrySink struct {
type ProxyPrometheusProvider struct {
// Disable the Prometheus endpoint.
Disable bool `json:"disable,omitempty"`
// Configure the compression on Prometheus endpoint. Compression is useful in situations when bandwidth is scarce and large payloads can be effectively compressed at the expense of higher CPU load.
// +optional
Compression *Compression `json:"compression,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the BTP API instantiated this as a list

Compression []*Compression `json:"compression,omitempty"`
to support multiple compression settings, to support heterogenous clients (and compression settings), do we also need to make it a list here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could 1 envoy route support multiple compression libraries? AFAIK we need to select only 1 library right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it to list but limit maxitem to 1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that doesn't make sense IMO

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think envoy can support multiple compressor libraries per route, which is the argument for making it a list
but realistically the persona who creates the EnvoyProxy resource with this setting, also knows beforehand the compression supported by the peer, so a single scalar value is fine here

}
7 changes: 6 additions & 1 deletion api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -6227,6 +6227,24 @@ spec:
description: Prometheus defines the configuration for Admin
endpoint `/stats/prometheus`.
properties:
compression:
description: Configure the compression on Prometheus endpoint.
Compression is useful in situations when bandwidth is
scarce and large payloads can be effectively compressed
at the expense of higher CPU load.
properties:
gzip:
description: The configuration for GZIP compressor.
type: object
type:
description: CompressorType defines the compressor
type to use for compression.
enum:
- Gzip
type: string
required:
- type
type: object
disable:
description: Disable the Prometheus endpoint.
type: boolean
Expand Down
24 changes: 19 additions & 5 deletions internal/xds/bootstrap/bootstrap.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,11 @@ type bootstrapParameters struct {
ReadyServer readyServerParameters
// EnablePrometheus defines whether to enable metrics endpoint for prometheus.
EnablePrometheus bool
// EnablePrometheusCompression defines whether to enable HTTP compression on metrics endpoint for prometheus.
EnablePrometheusCompression bool
// PrometheusCompressionLibrary defines the HTTP compression library for metrics endpoint for prometheus.
PrometheusCompressionLibrary string

// OtelMetricSinks defines the configuration of the OpenTelemetry sinks.
OtelMetricSinks []metricSink
// EnableStatConfig defines whether to to customize the Envoy proxy stats.
Expand Down Expand Up @@ -136,16 +141,23 @@ func (b *bootstrapConfig) render() error {
// GetRenderedBootstrapConfig renders the bootstrap YAML string
func GetRenderedBootstrapConfig(opts *RenderBootsrapConfigOptions) (string, error) {
var (
enablePrometheus = true
metricSinks []metricSink
StatsMatcher StatsMatcherParameters
enablePrometheus = true
enablePrometheusCompression = false
PrometheusCompressionLibrary = "gzip"
metricSinks []metricSink
StatsMatcher StatsMatcherParameters
)

if opts != nil && opts.ProxyMetrics != nil {
proxyMetrics := opts.ProxyMetrics

if proxyMetrics.Prometheus != nil {
enablePrometheus = !proxyMetrics.Prometheus.Disable

if proxyMetrics.Prometheus.Compression != nil {
enablePrometheusCompression = true
PrometheusCompressionLibrary = string(proxyMetrics.Prometheus.Compression.Type)
}
}

addresses := sets.NewString()
Expand Down Expand Up @@ -216,8 +228,10 @@ func GetRenderedBootstrapConfig(opts *RenderBootsrapConfigOptions) (string, erro
Port: EnvoyReadinessPort,
ReadinessPath: EnvoyReadinessPath,
},
EnablePrometheus: enablePrometheus,
OtelMetricSinks: metricSinks,
EnablePrometheus: enablePrometheus,
EnablePrometheusCompression: enablePrometheusCompression,
PrometheusCompressionLibrary: PrometheusCompressionLibrary,
OtelMetricSinks: metricSinks,
},
}
if opts != nil && opts.ProxyMetrics != nil && opts.ProxyMetrics.Matches != nil {
Expand Down
23 changes: 23 additions & 0 deletions internal/xds/bootstrap/bootstrap.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,29 @@ static_resources:
prefix: /stats/prometheus
route:
cluster: prometheus_stats
{{- if .EnablePrometheusCompression }}
typed_per_filter_config:
envoy.filters.http.compression:
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.CompressorPerRoute
{{- if eq .PrometheusCompressionLibrary "gzip"}}
compressor_library:
name: text_optimized
typed_config:
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
{{- end }}
{{- if eq .PrometheusCompressionLibrary "brotli"}}
compressor_library:
name: text_optimized
typed_config:
"@type": type.googleapis.com/envoy.extensions.compression.brotli.compressor.v3.Brotli
{{- end }}
{{- if eq .PrometheusCompressionLibrary "zstd"}}
compressor_library:
name: text_optimized
typed_config:
"@type": type.googleapis.com/envoy.extensions.compression.zstd.compressor.v3.Zstd
{{- end }}
{{- end }}
{{- end }}
http_filters:
- name: envoy.filters.http.health_check
Expand Down
12 changes: 12 additions & 0 deletions internal/xds/bootstrap/bootstrap_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ func TestGetRenderedBootstrapConfig(t *testing.T) {
},
},
},
{
name: "enable-prometheus-gzip-compression",
opts: &RenderBootsrapConfigOptions{
ProxyMetrics: &egv1a1.ProxyMetrics{
Prometheus: &egv1a1.ProxyPrometheusProvider{
Compression: &egv1a1.Compression{
Type: "gzip",
},
},
},
},
},
{
name: "otel-metrics",
opts: &RenderBootsrapConfigOptions{
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
layered_runtime:
layers:
- name: global_config
static_layer:
envoy.restart_features.use_eds_cache_for_ads: true
re2.max_program_size.error_level: 4294967295
re2.max_program_size.warn_level: 1000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
listeners:
- name: envoy-gateway-proxy-ready-0.0.0.0-19001
address:
socket_address:
address: 0.0.0.0
port_value: 19001
protocol: TCP
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: eg-ready-http
route_config:
name: local_route
virtual_hosts:
- name: prometheus_stats
domains:
- "*"
routes:
- match:
prefix: /stats/prometheus
route:
cluster: prometheus_stats
typed_per_filter_config:
envoy.filters.http.compression:
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.CompressorPerRoute
compressor_library:
name: text_optimized
typed_config:
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
pass_through_mode: false
headers:
- name: ":path"
string_match:
exact: /ready
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: prometheus_stats
connect_timeout: 0.250s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: prometheus_stats
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 19000
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- load_balancing_weight: 1
lb_endpoints:
- load_balancing_weight: 1
endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
explicit_http_config:
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: "envoy.resource_monitors.global_downstream_max_connections"
typed_config:
"@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
max_active_downstream_connections: 50000
2 changes: 2 additions & 0 deletions site/content/en/latest/api/extension_types.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,6 +455,7 @@ This can help reduce the bandwidth at the expense of higher CPU.

_Appears in:_
- [BackendTrafficPolicySpec](#backendtrafficpolicyspec)
- [ProxyPrometheusProvider](#proxyprometheusprovider)

| Field | Type | Required | Description |
| --- | --- | --- | --- |
Expand Down Expand Up @@ -2316,6 +2317,7 @@ _Appears in:_
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `disable` | _boolean_ | true | Disable the Prometheus endpoint. |
| `compression` | _[Compression](#compression)_ | false | Configure the compression on Prometheus endpoint. Compression is useful in situations when bandwidth is scarce and large payloads can be effectively compressed at the expense of higher CPU load. |


#### ProxyProtocol
Expand Down