e2e: Add backend health check e2e case via active http

[aoledk]

What this PR does / why we need it:

Check whether backend health check via active http works in e2e, inspired by https://github.com/projectcontour/contour/blob/main/test/e2e/httpproxy/http_health_checks_test.go.

EG use the default 50% from Envoy as cluster panic threshold. In case of health check fail, all endpoints in cluster become unhealthy and panic mode is activated, request is distributed to all unhealthy echo-basic endpoints and get response from echo-basic, with same behavior as case of health check pass.

To make those two cases distinguishable, in case of health check fail, Envoy should not distribute request to unhealthy endpoint, respond with 503 no healthy upstream directly. This requires cluster panic mode is disabled.

Since EG doesn't have knob to configure cluster panic threshold currently, I use EnvoyPatchPolicy to disable cluster panic mode in this PR.

Issues about panic threshold in contour:
projectcontour/contour#579.
projectcontour/contour#1846

Which issue(s) this PR fixes:

xref #2417

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.01%. Comparing base (5050e36) to head (bf9cabf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3677      +/-   ##
==========================================
- Coverage   69.02%   69.01%   -0.01%     
==========================================
  Files         176      176              
  Lines       21688    21688              
==========================================
- Hits        14971    14969       -2     
- Misses       5637     5638       +1     
- Partials     1080     1081       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[arkodg]

thanks for picking this one up @aoledk !
would be great to also make sure we can check if the membership_degraded stat https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats is also bumped
similar to whats done here

gateway/test/e2e/tests/connection_limit.go

Line 70 in d784c32

promQL := fmt.Sprintf(`envoy_connection_limit_limited_connections{envoy_connection_limit_prefix="%s",gateway_envoyproxy_io_owning_gateway_name="%s"}`, prefix, gtwName)

[aoledk]

We can use membership_healthy to tell whether active http health check works, instead of membership_degraded which requires upstream host to respond with special x-envoy-degraded header ¹, IMO it's out of this issue's scope.

• In case of health check pass, membership_healthy should > 0.
• In case of health check fail, membership_healthy should == 0.

# TYPE envoy_cluster_membership_degraded gauge
envoy_cluster_membership_degraded{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 0
envoy_cluster_membership_degraded{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 0

# TYPE envoy_cluster_membership_excluded gauge
envoy_cluster_membership_excluded{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 0
envoy_cluster_membership_excluded{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 0

# TYPE envoy_cluster_membership_healthy gauge
envoy_cluster_membership_healthy{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 0
envoy_cluster_membership_healthy{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 2

# TYPE envoy_cluster_membership_total gauge
envoy_cluster_membership_total{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 2
envoy_cluster_membership_total{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 2

Footnotes

1. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/health_checking#arch-overview-health-checking-degraded ↩

[arkodg]

membership_health works, thanks !

[arkodg]

shouldn't this be non zero ?

[arkodg]

you prob need to wait until non zero is reached

[aoledk]

I've turned to use non-zero cluster health check stats to verify whether backend health check works, for both pass and fail cases. And undo changes in e2e/utils/prometheus.sum to restore the behavior of only return nil when non-zero value is queried.

# TYPE envoy_cluster_health_check_attempt counter
envoy_cluster_health_check_attempt{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 2
envoy_cluster_health_check_attempt{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 2

# TYPE envoy_cluster_health_check_success counter
envoy_cluster_health_check_success{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 0
envoy_cluster_health_check_success{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 2

# TYPE envoy_cluster_health_check_failure counter
envoy_cluster_health_check_failure{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-fail/rule/0"} 2
envoy_cluster_health_check_failure{envoy_cluster_name="httproute/gateway-conformance-infra/http-with-health-check-active-http-pass/rule/0"} 0

[arkodg]

thanks !

[zirain]

/retest