[CONTINT-3558] Add e2e tests for APM origin detection on k8s #22969

AliDatadog · 2024-02-19T16:21:34Z

What does this PR do?

This PR adds e2e tests for origin detection on APM.

Motivation

Increase test coverage.

Additional Notes

N/A

Possible Drawbacks / Trade-offs

N/A

Describe how to test/QA your changes

Run tests using inv new-e2e-tests.run --targets=./tests/containers --run TestKindSuite

=== RUN   TestKindSuite/TestTraceTCP
=== RUN   TestKindSuite/TestTraceUDP

pr-commenter · 2024-02-19T17:52:15Z

Bloop Bleep... Dogbot Here

Regression Detector Results

Run ID: 7beef838-ee85-4b76-91ac-af28fbf4b2cb
Baseline: 9898528
Comparison: 0cc41ee

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	-2.06	[-8.56, +4.44]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	otel_to_otel_logs	ingress throughput	+1.31	[+0.68, +1.93]
➖	process_agent_real_time_mode	memory utilization	+0.12	[+0.08, +0.15]
➖	file_tree	memory utilization	+0.04	[-0.03, +0.11]
➖	process_agent_standard_check_with_stats	memory utilization	+0.04	[+0.01, +0.07]
➖	trace_agent_json	ingress throughput	+0.03	[+0.01, +0.06]
➖	trace_agent_msgpack	ingress throughput	+0.03	[+0.01, +0.04]
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.00, +0.00]
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.00, +0.00]
➖	process_agent_standard_check	memory utilization	-0.03	[-0.05, -0.00]
➖	tcp_syslog_to_blackhole	ingress throughput	-0.14	[-0.19, -0.09]
➖	idle	memory utilization	-0.40	[-0.43, -0.37]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.48	[-1.92, +0.95]
➖	basic_py_check	% cpu utilization	-0.62	[-2.81, +1.56]
➖	file_to_blackhole	% cpu utilization	-2.06	[-8.56, +4.44]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

AliDatadog · 2024-02-21T15:40:09Z

/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=auto

dd-devflow · 2024-02-21T15:40:45Z

🚂 Gitlab pipeline started

Started pipeline #28643033

AliDatadog · 2024-03-01T14:27:53Z

/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=auto

dd-devflow · 2024-03-01T14:28:27Z

🚂 Gitlab pipeline started

Started pipeline #29296220

test/new-e2e/tests/containers/k8s_test.go

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

test/new-e2e/tests/containers/k8s_test.go

AliDatadog · 2024-03-08T09:47:56Z

/trigger-ci --variable RUN_E2E_TESTS=true

dd-devflow · 2024-03-08T09:48:34Z

🚂 Gitlab pipeline started

Started pipeline #29787400

AliDatadog · 2024-03-08T13:21:47Z

/merge

dd-devflow · 2024-03-08T13:21:52Z

🚂 MergeQueue

Pull request added to the queue.

This build is next! (estimated merge in less than 28m)

Use /merge -c to cancel this operation!

* implement e2e test for APM origin detection * remove test of cgroup version * only test the namespace workload-tracegen * simplify the logic with two separate tests for TCP and UDS * iterate from the most recent traces * bump test infra definitions * fix image_tag issue * add docker_image and image_tag to ecs suite * rebump * hardcode the image tag * Update test/new-e2e/tests/containers/k8s_test.go Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com> * Update test/new-e2e/tests/containers/k8s_test.go Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com> * Remove the `require` func Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com> * Update error message Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com> * Iterate in normal order Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com> * Rename test --------- Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

AliDatadog added team/containers changelog/no-changelog [deprecated] qa/skip-qa - use other qa/ labels [DEPRECATED] Please use qa/done or qa/no-code-change to skip creating a QA card qa/done QA done before merge and regressions are covered by tests labels Feb 19, 2024

AliDatadog added this to the 7.51.0 milestone Feb 19, 2024

AliDatadog requested review from a team as code owners February 19, 2024 16:21

implement e2e test for APM origin detection

929ac8b

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from f8f99cc to 929ac8b Compare February 19, 2024 16:33

AliDatadog removed request for a team February 19, 2024 16:35

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from 5970f7c to 84e9fda Compare February 20, 2024 14:39

remove test of cgroup version

f3792f4

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from 84e9fda to f3792f4 Compare February 20, 2024 14:51

Merge branch 'main' into ali/e2e-test-apm-origin

8ad567c

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from f5045a7 to 8ad567c Compare February 20, 2024 15:15

kacper-murzyn modified the milestones: 7.51.0, 7.53.0 Feb 27, 2024

AliDatadog marked this pull request as draft February 27, 2024 16:09

only test the namespace workload-tracegen

a8e91f9

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from ee14dcb to a8e91f9 Compare February 27, 2024 16:17

AliDatadog marked this pull request as ready for review February 27, 2024 16:18

AliDatadog added 3 commits February 28, 2024 13:40

simplify the logic with two separate tests for TCP and UDS

8077b79

iterate from the most recent traces

49bc412

bump test infra definitions

731757e

AliDatadog marked this pull request as draft March 1, 2024 14:21

Merge branch 'ali/bump-test-infra-def' into ali/e2e-test-apm-origin

5608891

AliDatadog marked this pull request as ready for review March 1, 2024 14:27

AliDatadog added 5 commits March 1, 2024 17:54

fix image_tag issue

d85aae2

add docker_image and image_tag to ecs suite

2a7b7b7

rebump

fdc6fec

hardcode the image tag

fb8bf9c

Merge branch 'ali/bump-test-infra-def' into ali/e2e-test-apm-origin

f02130d

L3n41c reviewed Mar 4, 2024

View reviewed changes

AliDatadog and others added 5 commits March 4, 2024 14:31

Update test/new-e2e/tests/containers/k8s_test.go

eed623a

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

Update test/new-e2e/tests/containers/k8s_test.go

00571ad

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

Remove the require func

a60e839

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

Update error message

aa4d98b

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

Iterate in normal order

8c61097

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

L3n41c approved these changes Mar 4, 2024

View reviewed changes

AliDatadog commented Mar 6, 2024

View reviewed changes

test/new-e2e/tests/containers/k8s_test.go Outdated Show resolved Hide resolved

Rename test

5d1f69a

Base automatically changed from ali/bump-test-infra-def to main March 7, 2024 18:00

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from 9841a20 to baee5fe Compare March 8, 2024 09:46

Merge branch 'main' into ali/e2e-test-apm-origin

0cc41ee

AliDatadog force-pushed the ali/e2e-test-apm-origin branch from baee5fe to 0cc41ee Compare March 8, 2024 09:47

AliDatadog changed the title ~~[CONTINT-3558] Add e2e tests for APM origin detection~~ [CONTINT-3558] Add e2e tests for APM origin detection on k8s Mar 8, 2024

dd-mergequeue bot merged commit 5d68596 into main Mar 8, 2024
185 checks passed

dd-mergequeue bot deleted the ali/e2e-test-apm-origin branch March 8, 2024 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CONTINT-3558] Add e2e tests for APM origin detection on k8s #22969

[CONTINT-3558] Add e2e tests for APM origin detection on k8s #22969

AliDatadog commented Feb 19, 2024 •

edited

Loading

pr-commenter bot commented Feb 19, 2024 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

AliDatadog commented Feb 21, 2024

dd-devflow bot commented Feb 21, 2024

AliDatadog commented Mar 1, 2024

dd-devflow bot commented Mar 1, 2024

AliDatadog commented Mar 8, 2024

dd-devflow bot commented Mar 8, 2024

AliDatadog commented Mar 8, 2024

dd-devflow bot commented Mar 8, 2024

[CONTINT-3558] Add e2e tests for APM origin detection on k8s #22969

[CONTINT-3558] Add e2e tests for APM origin detection on k8s #22969

Conversation

AliDatadog commented Feb 19, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter bot commented Feb 19, 2024 • edited Loading

Bloop Bleep... Dogbot Here

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

AliDatadog commented Feb 21, 2024

dd-devflow bot commented Feb 21, 2024

AliDatadog commented Mar 1, 2024

dd-devflow bot commented Mar 1, 2024

AliDatadog commented Mar 8, 2024

dd-devflow bot commented Mar 8, 2024

AliDatadog commented Mar 8, 2024

dd-devflow bot commented Mar 8, 2024

AliDatadog commented Feb 19, 2024 •

edited

Loading

pr-commenter bot commented Feb 19, 2024 •

edited

Loading