DAOS-12946 test: fix telemetry_pool_metrics with ior error #12478

knard-intel · 2023-06-21T19:47:31Z

Description

When the ior command is failing, it is not possible to properly test the metrics as we do not know how much data have been transferred.
Thus we cancel the test when such error is occurring, as the purpose of this test is not to check the ior command.

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

daosbuild1

LGTM. No errors found by checkpatch.

github-actions · 2023-06-21T20:23:58Z

Bug-tracker data:
Ticket title is 'telemetry/telemetry_pool_metrics.py:TelemetryPoolMetrics.test_telemetry_pool_metrics - Aggregated value of the metric engine_pool_ops_fetch for oclass RP_3GX is invalid: got=4, wait_in=[519, 577]'
Status is 'In Progress'
Labels: 'ci_impact,daily_test,triaged'
https://daosio.atlassian.net/browse/DAOS-12946

daosbuild1 · 2023-06-21T20:28:45Z

Test stage Build RPM on Leap 15.4 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12478/1/execution/node/288/log

daosbuild1 · 2023-06-21T20:38:09Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12478/1/execution/node/347/log

daosbuild1 · 2023-06-21T20:39:08Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12478/1/execution/node/366/log

daosbuild1

LGTM. No errors found by checkpatch.

Fix management of ior errors: canceling the test as we do not know how much data have been transferred. Test-tag: pr test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning · 2023-06-23T16:50:29Z

src/tests/ftest/telemetry/telemetry_pool_metrics.py

+        except TestFail as error:
+            # NOTE DAOS-12946:  it is not possible to properly test the metrics as we do not know
+            # how much data have been transferred.  Thus we cancel the test, as the purpose of this
+            # test is not to check the ior command.
+            self.log.error(">>>>>> IOR command failed: %s", str(error))
+            self.log.info("------ Test canceled ------")
+            self.cancel("IOR command failed: {}".format(error))


I wonder if the test should just fail here - self.fail - since if IOR fails that's probably a different problem that needs to be investigated. I commented on the ticket as well

Good point, I will change the PR according to that.

Fail on ior error

daosbuild1 · 2023-06-23T19:52:44Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12478/4/execution/node/1290/log

Integrate Reviewers comments. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

…/daos-12946 Required-githooks: true

Fix flake8 code formating. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

Typo hotfix. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-23T20:54:52Z

Test stage Unit Test on EL 8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-12478/6/display/redirect

…/daos-12946 Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning · 2023-06-26T16:27:30Z

src/tests/ftest/telemetry/telemetry_pool_metrics.py

+        except TestFail as error:
+            # NOTE DAOS-12946:  it is not possible to properly test the metrics as we do not know
+            # how much data have been transferred.  However, the reason of the ior failure should be
+            # investigated and thus we fail the test.
+            self.log.error(
+                ">>>>>> Failure of IOR is unexpected and should be investigated: %s",
+                str(error))
+            self.fail("IOR command failed: {}".format(error))



Sorry, I didn't realize before that this is TestFail. self.fail() will raise another TestFail. So we could just remove the try..except to get the same behavior. I think Failure of IOR is unexpected and should be investigated is implied. I.e. if IOR fails, the test fails and it's understood that it needs to be investigated.

Remove useless exception catch

…/daos-12946 Required-githooks: true

Integrate reviewers comments. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning

LGTM. Thanks!

shimizukko

You might want to remove from avocado.core.exceptions import TestFail

daltonbohning · 2023-06-26T23:47:02Z

You might want to remove from avocado.core.exceptions import TestFail

Good catch! That pylint check is failing:
https://github.com/daos-stack/daos/actions/runs/5383214549

knard-intel · 2023-06-29T07:38:27Z

You might want to remove from avocado.core.exceptions import TestFail

Remove useless package import

…/daos-12946 Required-githooks: true

Integrate reviewers comments: * Remove useless package import #12478 (comment) Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

knard-intel · 2023-06-30T07:50:14Z

@daos-stack/daos-gatekeeper Test-Tag was used to limit functional tests to the fixed one.

When the ior command is failing, it is not possible to properly test the metrics as we do not know how much data have been transferred. Thus we cancel the test when such error is occurring, as the purpose of this test is not to check the ior command. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

…12478 #13122 (#13229) When the ior command is failing, it is not possible to properly test the metrics as we do not know how much data have been transferred. Thus we cancel the test when such error is occurring, as the purpose of this test is not to check the ior command. Update engine_pool_xferred_fetch metric threshold to manage extra networking control traffic. Add some comments on the limitation of the tests. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1 reviewed Jun 21, 2023

View reviewed changes

knard-intel force-pushed the ckochhof/fix/master/daos-12946 branch from a2e0cf2 to f6b7187 Compare June 22, 2023 06:56

daosbuild1 reviewed Jun 22, 2023

View reviewed changes

knard-intel force-pushed the ckochhof/fix/master/daos-12946 branch from f6b7187 to 5414f06 Compare June 22, 2023 07:02

knard-intel force-pushed the ckochhof/fix/master/daos-12946 branch from 5414f06 to 8456e86 Compare June 22, 2023 07:04

daosbuild1 reviewed Jun 22, 2023

View reviewed changes

knard-intel requested review from daltonbohning, phender and shimizukko June 23, 2023 08:45

knard-intel marked this pull request as ready for review June 23, 2023 08:45

knard-intel requested a review from a team as a code owner June 23, 2023 08:45

daltonbohning reviewed Jun 23, 2023

View reviewed changes

kanard38 added 4 commits June 23, 2023 20:21

DAOS-12946 test: fix telemetry_pool_metrics with ior error

e397fbb

Integrate Reviewers comments. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

0baa7eb

…/daos-12946 Required-githooks: true

DAOS-12946 test: fix telemetry_pool_metrics with ior error

9ba1c40

Fix flake8 code formating. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

DAOS-12946 test: fix telemetry_pool_metrics with ior error

71525b5

Typo hotfix. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

daosbuild1 reviewed Jun 23, 2023

View reviewed changes

knard-intel requested a review from daltonbohning June 23, 2023 20:29

daosbuild1 reviewed Jun 23, 2023

View reviewed changes

shimizukko previously approved these changes Jun 23, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

ee92e93

…/daos-12946 Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true

daosbuild1 reviewed Jun 26, 2023

View reviewed changes

daltonbohning reviewed Jun 26, 2023

View reviewed changes

kanard38 added 2 commits June 26, 2023 21:25

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

c5a749a

…/daos-12946 Required-githooks: true

DAOS-12946 test: fix telemetry_pool_metrics with ior error

8ee9741

Integrate reviewers comments. Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

knard-intel dismissed shimizukko’s stale review via 8ee9741 June 26, 2023 21:27

daosbuild1 reviewed Jun 26, 2023

View reviewed changes

knard-intel requested review from daltonbohning and shimizukko June 26, 2023 21:44

daltonbohning previously approved these changes Jun 26, 2023

View reviewed changes

shimizukko previously approved these changes Jun 26, 2023

View reviewed changes

kanard38 added 2 commits June 29, 2023 07:40

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

9711b94

…/daos-12946 Required-githooks: true

DAOS-12946 test: fix telemetry_pool_metrics with ior error

8a47ce7

Integrate reviewers comments: * Remove useless package import #12478 (comment) Test-tag: TelemetryPoolMetrics,test_telemetry_pool_metrics Required-githooks: true Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com>

knard-intel dismissed stale reviews from shimizukko and daltonbohning via 8a47ce7 June 29, 2023 07:42

daosbuild1 reviewed Jun 29, 2023

View reviewed changes

knard-intel requested review from shimizukko and daltonbohning June 29, 2023 07:47

shimizukko approved these changes Jun 29, 2023

View reviewed changes

daltonbohning approved these changes Jun 29, 2023

View reviewed changes

knard-intel requested a review from a team June 30, 2023 07:49

phender merged commit 54b6e0f into master Jun 30, 2023

phender deleted the ckochhof/fix/master/daos-12946 branch June 30, 2023 14:58

knard-intel mentioned this pull request Oct 25, 2023

DAOS-14220 test: Fix ior errors management and fetch metric threshold #12478 #13122 #13229

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-12946 test: fix telemetry_pool_metrics with ior error #12478

DAOS-12946 test: fix telemetry_pool_metrics with ior error #12478

knard-intel commented Jun 21, 2023 •

edited

Loading

daosbuild1 left a comment

github-actions bot commented Jun 21, 2023 •

edited

Loading

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daltonbohning Jun 23, 2023

knard-intel Jun 23, 2023 •

edited

Loading

daosbuild1 commented Jun 23, 2023

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jun 23, 2023

daosbuild1 left a comment

daltonbohning Jun 26, 2023

knard-intel Jun 26, 2023 •

edited

Loading

daosbuild1 left a comment

daltonbohning left a comment

shimizukko left a comment

daltonbohning commented Jun 26, 2023

knard-intel commented Jun 29, 2023 •

edited

Loading

daosbuild1 left a comment

knard-intel commented Jun 30, 2023 •

edited

Loading

DAOS-12946 test: fix telemetry_pool_metrics with ior error #12478

DAOS-12946 test: fix telemetry_pool_metrics with ior error #12478

Conversation

knard-intel commented Jun 21, 2023 • edited Loading

Description

Before requesting gatekeeper:

Gatekeeper:

daosbuild1 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 21, 2023 • edited Loading

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daltonbohning Jun 23, 2023

Choose a reason for hiding this comment

knard-intel Jun 23, 2023 • edited Loading

Choose a reason for hiding this comment

daosbuild1 commented Jun 23, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 23, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daltonbohning Jun 26, 2023

Choose a reason for hiding this comment

knard-intel Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daltonbohning left a comment

Choose a reason for hiding this comment

shimizukko left a comment

Choose a reason for hiding this comment

daltonbohning commented Jun 26, 2023

knard-intel commented Jun 29, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

knard-intel commented Jun 30, 2023 • edited Loading

knard-intel commented Jun 21, 2023 •

edited

Loading

github-actions bot commented Jun 21, 2023 •

edited

Loading

knard-intel Jun 23, 2023 •

edited

Loading

knard-intel Jun 26, 2023 •

edited

Loading

knard-intel commented Jun 29, 2023 •

edited

Loading

knard-intel commented Jun 30, 2023 •

edited

Loading