DAOS-13672 control: Bump system_ram_reserved to reduce OOM occurrences #12430

tanabarr · 2023-06-16T16:02:43Z

Attempt to reduce the chance of OOM killer terminating an engine
process when maximum pool space is allocated by slightly increasing
the system_ram_reserved default value from 6->16gib.

Some test yaml system_ram_reserved values have been reduced to 6 to
prevent the increase in the default from causing an available memory
check failure on engine start-up in memory constrained (VM)
environments. Increasing the default value should also resolve
DAOS-13918 by providing a larger memory buffer to reduce the chance
of intermittent failures related to this check.

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

daosbuild1

LGTM. No errors found by checkpatch.

github-actions · 2023-06-16T16:32:11Z

Bug-tracker data:
Ticket title is 'Tighten validation rules for tmpfs memory usage'
Status is 'In Progress'
Labels: 'md_on_ssd2'
https://daosio.atlassian.net/browse/DAOS-13672

daosbuild1 · 2023-06-16T16:44:47Z

Test stage Unit Test on EL 8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12430/1/testReport/(root)/

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-19T10:32:47Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12430/2/execution/node/1095/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-19T22:25:17Z

Test stage Unit Test on EL 8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12430/3/testReport/

daosbuild1 · 2023-06-21T04:37:11Z

Test stage Functional on EL 8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12430/4/testReport/

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

tanabarr · 2023-06-21T17:16:26Z

merged 6e16c1c into this PR so number of targets are taken into account when calculating engine memory reservation during scm auto sizing

daosbuild1 · 2023-06-21T17:18:19Z

Test stage Build RPM on Leap 15.4 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12430/7/execution/node/353/log

daosbuild1 · 2023-06-21T17:24:23Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12430/7/execution/node/360/log

kjacque

Current changes look OK to me. I'll hold off on approval until it runs with the correct pragma.

Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

…srsvd-bump Allow-unstable: true Test-tag: pr daily_regression Test-nvme: auto_md_on_ssd Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-07-27T09:05:44Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12430/19/execution/node/1315/log

tanabarr · 2023-07-27T11:05:31Z

https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-12430/19/tests/

Shows the following identified failures:
2 * PoolCreateTests - DAOS-13967,DAOS-13916
DmgSystemReformatTest - DAOS-13613

And 2 new failures:
DmgPoolQueryTest - not specific to this PR, seen on https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/md-on-ssd-testing/63/tests/
SnapshotAggregation - DmgJsonCommandFailure('pool create failed: DER_NOSPACE(-1007): No space on storage target',) looks specific to this PR and possibly to do with the system_ram_reserved default increase to 16gib

Skip-func-test-vm: true Test-tag: test_snapshot_aggregation Test-nvme: auto_md_on_ssd Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

daosbuild1 · 2023-07-27T16:58:09Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12430/20/execution/node/328/log

daosbuild1

LGTM. No errors found by checkpatch.

tanabarr · 2023-07-27T16:59:23Z

reducing system_ram_reserved to try to make test_snapshot_aggregation pass

tanabarr · 2023-07-28T10:20:00Z

snapshot aggregation test now passing as per https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-12430/21/testReport/FTEST_container/SnapshotAggregation/

combining runs 19 & 21 there are only expected failures

identified failures:
2 * PoolCreateTests - DAOS-13967,DAOS-13916
DmgSystemReformatTest - DAOS-13613
DmgPoolQueryTest - DAOS-14072 (PR just created, seen on https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/md-on-ssd-testing/63/tests/)

@phender can you take a look to see if we are at the stage where this could be force landed? obviously pending review approvals

ashleypittman · 2023-08-01T07:26:49Z

The ARM CI testing on master failed for this PR, please fix insufficient ram to meet minimum requirements (mem stats: total 16 GiB (16739012608) - (hugepages 0 B + sys rsvd 16 GiB + (engine rsvd 1.0 GiB * nr engines 1). 4 tgts-per-engine))

https://github.com/daos-stack/daos/actions/runs/5721566985/job/15503353047

…#12597) During the evaluation of an optimum RAM-disk size, update per-engine memory reservation per-engine calculation to take into account the number of targets. Reserve 128mib of RAM per target. control: Bump system_ram_reserved to reduce OOM occurrences (#12430) Attempt to reduce the chance of OOM killer terminating an engine process when maximum pool space is allocated by slightly increasing the system_ram_reserved default value from 6->16gib. Some test yaml system_ram_reserved values have been reduced to 6 to prevent the increase in the default from causing an available memory check failure on engine start-up in memory constrained (VM) environments. Increasing the default value should also resolve DAOS-13918 by providing a larger memory buffer to reduce the chance of intermittent failures related to this check. test: Reduce system_ram_reserved for GHA ARM build (#12746) Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

…#12597) (#12755) During the evaluation of an optimum RAM-disk size, update per-engine memory reservation per-engine calculation to take into account the number of targets. Reserve 128mib of RAM per target. control: Bump system_ram_reserved to reduce OOM occurrences (#12430) Attempt to reduce the chance of OOM killer terminating an engine process when maximum pool space is allocated by slightly increasing the system_ram_reserved default value from 6->16gib. Some test yaml system_ram_reserved values have been reduced to 6 to prevent the increase in the default from causing an available memory check failure on engine start-up in memory constrained (VM) environments. Increasing the default value should also resolve DAOS-13918 by providing a larger memory buffer to reduce the chance of intermittent failures related to this check. test: Reduce system_ram_reserved for GHA ARM build (#12746) Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

tanabarr requested a review from a team as a code owner June 16, 2023 16:02

tanabarr requested review from mjmac and removed request for a team June 16, 2023 16:02

tanabarr self-assigned this Jun 16, 2023

tanabarr added the control-plane work on the management infrastructure of the DAOS Control Plane label Jun 16, 2023

tanabarr requested review from NiuYawei and a team June 16, 2023 16:03

daosbuild1 reviewed Jun 16, 2023

View reviewed changes

daosbuild1 reviewed Jun 18, 2023

View reviewed changes

NiuYawei previously approved these changes Jun 19, 2023

View reviewed changes

tanabarr dismissed NiuYawei’s stale review via 7fe0777 June 19, 2023 21:50

tanabarr requested a review from a team as a code owner June 19, 2023 21:50

daosbuild1 reviewed Jun 19, 2023

View reviewed changes

daltonbohning previously approved these changes Jun 20, 2023

View reviewed changes

tanabarr requested a review from NiuYawei June 20, 2023 21:02

NiuYawei previously approved these changes Jun 21, 2023

View reviewed changes

daosbuild1 reviewed Jun 21, 2023

View reviewed changes

tanabarr dismissed stale reviews from daltonbohning and NiuYawei via 6e16c1c June 21, 2023 17:11

daosbuild1 reviewed Jun 21, 2023

View reviewed changes

tanabarr requested review from NiuYawei and daltonbohning June 21, 2023 17:18

kjacque reviewed Jul 25, 2023

View reviewed changes

tanabarr added 2 commits July 26, 2023 08:00

increase system_ram_reserved default to 16 as per code review comment

3f3692d

Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

Merge remote-tracking branch 'origin/master' into tanabarr/control-sy…

a8a280c

…srsvd-bump Allow-unstable: true Test-tag: pr daily_regression Test-nvme: auto_md_on_ssd Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

tanabarr removed the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jul 26, 2023

tanabarr dismissed knard-intel’s stale review via a8a280c July 26, 2023 15:36

tanabarr requested review from kjacque and knard-intel July 26, 2023 15:37

daosbuild1 reviewed Jul 26, 2023

View reviewed changes

knard-intel previously approved these changes Jul 27, 2023

View reviewed changes

attempt to resolve snapshot aggregation test failure

a25515c

Skip-func-test-vm: true Test-tag: test_snapshot_aggregation Test-nvme: auto_md_on_ssd Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

tanabarr dismissed knard-intel’s stale review via a25515c July 27, 2023 16:56

daosbuild1 reviewed Jul 27, 2023

View reviewed changes

tanabarr requested a review from knard-intel July 28, 2023 09:45

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jul 28, 2023

mjmac approved these changes Jul 28, 2023

View reviewed changes

knard-intel approved these changes Jul 28, 2023

View reviewed changes

kjacque approved these changes Jul 28, 2023

View reviewed changes

mchaarawi removed their request for review July 28, 2023 17:26

tanabarr requested a review from a team July 28, 2023 17:30

phender merged commit f9f935e into master Aug 1, 2023
32 checks passed

phender deleted the tanabarr/control-sysrsvd-bump branch August 1, 2023 01:42

tanabarr mentioned this pull request Aug 1, 2023

DAOS-13672 control: Calculate engine memory reservation on nr targets… #12755

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-13672 control: Bump system_ram_reserved to reduce OOM occurrences #12430

DAOS-13672 control: Bump system_ram_reserved to reduce OOM occurrences #12430

tanabarr commented Jun 16, 2023 •

edited

Loading

daosbuild1 left a comment

github-actions bot commented Jun 16, 2023

daosbuild1 commented Jun 16, 2023

daosbuild1 left a comment

daosbuild1 commented Jun 19, 2023

daosbuild1 left a comment

daosbuild1 commented Jun 19, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 left a comment

daosbuild1 left a comment

tanabarr commented Jun 21, 2023 •

edited

Loading

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

kjacque left a comment

daosbuild1 left a comment

daosbuild1 commented Jul 27, 2023

tanabarr commented Jul 27, 2023

daosbuild1 commented Jul 27, 2023

daosbuild1 left a comment

tanabarr commented Jul 27, 2023

tanabarr commented Jul 28, 2023

ashleypittman commented Aug 1, 2023 •

edited

Loading

DAOS-13672 control: Bump system_ram_reserved to reduce OOM occurrences #12430

DAOS-13672 control: Bump system_ram_reserved to reduce OOM occurrences #12430

Conversation

tanabarr commented Jun 16, 2023 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

daosbuild1 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 16, 2023

daosbuild1 commented Jun 16, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 19, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 19, 2023

daosbuild1 commented Jun 21, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

tanabarr commented Jun 21, 2023 • edited Loading

daosbuild1 commented Jun 21, 2023

daosbuild1 commented Jun 21, 2023

kjacque left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jul 27, 2023

tanabarr commented Jul 27, 2023

daosbuild1 commented Jul 27, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

tanabarr commented Jul 27, 2023

tanabarr commented Jul 28, 2023

ashleypittman commented Aug 1, 2023 • edited Loading

tanabarr commented Jun 16, 2023 •

edited

Loading

tanabarr commented Jun 21, 2023 •

edited

Loading

ashleypittman commented Aug 1, 2023 •

edited

Loading