DAOS-14408 common: enable NDCTL for DCPM #14371

grom72 · 2024-05-15T06:35:00Z

This PR prepares DAOS to be used with NDCTL enabled in PMDK, which means:

NDCTL must not be used when non-DCPM (simulate PMem) - storage class: "ram" is used:
PMEMOBJ_CONF=sds.at_create=0 env variable disables NDCTL features in the PMDK
The default ULT stack size must be at least 20KiB to avoid stack overuse by PMDK with NDCTL enabled and be aligned with Linux page size.
ABT_THREAD_STACKSIZE=20480 env variable is used to increase the default ULT stack size.

This modification shall not affect md-on-ssd mode as long as storage class: "ram" is used for the first tier in the storage configuration.
This change does not require any configuration changes to existing systems.

The new PMDK package with NDCTL enabled (daos-stack/pmdk#38) will be delivered as soon as this PR is merged and backported to stable/2.6.

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-05-15T06:35:22Z

Ticket title is 'NDCTL must be enabled to provide support for RAS functionality in PMDK'
Status is 'In Review'
Labels: 'scrubbed_2.8,triaged'
https://daosio.atlassian.net/browse/DAOS-14408

daosbuild1 · 2024-05-15T16:51:14Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/5/execution/node/886/log

daosbuild1 · 2024-05-17T23:44:01Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/7/execution/node/329/log

daosbuild1 · 2024-05-17T23:44:21Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/7/execution/node/366/log

daosbuild1 · 2024-05-17T23:46:17Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/7/execution/node/363/log

daosbuild1 · 2024-05-17T23:46:24Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/7/execution/node/310/log

daosbuild1 · 2024-05-18T12:49:39Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/8/execution/node/1176/log

daosbuild1 · 2024-05-18T17:15:48Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/9/execution/node/1176/log

daosbuild1 · 2024-05-19T19:05:58Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/9/execution/node/1417/log

daosbuild1 · 2024-05-19T23:15:26Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/9/execution/node/1509/log

daosbuild1 · 2024-05-19T23:15:34Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/10/execution/node/1152/log

daosbuild1 · 2024-05-20T18:57:54Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/10/execution/node/1463/log

daosbuild1 · 2024-05-20T22:18:42Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/10/execution/node/1417/log

daosbuild1 · 2024-05-20T23:38:16Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/9/execution/node/1601/log

daosbuild1 · 2024-05-21T20:47:50Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/10/execution/node/1602/log

daosbuild1 · 2024-08-08T07:12:01Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/11/execution/node/273/log

daosbuild1 · 2024-08-08T07:12:24Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/11/execution/node/367/log

daosbuild1 · 2024-08-08T07:14:33Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/11/execution/node/343/log

daosbuild1 · 2024-08-08T07:15:04Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/11/execution/node/383/log

daosbuild1 · 2024-08-08T20:30:34Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/15/execution/node/758/log

daosbuild1 · 2024-08-09T12:22:55Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/17/execution/node/920/log

daosbuild1 · 2024-08-09T15:53:12Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/18/execution/node/920/log

daosbuild1 · 2024-08-09T20:32:12Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/18/execution/node/904/log

daosbuild1 · 2024-08-09T22:30:50Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/19/execution/node/870/log

daosbuild1 · 2024-08-10T00:56:14Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/16/execution/node/968/log

daosbuild1 · 2024-08-12T08:55:27Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/20/execution/node/962/log

brianjmurrell · 2024-09-23T12:39:13Z

@brianjmurrell retested with
Skip-func-test-leap15: false
Skip-func-test-el9: false
Skip-test-leap-15.4-rpms: false
Skip-test-el9-rpms: false
No test failure: build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14371/123/pipeline-graph

@grom72 Don't you also need the Test-tag: DaosBuild commit pragma to run the tests affected (by the daos-client-tests Requires: changes) in the el9 and leap15 stages?

Test-tag: DaosBuild PR-repos: pmdk@PR-38:11 Priority: 2 Cancel-prev-build: false Skip-func-test-leap15: false Skip-func-test-el9: false Skip-test-leap-15.4-rpms: false Skip-test-el9-rpms: false Allow-unstable-test: true Skip-func-hw-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

daosbuild1 · 2024-09-24T07:32:54Z

Test stage Functional on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/124/execution/node/1310/log

daosbuild1 · 2024-09-24T11:50:15Z

Test stage Functional on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/125/execution/node/538/log

Test-tag: DaosBuild PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_il:DAOS-16556 Priority: 2 Cancel-prev-build: false Skip-func-test-leap15: false Skip-func-test-el9: false Skip-test-leap-15.4-rpms: false Skip-test-el9-rpms: false Allow-unstable-test: true Skip-func-hw-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

daosbuild1 · 2024-09-24T22:25:57Z

Test stage Functional on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14371/126/execution/node/1232/log

Test-tag: DaosBuild PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Cancel-prev-build: false Skip-func-test-leap15: false Skip-func-test-el9: false Skip-test-leap-15.4-rpms: false Skip-test-el9-rpms: false Allow-unstable-test: true Skip-func-hw-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

daosbuild1 · 2024-09-25T06:43:45Z

Test stage Functional on Leap 15.5 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14371/127/testReport/

brianjmurrell

As you can see in your latest test run, DaosBuild.test_dfuse_daos_build_wt_il failed on Leap15 with:

06:40:54 DEBUG|     src/common.inc:336: *** Please install libndctl-dev/libndctl-devel/ndctl-devel >= 63.  Stop.

This is because of the concern I raised previously.

I suspect you also need to add this requirement (lib{nd,dax}ctl-dev packages) to the debian/control file's Package: daos-client-tests's Depends: clause. To be sure, there lots of others missing there that have been added to the daos.spec in the past without adding them to debian/control but since we are identifying this particular one missing, let's take the opportunity to add it in this PR so as not to increase technical debt. Indeed, it would be nice if we were doing even minimal testing on Ubuntu to help identify these kinds of gaps as they happen.

utils/rpms/daos.spec

brianjmurrell · 2024-09-25T12:54:07Z

utils/scripts/install-leap15.sh

Just curious why daxctl-devel is not needed on Leap15.

Any answer here @grom72?

debian/changelog

Test-tag: DaosBuild PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Cancel-prev-build: false Skip-func-test-leap15: false Skip-func-test-el9: false Skip-test-leap-15.4-rpms: false Skip-test-el9-rpms: false Allow-unstable-test: true Skip-func-hw-test: false Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

grom72 · 2024-09-25T19:05:42Z

I suspect you also need to add this requirement (lib{nd,dax}ctl-dev packages) to the debian/control file's Package: daos-client-tests's Depends: clause. To be sure, there lots of others missing there that have been added to the daos.spec in the past without adding them to debian/control but since we are identifying this particular one missing, let's take the opportunity to add it in this PR so as not to increase technical debt. Indeed, it would be nice if we were doing even minimal testing on Ubuntu to help identify these kinds of gaps as they happen.

I wonder why we have to add PMem-related dependencies to daos-client-tests. DAOS client has nothing to do with PMem directly.

Test-tag: DaosBuild PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Cancel-prev-build: false Skip-build-ubuntu20-rpm: false Skip-build-leap15-rpm: true Skip-build-leap15-icc: true Skip-build-el9-rpm: true Skip-nlt: true Skip-unit-tests: true Skip-func-test-vm: true Skip-test-rpms: true Skip-unit-test-memcheck: true Skip-func-test: true Skip-unit-tests: true Allow-unstable-test: true Skip-func-hw-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

grom72 · 2024-09-25T19:54:53Z

@brianjmurrell Leap15 has been fixed:
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/128/pipeline-console/?selected-node=1022

brianjmurrell · 2024-09-25T20:02:58Z

I wonder why we have to add PMem-related dependencies to daos-client-tests. DAOS client has nothing to do with PMem directly.

Because one of the daos (client) tests is to clone and build daos itself on daos (i.e. dfuse IIUC). It builds server+client. So the dependencies there are not actually for the client but for the building of daos.

I have more than once questioned the value of building daos specifically as a test of daos. While building something with a lot of files is probably a good test of daos/dfuse, I wonder if it needs to be as complicated as daos. Maybe the Linux kernel, for example, which is entirely self-contained in it's own source tree and doesn't have a brazillion dependencies is a better project. But that's just my opinion.

grom72 · 2024-09-25T20:12:18Z

I wonder why we have to add PMem-related dependencies to daos-client-tests. DAOS client has nothing to do with PMem directly.

Because one of the daos (client) tests is to clone and build daos itself on daos (i.e. dfuse IIUC). It builds server+client. So the dependencies there are not actually for the client but for the building of daos.

I have more than once questioned the value of building daos specifically as a test of daos. While building something with a lot of files is probably a good test of daos/dfuse, I wonder if it needs to be as complicated as daos. Maybe the Linux kernel, for example, which is entirely self-contained in it's own source tree and doesn't have a brazillion dependencies is a better project. But that's just my opinion.

Does it mean that in the case of all client tests, we install dependences required for DAOS build (*-dev)?
If yes, how can we know that the client works properly as we use it in a "dirty" environment with some development packages?

brianjmurrell · 2024-09-25T20:49:37Z

Does it mean that in the case of all client tests, we install dependences required for DAOS build (*-dev)?

Yes, *-devel to be accurate. Most specifically, we (want to) install the result of dnf builddep utils/rpms/daos.spec so we effectively mirror the BuildRequires: for EL8 in utils/rpms/daos.spec as Requires: in the daos-client-tests subpackage.

Yes, it's messy.

If yes, how can we know that the client works properly as we use it in a "dirty" environment with some development packages?

This is a valid question and point. Fortunately the tendency is to miss Requires: in the daos-client-tests package and not the daos-client package. But this is another good reason to switch from trying to build daos on daos to building something else that has no such large pool of dependencies. I propose[d] the Linux kernel. It seems to be a popular target for filesystem testing.

brianjmurrell · 2024-09-25T20:57:15Z

Skip-build-leap15-rpm: true
Skip-build-leap15-icc: true
Skip-build-el9-rpm: true
Skip-nlt: true
Skip-unit-tests: true
Skip-func-test-vm: true
Skip-test-rpms: true
Skip-unit-test-memcheck: true
Skip-func-test: true
Skip-unit-tests: true
Allow-unstable-test: true
Skip-func-hw-test: true

Maybe gatekeepers will disagree but I don't think it's valid to skip all of the build and test that is required to show the validity of the PR in what you expect to be the final commit before requesting landing.

All requested changes have been made.

PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Cancel-prev-build: false Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

grom72 · 2024-09-26T05:34:31Z

Skip-build-leap15-rpm: true
Skip-build-leap15-icc: true
Skip-build-el9-rpm: true
Skip-nlt: true
Skip-unit-tests: true
Skip-func-test-vm: true
Skip-test-rpms: true
Skip-unit-test-memcheck: true
Skip-func-test: true
Skip-unit-tests: true
Allow-unstable-test: true
Skip-func-hw-test: true

Maybe gatekeepers will disagree but I don't think it's valid to skip all of the build and test that is required to show the validity of the PR in what you expect to be the final commit before requesting landing.

The last builds were only to confirm that Leap rpms/Ubuntu pkg are properly built in a test environment.
9175174...e444b1f

The full validation has been done in the following builds:
w/o NDCTL
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/121/pipeline-graph/
w/ NDCTL
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/122/pipeline-graph/
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/123/pipeline-graph/

Anyhow I have triggered full validation if we want to have a consistent picture of validation in one place:
with NDCTL:
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/130/
without NDCTL:
https://build.hpdd.intel.com/job/daos-stack/job/daos/view/change-requests/job/PR-14371/131/

grom72 force-pushed the grom72/ndctl-validation branch from 38bd529 to 96548d9 Compare May 17, 2024 07:59

grom72 force-pushed the grom72/ndctl-validation branch from 1fb603d to cd0ed94 Compare August 8, 2024 07:06

grom72 force-pushed the grom72/ndctl-validation branch from a55f41f to e32501e Compare August 8, 2024 20:03

grom72 force-pushed the grom72/ndctl-validation branch from e32501e to 56669f2 Compare August 9, 2024 06:03

brianjmurrell previously requested changes Sep 25, 2024

View reviewed changes

grom72 dismissed stale reviews from kjacque, NiuYawei, and tanabarr via c6f4853 September 25, 2024 17:46

grom72 requested a review from brianjmurrell September 25, 2024 19:54

grom72 requested review from kjacque, NiuYawei and tanabarr September 25, 2024 20:14

grom72 added 2 commits September 26, 2024 06:39

Final validation.

e444b1f

PR-repos: pmdk@PR-38:11 Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

Final validation (no NDCTL).

8d7da45

Skip-list: test_dfuse_daos_build_wt_pil4dfs:DAOS-16556 Priority: 2 Cancel-prev-build: false Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@intel.com>

grom72 requested a review from a team September 26, 2024 05:35

tanabarr approved these changes Sep 26, 2024

View reviewed changes

DAOS-14408 common: enable NDCTL for DCPM #14371

Are you sure you want to change the base?

DAOS-14408 common: enable NDCTL for DCPM #14371

Conversation

grom72 commented May 15, 2024 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented May 15, 2024 • edited Loading

daosbuild1 commented May 15, 2024

daosbuild1 commented May 17, 2024

daosbuild1 commented May 17, 2024

daosbuild1 commented May 17, 2024

daosbuild1 commented May 17, 2024

daosbuild1 commented May 18, 2024

daosbuild1 commented May 18, 2024

daosbuild1 commented May 19, 2024

daosbuild1 commented May 19, 2024

daosbuild1 commented May 19, 2024

daosbuild1 commented May 20, 2024

daosbuild1 commented May 20, 2024

daosbuild1 commented May 20, 2024

daosbuild1 commented May 21, 2024

daosbuild1 commented Aug 8, 2024

daosbuild1 commented Aug 8, 2024

daosbuild1 commented Aug 8, 2024

daosbuild1 commented Aug 8, 2024

daosbuild1 commented Aug 8, 2024

daosbuild1 commented Aug 9, 2024

daosbuild1 commented Aug 9, 2024

daosbuild1 commented Aug 9, 2024

daosbuild1 commented Aug 9, 2024

daosbuild1 commented Aug 10, 2024

daosbuild1 commented Aug 12, 2024

brianjmurrell commented Sep 23, 2024

daosbuild1 commented Sep 24, 2024

daosbuild1 commented Sep 24, 2024

daosbuild1 commented Sep 24, 2024

daosbuild1 commented Sep 25, 2024

brianjmurrell left a comment

Choose a reason for hiding this comment

brianjmurrell Sep 25, 2024

Choose a reason for hiding this comment

brianjmurrell Sep 25, 2024

Choose a reason for hiding this comment

grom72 commented Sep 25, 2024 • edited Loading

grom72 commented Sep 25, 2024

brianjmurrell commented Sep 25, 2024

grom72 commented Sep 25, 2024

brianjmurrell commented Sep 25, 2024

brianjmurrell commented Sep 25, 2024

grom72 commented Sep 26, 2024

grom72 commented May 15, 2024 •

edited

Loading

github-actions bot commented May 15, 2024 •

edited

Loading

grom72 commented Sep 25, 2024 •

edited

Loading