DAOS-16585 tests: Fix NLT handling of __fxstat detection

[techbasset]

Use strace to determine whether calls to __fxstat actually happen when using a utility/command the IL is being tested on, and stop treating it as an error to not see __fxstat when it's not used.

Required-githooks: true

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is 'NLT test failures under Ubuntu 22.04'
Status is 'Open'
Labels: 'google-cloud-daos'
https://daosio.atlassian.net/browse/DAOS-16585

[daosbuild1]

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/3/display/redirect

[daosbuild1]

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/3/display/redirect

[daosbuild1]

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/3/display/redirect

[daosbuild1]

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/3/display/redirect

[daosbuild1]

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/3/display/redirect

[daosbuild1]

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/4/display/redirect

[daosbuild1]

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/4/display/redirect

[daosbuild1]

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/4/display/redirect

[daosbuild1]

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/4/display/redirect

[daosbuild1]

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/4/display/redirect

[daosbuild1]

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/6/display/redirect

[daosbuild1]

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/6/display/redirect

[daosbuild1]

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/6/display/redirect

[daosbuild1]

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/6/display/redirect

[daosbuild1]

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15150/6/display/redirect

[jolivier23]

I don't recall if pylint will still complain on every PR thereafter but it used to be the case. You could either remove strace from the comment or add a comment such as

daos/utils/node_local_test.py

Line 3402 in b5d2047

# pylint: disable=wrong-spelling-in-comment

[jolivier23]

this one is simple enough to address, then you don't need explicit close

[ashleypittman]

This would work in as much as if glibc is behaving differently then the specific check that's failing will be disabled. What we should be testing however is if fstat is being intercepted properly, historically and what the code currently does is check the logs for the wrapper function however dfuse now has per operation statistics available to the client, a more comprehensive solution would be to sample the fstat count before and after the command is invoked as a way of knowing if it had been intercepted or not.

One complexity here is that the first fstat of every file is forwarded so that the st_dev value can be loaded/cached so in order to properly fix this il_stat may need to be passed in the number of files which are accessed. I'll see if I can get a PR together on this basis.

[ashleypittman]

This would be called for every il_cmd invocation and there are probably dozens so it could/should be saved in conf

[techbasset]

I thought about that and wanted to not assume different executables ended up using the same libraries. And, I don't think it's going to affect overall runtime to just do this every time. Do you feel differently?

[ashleypittman]

I'd missed that it was running the actual command rather than just a generic unix command here. This means the command has to be idempotent and perform the same operations on subsequent invocations but looking at the places where this is called it seems it probably is.

[ashleypittman]

Is this a part of the fix or something else?

[jolivier23]

Is this a part of the fix or something else?

there was a recent change that made it such that made go 1.22 a requirement and ubuntu 22 has 1.18. Not sure why the builds didn't fail on that patch.

[ashleypittman]

This would probably be better landing as part of #15174 where the go developers can review it.

That said, this script normally just installs packages, https://github.com/daos-stack/daos/blob/master/utils/docker/Dockerfile.ubuntu would be a better place for this code, or perhaps a utils/scripts/helpers/repo-helper-debian.sh script to match what rocky does.

[techbasset]

This would work in as much as if glibc is behaving differently then the specific check that's failing will be disabled. What we should be testing however is if fstat is being intercepted properly, historically and what the code currently does is check the logs for the wrapper function however dfuse now has per operation statistics available to the client, a more comprehensive solution would be to sample the fstat count before and after the command is invoked as a way of knowing if it had been intercepted or not.

One complexity here is that the first fstat of every file is forwarded so that the st_dev value can be loaded/cached so in order to properly fix this il_stat may need to be passed in the number of files which are accessed. I'll see if I can get a PR together on this basis.

FWIW I kind of like the strace approach as a general solution as it lets you establish a ground truth about what's actually happening without making assumptions. One can imagine extending this to have the strace dictate all the calls you should be intercepting. @jolivier23 pointed out, for instance, that "newfstat" shows up and is probably not being intercepted right now? shrug

Meantime we (Google) would like some fix here ASAP to unblock our own client testing. So: request to separate a short term fix from a longer term more complete solution?

[ashleypittman]

FWIW I kind of like the strace approach as a general solution as it lets you establish a ground truth about what's actually happening without making assumptions. One can imagine extending this to have the strace dictate all the calls you should be intercepting. @jolivier23 pointed out, for instance, that "newfstat" shows up and is probably not being intercepted right now? shrug

I think we're for different points but with the same end. The current code checks is a particular glibc implementation of fstat is being intercepted, if it's not then is that because the interception is broken or because a different implementation is in use? Using strace will hide the second failure mode.

Overall this is really just a quick smoke test and tracking for filenames in the log file is a bit of a hack, to do this properly we'd write custom c code and call it from ftest with an appropriate harness. For now, and to make progress here I'd be happy to simply disable the check_fstat check, I can re-work NLT to re-enable this smoke test using more modern dfuse/ioil features and having it disabled for a short period of time won't be an issue IMO.

Meantime we (Google) would like some fix here ASAP to unblock our own client testing. So: request to separate a short term fix from a longer term more complete solution?