Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16167 test: update soak test to use internal job scheduler #14775

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
Open

DAOS-16167 test: update soak test to use internal job scheduler #14775

wants to merge 45 commits into from
+550 −182

Conversation

mjean308
Copy link
Contributor

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@github-actions GitHub Actions
Copy link

Ticket title is 'Soak: update soak test to use internal job scheduler instead of depending on slurm'
Status is 'In Progress'
Labels: 'daos_framework,soak'
https://daosio.atlassian.net/browse/DAOS-16167

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/415/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/313/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/357/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/373/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/414/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/364/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/360/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/361/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/367/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/383/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/387/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/363/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/365/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/364/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/353/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/359/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/341/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/344/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/467/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/398/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/336/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/383/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/351/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/350/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/357/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/518/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/352/log

@mjean308
DAOS-16167 test: update soak test to use internal job scheduler
47e9f7c 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
include HDF5 vol env vars for container
be071c6 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/30/execution/node/812/log

@mjean308
Merge branch 'master' into mjean/DAOS-16167
e0f7950
@mjean308
Include updates fro feedback
c0a0656 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/31/execution/node/922/log

@mjean308
fix linting
d3d39f1 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
fix quotes
2dc93d9 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
change check for down nodes
5fa1404 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Add pragmas
8fde68e 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
exit soak if not enough nodes
7e608a6 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Add pragmas
5927833 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
069d32a 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
add logging message when job is cancelled
5ae46a6 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
cb74b98 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
fix logging
0798663 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
daltonbohning
daltonbohning reviewed Sep 24, 2024
View reviewed changes
src/tests/ftest/util/soak_utils.py
from test_utils_container import add_container

H_LOCK = threading.Lock()
id_counter = count(start=1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion - you might want to start with a larger number so the job ids are easily sortable (if you need that). E.g. with start= you'll get

1
2
...
9
10  <-- throws off sorting / printing ids in a list format

But e.g. start=1000, numbers will be aligned from 1000 to 9999

1001
1002
...
1009
1010
...
9999
10000 # <-- finally not aligned

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I need that because the id is added into the log names or it is used in a dictionary

src/tests/ftest/util/soak_utils.py Outdated
Comment on lines 441 to 444
job_queue.put(results)
# give time to update the queue before exiting
time.sleep(0.5)
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When job_queue.put returns, it should be updated with no need to wait. Did you find otherwise?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not remove it to try without the sleep yet. I will test on wolf

src/tests/ftest/util/soak_utils.py Outdated

for cmd in list(job_cmds):
script_file.write(cmd + "\n")
script_file.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be necessary, since you opened like with open(scriptfile, 'w') as script_file

Suggested change
script_file.close()

@mjean308
Merge branch 'master' into mjean/DAOS-16167
e61f9b5 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
Merge branch 'master' into mjean/DAOS-16167
ca4c755 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
include feedback
950f708 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/43/execution/node/936/log

@mjean308
Fix typo
e94e2b4 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true
Skipped-githooks: flake,pylint

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
fd7d289 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
Remove server logging from preteardown
42436bf 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true
Skipped-githooks: flake,pylint

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
c7c762a 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
5570550 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
Merge branch 'master' into mjean/DAOS-16167
7d6084d 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/49/execution/node/825/log

@mjean308
Merge branch 'master' into mjean/DAOS-16167
86dab47 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/50/execution/node/936/log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saurabhtandan saurabhtandan Awaiting requested review from saurabhtandan

@phender phender Awaiting requested review from phender

@rpadma2 rpadma2 Awaiting requested review from rpadma2

@daltonbohning daltonbohning Awaiting requested review from daltonbohning

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mjean308 @daosbuild1 @daltonbohning @rpadma2