Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16167 test: update soak test to use internal job scheduler #14775

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open

DAOS-16167 test: update soak test to use internal job scheduler #14775

wants to merge 35 commits into from

Conversation

mjean308
Copy link
Contributor

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@github-actions GitHub Actions
Copy link

Ticket title is 'Soak: update soak test to use internal job scheduler instead of depending on slurm'
Status is 'In Progress'
Labels: 'daos_framework,soak'
https://daosio.atlassian.net/browse/DAOS-16167

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/415/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/313/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/357/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/373/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/414/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/364/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/360/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/361/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/367/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/383/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/387/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/363/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/365/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/364/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/353/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/359/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/341/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/344/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/467/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/398/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/336/log

@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/383/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/351/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/350/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/357/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/518/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/352/log

@mjean308
DAOS-16167 test: update soak test to use internal job scheduler
47e9f7c 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
debug
e784fa3 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
daltonbohning
daltonbohning reviewed Aug 7, 2024
View reviewed changes
src/tests/ftest/util/soak_utils.py Outdated
Comment on lines 283 to 284
run_local(self.log, cmd3, timeout=600)
run_local(self.log, cmd4, timeout=600)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll want to pass check=True or it won't raise an exception

src/tests/ftest/util/soak_utils.py Outdated
Comment on lines 289 to 290
run_local(self.log, cmd, timeout=600)
run_local(self.log, cmd2, timeout=600)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll want to pass check=True or it won't raise an exception

src/tests/ftest/util/job_manager_utils.py Show resolved Hide resolved
src/tests/ftest/util/soak_test_base.py Outdated
Comment on lines 311 to 313
env = ";".join([f"export LD_LIBRARY_PATH={lib_path}",
f"export PATH={path}",
f"export VIRTUAL_ENV={v_env}"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend using EnvironmentVariables here. It's just a dictionary with some helper methods

from command_utils_base import EnvironmentVariables

env = EnvironmentVariables()
env['FOO'] = 'bar'
env_str = env.to_export_str()  # export FOO=bar;

src/tests/ftest/util/soak_test_base.py
Comment on lines +363 to +364
while not job_queue.empty():
job_results = job_queue.get()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this submits potentially multiple jobs and doesn't schedule anymore jobs until all of those are complete. I think you'd rather submit and complete asynchronously? I could help with that improvement if needed. There are a few approaches we could take

src/tests/ftest/util/soak_test_base.py Outdated
else:
# update soak_results to include job id NOT run and set state = CANCELLED
for job in job_id_list:
if job not in list(self.soak_results.keys()):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if job not in list(self.soak_results.keys()):
if job not in self.soak_results:

src/tests/ftest/util/soak_utils.py Outdated
@@ -233,7 +254,7 @@ def get_daos_server_logs(self):
self (obj): soak obj
"""
daos_dir = self.outputsoak_dir + "/daos_server_logs"
logs_dir = os.environ.get("DAOS_TEST_LOG_DIR", "/var/tmp/daos_testing/")
logs_dir = os.environ.get("DAOS_TEST_LOG_DIR", "/tmp/")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should always be defined now

Suggested change
logs_dir = os.environ.get("DAOS_TEST_LOG_DIR", "/tmp/")
logs_dir = self.test_env.log_dir

src/tests/ftest/util/soak_utils.py
Comment on lines +470 to +472
job_queue.put(results)
# give time to update the queue before exiting
time.sleep(0.5)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? As far as I'm aware, the queue should be updated as soon as put returns

src/tests/ftest/util/soak_utils.py Outdated
for key, value in list(sbatch_params.items()):
if value is not None:
if key == "error":
value = value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
value = value

@mjean308
Check for all jobs to be done
cbae14b 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
cef7710
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/24/execution/node/1446/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/24/execution/node/1551/log

@mjean308
Merge branch 'master' into mjean/DAOS-16167
a0a4aca 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
update jobscript VIRTUAL_ENV check
4cd4cb1 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
2f0eb23 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
include join
f4a8e3c 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/28/execution/node/944/log

@mjean308
Merge branch 'master' into mjean/DAOS-16167
3b91628
@mjean308
include HDF5 vol env vars for container
be071c6 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/30/execution/node/812/log

@mjean308
Merge branch 'master' into mjean/DAOS-16167
e0f7950
@mjean308
Include updates fro feedback
c0a0656 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/31/execution/node/922/log

@mjean308
fix linting
d3d39f1 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
fix quotes
2dc93d9 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
change check for down nodes
5fa1404 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Add pragmas
8fde68e 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
exit soak if not enough nodes
7e608a6 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Add pragmas
5927833 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
069d32a 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
add logging message when job is cancelled
5ae46a6 
Signed-off-by: Maureen Jean <maureen.jean@intel.com>
@mjean308
Merge branch 'master' into mjean/DAOS-16167
cb74b98 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke
@mjean308
fix logging
0798663 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Signed-off-by: Maureen Jean <maureen.jean@intel.com>
daltonbohning
daltonbohning reviewed Sep 24, 2024
View reviewed changes
src/tests/ftest/util/soak_utils.py
from test_utils_container import add_container

H_LOCK = threading.Lock()
id_counter = count(start=1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion - you might want to start with a larger number so the job ids are easily sortable (if you need that). E.g. with start= you'll get

1
2
...
9
10  <-- throws off sorting / printing ids in a list format

But e.g. start=1000, numbers will be aligned from 1000 to 9999

1001
1002
...
1009
1010
...
9999
10000 # <-- finally not aligned

src/tests/ftest/util/soak_utils.py
Comment on lines +441 to +444
job_queue.put(results)
# give time to update the queue before exiting
time.sleep(0.5)
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When job_queue.put returns, it should be updated with no need to wait. Did you find otherwise?

src/tests/ftest/util/soak_utils.py

for cmd in list(job_cmds):
script_file.write(cmd + "\n")
script_file.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be necessary, since you opened like with open(scriptfile, 'w') as script_file

Suggested change
script_file.close()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daltonbohning daltonbohning daltonbohning left review comments

@saurabhtandan saurabhtandan Awaiting requested review from saurabhtandan

@phender phender Awaiting requested review from phender

@rpadma2 rpadma2 Awaiting requested review from rpadma2

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mjean308 @daosbuild1 @daltonbohning @rpadma2