Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-14261 engine: Add dss_chore for I/O forwarding (#13372) #14158

Merged
merged 14 commits into from
Apr 19, 2024
Merged

DAOS-14261 engine: Add dss_chore for I/O forwarding (#13372) #14158

merged 14 commits into from
Apr 19, 2024

Conversation

jolivier23
Copy link
Contributor

As requested by the Jira ticket, add a new I/O forwarding mechanism, dss_chore, to avoid creating a ULT for every forwarding task.

  • Forwarding of object I/O and DTX RPCs is converted to chores.

  • Cancelation is not implemented, because the I/O forwarding tasks themselves do not support cancelation yet.

  • In certain engine configurations, some xstreams do not need to initialize dx_chore_queue. This is left to future work.

Required-githooks: true
Skipped-githooks: clang

Change-Id: I8d6f9889f5562a8bc3683d26cb830672a8aa40f3

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@liw @jolivier23
DAOS-14261 engine: Add dss_chore for I/O forwarding (#13372)
9523ff1 
As requested by the Jira ticket, add a new I/O forwarding mechanism,
dss_chore, to avoid creating a ULT for every forwarding task.

  - Forwarding of object I/O and DTX RPCs is converted to chores.

  - Cancelation is not implemented, because the I/O forwarding tasks
    themselves do not support cancelation yet.

  - In certain engine configurations, some xstreams do not need to
    initialize dx_chore_queue. This is left to future work.

Required-githooks: true
Skipped-githooks: clang

Change-Id: I8d6f9889f5562a8bc3683d26cb830672a8aa40f3
Signed-off-by: Li Wei <wei.g.li@intel.com>
@jolivier23 jolivier23 requested review from a team as code owners April 12, 2024 18:35
@github-actions GitHub Actions
Copy link

Bug-tracker data:
Ticket title is 'limit ULT creation on helper xstream'
Status is 'Resolved'
https://daosio.atlassian.net/browse/DAOS-14261

@daosbuild1
Copy link
Collaborator

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14158/1/execution/node/164/log

daosbuild1
daosbuild1 reviewed Apr 12, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14158/1/execution/node/379/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14158/1/execution/node/405/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14158/1/execution/node/392/log

jolivier23 and others added 4 commits April 16, 2024 09:30
@jolivier23
b/334920724 - Fix google/2.4 pylint issues
5f61573 
disable CODEOWNERS for google branch
disable upstream hardware tests on branch by default
remove bad merge block
fix ordering of imports
Rename google-changeId.py
set option for dynamic fuse

Required-githooks: true

Change-Id: I8b0edfdd18a48c2f132102a49ddc44ee04b02586
Signed-off-by: Jeff Olivier <jeffolivier@google.com>
@mjmac @jolivier23
DAOS-15429 test: Fix Go unit tests (#13981)
fade6d6 
Fixes a couple of Go unit test failures:
  * Adjust gRPC client tests to behave correctly
    with newer gRPC versions
  * Don't run the syslogger test if syslogd is not
    running
  * Adjust some system tests to use paths that are
    more likely to exist on most systems

Change-Id: Ifb3198435113f3dc251c6c0822b66c157aa1a369
Signed-off-by: Michael MacDonald <mjmac@google.com>
@ashleypittman @jolivier23
DAOS-13490 test: Update valgrind suppressions. (#13142)
dd90047 
A go change has introduced some new failure traces, suppress them.

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>
@jolivier23
Merge branch 'jvolivie/patch' into jvolivie/backport_io_forward
fd4fc5a 
Change-Id: Ieb0610107d1f0b6c9cd5f84c9eca9013a07bac17
@jolivier23 jolivier23 requested review from a team as code owners April 16, 2024 21:42
daosbuild1
daosbuild1 reviewed Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

daltonbohning
daltonbohning reviewed Apr 16, 2024
View reviewed changes
.github/CODEOWNERS Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional to comment these out?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Just realized it's for the google branch :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, just making it so we don't bug everyone.. Unfortunately, until CODEOWNERS lands, it will still do so.

Jenkinsfile Outdated
@@ -289,16 +289,16 @@ pipeline {
description: 'Run the Functional on Ubuntu 20.04 test stage' +
' Requires CI_MORE_FUNCTIONAL_PR_TESTS')
booleanParam(name: 'CI_medium_TEST',
defaultValue: true,
defaultValue: false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debugging?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I'm disabling hardware testing on the google branch so we don't use too much resources

mchaarawi and others added 6 commits April 16, 2024 17:02
@mchaarawi @jolivier23
DAOS-15159 test: add a supression for new valgrind warning in NLT (#1…
1406159 
…3782)

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
@soumagne @jolivier23
DAOS-14669 test: switch tcp;ofi_rxm testing to tcp (#13365)
7f9d8c8 
Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>
Signed-off-by: Jeff Olivier <jeffolivier@google.com>
@mchaarawi @jolivier23
DAOS-15548 test: add new valgrind suppression for daos tool (#14081)
5915a99 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
@mchaarawi @jolivier23
DAOS-15159 test: add a supression for new valgrind warning in NLT (#1…
6dafd4a 
…3782)

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
@soumagne @jolivier23
DAOS-14669 test: switch tcp;ofi_rxm testing to tcp (#13365)
0a668b3 
Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>
@mchaarawi @jolivier23
DAOS-15548 test: add new valgrind suppression for daos tool (#14081)
b8ce6c4 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
@jolivier23
Merge branch 'jvolivie/patch' into jvolivie/backport_io_forward
618871c 
Change-Id: I13881b71e4baf1fb4fc85bf17161c523a155f728
@jolivier23 jolivier23 removed request for a team April 16, 2024 23:10
daosbuild1
daosbuild1 reviewed Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@jolivier23 jolivier23 changed the base branch from google/2.4 to jvolivie/patch April 16, 2024 23:12
daosbuild1
daosbuild1 reviewed Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14158/4/testReport/

Base automatically changed from jvolivie/patch to google/2.4 April 17, 2024 14:11
daosbuild1
daosbuild1 reviewed Apr 17, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@jolivier23
Merge branch 'google/2.4' into jvolivie/backport_io_forward
974f213 
Required-githooks: true

Change-Id: I8576dcb1cd42a20d1371132bdee10af0be4da798
Signed-off-by: Jeff Olivier <jeffolivier@google.com>
daosbuild1
daosbuild1 reviewed Apr 17, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14158/6/testReport/

@jolivier23 jolivier23 requested a review from liw April 18, 2024 22:40
@jolivier23
Copy link
Contributor Author

@liw would you mind checking that my cherry-pick (to google branch ) is correct?

liw
liw approved these changes Apr 18, 2024
View reviewed changes
Copy link
Contributor

@liw liw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem. It looks right to me. @jolivier23, make sure you also cherry-pick e09dae8, which is a critical follow-up fix for this patch.

@jolivier23
Merge branch 'google/2.4' into jvolivie/backport_io_forward
aa19cdb 
Change-Id: I9df7b2d0d0ed43a9e8ab19817faf37f2c509d293
daosbuild1
daosbuild1 reviewed Apr 19, 2024
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14158/7/testReport/

@jolivier23 jolivier23 requested a review from mjmac April 19, 2024 15:00
@jolivier23 jolivier23 merged commit dfd7211 into google/2.4 Apr 19, 2024
31 of 34 checks passed
@jolivier23 jolivier23 deleted the jvolivie/backport_io_forward branch April 19, 2024 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daosbuild1 daosbuild1 daosbuild1 left review comments

@daltonbohning daltonbohning daltonbohning left review comments

@liw liw liw approved these changes

@mjmac mjmac Awaiting requested review from mjmac

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@jolivier23 @daosbuild1 @liw @daltonbohning @mjmac @ashleypittman @mchaarawi @soumagne