Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PushPull benchmark test #658

Merged
merged 4 commits into from
Nov 6, 2023
Merged

Add PushPull benchmark test #658

merged 4 commits into from
Nov 6, 2023

Conversation

sejongk
Copy link
Contributor

@sejongk sejongk commented Nov 2, 2023

What this PR does / why we need it:

This PR introduces a benchmark test for PushPull in Yorkie.

While Yorkie already features gRPC benchmarks for UpdateAndSync and UpdateProject, they may not provide accurate measurements of database I/O performance. As a result, these PushPull benchmarks are specifically designed to assess the performance of the server's PushPull functionality.

To ensure accurate results, we make use of b.StartTimer(), b.StopTimer(), and b.ResetTimer() to exclude time spent on costly setup operations from the benchmark results.

These benchmarks focus on various dimensions, including the type of operation (push/pull), the number of changes (100/1000) within a change pack, and the size of the snapshot in the change pack (3KB/30KB). It's important to note that the benchmark target database is currently only MongoDB.

We believe that these benchmarks will be valuable in estimating the potential improvements proposed in issue #597, assessing the effectiveness of database sharding (#530), and determining suitable snapshot intervals and thresholds for various CRDTs.

Plus, the benchmark task in CI is separated from the build task to run this in parallel.

Which issue(s) this PR fixes:

Fixes #600

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


Additional documentation:


Checklist:

  • Added relevant tests or not required
  • Didn't break anything

@sejongk sejongk marked this pull request as draft November 2, 2023 12:17
Copy link

codecov bot commented Nov 2, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (107498e) 48.89% compared to head (952b36a) 48.96%.
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #658      +/-   ##
==========================================
+ Coverage   48.89%   48.96%   +0.07%     
==========================================
  Files          69       69              
  Lines       10076    10090      +14     
==========================================
+ Hits         4927     4941      +14     
- Misses       4629     4631       +2     
+ Partials      520      518       -2     

see 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hackerwins hackerwins force-pushed the main branch 3 times, most recently from 7892142 to fdc2e1c Compare November 4, 2023 10:22
Copy link
Member

@hackerwins hackerwins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution.
I left a few comments.

Makefile Outdated Show resolved Hide resolved
test/bench/push_pull_bench_test.go Outdated Show resolved Hide resolved
@sejongk sejongk force-pushed the add-push-pull-benchmark branch from c4b13a3 to 952b36a Compare November 6, 2023 00:00
@sejongk sejongk marked this pull request as ready for review November 6, 2023 00:22
@hackerwins hackerwins self-requested a review November 6, 2023 03:08
Copy link
Member

@hackerwins hackerwins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

@hackerwins hackerwins merged commit 5500afb into main Nov 6, 2023
2 checks passed
@hackerwins hackerwins deleted the add-push-pull-benchmark branch November 6, 2023 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Suggest to add PushPull benchmark test
2 participants