Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce generic consolidator #480

Closed
wants to merge 1 commit into from

Conversation

antiguru
Copy link
Member

@antiguru antiguru commented May 1, 2024

Currently slightly slower than the existing in-place consolidate, mostly because it needs to copy the keys.

test consolidation::tests::test_consolidator_duration ... elapsed consolidator 3.956388988s ok
test consolidation::tests::test_consolidator_duration_vec ... elapsed vec 3.652386365s

@antiguru
Copy link
Member Author

antiguru commented May 3, 2024

It's improved:

➜  differential-dataflow git:(consolidator) taskset -c 4 cargo test --release --lib -- --nocapture duration --test-threads=1
   Compiling differential-dataflow v0.12.0 (/home/moritz/dev/repos/differential-dataflow)
    Finished `release` profile [optimized + debuginfo] target(s) in 36.85s
     Running unittests src/lib.rs (target/release/deps/differential_dataflow-59cc925a33da4197)

running 2 tests
test consolidation::tests::test_consolidator_duration ... elapsed consolidator 2.036071091s
ok
test consolidation::tests::test_consolidator_duration_vec ... elapsed vec 8.337145545s
ok

Currently slightly slower than the existing in-place consolidate, mostly
because it needs to copy the keys.

test consolidation::tests::test_consolidator_duration ... elapsed consolidator 3.956388988s
ok
test consolidation::tests::test_consolidator_duration_vec ... elapsed vec 3.652386365s

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Move instead of clone

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Correct logic

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Consolidator does not sort

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Explicitly specify target container

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Make it a trait, not sure if better

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

Consolidator on batcher

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>
@antiguru
Copy link
Member Author

Closing in favor of #494.

@antiguru antiguru closed this Jun 13, 2024
@antiguru antiguru deleted the consolidator branch June 13, 2024 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant