Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert into arrow without going via into_canonical #1736

Merged
merged 4 commits into from
Jan 2, 2025
Merged

Convert into arrow without going via into_canonical #1736

merged 4 commits into from
Jan 2, 2025

Conversation

joseph-isaacs
Copy link
Member

No description provided.

@joseph-isaacs joseph-isaacs requested a review from a10y December 20, 2024 16:18
@joseph-isaacs joseph-isaacs enabled auto-merge (squash) December 20, 2024 16:19
@a10y a10y added the benchmark Run benchmarks on this branch label Dec 20, 2024
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Dec 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: random_access

Table of Results
name PR ef45916 base 0cc9df2 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 3.94941e+06 3.90163e+06 1.01225 ns
random-access/vortex-local-fs 3.70636e+06 3.59683e+06 1.03045 ns
random-access/parquet-tokio-local-disk 2.35399e+08 2.29978e+08 1.02357 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H

Table of Results
name PR ef45916 base 0cc9df2 ratio (PR/base) unit
tpch_q01/arrow 523689474 5.79769e+08 0.903273 ns
tpch_q01/parquet 752016015 7.91031e+08 0.950678 ns
tpch_q01/vortex-file-compressed 458848320 4.75948e+08 0.964073 ns
tpch_q02/arrow 141445820 1.49857e+08 0.943872 ns
tpch_q02/parquet 172473148 1.87972e+08 0.917545 ns
tpch_q02/vortex-file-compressed 143036956 1.67289e+08 0.855031 ns
tpch_q03/arrow 166698618 1.96109e+08 0.850028 ns
tpch_q03/parquet 365403504 3.98109e+08 0.917848 ns
tpch_q03/vortex-file-compressed 207740270 2.31411e+08 0.897712 ns
tpch_q04/arrow 176093060 1.92567e+08 0.914453 ns
tpch_q04/parquet 214843367 2.30018e+08 0.93403 ns
tpch_q04/vortex-file-compressed 157027918 1.72879e+08 0.908309 ns
tpch_q05/arrow 313179694 3.71855e+08 0.842209 ns
tpch_q05/parquet 484630383 5.62277e+08 0.861907 ns
tpch_q05/vortex-file-compressed 343921911 3.93738e+08 0.873478 ns
tpch_q06/arrow 25719729 2.90057e+07 0.886713 ns
tpch_q06/parquet 148834385 1.56556e+08 0.950677 ns
tpch_q06/vortex-file-compressed 61744107 6.5542e+07 0.942054 ns
tpch_q07/arrow 620894252 6.62585e+08 0.937079 ns
tpch_q07/parquet 756458300 8.60542e+08 0.879049 ns
tpch_q07/vortex-file-compressed 619463653 6.95405e+08 0.890795 ns
tpch_q08/arrow 258174366 2.81347e+08 0.917636 ns
tpch_q08/parquet 529697057 5.89354e+08 0.898776 ns
tpch_q08/vortex-file-compressed 323410248 3.70064e+08 0.873931 ns
tpch_q09/arrow 461269433 5.24889e+08 0.878794 ns
tpch_q09/parquet 773472403 8.208e+08 0.942339 ns
tpch_q09/vortex-file-compressed 564207971 6.32619e+08 0.89186 ns
tpch_q10/arrow 260172378 2.94017e+08 0.884888 ns
tpch_q10/parquet 509486576 5.61592e+08 0.907218 ns
tpch_q10/vortex-file-compressed 265017047 3.08812e+08 0.858181 ns
tpch_q11/arrow 139727416 1.58358e+08 0.882352 ns
tpch_q11/parquet 144829133 1.74141e+08 0.831676 ns
tpch_q11/vortex-file-compressed 125743461 1.43285e+08 0.877575 ns
tpch_q12/arrow 182248559 1.91653e+08 0.950929 ns
tpch_q12/parquet 325568422 3.46932e+08 0.938421 ns
tpch_q12/vortex-file-compressed 225581187 2.47698e+08 0.910711 ns
tpch_q13/arrow 169617438 1.97682e+08 0.858032 ns
tpch_q13/parquet 306561910 3.42726e+08 0.894482 ns
tpch_q13/vortex-file-compressed 179432948 1.9856e+08 0.903669 ns
tpch_q14/arrow 36081726 4.20808e+07 0.857439 ns
tpch_q14/parquet 228637613 2.41163e+08 0.948061 ns
tpch_q14/vortex-file-compressed 67055338 6.9987e+07 0.958111 ns
tpch_q15/arrow 65911429 7.17694e+07 0.918377 ns
tpch_q15/parquet 323904236 3.38803e+08 0.956024 ns
tpch_q15/vortex-file-compressed 106562230 1.22543e+08 0.869588 ns
tpch_q16/arrow 101457231 1.17233e+08 0.865431 ns
tpch_q16/parquet 116603174 1.3126e+08 0.888338 ns
tpch_q16/vortex-file-compressed 106519383 1.18494e+08 0.898946 ns
tpch_q17/arrow 605206240 7.2813e+08 0.831178 ns
tpch_q17/parquet 672610974 7.71629e+08 0.871677 ns
tpch_q17/vortex-file-compressed 573854918 6.41732e+08 0.894228 ns
tpch_q18/arrow 1122013936 1.27986e+09 0.876667 ns
tpch_q18/parquet 1344204482 1.56972e+09 0.856331 ns
tpch_q18/vortex-file-compressed 1153066670 1.33024e+09 0.866813 ns
tpch_q19/arrow 151915017 1.57567e+08 0.964131 ns
tpch_q19/parquet 416069282 4.38714e+08 0.948383 ns
tpch_q19/vortex-file-compressed 116227703 1.15597e+08 1.00545 ns
tpch_q20/arrow 206230093 2.37379e+08 0.868781 ns
tpch_q20/parquet 340916553 3.71667e+08 0.917264 ns
tpch_q20/vortex-file-compressed 230534118 2.68836e+08 0.857526 ns
tpch_q21/arrow 972285568 1.0955e+09 0.88753 ns
tpch_q21/parquet 1094191076 1.27173e+09 0.860397 ns
tpch_q21/vortex-file-compressed 906879267 1.02646e+09 0.883504 ns
tpch_q22/arrow 76743925 8.12659e+07 0.944355 ns
tpch_q22/parquet 107393658 1.149e+08 0.934667 ns
tpch_q22/vortex-file-compressed 80811189 8.74517e+07 0.924067 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: Clickbench

Table of Results
name PR ef45916 base 0cc9df2 ratio (PR/base) unit
clickbench_q00/parquet 1862574 1.83278e+06 1.01625 ns
clickbench_q01/parquet 59903376 6.00797e+07 0.997065 ns
clickbench_q02/parquet 117410907 1.20173e+08 0.977013 ns
clickbench_q03/parquet 83719298 8.43135e+07 0.992953 ns
clickbench_q04/parquet 660321139 6.51036e+08 1.01426 ns
clickbench_q05/parquet 836001790 8.1751e+08 1.02262 ns
clickbench_q06/parquet 1963445 1.93515e+06 1.01462 ns
clickbench_q07/parquet 62379265 6.23483e+07 1.0005 ns
clickbench_q08/parquet 1023818855 1.01317e+09 1.01051 ns
clickbench_q09/parquet 1042220027 1.03282e+09 1.0091 ns
clickbench_q10/parquet 268893451 2.59684e+08 1.03546 ns
clickbench_q11/parquet 315608664 3.13208e+08 1.00766 ns
clickbench_q12/parquet 833526863 8.34985e+08 0.998254 ns
clickbench_q13/parquet 1092717112 1.17952e+09 0.926408 ns
clickbench_q14/parquet 949395134 9.4991e+08 0.999458 ns
clickbench_q15/parquet 740213227 7.51022e+08 0.985608 ns
clickbench_q16/parquet 1938289480 1.9736e+09 0.982107 ns
clickbench_q17/parquet 1716174519 1.69507e+09 1.01245 ns
clickbench_q18/parquet 3899204712 3.84887e+09 1.01308 ns
clickbench_q19/parquet 65287815 6.48957e+07 1.00604 ns
clickbench_q20/parquet 1219354855 1.22272e+09 0.997245 ns
clickbench_q21/parquet 1461920917 1.48325e+09 0.985621 ns
clickbench_q22/parquet 2477752463 2.47175e+09 1.00243 ns
clickbench_q23/parquet 8376112376 8.31425e+09 1.00744 ns
clickbench_q24/parquet 532984350 5.29351e+08 1.00686 ns
clickbench_q25/parquet 514384295 5.07067e+08 1.01443 ns
clickbench_q26/parquet 590207035 5.96233e+08 0.989893 ns
clickbench_q27/parquet 1653928186 1.65855e+09 0.997214 ns
clickbench_q28/parquet 11084981235 1.15534e+10 0.959459 ns
clickbench_q29/parquet 424028679 4.15094e+08 1.02152 ns
clickbench_q30/parquet 850784574 8.42071e+08 1.01035 ns
clickbench_q31/parquet 860422699 8.82576e+08 0.9749 ns
clickbench_q32/parquet 3290095364 3.21487e+09 1.0234 ns
clickbench_q33/parquet 2841572025 2.80578e+09 1.01276 ns
clickbench_q34/parquet 2797575155 2.78542e+09 1.00436 ns
clickbench_q35/parquet 1260792730 1.25584e+09 1.00395 ns
clickbench_q36/parquet 172078990 1.72904e+08 0.995231 ns
clickbench_q37/parquet 86710950 8.79277e+07 0.986161 ns
clickbench_q38/parquet 116756247 1.15177e+08 1.01371 ns
clickbench_q39/parquet 364181003 3.57456e+08 1.01881 ns
clickbench_q40/parquet 51866083 5.26903e+07 0.984357 ns
clickbench_q41/parquet 47963206 4.91058e+07 0.976732 ns
clickbench_q42/parquet 67349870 6.7518e+07 0.99751 ns
clickbench_q00/vortex-file-compressed 2028333 1.99323e+06 1.01761 ns
clickbench_q01/vortex-file-compressed 29716552 2.70084e+07 1.10027 ns
clickbench_q02/vortex-file-compressed 114529422 9.27529e+07 1.23478 ns
clickbench_q03/vortex-file-compressed 78219094 7.2729e+07 1.07549 ns
clickbench_q04/vortex-file-compressed 614599709 5.77031e+08 1.06511 ns
clickbench_q05/vortex-file-compressed 604666109 6.09085e+08 0.992745 ns
clickbench_q06/vortex-file-compressed 2052715 2.04213e+06 1.00518 ns
clickbench_q07/vortex-file-compressed 47196083 4.94384e+07 0.954644 ns
clickbench_q08/vortex-file-compressed 985233448 9.61888e+08 1.02427 ns
clickbench_q09/vortex-file-compressed 872589009 8.59088e+08 1.01571 ns
clickbench_q10/vortex-file-compressed 199677738 2.04217e+08 0.977773 ns
clickbench_q11/vortex-file-compressed 235628917 2.43149e+08 0.969074 ns
clickbench_q12/vortex-file-compressed 563130172 5.79523e+08 0.971714 ns
clickbench_q13/vortex-file-compressed 913460971 9.20046e+08 0.992843 ns
clickbench_q14/vortex-file-compressed 688375041 6.86666e+08 1.00249 ns
clickbench_q15/vortex-file-compressed 697358218 6.98599e+08 0.998224 ns
clickbench_q16/vortex-file-compressed 1646312624 1.61229e+09 1.0211 ns
clickbench_q17/vortex-file-compressed 1538099710 1.51995e+09 1.01194 ns
clickbench_q18/vortex-file-compressed 3716678200 3.68334e+09 1.00905 ns
clickbench_q19/vortex-file-compressed 52375305 4.90437e+07 1.06793 ns
clickbench_q20/vortex-file-compressed 721837566 7.31467e+08 0.986835 ns
clickbench_q21/vortex-file-compressed 991392225 9.86588e+08 1.00487 ns
clickbench_q22/vortex-file-compressed 2226021157 2.20208e+09 1.01087 ns
clickbench_q23/vortex-file-compressed 5372577963 5.11675e+09 1.05 ns
clickbench_q24/vortex-file-compressed 381743143 3.66385e+08 1.04192 ns
clickbench_q25/vortex-file-compressed 336072572 3.28066e+08 1.02441 ns
clickbench_q26/vortex-file-compressed 433809732 4.13145e+08 1.05002 ns
clickbench_q27/vortex-file-compressed 1449071772 1.4433e+09 1.004 ns
clickbench_q28/vortex-file-compressed 10608828606 1.05503e+10 1.00555 ns
clickbench_q29/vortex-file-compressed 669451756 6.57352e+08 1.01841 ns
clickbench_q30/vortex-file-compressed 582484328 5.86645e+08 0.992909 ns
clickbench_q31/vortex-file-compressed 599953379 5.79179e+08 1.03587 ns
clickbench_q32/vortex-file-compressed 3126405281 3.08982e+09 1.01184 ns
clickbench_q33/vortex-file-compressed 2206485498 2.20039e+09 1.00277 ns
clickbench_q34/vortex-file-compressed 2208802588 2.19e+09 1.00859 ns
clickbench_q35/vortex-file-compressed 1386738803 1.38274e+09 1.00289 ns
clickbench_q36/vortex-file-compressed 72458862 7.27528e+07 0.99596 ns
clickbench_q37/vortex-file-compressed 57341850 5.13798e+07 1.11604 ns
clickbench_q38/vortex-file-compressed 51552920 5.03679e+07 1.02353 ns
clickbench_q39/vortex-file-compressed 146297248 1.21613e+08 1.20298 ns
clickbench_q40/vortex-file-compressed 35575051 3.66238e+07 0.971364 ns
clickbench_q41/vortex-file-compressed 34789410 3.44813e+07 1.00893 ns
clickbench_q42/vortex-file-compressed 41261554 3.60881e+07 1.14336 ns

@joseph-isaacs joseph-isaacs requested a review from a10y January 2, 2025 10:07
@joseph-isaacs joseph-isaacs merged commit 14f51e8 into develop Jan 2, 2025
21 checks passed
@joseph-isaacs joseph-isaacs deleted the ji/into-arrow-wo-canon branch January 2, 2025 13:35
AdamGS added a commit that referenced this pull request Jan 3, 2025
@AdamGS
Revert "Convert into arrow without going via into_canonical (#1736)"
b6ef238 
This reverts commit 14f51e8.
@AdamGS AdamGS mentioned this pull request Jan 3, 2025
Merged
AdamGS added a commit that referenced this pull request Jan 3, 2025
@AdamGS
Revert "Convert into arrow without going via into_canonical" (#1797)
5a704cf 
Reverts #1736
@AdamGS AdamGS mentioned this pull request Jan 3, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@a10y a10y a10y approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@joseph-isaacs @a10y