Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmark numbers #411

Merged
merged 31 commits into from
Oct 14, 2024
+13 −13
Merged

Update benchmark numbers #411

Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
464e3eb
Update benchmark numbers
yiweny Jun 10, 2024
2742089
Update README.md
yiweny Jun 10, 2024
088be51
Update README.md
yiweny Jun 11, 2024
4b1967e
Update resnet
yiweny Aug 18, 2024
39b7b6f
Update README.md
yiweny Aug 18, 2024
120f3a0
Update README.md
yiweny Aug 20, 2024
1baa314
Merge branch 'master' into yiweny-patch-1
yiweny Aug 21, 2024
83d3025
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 21, 2024
dfcbe6e
Update README.md
yiweny Aug 21, 2024
d1b07a3
update
akihironitta Aug 28, 2024
e1ec94e
Merge branch 'master' into yiweny-patch-1
akihironitta Aug 28, 2024
4c073d6
Update README.md
yiweny Aug 31, 2024
96f4b51
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 31, 2024
e6646f0
Update README.md
yiweny Sep 4, 2024
bcee053
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2024
f11bc44
Update benchmark/README.md
akihironitta Sep 4, 2024
5ddd7f3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2024
ed65f25
Update benchmark/README.md
akihironitta Sep 4, 2024
feb64c3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2024
6b57c04
Update benchmark/README.md
akihironitta Sep 4, 2024
ccdb4c9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2024
e7355a5
Update benchmark/README.md
akihironitta Sep 9, 2024
9cf5308
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
7b55c70
Update benchmark/README.md
akihironitta Sep 9, 2024
86b1677
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
e55f4d1
Update README.md
yiweny Sep 14, 2024
c9ec325
Update README.md
yiweny Sep 14, 2024
0dac656
Update benchmark/README.md
akihironitta Sep 14, 2024
f11b441
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 14, 2024
8bdc5e9
Update README.md
yiweny Sep 18, 2024
5d82a1f
Merge branch 'master' into yiweny-patch-1
akihironitta Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,19 +154,19 @@ Experimental setting: 20 Optuna search trials for XGBoost and CatBoost. 5 Optuna

Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM. 3 Optuna search trials and 10 epochs training for deep learning models.

| | dataset_0 |
| :------------------ | :-------------------- |
| XGBoost | 0.966±0.000 (19327s) |
| CatBoost | 0.971±0.000 (223s) |
| LightGBM | **0.965±0.000 (67s)** |
| Trompt | 0.970±0.000 (12358s) |
| ResNet | 0.970±0.000 (672s) |
| MLP | 0.973±0.000 (223s) |
| FTTransformerBucket | 0.970±0.000 (2071s) |
| ExcelFormer | 0.969±0.000 (1785s) |
| FTTransformer | 0.971±0.000 (2918s) |
| TabNet | 0.970±0.000 (323s) |
| TabTransformer | 0.984±0.000 (318s) |
| | dataset_0 | dataset_1 | dataset_2 | dataset_3 | dataset_4 | dataset_5 |
| :------------------ | :-------------------- | :-------------------- | :------------------- | :-------------------- | :-------------------- | :-------------------- |
| XGBoost | 0.966±0.000 (19327s) | | | | | |
| CatBoost | 0.971±0.000 (223s) | | | | | |
| LightGBM | **0.965±0.000 (67s)** | | | | | |
| Trompt | 0.970±0.000 (12358s) | Too slow\* | Too slow\* | Too slow\* | 0.796±0.001 (157380s) | 0.799±0.001 (55577s) |
| ResNet | 0.969±0.000 (5800s) | 0.976±0.000 (57298s) | 0.655±0.000 (4612s) | 0.880±0.000 (20101s) | 0.757±0.000 (53493s) | 0.772±0.001 (3906s) |
| MLP | 0.973±0.000 (223s) | 0.980±0.000 (1104s) | 0.681±0.002 (91s) | 0.908±0.001 (529s) | 0.792±0.001 (680s) | 0.810±0.002 (117s) |
| FTTransformerBucket | 0.970±0.000 (2071s) | 0.977±0.000 (10953s) | 0.776±0.004 (1608s) | 0.896±0.002 (33883s) | 0.785±0.002 (145035s) | 0.808±0.003 (10407s) |
| ExcelFormer | 0.969±0.000 (1785s) | 0.978±0.001 (47745s) | 0.649±0.000 (15888s) | Too slow\* | Too slow\* | 0.782±0.001 (14729s) |
| FTTransformer | 0.969±0.000 (25604s) | 0.975±0.000 (219716s) | 0.670±0.004 (17119s) | 0.888±0.002 (323939s) | Too slow\* | 0.788±0.001 (103434s) |
| TabNet | 0.968±0.001 (3177s) | 0.974±0.000 (44035s) | 0.655±0.001 (2679s) | 0.891±0.001 (6800s) | 0.767±0.001 (6242s) | 0.784±0.001 (2307s) |
| TabTransformer | 0.984±0.000 (2813s) | 0.986±0.000 (13457s) | 0.713±0.002 (3767s) | 0.895±0.001 (5679s) | 0.764±0.001 (3763s) | 0.781±0.000 (3540s) |

### `task_type: multiclass_classification`

Expand Down
Loading