Add SapientML to automl benchmark #630
Merged
+232
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kosaku Kimura <kimura.kosaku@fujitsu.com>
PGijsbers
requested changes
Jul 29, 2024
|
@kimusaku Are you still planning to work on this? |
|
@PGijsbers Yes, we are still working on this, especially in making SapientML Python 3.9-compatible. We will be able to update the PR soon. |
* WIP Signed-off-by: Kosaku Kimura <kimura.kosaku@fujitsu.com> * Issue fix in setup.sh for SapientML * Updation of config file as Sapientml supports Python version 3.9 Signed-off-by: HimanshuRRai <himanshu.rai@fujitsu.com> --------- Signed-off-by: Kosaku Kimura <kimura.kosaku@fujitsu.com> Signed-off-by: HimanshuRRai <himanshu.rai@fujitsu.com> Co-authored-by: muhammed-nafi-k-a <muhammednafi.a@fujitsu.com> Co-authored-by: HimanshuRRai <himanshu.rai@fujitsu.com>
PGijsbers
reviewed
Oct 10, 2024
Signed-off-by: HimanshuRRai <himanshu.rai@fujitsu.com>
PGijsbers
approved these changes
Nov 9, 2024
There was a problem hiding this comment.
Very sorry for the slow reply. I finally had a chance to run it locally and it seems to work fine... except when running the validation benchmark (python runbenchmark.py sapientml validation -f 0):
Summing up scores for current run:
id task fold framework constraint result metric duration seed info
openml.org/t/9910 bioresponse 0 SapientML test 0.797310 auc 59.1 432708598 NaN
openml.org/t/125920 dresses-sales 0 SapientML test 0.597701 auc 19.3 432708598 NaN
openml.org/t/2079 eucalyptus 0 SapientML test -11.689800 neg_logloss 20.6 432708598 NaN
openml.org/t/167125 internet-advertisements 0 SapientML test 0.929463 auc 59.0 432708598 NaN
openml.org/t/9950 micro-mass 0 SapientML test -1.242880 neg_logloss 35.2 432708598 NaN
openml.org/t/3917 kc1 0 SapientML test 0.720932 auc 23.5 432708598 NaN
openml.org/t/168868 APSFailure 0 SapientML test 0.897341 auc 206.1 432708598 NaN
openml.org/t/168877 diabetes130US 0 SapientML test NaN neg_logloss 65.8 432708598 NoResultError: All candidate scripts failed. Final script is not saved.
openml.org/t/211985 census-income 0 SapientML test 0.709671 auc 92.7 432708598
The scripts generated can be found here:https://gist.github.com/PGijsbers/ed825dab98b85ab43b7bbbc00781b694
I could merge it (sapient is not part of CI currently), but perhaps you'd prefer to fix it first? If it's reproducible for you.
|
@PGijsbers thanks so much. We have fixed the issue, and now this PR can be merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SapientML is an AutoML technology that can learn from a corpus of existing datasets and their human-written pipelines, and efficiently generate a high-quality pipeline for a predictive task on a new dataset.