Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

predict type should be also visible in flow.name #292

Open
giuseppec opened this issue Nov 21, 2016 · 0 comments
Open

predict type should be also visible in flow.name #292

giuseppec opened this issue Nov 21, 2016 · 0 comments

Comments

@giuseppec
Copy link
Member

OpenML computes the AUC also when the predictions are not probabilities and from
the list below, we don't really see that mlr.classif.randomForest(17) uses predict.type = "response" and mlr.classif.randomForest(21) uses predict.type = "prob".

res = listOMLRunEvaluations(task.id = 37)
res = res[grep("mlr.classif.randomForest", res$flow.name), ]
res[,c(1:5,8)]
#      run.id task.id setup.id flow.id                    flow.name area.under.roc.curve
#1297 1816243      37     9405    4782 mlr.classif.randomForest(17)             0.736701
#1612 1839540      37    24112    4849 mlr.classif.randomForest(21)             0.825437
#1613 1839541      37    24113    4849 mlr.classif.randomForest(21)             0.826119
#1614 1839542      37    24114    4849 mlr.classif.randomForest(21)             0.812451

One can only see this after

convertOMLFlowToMlr(getOMLFlow(4849))
#Learner classif.randomForest from package randomForest
#Type: classif
#Name: Random Forest; Short name: rf
#Class: classif.randomForest
#Properties: twoclass,multiclass,numerics,factors,ordered,prob,class.weights
#Predict-Type: prob
#Hyperparameters: 

convertOMLFlowToMlr(getOMLFlow(4782))
#Learner classif.randomForest from package randomForest
#Type: classif
#Name: Random Forest; Short name: rf
#Class: classif.randomForest
#Properties: twoclass,multiclass,numerics,factors,ordered,prob,class.weights,featimp
#Predict-Type: response
#Hyperparameters: 

My suggestions would be to modify the flow name so that one can see this info directly from the flow.name, e.g. something like mlr.prob.classif.randomForest(21).

@giuseppec giuseppec added this to the 1.4 milestone Apr 4, 2017
@giuseppec giuseppec removed this from the 1.4 milestone Feb 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants