Skip to content

BenchMetrics Prob: Benchmarking of probabilistic error performance evaluation instruments for binary-classification problems

License

Notifications You must be signed in to change notification settings

gurol/BenchMetricsProb

Repository files navigation

A reproducible research compedium of

BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems

Last-changedate License: AGPL v3 ORCiD

Gürol Canbek (2023). BenchMetrics Prob: Benchmarking of probabilistic error / loss performance evaluation instruments for binary-classi cation problems. International Journal of Machine Learning and Cybernetics. doi: 10.1007/s13042-023-01826-5

🔗 Access the free-access full text here.

This repository provides BenchMetrics Prob probabilistic error/loss instrument calculator and simulation tool for benchmarking the robustness of 31 probabilistic error/loss measures/metrics via five criteria and seven simulation cases proposed in my article above.

The proposed method was tested on 30 probabilistic error instruments given in the table below and LogLoss as a probabilistic loss instrument.

Note: Please, cite my article if you would like to use and/or adapt the tool, simulation cases, methodology, and other materials provided and let me know. Thank you for your interest.

Subtype Instrument Name Abbreviation
(Raw) Mean Error ME
Squared Mean Squared Error MSE
Root Mean Square Error RMSE
Median Squared Error MdSE
Sum Squared Error SSE
Normalized Mean Squared Error (v1-5) nMSE
Absolute Mean Absolute Error MAE
Median Absolute Error MdAE
Maximum Absolute Error MxAE
Geometric Mean Absolute Error GMAE
Relative Mean Relative Absolute Error MRAE
Median Relative Absolute Error MdRAE
Relative Geometric Mean Relative Absolute Error GMRAE
Relative Absolute Error RAE
Relative Squared Error RSE
Percentage Mean Percentage Error MPE
Mean Absolute Percentage Error MAPE
Median Absolute Percentage Error MdAPE
Root Mean Square Percentage Error RMSPE
Root Median Square Percentage Error RMdSPE
Percentage (Symmetric) Symmetric Mean Absolute Percentage Error sMAPE
Normalized Symmetric Mean Absolute Percentage Error nsMAPE
Normalized Symmetric Median Absolute Percentage Error nsMdAPE
Scaled Mean Absolute Scaled Error MASE
Median Absolute Scaled Error MdASE
Root Mean Squared Scaled Error RMSSE