This repository hosts a tool that simplifies and accelerates the model comparison process. 🔄
How It Works:
- Provide Your Dataset: Input your dataset.
- Bring Your Models: Include your pre-trained models.
- Automated Insights: Easily generate metrics, visualizations, and insights to compare model performances.
This tool is designed to make model evaluation straightforward and efficient. 📈🤖
The actual reason for building this repo was because I mostly work on regression tasks and constantly find myself frustrated with the kaggle-ization of work. I find people often pick a metric like RMSE at random and then go all in on optimizing it. I like to view a range of metrics (rather than just one) and I often implement methods that examine the performance without aggregating. I was inspired particularly by Vincent Warmerdams DoubtLab.
- Automated model comparison.
- Computes a variety of standard metrics.
- Easily extensible to include extra metrics.
- Returns data rows for deeper investigation.
- Automatically generates and saves wide range of useful visualisations.
To install the speed-insights
package, you can use Poetry, a dependency management tool for Python.
-
Make sure you have Poetry installed. If not, you can install it by following the official Poetry installation guide.
-
Clone the repository to your local machine:
git clone https://github.com/adamkells/speed-insights.git
-
Navigate to the project directory:
cd speed-insights
-
Install the project dependencies using Poetry:
poetry install
This will create a virtual environment and install all the required dependencies specified in the
pyproject.toml
file. -
Activate the virtual environment:
poetry shell
This will activate the virtual environment and allow you to run the
speed-insights
commands. -
You're all set! You can now start using the
speed-insights
package.from speed_insights import SpeedInsights
If you encounter any issues during the installation process, please refer to the Poetry documentation or raise an issue in the speed-insights repository.
SpeedInsights is a designed to simplify and streamline the process of model evaluation. It includes a range of features that make it easy to compare the performance of different models, compute various standard metrics, and generate insightful visualizations.
With SpeedInsights, you can input your dataset, include pre-trained models, and generate metrics and insights to assess and compare model performances. At present it is specifically geared towards regression tasks but will be expanding in Q1 2024.
In the following example, we load a dataframe and train an sklearn fit/predict model as per the usual ML workflow. We then create a SpeedInsights object from X and y data as well as dictionary of my model.
import pandas as pd
from speed_insights import SpeedInsights
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
# Load the dataset
data = pd.read_csv('my_data.csv')
y = data['target']
X = data.drop('target', axis=1)
# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create an instance of the LinearRegression model
model = LinearRegression()
# Fit the model to the training data
model.fit(X_train, y_train)
# Create an instance of the SpeedInsights class
insights = SpeedInsights(X_test, y_test, {'linear-model': model})
# Generate metrics, visualizations, and insights
insights.generate_metrics()
If we want to compare a range of models we can simply extend our dictionary:
model_1 = LinearRegression()
model_2 = RandomForest()
model_1.fit(X_train, y_train)
model_2.fit(X_train, y_train)
insights = SpeedInsights(X_test, y_test, {'linear-model': model_1, 'rf-model': model_2})
insights.generate_metrics()
We can also generate visualisations of our predictions vs our ground truth with the generate_prediction_visualisations method.
insights = SpeedInsights(X_test, y_test, {'linear-model': model_1, 'rf-model': model_2})
insights.generate_prediction_visualisations('output_folder')
The following items are on the roadmap for Q1 2024.
- Expanding the list of metrics (SMAPE) as well as some pragmatic types of errors (largest error, std of error distribution).
- Adding a method to return potentially interesting rows for inspection.
- Expanding the visualisations (distribution of errors)
- Fairness investigation (groupwise inspection of metrics and errors)
Contributions are more than welcome! Feel free to raise an issue if you have any suggestions for additional visualisations and metrics.