Oncology FM Evaluation Framework by kaiko.ai

Installation • How To Use • Quick Start • Documentation • Datasets • Benchmarks
Contribute • Acknowledgements

eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

• Easy and reliable benchmark of Oncology FMs
• Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
• Automatic embedding inference and evaluation of a downstream task
• Native support of popular medical datasets and models
• Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 

eva uses jsonargparse to make it easily configurable by automatically generating command line interfaces (CLIs), which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva can be used either directly from Python or as a CLI tool (recommended).

For more information, please refer to the documentation.

Learn about Configs

The following interfaces are identical:

Python interface

Configuration file

# main.py # execute with: `python main.py` from torch import nn from eva import core from eva.vision import datasets, transforms # initialize trainer trainer = core.Trainer(max_steps=100) # initialize model model = core.HeadModule( backbone=nn.Flatten(), head=nn.Linear(150528, 4), criterion=nn.CrossEntropyLoss(), ) # initialize data data = core.DataModule( datasets=core.DatasetsSchema( train=datasets.BACH( root="data/bach", split="train", download=True, transforms=transforms.ResizeAndCrop(), ), ), dataloaders=core.DataloadersSchema( train=core.DataLoader(batch_size=32), ), ) # perform fit pipeline = core.Interface() pipeline.fit(trainer, model=model, data=data)

# main.yaml # execute with: `eva fit --config main.yaml` --- trainer: class_path: eva.Trainer init_args: max_steps: 100 model: class_path: eva.HeadModule init_args: backbone: torch.nn.Flatten head: class_path: torch.nn.Linear init_args: in_features: 150528 out_features: 4 criterion: torch.nn.CrossEntropyLoss data: class_path: eva.DataModule init_args: datasets: train: class_path: eva.vision.datasets.BACH init_args: root: ./data/bach split: train download: true transforms: eva.vision.transforms.ResizeAndCrop dataloaders: train: batch_size: 32

The .yaml file defines the functionality of eva by parsing and translating its content to Python objects directly. Native supported configs can be found at the configs directory of the repo, which can be both locally stored or remote.

Quick Start

We define two types of evaluations: online and offline. While online fit uses the backbone (FM) to perform forward passes during the fitting process, offline fit first generates embeddings with the backbone and then fits the model using these embeddings as input, resulting in a faster evaluation.

Here are some examples to get you started:

• Perform a downstream offline classification evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and then performing 5 sequential fits:

  export DOWNLOAD_DATA=true
  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml

• Perform a downstream online segmentation evaluation of DINO ViT-S/16 on the MoNuSAC dataset with the ConvDecoderMS decoder:

  export DOWNLOAD_DATA=true
  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml

For more examples, take a look at the configs and tutorials.

Note

All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set the environmental variable DOWNLOAD_DATA=true or in the configuration file download=true.

Leaderboards

In this section you will find model benchmarks which were generated with eva.

Table I: WSI and microscopy image tasks

Model	BACH	CRC	MHIST	PCam	Camelyon16	PANDA	CoNSeP	MoNuSAC
ViT-S/16 (random) [1]	0.411	0.613	0.5	0.752	0.551	0.347	0.489	0.394
ViT-S/16 (ImageNet) [1]	0.675	0.936	0.827	0.861	0.751	0.676	0.54	0.512
DINO(p=16) [2]	0.77	0.936	0.751	0.905	0.869	0.737	0.625	0.549
Phikon [3]	0.715	0.942	0.766	0.925	0.879	0.784	0.68	0.554
UNI [4]	0.797	0.95	0.835	0.939	0.933	0.774	0.67	0.575
ViT-S/16 (kaiko.ai) [5]	0.8	0.949	0.831	0.902	0.897	0.77	0.622	0.573
ViT-S/8 (kaiko.ai) [5]	0.825	0.948	0.826	0.887	0.879	0.741	0.677	0.617
ViT-B/16 (kaiko.ai) [5]	0.846	0.959	0.839	0.906	0.891	0.753	0.647	0.572
ViT-B/8 (kaiko.ai) [5]	0.867	0.952	0.814	0.921	0.939	0.761	0.706	0.661
ViT-L/14 (kaiko.ai) [5]	0.862	0.935	0.822	0.907	0.941	0.769	0.686	0.599

Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report balanced accuracy for classification tasks and generalized Dice score for semgetnation tasks, averaged over 5 runs. Results are reported on the "test" split if available and otherwise on the "validation" split.

References:

1. "Emerging properties in self-supervised vision transformers”, arXiv
2. "Benchmarking self-supervised learning on diverse pathology datasets”, arXiv
3. "Scaling self-supervised learning for histopathology with masked image modeling”, medRxiv
4. "A General-Purpose Self-Supervised Model for Computational Pathology”, arXiv
5. "Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”, arXiv

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

Acknowledgements

Our codebase is built using multiple opensource contributions

Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}