Releases · kaiko-ai/eva

What's Changed

Remove CI badge by @roman807 in #345

Minor fix on downloading configs instructions by @ioangatop in #344

Update email in CoC by @ioangatop in #347

Print results table at the end of an evaluation session by @nkaenzig in #337

update installation instructions by @roman807 in #349

Add MultiEmbeddingsClassificationDataset by @nkaenzig in #351

Disable Loggers that don't support saving to remote storage at runtime by @nkaenzig in #352

Fix typos and inconsistent formatting by @roman807 in #353

Add gitleaks to CI workflow by @a-thiery in #357

add dinov2 to documentation by @roman807 in #355

update onnx to version 1.16.0 by @roman807 in #359

Add UNI results & replication instructions by @roman807 in #362

#366 additional Renovate settings by @b-abderrahmane in #367

Update lightning to version 2.2.2 by @roman807 in #371

Reformat image segmentation datasets by @ioangatop in #380

Pin dependencies by @renovate in #369

Update pdm-project/setup-pdm digest to 568ddd6 by @renovate in #400

Update wntrblm/nox action to v2024.04.15 by @renovate in #372

Update actions/checkout digest to 0ad4b8f by @renovate in #399

Update TotalSegmentator output binary masks to a semantic mask by @ioangatop in #390

Add support for 8bit images in TotalSegmentator by @ioangatop in #398

Add read_nifti_slice to optionally cast image to stored input type by @ioangatop in #392

Update CI triggering events by @ioangatop in #406

Move model freezing functionality to configure_model by @ioangatop in #409

Update DEVELOPER_GUIDE.md by @nkaenzig in #418

Update TotalSegmentator2D dataset to fetch all the slices by @ioangatop in #416

Move metrics to CPU when using single device by @ioangatop in #446

Remove total segmentator classification dataset by @ioangatop in #450

updated eva logo by @roman807 in #454

Update actions/checkout digest to a5ac7e5 by @renovate in #458

Add configuration logger by @ioangatop in #466

Update README with paper citation by @ioangatop in #474

fix config link in docs by @roman807 in #482

Update img shields of README by @ioangatop in #480

Fix torch and jsonargparse versions by @ioangatop in #483

Oncology FM Evaluation Framework by kaiko.ai

Installation • How To Use • Documentation • Datasets • Benchmarks
Contribute • Acknowledgements

eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai.
Check out the documentation for more information.

Highlights:

Easy and reliable benchmark of Oncology FMs
Automatic embedding inference and evaluation of a downstream task
Native support of popular medical datasets and models
Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml

When used as a CLI tool, eva supports configuration files (.yaml) as an argument to define its functionality.
Native supported configs can be found at the configs directory
of the repo. Apart from cloning the repo, you can download the latest config folder as .zip from your browser from
here. Alternatively,
from a specific release the configs can be downloaded from the terminal as follows:

curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs.zip

For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
linear probing by first inferring the embeddings and performing 5 sequential fits, execute:

# from a locally stored config file
eva predict_fit --config ./configs/vision/dino_vit/offline/bach.yaml

# from a remote stored config file
eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml

Note

All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
For automatic download you have to manually set download=true.

To view all the possibles, execute:

eva --help

For more information, please refer to the documentation
and tutorials.

Benchmarks

In this section you will find model benchmarks which were generated with eva.

Table I: WSI patch-level benchmark

Model	BACH	CRC	MHIST	PCam/val	PCam/test
ViT-S/16 (random) ^[1]	0.410	0.617	0.501	0.753	0.728
ViT-S/16 (ImageNet) ^[1]	0.695	0.935	0.831	0.864	0.849
ViT-B/8 (ImageNet) ^[1]	0.710	0.939	0.814	0.870	0.856
DINO_(p=16) ^[2]	0.801	0.934	0.768	0.889	0.895
Phikon ^[3]	0.725	0.935	0.777	0.912	0.915
ViT-S/16 (kaiko.ai) ^[4]	0.797	0.943	0.828	0.903	0.893
ViT-S/8 (kaiko.ai) ^[4]	0.834	0.946	0.832	0.897	0.887
ViT-B/16 (kaiko.ai) ^[4]	0.810	0.960	0.826	0.900	0.898
ViT-B/8 (kaiko.ai) ^[4]	0.865	0.956	0.809	0.913	0.921
ViT-L/14 (kaiko.ai) ^[4]	0.870	0.930	0.809	0.908	0.898

Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report averaged balanced accuracy
over 5 runs, with an average standard deviation of ±0.003.

References:

"Emerging properties in self-supervised vision transformers”
"Benchmarking self-supervised learning on diverse pathology datasets”
"Scaling self-supervised learning for histopathology with masked image modeling”
"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer
and contributing guide for help on how to do so.

All contributors must follow the code of conduct.