Generalization After Transfer Evaluation (GATE)

GATE is a comprehensive benchmarking suite that aims to fill a gap in the evaluation of foundation models. Typically, foundation model evaluation overlooks the diverse applicability found in real-world settings. GATE is designed for multi-modal and multi-task scenarios, stress-testing models in a variety of domains beyond their initial training.

Features

• Facilitates the transfer of neural network trunks across different modalities, domains, and tasks, promoting robust fine-tuning for visual tasks.
• Maximizes research signal per GPU hour through carefully selected scenarios.
• Enables straightforward replacement of transferable trunks with minimal effort.

Installation

git clone https://github.com/AntreasAntoniou/GATE.git
cd GATE
pip install -r requirements.txt
pip install -e . # for local dev editable mode
or 
pip install .

For development purposes, use the requirements_dev.txt file:

pip install -r requirements_dev.txt

Once you install gate, you should configure it using

gate config

Follow the onscreen instructions to acquire all necessary API keys and set them up. 😸

Usage

1. Run the GATE tiers on the models presented in the original paper

python gate/tiers/builder.py --experiment_type=all | python gate/tiers/run_experiments.py 1;

Which will build the full GATE tier and run it on one GPU. Small, Base And Big GATE will be added very shortly as an explicit option.

2. Run single experiment using the gate/run.py entry point

accelerate launch gate/run.py exp_name=beta-coco-10k-BART-123 encoder=bart encoder.image_size=224 encoder.pretrained=True encoder.clip_model_name=openai/clip-vit-base-patch16 encoder.bart_model_name=facebook/bart-base encoder.num_projection_features=768 adapter=segmentation-adapter adapter.loss_type_id=default dataset=coco_10k optimizer.lr=6e-06 optimizer.weight_decay=0.01 trainer=image_semantic_segmentation evaluator=image_semantic_segmentation num_workers=24 seed=123 train_batch_size=8 eval_batch_size=8 train_iters=10000 learner.evaluate_every_n_steps=500#### 1. Run the GATE tiers on the models presented in the original paper

2. Run the GATE tiers on custom encoder models

Section under construction.

3. Use GATE as a template for your research project

GATE can be used as a template for your research project. It provides full Hydra integration and includes boilerplate, models, datasets, trackers, and more. Note that using GATE as a template may involve a lot of overhead and time to learn, and may be complex.

4. Use GATE as a library

GATE can be used as a library in your Python projects. Here is a basic example:

## Importing the Dataset

# To import and use the Happywhale dataset, you can use the `build_dataset` and `build_gate_dataset` functions from the GATE library. Here's how to do it:

import os
import torch
import torchvision.transforms as T
from gate.data.image.classification.happywhale import build_dataset, build_gate_dataset

# Set the data directory
data_dir = "path/to/your/data"

# Build the main dataset
main_dataset = build_dataset(data_dir=data_dir)

# Check if the train set is available
assert main_dataset["train"] is not None, "Train set should be available"

# Define a transform function
def default_transforms(input_dict):
    input_dict["image"] = T.ToTensor()(input_dict["image"])
    return input_dict

# Build the GATE dataset with transforms
gate_dataset = build_gate_dataset(
    data_dir=data_dir,
    transforms=default_transforms,
)

# Create a DataLoader for the training set
gate_dataloader = torch.utils.data.DataLoader(
    gate_dataset["train"], batch_size=64, shuffle=True, num_workers=24
)

# Verify the dataset splits
assert gate_dataset["train"] is not None, "Train set should be available"
assert gate_dataset["val"] is not None, "Validation set should be available"
assert gate_dataset["test"] is not None, "Test set should be available"

# Iterate through the dataset
for item in gate_dataloader:
    # Access the data
    images = item["image"]
    individual_labels = item["labels"]["individual"]
    species_labels = item["labels"]["species"]
    
    # Your processing code here
    ...

    break  # Remove this line to process all batches

GATE Models Usage

This section demonstrates how to use various models provided by the GATE library for different modalities and tasks.

Available Models

GATE provides a variety of model adapters for different modalities:

• Image: TimmCLIPAdapter, CLIPVisionAdapter
• Text: CLIPTextAdapter, BertAdapter, MPNetAdapter, BartAdapter
• Audio: WhisperAdapter, Wav2VecV2Adapter

Example Usage

Here's an example of how to use these models with a Prototypical Network for few-shot classification:

import torch
from gate.models.backbones.timm import CLIPModelPaths, TimmCLIPAdapter
from gate.models.task_adapters.few_shot_classification.protonet import PrototypicalNetwork
from gate.models.core import GATEModel

# Initialize the encoder
encoder = TimmCLIPAdapter(
    timm_model_name="tf_efficientnetv2_s_in21ft1k",
    clip_model_name=CLIPModelPaths.openai_b_16,
    num_projection_features=64
)

# Create the Prototypical Network model
model = PrototypicalNetwork(encoder=encoder, num_output_features=512)

# Wrap the model with GATEModel
gate_model = GATEModel(config=model.modality_config, model=model)

# Prepare input data
support_set_inputs = torch.rand(2, 2, 3, 224, 224)
query_set_inputs = torch.rand(2, 2, 3, 224, 224)
support_set_labels = torch.randint(0, 2, (2, 2))
query_set_labels = torch.randint(0, 2, (2, 2))

# Create input dictionary
input_dict = {
    "image": {
        "support_set": support_set_inputs,
        "query_set": query_set_inputs,
    },
    "labels": {
        "support_set": support_set_labels,
        "query_set": query_set_labels,
    },
}

# Apply transforms
input_dict = model.adapter_transforms(input_dict)

# Forward pass
output = gate_model.forward(input_dict)

# Access results
logits = output["image"]["image"]["logits"]
loss = output["image"]["image"]["loss"]

# Backward pass
loss.backward()

Use GATE as a library, as a source of experiment generation

GATE can be used as a library to generate experiments. Here is an example:

builder = gate.build_experiments(model=GATEModel(), gate_flavour="base")
experiments = builder.generate_experiments()
builder.run_experiments()

Project Structure

A high-level overview of the project's structure is given below:

.
├── boilerplate/
│   ├── callbacks.py
│   ├── convenience.py
│   ├── core.py
│   ├── decorators.py
│   ├── utils.py
│   └── wandb_utils.py
├── config/
│   ├── config.py
│   └── variables.py
├── data/
│   ├── few_shot/
│   ├── image/
│   ├── image_text/
│   ├── medical/
│   ├── tasks/
│   ├── transforms/
│   ├── video/
│   └── core.py
├── menu/
│   ├── configs/
│   ├── builder.py
│   ├── collector.py
│   ├── core.py
│   └── utils.py
├── metrics/
│   ├── core.py
│   ├── glossary.py
│   ├── multi_class_classification.py
│   ├── segmentation.py
│   └── vqa_eval.py
├── models/
│   ├── backbones/
│   ├── blocks/
│   ├── task_adapters/
│   └── core.py
├── orchestration/
│   ├── evaluators/
│   ├── trainers/
│   └── utils/
├── dummy_module.py
└── run.py

For a more detailed description of the individual files and directories, please refer to the comments in the respective files.