Setting Up an Experiment

Each time you run an experiment on DaCapo, you have to tell it where to store the output, where to find your data (DataSplit), how to configure the neural network structure (Architecture), what you want the network to learn (Task), and how you want the network to learn (Training). Finally, you have to configure the training run.

Data Storage

What data are stored

DaCapo has the following data storage components:

• Loss stats: The loss per training run iteration is stored along with a couple other statistics such as how long that iteration took to compute. These will be stored in the MongoDB if available.
• Validation scores: For each validation step. These will be stored in the MongoDB if available.
• Validation volumes: The results of validation (images with presumptive organelles labeled) are stored in zarr datasets so you visually inspect the best predictions on your validation out data according to the validation metric of your choice. This data will be stored on disk.
• Checkpoints: Copies of your model are stored at various intervals during training. This lets you retrieve the best performing model according to the validation metric of your choice. This data will be stored on disk.
• Training Snapshots: Every n iterations (where n corresponds to the snapshot_interval defined in the Training configuration) a snapshot that includes the inputs and outputs of the model at that iteration is stored along with some extra results that can be very helpful for debugging. The saved arrays include: Ground Truth, Target (Ground Truth transformed by Task), Raw, Prediction, Gradient, and Weights (for modifying the loss). This data will be stored on disk.
• Configurations: To make runs easily reproducible, the configuration files used to execute experiments are saved. This way other people can use the exact same configuration files or change single parameters and get comparable results. This data will be stored in the MongoDB if available.

Defining Storage Locations

To define where this data goes, use a text editor to create a configuration file called dacapo.yaml. Here is a template for the content of the file:

mongodbhost: mongodb://dbuser:dbpass@dburl:dbport/ mongodbname: dacapo runs_base_dir: /path/to/my/data/storage

The **runs_base_dir ** command defines where your on-disk data will be stored. The mongodbhost and mongodbname commands define the mongodb host and database that will store your cloud data. If you want to store everything on disk, replace mongodbhost and mongodbname with:

type: files runs_base_dir: /path/to/my/data/storage

Experiment Configuration

Configuration files for your experiments are created in python, using a Jupiter notebook. The first thing you need to do is launch the dacapo and logging platforms by typing in the following commands:

import dacapo import logging

????

You also need to retrieve your decapo.yaml by typing:

from dacapo.store.create_store import create_config_store

config_store = create_config_store()

Now you can create your configurations.

DataSplit

The DataSplit configuration file indicates where your training, validation, and testing data are stored, what forma it is in, and whether it needs to be normalized.

????

Instructions for doing this are coming soon!

Architecture

The neural network architecture used is usually one called a UNet. You will need to specify parameters such as how much you want to downsample your image and h how many layers you want in the neural network.

To configure the architecture, use this template code:

from dacapo.experiments.architectures import CNNectomeUNetConfig

architecture_config = CNNectomeUNetConfig(
    name="example_attention-upsample-unet",
    input_shape=(216, 216, 216),
    fmaps_out=72,
    fmaps_in=1,
    num_fmaps=12,
    fmap_inc_factor=6,
    downsample_factors=[(2, 2, 2), (3, 3, 3), (3, 3, 3)],
    kernel_size_down=None,
    kernel_size_up=None,
    eval_shape_increase=(72, 72, 72),
    upsample_factors=[(2, 2, 2)],
    constant_upsample=True,
    padding="valid",
    use_attention=True,
)
config_store.store_architecture_config(architecture_config)

Task

The Task configuration defines what and how you want the network to learn.

What types of tasks are available to use?

DaCapo can be used to train a network model to perform either semantic segmentation (e.g., Is this a mitochondrion?) or instance segmentation (e.g., Which mitochondrion is this?). The following approaches are available to performing these tasks:

**Semantic segmentation approaches

• Signed distances---Predicts how far inside or outside of an object type a voxel is
  
• One-hot encoding of different types of objects---each output unit predicts a single, unique object type

**Instance segmentation approaches

• Affinities---Predicts whether voxels are part off the same structure as neighboring voxels
  
• Local Shape Descriptors---Predicts organelle boundaries using size, offset to center of mass, and directionality

To configure the task, use the following template code:

from dacapo.experiments.tasks import DistanceTaskConfig

task_config = DistanceTaskConfig(
    name="example_distances_8nm_mito_proxisome_many",
    channels=["mito", "peroxisome"],
    clip_distance=80.0,
    tol_distance=80.0,
    scale_factor=160.0,
    mask_distances=True,
    clipmin=0.05,
    clipmax=0.95,
)
config_store.store_task_config(task_config)

Trainer

The Trainer configuration defines the training loop, what sort of Augmentations to apply during training, what learning rate and optimizer to use, and what batch size to train with.

To configure the trainer, use the following template code:

from dacapo.experiments.trainers import GunpowderTrainerConfig
from dacapo.experiments.trainers.gp_augments import (
    ElasticAugmentConfig,
    GammaAugmentConfig,
    IntensityAugmentConfig,
    IntensityScaleShiftAugmentConfig,
)

trainer_config = GunpowderTrainerConfig(
    name="default",
    batch_size=2,
    learning_rate=0.0001,
    num_data_fetchers=20,
    augments=[
        ElasticAugmentConfig(
            control_point_spacing=[100, 100, 100],
            control_point_displacement_sigma=[10.0, 10.0, 10.0],
            rotation_interval=(0.0, 1.5707963267948966),
            subsample=8,
            uniform_3d_rotation=True,
        ),
        IntensityAugmentConfig(scale=(0.25, 1.75), shift=(-0.5, 0.35), clip=True),
        GammaAugmentConfig(gamma_range=(0.5, 2.0)),
        IntensityScaleShiftAugmentConfig(scale=2.0, shift=-1.0),
    ],
    snapshot_interval=10000,
    min_masked=0.05,
    clip_raw=True,
)
config_store.store_trainer_config(trainer_config)

Run Configuration

The last step before executing a run is to configure the run. You need to specify

• the starting point of the network (from scratch or from a partially trained network (start_config)
• How many times the data should be passed through the network (iterations)
• How often to run a validation step
• How many times to repeat the run
• Which DataSplit, Architecture, Task, and Trainer configuration files to use

from dacapo.experiments.starts import StartConfig
from dacapo.experiments import RunConfig
from dacapo.experiments.run import Run

# start_config = None

# Uncomment to start from a pretrained model
start_config = StartConfig(
    "setup04",
    "best",
)

iterations = 200000
validation_interval = 5
repetitions = 3
for i in range(repetitions):
    run_config = RunConfig(
        name=("_").join(
            [
                "example",
                "scratch" if start_config is None else "finetuned",
                datasplit_config.name,
                task_config.name,
                architecture_config.name,
                trainer_config.name,
            ]
        )
        + f"__{i}",
        datasplit_config=datasplit_config,
        task_config=task_config,
        architecture_config=architecture_config,
        trainer_config=trainer_config,
        num_iterations=iterations,
        validation_interval=validation_interval,
        repetition=i,
        start_config=start_config,
    )

    print(run_config.name)
    config_store.store_run_config(run_config)