FeTS Challenge Task 2 Workflows

Overview

Evaluating hospitals create MedPerf accounts.
Evaluating hospitals are the data owners.
DKFZ has an account and is a benchmark owner and model owner
MLCommons are admins (Alejandro, Johnu, Micah)
FeTS challenge has it's own dedicated MedPerf server (www.medperf.org/fets or similar)
DKFZ registers the reference benchmark pipeline and test dataset
Evaluating hospitals register their datasets
Evaluating hospitals run the test pipeline against their data to test compatibility/compute resources
Competitors implement MLCubes according to template provided
Competitors submit their MLCubes to DKFZ via Synapse
DKFZ evaluates submissions for compliance, hosts and registers them
MedPerf team provides script to evaluating hospitals to download and verify submissions
MedPerf team provides script to evaluating hospitals to run each model in the benchmark (results are not uploaded at this point)
Evaluating sites (optionally) review and upload all results
DKFZ can view the results in MedPerf for scoring and analysis

Work with MLCube and MedPerf teams to create competitor instructions and materials
Create DKFZ medperf account
Collect MedPerf account names from evaluating hospitals and pass list to MedPerf admins
Create FeTS Challenge Benchmark (expand this)
- Host benchmark mlcubes
Collect submissions
Vet submissions
- For non-compliant submissions, contact competitors as able
Host submissions
Register submissions in MedPerf
Associate submissions to the benchmark
Send list of submission URLs and hashes to MedPerf team
Wait for results
Analyze results in MedPerf to determine challenge outcome

Gather their QC'd/labeled data onto one system, in file structure defined by DKFZ
Download FeTS Challenge package (includes MedPerf, extra scripts, scripts can download benchmark cubes)
Provide DKFZ desired MedPerf user account name
Receive account name and password from MedPerf admins (via DKFZ??)
Test MedPerf login
Change user password (password change API required)
Use script to download and verify reference mlcubes
Test Benchmark reference pipeline
- check for correct use of GPU
Run data preparation step and register with medperf
Request benchmark association (this exercises full pipeline on their data)
Use script to download and verify all competitor MLCubes
Use script to evaluate all competitor MLCubes. Results are not uploaded in this step.
Sanity check results to ensure no leaked information
Use script to upload results to MedPerf