CM interface to reproduce experiments from research projects

About

This CM repository contains a unified Collective Mind interface (MLCommons CM) to access, run and reproduce experimetns from research projects and papers (ACM, IEEE, NeurIPS, ICML ...) using artifact evaluation methodology and automation from MLCommons, ACM/cTuning and NeurIPS.

Why?

While working with the community to reproduce or replicate 150+ research papers during artifact evaluation, we have seen that reviewers spend most of their time at the kick-the-tires phase deciphering various READMEs and scripts to figure out how to prepare and run experiments.

This experience motivated us to develop a simple, technology-agnostic and human-friendly interface and automation language (MLCommons Collective Mind) to provide the same common interface to prepare, run and visualize experiments from any paper or research project.

The goal is to make it easier for the community and evaluators to start reproducing/replicating research results and even fully automate this process in the future.

Note that it's a pilot project - we are very interested in your feedback to improve this interface and simplify and automate artifact evaluation. Feel free to contact us via this Discord server.

License

MIT

How to use

Install CM automation language as described here.

Install common research automations from MLCommons via CM

cm pull repo mlcommons@ck

Install this repository

cm pull repo ctuning@cm-reproduce-research-projects

List available interfaces for research projects

cm find script --tags=reproduce,project

Set up virtual environment before running experiments

If you use python, we suggest you to set up Python virtual environment via CM as follows:

cm run script "install python-venv" --name=reproducibility
export CM_SCRIPT_EXTRA_CMD="--adr.python.name=reproducibility"

It will be automatically activated for all further CM commands while keeping your Python installation intact.

Run experiments

Each CM script wrapper should have a CM script with the following variations: install_deps, run, analyze, plot, validate, reproduce

It's possible to run them as follows:

cm find script --tags=reproduce,project
cm run script {name from above list} --tags=_{variation}

Check/clean CM cache

Many CM commands download or produce new artifacts that are stored in CM cache (including CM Python virtual environments) to avoid polluting user system.

You can view current CM cache as follows:

cm show cache
cm show cache --tags=python

You can clean CM cache and start from scratch at any time using the following command:

cm rm cache -f

Add CM interface for new projects and papers

You can just copy any CM script and update alias in _cm.yaml. You can then extend it based on the following examples:

• CM script to reproduce MICRO paper to reproduce results from a MICRO paper.
• CM script to reproduce IPOL journal paper
  • CM tutorial to reproduce an IPOL journal paper.
• _cm.yaml description to download and extract files from Zenodo/Dropbox via CM

After manually updating meta description of CM artifacts (_cm.yaml or _cm.json) you must reindex them as follows:

cm reindex repo

We are updating a tutorial to add CM interface to new projects and papers and will share it soon!

Contact us

Don't hesitate to reach the MLCommons Task Force on Automation and Reproducibility and the cTuning foundation (developers of the open-source CM automation language and CK playground) to provide your feedback, get help or brainstorm ideas using this public Discord server.