Linux/MacOS: 
-
We have moved this code back to the CK framework and will continue developments there!
-
Join us at the upcoming ACM TechTalk (Feb.11).
-
We have successfully completed the prototyping phase of the Collective Knowledge technology to make it easier to reproduce AI&ML and deploy it in production with the help of portable CK workflows, reusable artifacts and MLOps as described in this white paper and the CK presentation. We are now preparing the second phase of this project to make CK simpler to use, more stable and more user friendly - don't hesitate to get in touch with the CK author to know more!
cBench is a small and cross-platform framework connected with the open Collective Knowledge portal to help researchers and practitioners reproduce ML&systems research on their own bare-metal platforms, participate in collaborative benchmarking and optimization, and share results on live scoreobards.
You can try to reproduce MLPerf inference benchmark on your machine using this solution and see public results from the community on this scoreboard.
cBench is a part of the Collective Knowledge project (CK) and uses portable CK solutions to describe how to download, build, benchmark and optimize applications across different hardware, software, models and data sets.
| As a host platform | As a target platform | |
|---|---|---|
| Generic Linux | ✓ | ✓ |
| Linux (Arm) | ✓ | ✓ |
| Raspberry Pi | ✓ | ✓ |
| MacOS | ✓ | ± |
| Windows | ✓ | ✓ |
| Android | ± | ✓ |
| iOS | TBD | TBD |
Install prerequisites:
sudo apt update
sudo apt install git wget libz-dev curl cmake
sudo apt install gcc g++ autoconf autogen libtool
sudo apt install libfreetype6-dev
sudo apt install python3.7-dev
sudo apt install -y libsm6 libxext6 libxrender-dev
Install cbench:
python3 -m pip install cbench
Initialize the CK solution for MLPerf:
cb init demo-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows
Participate in crowd-benchmarking:
cb benchmark demo-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows
See your results on a public SOTA dashboard.
You can also use the stable Docker image to participate in crowd-benchmarking:
sudo docker run ctuning/cbench-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows /bin/bash -c "cb benchmark demo-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows"
You can also check all dependencies for this solution.
- This is an ongoing project - don't hesitate to contact us if you have any feedback and suggestions!
We would like to thank all CK partners for fruitful discussions and feedback!
Copyright 2020 cTuning foundation