Synthesis, placement, and routing turnaround times are some of the major bottlenecks in digital design productivity. Engineers usually wait several hours to get accurate results to design changes that are often quite small. Incremental synthesis has emerged as an attempt to reduce these long times, but research in incremental synthesis currently lacks a consistent benchmark to enable comparison between different flows and is reflective of real design changes. ANUBIS is the first benchmark suite intended for evaluating incremental synthesis techniques and flows. It has been presented in the 2017 International Workshop of Logic and Synthesis. This repository includes the verilog code for the benchmarks with code changes that can be activated through define statements. We also included scripts to run sample flows based that already include incremental steps.
To run the sample flow, you need to first build yosys and run the benchmarks with yosys, and then run the appropriate flow. The instructions here assume you have all the dependencies installed and vivado (or quartus) binaries in your PATH environment variable. The Makefiles are meant to be run in bash.
cd yosys
make
cd ../scripts/yosys
make complete
cd ../vivado # or quartus
make complete
The ANUBIS table for the flow will be dumped to STDOUT.
Start from an existing sample case. You need to provide a script for baseline synthesis, full synthesis of each delta, and incremental synthesis for each delta (see the appropriate makefile lines). Then you need to provide a script to collect (for full and incremental synthesis):
- Runtime for synthesis, placement, and routing
- Delay after routing
- Energy (Power times Frequency) after routing
- Area after routing
For power, it is fine to use static power analysis (ie. estimated switching activities) and no leakage power as long as the assumptions are the same for both the incremental and full synthesis flows.
The benchmarks consists of a set of designs, design changes and a standard scoring system. Changes are based on real-world changes collected through history in the repository of the designs and/or commented out code.
The ANUBIS score depends on the runtime, normalized to the YOSYS runtime, and corrected with the Quality-of-Results (QoR) degradation for delay, energy (power x frequency) and area. For each change, the corrected runtime is calculated as
tau(an) = t_{i}(a_n) + (1+\alpha) x (t_{f}(a_n))/(\alpha + e^((\beta Q_(f)(a_n)) / (Q(i)(a_n))))
Where \(\alpha ## 10^8\) and \(\beta ## 26\) are two normalization constants. Then the score for each task (synthesis, placement, routing) and for each QoR metric (delay, energy, area) is calculated as the geometric mean of the taus of all the changes and the baseline runtime.
Sample scripts to run and calculate the scores are provided with ANUBIS with
To report the scores of an incremental flow, a standard table with breakdown per task and QoR target is used, with a overall ANUBIS Value (larger is better). A sample table is show below.
| Task | Delay | Energy | Area | \gmean | Full |
|---|---|---|---|---|---|
| Synth | \gmean(s) | full(s) | |||
| Place | \gmean(p) | full(p) | |||
| Route | \gmean(r) | full(r) | |||
| \gmean | \gmean(d) | \gmean(e) | \gmean(a) | \gmean(\gmean) | \gmean(full) |
If you have a new record and want your results to be displayed here, send your results, with a binary (or source code) and if we can reproduce your results, we will include your score in this page with a link to your work.
For more details, check the ANUBIS paper.
If you use ANUBIS to evaluate your flow, please use the following citation:
@inproceedings{anubis:iwls2017,
author = {Possignolo, Rafael T. and Kabylkas, Nursultan and Renau, Jose},
title = {{Anubis}: A New Benchmark for Incremental Synthesis},
booktitle = {Logic Synthesis (IWLS), Proceedings of the 2017 International Workshop on},
year = {2017},
month = {Jun},
}All authors are affiliated to the MASC lab at UCSC.
ANUBIS is distributed under BSD License 2.0, individual benchmarks however may hold a different license. ANUBIS include the following third part benchmarks:
- YOSYS:
- Distributed under ISC License
- ABC:
- Distributed under University of California License
- MOR1KX :
- Distributed under Open Hardware Description License Version 1.0 (Based on the MPL 2.0 RC2)
- OR1200:
- Distributed under GNU Lesser General Public License version 2.1
- ALPHA / DLX:
- Distributed under the Bug UnderGround UMich project. Authors do not warrant or assume any legal responsibility for the accuracy, completeness, or usefulness of this software, nor for any damage derived by its use.
A copy of the copyright notes is found within the directory of each project.
Copyright (c) 2017 Regents of the University of California All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the University of California nor the
names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL University of California or its affiliates BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.