Offline Evaluation

In this page, we will show you how to reproduce our off-line evaluation results in our CoNEXT 2012 paper (to appear). You only need a reasonably modern computer that can run Python 2.7 (of course, you may also need git to download the source file).

Get the source

If you're reading this page, you may already have the source. If you are reading this page, you may already know how to do that.

git clone https://github.com/eastzone/atpg.git
cd atpg/

All the directories below uses atpg/ (or the directory you clone into) as the reference point.

Generate transfer functions

A transfer function models the behavior of a network device. Before generating test packets, we need to transform vendor-specific (Cisco/Juniper) configuration files into vendor-independent transfer functions.

Two data sets are built in our codebase in data/: Stanford and Internet2. You may run the following commands to generate these two sets of transfer functions.

cd atpg/utils
python generate_stanford_backbone_tf.py
python generate_internet2_backbone_tf.py

The outputs are in work/. Now we have all the transfer functions.

Count the packets and coverage

In the paper, we show offline evaluation results in Table 5. This table can be reproduce by running a simple bash script in scripts/

cd scripts
./stanford.sh
./internet2.sh

The resulting outputs are stored in work/.

Rule clustering

In Figure 8, we show the internal structure of two data sets, and provide a potential explanation of why so few packets are sufficient. The figure can be reproduced by running the following command

cd atpg/
./rule_clustering.py