Templated verilator unit test generator

[jbush001]

Currently the Verilator unit tests (in tests/unit) are unwieldy to maintain and extend. Because Verilator doesn't support behavioral delays, each test uses a cycle counter and a huge case statement. This makes it hard to rearrange tests, because it requires renumbering the cases. The tests are also interdependent because it is difficult to add a common test initialization routine. Ideally, some kind of macro processor, DSL, or template system could extract fragments of Verilog code and put it into the appropriate structure:

1. Automatically create the case statements. The test case could contain a statement like 'clock' that would indicate where the division points should be.
2. When a single file contains multiple test cases, the templating system could automatically create a separate test module for each one and insert common initialization code before each one to start in a known state.

A few ideas:

1. Use an existing macro processor like m4 to generate the test cases.
2. Implement the system as python code in the existing runtest.py.

[jbush001]

The system would ideally allow specifying the following fragments of code:

• One that instantiates the module under test (MUT), and declares all of the signals attached to it.
• Initialization code that runs before each cycle to set MUT inputs to known values.
• Code that is executed before each test case to initialize the MUT.
• Individual test cases.

[jbush001]

Another thing that would be nice would be to emulate blocking assignments. Assume we wanted to test an adder:

module adder(input clk, input[31:0] a, input[31:0] b, output logic[31:0] sum)
    always_ff @(posedge clk)
        sum <= a + b;
endmodule

With a verilog simulator that supports delays, the testbench could be structured like this:

initial
begin
    a = 1;
    b = 2;
    clk = 0;
    #5 clk = 1;
    #5 assert(sum == 3);
end

But in Verilator, assignments in this loop should be non-blocking (because it's always_ff). This means we'd need to do this:

    a <= 1;
    b <= 2;
    clock();
    clock();
    assert(sum == 3);

The input signals are asserted after the first clock, and the module latches the result on the second. With more sophisticated use cases, this can become harder to reason about and error prone. It would be nice if the input to this system would appear like a non-blocking assignment, but the system would automatically create a non-blocking assignment and emit it in the previous cycle, so this:

    a = 1;
    b = 2;
    clock();
    assert(sum == 3);

Would be converted to this:

    case (clock)
    0:
    begin
        a <= 1;
        b <= 2;
    end
    
    // skip 1

    2:
    begin
        assert(sum == 3);
    end

    endcase

It's not sufficient to make the clock() macro always skip a cycle, as this would preclude testing multi-stage pipelined or asynchronous modules.

[jbush001]

Another alternative would be to build only the MUT with Verilator and interface with the signals from the c++ driver. This would be much simpler, because it wouldn't need to have case statement loop: it could drive the clock directly when it wanted to like a behavioral simulation. The c++ shim could either be:

• Generated from some higher level script/DSL.
• Have a native language interface from a scripting language (e.g. ctypes or SWIG)

Chisel uses a similar mechanism for building unit tests. It creates an executable that has a command interface over stdio that can set and read signals. A script talks to this program to run tests.

[jbush001]

This project generates testbenches automatically:
https://github.com/xfguo/tbgen/blob/master/tbgen.py

[jbush001]

Also see cocotb: https://cocotb.readthedocs.io/en/latest/introduction.html

Unfortunately, this does not appear to support Verilator.

[jbush001]

SystemPerl exposes the parser from verilator. This could be used to generate the testbench: https://www.veripool.org/wiki/systemperl. However, that would add another dependency to the project.

[olofk]

Also see cocotb: https://cocotb.readthedocs.io/en/latest/introduction.html

Unfortunately, this does not appear to support Verilator.

It does now :)

[jbush001]

Oh, cool! I'll have to check that out.