Before following this guide, make sure you've followed the dependency installation instructions.
Verilator is a cycle-accurate simulation tool. It translates synthesizable Verilog code into a simulation program in C++, which is then compiled and executed.
Even though Verilator is packaged for most Linux distributions, these versions tend to be too old to be usable. We recommend compiling Verilator from source, as outlined here.
Fetch, build and install Verilator itself (this should be done outside the $REPO_TOP directory).
Note that Verilator 4.210 will not build with GCC 12.0 or later, so it will need to be built with an older toolchain.
The example below assumes gcc-11 and g++-11 are installed on the system.
export VERILATOR_VERSION={{#tool-version verilator }}
git clone https://github.com/verilator/verilator.git
cd verilator
git checkout v$VERILATOR_VERSION
autoconf
CC=gcc-11 CXX=g++-11 ./configure --prefix=/tools/verilator/$VERILATOR_VERSION
CC=gcc-11 CXX=g++-11 make
sudo CC=gcc-11 CXX=g++-11 make installThe make step can take several minutes.
After installation you need to add /tools/verilator/$VERILATOR_VERSION/bin to your PATH environment variable.
Also add it to your ~/.bashrc or equivalent so that it's on the PATH in the future, like this:
export PATH=/tools/verilator/$VERILATOR_VERSION/bin:$PATHCheck your installation by running:
$ verilator --version
Verilator 4.210 2021-07-07 rev v4.210 (mod)If you need to install to a different location than /tools/verilator/..., you can pass a different directory to ./configure --prefix above and add your/install/location/bin to PATH instead.
First the RTL must be built into a simulator binary. This is done by running fusesoc, which collects up RTL code and passes it to Verilator to generate and then compile a C++ model. Next software must be built to run on the simulated hardware. There are 4 memory types on OpenTitan hardware: ROM, Flash, OTP, and SRAM. Software images need to be provided for ROM, Flash, and OTP (SRAM is populated at runtime). By default, the system will first execute out of ROM and then jump to Flash. The OTP image does not contain executable code, rather it contains root secrets, runtime configuration data, and life cycle state. (By default, the life cycle state is set to RMA, which enables debugging features such as the JTAG interface for the main processor.) Lastly, the Verilator simulation binary must be run with the correct arguments.
Thankfully, Bazel (and opentitantool) simplify this process by providing a single interface for performing all of the above steps.
Moreover, Bazel automatically connects to the simulated UART (via opentitantool) to print the test output in real time.
For example, to run the UART smoke test on Verilator simulated hardware, and see the output in real time, use
cd $REPO_TOP
bazel test --test_output=streamed //sw/device/tests:uart_smoketest_sim_verilatoror
cd $REPO_TOP
bazel test --test_tag_filters=verilator --test_output=streamed //sw/device/tests:uart_smoketestYou should expect to see something like:
Invoking: sw/host/opentitantool/opentitantool --rcfile= --logging=info --interface=verilator --verilator-bin=hw/build.verilator_real/sim-verilator/Vchip_sim_tb --verilator-rom=sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem --verilator-flash=sw/device/tests/uart_smoketest_prog_sim_verilator.64.scr.vmem --verilator-otp=hw/ip/otp_ctrl/data/img_rma.vmem console --exit-failure=(FAIL|FAULT).*\n --exit-success=PASS.*\n --timeout=3600s
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::subprocess] Spawning verilator: "hw/build.verilator_real/sim-verilator/Vchip_sim_tb" ["--meminit=rom,sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem", "--meminit=flash,sw/device/tests/uart_smoketest_prog_sim_verilator.64.scr.vmem", "--meminit=otp,hw/ip/otp_ctrl/data/img_rma.vmem"]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] Simulation of OpenTitan Earl Grey
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] =================================
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] Tracing can be toggled by sending SIGUSR1 to this process:
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ kill -USR1 3422749
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] GPIO: FIFO pipes created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read (read) and $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write (write) for 32-bit wide GPIO.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] GPIO: To measure the values of the pins as driven by the device, run
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ cat $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read # '0' low, '1' high, 'X' floating
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] GPIO: To drive the pins, run a command like
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ echo 'h09 l31' > $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write # Pull the pin 9 high, and pin 31 low.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] SPI: Created /dev/pts/9 for spi0. Connect to it with any terminal program, e.g.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ screen /dev/pts/9
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] NOTE: a SPI transaction is run for every 4 characters entered.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] SPI: Monitor output file created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/spi0.log. Works well with tail:
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ tail -f $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/spi0.log
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] UART: Created /dev/pts/10 for uart0. Connect to it with any terminal program, e.g.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ screen /dev/pts/10
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] UART: Additionally writing all UART output to 'uart0.log'.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] USB: Monitor output file created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/usb0.log. Works well with tail:
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] $ tail -f $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/usb0.log
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] JTAG: Virtual JTAG interface dmi0 is listening on port 44853. Use
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] OpenOCD and the following configuration to connect:
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] interface remote_bitbang
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] remote_bitbang_host localhost
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] remote_bitbang_port 44853
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout] Simulation running, end by pressing CTRL-c.
[2022-06-09T08:08:16Z INFO opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:17Z INFO opentitanlib::transport::verilator::transport] Verilator started with the following interaces:
[2022-06-09T08:08:17Z INFO opentitanlib::transport::verilator::transport] gpio_read = $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read
[2022-06-09T08:08:17Z INFO opentitanlib::transport::verilator::transport] gpio_write = $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write
[2022-06-09T08:08:17Z INFO opentitanlib::transport::verilator::transport] uart = /dev/pts/10
[2022-06-09T08:08:17Z INFO opentitanlib::transport::verilator::transport] spi = /dev/pts/9
Starting interactive console
[CTRL+C] to exit.
I00000 test_rom.c:81] Version: earlgrey_silver_release_v5-5775-gefa09d3b8
Build Date: 2022-06-09, 00:12:35
I00001 test_rom.c:118] Test ROM complete, jumping to flash!
I00000 status.c:28] PASS!
Exiting interactive console.
[2022-06-09T08:09:38Z INFO opentitantool::command::console] ExitSuccess("PASS!\r\n")For most use cases, interacting with the UART is all you will need and you can stop here. However, if you want to interact with the simulation in additional ways, there are more options listed below.
All executed instructions in the loaded software into Verilator simulations are logged to the file trace_core_00000000.log.
By default this file is stored in a subdirectory of ~/.cache/bazel.
You can find it using the following command:
find ~/.cache/bazel -name "trace_core_00000000.log"The columns in this file are tab separated; change the tab width in your editor if the columns don't appear clearly, or open the file in a spreadsheet application.
The simulation includes a DPI module to map general-purpose I/O (GPIO) pins to two POSIX FIFO files: one for input, and one for output.
Observe the gpio0-read file for outputs (in the same directory as the trace):
cat gpio0-readTo drive input pins write to the gpio0-write file.
A command consists of the desired state: h for high, and l for low, and the decimal pin number.
Multiple commands can be issued by separating them with a single space.
echo 'h09 l31' > gpio0-write # Pull the pin 9 high, and pin 31 low.The simulation includes a "virtual JTAG" port to which OpenOCD can connect using its remote_bitbang driver.
All necessary configuration files are included in this repository.
For more guidance on using OpenOCD, see Using OpenOCD.
Run the simulation with Bazel, making sure to build the device software with debug symbols using
cd $REPO_TOP
bazel test --copt=-g --test_output=streamed //sw/device/tests:uart_smoketest_sim_verilatorThen, connect with OpenOCD using the following command.
cd $REPO_TOP
/tools/openocd/bin/openocd -s util/openocd -f board/lowrisc-earlgrey-verilator.cfgLastly, connect GDB using the following command (noting it needs to be altered to point to the sw binary in use).
cd $REPO_TOP
riscv32-unknown-elf-gdb -ex "target extended-remote :3333" -ex "info reg" \
"$(./bazelisk.sh outquery --config=riscv32 //sw/device/tests:uart_smoketest_prog_sim_verilator.elf)"The simulation contains code to monitor the SPI bus and provide a host interface to allow interaction with the spi_device.
When starting the simulation you should see a message like
SPI: Created /dev/pts/4 for spi0. Connect to it with any terminal program, e.g.
$ screen /dev/pts/4
NOTE: a SPI transaction is run for every 4 characters entered.
SPI: Monitor output file created at /auto/homes/mdh10/github/opentitan/spi0.log. Works well with tail:
$ tail -f /auto/homes/mdh10/github/opentitan/spi0.logUse any terminal program, e.g. screen or microcom to connect to the simulation.
screen /dev/pts/4Microcom seems less likely to send unexpected control codes when starting:
microcom -p /dev/pts/4The terminal will accept (but not echo) characters.
After 4 characters are received a 4-byte SPI packet is sent containing the characters.
The four characters received from the SPI transaction are echoed to the terminal.
The hello_world code will print out the bytes received from the SPI port (substituting _ for non-printable characters).
The hello_world code initially sets the SPI transmitter to return SPI! (so that should echo after the four characters are typed) and when bytes are received it will invert their bottom bit and set them for transmission in the next transfer (thus the Nth set of four characters typed should have an echo of the N-1th set with bottom bit inverted).
The SPI monitor output is written to a file.
It may be monitored with tail -f which conveniently notices when the file is truncated on a new run, so does not need restarting between simulations.
The output consists of a textual "waveform" representing the SPI signals.
With the --trace argument the simulation generates a FST signal trace which can be viewed with Gtkwave (only).
An argument may be provided to --trace to specify where the trace file will be saved.
Note that for automated tests, opentitantool interfaces with the simulator, and verilator arguments must be passed via --verilator-args=<arg>.
In addition, it may be necessary to adjust the test timeout values when using bazel test with waveforms.
Tracing slows down the simulation by roughly factor of 1000.
The timeout adjustment may be done via bazel's --test_timeout option.
Putting these all together for the UART smoke test yields the following command line:
cd $REPO_TOP
bazel test //sw/device/tests:uart_smoketest_sim_verilator \
--test_output=streamed \
--test_timeout=1000 \
--test_arg=--verilator-args=--trace=/tmp/sim.fst
gtkwave /tmp/sim.fstWithout an argument to --trace, the waveform file would be named sim.fst and be placed in the test's runfiles tree.
It would appear alongside the simulator's other outputs in the test's working directory.