This file explains the zkEVM Prover debug tips & tricks. Please read README.md first. All configuration parameters are documented in ./src/config/README.md . Although the config file can be specified as an argument, we will assume for simplicity that it is named config.json, the default config file name.
The first place where to find debugging information is the process logs.
You can redirect the error channel to the output channel and save all logs to a file named 'logs' by calling the zkEVM Prover with the following command:
./build/zkProver 2>&1 | tee logs
The main state machine is implemented both in native C code and in generated C code.
The native code is slower and easier to debug, while the generated code is faster and more difficult to debug.
The result of both native and generated code should be identical, apart from the performance.
You can switch from one to the other by editing the config.json file and rebooting the zkEVM Prover:
"useMainExecGenerated": true,
'true' will configure the zkEVM Prover to use generated code, and it is the default value for performance reasons. 'false' will configure the zkEVM Prover to use native code.
When the zkEVM Prover is configured to use native code, you can get a log describing what every ROM instruction does. This mechanism can be useful to understand the context under which a certain situation happens, e.g. an error. Warning: this mechanism can generate a lot of traces, so use it with caution. You can enable this mechanism by editing the config.json file and rebooting the zkEVM Prover:
"executorROMLineTraces": true,
The zkEVM Prover can be configured to save to file the input or the output, in different formats. Files are generated in the directory './runtime/output' by default. You can change the output directory by editing the config.json file and rebooting the zkEVM Prover:
"outputPath": "runtime/output",
The most used format is JSON.
You can save executor GRPC input to a JSON file by editing the config.json file and rebooting the zkEVM Prover:
"saveInputToFile": true,
With this input file you can re-submit the same batch later, as long as you have access to the same database, by editing the config.json file and rebooting the zkEVM Prover:
"runExecutorClient": true,
"inputFile": "./runtime/output/<filename>.json",
If you don't have access to the database when debugging, the execution will fail with a 'key not found" error. If you don't want to depend on the database access, you must save all DB reads to the same JSON file, by editing the config.json file and rebooting the zkEVM Prover:
"saveDbReadsToFile": true,
This configuration will save not only the input parameters, but also the DB read data. However, this file is saved after the batch has been processed. If the process is killed before you reach the point where this file is saved, you can save it after every DB read. This slows down the processing, but ensures having the required DB data to reach the problematic point of the execution. If you want to get the input file continously updated, you can enable this feature by editing the config.json file and rebooting the zkEVM Prover:
"saveDbReadsToFileOnChange": true,
You can also save the GRPC input request to a text file, by editing the config.json file and rebooting the zkEVM Prover:
"saveRequestToFile": true,
The content of this file is directly generated by the GRPC library, which can be useful if you are familiar with this format. The numeric values that are zero are not logged, since it is their default value. The binary non-textual values are logged in octal escaped format. This format is useful to ensure that some data sent by the service client actually reached the zkEVM Prover.
You can save executor GRPC output data to a JSON file by editing the config.json file and rebooting the zkEVM Prover:
"saveOutputToFile": true,
You can save executor GRPC output response to a text file by editing the config.json file and rebooting the zkEVM Prover:
"saveResponseToFile": true,
Some of the described configuration option generate one file per batch. If you have several of this configuration options enabled, you can get several files per batch. This can represent a high number of files after some time of operation. Once you have completed your debugging, consider disabling these options to avoid storage issues and low execution performance.