To utilize the custom checker, please ensure that the manifest.yaml file does not include the checker attribute.
The custom checker, which should be named checker, must be stored in the main directory of the task.
The checker script must be provided by the user and takes in the following command-line arguments:
- Absolute path to the input file of the test case
- Absolute path to the output file generated by the user's program for the test case
- Absolute path to the solution file of the test case
Of course, the checker script is passed to itself as the 0-th argument, but it can be safely ignored.
The checker must then write two lines to standard output. The first line denotes the verdict of the user's program, which one be either of the following:
- Correct
- Partially correct
- Incorrect
- Time Limit Exceeded
- Memory Limit Exceeded
- Runtime Error
- Signal Error
- Judge Error
In a custom checker, metrics about the user's program on the current test case must be printed on the second line. If a custom grouper is used, then any string can be printed on the second line. Otherwise, if one of the default groupers is used, you must conform to its protocol (see Default Groupers for more information). In a custom checker, the score of the user's program on the current test case must then be printed on the second line. Finally, on the last line, the checker can optionally output a message describing the result of the test case. If no message is provided, the default message specified in the global configuration (see Global Configuration) will be automatically added instead if it exists.
For example, the following are valid outputs from a custom checker:
Correct
20
Target reached in 25 moves
Partially Correct
[s1]asdf
Incorrect
!!!
Wrong format
The "Judge Error" verdict can be output from both the grader and a custom checker. The grader will output "Judge Error" when there is an internal error of the grader. On the other hand, the custom checker should output "Judge Error" when there is an internal problem of the custom checker. In the case that the judging error comes from the grader, the following will be output:
Judge Error
0
{DEFAULT_MESSAGE}
If you intend to utilize C++ code without the need for compilation, you can incorporate the provided code snippet into the checker file:
#!/bin/bash
FILENAME="checker.cpp"
BINNAME="output"
DIR="$( cd "$( dirname "$0" )" && pwd )"
CHECKER="${DIR}/${BINNAME}"
LAST_CHECKER="$(stat -c%Y ${DIR}/${FILENAME})"
LAST_BIN="$(stat -c%Y ${CHECKER} 2> /dev/null)"
if [[ ! -f "${CHECKER}" ]] || [[ $LAST_CHECKER -ge $LAST_BIN ]]; then
( cd "${DIR}" && g++ -std=c++11 -O2 -Wall -Wextra -pedantic -o "${BINNAME}" "${FILENAME}")
fi
( cd "${DIR}" && "./${BINNAME}" $1 $2 $3 )Additionally, remember to place the checker.cpp file inside the main directory of the task.
Default checkers are provided with the grader that can easily be used by specifying them as the checker in task manifests. This removes the hassle of having to write checkers for typical tasks. All checkers output the verdict in the first line, a score out of 100 for the second line, and the default message of the corresponding verdict on the third line. If first line "Correct", then the second line will be 100. Otherwise, it will be 0. Note that default checkers will never emit the "Partially Correct" verdict. Each default checker will only emit the "Correct" verdict if all tokens match between the user's output and the solution's output.
Each default checker will split output into tokens as follows:
-
ncmp: single or more 64-bit integers, ignores whitespaces -
wcmp: sequence of tokens -
nyesno: single or more yes/no tokens, case insensitive -
lcmp: lines, ignores whitespaces -
fcmp: lines, doesn't ignore whitespaces -
rcmp6: single or more floating point numbers, maximum error$10^{-6}$ -
rcmp9: single or more floating point numbers, maximum error$10^{-9}$