42ShellTester is an integration testing framework wrote in Bash and designed for the pedagogical projects of the Shell branch at School 42 (Paris) listed bellow:
- minishell
- 21sh
- 42sh
It brings you an easy way to add, maintain and run integration tests, helping you to work step by step on your Shell implementation.
42ShellTester is currently packaged with 289 tests.
git clone https://github.com/we-sh/42ShellTester ~/42ShellTesterAdd the path to your Shell as argument:
bash ~/42ShellTester/42ShellTester.sh "/ABSOLUTE/PATH/TO/YOUR/SHELL"Run tests that matches with the specified regular expression (e.g. --filter "builtins").
Run tests that does not fail with the specified Shell binary (e.g. --reference "bash").
Run tests that are POSIX compliant only (run all by default).
Run tests that are marked as « hard » (omitted by default).
Also run pending tests.
Equivalent to use the two options --pending and --hard together.
Also display tests that succeed (hidden by default).
- 21sh/
- 42sh/
- builtins/
- export/
- 001-display-env
- 002-export-basic-key-value-1
- 003-export-basic-key-value-2
- 004-export-empty-variable-1
- 005-export-empty-variable-2
- 006-export-update-env-variable
- 007-existing-environment-variable
- 008-local-to-environment
- 009-export-with-equal-but-no-value-part1
- 010-export-with-equal-but-no-value-part2
- errors/
- mixed/
- options/
- export/
- escaping/
- globbing/
- brace-expansion/
- ascii-range/
- errors/
- list-of-values/
- numeric-range/
- 001-simple-ascending-1
- 002-simple-ascending-2
- 003-simple-ascending-3
- 004-simple-ascending-4
- 005-simple-ascending-5
- 006-simple-descending-1
- 007-simple-descending-2
- 008-simple-descending-3
- 009-simple-descending-4
- 010-simple-descending-5
- 011-identical-positive-start-and-end
- 012-identical-negative-start-and-end
- 013-multiple-1
- 014-multiple-2
- 015-big-range
- bracket-expansion/
- brace-expansion/
- local-variable/
- 001-declare-and-expand-1
- 002-declare-and-expand-2
- 003-unknown-variable-does-not-result-in-new-argument
- 004-existing-variable-in-environment-1
- 005-existing-variable-in-environment-2
- 006-existing-variable-in-environment-3
- 007-multiple-declaration-at-a-time
- 008-multiple-declaration-with-same-name
- 009-last-exit-status
- mixed/
- quoting/
- double-quotes/
- mixed/
- simple-quotes/
- subshell/
- builtins/
- bonuses/
- minishell/
- binary/
- builtins/
- cd/
- 001-no-arg
- 002-current-directory
- 003-current-directory-2
- 004-parent-directory
- 005-root-path
- 006-root-path-2
- 007-symbolic-link
- 008-symbolic-link-2
- 009-following-links
- 010-update-OLDPWD
- 011-dotdot
- 012-dot
- 013-absolute-path
- errors/
- 001-not-a-directory
- 002-not-a-directory-2
- 003-permission-denied
- 004-permission-denied-2
- 005-too-many-symbolic-links-encountered
- 006-too-many-symbolic-links-encountered-2
- 007-no-such-file-or-directory
- 008-no-such-file-or-directory-2
- 009-no-such-file-or-directory-symlink
- 010-no-such-file-or-directory-symlink-2
- options/
- env/
- exit/
- mixed/
- setenv/
- unsetenv/
- cd/
- misc/
- Scope indentation must be done with 2 spaces
- Variable names must be upper case (e.g.
INDEX) - Global variables must be prefixed with
GLOBAL_(e.g.GLOBAL_TOKEN) - Variable expansion must be surrounded by curly braces (e.g.
${VARIABLE}) - Arguments of functions and commands must be surrounded by double or simple quotes (e.g.
run_assert "STDERR") - Semicolon are banned so that words like
thenanddoare always at start of lines (e.g. wrong inline code:if [ ... ]; then cmd; fi) - Folder names may only contain alphanumeric and
-
An integration test must be self-sufficient, that means executing the full test suite or only one test must result in the same failed or success status. The framework 42ShellTester brings you tools for that!
Firstly, tests are executed inside a temporary folder tmp/ that is created at launch time and placed at the root installation folder of the framework. You may generate temporary files, binaries and folders that are needed for your test, but pay attention to not touch external folders. Use the before_exec callback to generate these resources.
Secondly, each test is executed within a sub-shell, so that you may modify the environment without disrupting the test suite. Use the before_exec callback to modify the environment.
Thirdly, a test must concern one single feature at a time, that means wherever possible you must avoid the use of multiple builtins or capabilities (e.g. do not use a pipe | within a test that concerns the builtin env, or again use absolute paths to binaries like /bin/ls to let the Shell implementation not support the PATH, except if you precisely test this feature!).
Fourthly, when a test need binaries like /bin/env or /bin/echo, prefer to recode your own, simplier and multi-platform, and place it in support/ folder. Then use the before_exec callback to compile it and make it available for your test.
Sixthly, a test that is not POSIX compliant must contain a file named non-posix containing a small explanation of why.
Finally, don't write a README and let the task generate_readmes do it for you :-) A description may be added in a file named description that will appear at the top of the README.
Follow the guideline to add a new test:
- Create a sub-folder in
spec/(e.g.spec/minishell/builtin/cd/new-test/) - If necessary, create a file
before_execthat contains the shell commands that prepare the environment and the temporary resources (e.g.mkdir valid_folder) - Create a file
stdinthat contains the shell command you want to test (e.g.cd invalid_folder) - Create the files
stdoutand/orstderrthat contain the expected output assertions (e.g. in stderr:expected_to_not be_empty) (see available assertions and verbs bellow) - You may also create a file
miscthat contains special expectations not concerning output on standard and error (e.g.expected_to_not exit_with_status 0) - If necessary, create a file
descriptionthat describes more precisely the purpose of the test (e.g.Trying to access invalid folder must display an error on standard error and result in a failure status code) (the description will be included at top of the auto-generated README) - If the test is not POSIX compliant, create a file
non-posixthat explains why.
-
expected_to/expected_to_not+verb: An assertion beginning with expected_to (or its opposite expected_to_not) makes the test resulting in failure status if the expectation that follows does not comply. -
might/might_not+verb: An assertion beginning with might (or its opposite might_not) always makes the test resulting in success status. When the expectation that follows may not comply, it is nevertheless considered as success but it displays a warning message.
be_empty: Actual output is empty.create_file+$filename: Actual command creates a file named $filename. May also be followed with a file test:matching_regex+$regex: At least one line of the file matches with the regular expression $regex.not_matching_regex+$regex: Any line of the file does match with the regular expression $regex.with_nb_of_lines+$int: The file contains exactly $int lines.
exit_with_status+$int: The Shell termination results in the exit status $int.have_nb_of_lines+$int: Actual output contains exactly $int lines.match_regex+$regex: At least one line of actual output does match with the regular expression $regex.once: The matching is limited to only one occurrence.$inttimes: The matching must exactly occur $int times.
match_each_regex_of_file+$filename: Actual output does match with each regular expression contained in the file named $filename (in an indifferent order).
A verb is a function that is prefixed by run_verb_ and that returns 0 or 1 according to the tested behavior. It may return a status 255 when bad or missing argument.
At runtime, the framework provides a list of variables that can be used by the verbs:
RESPONSE: The path to the file containing actual output (STDOUT or STDERR)RESPONSE_EXIT_STATUS: The exit status of the Shell terminationEXPECTED_TO_ARGS[]: An array containing the arguments following the verb
Follow the guideline to add a new verb:
- Choose the best name that respects the CamelCase convention and that can be human-readable when used with an assertion (e.g.
expected_to be_emptycan be readactual output is expected to be empty) - Create a file in
lib/verbs/with the exact name of the verb and that is prefixed withrun_verb_(e.g.lib/verbs/run_verb_be_empty.sh - Add a shebang:
#!/bin/shand a comment that describes the tested behavior - Create a function with the exact name of the verb and that is prefixed with
run_verb_(the same as the file name) and make it respect the following rules:
- Local variables must be declared with
local - No output can be done with
echoorprintf - Function returns
0on succes,1on fail or255on bad use - Use the array
EXPECTED_TO_ARGS[]to take advantage of arguments (e.g.expected_to match_regex "regex", thenEXPECTED_TO_ARGS[0]containsregex)
The framework 42ShellTester provides several binaries to be used within the tests. Using them instead of using Unix binaries prevents from undefined behaviors and compatibility errors.
Find the available list of support binaries bellow:
- ./display_env: A binary that iterates on
**envpand write each element on standard output. - ./display_program_name: A binary that writes its name on standard ouput.
- ./display_pwd: A binary that writes on standard output the absolute path of the current directory returned by
getcwd(3), encountered with the stringsPWD:and:PWD. - ./exit_with_status: A binary that immediately exits with the status given as first argument.
- ./read_on_stdin: A binary that reads on standard entry and write each line on standard output suffixed with the character
@(e.g. same behavior ascat -eand the newline character). Whenread(2)returns-1, then the stringSTDIN READ ERRORis written on standard error. - ./sleep_and_exit_with_status: A binary that sleeps for a duration in seconds given as first argument and then exits with status given as second argument.
- ./sleep_and_write_on_stderr: A binary that sleeps for a duration in seconds given as first argument and then writes on STDERR the string given as second argument without EOL.
- ./write_all_arguments_on_stdout: A binary that writes on standard output each argument separated by the symbol
@. If no argument is given, it writes the string "nothing to be written on stdout". - ./write_on_stderr: A binary that writes on standard error the first given argument (the same behavior as
echobut with only one argument) and exits with an error status code given as second argument. If no argument is given, it writes the string "write on stderr" and exit with status1. - ./write_on_stdout: A binary that writes on standard output the first given argument (the same behavior as
echobut with only one argument). If no argument is given, it writes the string "write on stdout". - ./write_on_stdout_and_stderr: A binary that writes on standard output the first given argument, and writes on standard error the second given argument. If an argument is missing, it writes the strings "write on stdout" and "write on stderr".
bash ./tasks/generate_readmes.sh(only on master branch) to automaticaly generate the README files of tests
Edouard Audeguy
Illustrateur / Infographiste
https://edouardaudeguy.wix.com/portfolio