-
Notifications
You must be signed in to change notification settings - Fork 7.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Start documenting testing section
- Loading branch information
Showing
6 changed files
with
185 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
################# | ||
Testing | ||
################# | ||
|
||
.. toctree:: | ||
:hidden: | ||
|
||
running-tests | ||
writing-tests | ||
|
||
This section provides an overview of the testing infrastructure used for php-src. | ||
|
||
******* | ||
TODOS | ||
******* | ||
|
||
- Talk about bless-tests | ||
- Writing conflict free tests | ||
- Describe all sections from https://qa.php.net/phpt_details.php | ||
- How to invoke the test runner directly | ||
- Files generated when a test fails | ||
- Invoke GDB directly via the generated ``.sh`` file. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
################# | ||
--FILE-- | ||
################# | ||
|
||
The most common way to provide the script of the test being tested. | ||
The script within the ``--FILE--`` section should be wrapped by an opening *and* closing PHP tag. | ||
As this allows to execute the ``.phpt`` directly without invoking the test runner. | ||
|
||
********* | ||
Example | ||
********* | ||
|
||
.. code:: php | ||
--FILE-- | ||
<?php | ||
$r = 20 + 36; | ||
var_dump($r); | ||
?> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
################# | ||
--TEST-- | ||
################# | ||
|
||
The mandatory ``--TEST--`` section that must be at the beginning of a PHPT. | ||
|
||
It should contain the title of the test as plain text on a single line. | ||
|
||
********* | ||
Example | ||
********* | ||
|
||
.. code:: plaintext | ||
--TEST-- | ||
Test filter_input() with GET and POST data. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
######################## | ||
Running the test suite | ||
######################## | ||
|
||
php-src has a custom test runner that is the ``run-tests.php`` file at the root of the repository. | ||
|
||
There are two ways of executing the test runner, either via the makefile, or by invoking the test runner directly. | ||
|
||
****************** | ||
Via the makefile | ||
****************** | ||
|
||
After having compiled PHP a prompt to execute ``make test`` appears, | ||
doing so will run all the tests (a bit less than 18 000 at the time of writing) | ||
with a single thread. | ||
This is slow, but there are various ways to speed this up! | ||
|
||
First of all it is possible to set a ``TEST_PHP_ARGS`` variable with the make command | ||
that will pass the defined arguments to the test runner. | ||
One important argument is the ``-jN`` argument which allows the tests to be run in parallel (if possible). | ||
Some other important arguments are ``-q``, which makes the test runner silent (i.e. stops prompting to send the test results via email), | ||
and the ``--asan`` argument which is required when running tests with a build of PHP compiled with ASAN. | ||
|
||
Secondly, one can reduce the number of tests being run by specifying which individual tests, or test folders | ||
should be run via the ``TESTS`` variable. | ||
|
||
****************************************** | ||
Via direct invocation of the test runner | ||
****************************************** | ||
|
||
TODO | ||
|
||
********************* | ||
Test runner options | ||
********************* | ||
|
||
The test runner accepts various options that can be discovered by invoking it with ``--help`` | ||
Here is a list of the most common and useful options: | ||
|
||
- ``-g``: to filter which tests are shown, e.g. ``-g FAIL`` to only show failing tests | ||
- ``-x``: to skip tests marked as slow | ||
- ``--offline``: to skip tests which require an internet connection | ||
- ``-d``: to set an INI option, useful to enable/disable opcache and the JIT, e.g. ``-d opcache.enable=0`` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
######################## | ||
Writing tests | ||
######################## | ||
|
||
php-src's test runner uses a custom format for tests, named PHPT. | ||
Each PHPT test is composed of multiple sections, | ||
however each test must have at least 3 sections: | ||
|
||
- A title | ||
- A script to test | ||
- The expected output of the test | ||
|
||
A complete description of the available section is available TODO. | ||
|
||
.. toctree:: | ||
|
||
phpt-sections/test | ||
phpt-sections/file | ||
|
||
******************* | ||
Naming convention | ||
******************* | ||
|
||
The name of a test file should ideally be descriptive, | ||
however a lot of old tests files do not follow this convention as the old naming convention was: | ||
|
||
:Tests for a function's basic behaviour: ``<function_name>_basic.phpt`` (e.g. ``dba_open_basic.phpt``) | ||
|
||
:Tests for a function's error behaviour: ``<function_name>_error.phpt`` (e.g. ``dba_open_error.phpt``) | ||
|
||
:Tests for variations in a function's behaviour: ``<function_name>_variation.phpt`` (e.g. ``dba_open_variation.phpt``) | ||
|
||
:General tests for extensions: ``<extname><no>.phpt`` (e.g. ``dba_003.phpt``) | ||
|
||
For bugs the naming convention is as follows: | ||
|
||
- If the bug is a GitHub issue: ``gh<issue_no>.phpt`` (e.g. ``gh9032.phpt``) | ||
- If the bug is a bugsnet bug: ``bug<bug_no>.phpt`` (e.g. ``bug20528.phpt``) | ||
|
||
********************** | ||
Writing a basic test | ||
********************** | ||
|
||
|
||
.. code:: php | ||
--TEST-- | ||
echo - basic test for echo language construct | ||
--FILE-- | ||
<?php | ||
echo 'This works ', 'and takes args!'; | ||
?> | ||
--EXPECT-- | ||
This works and takes args! | ||
****************************************** | ||
An example of a more complicated test | ||
****************************************** | ||
|
||
Let's write a test which checks that 64 bit integers are usable with the X function of the DateTime extension | ||
|
||
TODO | ||
|
||
.. code:: php | ||
--TEST-- | ||
Test that X works with 64 bit integers | ||
--EXTENSIONS-- | ||
date | ||
--SKIPIF-- | ||
<?php | ||
if (PHP_INT_SIZE != 8) die("skip this test is for 64bit platform only"); | ||
?> | ||
--FILE-- | ||
<?php | ||
echo 'TODO'; | ||
?> | ||
--EXPECT-- | ||
TODO |