Use dune's include stanza for generated tests #729
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hra687261
approved these changes
Jul 12, 2023
Fine by me, but could be worth getting a second opinion form @Halbaroth or @Gbury |
|
OK, I'll wait until either of them chimes in. |
|
In my opinion, we shouldn't force the update to dune 3.5 for such a tiny benefit. |
|
OK, I will duplicate everything (athough I think most people already use a version of dune higher than 3.5 if they are using opam). |
|
Could rebase this PR please? |
|
Looks like the CI doesn't see the test files for some reason, I will have to investigate... |
Merged
The current implementation of test generation is quite high friction: - When adding new tests, running tests directly with dune doesn't work, because the dune files must be generated by `make gentest` beforehand. There is `make runtest`, but that always runs all the tests, and does not allow specifying only some tests to run (which, given that our test suite is quite big, is sometimes needed for quick iterations). - The generated dune files are not commited to the repository, which means that it is easy to forget to run `make gentest` when switching branches -- which either causes some tests to silently not run, or errors because some files are missing. This patch solves both issues by integrating the test generation with dune, using the `include` stanza. More precisely: - The test generation script (gentest.ml) is modified to write all the test cases on the standard output as a single file, rather than creating the dune test files by itself. The generated file makes use of the `subdir` stanza to locate the targets appropriately. - There is a new `dune.inc` file that is included by the `dune` file in `tests`, and a `diff` rule that generates the content of that `dune.inc` file using the `gentest.ml` script. This is the way rules generation must work in dune, and it means that the `dune.inc` file must be commited to the repository --- which may not be such a bad thing, see above. The rule generation is added to the `gentest`, `runtest`, `runtest-quick` and `runtest-ci` aliases. The advantage of the new approach is that since the dependency is known to dune (and not only to make), running a command such as `dune build @tests/issues/XXX/runtest-quick` will always regenerate the tests, and also that we won't have to delete `dune` files manually when switching branches, since everything is in the file that gets re-generated. It means that running tests is slightly slower, because it always requires running the `gentest` script --- but that is probably fine, until we have many more tests than currently.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the dune files for tests are generated using the Makefile, and not included in the repository. There are often issues when changing branches that add new tests, because the dune files easily get out of sync, and regularly require manual intervention to remove them when new test directories are added.
This patch uses dune's auto-generated files and include mechanism for test generation instead. More precisely:
The test generation script (gentest.ml) is modified to write all the test cases on the standard output as a single file, rather than creating the dune test files by itself. The generated file makes use of the
subdirstanza to locate the targets appropriately.There is a new
dune.incfile that is included by thedunefile intests, and adiffrule that generates the content of thatdune.incfile using thegentest.mlscript. This is the way rules generation must work in dune, and it means that thedune.incfile must be commited to the repository --- which may not be such a bad thing, see above.The rule generation is added to the
gentest,runtest,runtest-quickandruntest-cialiases through thealiasesstanza, which requires bumping our dune dependency to 3.5. If that is an issue, we can duplicate the rule instead.