Test suite clean up #3385

JDBetteridge · 2024-02-02T16:52:43Z

Description

This PR started as an experiment to "cheaply" speed up the test suite by calling mpiexec wrapping pytest, rather than forking a subprocess which calls mpiexec (which is also problematic for other reasons).

This PR now carries around multiple test suite fixes that should be merged back to master and includes fixes including:

Adding comm arguments to function calls that need them.
Freeing comms that are created.
Disabling a test that pollutes the tape.
"Fixing" ensemble parallel tests by using the simple partitioner (just in tests, Ensemble needs a proper fix!)
This work had to be rebased on JDBetteridge/update caching #3730 and uses PyOP2 #724 and FInAT #134 due to the deadlocks that they call.

We need to consider what aspects of this experiment we want to incorporate back into master.

Some timings for the actual speed-up (the original intention):

Results

(Real only)

Master

This week's scheduled execution:

Total (inc install): 50m 45s

This branch

With fixed caches, mpispawn, fixed FInAT hashes and pytest-split based on a timed execution.
NB: We tweak vertexonly/test_poisson_inverse_conductivity.py to only do 3 iterations (see diff)

Serial: 17m51s
2: 2m59s
3: 6m43s
4: 45s
6: 19s
7: 48s
8: 12s
Total (inc install): 46m 6s

Important, this branch only runs a maximum of 12 ranks/threads!

connorjward · 2024-02-02T17:38:33Z

~~This is cool, but isn't it a bad idea to effectively remove test coverage? If CI doesn't run all the tests no one will.~~

~~I can see this being useful in the context of a bigger change where we run the test suite with a number of Firedrake configurations and only one of them would run these slow tests.~~

tests/regression/test_ensembleparallelism.py

.github/workflows/build.yml

github-actions · 2024-09-12T13:53:27Z

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake complex	8094 ran	6553 passed	1541 skipped	0 failed

github-actions · 2024-09-12T13:59:51Z

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake real	6986 ran	6542 passed	427 skipped	17 failed

connorjward

Generally very happy with this.

.github/workflows/build.yml

.test_durations

firedrake/parameters.py

firedrake/slate/slac/compiler.py

firedrake/tsfc_interface.py

tests/demos/test_demos_run.py

tests/output/test_io_mesh.py

tests/slate/test_hdg_poisson.py

.github/workflows/build.yml

firedrake/tsfc_interface.py

.github/workflows/build.yml

connorjward

Leaving notes for someone (most likely me) to refer to in future. In summary:

Need to rebase/merge in master.
Tweaks to Makefile and build.yml.

connorjward · 2024-10-31T14:14:07Z

.github/workflows/build.yml

+            -o faulthandler_timeout=1860 \
+            --junit-xml=firedrake2_\$MPISPAWN_TASK_ID1.xml \
+            -m "parallel[\$MPISPAWN_WORLD_SIZE] and not broken" \
+            -v tests


TODO: "dogfood" (bleh) Makefile and use a matrix to massively cut down on boilerplate

connorjward · 2024-10-31T14:17:08Z

Makefile

+.PHONY: test_smoke
+test_smoke:
+	@echo "    Running the bare minimum smoke tests"
+	@python -m pytest -k "poisson_strong or stokes_mini or dg_advection" -v tests/regression/


It would be better to use MPI on the "outside" here for the parallel tests so this can be run to check things on HPC

connorjward · 2024-10-31T14:17:43Z

Makefile

-endif
+# Requires pytest and pytest-mpi only
+.PHONY: test_serial
+test_serial:


Terrible name! This runs all the parallel tests too!

connorjward · 2024-10-31T14:18:39Z

Makefile

+
+# Requires pytest and pytest-mpi only
+.PHONY: test_smoke
+test_smoke:


bikeshedding: I prefer make smoke_tests or make smoketests

connorjward · 2024-10-31T14:19:28Z

Makefile

+	done
+
+.PHONY: _test_large_world_test
+_test_large_world_tests:


I'm not sure why we have small_world and large_world tests separately.

firedrake/slate/slac/kernel_builder.py

…_tests

connorjward · 2025-01-09T17:21:40Z

tests/firedrake/conftest.py

    config.addinivalue_line(
        "markers",
-        "skipreal: mark as skipped unless in complex mode")
+        "broken: mark a test that is broken"


remove this

connorjward · 2025-01-09T17:22:01Z

tests/firedrake/conftest.py

+    )
+    config.addinivalue_line(
+        "markers",
+        "skipcomplexnoslate: mark as skipped in complex mode due to lack of Slate"


connorjward · 2025-01-09T17:22:25Z

tests/firedrake/conftest.py

    config.addinivalue_line(
        "markers",
-        "skipvtk: mark as skipped if vtk is not installed")
+        "skipnetgen: mark as skipped if netgen and ngsPETSc is not installed"


connorjward · 2025-01-09T17:26:15Z

tests/firedrake/multigrid/test_netgen_gmg.py

@@ -7,6 +7,7 @@
    import ngsPETSc
    del ngsPETSc
 except ImportError:
+    # Netgen is not installed


Suggested change

# Netgen is not installed

connorjward · 2025-01-09T17:30:23Z

tests/firedrake/vertexonly/test_poisson_inverse_conductivity.py

+@pytest.mark.markif_fixture(pytest.mark.slow, num_points="sparse")
+@pytest.mark.markif_fixture(pytest.mark.slow, num_points="dense")


Suggested change

@pytest.mark.markif_fixture(pytest.mark.slow, num_points="sparse")

@pytest.mark.markif_fixture(pytest.mark.slow, num_points="dense")

connorjward · 2025-01-09T17:34:39Z

tests/firedrake/equation_bcs/test_equation_bcs.py

@@ -321,6 +321,7 @@ def test_EquationBC_mixedpoisson_matrix_fieldsplit():
    assert abs(math.log2(err[0][0]) - math.log2(err[1][0]) - (porder+1)) < 0.05


+@pytest.mark.slow


Suggested change

@pytest.mark.slow

connorjward · 2025-01-09T17:34:57Z

tests/firedrake/equation_bcs/test_equation_bcs.py

@@ -231,7 +231,7 @@ def test_EquationBC_poisson_matrix(eq_type, with_bbc):
    assert abs(math.log2(err[0]) - math.log2(err[1]) - (porder+1)) < 0.05


-@pytest.mark.parametrize("with_bbc", [False, True])
+@pytest.mark.parametrize("with_bbc", [False, pytest.param(True, marks=pytest.mark.slow)])


Suggested change

@pytest.mark.parametrize("with_bbc", [False, pytest.param(True, marks=pytest.mark.slow)])

@pytest.mark.parametrize("with_bbc", [False, True])

connorjward · 2025-01-09T17:35:12Z

tests/firedrake/demos/test_notebooks_run.py

+@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="09-hybridisation.ipynb")
+@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="10-sum-factorisation.ipynb")
+@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="12-HPC_demo.ipynb")


Suggested change

@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="09-hybridisation.ipynb")

@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="10-sum-factorisation.ipynb")

@pytest.mark.markif_fixture(pytest.mark.slow, ipynb_file="12-HPC_demo.ipynb")

connorjward · 2025-01-09T17:35:53Z

tests/firedrake/conftest.py

+        markif_fixtures = [m for m in item.own_markers if m.name == "markif_fixture"]
+        for mark in markif_fixtures:
+            '''@pytest.mark.markif_fixture(*marks, **conditions)
+            marks: str | pytest.mark.structures.Mark
+                marks to apply if conditions are met
+            conditions: dict
+                dictionary of conditions; consisting of function argument keys
+                and fixture values or ids
+            '''
+            # (function argument names, fixture ids) in a list
+            fixtures = [(name, id_) for name, id_ in zip(item.callspec.params.keys(), item.callspec._idlist)]
+            # If all the fixtures are in the dictionary of conditions apply all of the marks
+            if all((k, str(v)) in fixtures for k, v in mark.kwargs.items()):
+                for label in mark.args:
+                    if isinstance(label, str):
+                        item.add_marker(getattr(pytest.mark, label)())
+                    else:
+                        item.add_marker(label())
+


Suggested change

markif_fixtures = [m for m in item.own_markers if m.name == "markif_fixture"]

for mark in markif_fixtures:

'''@pytest.mark.markif_fixture(*marks, **conditions)

marks: str | pytest.mark.structures.Mark

marks to apply if conditions are met

conditions: dict

dictionary of conditions; consisting of function argument keys

and fixture values or ids

'''

# (function argument names, fixture ids) in a list

fixtures = [(name, id_) for name, id_ in zip(item.callspec.params.keys(), item.callspec._idlist)]

# If all the fixtures are in the dictionary of conditions apply all of the marks

if all((k, str(v)) in fixtures for k, v in mark.kwargs.items()):

for label in mark.args:

if isinstance(label, str):

item.add_marker(getattr(pytest.mark, label)())

else:

item.add_marker(label())

connorjward · 2025-01-09T17:36:36Z

tests/firedrake/conftest.py

+    config.addinivalue_line(
+        "markers",
+        "slow: mark a test that takes a while to run"
+    )


JDBetteridge · 2025-01-09T17:40:22Z

Why would you remove the slow markers? They are really useful if you want to run the test suite faster (especially in parallel!) and are also there to highlight tests that can be sped up.

connorjward · 2025-01-09T17:49:04Z

Why would you remove the slow markers?

Sorry to undo your hard work but I don't see the benefit in having them:

They are really useful if you want to run the test suite faster (especially in parallel!)

But it doesn't run all of the tests. It is useful to be able to run only a few fast tests (a la make check) but I can't imagine a situation where I would want to run most-but-not-all of the tests.

and are also there to highlight tests that can be sped up.

We already know which tests are slow from .test_durations and from the

============================ slowest 200 durations =============================

output from pytest. This is another source of truth and I think it will immediately bit-rot.

JDBetteridge added the DO NOT MERGE label Feb 2, 2024

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from ecb9628 to f3685b7 Compare February 9, 2024 19:21

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from f3685b7 to f848c6f Compare March 5, 2024 13:40

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 93a0aae to a20fc9d Compare June 7, 2024 13:52

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from a5f614f to 27b9af3 Compare July 19, 2024 16:25

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from 2a7f468 to 6ed1774 Compare August 18, 2024 13:44

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 8132b64 to 1122a3b Compare August 31, 2024 18:37

JDBetteridge added enhancement performance and removed DO NOT MERGE labels Sep 4, 2024

JDBetteridge self-assigned this Sep 4, 2024

JDBetteridge added the bug label Sep 4, 2024

JHopeCollins requested changes Sep 10, 2024

View reviewed changes

tests/regression/test_ensembleparallelism.py Outdated Show resolved Hide resolved

JDBetteridge commented Sep 11, 2024

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 9d5f056 to df4aea3 Compare September 12, 2024 13:04

JDBetteridge marked this pull request as ready for review September 24, 2024 14:23

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from bf317f5 to ef87021 Compare October 2, 2024 21:57

connorjward changed the title ~~Mark and skip slow tests~~ Test suite clean up Oct 3, 2024

connorjward reviewed Oct 3, 2024

View reviewed changes

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from 8435a69 to 10f7f0c Compare October 8, 2024 14:02

JDBetteridge and others added 4 commits October 10, 2024 15:38

Draft changes to pyop2.caching

fcdeace

Change package branch

056ba30

Just notes

314ff8e

WIP

3f272d6

JDBetteridge added 3 commits October 10, 2024 15:38

More smothing to improve solutions

ae6a442

Try to prevent pytest overwriting xml files in parallel

0f31713

Dog food flavoured makefile

f493334

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from 10f7f0c to f493334 Compare October 10, 2024 14:38

JDBetteridge commented Oct 10, 2024

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

Remove package branch

8cee812

JDBetteridge linked an issue Oct 10, 2024 that may be closed by this pull request

INSTALL: Tests not passing on fresh install M2 Mac #3793

Closed

JDBetteridge commented Oct 15, 2024

View reviewed changes

firedrake/tsfc_interface.py Outdated Show resolved Hide resolved

JDBetteridge added 6 commits October 15, 2024 15:50

Error in loop on failure

c3dafc5

Give mesh session scope

b9205bb

Apply reviewers comments

9ec599e

Icosahedral radial mesh appears fixed?

2b8938c

Remove duplication from rebase

d26ea36

Test to see if Constant magically works again

fd3f337

JDBetteridge mentioned this pull request Oct 16, 2024

BUG: Use of constant in tests/slate/test_hdg_poisson.py::test_hdg_convergence causes errors #3802

Closed

connorjward reviewed Oct 18, 2024

View reviewed changes

.github/workflows/build.yml Show resolved Hide resolved

.github/workflows/build.yml Outdated Show resolved Hide resolved

JDBetteridge mentioned this pull request Oct 21, 2024

BUG: Performance regression on CI #3603

Closed

Fix constant numbering in SLATE

200682f

JDBetteridge commented Oct 21, 2024

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

JDBetteridge and others added 2 commits October 21, 2024 16:47

Mark another slow demo

6d8c93c

Stop using pytest-mpi branch

130b7b8

connorjward requested changes Oct 31, 2024

View reviewed changes

JHopeCollins mentioned this pull request Nov 18, 2024

BUG: Mesh partitioners are not deterministic across different Ensemble members #3866

Open

Merge remote-tracking branch 'origin/master' into JDBetteridge/faster…

ea03f73

…_tests

connorjward assigned connorjward and unassigned JDBetteridge Jan 9, 2025

connorjward reviewed Jan 9, 2025

View reviewed changes

connorjward added the macOS label Jan 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test suite clean up #3385

Test suite clean up #3385

JDBetteridge commented Feb 2, 2024 •

edited

Loading

connorjward commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

connorjward left a comment

connorjward left a comment

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

connorjward Jan 9, 2025

JDBetteridge commented Jan 9, 2025

connorjward commented Jan 9, 2025

		@pytest.mark.markif_fixture(pytest.mark.slow, num_points="sparse")
		@pytest.mark.markif_fixture(pytest.mark.slow, num_points="dense")

		@@ -321,6 +321,7 @@ def test_EquationBC_mixedpoisson_matrix_fieldsplit():
		assert abs(math.log2(err[0][0]) - math.log2(err[1][0]) - (porder+1)) < 0.05


		@pytest.mark.slow

	@pytest.mark.parametrize("with_bbc", [False, pytest.param(True, marks=pytest.mark.slow)])
	@pytest.mark.parametrize("with_bbc", [False, True])

Test suite clean up #3385

Are you sure you want to change the base?

Test suite clean up #3385

Conversation

JDBetteridge commented Feb 2, 2024 • edited Loading

Description

Results

Master

This branch

connorjward commented Feb 2, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

connorjward left a comment

Choose a reason for hiding this comment

connorjward left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JDBetteridge commented Jan 9, 2025

connorjward commented Jan 9, 2025

JDBetteridge commented Feb 2, 2024 •

edited

Loading

connorjward commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading