BUG: Performance regression on CI

[JDBetteridge]

Describe the bug
We have hit yet another performance regression!
This PR showed a performance regression:

PR: #3577

CI (tests 1hr): https://github.com/firedrakeproject/firedrake/actions/runs/9288922418/job/25561549540

When merged to master the same behaviour was observed:

CI (tests 52mins): https://github.com/firedrakeproject/firedrake/actions/runs/9316886159/job/25646023795

When that commit is reverted the execution time drops again:

CI (tests 42mins): https://github.com/firedrakeproject/firedrake/actions/runs/9319667894/job/25654727405?pr=3602
(42mins is longer than the scheduled tests which run for 35mins, but usually these have no resource contention https://github.com/firedrakeproject/firedrake/actions/runs/9239093106/job/25417813619)

This PR is keeping track of various fixes #3602

I'm going to wait for the next scheduled CI run to see how significant this issue is.

Steps to Reproduce
Steps to reproduce the behavior:

1. Use CI

Expected behavior
Same performance as before.

Error message
N/A

Environment:

• CI

Additional Info
Add any other context about the problem here.

[JDBetteridge]

Maybe everything is fine...
https://github.com/firedrakeproject/firedrake/actions/runs/9334091136/job/25727536478

[JDBetteridge]

These regressions may be related to the disks on the runners being almost full!

I will continue to monitor the situation

[connorjward]

Is this resolved?

[JDBetteridge]

Questionable, I think the issue needs to be potentially rephrased as it's unclear what a solution looks like. It could potentially be resolved by #3385

[connorjward]

Closing as this is very old now. We don't really seem to be observing any performance issues at the moment.