Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve testing speed #244

Open
jdeschamps opened this issue Sep 9, 2024 · 2 comments
Open

Improve testing speed #244

jdeschamps opened this issue Sep 9, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@jdeschamps
Copy link
Member

Since the addition of LVAE tests, testing is taking 10 to 30 minutes per PR / commit. That is way too long...

We could add a pytest.mark to signify tests that take a long time and only have them run on a single platform and only fully on main. That's a bit dangerous because they might escape platform specific problems.

We could also revisit all the tests that take too long and see whether they are entirely necessary.
@jdeschamps jdeschamps added the enhancement New feature or request label Sep 9, 2024
@jdeschamps
Copy link
Member Author

Here are the 20 slowest tests for instance:

14.84s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-None-5-1]
13.35s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-None-5-1]
11.20s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-None-5-8]
11.18s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-pixelwise-5-1]
10.66s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-None-5-8]
10.17s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-pixelwise-5-8]
8.95s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-pixelwise-5-1]
8.26s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-pixelwise-5-8]
4.94s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-None-1-1]
3.86s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-None-1-8]
3.81s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-None-1-1]
3.67s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_denoisplit[1-None-1-denoisplit_musplit]
3.50s call     tests/test_careamist.py::test_predict_on_array_tiled[1-4]
3.40s call     tests/test_careamist.py::test_predict_on_array_tiled[2-4]
3.36s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_denoisplit[1-None-3-denoisplit_musplit]
3.18s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-None-1-8]
3.14s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_denoisplit[1-pixelwise-1-denoisplit_musplit]
3.13s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[3-pixelwise-1-1]
3.10s call     tests/dataset/test_iterable_dataset.py::test_compute_mean_std_transform_welford_with_targets[shape2-ZYX-patch_size2]
2.97s call     tests/lightning/test_LVAE_lightning_module.py::test_training_loop_musplit[1-pixelwise-1-1]

@jdeschamps jdeschamps mentioned this issue Sep 9, 2024
Merged
jdeschamps added a commit that referenced this issue Sep 9, 2024
@jdeschamps
Chore: Split test CIs into CAREamics official and LVAE (#245)
1bd1c6a 
### Description

Currently the CI is extremely slow
(#244), to a point where it
requires 30 mins between pushing a fix and seeing whether all tests
pass... It is in particular due to the addition of numerous LVAE tests.

Until we go over all the tests and sanitize them a bit, this should
allow faster execution.

This PR is a temporary solution to make it easier to work:
- Split the CI between the "officially supported" algorithms and the
LVAE.
- Make LVAE CI run on a single python version and two runners (ubuntu
and windows, macos being super slow)
- Add codecov CI to have the code coverage computed from running all
tests (single runner, single python version)
- Add codecov threshold to not fail if there difference in coverage is
less than 2%
- Reduce `n_images` from `100` to `10` in welford tests

Summary:
- **What**: Make CI run faster.
- **Why**: Currently, CI is excruciatingly slow. 
- **How**: Split slow LVAE tests into their own CI running on fewer
runners.

### Changes Made

- **Added**: 
    - `lvae.yml` CI
    - `codecov.yml` CI
    - `codecov.yml` configuration file (in root)
- **Modified**:
    - all LVAE test files with the marker `lvae`
    - `pyproject.toml` to register the `lvae` marker
    - welford tests to reduce their length

### Related Issues

#244
@jdeschamps
Copy link
Member Author

Merged #245

Current solution has three CIs:

  • CI for officially supported code
  • CI for LVAE (running on fewer runners to gain time)
  • CI for codecov (for the day we want to have coverage from the LVAE)

This is a temporary fix, in the future we should include the LVAE code in the main CI (once it has been refactored) and make sure that the tests are sane.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jdeschamps