Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy eval refactor #6257

Closed
wants to merge 141 commits into from
Closed
Show file tree
Hide file tree
Changes from 122 commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
06d5842
Initial commit
atbenmurray Mar 27, 2023
2507a0b
Refactors and simplification of api / calls
atbenmurray Mar 28, 2023
56b618e
Refactoring functional spatial transform to support lazy_evaluation p…
atbenmurray Mar 28, 2023
315e5b7
Making lazy_evaluation parameter consistent across spatial array lazy…
atbenmurray Mar 28, 2023
4e69313
Updating apply and dictionary spatial transforms to support
atbenmurray Mar 29, 2023
99fa35b
Missing changes to spatial dictionary
atbenmurray Mar 29, 2023
a798e0e
Enable call-time overriding of lazy_evaluation in croppad/functional and
atbenmurray Mar 29, 2023
114b8cc
Updating croppad dictionary and croppad array for call-time laziness
atbenmurray Mar 29, 2023
5df4261
Fix for push_transform; updating the lazy integration test
atbenmurray Mar 29, 2023
84d65f7
removing issue.txt
atbenmurray Mar 30, 2023
7ec442f
. Renaming apply_transform to apply_pending as it is more semantically
atbenmurray Mar 31, 2023
8f62776
Tweaked array ApplyPending and ApplyPendingd for array and dict data
atbenmurray Mar 31, 2023
2492223
Added missing license text.
atbenmurray Mar 31, 2023
0064326
Missing import for apply_pending
atbenmurray Mar 31, 2023
0646602
Merge branch 'dev' into lazy_eval_refactor
atbenmurray Apr 14, 2023
05eb3f9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 14, 2023
788cd74
Fixing syntax error created by merge of dev to PR
atbenmurray Apr 14, 2023
4bd22f8
Fixing issues with pad_func post merge
atbenmurray Apr 14, 2023
1e0510c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 14, 2023
c36b69f
Fixing issues in crop while running unit tests
atbenmurray Apr 14, 2023
8a4f310
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 14, 2023
3c69776
Fixing issues with OneOf/SomeOf/RandomOrder after dev merge
atbenmurray Apr 14, 2023
6b94aa1
More croppad fixes post dev merge
atbenmurray Apr 14, 2023
bf1c01a
Fixing test_nvtx_decorator after def merge
atbenmurray Apr 14, 2023
19745f9
Removed use of **kwargs parameter in tests as it isn't supported by a…
atbenmurray Apr 23, 2023
b219f13
Setting use of lazy_evaluation to False for Rand2D/Rand3DElastic
atbenmurray Apr 23, 2023
6af8f24
Fixed test_spatial_pad
atbenmurray Apr 23, 2023
173c161
Renaming lazy_evaluation to lazy
atbenmurray Apr 23, 2023
2262ba9
Fixed all outstanding usages of apply_pending in tests. Removed the
atbenmurray Apr 23, 2023
aaaea13
Merge branch 'dev' into lazy_eval_refactor
atbenmurray Apr 24, 2023
088d14e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 24, 2023
3e6a637
Accepting formatting auto-fixes
atbenmurray Apr 24, 2023
98a53bf
Renaming surviving lazy_evaluation instances to lazy.
atbenmurray Apr 24, 2023
28a0175
Renamed the last errant 'lazy_evaluation' instances to 'lazy'
atbenmurray Apr 24, 2023
4e9b6c6
Fixing ruff errors for over-long line lengths
atbenmurray Apr 24, 2023
ad23bea
Fix for issue when the 'lazy' property is set to None
atbenmurray Apr 24, 2023
81549e2
Fix for _apply_transform when lazy is None, post merge tidy up
atbenmurray Apr 24, 2023
77eff5b
Removing the redundant check for log_stats from
atbenmurray Apr 24, 2023
7c53d26
Further formatting auto-fixes
atbenmurray Apr 24, 2023
ee5378f
Making default for overrides be None instead of {}.
atbenmurray Apr 24, 2023
28c05d0
Changed execute_pending_transforms to not make a shallow copy of dict…
atbenmurray Apr 24, 2023
d96fef0
Fixing docstring for Compose
atbenmurray Apr 24, 2023
654cf48
Fixing type complaints
atbenmurray Apr 24, 2023
64ec242
Fix for tests/tests_integration_lazy_samples. The modified lazy
atbenmurray Apr 25, 2023
f38eb9b
Flake8 fix
atbenmurray Apr 25, 2023
14349ff
Flake8 fixes
atbenmurray Apr 25, 2023
51fda45
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 25, 2023
c5742c0
Autoformat style fixes
atbenmurray Apr 25, 2023
ef5173e
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 25, 2023
916b0a9
Fixing bug introduced by flake8 fix
atbenmurray Apr 25, 2023
20c1cfc
Merge branch 'dev' into lazy_eval_refactor
atbenmurray Apr 25, 2023
c94341f
Removal of obsolete lazy eval code
atbenmurray Apr 25, 2023
cc51f2f
Autoformat shuffle
atbenmurray Apr 25, 2023
06aad01
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 25, 2023
3ae46e9
Resolving mypy issues
atbenmurray Apr 25, 2023
95a6a5f
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 25, 2023
81559fa
Finally got to grips with mypy troubles
atbenmurray Apr 25, 2023
af9cad1
Final mypy complaints
atbenmurray Apr 25, 2023
a9c8849
Chasing autofixes again
atbenmurray Apr 25, 2023
aadbae3
test_invert_warn_pending is now obsolete due to this no longer being an
atbenmurray Apr 25, 2023
1cccf9b
Adding documentation for ApplyPending/ApplyPendingd. Adding more
atbenmurray Apr 25, 2023
9ae1140
More autofix
atbenmurray Apr 25, 2023
bb2ad0f
Restoring lazy.setters for croppad and spatial lazy transforms
atbenmurray Apr 25, 2023
5abe294
Undoing the previous commit until I get to the bottom of the randaffined
atbenmurray Apr 25, 2023
c2bf4a7
Making apis for OneOf/SomeOf/RandomOrder consistent with Compose
atbenmurray Apr 26, 2023
b16a87f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 26, 2023
816666e
Removing partial docstring for 'lazy', which isn't on the init parameter
atbenmurray Apr 26, 2023
8318121
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 26, 2023
0e10c24
Adding execute_pending_transforms to lazy/functional __all__
atbenmurray Apr 26, 2023
f323ff9
Removing LazyMode; lazy is now True / False / None
atbenmurray Apr 26, 2023
4b69ec6
Renamed execute_pending_transforms to apply_pending_transforms.
atbenmurray Apr 26, 2023
29ce2b7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 26, 2023
6774aaa
Removed "LazyMode" from enums.py __all__
atbenmurray Apr 26, 2023
de5b9b8
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 26, 2023
1c7d172
Modified ApplyPending/ApplyPendingd in the light of the override issue
atbenmurray Apr 26, 2023
bb22c13
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 26, 2023
2159ce8
Correction to type info for execute_compose lazy parameter.
atbenmurray Apr 26, 2023
2c3172d
Removal of comment in docstring about IdentityD being used to cause
atbenmurray Apr 26, 2023
bb1214c
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 26, 2023
5a7820c
Logging refactored with pending pipeline logging enabled
atbenmurray Apr 26, 2023
503db85
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 26, 2023
1af35fd
All logging_names are now None by default to prevent massive spam in
atbenmurray Apr 26, 2023
75771a8
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 26, 2023
d97095a
Merge branch 'dev' into lazy_eval_refactor
atbenmurray Apr 26, 2023
af6cd62
More autofixes
atbenmurray Apr 26, 2023
cd246a8
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 26, 2023
1403d77
Making apply_transform logger_name None by default
atbenmurray Apr 26, 2023
5c4c10a
Added a dictionary test to the compose logging test suite
atbenmurray Apr 26, 2023
5755a65
Updated docstring for _apply_transform
atbenmurray Apr 27, 2023
ca0905a
Removed spurious empty string at the end of one of the logging test case
atbenmurray Apr 27, 2023
e8fdf5a
All spatial transforms now initialize LazyTransform correctly
atbenmurray Apr 27, 2023
70d3517
Adding setters for lazy back in.
atbenmurray Apr 27, 2023
b361a13
Merge branch 'dev' into lazy_eval_refactor
atbenmurray Apr 27, 2023
a6ad51e
Sorting out line length complaint in compose logging tests
atbenmurray Apr 27, 2023
7ec8429
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 27, 2023
305b88e
Fix for old lazy_evaluation name creeping in during merge
atbenmurray Apr 27, 2023
2aa0ada
Autofix
atbenmurray Apr 27, 2023
7b092ca
Merge branch 'Project-MONAI:dev' into lazy_eval_refactor
atbenmurray Apr 27, 2023
62431ff
Using get_logger so that named loggers all get set up correctly
atbenmurray Apr 27, 2023
6cdf4c6
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray Apr 27, 2023
2ea89c1
Fixes for logging; additional compose logging tests; missing imports
atbenmurray Apr 28, 2023
628a7f8
Adding ApplyPending and ApplyPendingd to transforms.rst
atbenmurray Apr 28, 2023
5b34755
Fixed transforms.rst modifications
atbenmurray Apr 28, 2023
d547b81
Refactor of lazy to provide in_order and out_of_order lazy execution
atbenmurray May 2, 2023
03386dc
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 2, 2023
d2fa521
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
eb641c6
Tidied up __all__ issues caused by refactor
atbenmurray May 2, 2023
d7d7cdb
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 2, 2023
29b37bf
Removed can_invert flag from compose compiler policy
atbenmurray May 2, 2023
3194980
Moved to use of a TraceKey.STATUS flag. This can be used to track
atbenmurray May 3, 2023
5ff516d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 3, 2023
eebb3de
Formatting autofixes
atbenmurray May 3, 2023
d966856
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 3, 2023
e39e0a3
Lazy {reorder: lazy_last_nosync} now passes
atbenmurray May 3, 2023
cf58b9f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 3, 2023
883f8c5
Fixed broken call to isinstance in lazy executors
atbenmurray May 4, 2023
e81f359
Autofix
atbenmurray May 4, 2023
db1486b
Resolving lint issue
atbenmurray May 4, 2023
b151607
Missing passing logger_name through to _apply_transform when mapping …
atbenmurray May 4, 2023
24286f1
Autofix
atbenmurray May 4, 2023
43f7bc0
Added missing lazy=None option from integration tests for lazy
atbenmurray May 4, 2023
b217dbe
Autofix
atbenmurray May 4, 2023
ed1d952
Fixing lint issues
atbenmurray May 5, 2023
e7f3a07
Autofixes
atbenmurray May 5, 2023
0751c52
lint fix
atbenmurray May 5, 2023
6114a67
started on the lazy resampling topic page
atbenmurray May 5, 2023
240059b
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 5, 2023
23ff89c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 5, 2023
5e1b81d
Removing unused Sequence import from lazy/executors
atbenmurray May 6, 2023
5dd6113
Renaming ComposeCompiler to ExecuteOptions in test_compose
atbenmurray May 6, 2023
74bed34
Update tests/test_compose.py
atbenmurray May 10, 2023
76e4b85
Execution options is no longer a poltergeist. Refactored is_tensor_in…
atbenmurray May 10, 2023
1c4fa6c
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 10, 2023
8dde40d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 10, 2023
7fd32e2
Removing unnecessary active_key property
atbenmurray May 10, 2023
10d95ce
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 10, 2023
0d6a847
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 10, 2023
1ceeb1a
Fix for multi-sampled data for transforms that aceept *args; refactor…
atbenmurray May 11, 2023
38bdef6
Merge branch 'lazy_eval_refactor' of github.com:atbenmurray/monai int…
atbenmurray May 11, 2023
6e0cfd9
Renaming LazyTrait partially_lazy -> checks_data
atbenmurray May 21, 2023
f2fb06a
Merge branch 'Project-MONAI:dev' into lazy_eval_refactor
atbenmurray May 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions docs/source/transforms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -958,6 +958,17 @@ MRI Transforms
:special-members: __call__


Lazy
^^^^

`ApplyPending`
""""""""""""""

.. autoclass:: ApplyPending
:members:
:special-members: __call__


Utility
^^^^^^^

Expand Down Expand Up @@ -1912,6 +1923,17 @@ Smooth Field (Dict)
:special-members: __call__


Lazy (Dict)
^^^^^^^^^^^

`ApplyPendingd`
"""""""""""""""

.. autoclass:: ApplyPendingd
:members:
:special-members: __call__


Utility (Dict)
^^^^^^^^^^^^^^

Expand Down Expand Up @@ -2211,9 +2233,3 @@ Utilities

.. automodule:: monai.transforms.utils_pytorch_numpy_unification
:members:

Lazy
----
.. automodule:: monai.transforms.lazy
:members:
:imported-members:
2 changes: 1 addition & 1 deletion monai/apps/auto3dseg/data_analyzer.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ def get_all_case_stats(self, key="training", transform_list=None):
manager_list = manager.list()
processes = []
for rank in range(nprocs):
p = tmp_ctx.Process(
p = tmp_ctx.Process( # type: ignore[attr-defined]
target=self._get_all_case_stats, args=(rank, nprocs, manager_list, key, transform_list)
)
processes.append(p)
Expand Down
7 changes: 6 additions & 1 deletion monai/apps/detection/transforms/dictionary.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

from __future__ import annotations

import warnings
from collections.abc import Hashable, Mapping, Sequence
from copy import deepcopy
from typing import Any
Expand Down Expand Up @@ -1326,7 +1327,11 @@ def __init__(
super().__init__(self.image_keys + self.box_keys, prob, max_k, spatial_axes, allow_missing_keys)
self.box_ref_image_keys = ensure_tuple_rep(box_ref_image_keys, len(self.box_keys))

def __call__(self, data: Mapping[Hashable, torch.Tensor]) -> Mapping[Hashable, torch.Tensor]:
def __call__(
self, data: Mapping[Hashable, torch.Tensor], lazy: bool | None = None
) -> Mapping[Hashable, torch.Tensor]:
if lazy is True:
wyli marked this conversation as resolved.
Show resolved Hide resolved
warnings.warn("RandRotateBox90d cannot currently execute lazily; ignoring lazy=True")
self.randomize()
d = dict(data)

Expand Down
8 changes: 6 additions & 2 deletions monai/apps/reconstruction/transforms/dictionary.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

from __future__ import annotations

import warnings
from collections.abc import Hashable, Mapping, Sequence

import numpy as np
Expand Down Expand Up @@ -213,10 +214,10 @@ class ReferenceBasedSpatialCropd(Cropd):
"""

def __init__(self, keys: KeysCollection, ref_key: str, allow_missing_keys: bool = False) -> None:
super().__init__(keys, cropper=None, allow_missing_keys=allow_missing_keys) # type: ignore
super().__init__(keys, cropper=None, allow_missing_keys=allow_missing_keys, lazy=False) # type: ignore
self.ref_key = ref_key

def __call__(self, data: Mapping[Hashable, Tensor]) -> dict[Hashable, Tensor]:
def __call__(self, data: Mapping[Hashable, Tensor], lazy: bool | None = None) -> dict[Hashable, Tensor]:
"""
This transform can support to crop ND spatial (channel-first) data.
It also supports pseudo ND spatial data (e.g., (C,H,W) is a pseudo-3D
Expand All @@ -229,6 +230,9 @@ def __call__(self, data: Mapping[Hashable, Tensor]) -> dict[Hashable, Tensor]:
Returns:
the new data dictionary
"""
if lazy is True:
wyli marked this conversation as resolved.
Show resolved Hide resolved
warnings.warn("ReferenceBasedSpatialCropd cannot currently execute lazily; ignoring lazy=True")

d = dict(data)

# compute roi_size according to self.ref_key
Expand Down
4 changes: 3 additions & 1 deletion monai/transforms/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,9 @@
from .inverse_batch_transform import BatchInverseTransform, Decollated, DecollateD, DecollateDict
from .io.array import SUPPORTED_READERS, LoadImage, SaveImage
from .io.dictionary import LoadImaged, LoadImageD, LoadImageDict, SaveImaged, SaveImageD, SaveImageDict
from .lazy.functional import apply_transforms
from .lazy.array import ApplyPending
from .lazy.dictionary import ApplyPendingd, ApplyPendingD, ApplyPendingDict
from .lazy.functional import apply_pending
from .lazy.utils import combine_transforms, resample
from .meta_utility.dictionary import (
FromMetaTensord,
Expand Down
Loading