Skip to content

Commit

Permalink
Fix ImageStitching plugin hanging in multi-node MPI runs
Browse files Browse the repository at this point in the history
In multi-node MPI runs via `savu_launcher.sh` (and possible other cases
involving MPI) which included the `ImageStitching` plugin, it was
observed that the `ImageStitching` plugin never completed its processing
and instead was "hanging" indefinitely.

The reason for this was found to be related to the calculation of
statistics. There was a check for 4D data in the calculation of
statistics, and if the data was labelled by Savu as 4D, to then perform
some extra statistics calculations on the 4D data assuming that the 4D
data contained volume data (most likely, assuming that the 4D data was a
group of multiple 3D volumes).

Savu labels the output of `LfovLoader` as being 4D, to mean either "two
stacks of projections to stitch together" or "two stacks of sinograms to
stitch together". `ImageStitching` takes this "4D" output from
`LfovLoader` and then converts it to 3D.

In the `Statistics` class, the `setup_4d()` method applies some logic to
determine if the plugin outputs 4D slices or not. The logic as it is
currently concludes that `ImageStitching` outputs 4D slices, which is
incorrect. Statistic calculations based on the assumption that the
slices are volume data were then attempted to be applied to non-volume
data.

Somewhere in the volume statistics calculations, at least one of the MPI
processes "hangs", and the other MPI processes which encountered no
problem (for some unknown reason) are left waiting indefinitely at the
MPI barrier in `PluginDriver._run_plugin_instances()`.

Regarding the "fix" for this: the reasoning behind why the logic for
determining if the output is 4D slices is set up this particular way is
not known. So, there are some reservations in "correcting" this for the
case of `ImageStitching`, out of concern for potentially breaking other
cases where 4D slices are outputted/relevant.

Therefore, the strategy to fix this situation is to instead:
- query the pattern(s) of the input dataset(s)
- check if any of them are the patterns that `ImageStitching` works on,
  which are considered 4D by Savu but are *not* volumes
- if any of the patterns of the input datasets match the patterns that
  are connected to stitching images, do *not* perform volume statistics
  calculations on the output slices
  • Loading branch information
yousefmoazzam committed Oct 16, 2023
1 parent 6b1fe63 commit a9fe7c9
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion savu/data/stats/statistics.py
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,14 @@ def set_slice_stats(self, my_slice, base_slice=None, pad=True):
if slice_stats is not None:
for key, value in slice_stats.items():
self.stats[key].append(value)
if self._4d:
in_datasets = self.plugin.get_in_datasets()
in_pData = [d._get_plugin_data() for d in in_datasets]
patterns = [pData.get_pattern_name() for pData in in_pData]
EXCLUDED_4D_PATTERNS = ["PROJECTION_STACK", "SINOGRAM_STACK"]
any_stitch_patterns = list(
map(lambda pattern: pattern in EXCLUDED_4D_PATTERNS, patterns)
)
if self._4d and not any(any_stitch_patterns):
if sum(self.stats["data_points"]) >= self._volume_total_points:
self.set_volume_stats()
else:
Expand Down

0 comments on commit a9fe7c9

Please sign in to comment.