Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix
ImageStitching
plugin hanging in multi-node MPI runs
In multi-node MPI runs via `savu_launcher.sh` (and possible other cases involving MPI) which included the `ImageStitching` plugin, it was observed that the `ImageStitching` plugin never completed its processing and instead was "hanging" indefinitely. The reason for this was found to be related to the calculation of statistics. There was a check for 4D data in the calculation of statistics, and if the data was labelled by Savu as 4D, to then perform some extra statistics calculations on the 4D data assuming that the 4D data contained volume data (most likely, assuming that the 4D data was a group of multiple 3D volumes). Savu labels the output of `LfovLoader` as being 4D, to mean either "two stacks of projections to stitch together" or "two stacks of sinograms to stitch together". `ImageStitching` takes this "4D" output from `LfovLoader` and then converts it to 3D. In the `Statistics` class, the `setup_4d()` method applies some logic to determine if the plugin outputs 4D slices or not. The logic as it is currently concludes that `ImageStitching` outputs 4D slices, which is incorrect. Statistic calculations based on the assumption that the slices are volume data were then attempted to be applied to non-volume data. Somewhere in the volume statistics calculations, at least one of the MPI processes "hangs", and the other MPI processes which encountered no problem (for some unknown reason) are left waiting indefinitely at the MPI barrier in `PluginDriver._run_plugin_instances()`. Regarding the "fix" for this: the reasoning behind why the logic for determining if the output is 4D slices is set up this particular way is not known. So, there are some reservations in "correcting" this for the case of `ImageStitching`, out of concern for potentially breaking other cases where 4D slices are outputted/relevant. Therefore, the strategy to fix this situation is to instead: - query the pattern(s) of the input dataset(s) - check if any of them are the patterns that `ImageStitching` works on, which are considered 4D by Savu but are *not* volumes - if any of the patterns of the input datasets match the patterns that are connected to stitching images, do *not* perform volume statistics calculations on the output slices
- Loading branch information