Merge pull request #212 from rapidsai/branch-22.02

[RELEASE] cucim v22.02

.gitignore

@@ -153,3 +153,5 @@ conda-bld
 # Large Images
 *.svs
 
+# Custom debug environment setup script for VS Code (used by scripts/debug_python)
+/scripts/debug_env.sh

3rdparty/LICENSE.taskflow

@@ -0,0 +1,21 @@
+TASKFLOW MIT LICENSE
+
+Copyright (c) 2018-2021 Dr. Tsung-Wei Huang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

CHANGELOG.md

@@ -1,6 +1,145 @@
-# cuCIM 21.12.00 (Date TBD)
+# cuCIM 22.02.00 (2 Feb 2022)
 
-Please see https://github.com/rapidsai/cucim/releases/tag/v21.12.00a for the latest changes to this development branch.
+## 🚨 Beaking Changes
+
+- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
+
+## 📖 Documentation
+
+- PyPI v21.12.00 elease ([#182](https://github.com/rapidsai/cucim/pull/182)) [@gigony](https://github.com/gigony)
+
+## 🚀 New Featues
+
+- Allow CuPy 10 ([#195](https://github.com/rapidsai/cucim/pull/195)) [@jakikham](https://github.com/jakikham)
+- Suppot multi-theads and batch, and suppot nvJPEG fo JPEG-compessed images ([#191](https://github.com/rapidsai/cucim/pull/191)) [@gigony](https://github.com/gigony)
+- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
+
+## 🛠️ Impovements
+
+- Upgade Numpy to 1.19 fo Python 3.9 suppot ([#203](https://github.com/rapidsai/cucim/pull/203)) [@Ethyling](https://github.com/Ethyling)
+- Allow installation with CuPy 10 ([#197](https://github.com/rapidsai/cucim/pull/197)) [@glee77](https://github.com/glee77)
+- Upgade Numpy to 1.18 fo Python 3.9 suppot ([#196](https://github.com/rapidsai/cucim/pull/196)) [@Ethyling](https://github.com/Ethyling)
+- Add missing impots tests ([#183](https://github.com/rapidsai/cucim/pull/183)) [@Ethyling](https://github.com/Ethyling)
+
+# cuCIM 21.12.00 (9 Dec 2021)
+
+## 🚀 New Features
+
+1. Support Aperio SVS with CPU LZW and jpeg2k decoder ([#141](https://github.com/rapidsai/cucim/pull/141)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+2. Add NVTX support for performance analysis ([#144](https://github.com/rapidsai/cucim/pull/144)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+3. Normalize operation ([#150](https://github.com/rapidsai/cucim/pull/150)) [[@shekhardw](https://github.com/shekhardw)](https://github.com/shekhardw](https://github.com/shekhardw))
+
+### 1. Support Aperio SVS (.svs)
+
+cuCIM now supports [Aperio SVS format](https://openslide.org/formats/aperio/) with help of [OpenJpeg](https://www.openjpeg.org/) for decoding jpeg2k-compressed data.
+
+Please check [this notebook](https://nbviewer.org/github/rapidsai/cucim/blob/branch-21.12/notebooks/Supporting_Aperio_SVS_Format.ipynb) to see how to use the feature.
+
+#### Unaligned Case (`per_process`, JPEG-compressed SVS file)
+
+![image](https://user-images.githubusercontent.com/1928522/141350490-06fdd8cb-5be2-42e4-9774-c7b76fab6f9a.png)
+
+#### Unaligned Case (`per_process`, JPEG2000 RGB-compressed SVS file)
+
+![image](https://user-images.githubusercontent.com/1928522/141093324-574b532e-ad42-4d61-8473-4c3e07e3feae.png)
+
+#### Unaligned Case (`per_process`, JPEG2000 YCbCr-compressed SVS file)
+
+![image](https://user-images.githubusercontent.com/1928522/141093381-8ab0161d-1b17-4e80-a680-86abfbf2fa65.png)
+
+The detailed data is available [here](https://docs.google.com/spreadsheets/d/15D1EqNI_E9x_S8i3kJLwBxMcEmwk8SafW0WryMrAm6A/edit#gid=369408723).
+
+### 2. Add NVTX support for performance analysis
+
+Important methods in cuCIM are instrumented with [NVTX](https://docs.nvidia.com/gameworks/index.html#gameworkslibrary/nvtx/nvidia_tools_extension_library_nvtx.htm) so can see performance bottlenecks easily with [NSight systems](https://developer.nvidia.com/nsight-systems).
+
+Tracing can be enabled through config file or environment variable or through API and less than 1% performance overheads in normal execution.
+
+#### Enabling Tracing
+##### Through `.cucim.json` file
+
+```json
+{
+        "profiler" : { "trace": true }
+}
+```
+
+##### Through Environment variable
+
+```bash
+CUCIM_TRACE=1 python
+```
+
+##### Through API
+
+```python
+from cucim import CuImage
+
+CuImage.profiler(trace=True)
+#or
+CuImage.profiler().trace(True)
+
+CuImage.profiler().config
+# {'trace': True}
+CuImage.profiler().trace()
+# True
+CuImage.is_trace_enabled # this is simpler method.
+# True
+```
+
+#### Profiling with NVIDIA Nsight Systems
+
+```bash
+nsys profile -f true -t cuda,nvtx,osrt -s cpu -x true --trace-fork-before-exec true -o my_profile `which python` benchmark.py
+# can add `--stats true`
+```
+
+Then, execute `nsight-sys` to open the profile results (my_profile.qdrep).
+
+![image](https://user-images.githubusercontent.com/1928522/141221297-2ff5224b-e99b-4fe6-af7d-69452141d71d.png)
+
+With this feature, a bug in cuCIM [v21.10.01](https://github.com/rapidsai/cucim/wiki/release_notes_v21.10.01) (thread contention in Cache) was found and fixed ([#145](https://github.com/rapidsai/cucim/pull/145)).
+
+### 3. Normalize operation
+
+CUDA-based normalization operation is added. Normalization supports the following types.
+
+1. Simple range based normalization
+2. Arctangent based normalization
+
+Arctangent-based normalization helps to stretch lower intensity pixels in the image slightly better than range-based normalization. If you look at its [graph](https://mathworld.wolfram.com/InverseTangent.html), there is a huge variation at certain lower intensities, but as intensities become higher, the curve becomes flatter. This helps in isolating regions like lungs (and regions within lungs) more efficiently. There can be separate use cases depending on the modality and the application.
+
+Please check the [test cases](https://github.com/rapidsai/cucim/blob/branch-21.12/python/cucim/src/cucim/core/operations/intensity/tests/test_normalize.py) to see how you can use the operation.
+
+
+## 🐛 Bug Fixes
+
+- Load libcufile.so with RTLD_NODELETE flag ([#177](https://github.com/rapidsai/cucim/pull/177)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Remove rmm/nvcc dependencies to fix cudaErrorUnsupportedPtxVersion error ([#175](https://github.com/rapidsai/cucim/pull/175)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Do not compile code with nvcc if no CUDA kernel exists ([#171](https://github.com/rapidsai/cucim/pull/171)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Fix a segmentation fault due to unloaded libcufile ([#158](https://github.com/rapidsai/cucim/pull/158)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Fix thread contention in Cache ([#145](https://github.com/rapidsai/cucim/pull/145)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Build with NumPy 1.17 ([#148](https://github.com/rapidsai/cucim/pull/148)) [[@jakirkham](https://github.com/jakirkham)](https://github.com/jakirkham](https://github.com/jakirkham))
+
+## 📖 Documentation
+
+- Add Jupyter notebook for SVS Support ([#147](https://github.com/rapidsai/cucim/pull/147)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Update change log for v21.10.01 ([#142](https://github.com/rapidsai/cucim/pull/142)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- update docs theme to pydata-sphinx-theme ([#138](https://github.com/rapidsai/cucim/pull/138)) [[@quasiben](https://github.com/quasiben)](https://github.com/quasiben](https://github.com/quasiben))
+- Update Github links in README.md through script ([#132](https://github.com/rapidsai/cucim/pull/132)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Fix GDS link in Jupyter notebook ([#131](https://github.com/rapidsai/cucim/pull/131)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Update notebook for the interoperability with DALI ([#127](https://github.com/rapidsai/cucim/pull/127)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+
+
+
+## 🛠️ Improvements
+
+- Update `conda` recipes for Enhanced Compatibility effort by ([#164](https://github.com/rapidsai/cucim/pull/164)) [[@ajschmidt8](https://github.com/ajschmidt8)](https://github.com/ajschmidt8](https://github.com/ajschmidt8))
+- Fix Changelog Merge Conflicts for `branch-21.12` ([#156](https://github.com/rapidsai/cucim/pull/156)) [[@ajschmidt8](https://github.com/ajschmidt8)](https://github.com/ajschmidt8](https://github.com/ajschmidt8))
+- Add cucim.kit.cumed plugin with skeleton ([#129](https://github.com/rapidsai/cucim/pull/129)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Update initial cpp unittests ([#128](https://github.com/rapidsai/cucim/pull/128)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Optimize zoom out implementation with separate padding kernel ([#125](https://github.com/rapidsai/cucim/pull/125)) [[@chirayuG-nvidia](https://github.com/chirayuG-nvidia)](https://github.com/chirayuG-nvidia](https://github.com/chirayuG-nvidia))
+- Do not force install linux-64 version of openslide-python ([#124](https://github.com/rapidsai/cucim/pull/124)) [[@Ethyling](https://github.com/Ethyling)](https://github.com/Ethyling](https://github.com/Ethyling))
 
 # cuCIM 21.10.00 (7 Oct 2021)
 

CMakeLists.txt

@@ -126,6 +126,7 @@ superbuild_depend(json)
 superbuild_depend(libcuckoo)
 superbuild_depend(boost-header-only)
 superbuild_depend(nvtx3)
+superbuild_depend(taskflow)
 
 ################################################################################
 # Define some names

LICENSE-3rdparty.md

@@ -85,7 +85,7 @@ DLPack
   - https://github.com/dmlc/dlpack/blob/main/LICENSE
 - Copyright: DLPack Contributors
 
-NVIDIA CUDA TOOLKIT (including libcufile)
+NVIDIA CUDA TOOLKIT (including libcufile/libnvjpeg)
 - License: NVIDIA License
   - https://docs.nvidia.com/cuda/pdf/EULA.pdf
 - Copyright: NVIDIA Corporation
@@ -269,3 +269,9 @@ NVTX (NVIDIA Tool Extension Library)
   - https://raw.githubusercontent.com/NVIDIA/NVTX/release-v3/LICENSE.txt
 - Copyright: NVIDIA Corporation
 - Usage: Support for profiling with NVIDIA Nsight Systems
+
+Taskflow
+- License: MIT License
+  - https://github.com/taskflow/taskflow
+- Copyright: Dr. Tsung-Wei Huang
+- Usage: Threadpool implementation for batch processing.

README.md

@@ -16,21 +16,30 @@
 
 ### Conda
 
-#### Conda (stable)
+#### [Conda (stable)](https://anaconda.org/rapidsai/cucim)
 
 > conda create -n cucim -c rapidsai -c conda-forge cucim cudatoolkit=`<CUDA version>`
 
 `<CUDA version>` should be 11.0+ (e.g., `11.0`, `11.2`, etc.)
 
-#### Conda (nightlies)
+#### [Conda (nightlies)](https://anaconda.org/rapidsai-nightly/cucim)
 
 > conda create -n cucim -c rapidsai-nightly -c conda-forge cucim cudatoolkit=`<CUDA version>`
 
 `<CUDA version>` should be 11.0+ (e.g., `11.0`, `11.2`, etc)
 
+### [PyPI](https://pypi.org/project/cucim/)
+
+```bash
+pip install cucim
+
+# Install dependencies for `cucim.skimage` (assuming that CUDA 11.0 is used for CuPy)
+pip install scipy scikit-image cupy-cuda110
+```
+
 ### Notebooks
 
-Please check out our [Welcome](notebooks/Welcome.ipynb) notebook ([NBViewer](https://nbviewer.jupyter.org/github/rapidsai/cucim/blob/branch-21.12/notebooks/Welcome.ipynb))
+Please check out our [Welcome](notebooks/Welcome.ipynb) notebook ([NBViewer](https://nbviewer.jupyter.org/github/rapidsai/cucim/blob/branch-22.02/notebooks/Welcome.ipynb))
 
 #### Downloading sample images
 

VERSION

@@ -1 +1 @@
-21.12.00
+22.02.00

benchmarks/main.cpp

@@ -65,8 +65,7 @@ static void test_cucim(benchmark::State& state)
 
         cucim::CuImage image = cucim::CuImage(input_path.c_str());
         cucim::CuImage region =
-            image.read_region({ request_location[0], request_location[1] }, { state.range(0), state.range(0) }, 0,
-                              cucim::DimIndices{}, "cpu", nullptr, "");
+            image.read_region({ request_location[0], request_location[1] }, { state.range(0), state.range(0) }, 0);
     }
 }
 

benchmarks/skimage/cucim_exposure_bench.py

@@ -112,13 +112,13 @@ def set_args(self, dtype):
 for shape in [(512, 512), (3840, 2160), (3840, 2160, 3), (192, 192, 192)]:
     ndim = len(shape)
 
-    multichannel = shape[-1] in [3, 4]
+    channel_axis = -1 if shape[-1] in [3, 4] else None
 
     B = MatchHistogramBench(
         function_name="match_histograms",
         shape=shape,
         dtypes=dtypes,
-        fixed_kwargs=dict(multichannel=multichannel),
+        fixed_kwargs=dict(channel_axis=channel_axis),
         var_kwargs=dict(),
         module_cpu=skimage.exposure,
         module_gpu=cucim.skimage.exposure,

benchmarks/skimage/cucim_feature_bench.py

@@ -74,7 +74,7 @@ def set_args(self, dtype):
             continue
 
         if function_name == "multiscale_basic_features":
-            fixed_kwargs["multichannel"] = shape[-1] == 3
+            fixed_kwargs["channel_axis"] = -1 if shape[-1] == 3 else None
             if ndim == 3 and shape[-1] != 3:
                 # Omit texture=True case to avoid excessive GPU memory usage
                 var_kwargs["texture"] = [False]

benchmarks/skimage/cucim_filters_bench.py

@@ -128,15 +128,17 @@
             var_kwargs["frequency"] = [f for f in var_kwargs["frequency"] if f >= 0.1]
 
         if function_name == "median":
-            selems = []
+            footprints = []
             ndim = len(shape)
-            selem_sizes = [3, 5, 7, 9] if ndim == 2 else [3, 5, 7]
-            for selem_size in [3, 5, 7, 9]:
-                selems.append(np.ones((selem_size,) * ndim, dtype=bool))
-            var_kwargs["selem"] = selems
+            footprint_sizes = [3, 5, 7, 9] if ndim == 2 else [3, 5, 7]
+            for footprint_size in [3, 5, 7, 9]:
+                footprints.append(
+                    np.ones((footprint_size,) * ndim, dtype=bool)
+                )
+            var_kwargs["footprint"] = footprints
 
         if function_name in ["gaussian", "unsharp_mask"]:
-            fixed_kwargs["multichannel"] = True if shape[-1] == 3 else False
+            fixed_kwargs["channel_axis"] = -1 if shape[-1] == 3 else None
 
         B = ImageBench(
             function_name=function_name,

benchmarks/skimage/cucim_measure_bench.py

@@ -246,15 +246,17 @@ def set_args(self, dtype):
             fixed_kwargs["dst"] = (shape[0] - 32, shape[1] + 9)
 
         if function_name == "median":
-            selems = []
+            footprints = []
             ndim = len(shape)
-            selem_sizes = [3, 5, 7, 9] if ndim == 2 else [3, 5, 7]
-            for selem_size in [3, 5, 7, 9]:
-                selems.append(np.ones((selem_sizes,) * ndim, dtype=bool))
-            var_kwargs["selem"] = selems
+            footprint_sizes = [3, 5, 7, 9] if ndim == 2 else [3, 5, 7]
+            for footprint_size in [3, 5, 7, 9]:
+                footprints.append(
+                    np.ones((footprint_sizes,) * ndim, dtype=bool)
+                )
+            var_kwargs["footprint"] = footprints
 
         if function_name in ["gaussian", "unsharp_mask"]:
-            fixed_kwargs["multichannel"] = True if shape[-1] == 3 else False
+            fixed_kwargs["channel_axis"] = -1 if shape[-1] == 3 else None
 
         B = FiltersBench(
             function_name=function_name,

benchmarks/skimage/cucim_metrics_bench.py

@@ -68,7 +68,7 @@ def set_args(self, dtype):
             continue
 
         if function_name in ["structural_similarity"]:
-            fixed_kwargs["multichannel"] = True if shape[-1] == 3 else False
+            fixed_kwargs["channel_axis"] = -1 if shape[-1] == 3 else None
 
         B = MetricsBench(
             function_name=function_name,