Add opening VSI file activity and corresponding data

_config.yml

@@ -33,6 +33,7 @@ module_order:
 - ome_zarr
 - table_file_formats
 - image_inspection_and_presentation
+- correlative_image_rendering
 - volume_slicing
 - projections
 - volume_viewer
@@ -62,13 +63,12 @@ module_order:
 - script_variables
 - string_concat
 - output_saving
+- script_functions
+- script_for_loop
 - batch_processing
+- batch_qc_and_exploration
 - fetching_user_input
-- commenting
 - script_env
-- script_functions
-- script_for_loop
-- correlative_image_rendering
 - deep_learning_run_segmentation
 - manual_segmentation
 - workflow_golgi_per_cell

_includes/batch_qc_and_exploration/batch_explore_ilastik_TODO_fiji_mobie.md

@@ -0,0 +1,2 @@
+- Open Fiji with the [MoBIE update site installed](https://github.com/mobie/mobie-viewer-fiji?tab=readme-ov-file#install).
+- TODO

_includes/batch_qc_and_exploration/batch_explore_ilastik_output.md

@@ -0,0 +1,5 @@
+<h4 id="ilastik_output"><a href="#ilastik_output">Batch explore ilastik output</a></h4>
+
+This activity demonstrates how to efficiently batch explore output generated with the [ilastik]() software.
+
+Please [download the example data](TODO).

_includes/batch_qc_and_exploration/batch_explore_objects_table_fiji_mobie.md

@@ -0,0 +1,17 @@
+- Open Fiji with the [MoBIE update site installed](https://github.com/mobie/mobie-viewer-fiji?tab=readme-ov-file#install).
+- Open `All_Wells_cells.txt` and identify the columns that contain file names, as those will be needed to open the table with MoBIE
+- [ Plugins › MoBIE › Open › Open Table... ]
+   - Table Path: [ Browse ] to `All_Wells_cells.txt`
+   - Image Path Column(s): `FileName_DNA, FileName_PLA`
+   - Labels Path Column(s): `FileName_CellLabels, FileName_PLALabels`
+   - Data Root Folder: [ Browse ] to data folder
+   - SpatialCalibration: Does not matter as everything is in pixel units
+   - Grid: Transformed (Stitched performs better if you have >100 images)
+   - Click [ OK ]
+- The MoBIE UI and BigDataViewer will open allowing you to conveniently browse all data
+- Browsing suggestions:
+   - Cell table: `Color > Color by Column`: `Children_PLA_dots_Count` with `blueWhiteRed` LUT
+      - This will paint cells red that have a lot of PLA spots
+   - Image table: `Color > Color by Column`: `Mean [Object Number]` with `blueWhiteRed` LUT
+      - This will outline images red that have a lot of cells
+

_includes/batch_qc_and_exploration/batch_explore_segmented_images.md

@@ -0,0 +1,7 @@
+<h4 id="segmented_images"><a href="#segmented_images">Batch explore segmented images</a></h4>
+
+This activity will show how to efficiently batch explore potentially many segmented images.
+
+Please [download the example data](https://github.com/tischi/cellprofiler-practical-NeuBIAS-Lisbon-2017/archive/refs/heads/mobie.zip) (including the CellProfiler pipeline used to create the segmentations).
+
+After unzipping, the data for this activity can be found in the `batch_qc_practical` sub-folder.

_includes/batch_qc_and_exploration/batch_explore_segmented_images_and_object_measurements.md

@@ -0,0 +1,7 @@
+<h4 id="segmented_images_and_measurements"><a href="#segmented_images_and_measurements">Batch explore segmented images and object measurements</a></h4>
+
+This activity will show how to efficiently batch explore, potentially many, segmented images and the corresponding object measurements.
+
+Please [download the example data](https://github.com/tischi/cellprofiler-practical-NeuBIAS-Lisbon-2017/archive/refs/heads/mobie.zip) (including the CellProfiler pipeline used to create the segmentations and object measurements).
+
+After unzipping, the data for this activity can be found in the `batch_qc_practical` sub-folder.

_includes/batch_qc_and_exploration/batch_explore_segmented_images_and_object_measurements_fiji_mobie.md

@@ -0,0 +1,13 @@
+- Open Fiji with the [MoBIE update site installed](https://github.com/mobie/mobie-viewer-fiji?tab=readme-ov-file#install).
+- Use MoBIE to inspect segmented nuclei and nuclei measurements
+- [ Plugins › MoBIE › Open › Open Image and Labels... ]
+   - Image URI: [ Browse ] to `Well_1_C0.tif` and then, to open all wells, replace `_1_` by `_.*_` such that it reads `.../Well_.*_C0.tif`
+   - Label Mask URI: [ Browse ] to `Well_1_nuclei_labels.tif` and, as above, replace `_1_` by `_.*_`
+   - Label Mask Table URI: [ Browse ] to `Well_1_nuclei.txt` and, as above, replace `_1_` by `_.*_`
+   - SpatialCalibration: Does not matter here, because the images are only in pixel units anyway
+   - Grid: Transformed (Stitched performs better if you have >100 images)
+   - Click [ OK ]
+- The MoBIE UI and BigDataViewer will open allowing you to conveniently browse all data
+- Browsing suggestions:
+   - Sort the nuclei table by `AreaShape_Area` and browse to the largest and smallest nuclei, as those are likely representing segmentation errors
+   - In the table, use "Color by Column", select a measurement and color it using the blue-white-red LUT 

_includes/batch_qc_and_exploration/batch_explore_segmented_images_fiji_mobie.md

@@ -0,0 +1,18 @@
+- Open Fiji with the [MoBIE update site installed](https://github.com/mobie/mobie-viewer-fiji?tab=readme-ov-file#install).
+- Open `Well_1_C0.tif` and `Well_1_nuclei_labels.tif` in Fiji
+  - Optional: Change the LUT for the labels to `glasbey_inverted`
+- Check for segmentation errors
+- Appreciate that manually inspecting all nuclei segmentations like this would be tedious 
+- Close the two images in Fiji
+- Now use MoBIE to open all images and segmentations
+- [ Plugins › MoBIE › Open › Open Image and Labels... ]
+   - Image URI: [ Browse ] to `Well_1_C0.tif` and then, to open all wells, replace `_1_` by `_.*_` such that it reads `.../Well_.*_C0.tif`
+   - Label Mask URI: [ Browse ] to `Well_1_nuclei_labels.tif` and, same as above, replace `_1_` by `_.*_`
+   - Label Mask Table URI: Leave empty
+   - SpatialCalibration: Does not matter here, because the images are only in pixel units anyway
+   - Grid: Transformed (Stitched performs better if you have >100 images)
+   - Click [ OK ]
+- The MoBIE UI and BigDataViewer will open allowing you to conveniently browse all data
+- Browsing suggestions
+   - [ Ctrl L ] to shuffle the label colors
+   - Use the table to move between images

_includes/batch_qc_and_exploration/batch_explore_segmented_images_napari.py

@@ -0,0 +1,2 @@
+# TODO
+# If someone has a solution please reach out :-)

_includes/big_image_file_formats/create_imaris.md

@@ -0,0 +1,3 @@
+<h4 id="create_imaris"><a href="#create_imaris">Create Imaris files</a></h4>
+
+Create chunked multi-resolution HDF5 based Imaris files.

_includes/big_image_file_formats/create_imaris_imaris_file_converter.md

@@ -0,0 +1,6 @@
+- Install Imaris File Converter
+  - Open this website: https://imaris.oxinst.com/microscopy-imaging-software-free-trial
+  - Scroll all the way to the bottom
+  - Download and install the Imaris File Converter for your OS
+- TODO
+

_includes/image_file_formats/open_lif.md

@@ -1,4 +1,4 @@
-<h4 id="open_lif"><a href="#open_lif">Open LIF image data</a></h4>
+<h4 id="open_lif"><a href="#open_lif">Open Leica LIF image data</a></h4>
 
 - Open a LIF (Leica Image File) file and inspect its pixel and metatdata content
 - Observe the data is presented as one file on disk

_includes/image_file_formats/open_vsi.md

@@ -0,0 +1,9 @@
+<h4 id="open_lif"><a href="#open_lif">Open LIF image data</a></h4>
+
+- Open a LIF (Leica Image File) file and inspect its pixel and metatdata content
+- Observe the data is presented as one file on disk
+- Observe that this one file contains multiple image datasets
+
+##### Data
+
+- [LIF image file](https://github.com/NEUBIAS/training-resources/raw/master/image_data/xy_xyc__two_images.lif)

_includes/image_file_formats/open_vsi_bioio.py

@@ -0,0 +1,48 @@
+# TODO: Change the below code to open the VSI dataset
+
+# %% 
+# Open a VSI image file
+# minimal conda env for this module
+# conda create -n ImageFileFormats python=3.10
+# activate ImageFileFormat
+# pip install bioio bioio-tifffile bioio-lif bioio-czi bioio-ome-tiff bioio-ome-zarr notebook
+# Note: for only dealing with .lif just do pip install bioio bioio-lif
+
+
+# %%
+# Load .lif file
+# - Observe that BioImage chooses the correct reader plugin
+from bioio import BioImage
+image_url = "https://github.com/NEUBIAS/training-resources/raw/master/image_data/xy_xyc__two_images.lif"
+bioimage = BioImage(image_url)
+print(bioimage)
+print(type(bioimage))
+
+# %%
+# Inspect number of images in object
+print(bioimage.scenes)
+
+# %%
+# Inspect both images in the object
+for image in bioimage.scenes:
+    print(f'Image name: {image}')
+    # Select image:
+    bioimage.set_scene(image)
+    # Inspect dimension and shape of image
+    print(f'Image dimension: {bioimage.dims}')
+    print(f'Dimension order is: {bioimage.dims.order}')
+    print(f'Image shape: {bioimage.shape}')
+    # Extract image data (5D)
+    image_data = bioimage.data
+    print(f'Image type: {type(image_data)}')
+    print(f'Image array shape: {image_data.shape}')
+    # Extract specific image part
+    image_data = bioimage.get_image_data('YX')
+    print(f'Image type: {type(image_data)}')
+    print(f'Image array shape: {image_data.shape}')
+    # Read pixel size
+    print(f'Pixel size: {bioimage.physical_pixel_sizes}')
+    # Read metadata
+    print(f'Metadata type: {type(bioimage.metadata)}')
+    print(f'Metadata: {bioimage.metadata}')
+    print('\n')

_includes/image_file_formats/open_vsi_imagejgui.md

@@ -0,0 +1,12 @@
+- Drag and drop the .VSI file on Fiji
+  - What happens?
+  - For us, it seemingly opened the image data wrongly, showing some weird RGB data
+- Now open the VSI file using [Plugins > Bio-Formats > Bio-Formats Importer]
+  - [X] Display metadata
+  - [X] Display OME-XML Metadata
+  - Press [OK]
+  - In the upcoming dialog select the first image "series", which represents the actual image data
+  - Press [OK]
+- The image data and metadata will open and can be inspected
+
+Copy the VSI file to a different location and try again opening it. This should NOT work anymore, because the link to actual data, which is in the ETS file, is now broken

_includes/ome_tiff/inspect_metadata.md

@@ -1,4 +1,4 @@
-<h4 id="inspect_metadata"><a href="inspect_metadata">Inspect OME-TIFF metadata</a></h4>
+<h4 id="inspect_metadata"><a href="#inspect_metadata">Inspect OME-TIFF metadata</a></h4>
 
 - Download the OME-TIFF image file [xy_xyc__two_images.ome.tiff](https://github.com/NEUBIAS/training-resources/raw/master/image_data/xy_xyc__two_images.ome.tiff)
 - Open the OME-XML metadata of this file and try to understand it

_includes/ome_tiff/lif2ometiff.md

@@ -1,4 +1,4 @@
-<h4 id="lif2ometiff"><a href="lif2ometiff">Convert LIF to OME-TIFF</a></h4>
+<h4 id="lif2ometiff"><a href="#lif2ometiff">Convert LIF to OME-TIFF</a></h4>
 
 - Download the Leica image file [image.lif](https://github.com/NEUBIAS/training-resources/raw/master/image_data/xy_xyc__two_images.lif)
 - Open the file

_includes/ome_zarr/ome_zarr_open_java.md

@@ -1,4 +1,4 @@
-<h4 id="ome-zarr-open-java"><a href="ome-zarr-open-java">Open OME-Zarr in Java ecosystem</a></h4>
+<h4 id="ome-zarr-open-java"><a href="#ome-zarr-open-java">Open OME-Zarr in Java ecosystem</a></h4>
 
 Open OME-Zarr data in the Java ecosystem.
 

_layouts/module.html

@@ -289,6 +289,14 @@ <h2>Figure</h2>
   <br>
   <br>
 
+  <div class="module-content">
+    {{ content }}
+  </div>
+
+  <br>
+  <br>
+  <br>
+
   {% if page.multiactivities %}
     <h2 id="activities"><a href="#activities">Activities</a></h2>
     {%- assign idx_0 = 0 -%}
@@ -306,7 +314,7 @@ <h2 id="activities"><a href="#activities">Activities</a></h2>
       <select id="{{ iap }}" name="activityplatformlist" onchange="{{ fun }}">
         {% if site.default-platform == "NONE" %}
           <option value="null-activity-{{ idx_0 }}" id="default-activity-option-{{ idx_0 }}" selected="selected">
-            Select a platform...
+            Select an implementation...
           </option>
         {% else %}
           <option value="{{ site.default-platform | slugify }}-activity-{{ idx_0 }}" id="{{ site.default-platform | slugify }}-option-{{ idx_0 }}" selected="selected">
@@ -372,7 +380,7 @@ <h2>Activities</h2>
       <select id="id_activity_platform" name="activityplatformlist" onchange="change_activity_content_by_platform('id_activity_platform');return false;">
         {% if site.default-platform == "NONE" %}
           <option value="null-activity" id="default-activity-option" selected="selected">
-            Select a platform...
+            Select an implementation...
           </option>
         {% else %}
           <option value="{{ site.default-platform | slugify }}-activity" id="{{ site.default-platform | slugify }}-option" selected="selected">
@@ -431,7 +439,7 @@ <h2>Exercises</h2>
     <select id="id_exercises_platform" name="exercisesplatformlist" onchange="change_exercises_content_by_platform('id_exercises_platform');return false;">
       {%- if site.default-platform == "NONE" -%}
         <option value="null-exercises" id="null-exercises-option" selected="selected">
-          Select a platform...
+          Select an implementation...
         </option>
       {%- else -%}
         <option value="{{ site.default-platform | slugify }}-exercises" id="{{ site.default-platform | slugify }}-exercises-option" selected="selected">
@@ -497,11 +505,6 @@ <h2>Key points</h2>
   {% endif %}
 
 
-  <h2>Explanations</h2>
-  <div class="module-content">
-    {{ content }}
-  </div>
-
   <br>
   <br>
   <br>

_modules/batch_qc_and_exploration.md

@@ -0,0 +1,46 @@
+---
+title: Batch exploration of segmentation results
+layout: module
+tags:
+prerequisites:
+  - "[Segment and measure nuclei shapes](../workflow_segment_2d_nuclei_measure_shape)"
+objectives:
+  - "Use various tools to efficiently inspect segmented images and corresponding object measurements."
+motivation: |
+  Deriving scientifically sound conclusions from microscopy experiments typically requires batch analysis of large image data sets. Once the analysis has been conducted it is critical to visually inspect the results to identify errors and to make scientific discoveries. To do so efficiently requires making oneself familiar with appropriate tools.
+
+concept_map: >
+  graph TD
+    I("Images") --> BA("Batch analysis")
+    BA --> S("Segmentations") 
+    S --> M("Object measurements")
+    I --> Q("Visual QC")
+    S --> Q
+    M --> Q
+
+figure: /figures/batch_qc_and_exploration.png
+figure_legend: "Depiction of a typical bioimage analysis workflow, where batch analysis of many input images yields object segmentation images and measurements, which must be quality controlled and explored for scientific discovery."
+
+multiactivities:
+  - ["batch_qc_and_exploration/batch_explore_segmented_images.md", [["Fiji MoBIE", "batch_qc_and_exploration/batch_explore_segmented_images_fiji_mobie.md"], ["napari (TODO)", "batch_qc_and_exploration/batch_explore_segmented_images_napari.py"]]]
+  - ["batch_qc_and_exploration/batch_explore_segmented_images_and_object_measurements.md", [["Explore Images & Labels & Tables - Fiji MoBIE", "batch_qc_and_exploration/batch_explore_segmented_images_and_object_measurements_fiji_mobie.md"], ["Explore Objects Table - Fiji MoBIE", "batch_qc_and_exploration/batch_explore_objects_table_fiji_mobie.md"]]]
+
+assessment: >
+
+  ### Fill in the blanks
+
+    1. TODO ___ .
+    1. TODO ___ .
+    
+    > ## Solution
+    >   1. TODO
+    >   1. TODO
+    {: .solution}
+
+learn_next:
+  - 
+
+external_links:
+  - "[MoBIE documentation](https://mobie.github.io/)"
+---
+

_modules/big_image_file_formats.md

@@ -21,26 +21,8 @@ figure_legend: "Big image data formats typically support flexible chunking of da
 
 multiactivities:
   - ["big_image_file_formats/lazy_load_tiff_stack.md", [["ImageJ GUI", "big_image_file_formats/lazy_load_tiff_stack_imagej_gui.md"]]]
+  - ["big_image_file_formats/create_imaris.md", [["Imaris File Converter", "big_image_file_formats/create_imaris_imaris_file_converter.md"]]]
 
-activity_preface: |
-  - Inspect the size of a large TIFF file on disk.
-  - Compare the file size to the size of your computer's memory.
-  - Open this file in an image viewer.
-    - Observe that this takes some time.
-    - Observe that your memory fills up.
-  - Open a big image (several GB) from a TIFF file.
-  - Lazy-load a big image (several GB) from a TIFF file.
-    - Observe that TIFF chunking is plane-wise.
-      - Observe that slicing "at an angle" requires loading everything.
-  - Lazy-load from a file format with resolution pyramid and 3-D chunking (e.g. OME-Zarr, BDV/XML/HDF5)
-    - Observe that it opens very quickly (thanks to resolution pyramid and chunking).
-    - Observe that one can swiftly slice the data at all angles (thanks to 3-D chunking).
-    - Inspect the file format to see the resolution pyramid and chunks.
-    - Only load one specific resolution level.
-      - This is useful as most software does not support multi-resolution data for computations.
-
-activities:
-  - ["ImageJ GUI", "big_image_file_formats/activities/big_image_file_formats_imagejgui.md", "markdown"]
 
 exercises:
 
@@ -61,18 +43,20 @@ assessment: >
 learn_next:
 
 external_links:
-
+  - "[Imaris file format](https://imaris.oxinst.com/support/imaris-file-format)"
 ---
 
-### Analogy of big image data with Google Maps
+### Similarities of big microscopy data with Google maps
+
+We can think of the data in Google maps as one very big 2D image. Loading all the data in Google maps into your phone or computer is not possible, because it would take to long and your device would run out of memory. 
 
-When thinking about big image data it is helpful to think about Google Maps. There, you often don't want to see the whole image (planet earth) but rather zoom in on a region (e.g., country). And, if you are zoomed out (e.g., on a continent) you don't care about seeing individual houses in a city; in fact, your computer monitor would not have enough pixels do display all these details while still showing a whole continent, and it would probably not have enough memory to even download all this data.
+Another important aspect is that if you are currently looking at a whole country, it is not useful to load very detailed data about individual houses in one city, because the monitor of your device would not have enough pixels to display this information.
 
-Thus, to offer you a smooth experience, Google Maps **lazy loads** only the part of the world (**chunk**) that you currently look at, at an **resolution level** that is approriate for the number of pixels of your viewing device (phone or computer monitor).
+Thus, to offer you a smooth browsing experience, Google Maps **lazy loads** only the part of the world (**chunk**) that you currently look at, at an **resolution level** that is approriate for the number of pixels of your phone or computer monitor.
 
 ### Chunking
 
-The efficiency with which parts (chunks) of image data can be loaded from your hard disk into your computer memory depends on how the image data is layed out (chunked) on the hard disk. This is a longer, very technical, discussion and what is most optimal probably also depends on the exact storage medium that you are using. Essentially, you want to have the size of your chunks small enough such that your hardware can load one chunk very fast, but you also want the chunks big enough in order to minimise the number of chunks that you need to load. The reason for the latter is that for each chunk your software has to tell your computer "please go and load this chunk", which in itself takes time even if the chunk is very small. Thus, big image data formats typically offer you to choose the chunking such that you can optimise it for your hardware and access patterns.
+The efficiency with which parts (chunks) of image data can be loaded from your hard disk into your computer memory depends on how the image data is layed out (chunked) on the hard disk. This is a longer, very technical, discussion and what is most optimal probably also depends on the exact storage medium that you are using. Essentially, you want to have the size of your chunks small enough such that your hardware can load one chunk very fast, but you also want the chunks big enough in order to minimise the number of chunks that you need to load. The reason for the latter is that for each chunk your software has to tell your computer "please go and load this chunk", which in itself takes time, even if the chunk is very small. Thus, big image data formats typically offer you to choose the chunking such that you can optimise it for your hardware and access patterns.
 
 ### Resolution pyramids