Add confocor3 support (#30)

* upgrade pip * Upload linux wheels * Upload non master to testpypi * Add windows wheel * windows wheel * windows wheel * Windows wheel * Skip existing wheels * Use prebuild docker img for windows * Windows wheels * skip win32 and pypy * FFTW dir in windows build * Compiler flags for HDF5 and fftw * Use fftw only with ICS * Add swig * Mask hdf5 dependency * Mask hdf5 * Fix if close * Hdf5 issue * Static boost linking * Windows boost linking * win boost * remove boost system * static boost * boost exception issue * Use vcpkg on win * Static boost builds * avoid boost * Fix typo * auto link * linking * Build photon hdf by default * Remove openmp dependency * remove boost::filesystem and locale depency * remove boost:locale * Enable language * remove vcpkg * only boost header * remove pip swig cmake boost * add cmake * Use cmake findpython * switch findpzthon3 for findpython * Set DPython_ROOT_DIR in setup.py * Add swig * remove cmake swig * remove cmake submodule * move cmake files from submodule to folder * remove cmake submodule * Add pocketfft * Reduce boost dependency * Remove unused import * Move other peoples code to external * Move other peoples code to external * Move to external * Refactoring * refactor * Add itlib * Restore * Add aglib * Add aglib * Add optimization wrapper * Add optimization wrapper * Add msstl * use itlib flat_set * Cleanup * Cleanup * bump version * separate external include files * cleanup * Add documentation * win issue * Use BUILD_PYTHON_DOCS * Win issue * win build issue * Move external thirdparty code to highlight other projects code external third party code is moved to thirdparty folder. * Remove boost dependency * Static linking of boost remove uneeded boost dependency and link (just in case) boost statically * switch fftw for pocketfftw * switch fftw for pocketfftw * fftw -> pocketfft * Fix typos * remove boost fftw * update license info * remote fftw * cleanup namespace * Add windows wheel Dockerfile * Add wheel:win build * Remove unused config * fix indentation * Fix typo * use cmd.exe for wheel:win * use cmd for whee:win * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Simplify test * fix skip on linux * fix wheel:win pypi upload * Add pwsh to docker * Update .gitlab-ci.yml file * fix * xx * ff * hfd * Update file .gitlab-ci.yml * Update file pocketfft_hdronly.h * Add example writing CLSM data * Windows * new c++ on osx * Dynamic hdf5 lib * fix call * wheel build * add header setter * doc fixes * add windows wheel * docs * docs * add osx:wheel * macos wheel * osx fixes * macos wheel * use modern c++ on osx * wheel osx fix * doc fix * Fix badges and license * Removed TODO * doc fixes * DOC FIX * Also upload development docs * doc build fix * Docs buil * docs building * fix doc * fix doc * Redirect RTD page to docs * Support more py versions * docs deposit * pin boost * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Compile wheel for py311 * Set min macos version * Update .gitlab-ci.yml * Update .gitlab-ci.yml * Specify mmacosx version * Update .gitlab-ci.yml * Cleanup * Cleanup * Add read clsm settings method * Add mask flip * Fix typo * Add option for mask * Change default masking. * Bump version * Pypi 2fa requires tokens * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Replace old np.float types * Replace old np.float types * Fix TTTRMask issue need to flip * Improved documentation * Separate doc from doc-tttrlib env * Fix typo * Fix typo * Fix bixtex issue * Fix doxygen issue * Bump version * Improve docstrings * Additional sphinx modules * Fix typo * Fix typo * Fix issue of deprecated std * Remove missing sphinx-extension * Fix windows build * Fix doc deploy * Fix doc deploy * Fix doc deploy * Fix doc deploy * Fix Pda histogram_function * fix wrong path * CI gitlab github sync * Fix doc issues * Add stubs for doc * Add stubs for doc * Fix wrong path * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml file * Update .gitlab-ci.yml * Update pyproject.toml * Add github action * Remove gitlab webide config * Remove gitlab webide config * Remove github action demo * Add Zeiss Confocor3 reader * Fix set_header issue Assigning a raw pointer when using set_header can cause a double free. Use copy constructor instead. * Add x, y attribute to Correlator.py * Also run Correlator when accessing axis * Add Confocor3 support * Add Confocor3 support * Cleanup * Cleanup * Fix wrong link * Remove old interface * Remove work in progress file * Fix plot Remove old interface Adjust plot range * Fix plot Remove old interface Adjust plot range * Fix typo * Fix file path * Fix plot Increase number of photons for MLE and measure time. * Bump version * Build on all platforms * Update plot * Update recipe Use libboost-devel instead of boost-cpp Use matplotlib-base instead of matplotlib -> no qt5 dependency * Ignore .cache dir * No gpg check on yum * Fix header issue * Build issues on Windows * Do not use git in conda builds * Fix MY_INSTALL_PREFIX * Conda windows build issues * Remove tools submodule * Add tools submodule --------- Co-authored-by: tpeulen <thomas.peulen@posteo.de> Co-authored-by: Thomas-Otavio Peulen <tpeulen@peulen.xyz> Co-authored-by: Administrator <admin@example.com>
Fluorescence-Tools · Jan 31, 2024 · 5a7dc40 · 5a7dc40
1 parent 8b30997
commit 5a7dc40
Show file tree

Hide file tree

Showing 27 changed files with 362 additions and 151 deletions.
diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,7 @@ dist/
 doc/auto_examples
 tmp/
 .vscode
+.cache
 .vs
 .ipynb_checkpoints
 cmake-build-*/

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -2,23 +2,10 @@ variables:
  GIT_SSL_NO_VERIFY: "1"
 
 stages:
-# - sync
  - build
  - test
  - deploy
 
-# github:push:
-# stage: sync
-# image: condaforge/mambaforge
-# tags:
-# - linux
-# before_script:
-# - git submodule update --init --recursive --remote
-# script:
-# - git switch -c ${CI_COMMIT_REF_NAME}
-# - git push --force https://${GITHUB_TOKEN}@github.com/fluorescence-tools/imp.bff
-
-
 # general definitions
 .build: &build
  stage: build
@@ -47,21 +34,21 @@ conda:build:lnx:x86:
  - linux
  - x86_64
 conda:build:lnx:ppc64le:
-  <<: *conda_build_lnx
-  image:
-  entrypoint: [ "/bin/bash", "-i", "-c" ]
-  name: condaforge/linux-anvil-ppc64le
-  tags:
-  - linux
-  - ppc64le
+ <<: *conda_build_lnx
+ image:
+ entrypoint: [ "/bin/bash", "-i", "-c" ]
+ name: condaforge/linux-anvil-ppc64le
+ tags:
+ - linux
+ - ppc64le
 conda:build:lnx:aarch64:
-  <<: *conda_build_lnx
-  image:
-  entrypoint: [ "/bin/bash", "-i", "-c" ]
-  name: condaforge/linux-anvil-aarch64
-  tags:
-  - linux
-  - aarch64
+ <<: *conda_build_lnx
+ image:
+ entrypoint: [ "/bin/bash", "-i", "-c" ]
+ name: condaforge/linux-anvil-aarch64
+ tags:
+ - linux
+ - aarch64
 
 conda:build:osx:
  <<: *conda_build_posix

diff --git a/conda-recipe/bld.bat b/conda-recipe/bld.bat
@@ -1,5 +1,4 @@
 cd %SRC_DIR%
-git submodule update --recursive --init --remote
 
 echo "Build app wrapper"
 :: build app wrapper
@@ -10,20 +9,25 @@ if errorlevel 1 exit 1
 rmdir b2 /s /q
 mkdir b2
 cd b2
-for /f %%A in ('python -c "import platform; print(platform.python_version())"') do set python_version=%%A
-echo Python version: %python_version%
+
+REM Call Python with the --version flag to get the version information
+for /f "tokens=2 delims= " %%v in ('%PYTHON% --version 2^>^&1') do set PYTHON_VERSION=%%v
+
+REM Extract only the numeric part of the version
+for /f "tokens=1-3 delims=." %%a in ("%PYTHON_VERSION%") do set PYTHON_VERSION_NUMERIC=%%a.%%b.%%c
+
+echo Python version: %PYTHON_VERSION_NUMERIC%
 cmake .. -G "NMake Makefiles" ^
  -DCMAKE_INSTALL_PREFIX="%LIBRARY_PREFIX%" ^
  -DCMAKE_PREFIX_PATH="%PREFIX%" ^
  -DBUILD_PYTHON_INTERFACE=ON ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DCMAKE_LIBRARY_OUTPUT_DIRECTORY="%SP_DIR%" ^
  -DCMAKE_SWIG_OUTDIR="%SP_DIR%" ^
- -DPYTHON_VERSION="%python_version%" ^
+ -DPYTHON_VERSION="%PYTHON_VERSION_NUMERIC%" ^
  -DBUILD_LIBRARY=ON
 nmake install
 
-
 :: Add wrappers to path for each Python command line tool
 :: (all files without an extension)
 cd %SRC_DIR%\bin

diff --git a/conda-recipe/build.sh b/conda-recipe/build.sh
@@ -1,4 +1,3 @@
-git submodule update --recursive --init --remote
 mkdir b2 && cd b2
 
 if [[ "${target_platform}" == osx-* ]]; then

diff --git a/conda-recipe/meta.yaml b/conda-recipe/meta.yaml
@@ -13,41 +13,37 @@ build:
  number: 1
  detect_binary_files_with_prefix: True # [not win]
 
+requirements:
 requirements:
  build:
- - python
- - numpy
- - doxygen
  - {{ compiler('c') }}
  - {{ compiler('cxx') }}
  - llvm-openmp # [osx]
  - libgomp # [linux]
  - pkg-config # [not win]
- - swig
- - ninja
  - cmake
+ - ninja
  - make # [linux]
- - typing # [py27]
+ - doxygen
+ - swig
+ - python
+ - numpy
  host:
- - boost-cpp 1.78 # [py > 27]
- - boost-cpp 1.69 # [py27]
+ - libboost-devel
  - hdf5
  - python
  - setuptools
  - typing # [py27]
  - numpy
  run:
  - python
- - tqdm # [py > 27]
- - tqdm <=4.14 # [py27] 
+ - tqdm
  - click
  - click-didyoumean
  - {{ pin_compatible('numpy') }}
  - scikit-image
- - matplotlib
+ - matplotlib-base
  - hdf5
- - typing # [py27]
- - importlib_metadata # [py27]
 
 test:
  imports:
@@ -64,3 +60,4 @@ about:
 extra:
  recipe-maintainers:
  - tpeulen
+ - khemmen
diff --git a/.../correlation/plot_filtered_correlation.py → ...on/not_ready/plot_filtered_correlation.py b/.../correlation/plot_filtered_correlation.py → ...on/not_ready/plot_filtered_correlation.py
diff --git a/examples/correlation/plot_confocor3_two_ch_correlation.py b/examples/correlation/plot_confocor3_two_ch_correlation.py
@@ -0,0 +1,121 @@
+"""
+=======================================
+Confocor3 two channel cross-correlation
+=======================================
+
+The raw FCS data format of the Zeiss Confocor3 is relatively simple.
+Zeiss Confocor3 raw files store time-difference between photons.
+A relatively small header is followed by a set of 32-bit integers that
+contain the time difference to the previous registered photon. Photons
+registered by different channels are stored in separate files.
+
+In this example two raw files of a Confocor3 are read and merged into
+a single photon stream. Next, the merged photon stream is used to compute
+the cross-correlation between the two channels.
+
+"""
+import pathlib
+import numpy as np
+
+import pylab as plt
+import tttrlib
+
+#%%
+# Reading data
+# ------------
+# The photon data registered by different detectors are saved in separate files.
+# Read the data of all channels that should be correlated into separate containers.
+fns = [str(p) for p in pathlib.Path('../../tttr-data/cz/fcs').glob('5a6ce6a348a08e3da9f7c0ab4ee0ce94_R1_P1_K1_Ch*.raw')]
+tttr_data = [tttrlib.TTTR(fn, 'CZ-RAW') for fn in fns]
+
+#%%
+# We combine the data in different files into a single TTTR container using the header
+# of first file as template.
+header = tttr_data[0].header
+channels = [t.routing_channels[0] for t in tttr_data]
+print("Used channels:", channels)
+
+#%%
+# You can check the count rates of the channels using the macro time resolution contained
+# in the header
+macro_time_resolution = header.macro_time_resolution
+count_rates = [len(t) / (t.macro_times[-1] * macro_time_resolution) for t in tttr_data]
+print("Count rates:", count_rates)
+
+#%%
+# Now we merge the data of the two detectors in a single container. The
+# marco times need to be sorted first.
+macro_times = np.concatenate([t.macro_times for t in tttr_data])
+routing_channels = np.concatenate([t.routing_channels for t in tttr_data])
+sorted_indices = np.argsort(macro_times)
+
+#%%
+# Using the sorted macro times we sort the routing channel numbers
+# and the macro times.
+routing_channels = routing_channels[sorted_indices]
+macro_times = macro_times[sorted_indices]
+
+#%%
+# Note: no micro time and no event type in the raw Confocor3 format.
+# Thus, we use ones for the micro time and the event type.
+micro_times = np.ones_like(macro_times, dtype=np.uint16)
+event_types = np.ones_like(macro_times, dtype=np.int8)
+
+#%%
+# Using the merged marcro times and channel numbers, we create a new
+# TTTR container.
+tttr_merged = tttrlib.TTTR()
+tttr_merged.set_header(header)
+tttr_merged.append_events(
+ macro_times=macro_times,
+ micro_times=micro_times,
+ routing_channels=routing_channels,
+ event_types=event_types
+)
+
+#%%
+# The container can be used for standard analysis, e.g., correlations.
+settings = {
+ "n_bins": 9, # n_bins and n_casc defines the settings of the multi-tau
+ "n_casc": 19, # correlation algorithm
+}
+
+# Create correlator
+# Caution: x-axis in units of macro time counter
+# tttrlib.Correlator is unaware of the calibration in the TTTR object
+correlator = tttrlib.Correlator(
+ channels=([1], [2]),
+ tttr=tttr_merged,
+ **settings
+)
+plt.semilogx(
+ correlator.x_axis,
+ correlator.correlation,
+ label="Corr(1,2)"
+)
+
+correlator = tttrlib.Correlator(
+ channels=([1], [1]),
+ tttr=tttr_merged,
+ **settings
+)
+plt.semilogx(
+ correlator.x_axis,
+ correlator.correlation,
+ label="Corr(1,1)"
+)
+
+correlator = tttrlib.Correlator(
+ channels=([2], [2]),
+ tttr=tttr_merged,
+ **settings
+)
+plt.semilogx(
+ correlator.x_axis,
+ correlator.correlation,
+ label="Corr(2,2)"
+)
+
+plt.ylim(1.032, 1.050)
+plt.show()
+
diff --git a/examples/correlation/plot_correlation_cr_filter.py b/examples/correlation/plot_correlation_cr_filter.py
@@ -42,7 +42,6 @@
 # ``n_bins`` and ``n_casc`` define the settings of the multi-tau correlation steps.
 # If ``make_fine`` is set to false the micro time is not used for correlation.
 settings = {
- "method": "default",
  "n_bins": 3,
  "n_casc": 27,
  "make_fine": False

diff --git a/examples/correlation/plot_full_correlation.py b/examples/correlation/plot_full_correlation.py
@@ -58,7 +58,6 @@
 # times as time information, the default value of parameters ``make_fine``
 # when creating a new Correlator needs to by modified:
 full_corr_settings = {
- "method": 'default',
  "n_casc": 37, # n_bins and n_casc defines the settings of the multi-tau
  "n_bins": 7, # correlation algorithm
  "make_fine": True # Use the microtime information (also called "fine" correlation)
@@ -103,5 +102,6 @@
 ax.set_xlabel('corr. time (ms)')
 ax.set_ylabel('Correlation Amplitude')
 ax.legend()
+ax.set_ylim(0, 12) # Set y-axis range to 0-12
 
 plt.show()
diff --git a/examples/correlation/plot_gated_correlation.py b/examples/correlation/plot_gated_correlation.py
diff --git a/examples/correlation/plot_normal_correlation.py b/examples/correlation/plot_normal_correlation.py
@@ -39,7 +39,6 @@
 
 # Correlator settings, if the same settings are used repeatedly it is useful to define them once
 settings = {
- "method": "default",
  "n_bins": 7, # n_bins and n_casc defines the settings of the multi-tau
  "n_casc": 27, # correlation algorithm
  "make_fine": False # Do not use the microtime information
@@ -64,8 +63,8 @@
 correlator.set_events(t1, w1, t2, w2)
 
 # scale the x-axis to have units in milliseconds (common unit in FCS)
-x = correlator.curve.x * (data.header.macro_time_resolution)
-y = correlator.curve.y
+x = correlator.x * data.header.macro_time_resolution
+y = correlator.y
 
 plt.semilogx(x, y, label="Gp/Gs")
 
@@ -82,8 +81,8 @@
 
 # no need to scale axis - correlator aware of macro time units
 plt.semilogx(
- correlator.curve.x,
- correlator.curve.y,
+ correlator.x,
+ correlator.y,
  label="Gp,Gs/Rp,Rs"
 )
 
@@ -95,8 +94,8 @@
 )
 
 plt.semilogx(
- correlator.x_axis,
- correlator.correlation,
+ correlator.x,
+ correlator.y,
  label="pR,sR"
 )
 
@@ -108,8 +107,8 @@
 )
 
 plt.semilogx(
- correlator.x_axis,
- correlator.correlation,
+ correlator.x,
+ correlator.y,
  label="pRsR,pGsG"
 )
 

diff --git a/examples/correlation/plot_sliced_correlation.py b/examples/correlation/plot_sliced_correlation.py
@@ -44,7 +44,7 @@
 #%%
 # Next, we plan to split the TTTR data into separate data chunks. Here, chunk
 # the data into at least 5 seconds long pieces. The method ``get_ranges_by_time_window``
-# returns an one-dimensional array with the beginning and the end index of each
+# returns a one-dimensional array with the beginning and the end index of each
 # chunk. Later, we will use these start/stop indices to define TTTR slices
 # that will be correlated. To slice the data into time windows we use the
 # macro time calibration we computed previously. Note, the last chunk can