193 accelerate piv (#196)

* WIP #193 * fix #194 * Refactor coordinate assignments and projections #193 * Implement dual PIV engines: OpenPIV and FF-PIV #193 Refactored the PIV implementation to support both 'OpenPIV' and 'FF-PIV' engines, allowing selection via the 'engine' parameter. * Refactor PIV computation and documentation. #193 * robustness increase for cases where last frames are not seekable #193 * bug fix with time selection with missing frames #193 * chunked frame reading WIP #193 * Handle invalid frames and update frame count logic #193 * Minor updates in documentation for installation instructions * Remove debug prints and commented code #193 * Small bug fixes and bump version 0.7.0 #193 * change order of import cli #193 * Refactor and clean up code for code coverage #193
localdevices · Dec 10, 2024 · 26dbcb7 · 26dbcb7
1 parent 83019dc
commit 26dbcb7
Show file tree

Hide file tree

Showing 27 changed files with 1,513 additions and 827 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,16 @@
+## [0.7.0] - 2024-12-10
+### Added
+`get_piv` now uses several engines, `engine="numba"` is a lot fastr
+### Changed
+Reading frames is now a lot more efficient as they are read in bulks (20 by default). As a result, very large videos
+can be processed efficiently.
+### Deprecated
+openpiv is still default, but may become deprecated in future versions.
+### Removed
+### Fixed
+### Security
+
+
 ## [0.6.1] - 2024-09-26
 ### Added
 ### Changed

diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 <figure>
-    <img src="https://raw.githubusercontent.com/localdevices/pyorc/main/docs/_static/orc_logo_color.svg" 
+    <img src="https://raw.githubusercontent.com/localdevices/pyorc/main/docs/_static/orc_logo_color.svg"
 width=100 align="right">
 </figure>
 <br>
@@ -13,12 +13,12 @@ width=100 align="right">
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/localdevices/pyorc.git/main?labpath=examples)
 [![License](https://img.shields.io/github/license/localdevices/pyorc?style=flat)](https://github.com/localdevices/pyorc/blob/main/LICENSE)
 
-**pyorc**, short for "pyOpenRiverCam" is a fully Open Source library for performing image-based river flow analysis. It is the underlying library for 
+**pyorc**, short for "pyOpenRiverCam" is a fully Open Source library for performing image-based river flow analysis. It is the underlying library for
 computations on the fully open software stack OpenRiverCam. **pyorc** can only be successful if the underlying methods
 are made available openly for all. Currently **pyorc** implements Large-scale Particle Image Velocimetry (LSPIV) based
-flow analysis using the OpenPIV library and reprojections and image pre-processing with OpenCV. We wish to extend this 
+flow analysis using the OpenPIV library and reprojections and image pre-processing with OpenCV. We wish to extend this
 to Large-scale Particle Tracking Velocimetry (LSPTV) and Space-Time Image Velocimetry (STIV) for conditions that are less favourable for LSPIV using open
-libraries or extensions to this code. 
+libraries or extensions to this code.
 
 ![example_image](https://raw.githubusercontent.com/localdevices/pyorc/main/docs/ngwerere.jpg)
 Image: Example of pyorc velocimetry over Ngwerere river at the Zambezi Road crossing - Lusaka, Zambia.
@@ -29,15 +29,15 @@ Current capabilities are:
 * Discharge estimation over provided cross-section
 * Plotting of velocimetry results and cross-section flows in camera, geographical and orthoprojected perspectives.
 
-We use the well-known **xarray** data models and computation pipelines (with dask) throughout the entire library to 
-guarantee an easy interoperability with other tools and methods, and allow for lazy computing. 
+We use the well-known **xarray** data models and computation pipelines (with dask) throughout the entire library to
+guarantee an easy interoperability with other tools and methods, and allow for lazy computing.
 
 We are seeking funding for the following frequently requested functionalities:
-* A command-line interface for processing single or batch videos
-* Implementation of better filtering in pre-processing
-* Improved efficiency of processing
-* Establishing on-site edge computation through a raspberry-pi camera setup
+* Exports to simple text formats and GIS-compatible layers
+* Exports to augmented reality videos
 * Implementation of additional processing algorithms (STIV and LSPTV)
+* Implementation of several optical methods for reading water levels
+* Improved nighttime / poor weather conditions processing through learning approaches
 
 If you wish to fund this or other work on features, please contact us at info@rainbowsensing.com.
 
@@ -48,28 +48,60 @@ If you wish to fund this or other work on features, please contact us at info@ra
 > **_compatibility:_** At this moment **pyorc** works with any video compatible with OpenCV as long as it has proper metadata.
 
 ## Installation
+You need a python environment. We recommend using the Miniforge project. Download
+the latest miniforge installer from https://github.com/conda-forge/miniforge and follow the installation instructions
+before continuing installing **pyorc**.
 
-To get started with **pyorc**, we recommend to setup a python virtual environment. 
-We recommend using a Miniconda or Anaconda environment as this will ease installation, and will allow you to use all
-functionalities without any trouble. Especially geographical plotting with `cartopy` can be difficult to get installed. 
-With a `conda` environment and our `conda-forge` package this is solved. In the subsections below, you can find specific
-instructions for different use cases. 
+### virtual environment
+To get started with **pyorc**, we recommend to set up a python virtual environment. This ensures that installed libraries
+**pyorc** will not conflict with other libraries or library versions which you may need for other projects.
+
+Setting up a virtual environment can be done with:
+```shell
+python -m venv pyorc_env
+```
+this creates a new folder `pyorc_env` on your disk which contains your virtual environment.
+After activating the environment, any package you install will be installed in this environment only.
+Activating in Unix/Linux is done as follows:
+
+```shell
+source pyorc_env/bin/activate
+```
+In Windows, the activation script is in a different folder. Type the following to activate the environment.
+```shell
+pyorc_env\Scripts\activate
+```
 
 ### Installation for direct use
 
-If you simply want to add **pyorc** to an existing python installation or virtual environment, then follow these 
+If you simply want to add **pyorc** to an existing python installation or virtual environment, then follow these
 instructions.
 
-First activate the environment you want **pyorc** to be installed in (if you don't care about virtual environments, then 
-simply skip this step). You can simply install pyorc with all its dependencies as follows:
+First activate the environment you want **pyorc** to be installed in (if you don't care about virtual environments, then
+simply skip this step). See the above sub-section for information. You can simply install pyorc with all its
+dependencies as follows:
 
+```shell
+pip install pyopenrivercam[extra]
 ```
-conda activate <name-of-your-environment>
-conda install -c conda-forge pyopenrivercam
-```
+The `[extra]` section ensures that also geographical plotting is supported, which we recommend especially for the
+set up of a camera configuration.
+
+### Upgrading from pypi with pip
 
-If you use mamba as a package mananager, then the steps are the same, except for the installation step, which is:
+Did you read about a new version and you want to upgrade? Simply activate your virtual environment, type
+
+```shell
+pip install --upgrade pyopenrivercam[extra]
 ```
+
+and then enjoy the latest features.
+
+### installation with mamba package manager
+
+If you use `mamba` as a package manager, then the steps are the same, except for the installation step, which is:
+
+```shell
 mamba install pyopenrivercam
 ```
 
@@ -85,7 +117,7 @@ git clone https://github.com/localdevices/pyorc.git
 cd pyorc
 ```
 
-Setup a virtual environment with all dependencies as follows:
+Set up a virtual environment with all dependencies as follows:
 ```
 conda env create -f envs/pyorc-dev.yml
 conda activate pyorc-dev
@@ -117,34 +149,36 @@ pip install -e .
 ```
 
 ## Using pyorc
-To use **pyorc**, you can use the API for processing. A command-line interface is forthcoming pending funding. 
+To use **pyorc**, you can use the API for processing. A command-line interface is forthcoming pending funding.
 A manual is also still in the making.
 
 ## Acknowledgement
-The first development of pyorc has been supported by the World Meteorological Organisation - HydroHub. 
+The first development of pyorc has been supported by the World Meteorological Organisation - HydroHub.
 
 ## License
 **pyorc** is licensed under AGPL Version 3 (see [LICENSE](./LICENSE) file).
 
-**pyorc** uses the following libraries and software with said licenses.
+**pyorc** uses the following important libraries and software with said licenses.
 
 | Package    | Version | License                            |
 |------------|---------|------------------------------------|
-| numpy      | 1.23.2  | BSD License                        |
-| opencv2    | 4.6.0   | MIT License                        |                                                                                      
-| openpiv    | 0.23.8  | GPLv3                              |                                                                                      
-| matplotlib | 3.5.3   | Python Software Foundation License |                                                               
-| geopandas  | 0.10.2  | BSD License                        |                                                                                              
- | pandas     | 1.4.3   | BSD License                        |                                                                                      
+| ffpiv      | 0.1.2   | AGPLv3                             |
+| numpy      | 1.26.4  | BSD License                        |
+| opencv2    | 4.10.0  | MIT License                        |
+| openpiv    | 0.25.3  | GPLv3                              |
+| matplotlib | 3.9.2   | Python Software Foundation License |
+| geopandas  | 1.0.1   | BSD License                        |
+| pandas     | 2.2.2   | BSD License                        |
 
 Project organisation
 --------------------
 
     .
-    ├── README.md
-    ├── LICENSE
-    ├── TRADEMARK.md
-    ├── setup.py            <- setup script compatible with pip
+    ├── CHANGELOG.md        <- Version-based changelog documentation
+    ├── README.md           <- This file
+    ├── LICENSE             <- License file containing AGPLv3.0 license terms
+    ├── TRADEMARK.md        <- Trademark guidelines
+    ├── pyproject.toml      <- setup pipeline compatible with pip
     ├── environment.yml     <- YML-file for setting up a conda environment with dependencies
     ├── docs                <- Sphinx documentation source code
         ├── ...             <- Sphinx source code files
@@ -154,4 +188,3 @@ Project organisation
         ├── ...             <- pyorc functions and API files
     ├── tests               <- pytest suite
         ├── ...             <- pytest functions on API level
-
diff --git a/docs/_images/GCPs_interactive.jpg b/docs/_images/GCPs_interactive.jpg
diff --git a/docs/_images/bbox_interactive.jpg b/docs/_images/bbox_interactive.jpg
diff --git a/docs/api.rst b/docs/api.rst
@@ -9,9 +9,9 @@ API reference
 **pyorc**'s API consists of several subclasses of the ``xarray.Dataset`` and ``xarray.DataArray`` data models.
 In a nutshell, xarray_'s data models are meant to store and analyze scientific datasets with multiple
 dimensions. A ``xarray.DataArray`` contains one variable with possibly several dimensions and coordinates
-within those dimensions. A ``xarray.Dataset`` may contain multiple ``xarray.DataArray`` objects, with shared 
+within those dimensions. A ``xarray.Dataset`` may contain multiple ``xarray.DataArray`` objects, with shared
 coordinates. In **pyorc** typically the coordinates are ``time`` for time epochs measured in seconds since
-the beginning of a video, ``x`` for horizontal grid spacing (in meters), ``y`` for vertical grid spacing 
+the beginning of a video, ``x`` for horizontal grid spacing (in meters), ``y`` for vertical grid spacing
 (in meters). Operations you can apply on both data models are very comparable with operations you may
 already use in pandas_, such as resampling, reindexing, aggregations, and so on.
 
@@ -47,6 +47,7 @@ Setting of properties and attributes
     CameraConfig.set_bbox_from_corners
     CameraConfig.set_gcps
     CameraConfig.set_lens_pars
+    CameraConfig.set_intrinsic
     CameraConfig.set_lens_calibration
     CameraConfig.set_lens_position
 
@@ -57,6 +58,7 @@ Exporting
     :toctree: _generated
 
     CameraConfig.to_dict
+    CameraConfig.to_dict_str
     CameraConfig.to_file
     CameraConfig.to_json
 
@@ -68,8 +70,18 @@ Retrieve geometrical information
     :toctree: _generated
 
     CameraConfig.get_M
+    CameraConfig.get_bbox
+    CameraConfig.get_camera_coords
+    CameraConfig.get_dist_shore
+    CameraConfig.get_dist_wall
     CameraConfig.get_depth
+    CameraConfig.get_z_a
+    CameraConfig.project_grid
+    CameraConfig.project_points
+    CameraConfig.unproject_points
     CameraConfig.z_to_h
+    CameraConfig.h_to_z
+    CameraConfig.estimate_lens_position
 
 Plotting methods
 ----------------
@@ -93,11 +105,27 @@ Class and properties
 
     Video
     Video.camera_config
-    Video.fps
     Video.end_frame
+    Video.fps
+    Video.frames
+    Video.freq
+    Video.h_a
+    Video.lazy
+    Video.mask
+    Video.rotation
+    Video.stabilize
     Video.start_frame
     Video.corners
 
+Setting properties
+------------------
+
+.. autosummary::
+    :toctree: _generated
+
+    Video.set_mask_from_exterior
+
+
 Getting frames from video objects
 ---------------------------------
 
@@ -106,6 +134,8 @@ Getting frames from video objects
 
     Video.get_frame
     Video.get_frames
+    Video.get_frames_chunk
+    Video.get_ms
 
 .. _frames:
 
@@ -132,9 +162,11 @@ Enhancing frames
     :toctree: _generated
 
     Frames.edge_detect
+    Frames.minmax
     Frames.normalize
-    Frames.time_diff
+    Frames.reduce_rolling
     Frames.smooth
+    Frames.time_diff
 
 Projecting frames to planar views
 ---------------------------------
@@ -178,6 +210,7 @@ Class and properties
     Velocimetry.camera_config
     Velocimetry.camera_shape
     Velocimetry.h_a
+    Velocimetry.is_velocimetry
 
 .. _masks:
 
@@ -274,6 +307,10 @@ Derivatives
 
     Transect.vector_to_scalar
     Transect.get_xyz_perspective
+    Transect.get_depth_perspective
+    Transect.get_bottom_surface_z_perspective
+    Transect.get_transect_perspective
+    Transect.get_wetted_perspective
 
 .. _river_flow:
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -54,12 +54,10 @@ Support us
 We are seeking funding for the following frequently requested functionalities:
 
 * Exports to simple text formats and GIS-compatible layers
-* Compiled windows executables
-* Implementation of better filtering in pre-processing
-* Improved efficiency of processing (e.g. through GPU/TPU acceleration)
-* Establishing on-site edge computation through a raspberry-pi camera setup
+* Exports to augmented reality videos
 * Implementation of additional processing algorithms (STIV and LSPTV)
-* Establishment of dashboard environments for several use cases
+* Implementation of several optical methods for reading water levels
+* Improved nighttime / poor weather conditions processing through learning approaches
 
 Please contact us at info@rainbowsensing.com for further information.
 
@@ -72,7 +70,8 @@ Please contact us at info@rainbowsensing.com for further information.
 .. note::
 
    Acknowledgement: the development of **pyorc** has been funded partly by the World Meteorological Organisation,
-   Rijkswaterstaat and in-kind contributions of `Rainbow Sensing <https://rainbowsensing.com>`_.
+   Rijkswaterstaat, the European Commission TEMBO Africa project, Grant Agreement No. 101086209
+   and in-kind contributions of `Rainbow Sensing <https://rainbowsensing.com>`_.
 
 .. toctree::
    :maxdepth: 1