Small bug fixes and bump version 0.7.0 #193

CHANGELOG.md

@@ -1,3 +1,16 @@
+## [0.7.0] - 2024-12-10
+### Added
+`get_piv` now uses several engines, `engine="numba"` is a lot fastr
+### Changed
+Reading frames is now a lot more efficient as they are read in bulks (20 by default). As a result, very large videos
+can be processed efficiently.
+### Deprecated
+openpiv is still default, but may become deprecated in future versions.
+### Removed
+### Fixed
+### Security
+
+
 ## [0.6.1] - 2024-09-26
 ### Added
 ### Changed

README.md

@@ -33,6 +33,8 @@ We use the well-known **xarray** data models and computation pipelines (with das
 guarantee an easy interoperability with other tools and methods, and allow for lazy computing.
 
 We are seeking funding for the following frequently requested functionalities:
+* Exports to simple text formats and GIS-compatible layers
+* Exports to augmented reality videos
 * Implementation of additional processing algorithms (STIV and LSPTV)
 * Implementation of several optical methods for reading water levels
 * Improved nighttime / poor weather conditions processing through learning approaches
@@ -156,25 +158,27 @@ The first development of pyorc has been supported by the World Meteorological Or
 ## License
 **pyorc** is licensed under AGPL Version 3 (see [LICENSE](./LICENSE) file).
 
-**pyorc** uses the following libraries and software with said licenses.
+**pyorc** uses the following important libraries and software with said licenses.
 
 | Package    | Version | License                            |
 |------------|---------|------------------------------------|
-| numpy      | 1.23.2  | BSD License                        |
-| opencv2    | 4.6.0   | MIT License                        |
-| openpiv    | 0.23.8  | GPLv3                              |
-| matplotlib | 3.5.3   | Python Software Foundation License |
-| geopandas  | 0.10.2  | BSD License                        |
- | pandas     | 1.4.3   | BSD License                        |
+| ffpiv      | 0.1.2   | AGPLv3                             |
+| numpy      | 1.26.4  | BSD License                        |
+| opencv2    | 4.10.0  | MIT License                        |
+| openpiv    | 0.25.3  | GPLv3                              |
+| matplotlib | 3.9.2   | Python Software Foundation License |
+| geopandas  | 1.0.1   | BSD License                        |
+| pandas     | 2.2.2   | BSD License                        |
 
 Project organisation
 --------------------
 
     .
-    ├── README.md
-    ├── LICENSE
-    ├── TRADEMARK.md
-    ├── setup.py            <- setup script compatible with pip
+    ├── CHANGELOG.md        <- Version-based changelog documentation
+    ├── README.md           <- This file
+    ├── LICENSE             <- License file containing AGPLv3.0 license terms
+    ├── TRADEMARK.md        <- Trademark guidelines
+    ├── pyproject.toml      <- setup pipeline compatible with pip
     ├── environment.yml     <- YML-file for setting up a conda environment with dependencies
     ├── docs                <- Sphinx documentation source code
         ├── ...             <- Sphinx source code files

docs/api.rst

@@ -9,9 +9,9 @@ API reference
 **pyorc**'s API consists of several subclasses of the ``xarray.Dataset`` and ``xarray.DataArray`` data models.
 In a nutshell, xarray_'s data models are meant to store and analyze scientific datasets with multiple
 dimensions. A ``xarray.DataArray`` contains one variable with possibly several dimensions and coordinates
-within those dimensions. A ``xarray.Dataset`` may contain multiple ``xarray.DataArray`` objects, with shared 
+within those dimensions. A ``xarray.Dataset`` may contain multiple ``xarray.DataArray`` objects, with shared
 coordinates. In **pyorc** typically the coordinates are ``time`` for time epochs measured in seconds since
-the beginning of a video, ``x`` for horizontal grid spacing (in meters), ``y`` for vertical grid spacing 
+the beginning of a video, ``x`` for horizontal grid spacing (in meters), ``y`` for vertical grid spacing
 (in meters). Operations you can apply on both data models are very comparable with operations you may
 already use in pandas_, such as resampling, reindexing, aggregations, and so on.
 
@@ -47,6 +47,7 @@ Setting of properties and attributes
     CameraConfig.set_bbox_from_corners
     CameraConfig.set_gcps
     CameraConfig.set_lens_pars
+    CameraConfig.set_intrinsic
     CameraConfig.set_lens_calibration
     CameraConfig.set_lens_position
 
@@ -57,6 +58,7 @@ Exporting
     :toctree: _generated
 
     CameraConfig.to_dict
+    CameraConfig.to_dict_str
     CameraConfig.to_file
     CameraConfig.to_json
 
@@ -68,8 +70,18 @@ Retrieve geometrical information
     :toctree: _generated
 
     CameraConfig.get_M
+    CameraConfig.get_bbox
+    CameraConfig.get_camera_coords
+    CameraConfig.get_dist_shore
+    CameraConfig.get_dist_wall
     CameraConfig.get_depth
+    CameraConfig.get_z_a
+    CameraConfig.project_grid
+    CameraConfig.project_points
+    CameraConfig.unproject_points
     CameraConfig.z_to_h
+    CameraConfig.h_to_z
+    CameraConfig.estimate_lens_position
 
 Plotting methods
 ----------------
@@ -93,11 +105,27 @@ Class and properties
 
     Video
     Video.camera_config
-    Video.fps
     Video.end_frame
+    Video.fps
+    Video.frames
+    Video.freq
+    Video.h_a
+    Video.lazy
+    Video.mask
+    Video.rotation
+    Video.stabilize
     Video.start_frame
     Video.corners
 
+Setting properties
+------------------
+
+.. autosummary::
+    :toctree: _generated
+
+    Video.set_mask_from_exterior
+
+
 Getting frames from video objects
 ---------------------------------
 
@@ -106,6 +134,8 @@ Getting frames from video objects
 
     Video.get_frame
     Video.get_frames
+    Video.get_frames_chunk
+    Video.get_ms
 
 .. _frames:
 
@@ -132,9 +162,11 @@ Enhancing frames
     :toctree: _generated
 
     Frames.edge_detect
+    Frames.minmax
     Frames.normalize
-    Frames.time_diff
+    Frames.reduce_rolling
     Frames.smooth
+    Frames.time_diff
 
 Projecting frames to planar views
 ---------------------------------
@@ -178,6 +210,7 @@ Class and properties
     Velocimetry.camera_config
     Velocimetry.camera_shape
     Velocimetry.h_a
+    Velocimetry.is_velocimetry
 
 .. _masks:
 
@@ -274,6 +307,10 @@ Derivatives
 
     Transect.vector_to_scalar
     Transect.get_xyz_perspective
+    Transect.get_depth_perspective
+    Transect.get_bottom_surface_z_perspective
+    Transect.get_transect_perspective
+    Transect.get_wetted_perspective
 
 .. _river_flow:
 

docs/index.rst

@@ -54,12 +54,10 @@ Support us
 We are seeking funding for the following frequently requested functionalities:
 
 * Exports to simple text formats and GIS-compatible layers
-* Compiled windows executables
-* Implementation of better filtering in pre-processing
-* Improved efficiency of processing (e.g. through GPU/TPU acceleration)
-* Establishing on-site edge computation through a raspberry-pi camera setup
+* Exports to augmented reality videos
 * Implementation of additional processing algorithms (STIV and LSPTV)
-* Establishment of dashboard environments for several use cases
+* Implementation of several optical methods for reading water levels
+* Improved nighttime / poor weather conditions processing through learning approaches
 
 Please contact us at info@rainbowsensing.com for further information.
 
@@ -72,7 +70,8 @@ Please contact us at info@rainbowsensing.com for further information.
 .. note::
 
    Acknowledgement: the development of **pyorc** has been funded partly by the World Meteorological Organisation,
-   Rijkswaterstaat and in-kind contributions of `Rainbow Sensing <https://rainbowsensing.com>`_.
+   Rijkswaterstaat, the European Commission TEMBO Africa project, Grant Agreement No. 101086209
+   and in-kind contributions of `Rainbow Sensing <https://rainbowsensing.com>`_.
 
 .. toctree::
    :maxdepth: 1

docs/user-guide/velocimetry/index.rst

@@ -61,9 +61,9 @@ in :ref:`frames_ug`.
         Here the ``engine: numba`` ensures that the much faster numba implementation is used. ``window_size: 64``
         overrides any window size provided in the camera configuration and sets it to 64. When using the engine
         parameter with ``numba`` or ``numpy``, you can also provide memory safety margins by manually adjusting the
-        amount of chunks in which the problem is sub-divided. If you notice a list of 20 chunks is made and the
-        calculations crash, try manually setting it to twice the amount with ``chunks: 40``. The default should be
-        reasonably safe, so we hope you never have to touch this parameter at all :-).
+        chunk size with the ``chunksize`` parameter. If you notice a memory warning is given, and ``chunksize`` is
+        set to 5, try manually setting it to e.g. 3 or 2. The default should be reasonably safe, so we hope you never
+        have to touch this parameter at all :-) unless you process very very large videos with very large objectives.
 
         .. note::
 
@@ -91,11 +91,11 @@ in :ref:`frames_ug`.
         Use thew ``engine`` parameter to select a much faster computational engine. With ``engine="numba"`` a very fast
         numba-based computation will be used. The computation will be chunked into several batches based on available
         memory. If you find out your computations crash it is likely due to lack of memory. In this case you can
-        override automatically computed chunk amounts by setting ``chunks`` to a larger amount than automatically
-        selected, or provide ``memory_factor`` and set this to a higher amount than the default (4). ``memory_factor``
-        decides on the fraction of the memory reserved for one entire chunk of computation. By setting it to 4 only
-        1/4th of the available memory is used. In practice, for large problems, more temporary memory storage is needed
-        within the lazy subprocessing.
+        override automatically computed chunk amounts by setting ``chunksize`` to a smaller or user-defined amount.
+        You may also set ``memory_factor`` to a higher amount than the default (2). ``memory_factor``
+        decides on the fraction of the memory reserved for one entire chunk of computation. E.g. by setting it to 4 only
+        1/4th of the available memory is assumed to be available. In practice, for large problems, more temporary
+        memory storage is needed within the subprocessing.
 
 Interrogating and storing PIV results
 -------------------------------------
@@ -146,24 +146,14 @@ These results can be stored so that they can be interrogated later, by other sof
             # store results in a file, this will take a while
             piv.to_netcdf("piv_results.nc")
 
-        Only when you store or otherwise retrieve data resulting from ``get_piv``, the computations will actually be performed.
-        Therefore it is normal that only after calling a command that retrieves data, you will need to wait for a while before
-        data is returned. This may take several minutes for small problems, but for large areas of interest or large amounts of
-        time steps (or a slow machine) it can also take half an hour or longer. To keep track of progress you can also first
-        prepare the storage process and the wrap a ``ProgressBar`` from the ``dask.diagnostics`` library.
-        Below you can find an example how to store data with such a progress bar.
+        PIV computations are the longest in time. Therefore it is normal that you will need to wait for a while before
+        data is returned. This may take a few seconds for small problems, but for large areas of interest or large
+        amounts of time steps (or a slow machine with little memory) it can also take half an hour or longer. With
+        the new (>=0.7.0) numba and numpy engines, you can keep track of progress with a progress bar which is
+        automatically displayed during the processing.
 
-        .. code:: python
-
-            # import ProgressBar
-            from dask.diagnostics import ProgressBar
-            # store results with a progress bar
-            delayed_obj = piv.to_netcdf("piv_results.nc", compute=False)
-            with ProgressBar():
-                results = delayed_obj.compute()
-
-        You should then see a progressing bar on your screen while data is stored. If you wish to load your results into
-        memory after having stored it in a previous session, you can simply use ``xarray`` functionality to do so.
+        If you wish to load your results into memory after having stored it in a previous session, you can simply
+        use ``xarray`` functionality to do so.
 
         .. code:: python
 

examples/02_Process_velocimetry.ipynb

@@ -238,9 +238,9 @@
    "metadata": {},
    "source": [
     "### Velocimetry estimates\n",
-    "Now that we have real-world projected frames, with contrast enhanced, let's do some velocimetry! For Particle Image Velocimetry, this is as simple as calling the `.get_piv` method on the frames. Again a lazy result is returned really fast. If you want to do the computations, you can either extract a single frame, or (as below) store the result in a nice NetCDF file. Note that this file can be loaded back into memory with the `xarray` API without any additional fuss. We use a delayed method for storing, just to see a progress bar. If you are not interested in that, you can also replace the last 3 lines by:\n",
+    "Now that we have real-world projected frames, with contrast enhanced, let's do some velocimetry! For Particle Image Velocimetry, this is as simple as calling the `.get_piv` method on the frames. We then store the result in a nice NetCDF file. This file can be loaded back into memory with the `xarray` API without any additional fuss. We also use our new `numba` engine to compute PIV (version >= 0.7.0). This is faster than the original `openpiv` engine. If you want to try `openpiv` you can do this by changing the engine.\n",
     "```python\n",
-    "piv.to_netcdf(\"ngwerere_piv.nc\")\n",
+    "piv = da_norm_proj.frames.get_piv(engine=\"openpiv\")\n",
     "```\n"
    ]
   },
@@ -251,15 +251,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import time\n",
-    "t1 = time.time()\n",
-    "da_norm_proj = da_norm_proj.load()\n",
-    "piv = da_norm_proj.frames.get_piv()\n",
-    "delayed_obj = piv.to_netcdf(\"ngwerere_piv.nc\", compute=False)\n",
-    "with ProgressBar():\n",
-    "    results = delayed_obj.compute()\n",
-    "t2 = time.time()\n",
-    "print(f\"write v took {t2-t1} secs.\")"
+    "piv = da_norm_proj.frames.get_piv(engine=\"numba\")\n",
+    "piv.to_netcdf(\"ngwerere_piv.nc\")\n"
    ]
   },
   {

pyorc/__init__.py

@@ -1,8 +1,9 @@
 """pyorc: free and open-source image-based surface velocity and discharge."""
 
-__version__ = "0.6.1"
+__version__ = "0.7.0"
 
-from .api import *
+from . import cli, service
+from .api import CameraConfig, Frames, Transect, Velocimetry, Video, get_camera_config, load_camera_config
 from .project import *
 
 __all__ = [
@@ -14,5 +15,5 @@
     "Velocimetry",
     "Transect",
     "service",
-    "cli"
+    "cli",
 ]

pyorc/api/frames.py

@@ -550,16 +550,34 @@ def animate(i):
     def to_video(self, fn, video_format=None, fps=None):
         """Write frames to a video file without any layout.
 
+        Frames from the input object are written into a video file. The format and frame
+        rate can be customized as per user preference or derived automatically from the
+        input object.
+
         Parameters
         ----------
         fn : str
-            Path to output file
+            Path to the output video file.
         video_format : cv2.VideoWriter_fourcc, optional
-            A VideoWriter preference, default is cv2.VideoWriter_fourcc(*"mp4v")
+            The desired video file format codec. If not provided, defaults to
+            `cv2.VideoWriter_fourcc(*"mp4v")`.
         fps : float, optional
-            Frames per second, if not provided, derived from original video
+            Frames per second for the output video. If not specified, it is estimated
+            from the time differences in the input frames.
 
         """
+        # """Write frames to a video file without any layout.
+        #
+        # Parameters
+        # ----------
+        # fn : str
+        #     Path to output file
+        # video_format : cv2.VideoWriter_fourcc, optional
+        #     A VideoWriter preference, default is cv2.VideoWriter_fourcc(*"mp4v")
+        # fps : float, optional
+        #     Frames per second, if not provided, derived from original video
+        #
+        # """
         if video_format is None:
             # set to a default
             video_format = cv2.VideoWriter_fourcc(*"mp4v")

pyorc/cli/main.py

@@ -240,7 +240,7 @@ def camera_config(
             logger=logger,
         )
     else:
-        stabilize = None
+        stabilize_pol = None
     service.camera_config(
         video_file=videofile,
         cam_config_file=output,

pyorc/cv.py

@@ -52,7 +52,7 @@ def _check_valid_frames(cap, frame_number):
         if not (ret):
             last_valid_idx = n
         n -= 1
-        if n == -len(frame_number):
+        if n == -len(frame_number) - 1:
             last_valid_idx = 0
             break