merge streamdaq_test

.github/workflows/tests.yml

@@ -1,12 +1,16 @@
 name: Run Python tests
 
-on: [push]
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
 
 jobs:
   build:
     name: Run tests
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: false
       matrix:
         os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
         python-version: ["3.9", "3.10", "3.11"]
@@ -17,10 +21,11 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
+          cache: 'pip'
       - name: Install dependencies
-        run: pip install .[tests] pytest pytest-md pytest-emoji
+        run: pip install .[tests] pytest pytest-md
       - uses: pavelzw/pytest-action@v2
         with:
-          emoji: true
           verbose: true
-          job-summary: true
+          emoji: false
+          job-summary: true

docs/api/models/index.md

@@ -20,4 +20,5 @@ buffer
 config
 mixins
 models
+stream
 ```

docs/api/models/stream.md

@@ -0,0 +1,7 @@
+# Stream
+
+```{eval-rst}
+.. automodule:: miniscope_io.models.stream
+    :members:
+    :undoc-members:
+```

docs/api/stream_daq.md

@@ -1,11 +1,60 @@
-# Stream Daq
+# Stream DAQ
+This module is a data acquisition module that captures video streams from Miniscopes based on the `Miniscope-SAMD-Framework` firmware. The firmware repository will be published in future updates but is currently under development and private.
 
-(Some description of what this module does, can happen either here in markdown
-or in the module docstring as RST)
+## Command
+After [installation](./installation.md) and customizing [configurations](#configuration) if necessary, run the following command in this Git repository to start the data acquisition process:
+```bash
+>>> streamDaq -c path/to/config/yaml/file.yml
+Connected to XEM7310-A75
+Successfully uploaded /miniscope-io/miniscope_io/devices/selected_bitfile.bit
+FrontPanel is supported
+[24-06-11T01:40:45] INFO     [miniscope_io.streamDaq.frame] frame: 1570, bits lost: 0                                    stream_daq.py:524
+[24-06-11T01:40:46] INFO     [miniscope_io.streamDaq.frame] frame: 1571, bits lost: 0                                    stream_daq.py:524
+```
+A window displaying the image transferred from the Miniscope should pop up. Additionally, the indexes of captured frames and their statuses will be logged in the terminal.
+
+## Prerequisites
+- **Data capture hardware:** Opal Kelly XEM7310-A75 FPGA board (connected via USB)
+- **Supported Operating Systems:** MacOS or Ubuntu PC (To do: specify version)
+- **Imaging hardware:** Miniscope based on the `Miniscope-SAMD-Framework` firmware. Hardware modules for feeding in data into the data capture hardware are also needed but these will be specified in future updates.
+
+## Configuration
+A YAML file is used to configure this module. The configuration needs to match the imaging and data capture hardware for proper operation. This file is used to set up hardware, define data formats, and set data preambles. The contents of this YAML file will be parsed into a model [miniscope_io.models.stream](../api/models/stream.md), which then configures the Stream DAQ.
+
+### Example Configuration
+Below is an example configuration YAML file. More examples can be found in `miniscope_io.data.config`.
+
+```yaml
+# capture device. "OK" (Opal Kelly) or "UART"
+device: "OK"
+
+# The configuration bitstream file to upload to the Opal Kelly board. This uploads a Manchester decoder HDL and different bitstream files are required to configure different data rates and bit polarity. This is a binary file synthesized using Vivado, and details for generating this file will be provided in later updates.
+bitstream: "USBInterface-6mhz-3v3-INVERSE.bit"
+
+# COM port and baud rate is only required for UART mode
+port: null
+baudrate: null
+
+# Preamble for each data buffer. This is actually converted to bytes in the StreamDaq but has to be imported as a string because .yaml doesn't support hexadecimal formats.
+preamble: 0x12345678
+
+# Image format. StreamDaq will calculate buffer size, etc. based on these parameters
+frame_width: 304
+frame_height: 304
+pix_depth: 8
+
+# Buffer data format. These have to match the firmware value
+header_len: 12
+buffer_block_length: 40
+block_size: 512
+num_buffers: 8
+
+# Temporary parameter to handle bit polarity. This is not actual LSB, so it needs to be fixed later.
+LSB: True
+```
 
 ```{eval-rst}
 .. automodule:: miniscope_io.stream_daq
     :members:
     :undoc-members:
-```
-
+```

docs/conf.py

@@ -8,8 +8,8 @@
 
 project = 'miniscope-io'
 copyright = '2023, Jonny'
-author = 'Jonny'
-release = '0.1.0'
+author = 'Jonny, Takuya'
+release = '0.1.8'
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

docs/guide/index.md

@@ -2,4 +2,5 @@
 
 ```{toctree}
 installation
+usage
 ```

docs/guide/usage.md

@@ -0,0 +1,2 @@
+```{include} ../api/stream_daq.md
+```

docs/meta/changelog.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 0.1.9 - 24-06-21
+
+- https://github.com/Aharoni-Lab/miniscope-io/pull/26
+
+Testing:
+
+- [@t-sasatani](https://github.com/t-sasatani) - add end-to-end test for {class}`~miniscope_io.stream_daq.streamDaq`
+- Add a mock class for {class}`~miniscope_io.devices.opalkelly.okDev`
+- replace `tmpdir` fixture and `tempfile` module with `tmp_path`
+
+New:
+
+- [@t-sasatani](https://github.com/t-sasatani) - allow use of okDev on Windows
+- {meth}`~miniscope_io.stream_daq.StreamDaq.capture` can export video :)
+- More specific exceptions:
+  - {class}`~miniscope_io.exceptions.StreamError`
+  - {class}`~miniscope_io.exceptions.StreamReadError`
+  - {class}`~miniscope_io.exceptions.DeviceError`
+  - {class}`~miniscope_io.exceptions.DeviceOpenError`
+  - {class}`~miniscope_io.exceptions.DeviceConfigurationError`
+- {func}`~miniscope_io.utils.hash_video` - hash decoded video frames, rather than encoded video file
+
+
+Fixed:
+
+- Removed `print` statements in {class}`~miniscope_io.devices.opalkelly.okDev`
+- {meth}`~miniscope_io.stream_daq.StreamDaq.capture`
+  - Don't require `config`
+  - Replace logging with {func}`~miniscope_io.logging.init_logger`
+  - Use of {attr}`~miniscope_io.stream_daq.StreamDaq.terminate` to control inner loops
+
+
+Models:
+
+- added `fs` and `show_video` to {class}`~miniscope_io.models.stream.StreamDaqConfig`
+
+CI:
+
+- [@t-sasatani](https://github.com/t-sasatani) - restore windows and mac tests (oops)
+- caching dependency installs
+- not using pytest-emoji, it was always annoying
+
+## 0.1.8 - 24-06-16
+
+- https://github.com/Aharoni-Lab/miniscope-io/pull/21
+- https://github.com/Aharoni-Lab/miniscope-io/pull/15
+
+New features:
+
+- **Support for Various Image Formats**: `streamDaq` now supports multiple image formats, including different image sizes, frame rates (FPS), and bit-depths. These configurations can be provided via a YAML file. Examples of these configurations can be found in `miniscope_io.data.config`.
+- **Pydantic Model for Configuration**: Added a Pydantic model to validate the configuration of `streamDaq`.
+- **Bitstream Loader**: Added a bitstream loader to automatically configure the Opal Kelly FPGA when running `streamDaq`.
+- **Updated Command Line Script**: The command line script for running `streamDaq` has been updated. Use `streamDaq -c path/to/config/yaml/file.yml` to run the process with your YAML configuration file.
+- **Logger Module**: Added a logger module that can be configured using environmental variables or a `.env` file.
+
+Note: Version 0.1.7 was skipped accidentally and does not exist.
+
 ## 0.1.6 - 24-04-09
 
 - https://github.com/Aharoni-Lab/miniscope-io/pull/14

miniscope_io/devices/opalkelly.py

@@ -3,6 +3,8 @@
 """
 
 from miniscope_io.vendor import opalkelly as ok
+from miniscope_io.exceptions import StreamReadError, DeviceConfigurationError, DeviceOpenError
+from miniscope_io.logging import init_logger
 
 
 class okDev(ok.okCFrontPanel):
@@ -18,13 +20,14 @@ class okDev(ok.okCFrontPanel):
 
     def __init__(self, serial_id: str = ""):
         super().__init__()
+        self.logger = init_logger('okDev')
         ret = self.OpenBySerial("")
         if ret != self.NoError:
-            raise ValueError(f"Cannot open device: {serial_id}")
+            raise DeviceOpenError(f"Cannot open device: {serial_id}")
         self.info = ok.okTDeviceInfo()
         ret = self.GetDeviceInfo(self.info)
         if ret == self.NoError:
-            print(f"Connected to {self.info.productName}")
+            self.logger.info(f"Connected to {self.info.productName}")
 
     def uploadBit(self, bit_file: str) -> None:
         """
@@ -36,10 +39,10 @@ def uploadBit(self, bit_file: str) -> None:
 
         ret = self.ConfigureFPGA(bit_file)
         if ret == self.NoError:
-            print(f"Succesfully uploaded {bit_file}")
+            self.logger.debug(f"Succesfully uploaded {bit_file}")
         else:
-            raise ValueError(f"Configuration of {self.info.productName} failed")
-        print(
+            raise DeviceConfigurationError(f"Configuration of {self.info.productName} failed")
+        self.logger.debug(
             "FrontPanel {} supported".format(
                 "is" if self.IsFrontPanelEnabled() else "not"
             )
@@ -61,9 +64,11 @@ def readData(self, length: int, addr: int = 0xA0, blockSize: int = 16) -> bytear
         buf = bytearray(length)
         ret = self.ReadFromBlockPipeOut(addr, data=buf, blockSize=blockSize)
         if ret < 0:
-            raise ValueError(f"Read failed: {ret}")
+            msg = f"Read failed: {ret}"
+            self.logger.error(msg)
+            raise StreamReadError(msg)
         elif ret < length:
-            print(f"Only {ret} bytes read")
+            self.logger.warning(f"Only {ret} bytes read")
         return buf
 
     def setWire(self, addr: int, val: int) -> None:
@@ -79,4 +84,4 @@ def setWire(self, addr: int, val: int) -> None:
         ret = self.SetWireInValue(addr, val)
         ret = self.UpdateWireIns()
         if ret != self.NoError:
-            raise ValueError(f"Wire update failed: {ret}")
+            raise DeviceConfigurationError(f"Wire update failed: {ret}")

miniscope_io/exceptions.py

@@ -13,3 +13,28 @@ class EndOfRecordingException(StopIteration):
     """
     Raised when :class:`.io.SDCard` is at the end of the available recording!
     """
+
+class StreamError(RuntimeError):
+    """
+    Base class for errors while streaming data
+    """
+
+class StreamReadError(StreamError):
+    """
+    Error while reading streaming data from a device
+    """
+
+class DeviceError(RuntimeError):
+    """
+    Base class for errors when communicating with or configuring devices
+    """
+
+class DeviceOpenError(DeviceError):
+    """
+    Error opening a connection to a device
+    """
+
+class DeviceConfigurationError(DeviceError):
+    """
+    Error while configuring a device
+    """

miniscope_io/io.py

@@ -378,8 +378,8 @@ def read(self, return_header: bool = False) -> Union[np.ndarray, Frame]:
 
     def to_video(
         self,
-        path: Union[Path, str],
-        fourcc: Literal["GREY", "mp4v", "XVID"] = "GREY",
+        path: Union[Path,str],
+        fourcc: Literal['GREY', 'mp4v', 'XVID'] = 'GREY',
         isColor: bool = False,
         force: bool = False,
         progress: bool = True,

miniscope_io/models/config.py

@@ -84,7 +84,8 @@ class Config(BaseSettings):
     def folder_exists(cls, v: Path) -> Path:
         v = Path(v)
         v.mkdir(exist_ok=True, parents=True)
-        assert v.exists()
+
+        assert v.exists(), f'{v} does not exist!'
         return v
 
     @model_validator(mode="after")

miniscope_io/models/stream.py

@@ -3,7 +3,7 @@
 """
 
 from pathlib import Path
-from typing import Optional, Union
+from typing import Optional, Union, Literal
 
 from pydantic import field_validator
 
@@ -25,13 +25,15 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
 
     Parameters
     ----------
+    mode: str
+    Operation modes. Maybe having DAQ (normal operation), RAW_RECORD (for getting testdata), and RAW_REPLAY (for plugging in recorded data for testing), BER_MEASURE, makes sense. Details TBD.
     device: str
         Interface hardware used for receiving data.
         Current options are "OK" (Opal Kelly XEM 7310) and "UART" (generic UART-USB converters).
         Only "OK" is supported at the moment.
     bitstream: str, optional
         Required when device is "OK".
-        Configuration file to be uploaded to Opal Kelly module.
+        The configuration bitstream file to upload to the Opal Kelly board. This uploads a Manchester decoder HDL and different bitstream files are required to configure different data rates and bit polarity. This is a binary file synthesized using Vivado, and details for generating this file will be provided in later updates.
     port: str, optional
         Required when device is "UART".
         COM port connected to the UART-USB converter.
@@ -42,6 +44,8 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
         Frame width of transferred image. This is used to reconstruct image.
     frame_height: int
         Frame height of transferred image. This is used to reconstruct image.
+    fs: int
+        Framerate of acquired stream
     preamble: str
         32-bit preamble used to locate the start of each buffer. The header and image data follows this preamble.
         This is used as a hex but imported as a string because yaml doesn't support hex format.
@@ -69,24 +73,29 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
         If `LSB`, then the incoming bitstream is in the format that each 32-bit words are bit-wise reversed on its own.
         Furthermore, the order of 32-bit words in the pixel data within the buffer is reversed (but the order of words in the header is preserved).
         Note that this format does not correspond to the usual LSB-first convention and the parameter name is chosen for the lack of better words.
+    show_video : bool, optional
+        Whether the video is showed in "real-time", by default True.
 
     ..todo::
         Takuya - double-check the definitions around blocks and buffers in the firmware and add description.
     """
 
+    mode: Literal["DAQ", "RAW_RECORD", "RAW_REPLAY"] = 'DAQ'
     device: str
     bitstream: Optional[Path]
     port: Optional[str]
     baudrate: Optional[int]
     frame_width: int
     frame_height: int
+    fs: int = 20
     preamble: bytes
     header_len: int
     pix_depth: int = 8
     buffer_block_length: int
     block_size: int
     num_buffers: int
     LSB: Optional[bool]
+    show_video: Optional[bool] = True
 
     @field_validator("preamble", mode="before")
     def preamble_to_bytes(cls, value: Union[str, bytes, int]) -> bytes: