merge main

.readthedocs.yml

@@ -17,9 +17,9 @@ sphinx:
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.9"
+    python: "3.11"
   jobs:
     post_create_environment:
       - pip install poetry
     post_install:
-      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs
+      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --extras docs

README.md

@@ -1,8 +1,15 @@
 # miniscope-io
 
-(Demonstration project for a lab workshop on making modular, testable python code.
-Functionality not guaranteed
--<3 jonny)
+![PyPI - Version](https://img.shields.io/pypi/v/miniscope-io)
+[![Documentation Status](https://readthedocs.org/projects/miniscope-io/badge/?version=latest)](https://miniscope-io.readthedocs.io/en/latest/?badge=latest)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/miniscope-io)
+![PyPI - Status](https://img.shields.io/pypi/status/miniscope-io)
+
+Generic i/o interfaces for miniscopes.
+
+In alpha - Fuller README forthcoming.
+
+See the docs for now: https://miniscope-io.readthedocs.io/
 
 
 

docs/api/models/index.md

@@ -20,5 +20,6 @@ buffer
 config
 mixins
 models
+sdcard
 stream
 ```

docs/api/stream_daq.md

@@ -2,7 +2,7 @@
 This module is a data acquisition module that captures video streams from Miniscopes based on the `Miniscope-SAMD-Framework` firmware. The firmware repository will be published in future updates but is currently under development and private.
 
 ## Command
-After [installation](./installation.md) and customizing [configurations](#configuration) if necessary, run the following command in this Git repository to start the data acquisition process:
+After [installation](../guide/installation.md) and customizing [configurations](stream-daq-config) if necessary, run the following command in this Git repository to start the data acquisition process:
 ```bash
 >>> streamDaq -c path/to/config/yaml/file.yml
 Connected to XEM7310-A75
@@ -18,6 +18,7 @@ A window displaying the image transferred from the Miniscope should pop up. Addi
 - **Supported Operating Systems:** MacOS or Ubuntu PC (To do: specify version)
 - **Imaging hardware:** Miniscope based on the `Miniscope-SAMD-Framework` firmware. Hardware modules for feeding in data into the data capture hardware are also needed but these will be specified in future updates.
 
+(stream-daq-config)=
 ## Configuration
 A YAML file is used to configure this module. The configuration needs to match the imaging and data capture hardware for proper operation. This file is used to set up hardware, define data formats, and set data preambles. The contents of this YAML file will be parsed into a model [miniscope_io.models.stream](../api/models/stream.md), which then configures the Stream DAQ.
 

docs/meta/changelog.md

@@ -1,6 +1,10 @@
 # Changelog
 
-## 0.1.9 - 24-06-21
+## 0.2
+
+### 0.2.0 - 24-06-21
+
+StreamDaq enhancements and testing
 
 - https://github.com/Aharoni-Lab/miniscope-io/pull/26
 
@@ -42,7 +46,9 @@ CI:
 - caching dependency installs
 - not using pytest-emoji, it was always annoying
 
-## 0.1.8 - 24-06-16
+## 0.1
+
+### 0.1.8 - 24-06-16
 
 - https://github.com/Aharoni-Lab/miniscope-io/pull/21
 - https://github.com/Aharoni-Lab/miniscope-io/pull/15
@@ -57,7 +63,7 @@ New features:
 
 Note: Version 0.1.7 was skipped accidentally and does not exist.
 
-## 0.1.6 - 24-04-09
+### 0.1.6 - 24-04-09
 
 - https://github.com/Aharoni-Lab/miniscope-io/pull/14
 
@@ -67,7 +73,7 @@ New features:
   initial version of code is present in `stream_daq.py`
 - Vendored opalkelly device drivers - see `devices` and `vendor`
 
-## 0.1.5 - 23-09-03
+### 0.1.5 - 23-09-03
 
 - https://github.com/Aharoni-Lab/miniscope-io/pull/9
 - https://github.com/Aharoni-Lab/miniscope-io/pull/10
@@ -86,7 +92,7 @@ Code structure:
 Tests:
 - Run tests on macos and windows
 
-## 0.1.4 - 23-09-03
+### 0.1.4 - 23-09-03
 
 https://github.com/Aharoni-Lab/miniscope-io/pull/8
 
@@ -104,16 +110,16 @@ Reverted:
 
 
 
-## 0.1.1 - 23-07-13
+### 0.1.1 - 23-07-13
 
-### Additions
+#### Additions
 
 - Added {class}`~miniscope_io.exceptions.EndOfRecordingException` when attempting to read past last frame
 - Added {attr}`~miniscope_io.io.SDCard.frame_count` property inferred from the number of buffers and buffers per frame
 - Return `self` when entering {class}`~.SDCard` context
 - Optionally return {class}`~miniscope_io.sd.DataHeader`s from frame when reading
 
-### Bugfixes
+#### Bugfixes
 
 - Index the position of the 0th frame in {attr}`~.SDCard.positions`
 - reset internal frame counter to 0 when exiting context

miniscope_io/devices/mocks.py

@@ -0,0 +1,71 @@
+"""
+Hardware mocks for devices.
+
+Used in testing, but kept in-package since for now some devices
+need modifications to their source (and we can't import from tests)
+
+Not to be considered part of the public interface of miniscope-io <3
+"""
+
+# ruff: noqa: D102
+
+import os
+from pathlib import Path
+from typing import Dict
+
+from miniscope_io.exceptions import EndOfRecordingException
+
+
+class okDevMock:
+    """
+    Mock class for :class:`~miniscope_io.devices.opalkelly.okDev`
+    """
+
+    DATA_FILE = None
+    """
+    Recorded data file to use for simulating read.
+    
+    Set as class variable so that it can be monkeypatched in tests that
+    require different source data files.
+    
+    Can be set using the ``PYTEST_OKDEV_DATA_FILE`` environment variable if 
+    this mock is to be used within a separate process.
+    """
+
+    def __init__(self, serial_id: str = ""):
+        self.serial_id = serial_id
+        self.bit_file = None
+
+        self._wires: Dict[int, int] = {}
+        self._buffer_position = 0
+
+        # preload the data file to a byte array
+        if self.DATA_FILE is None:
+            if os.environ.get("PYTEST_OKDEV_DATA_FILE", False):
+                # need to get file from env variables here because on some platforms
+                # the default method for creating a new process is "spawn" which creates
+                # an entirely new python session instead of "fork" which would preserve
+                # the classvar
+                okDevMock.DATA_FILE = Path(os.environ.get("PYTEST_OKDEV_DATA_FILE"))
+            else:
+                raise RuntimeError("DATA_FILE class attr must be set before using the mock")
+
+        with open(self.DATA_FILE, "rb") as dfile:
+            self._buffer = bytearray(dfile.read())
+
+    def uploadBit(self, bit_file: str) -> None:
+        assert Path(bit_file).exists()
+        self.bit_file = bit_file
+
+    def readData(self, length: int, addr: int = 0xA0, blockSize: int = 16) -> bytearray:
+        if self._buffer_position >= len(self._buffer):
+            # Error if called after we have returned the last data
+            raise EndOfRecordingException("End of sample buffer")
+
+        end_pos = min(self._buffer_position + length, len(self._buffer))
+        data = self._buffer[self._buffer_position : end_pos]
+        self._buffer_position = end_pos
+        return data
+
+    def setWire(self, addr: int, val: int) -> None:
+        self._wires[addr] = val

miniscope_io/models/stream.py

@@ -30,14 +30,6 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
 
     Parameters
     ----------
-    mode: str
-        Operation modes. Maybe having
-
-            - DAQ (normal operation),
-            - RAW_RECORD (for getting testdata), and
-            - RAW_REPLAY (for plugging in recorded data for testing),
-
-        makes sense. Details TBD.
     device: str
         Interface hardware used for receiving data.
         Current options are "OK" (Opal Kelly XEM 7310) and "UART" (generic UART-USB converters).
@@ -106,11 +98,10 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
 
     """
 
-    mode: Literal["DAQ", "RAW_RECORD", "RAW_REPLAY"] = "DAQ"
-    device: str
-    bitstream: Optional[Path]
-    port: Optional[str]
-    baudrate: Optional[int]
+    device: Literal["OK", "UART"]
+    bitstream: Optional[Path] = None
+    port: Optional[str] = None
+    baudrate: Optional[int] = None
     frame_width: int
     frame_height: int
     fs: int = 20
@@ -120,8 +111,8 @@ class StreamDaqConfig(MiniscopeConfig, YAMLMixin):
     buffer_block_length: int
     block_size: int
     num_buffers: int
-    LSB: Optional[bool]
-    show_video: Optional[bool] = True
+    LSB: bool = True
+    show_video: bool = True
 
     @field_validator("preamble", mode="before")
     def preamble_to_bytes(cls, value: Union[str, bytes, int]) -> bytes:

miniscope_io/plots/headers.py

@@ -2,9 +2,16 @@
 Plot headers from :class:`.SDCard`
 """
 
-import matplotlib.pyplot as plt
 import pandas as pd
 
+try:
+    import matplotlib.pyplot as plt
+except ImportError as e:
+    raise ImportError(
+        "matplotlib is not a required dependency of miniscope-io, "
+        "install it with the miniscope-io[plot] extra or manually in your environment :)"
+    ) from e
+
 
 def buffer_count(headers: pd.DataFrame, ax: plt.Axes) -> plt.Axes:
     """

miniscope_io/stream_daq.py

@@ -16,11 +16,11 @@
 from bitstring import Array, BitArray, Bits
 
 from miniscope_io import init_logger
+from miniscope_io.devices.mocks import okDevMock
 from miniscope_io.exceptions import EndOfRecordingException, StreamReadError
 from miniscope_io.formats.stream import StreamBufferHeader
 from miniscope_io.models.buffer import BufferHeader
 from miniscope_io.models.stream import StreamBufferHeaderFormat, StreamDaqConfig
-from tests.mock.opalkelly import okDevMock
 
 HAVE_OK = False
 ok_error = None
@@ -198,6 +198,7 @@ def _fpga_recv(
         serial_buffer_queue: multiprocessing.Queue,
         read_length: int = None,
         pre_first: bool = True,
+        capture_binary: Optional[Path] = None,
     ) -> None:
         """
         Function to read bitstream from OpalKelly device and store buffer in `serial_buffer_queue`.
@@ -217,6 +218,8 @@ def _fpga_recv(
             and is an integer multiple of 16 bytes (as recommended by OpalKelly).
         pre_first : bool, optional
             Whether preamble/header is returned at the beginning of each buffer, by default True.
+        capture_binary: Path, optional
+            save binary directly from the ``okDev`` to the supplied path, if present.
 
         Raises
         ------
@@ -259,12 +262,10 @@ def _fpga_recv(
                 self.terminate.value = True
                 break
 
-            if self.config.mode == "RAW_RECORD":
-                # Not sure where to define this because we don't want to overwrite.
-                fpga_raw_path = "data/fpga_raw.bin"
-                with open(fpga_raw_path, "wb") as file:
+            if capture_binary:
+                with open(capture_binary, "ab") as file:
                     file.write(buf)
-                self.terminate.value = True
+
             dat = BitArray(buf)
             cur_buffer = cur_buffer + dat
             pre_pos = list(cur_buffer.findall(pre))
@@ -461,6 +462,7 @@ def capture(
         read_length: Optional[int] = None,
         video: Optional[Path] = None,
         video_kwargs: Optional[dict] = None,
+        binary: Optional[Path] = None,
     ) -> None:
         """
         Entry point to start frame capture.
@@ -470,12 +472,15 @@ def capture(
         source : Literal[uart, fpga]
             Device source.
         read_length : Optional[int], optional
-            Passed to :function:`~miniscope_io.stream_daq.stream_daq._fpga_recv` when
+            Passed to :func:`~miniscope_io.stream_daq.stream_daq._fpga_recv` when
             `source == "fpga"`, by default None.
         video: Path, optional
             If present, a path to an output video file
-        video_options: dict, optional
+        video_kwargs: dict, optional
             kwargs passed to :meth:`.init_video`
+        binary: Path, optional
+            Save raw binary directly from ``okDev`` to file, if present.
+            Note that binary is captured in *append* mode, rather than rewriting an existing file.
 
         Raises
         ------
@@ -507,10 +512,7 @@ def capture(
             self.logger.debug("Starting fpga capture process")
             p_recv = multiprocessing.Process(
                 target=self._fpga_recv,
-                args=(
-                    serial_buffer_queue,
-                    read_length,
-                ),
+                args=(serial_buffer_queue, read_length, True, binary),
             )
         else:
             raise ValueError(f"source can be one of uart or fpga. Got {source}")