Docs update

README.md

@@ -24,7 +24,9 @@ Example notebooks are provided in the `docs/source/tutorials` folder and in the
 **If you use Kilosort1-4, please cite the [paper](https://www.biorxiv.org/content/10.1101/2023.01.07.523036v1):**     
 Pachitariu, M., Sridhar, S., Pennington, J., & Stringer, C. (2024). Spike sorting with Kilosort4.
 
-**Warning** :bangbang:: There was a bug in Kilosort 2.5 and 3 (but not 1,2 and 4) which caused fewer spikes to be detected in ~7ms periods at batch boundaries (every 2.1866s, issue #594). The patch0 releases fix this bug. It is also advised not to manually change the batch size in any Matlab-version of Kilosort (1-3). 
+**Warning** :bangbang:: There was a bug in Kilosort 2.5 and 3 (but not 1,2 and 4) which caused fewer spikes to be detected in ~7ms periods at batch boundaries (every 2.1866s, issue #594). The patch0 releases fix this bug. It is also advised not to manually change the batch size in any Matlab-version of Kilosort (1-3).
+
+**Note on multi-shank probes** : We are aware of some issues with sorting data from probes with multiple shanks. See [documentation here](https://kilosort.readthedocs.io/en/latest/multi_shank.html) for recommended workarounds until the code is updated to handle these probes.
 
 ## Installation
 
@@ -68,10 +70,20 @@ This [video](https://www.youtube.com/watch?v=gsixIQYvj3U) has step-by-step insta
 4. Hit `LOAD`. The data should now be visible.
 5. Hit `Run`. This will run the pipeline and output the results in a format compatible with Phy, the most popular spike sorting curating software.
 
-There is a warning that will always pop up when running Kilosort and/or using the BinaryFile class, but it's nothing to worry about:
+### Debugging qt.qpa.plugin error
+
+Some users have encountered the following error (or similar ones with slight variations) when attempting to launch the Kilosort4 GUI:
 ```
-UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\utils\tensor_numpy.cpp:205.)
+QObject::moveToThread: Current thread (0x2a7734988a0) is not the object's thread (0x2a77349d4e0).
+Cannot move to target thread (0x2a7734988a0)
+
+qt.qpa.plugin: Could not load the Qt platform plugin "windows" in "<FILEPATH>" even though it was found.
+This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
+
+Available platform plugins are: minimal, offscreen, webgl, windows.
 ```
+This is not specific to Kilosort4, it is a general problem with PyQt (the GUI library we chose to develop with) that doesn't appear to have a single cause or fix. We are looking into alternatives for the GUI code to avoid this issue, but in the meantime please check [issue 597](https://github.com/MouseLand/Kilosort/issues/597) and [issue 613](https://github.com/MouseLand/Kilosort/issues/613) for some suggested solutions.
+
 
 ## Integration with Phy GUI
 

docs/first_run.rst

@@ -12,6 +12,7 @@ Launching the GUI
 5. Hit :code:`Run`. This will run the pipeline and output the results in a
    format compatible with Phy, the most popular spike sorting curating software.
 
+See :ref:`gui_guide` for more detailed instructions.
 
 Using the API
 -------------

docs/gui_guide.rst

@@ -0,0 +1,79 @@
+.. _gui_guide:
+
+How to use the GUI
+==================
+This page explains how to use the basic functions of Kilosort4 GUI, in order of the steps you would take to start sorting.
+
+
+Select data
+-----------
+Start by selecting a binary data file (.bin, .bat, .dat, or .raw) to load, by clicking on the "Select Binary File" button near the top-left of the GUI. This will open a file dialog that will populate the neighboring text field after a file is selected. You can also paste a filepath directly into the text field.
+
+.. image:: https://www.kilosort.org/static/images/gui_select_binary.png
+   :width: 600
+
+
+Convert from other formats (optional)
+-------------------------------------
+If your data is not in one of the supported formats listed in the previous step, you can click the "Convert to Binary" button to open the data conversion tool. Using this tool requires the `SpikeInterface <https://spikeinterface.readthedocs.io/en/latest/>`_ package. To convert your data, you will need to select either a file OR a folder (not both) using the, choose the filetype from the list of supported options, and select the dtype. Then, click "Convert to Binary" (recommended) to copy the data to a new .bin file. Alternatively, you can click "Load As Wrapper" to use the data without copying to a new file, but you will not be able to view results in Phy.
+
+.. image:: https://www.kilosort.org/static/images/gui_convert_data.png
+   :width: 600
+
+Note: this tool is only intended to cover the most common formats and options for getting your data into Kilosort. If you don't see your data format or if you run into errors related to extra options that aren't in our GUI, we recommend using SpikeInterface directly to convert your data. Their `documentation here <https://spikeinterface.readthedocs.io/en/latest/modules_gallery/core/plot_1_recording_extractor.html>`_ shows an example of how to save recording traces to .raw format using their own tools.
+
+
+Choose where to save results
+----------------------------
+After a binary file is selected, the GUI will automatically populate the text field under "Select Results Dir." with the same path as your binary file, but ending in a "/kilosort4" directory instead of the binary file. If you wanted to change this, you can click the "Select Results Dir." button to open another file dialog, or edit the text field.
+
+.. image:: https://www.kilosort.org/static/images/gui_results_dir.png
+   :width: 600
+
+
+Select a probe
+--------------
+To select a probe, click the drop-down menu just below "Select Probe Layout." The list will include some default Neuropixels probe layouts. If you've already created your own probe file (.mat, .prb, or .json), you can select "other..." to open a file dialog and navigate to it.
+
+.. image:: https://www.kilosort.org/static/images/gui_select_probe.png
+   :width: 600
+
+If you need to create a new probe layout, select "[new]" to open the probe creation tool. Values for 'x-coordinates' and 'y-coordinates' need to be in microns, and can be specified with numpy expressions. For example, a 1-shank linear probe with 4 channels could have `np.ones(4)` in the 'x-coordinates' field instead of `1, 1, 1, 1`. Each field (except name) must have the same number of elements, corresponding to the number of ephys channels in the data. WHen you are finished setting the values, click "Check" to verify that your inputs are valid. If they are not, an error message will be displayed. Otherwise, the "OK" button will become clickable, which will save the probe to the Kilosort4 probes directory.
+
+.. image:: https://www.kilosort.org/static/images/gui_make_probe.png
+   :width: 600
+
+After a probe is selected, you can click "Preview Probe" to see a visualization and verify that the probe geometry looks correct.
+
+
+Load the data
+-------------
+After you select a probe, the GUI will attempt to automatically determine the correct value for 'number of channels.' Make sure this correctly reflects the number of channels in your datafile, including non-ephys channels. For example, Neuropixels 1 probes output data with 385 channels. Only 384 of those are the ephys data used for sorting, but 'number of channels' should still be set to 385. You may also need to change the dtype of the data (int16 by default) or the sampling rate (30000hz by default). Additionally, you can choose which computing device. By default, the GUI will select the first CUDA GPU detected by PyTorch, or CPU if no GPU is detected.
+
+When you are satisfied with these settings, click "LOAD" at the top left of the GUI to load the data.
+
+.. image:: https://www.kilosort.org/static/images/gui_data_settings.png
+   :width: 600
+
+
+Run spike sorting
+-----------------
+After loading the data, a heatmap will appear on the right half of the GUI showing a preprocessed version of the data. You can click "raw" at the bottom right to view the data without preprocessing applied. Make sure the data looks like what you expect, including the correct number of seconds along the bottom of the GUI. A common error to look for is diagonal lines in the heatmap, which usually indicates that 'number of channels' does not match the data. When everything looks good, click "Run" near the bottom left to begin spike sorting. When sorting is finished, the results will be saved to the directory indicated under "Select Results Dir."
+
+.. image:: https://www.kilosort.org/static/images/gui_run_sorting.png
+   :width: 600
+
+If you run into errors or the results look strange, you may need to tweak some of the other settings. A handful are shown below 'number of channels' and 'sampling frequency,' or you can click "Extra settings" to open a new window with more options. Mousing over the name of a setting for about half a second will show a description of what the setting does, along with information about which values are allowed. For more detailed suggestions, see :ref:`parameters`
+
+.. image:: https://www.kilosort.org/static/images/gui_extra_settings.png
+   :width: 600
+
+If you're still not sure how to proceed, check `issues page on our github <https://github.com/MouseLand/Kilosort/issues>`_ for more help.
+
+
+Resetting the GUI
+-----------------
+If the GUI gets stuck on a loading animation or some other odd behavior, try clicking on "Reset GUI" near the top right, which should reset it to the state shown in the first step on this page. If you want to make sure all previous settings are deleted, you can also click "Clear Cache" and then close and re-open the GUI.
+
+.. image:: https://www.kilosort.org/static/images/gui_reset.png
+   :width: 600

docs/index.rst

@@ -13,6 +13,9 @@ please see our pre-print on `BioArxiv <https://www.biorxiv.org/content/10.1101/2
 
    installation
    first_run
+   gui_guide
+   parameters
+   multi_shank
    hardware
    drift
 

docs/multi_shank.rst

@@ -0,0 +1,9 @@
+.. _multi_shank:
+
+Using probes with multiple shanks
+==================================
+Currently, Kilosort4 does not process shanks separately. Until this is added, we recommend changing your probe layout to artificially stack the shanks in a single column with a bit of vertical space (50-100um) between each shank.
+
+If that option isn't feasible for some reason, you can also try adjusting the `min_template_size` and/or `dminx` parameters in the GUI, or in the settings argument for `run_kilosort` if you're using the API. Setting `dminx` to around half the total width of the probe seems to be a good starting point, and you can adjust from there.
+
+See `issue 606 <https://github.com/MouseLand/Kilosort/issues/606>`_ and `issue 617 <https://github.com/MouseLand/Kilosort/issues/617>`_ for additional context.

docs/parameters.rst

@@ -0,0 +1,51 @@
+.. _parameters:
+
+When to adjust default settings
+===============================
+This page will give suggestions for when to change specific settings and why. Not every configurable setting will be covered here, but all settings include a short description in `kilosort/parameters.py <https://github.com/MouseLand/Kilosort/blob/main/kilosort/parameters.py>`_. The same description can also be seen in the GUI by mousing over the name of a setting for about half a second.
+
+
+``n_chan_bin`` (number of channels)
+-----------------------------------
+This should reflect the total number of channels in the binary file, `including non-ephys channels not used for sorting`. If you load your data in the GUI and see repeating diagonal patterns in the data, you probably need to change this value.
+
+
+``batch_size``
+--------------
+This sets the number of samples included in each batch of data to be sorted, with a default of 60000 corresponding to 2 seconds for a sampling rate of 30000. For probes with fewer channels (say, 64 or less), increasing ``batch_size`` to include more data may improve results because it allows for better drift estimation (more spikes to estimate drift from). 
+
+
+``nblocks``
+-----------
+This is the number of sections the probe is divided into when performing drift correction. The default of ``nblocks = 1`` indicates rigid registration (the same amount of drift is applied to the entire probe). If you see different amounts of drift in your data depending on depth along the probe, increasing ``nblocks`` will help get a better drift estimate. ``nblocks=5`` can be a good choice for single-shank Neuropixels probes. For probes with fewer channels (around 64 or less) or with sparser spacing (around 50um or more between contacts), drift estimates are not likely to be accurate, so drift correction should be skipped by setting ``nblocks = 0``.
+
+
+``Th_universal`` and ``Th_learned``
+-----------------------------------
+These control the threshold for spike detection when applying the universal and learned templates, respectively (loosely similar to Th(1) and Th(2) in previous versions). If few spikes are detected, or if you see neurons disappearing and reappearing over time when viewing results in Phy, it may help to decrease ``Th_learned``. To detect more units overall, it may help to reduce ``Th_universal``. Try reducing each threshold by 1 or 2 at a time.
+
+
+``tmin`` and ``tmax``
+---------------------
+This sets the start and end of data used for sorting (in seconds). By default, all data is included. If your data contains recording artifacts near the beginning or end of the session, you can adjust these to omit that data. "inf" and "np.inf" can be used for tmax to indicate end of session in the GUI and API respectively. 
+
+
+``dmin`` and ``dminx``
+----------------------
+These adjust the vertical and lateral spacing, respectively, of the universal templates used during spike detection, as well as the vertical size of channel neighborhoods used for clustering. By default, Kilosort will attempt to determine a good value based on the median distance between contacts, which tends to work for Neuropixels-like probes. However, if contacts are irregularly spaced, you may need to specify these manually.
+
+``min_template_size``
+---------------------
+This sets the standard deviation of the smallest Gaussian spatial envelope used to generate universal templates, with a default of 10 microns. You may need to increase this for probes with wider spaces between contacts.
+
+
+``nearest_chans`` and ``nearest_templates``
+-------------------------------------------
+This is the number of nearest channels and template locations, respectively, used when assigning templates to spikes during spike detection. ``nearest_chans`` cannot be larger than the total number of channels on the probe, so it will need to be reduced for probes with less than 10 channels. ``nearest_templates`` does not have this restriction. However, for probes with around 64 channels or less and sparsely spaced contacts, decreasing ``nearest_templates`` to be less than or equal to the number of channels helps avoid numerical instability.
+
+
+``duplicate_spike_bins``
+------------------------
+After sorting has finished, spikes that occur within this number of bins of each other, from the same unit, are assumed to be artifacts and removed. The default of 15 bins corresponds to 0.5ms for a sampling rate of 30000hz. If your sampling rate is different, you may need to increase or decrease this accordingly. If you see otherwise good neurons with large peaks around 0ms when viewing correlograms in Phy, increasing this value can help remove those artifacts.
+
+**Warning!!!** Do not increase this value beyond 0.5ms as it will interfere with the ACG and CCG refractory period estimations (which normally ignores the central 1ms of the correlogram).

kilosort/parameters.py

@@ -171,7 +171,8 @@
         'description': 
             """
             Sample index for aligning waveforms, so that their minimum 
-            or maximum value happens here. Default of 20.
+            or maximum value happens here. Defaults to 
+            `int(20 * settings['nt']/61)`.
             """
     },