Neural noise

[Pugavkomm]

At the moment, there is a lack of the ability to include neural noise in the neuron model. Usually, white Gaussian noise is added to spike neurons with small standard deviation and zero mean. Usually noise is added as an additional term of the membrane potential.

[Jegp]

Hi @Pugavkomm! Thanks for bringing this to our attention.
Do you mean noise for the input? Or as a post-hoc manipulation of the membrane voltage?

Do you have a link/reference to an implementation or a model describing this?

Concerning input noise, I think this could be added as a separate layer, which I would imagine being a little cleaner than adding it directly to the neurons (found this example https://discuss.pytorch.org/t/writing-a-simple-gaussian-noise-layer-in-pytorch/4694/8). But maybe you're talking about other dynamics?

[cpehle]

This is a good idea, I've considered integrating with https://github.com/google-research/torchsde. It would be relatively straightforward to implement a simplified version of that as well.

[Pugavkomm]

Reference
@Jegp, I meant noise in the neurons themselves. A model with such a noise can be viewed:
Spiking recurrent neural networks represent task-relevant neural sequences in rule-dependent computation (eq. 1a).

Explanation
Such noise simulates real neural noise and also allows the system to operate even with zero inputs, which can help implement complex memory systems.

Implementation
Since this noise is inherently an additional current, I think that somehow it can be implemented through some layer. But it is important to note that it is fed directly to the i-th neuron.

[Jegp]

Right, so it'll be a 1:1 mapping? I think that could be done as a separate module fairly easily. There is also the option of creating some kind of parameterized module within a neuron layer, but that quickly explodes in complexity if the goal is to implement it for all types of neurons.

[Pugavkomm]

Yes, it will. A 1:1 mapping + noise is a good idea.

[cpehle]

I think that makes a lot of sense, a clean way to add this would be to implement a naive https://en.wikipedia.org/wiki/Euler–Maruyama_method stepper, and then rewrite the time integration of the neuron models in terms of this.

[cpehle]

The advantage of this approach would be that it a) would be to eliminate the duplication of the solver implementation and that b) Euler and Euler-Maruyama are compatible by just setting the noise term to the "zero" function.