Reservoir Computing for .NET (RCNet)

This project is frozen and no further updates will be available. I leave it here only for inspiration and as a possible source of some implemented functionality. The successor project is EasyML, where you can find the good stuff from here + much more, in a simpler form and higher maturity.

This repository offers a .NET machine learning library primarily focused on prediction and classification of time series. RCNet library implementation is based on the "Reservoir Computing" concept where the two most well-known methods are Echo State Network (ESN) and Liquid State Machine (LSM). The main component of RCNet is called "State Machine". State Machine has quite wide configuration options of the reservoirs and except of both mentioned methods (ESN, LSM) it also supports "hybrid" recurrent reservoirs where hidden neurons having an analog and spiking activation can be synaptically connected. The correct synaptic linkage of hidden neurons having heterogeneous activations is enabled by a specific implementation of a hidden neuron, where each hidden neuron provides analog and spiking output regardless of the type of activation function is used. Dual output of hidden neuron also enables to always offer the full set of predictors on hidden neuron level (very important is the FiringTrace predictor). Firing a spike of the hidden neuron having analog activation is based on defined firing event depending on current and previous value of the activation. Use of spiking based predictor (FiringTrace) dramatically improves classification accuracy of the classical ESN. "Analog spiking" is an unique feature discovered and introduced by this project and thanks that, ESN is now capable to classify time series with high accuracy with relatively low resources and to be fully competetive to other classification algorithms.

The demo application (.NET Core 3.1)

A simple demo application demonstrates the StateMachine performance and it also includes several code examples showing how to use key RCNet components.
Application input data is standardly located in the "Data" sub-folder relative to the location of the executable DemoConsoleApp.exe. Data is expected in csv format and data delimiter can be a tab, semicolon or comma character.

• Continuous feeding regime requires a standard csv format, where the first line contains the names of the data fields and each next line contains the data. Here is an example.
• Patterned feeding regime requires specific logical csv format without colum names (header). Each data line contains values of steady (optional) and repetitive pattern features followed by expected output values at the end. Values of repetitive pattern features can be organized in two ways: groupped [v1(t1),v2(t1),v1(t2),v2(t2),v1(t3),v2(t3)] or sequential [v1(t1),v1(t2),v1(t3),v2(t1),v2(t2),v2(t3)]. Here is an example.

Application has no startup parameters and when started, it shows the root menu.
_Note that in some cases application may write data into "Examples" sub-folder relative to the location of the executable DemoConsoleApp.exe._

1. menu choice: Demonstration of the State Machine performance and configuration options

Application performs sequence of tasks defined in the SMDemoSettings.xml xml file, where each task is defined in the xml element "case" so you can easily insert new task or tune existing one by simple modification of xml content. SMDemoSettings.xml has to be located in the same folder as the executable DemoConsoleApp.exe.

Notes for SMDemoSettings.xml modification

If you modify SMDemoSettings.xml within the open solution in MS Studio, then for the correct connection of xml and xsd it is enough to have SMDemoSettings.xsd and RCNetTypes.xsd open at the same time. If you modify SMDemoSettings.xsd outside the open solution, then you must first add the schemaLocation = "RCNetTypes.xsd" attribute to the "import" element in the SMDemoSettings.xsd schema in the destination directory where the DemoConsoleApp.exe executable is located. Demo application has internally implemented the xml validation so connection of xml and xsd directly on file level is not necessary. But functional external connection of xml and xsd is useful for manual modifications, because consistency checks, annotations, selections from allowed values and checks of the range of entered values are then available instantly.

Time-series classification results comparison

SMDemoSettings.xml currently includes several classification problems from the: Anthony Bagnall, Jason Lines, William Vickers and Eamonn Keogh, The UEA & UCR Time Series Classification Repository, www.timeseriesclassification.com site and State Machine is able to achieve competetive results to the best classification algorithms referenced on that website.

Dataset	State Machine Accuracy	Best Ref. Accuracy	Best Ref. Algorithm
CricketX	82.05%	81.4%	COTE
Worms	83.12%	73.49%	BOSS
BeetleFly	100%	94.85%	BOSS
BirdChicken	100%	98.4%	BOSS
ProximalPhalanx	87.8%	88.09%	ST
Yoga	91.37%	90.99%	BOSS
Libras	92.78%	89.4%	DTWi

2. menu choice: Code examples related to State Machine

Switches to sub-menu with several code examples showing how to configure State Machine manually from scratch or using the StateMachineDesigner component to perform classification or forecasting, then how to train State Machine and verify its performance.

3. menu choice: Code examples related to State Machine

Switches to sub-menu with several code examples showing how to independently use non-recurrent network components. State Machine's readout layer consists of hirarchical structure of non-recurrent network components, but here is shown how to use them as the standalone components with no relation to State Machine.

4. menu choice: Playground

This choice runs the Playground module where you can quickly discover and try to use RCNet components.

RCNet library (.NET Standard 2.0)

Maintenance index	Cyclomatic complexity	Depth of inheritance	Code lines	Executable code lines
82 (green)	7482	3	53958	11949

Components by category:

Math

Component	Description
BasicStat	Implements the basic statistics of sample data.
WeightedAvg	Implements the weighted average.
MovingDataWindow	Implements the moving data window providing additional functions such as statistics, weighted average, etc.
ODENumSolver	Implements a simple numerical solver of the Ordinary Differential Equation(s).
Vector	Implements the vector.
Matrix	Implements the real matrix. It does not support the sparse matrix format.
EVD	Implements the Eigenvalue decomposition of a square matrix.
SVD	Implements the Singular Value decomposition of a matrix.
QRD	Implements the QR decomposition of a matrix.
LUD	Implements the LU (Lowed-Upper) decomposition of a square matrix.
ParamValFinder	Implements a simple iterative error-driven search for the parameter's optimal value.
HurstExpEstim	Implements the Hurst Exponent estimator using the rescaled range analysis.
"RandomValue"	Implements the random value. Supports Uniform, Gaussian, Exponential and Gamma distributions. Here is an extension code
Others	Set of small additional helper components like PhysUnit, Interval, Bitwise, Combinatorics, Discrete,...

XML handling

Component	Description
DocValidator	Implements the xml document loader and validator.

Data generators

Component	Description
PulseGenerator	Implements the constant pulse generator.
MackeyGlassGenerator	Implements the Mackey-Glass generator.
RandomGenerator	Implements the random signal generator.
SinusoidalGenerator	Implements the sinusoidal signal generator.

Data Filtering

Component	Description
BinFeatureFilter	Implements the binary feature filter.
RealFeatureFilter	Implements the real number feature filter.

Data holding

Component	Description
SimpleQueue	Implements a simple FIFO queue template. Supports access to enqueued elements so it can be also used as the moving data window.
DelimitedStringValues	Implements the single row of the delimited string values (csv format).
CsvDataHolder	Implements the holder of csv data and supports its loading and saving to a file.
VectorBundle	Implements a bundle of input and output data vector pairs.
InputPattern	Implements an input pattern. Pattern can be both univariate or multivariate. Supports data resampling (including simple detection of signal begin/end) and amplitude unification.
ResultBundle	Implements a bundle of input, computed and ideal (desired) data vector triplets.

Chainable Input Data Transformations

Component	Description
CDivTransformer	Implements the transformer of values from one input field. Divides the constant by the value from the input field.
DiffTransformer	Implements the transformer of values from one input field. Subtracts the previous value of the input field from the current value.
DivTransformer	Implements the transformer of values from two input fields. Divides the value of the first input field by the value from the second input field.
ExpTransformer	Implements the transformer of values from one input field. Raises the fixed base (by default the Euler's number) to the power of the value from the input field.
LinearTransformer	Implements the transformer of values from two input fields. Computes the linear equation (aX + bY) where "X" is the value from the first input field and "Y" is the value from the second input field. Coefficients "a" and "b" are specified constants.
LogTransformer	Implements the transformer of values from one input field. Computes the input field value's logarithm to the specified base.
MulTransformer	Implements the transformer of values from two input fields. Multiplies the value of the first input field by the value from the second input field.
MWStatTransformer	Implements the transformer of values from one input field. It keeps statistics of the input field recent values and provides specified statistical figure as the transformed value.
PowerTransformer	Implements the transformer of values from one input field. Raises the value from the input field to the power of fixed exponent.
YeoJohnsonTransformer	Implements the transformer of values from one input field. Calculates the Yeo-Johnson transformation. See the wiki pages.

Analog signal to spikes coders

Component	Description
A2SCoderSignalStrength	Implements the signal strength coder. Uses a novel coding algorithm to meet together two important spike-train conditions. Stronger signal leads to earlier first spike and higher spiking frequency.
A2SCoderGaussianReceptors	Implements the Gaussian Receptive Fields coder.
A2SCoderUpDirArrows	Implements the upward signal direction receptors coder. Each receptor is sensitive to upward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The positive difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency.
A2SCoderDownDirArrows	Implements the downward signal direction receptors coder. Each receptor is sensitive to downward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The negative difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency.

Analog activation functions (stateless)

See the wiki pages.

Component	Description
AFAnalogBentIdentity	Implements the Bent Identity activation function.
AFAnalogElliot	Implements the Elliot (aka Softsign) activation function.
AFAnalogGaussian	Implements the Gaussian activation function.
AFAnalogIdentity	Implements the Identity activation function (aka Linear).
AFAnalogISRU	Implements the ISRU (Inverse Square Root Unit) activation function.
AFAnalogLeakyReLU	Implements the LeakyReLU (Leaky Rectified Linear Unit) activation function.
AFAnalogSigmoid	Implements the Sigmoid activation function.
AFAnalogSinc	Implements the Sinc activation function.
AFAnalogSinusoid	Implements the Sinusoid activation function.
AFAnalogSoftExponential	Implements the Soft Exponential activation function.
AFAnalogSoftMax	Implements the Soft Max activation function.
AFAnalogSoftPlus	Implements the Soft Plus activation function.
AFAnalogSQNL	Implements the Square Nonlinearity activation function.
AFAnalogTanH	Implements the Hyperbolic Tangent activation function.

Spiking activation functions (stateful)

See the wiki pages.

Component	Description
AFSpikingAdExpIF	Implements the Adaptive Exponential Integrate and Fire neuron model.
AFSpikingExpIF	Implements the Exponential Integrate and Fire neuron model.
AFSpikingIzhikevichIF	Implements the Izhikevich Integrate and Fire neuron model.
AFSpikingLeakyIF	Implements the Leaky Integrate and Fire neuron model.
AFSpikingSimpleIF	Implements a very simple form of the Integrate and Fire neuron model.

Non-recurrent networks and trainers

Component	Description
FeedForwardNetwork	Implements the Feed Forward network supporting multiple hidden layers.
RPropTrainer	Implements the Resilient Backpropagation iRPROP+ trainer of the feed forward network.
QRDRegrTrainer	Implements the QRD regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation.
RidgeRegrTrainer	Implements the Ridge regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation.
ElasticRegrTrainer	Implements the Elastic linear regression trainer of the feed forward network.. The feed forward network to be trained must have no hidden layers and the Identity output activation.
ParallelPerceptron	Implements the Parallel Perceptron network.
PDeltaRuleTrainer	Implements the P-delta rule trainer of the parallel perceptron network.
TNRNet	Implements the holder of trained non-recurrent network and its error statistics.
TNRNetBuilder	Implements the builder of the trained non-recurrent network.
TNRNetCluster	Implements the cluster of trained non-recurrent networks.
TNRNetClusterBuilder	Implements the builder of trained non-recurrent networks cluster based on the cross-validation approach.
TNRNetClusterChain	Implements the chain of cooperating non-recurrent network clusters.
TNRNetClusterChainBuilder	Implements the builder of the chain of cooperating non-recurrent network clusters.

State Machine components

Component	Description
AnalogInputNeuron	Implements the input analog neuron. The input analog neuron is a special case of the neuron without an activation function. Its purpose is to provide an analog input for the reservoir's synapses.
SpikingInputNeuron	Implements the input spiking neuron. The input spiking neuron is a special case of the neuron without an activation function. Its purpose is to provide a spiking input for the reservoir's synapses.
InputEncoder	Implements a mediation layer between the external input data and the internal reservoirs of the neural preprocessor. Processes the external input data in the natural form and provides it's representation on analog and spiking input neurons for the next processing in the reservoirs. Allows to create new computed input fields using chainable transformations of existing external input fields, as well as adding independently generated input fields using various generators. Supports two input feeding modes: Continuous and Patterned. The Continuous feeding mode processes an input vector as the variable(s) values at the single time-point T. The Patterned feeding mode processes an input vector as an alone input pattern consisting of a time series of the variable(s) values. Supports three ways how to represent an analog value as the spikes: Horizontal, Vertical or Forbidden. The Horizontal way of coding means a simultaneous activity of the neuronal population where every input field is coded by several spiking input neurons (a horizontal spike-train). It is fast, it leads to a single computation cycle of the reservoirs per the input field value. The Vertical way of coding means that the input field value is coded as a spike-train on a single spiking input neuron. It is slower, it leads to multiple computation cycles of the reservoirs according to the spike-train length. The Forbidden way of coding means there is no coding of an analog value as the spikes. It is fast, it leads to a single computation cycle of the reservoirs per the input field value and it does not utilize any spiking input neuron(s).
Synapse	Implements the synapse. Supports the signal delaying and the short-term plasticity.
PredictorsProvider	Implements the provider of the unified set of computed predictors.
HiddenNeuron	Implements the hidden neuron. Supports engagement of both analog and spiking activation functions and provides unified set of available predictors.
ReservoirInstance	Implements the reservoir.
NeuralPreprocessor	Implements the neural preprocessor supporting multiple reservoirs.
ReadoutUnit	Implements the readout unit of the readout layer. It can do the Forecast or Classification.
OneTakesAllGroup	Implements the "One Takes All" group of readout units. Supports basic decision-making based directly on the results of readout units and also more advanced decision-making based on the result of a dedicated chain of network clusters.
PredictorsMapper	Implements the mapper of specific predictors to readout units.
ReadoutLayer	Implements the readout layer consisting of trained readout units (and "One Takes All" groups).

State Machine

The main serializable StateMachine component encapsulates the NeuralPreprocessor and ReadoutLayer components into the single component and adds support for routing of specific predictors and input fields to the specific readout units. State Machine also allows to bypass NeuralPreprocessor and to use input data directly as a predictors for the readout layer.

Setup

Each executive component that makes up StateMachine (including StateMachine itself) has its own related settings class providing configuration, which is required by the executive component's constructor.
Each settings class can be instantiated manually from scratch or from a xml element encapsulating all parameters. RCNetTypes.xsd defines all xml elements used in settings classes constructors.
Each settings class also implements the GetXml method so it can be instantiated from scratch and the initialization xml element can be exported by calling the GetXml method (and stored for later use). Using xml constructors is generally preferable because the initialization xml can be edited without the need to modify source code of the manual setup.
To make things easier, RCNet also implements helper component StateMachineDesigner for easier setup of simple ESN and LSM StateMachine configurations from the code (see the examples in demo application).

Documentation and contact

More detailed documentation will be gradually published here. Questions, ideas, suggestions for improvement and constructive comments are welcome at my email address oldrich.kozelsky@email.cz or newly you can use github discussions.