README.md

Bristol

Parallel random matrix tools and random matrix theory deep learning applications. Generate matrices from Circular Unitary Ensemble (CUE), Circular Ortogonal Ensemble (COE) and Circular Symplectic Ensemble (CSE). Additional spectral analysis utilities are also implemented, such as computation of spectral density and spectral ergodicity for complexity of deep learning architectures.

Features

• Generation of Circular Ensembles: CUE, COE and CSE.
• Random matrices: Reproducibility both in serial and parallel processing.
• Eigenvalue Spectra, spectral densitiy.
• Kullbach-Leibler divergence and spectral ergodicity measure functionality.
• Cascading Periodic Spectral Ergodicity (cPSE) : This is a complexity measure and could also detect when to stop addinig more layers.

Installation

Install with pip from pypi, as of 0.2.14 we prefer Python > 3.9

pip install bristol

Running tests

bash run_tests.py

To use the latest development version

pip install -upgrade git+https://github.com/msuzen/bristol.git

Documentation

Complexity of a deep learning model: cPSE

Vanilla case

In the vanilla case a list of matrices that are representative of ordered set of weight matrices can be used to compute cPSE over layers. As an examples:

from bristol import cPSE
import numpy as np
np.random.seed(42)
matrices = [np.random.normal(size=(64,64)) for _ in range(10)]
(d_layers, cpse) = cPSE.cpse_measure_vanilla(matrices)

Even for set of Gaussian matrices, d_layers decrease. Note that different layer types should be converted to a matrix format, i.e., CNNs to 2D matrices. See the main paper.

When should I stop adding more layers in deep learning?

d_layers is a decreasing vector, it will saturate at some point, that point is where adding more layers won’t improve the performance. This is data, learning and architecture independent measure.

For torch models

You need to put your model as pretrained model format of PyTorch. An example for vgg, and use cPSE.cpse_measure function simply:

from bristol import cPSE
import torchvision.models as models
netname = 'vgg11'
pmodel = getattr(models, netname)(pretrained=True)
(d_layers, cpse) = cPSE.cpse_measure(pmodel)

This would give cpse a single number expressing the complexity of your network and d_layers evolution of periodic spectral ergodicity withing layers as a vector, order matters.

Random Stream Chunking

Package employs a technique called random stream chunking to ensure reproducibility
between parallel and serial runs. A blog post explaining this:
Exact reproducibility of stochastic simulations for parallel and serial algorithms simultaneously Random Stream Chunking

Prototype notebooks

• Basics of circular ensembles ipynb.

• Computing spectral ergodicity for generated matrices ipynb. This is to reproduce the main figure from arXiv:1704.08693.

• The concept of cascading periodic ergodicity (cPSE) ipynb This is only to reproduce paper's results from arXiv:1911.07831.

• Empirical deviations of semicircle law in mixed-matrix ensembles,
  M. Suezen, hal-03464130 | ipynb Reproduces the work with the same title.

Contact

• Please create an issue for any type of questions or contact msuzen.

References

• Berry, M V & Pragya Shukla 2013, Hearing random matrices and random waves, New. J. Phys. 15 013026 (11pp) berry456

• Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices, Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1704.08693

• Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search, Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1911.07831

• Empirical deviations of semicircle law in mixed-matrix ensembles,
  M. Suezen, hal-03464130 | ipynb Reproduces the work with the same title.

Citation

If you use the ideas or tools from this package please do cite our manuscripts.

@article{suezen2021a,
    title={Empirical deviations of semicircle law in mixed-matrix ensembles},
    author={Mehmet Süzen},
    year={2021},
    eprint={hal-03464130},
    url={https://hal.archives-ouvertes.fr/hal-03464130}
}

@article{suezen2019a,
    title={Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search},
    author={Mehmet Süzen and Joan J. Cerdà and Cornelius Weber},
    year={2019},
    eprint={1911.07831},
    archivePrefix={arXiv},
    primaryClass={stat.ML}
}

@article{suezen2017a,
    title={Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices},
    author={Mehmet Süzen and Cornelius Weber and Joan J. Cerdà},
    year={2017},
    eprint={1704.08303},
    archivePrefix={arXiv},
    primaryClass={stat.ML}
}