Opening the black box of Deep Neural Networks via Information: reimplementation

Group project by Leonid Matyushin, Alexander Lyzhov, Alexey Ustimenko, Philip Blagoveschensky, Nikolay Skuratov

Code parts

• Testing different step size policies

• Exploring a minimalistic 3-neuron setup

• Tanh/ReLU information planes

• Compression of irrelevant information

Resources used

Papers and discussions:

• Opening the black box of Deep Neural Networks via Information

• On the Information Bottleneck Theory of Deep Learning

• Openreview discussion of opponent paper

• Blogpost critiquing the original paper

• Acolyer blog discussions: 1, 2

• Webema blogpost

Code:

• Original implementation

• Simplified implementation

• Opponent paper implementation