Highway Networks @ Tensorflow

Implement Highway Networks based on Tensorflow 0.12.1

• Paper:
  • Highway Networks
  • Training Very Deep Networks
• Dataset:
  • MNIST

Version

• CNN: Basic Convolutional Neural Networks implementation
• Highway-FCN: Fully-connected Highway Networks implementation
• Highway-CNN: Convolutional Highway Networks implementation

Execution

1. Install Python, TensorFlow and other dependencies:

# Ubuntu/Linux 64-bit
$ sudo apt-get install python-pip python-dev

# Mac OS X
$ sudo easy_install pip
$ sudo easy_install --upgrade six

# CPU only
$ pip install tensorflow

# GPU enabled
$ pip install tensorflow-gpu

You can also follow the official guide.

2. Clone the repository:

$ git clone https://github.com/wujysh/highway-networks-tensorflow.git && cd highway-networks-tensorflow

3. Run:

$ python {Version}.py

Replace {Version} with cnn, highway_fcn or highway_cnn.

Result

• Training convenience

• Accuracy

  • CNN: 92.14%
  • Highway-FCN: 96.77%
  • Highway-CNN: 97.68%

Reproduce Recurrent Highway Networks

• Paper: Recurrent Highway Networks
• Code: Github
• Experiment: (Perplexity - Epoch)

Acknowledgement

• Very Deep Learning with Highway Networks
• Recurrent Highway Networks
• MNIST For ML Beginners
• Highway Networks with TensorFlow