Attention and Augmented Recurrent Neural Networks

[YeonwooSung]

Abstract

• Augmenting RNN with Attention is a new trend.
• A human with a piece of paper is, in some sense, much smarter than a human without.
• Since vectors are the natural language of neural networks, the memory is an array of vectors

Details

• Neural Turing Machines

  • RNN with the external memory bank
  • reading and writing: instead of predicting where to read/write (discrete), model always read/writes on all area but simply learn the weight

• Attentional Interfaces

  • Basic attention

• Adaptive Computation Time

  • a way for RNNs to do different amounts of computation each step

• Neural Programmers

  • learns to create programs in order to solve a task

Personal Thoughts

• attention is the key to next-generation neural network

Link : https://distill.pub/2016/augmented-rnns/
Authors: Olah et al. 2016