Image_Captioning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a caption generator\n",
    "This notebook implements the Show and Tell caption generation model described in our corresponding article. The key portions of this notebook are loading the data with `get_data`, processing the text data with `preProBuildWordVocab`, building the `Caption_Generator` in `train` and tracking our progress.\n",
    "\n",
    "*Note:* create a directory to save your tensorflow models and assign this directory path to the `model_path` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import cv2\n",
    "import skimage\n",
    "import pickle as pkl\n",
    "\n",
    "import tensorflow.python.platform\n",
    "from keras.preprocessing import sequence\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading Data\n",
    "As mentioned in the README, in order to run this notebook, you will need VGG-16 image embeddings for the Flickr-30K dataset. These image embeddings are available from our [Google Drive](https://drive.google.com/file/d/0B5o40yxdA9PqTnJuWGVkcFlqcG8/view?usp=sharing).\n",
    "\n",
    "Additionally, you will need the corresponding captions for these images (`results_20130124.token`), which can also be downloaded from our [Google Drive](https://drive.google.com/file/d/0B2vTU3h54lTydXFjSVM5T2t4WmM/view?usp=sharing).\n",
    "\n",
    "Place all of these downloads in the `./data/` folder.\n",
    "\n",
    "The feature embeddings will be in `./data/feats.npy` and the embeddings' corresponding captions will be saved to `./data/results_20130124.token` ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "os.getcwd()\n",
    "path = \"/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master/data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.chdir = path\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_path = '/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master'\n",
    "model_path_transfer = '/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master/models/tf_final'\n",
    "feature_path = '/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master/data/feats.npy'\n",
    "annotation_path = '/home/niraj/Documents/artificial_intelligence/projects/oreilly image captioning/oreilly-captions-master/data/results_20130124.token'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data\n",
    "Parse the image embedding features from the Flickr30k dataset `./data/feats.npy`, and load the caption data via `pandas` from `./data/results_20130124.token`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_data(annotation_path, feature_path):\n",
    "     annotations = pd.read_table(annotation_path, sep='\\t', header=None, names=['image', 'caption'])\n",
    "     return np.load(feature_path,'r'), annotations['caption'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feats, captions = get_data(annotation_path, feature_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(158915, 4096)\n",
      "(158915,)\n"
     ]
    }
   ],
   "source": [
    "print(feats.shape)\n",
    "print(captions.shape)\n",
    "#print(captions_images.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.          0.          0.         ...,  0.          0.21412706\n",
      "  0.51223457]\n"
     ]
    }
   ],
   "source": [
    "print(feats[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two young guys with shaggy hair look at their hands while hanging out in the yard .\n"
     ]
    }
   ],
   "source": [
    "print(captions[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two young guys with shaggy hair look at their hands while hanging out in the yard .\n"
     ]
    }
   ],
   "source": [
    "print(captions[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def preProBuildWordVocab(sentence_iterator, word_count_threshold=30): # function from Andre Karpathy's NeuralTalk\n",
    "    print('preprocessing %d word vocab' % (word_count_threshold, ))\n",
    "    word_counts = {}\n",
    "    nsents = 0\n",
    "    for sent in sentence_iterator:\n",
    "      nsents += 1\n",
    "      for w in sent.lower().split(' '):\n",
    "        word_counts[w] = word_counts.get(w, 0) + 1\n",
    "    vocab = [w for w in word_counts if word_counts[w] >= word_count_threshold]\n",
    "    print('preprocessed words %d -> %d' % (len(word_counts), len(vocab)))\n",
    "\n",
    "# Giving every word a unique number and reversing the dictionary so every number will be key and word will be value    \n",
    "    ixtoword = {}\n",
    "    ixtoword[0] = '.'  \n",
    "    wordtoix = {}\n",
    "    wordtoix['#START#'] = 0 \n",
    "    ix = 1\n",
    "    for w in vocab:\n",
    "      wordtoix[w] = ix\n",
    "      ixtoword[ix] = w\n",
    "      ix += 1\n",
    "\n",
    "    word_counts['.'] = nsents\n",
    "    bias_init_vector = np.array([1.0*word_counts[ixtoword[i]] for i in ixtoword])\n",
    "    bias_init_vector /= np.sum(bias_init_vector) \n",
    "    bias_init_vector = np.log(bias_init_vector)\n",
    "    bias_init_vector -= np.max(bias_init_vector) \n",
    "    return wordtoix, ixtoword, bias_init_vector.astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Caption_Generator():\n",
    "    def __init__(self, dim_in, dim_embed, dim_hidden, batch_size, n_lstm_steps, n_words, init_b):\n",
    "\n",
    "        self.dim_in = dim_in\n",
    "        self.dim_embed = dim_embed\n",
    "        self.dim_hidden = dim_hidden\n",
    "        self.batch_size = batch_size\n",
    "        self.n_lstm_steps = n_lstm_steps\n",
    "        self.n_words = n_words\n",
    "        \n",
    "        # declare the variables to be used for our word embeddings\n",
    "        with tf.device(\"/cpu:0\"):\n",
    "            self.word_embedding = tf.get_variable(\"word_embedding\",initializer= tf.random_uniform([self.n_words, self.dim_embed], -0.1, 0.1))\n",
    "\n",
    "            self.embedding_bias = tf.get_variable(\"embedding_bias\",initializer= tf.zeros([dim_embed]))\n",
    "        \n",
    "        # declare the LSTM itself\n",
    "            self.lstm = tf.contrib.rnn.BasicLSTMCell(dim_hidden)\n",
    "        \n",
    "        # declare the variables to be used to embed the image feature embedding to the word embedding space\n",
    "            self.img_embedding = tf.get_variable(\"img_embedding\",initializer= tf.random_uniform([dim_in, dim_hidden], -0.1, 0.1))\n",
    "            self.img_embedding_bias = tf.get_variable(\"img_embedding_bias\",initializer= tf.zeros([dim_hidden]))\n",
    "\n",
    "        # declare the variables to go from an LSTM output to a word encoding output\n",
    "            self.word_encoding = tf.get_variable(\"word_encoding\",initializer= tf.random_uniform([dim_hidden, n_words], -0.1, 0.1))\n",
    "        # initialize this bias variable from the preProBuildWordVocab output\n",
    "            self.word_encoding_bias = tf.get_variable(\"word_encoding_bias\",initializer= init_b)\n",
    "\n",
    "    def build_model(self):\n",
    "        # declaring the placeholders for our extracted image feature vectors, our caption, and our mask\n",
    "        # (describes how long our caption is with an array of 0/1 values of length `maxlen`  \n",
    "        img = tf.placeholder(tf.float32, [self.batch_size, self.dim_in])\n",
    "        caption_placeholder = tf.placeholder(tf.int32, [self.batch_size, self.n_lstm_steps])\n",
    "        mask = tf.placeholder(tf.float32, [self.batch_size, self.n_lstm_steps])\n",
    "        \n",
    "        # getting an initial LSTM embedding from our image_imbedding\n",
    "        image_embedding = tf.matmul(img, self.img_embedding) + self.img_embedding_bias\n",
    "        \n",
    "        # setting initial state of our LSTM\n",
    "        state = self.lstm.zero_state(self.batch_size, dtype=tf.float32)\n",
    "\n",
    "        total_loss = 0.0\n",
    "        with tf.variable_scope(\"RNN\"):\n",
    "            for i in range(self.n_lstm_steps): \n",
    "                if i > 0:\n",
    "                   #if this isn’t the first iteration of our LSTM we need to get the word_embedding corresponding\n",
    "                   # to the (i-1)th word in our caption \n",
    "                    with tf.device(\"/cpu:0\"):\n",
    "                        current_embedding = tf.nn.embedding_lookup(self.word_embedding, caption_placeholder[:,i-1]) + self.embedding_bias\n",
    "                else:\n",
    "                     #if this is the first iteration of our LSTM we utilize the embedded image as our input \n",
    "                    current_embedding = image_embedding\n",
    "                if i > 0: \n",
    "                    # allows us to reuse the LSTM tensor variable on each iteration\n",
    "                    tf.get_variable_scope().reuse_variables()\n",
    "\n",
    "                    out, state = self.lstm(current_embedding, state)\n",
    "                        #out, state = self.tf.nn.dynamic_rnn(current_embedding, state)\n",
    "\n",
    "                \n",
    "                if i > 0:\n",
    "                    #get the one-hot representation of the next word in our caption \n",
    "                    labels = tf.expand_dims(caption_placeholder[:, i], 1)\n",
    "                    ix_range=tf.range(0, self.batch_size, 1)\n",
    "                    ixs = tf.expand_dims(ix_range, 1)\n",
    "                    concat = tf.concat([ixs, labels],1)\n",
    "                    onehot = tf.sparse_to_dense(\n",
    "                    concat, tf.stack([self.batch_size, self.n_words]), 1.0, 0.0)\n",
    "\n",
    "\n",
    "                    #perform a softmax classification to generate the next word in the caption\n",
    "                    logit = tf.matmul(out, self.word_encoding) + self.word_encoding_bias\n",
    "                    xentropy = tf.nn.softmax_cross_entropy_with_logits(logits=logit, labels=onehot)\n",
    "                    xentropy = xentropy * mask[:,i]\n",
    "\n",
    "                    loss = tf.reduce_sum(xentropy)\n",
    "                    total_loss += loss\n",
    "\n",
    "            total_loss = total_loss / tf.reduce_sum(mask[:,1:])\n",
    "            return total_loss, img,  caption_placeholder, mask\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Parameters ###\n",
    "dim_embed = 256\n",
    "dim_hidden = 256\n",
    "dim_in = 4096\n",
    "batch_size = 128\n",
    "momentum = 0.9\n",
    "n_epochs = 150\n",
    "\n",
    "def train(learning_rate=0.001, continue_training=False, transfer=True):\n",
    "    \n",
    "    tf.reset_default_graph()\n",
    "\n",
    "    feats, captions = get_data(annotation_path, feature_path)\n",
    "    wordtoix, ixtoword, init_b = preProBuildWordVocab(captions)\n",
    "\n",
    "    np.save('data/ixtoword', ixtoword)\n",
    "\n",
    "    index = (np.arange(len(feats)).astype(int))\n",
    "    np.random.shuffle(index)\n",
    "\n",
    "\n",
    "    sess = tf.InteractiveSession()\n",
    "    n_words = len(wordtoix)\n",
    "    maxlen = np.max( [x for x in map(lambda x: len(x.split(' ')), captions) ] )\n",
    "    caption_generator = Caption_Generator(dim_in, dim_hidden, dim_embed, batch_size, maxlen+2, n_words, init_b)\n",
    "\n",
    "    loss, image, sentence, mask = caption_generator.build_model()\n",
    "\n",
    "    saver = tf.train.Saver(max_to_keep=100)\n",
    "    global_step=tf.Variable(0,trainable=False)\n",
    "    learning_rate = tf.train.exponential_decay(learning_rate, global_step,\n",
    "                                       int(len(index)/batch_size), 0.95)\n",
    "    train_op = tf.train.AdamOptimizer(learning_rate).minimize(loss)\n",
    "    tf.global_variables_initializer().run()\n",
    "\n",
    "    if continue_training:\n",
    "        if not transfer:\n",
    "            saver.restore(sess,tf.train.latest_checkpoint(model_path))\n",
    "        else:\n",
    "            saver.restore(sess,tf.train.latest_checkpoint(model_path_transfer))\n",
    "    losses=[]\n",
    "    for epoch in range(n_epochs):\n",
    "        for start, end in zip( range(0, len(index), batch_size), range(batch_size, len(index), batch_size)):\n",
    "\n",
    "            current_feats = feats[index[start:end]]\n",
    "            current_captions = captions[index[start:end]]\n",
    "            current_caption_ind = [x for x in map(lambda cap: [wordtoix[word] for word in cap.lower().split(' ')[:-1] if word in wordtoix], current_captions)]\n",
    "\n",
    "            current_caption_matrix = sequence.pad_sequences(current_caption_ind, padding='post', maxlen=maxlen+1)\n",
    "            current_caption_matrix = np.hstack( [np.full( (len(current_caption_matrix),1), 0), current_caption_matrix] )\n",
    "\n",
    "            current_mask_matrix = np.zeros((current_caption_matrix.shape[0], current_caption_matrix.shape[1]))\n",
    "            nonzeros = np.array([x for x in map(lambda x: (x != 0).sum()+2, current_caption_matrix )])\n",
    "\n",
    "            for ind, row in enumerate(current_mask_matrix):\n",
    "                row[:nonzeros[ind]] = 1\n",
    "\n",
    "            _, loss_value = sess.run([train_op, loss], feed_dict={\n",
    "                image: current_feats.astype(np.float32),\n",
    "                sentence : current_caption_matrix.astype(np.int32),\n",
    "                mask : current_mask_matrix.astype(np.float32)\n",
    "                })\n",
    "\n",
    "            print(\"Current Cost: \", loss_value, \"\\t Epoch {}/{}\".format(epoch, n_epochs), \"\\t Iter {}/{}\".format(start,len(feats)))\n",
    "        print(\"Saving the model from epoch: \", epoch)\n",
    "        saver.save(sess, os.path.join(model_path, 'model'), global_step=epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocessing 30 word vocab\n",
      "preprocessed words 20326 -> 2942\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "Variable RNN/basic_lstm_cell/kernel does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-78-b6da0a27b701>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;31m#train(.001,False,False) #train from scratch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m     \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m.001\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m)\u001b[0m    \u001b[0;31m#continue training from pretrained weights @epoch500\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m     \u001b[0;31m#train(.001)  #train from previously saved weights\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-77-39693d0edd0a>\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(learning_rate, continue_training, transfer)\u001b[0m\n\u001b[1;32m     25\u001b[0m     \u001b[0mcaption_generator\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCaption_Generator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdim_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim_hidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim_embed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmaxlen\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minit_b\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m     \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mimage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcaption_generator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuild_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     28\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m     \u001b[0msaver\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSaver\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmax_to_keep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-76-d2e304449f23>\u001b[0m in \u001b[0;36mbuild_model\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     55\u001b[0m                     \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_variable_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreuse_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 57\u001b[0;31m                     \u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlstm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcurrent_embedding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     58\u001b[0m                         \u001b[0;31m#out, state = self.tf.nn.dynamic_rnn(current_embedding, state)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs, state, scope)\u001b[0m\n\u001b[1;32m    178\u001b[0m       with vs.variable_scope(vs.get_variable_scope(),\n\u001b[1;32m    179\u001b[0m                              custom_getter=self._rnn_get_variable):\n\u001b[0;32m--> 180\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mRNNCell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_rnn_get_variable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/layers/base.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs, *args, **kwargs)\u001b[0m\n\u001b[1;32m    448\u001b[0m         \u001b[0;31m# Check input assumptions set after layer building, e.g. input shape.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    449\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_assert_input_compatibility\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 450\u001b[0;31m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    451\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    452\u001b[0m         \u001b[0;31m# Apply activity regularization.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.pyc\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, inputs, state)\u001b[0m\n\u001b[1;32m    399\u001b[0m       \u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mh\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marray_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_or_size_splits\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    400\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 401\u001b[0;31m     \u001b[0mconcat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_linear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mh\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_units\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    402\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    403\u001b[0m     \u001b[0;31m# i = input_gate, j = new_input, f = forget_gate, o = output_gate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.pyc\u001b[0m in \u001b[0;36m_linear\u001b[0;34m(args, output_size, bias, bias_initializer, kernel_initializer)\u001b[0m\n\u001b[1;32m   1037\u001b[0m         \u001b[0m_WEIGHTS_VARIABLE_NAME\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtotal_arg_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_size\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m         \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m         initializer=kernel_initializer)\n\u001b[0m\u001b[1;32m   1040\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m       \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmath_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweights\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.pyc\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter)\u001b[0m\n\u001b[1;32m   1063\u001b[0m       \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1064\u001b[0m       \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1065\u001b[0;31m       use_resource=use_resource, custom_getter=custom_getter)\n\u001b[0m\u001b[1;32m   1066\u001b[0m get_variable_or_local_docstring = (\n\u001b[1;32m   1067\u001b[0m     \"\"\"%s\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.pyc\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(self, var_store, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter)\u001b[0m\n\u001b[1;32m    960\u001b[0m           \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    961\u001b[0m           \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 962\u001b[0;31m           use_resource=use_resource, custom_getter=custom_getter)\n\u001b[0m\u001b[1;32m    963\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m   def _get_partitioned_variable(self,\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.pyc\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter)\u001b[0m\n\u001b[1;32m    358\u001b[0m           \u001b[0mreuse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreuse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrainable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m           \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 360\u001b[0;31m           validate_shape=validate_shape, use_resource=use_resource)\n\u001b[0m\u001b[1;32m    361\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    362\u001b[0m       return _true_getter(\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.pyc\u001b[0m in \u001b[0;36m_rnn_get_variable\u001b[0;34m(self, getter, *args, **kwargs)\u001b[0m\n\u001b[1;32m    181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_rnn_get_variable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 183\u001b[0;31m     \u001b[0mvariable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    184\u001b[0m     trainable = (variable in tf_variables.trainable_variables() or\n\u001b[1;32m    185\u001b[0m                  (isinstance(variable, tf_variables.PartitionedVariable) and\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.pyc\u001b[0m in \u001b[0;36m_true_getter\u001b[0;34m(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource)\u001b[0m\n\u001b[1;32m    350\u001b[0m           \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrainable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    351\u001b[0m           \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 352\u001b[0;31m           use_resource=use_resource)\n\u001b[0m\u001b[1;32m    353\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    354\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mcustom_getter\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/niraj/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.pyc\u001b[0m in \u001b[0;36m_get_single_variable\u001b[0;34m(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape, use_resource)\u001b[0m\n\u001b[1;32m    680\u001b[0m       raise ValueError(\"Variable %s does not exist, or was not created with \"\n\u001b[1;32m    681\u001b[0m                        \u001b[0;34m\"tf.get_variable(). Did you mean to set reuse=None in \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 682\u001b[0;31m                        \"VarScope?\" % name)\n\u001b[0m\u001b[1;32m    683\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_fully_defined\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0minitializing_from_value\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    684\u001b[0m       raise ValueError(\"Shape of a new variable (%s) must be fully defined, \"\n",
      "\u001b[0;31mValueError\u001b[0m: Variable RNN/basic_lstm_cell/kernel does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    #train(.001,False,False) #train from scratch\n",
    "    train(.001,True,True)    #continue training from pretrained weights @epoch500\n",
    "    #train(.001)  #train from previously saved weights \n",
    "except KeyboardInterrupt:\n",
    "    print('Exiting Training')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}