summarize_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarizing Text with Amazon Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this project is to build a model that can create relevant summaries for reviews written about fine foods sold on Amazon. This dataset contains above 500,000 reviews, and is hosted on [Kaggle](https://www.kaggle.com/snap/amazon-fine-food-reviews).\n",
    "\n",
    "To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using bahdanau attention on the target data.\n",
    "\n",
    "The sections of this project are:\n",
    "- [1.Inspecting the Data](#1.-Insepcting-the-Data)\n",
    "- [2.Preparing the Data](#2.-Preparing-the-Data)\n",
    "- [3.Building the Model](#3.-Building-the-Model)\n",
    "- [4.Training the Model](#4.-Training-the-Model)\n",
    "- [5.Making Our Own Summaries](#5.-Making-Our-Own-Summaries)\n",
    "\n",
    "## Download data\n",
    "Amazon Reviews Data: [Reviews.csv](https://www.kaggle.com/snap/amazon-fine-food-reviews/downloads/Reviews.csv)\n",
    "\n",
    "word embeddings [numberbatch-en-17.06.txt.gz](https://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-en-17.06.txt.gz)\n",
    "after download, extract to **./model/numberbatch-en-17.06.txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.3.0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import time\n",
    "from tensorflow.python.layers.core import Dense\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors\n",
    "from tensorflow.python.ops import array_ops\n",
    "from tensorflow.python.ops import tensor_array_ops\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "def __pickleStuff(filename, stuff):\n",
    "    save_stuff = open(filename, \"wb\")\n",
    "    pickle.dump(stuff, save_stuff)\n",
    "    save_stuff.close()\n",
    "def __loadStuff(filename):\n",
    "    saved_stuff = open(filename,\"rb\")\n",
    "    stuff = pickle.load(saved_stuff)\n",
    "    saved_stuff.close()\n",
    "    return stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load those prepared data and skip to section \"[3. Building the Model](#3.-Building-the-Model)\"\n",
    "Once we have run through the \"[2.Preparing the Data](#2.-Preparing-the-Data)\" section, we should have those data, uncomment and run those lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clean_summaries = __loadStuff(\"./data/clean_summaries.p\")\n",
    "clean_texts = __loadStuff(\"./data/clean_texts.p\")\n",
    "\n",
    "sorted_summaries = __loadStuff(\"./data/sorted_summaries.p\")\n",
    "sorted_texts = __loadStuff(\"./data/sorted_texts.p\")\n",
    "word_embedding_matrix = __loadStuff(\"./data/word_embedding_matrix.p\")\n",
    "\n",
    "vocab_to_int = __loadStuff(\"./data/vocab_to_int.p\")\n",
    "int_to_vocab = __loadStuff(\"./data/int_to_vocab.p\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Insepcting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reviews = pd.read_csv(\"Reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568454, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B001E4KFG0</td>\n",
       "      <td>A3SGXH7AUHU8GW</td>\n",
       "      <td>delmartian</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1303862400</td>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B00813GRG4</td>\n",
       "      <td>A1D87F6ZCVE5NK</td>\n",
       "      <td>dll pa</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1346976000</td>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>B000LQOCH0</td>\n",
       "      <td>ABXLMWJIXXAIN</td>\n",
       "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1219017600</td>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>B000UA0QIQ</td>\n",
       "      <td>A395BORC6FGVXV</td>\n",
       "      <td>Karl</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1307923200</td>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>B006K2ZZ7K</td>\n",
       "      <td>A1UQRSCLF8GW1T</td>\n",
       "      <td>Michael D. Bigham \"M. Wassir\"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1350777600</td>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   ProductId          UserId                      ProfileName  \\\n",
       "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
       "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
       "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
       "3   4  B000UA0QIQ  A395BORC6FGVXV                             Karl   \n",
       "4   5  B006K2ZZ7K  A1UQRSCLF8GW1T    Michael D. Bigham \"M. Wassir\"   \n",
       "\n",
       "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
       "0                     1                       1      5  1303862400   \n",
       "1                     0                       0      1  1346976000   \n",
       "2                     1                       1      4  1219017600   \n",
       "3                     3                       3      2  1307923200   \n",
       "4                     0                       0      5  1350777600   \n",
       "\n",
       "                 Summary                                               Text  \n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...  \n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...  \n",
       "4            Great taffy  Great taffy at a great price.  There was a wid...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                         0\n",
       "ProductId                  0\n",
       "UserId                     0\n",
       "ProfileName               16\n",
       "HelpfulnessNumerator       0\n",
       "HelpfulnessDenominator     0\n",
       "Score                      0\n",
       "Time                       0\n",
       "Summary                   26\n",
       "Text                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for any nulls values\n",
    "reviews.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Remove null values and unneeded features\n",
    "reviews = reviews.dropna()\n",
    "reviews = reviews.drop(['Id','ProductId','UserId','ProfileName','HelpfulnessNumerator','HelpfulnessDenominator',\n",
    "                        'Score','Time'], 1)\n",
    "reviews = reviews.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568412, 2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Summary                                               Text\n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...\n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...\n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...\n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...\n",
       "4            Great taffy  Great taffy at a great price.  There was a wid..."
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review # 1\n",
      "Good Quality Dog Food\n",
      "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.\n",
      "\n",
      "Review # 2\n",
      "Not as Advertised\n",
      "Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as \"Jumbo\".\n",
      "\n",
      "Review # 3\n",
      "\"Delight\" says it all\n",
      "This is a confection that has been around a few centuries.  It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar.  And it is a tiny mouthful of heaven.  Not too chewy, and very flavorful.  I highly recommend this yummy treat.  If you are familiar with the story of C.S. Lewis' \"The Lion, The Witch, and The Wardrobe\" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.\n",
      "\n",
      "Review # 4\n",
      "Cough Medicine\n",
      "If you are looking for the secret ingredient in Robitussin I believe I have found it.  I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda.  The flavor is very medicinal.\n",
      "\n",
      "Review # 5\n",
      "Great taffy\n",
      "Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspecting some of the reviews\n",
    "for i in range(5):\n",
    "    print(\"Review #\",i+1)\n",
    "    print(reviews.Summary[i])\n",
    "    print(reviews.Text[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Preparing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A list of contractions from http://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python\n",
    "contractions = { \n",
    "\"ain't\": \"am not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he will\",\n",
    "\"he's\": \"he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how is\",\n",
    "\"i'd\": \"i would\",\n",
    "\"i'll\": \"i will\",\n",
    "\"i'm\": \"i am\",\n",
    "\"i've\": \"i have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it would\",\n",
    "\"it'll\": \"it will\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"needn't\": \"need not\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'll\": \"she will\",\n",
    "\"she's\": \"she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"that'd\": \"that would\",\n",
    "\"that's\": \"that is\",\n",
    "\"there'd\": \"there had\",\n",
    "\"there's\": \"there is\",\n",
    "\"they'd\": \"they would\",\n",
    "\"they'll\": \"they will\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we would\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what will\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where is\",\n",
    "\"who'll\": \"who will\",\n",
    "\"who's\": \"who is\",\n",
    "\"won't\": \"will not\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"you'd\": \"you would\",\n",
    "\"you'll\": \"you will\",\n",
    "\"you're\": \"you are\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_text(text, remove_stopwords = True):\n",
    "    '''Remove unwanted characters, stopwords, and format the text to create fewer nulls word embeddings'''\n",
    "    \n",
    "    # Convert words to lower case\n",
    "    text = text.lower()\n",
    "    \n",
    "    # Replace contractions with their longer forms \n",
    "    if True:\n",
    "        # We are not using \"text.split()\" here\n",
    "        #since it is not fool proof, e.g. words followed by punctuations \"Are you kidding?I think you aren't.\"\n",
    "        text = re.findall(r\"[\\w']+\", text)\n",
    "        new_text = []\n",
    "        for word in text:\n",
    "            if word in contractions:\n",
    "                new_text.append(contractions[word])\n",
    "            else:\n",
    "                new_text.append(word)\n",
    "        text = \" \".join(new_text)\n",
    "    \n",
    "    # Format words and remove unwanted characters\n",
    "    text = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', text, flags=re.MULTILINE)# remove links\n",
    "    text = re.sub(r'\\<a href', ' ', text)# remove html link tag\n",
    "    text = re.sub(r'&amp;', '', text) \n",
    "    text = re.sub(r'[_\"\\-;%()|+&=*%.,!?:#$@\\[\\]/]', ' ', text)\n",
    "    text = re.sub(r'<br />', ' ', text)\n",
    "    text = re.sub(r'\\'', ' ', text)\n",
    "    \n",
    "    # Optionally, remove stop words\n",
    "    if remove_stopwords:\n",
    "        text = text.split()\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        text = [w for w in text if not w in stops]\n",
    "        text = \" \".join(text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'great movie believe may'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_text(\"That's a great movie,Can you believe it?I've.But you may not.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the summaries and texts\n",
    "We will remove the stopwords from the texts because they do not provide much use for training our model. However, we will keep them for our summaries so that they sound more like natural phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries are complete.\n",
      "Texts are complete.\n"
     ]
    }
   ],
   "source": [
    "clean_summaries = []\n",
    "for summary in reviews.Summary:\n",
    "    clean_summaries.append(clean_text(summary, remove_stopwords=False))\n",
    "print(\"Summaries are complete.\")\n",
    "\n",
    "clean_texts = []\n",
    "for text in reviews.Text:\n",
    "    clean_texts.append(clean_text(text))\n",
    "print(\"Texts are complete.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean Review # 1\n",
      "good quality dog food\n",
      "bought several vitality canned dog food products found good quality product looks like stew processed meat smells better labrador finicky appreciates product better\n",
      "\n",
      "Clean Review # 2\n",
      "not as advertised\n",
      "product arrived labeled jumbo salted peanuts peanuts actually small sized unsalted sure error vendor intended represent product jumbo\n",
      "\n",
      "Clean Review # 3\n",
      "delight says it all\n",
      "confection around centuries light pillowy citrus gelatin nuts case filberts cut tiny squares liberally coated powdered sugar tiny mouthful heaven chewy flavorful highly recommend yummy treat familiar story c lewis lion witch wardrobe treat seduces edmund selling brother sisters witch\n",
      "\n",
      "Clean Review # 4\n",
      "cough medicine\n",
      "looking secret ingredient robitussin believe found got addition root beer extract ordered good made cherry soda flavor medicinal\n",
      "\n",
      "Clean Review # 5\n",
      "great taffy\n",
      "great taffy great price wide assortment yummy taffy delivery quick taffy lover deal\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the cleaned summaries and texts to ensure they have been cleaned well\n",
    "for i in range(5):\n",
    "    print(\"Clean Review #\",i+1)\n",
    "    print(clean_summaries[i])\n",
    "    print(clean_texts[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count the number of occurrences of each word in a set of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def count_words(count_dict, text):\n",
    "    for sentence in text:\n",
    "        for word in sentence.split():\n",
    "            if word not in count_dict:\n",
    "                count_dict[word] = 1\n",
    "            else:\n",
    "                count_dict[word] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Give the function a try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 2, 'dog': 2, 'great': 4, 'have': 1, 'is': 1, 'that': 1, 'you': 1}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mydict = {}\n",
    "count_words(mydict, [\"that is a great great great dog\",\"you have a great dog\"])\n",
    "mydict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Vocabulary: 125880\n"
     ]
    }
   ],
   "source": [
    "word_counts = {}\n",
    "count_words(word_counts, clean_summaries)\n",
    "count_words(word_counts, clean_texts)\n",
    "print(\"Size of Vocabulary:\", len(word_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how may \"hero\" occurs in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "114"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts[\"hero\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Load Conceptnet Numberbatch's (CN) embeddings, similar to GloVe, but probably better \n",
    " (https://github.com/commonsense/conceptnet-numberbatch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word embeddings: 417195\n"
     ]
    }
   ],
   "source": [
    "\n",
    "embeddings_index = {}\n",
    "with open('./model/numberbatch-en-17.06.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        values = line.split(' ')\n",
    "        word = values[0]\n",
    "        embedding = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = embedding\n",
    "\n",
    "print('Word embeddings:', len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a look at the CN embedding dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings_index[\"hero\"].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the number of words that are missing from CN, and are used more than our threshold.\n",
    "\n",
    "I use a **threshold** of 20, so that words not in CN can be added to our **word_embedding_matrix**, but they need to be common enough in the reviews so that the model can understand their meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words missing from CN: 2608\n",
      "Percent of words that are missing from vocabulary: 2.07%\n"
     ]
    }
   ],
   "source": [
    "missing_words = 0\n",
    "threshold = 20\n",
    "\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold:\n",
    "        if word not in embeddings_index:\n",
    "            missing_words += 1\n",
    "            \n",
    "missing_ratio = round(missing_words/len(word_counts),4)*100\n",
    "            \n",
    "print(\"Number of words missing from CN:\", missing_words)\n",
    "print(\"Percent of words that are missing from vocabulary: {}%\".format(missing_ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are those missing words in the CN\n",
    "Looks mostly products' brand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('wafu', 29),\n",
       " ('wasaibi', 24),\n",
       " ('sauage', 23),\n",
       " ('diabetisweet', 27),\n",
       " ('aerogrow', 99),\n",
       " ('lowfat', 298),\n",
       " ('deliverd', 21),\n",
       " ('bullysticks', 21),\n",
       " ('keurigs', 72),\n",
       " ('pepitas', 42),\n",
       " ('wellpet', 27),\n",
       " ('undertaste', 24),\n",
       " ('50g', 44),\n",
       " ('ammount', 45),\n",
       " ('400', 461),\n",
       " ('toniq', 21),\n",
       " ('gummis', 161),\n",
       " ('teasan', 81),\n",
       " ('27th', 28),\n",
       " ('iherb', 66),\n",
       " ('fage', 34),\n",
       " ('droste', 70),\n",
       " ('wholefoods', 145),\n",
       " ('marzanos', 29),\n",
       " ('discusting', 28),\n",
       " ('foojoy', 41),\n",
       " ('91', 75),\n",
       " ('indomie', 36),\n",
       " ('5hour', 64),\n",
       " ('ec155', 34)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_words = []\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold and word not in embeddings_index:\n",
    "        missing_words.append((word,count))\n",
    "missing_words[:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Words to indexes, indexes to words dicts\n",
    "Limit the vocab that we will use to words that appear ≥ threshold or are in CN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of unique words: 125880\n",
      "Number of words we will use: 59072\n",
      "Percent of words we will use: 46.93%\n"
     ]
    }
   ],
   "source": [
    "#dictionary to convert words to integers\n",
    "vocab_to_int = {} \n",
    "# Index words from 0\n",
    "value = 0\n",
    "for word, count in word_counts.items():\n",
    "    if count >= threshold or word in embeddings_index:\n",
    "        vocab_to_int[word] = value\n",
    "        value += 1\n",
    "\n",
    "# Special tokens that will be added to our vocab\n",
    "codes = [\"<UNK>\",\"<PAD>\",\"<EOS>\",\"<GO>\"]   \n",
    "\n",
    "# Add codes to vocab\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = len(vocab_to_int)\n",
    "\n",
    "# Dictionary to convert integers to words\n",
    "int_to_vocab = {}\n",
    "for word, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = word\n",
    "\n",
    "usage_ratio = round(len(vocab_to_int) / len(word_counts),4)*100\n",
    "\n",
    "print(\"Total number of unique words:\", len(word_counts))\n",
    "print(\"Number of words we will use:\", len(vocab_to_int))\n",
    "print(\"Percent of words we will use: {}%\".format(usage_ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create word embedding matrix\n",
    "It has shape (nb_words, embedding_dim) i.e. (59072, 300) in this case. 1st dim is word index, 2nd dim is from CN or random generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "59072\n"
     ]
    }
   ],
   "source": [
    "# Need to use 300 for embedding dimensions to match CN's vectors.\n",
    "embedding_dim = 300\n",
    "nb_words = len(vocab_to_int)\n",
    "\n",
    "# Create matrix with default values of zero\n",
    "word_embedding_matrix = np.zeros((nb_words, embedding_dim), dtype=np.float32)\n",
    "for word, i in vocab_to_int.items():\n",
    "    if word in embeddings_index:\n",
    "        word_embedding_matrix[i] = embeddings_index[word]\n",
    "    else:\n",
    "        # If word not in CN, create a random embedding for it\n",
    "        new_embedding = np.array(np.random.uniform(-1.0, 1.0, embedding_dim))\n",
    "        embeddings_index[word] = new_embedding\n",
    "        word_embedding_matrix[i] = new_embedding\n",
    "\n",
    "# Check if value matches len(vocab_to_int)\n",
    "print(len(word_embedding_matrix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to convert sentences to sequence of words indexes\n",
    "It also use `<UNK>` index to replace unknown words, append `<EOS>` (End of Sentence) to the sequences if eos is set True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_to_ints(text, word_count, unk_count, eos=False):\n",
    "    '''Convert words in text to an integer.\n",
    "       If word is not in vocab_to_int, use UNK's integer.\n",
    "       Total the number of words and UNKs.\n",
    "       Add EOS token to the end of texts'''\n",
    "    ints = []\n",
    "    for sentence in text:\n",
    "        sentence_ints = []\n",
    "        for word in sentence.split():\n",
    "            word_count += 1\n",
    "            if word in vocab_to_int:\n",
    "                sentence_ints.append(vocab_to_int[word])\n",
    "            else:\n",
    "                sentence_ints.append(vocab_to_int[\"<UNK>\"])\n",
    "                unk_count += 1\n",
    "        if eos:\n",
    "            sentence_ints.append(vocab_to_int[\"<EOS>\"])\n",
    "        ints.append(sentence_ints)\n",
    "    return ints, word_count, unk_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply convert_to_ints to clean_summaries and clean_texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of words in headlines: 26232576\n",
      "Total number of UNKs in headlines: 163594\n",
      "Percent of words that are UNK: 0.62%\n"
     ]
    }
   ],
   "source": [
    "\n",
    "word_count = 0\n",
    "unk_count = 0\n",
    "\n",
    "int_summaries, word_count, unk_count = convert_to_ints(clean_summaries, word_count, unk_count)\n",
    "int_texts, word_count, unk_count = convert_to_ints(clean_texts, word_count, unk_count, eos=True)\n",
    "\n",
    "unk_percent = round(unk_count/word_count,4)*100\n",
    "\n",
    "print(\"Total number of words in headlines:\", word_count)\n",
    "print(\"Total number of UNKs in headlines:\", unk_count)\n",
    "print(\"Percent of words that are UNK: {}%\".format(unk_percent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a look at what the sequence looks like\n",
    "Each number here represents a word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[32681, 40810, 26787, 54872],\n",
       " [2229, 54986, 47923],\n",
       " [23867, 38191, 14436, 39262]]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_summaries[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to get the length of each sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_lengths(text):\n",
    "    '''Create a data frame of the sentence lengths from a text'''\n",
    "    lengths = []\n",
    "    for sentence in text:\n",
    "        lengths.append(len(sentence))\n",
    "    return pd.DataFrame(lengths, columns=['counts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   counts\n",
       "0       4\n",
       "1       3\n",
       "2       4"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_lengths(int_summaries[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get statistic summary of the length of summaries and texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean        4.181208\n",
      "std         2.657212\n",
      "min         0.000000\n",
      "25%         2.000000\n",
      "50%         4.000000\n",
      "75%         5.000000\n",
      "max        48.000000\n",
      "\n",
      "Texts:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean       42.969429\n",
      "std        44.166421\n",
      "min         2.000000\n",
      "25%        18.000000\n",
      "50%        30.000000\n",
      "75%        51.000000\n",
      "max      2063.000000\n"
     ]
    }
   ],
   "source": [
    "lengths_summaries = create_lengths(int_summaries)\n",
    "lengths_texts = create_lengths(int_texts)\n",
    "\n",
    "print(\"Summaries:\")\n",
    "print(lengths_summaries.describe())\n",
    "print()\n",
    "print(\"Texts:\")\n",
    "print(lengths_texts.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### See what's the max squence length we can cover by percentile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84.0\n",
      "118.0\n",
      "216.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of texts\n",
    "print(np.percentile(lengths_texts.counts, 89.5))\n",
    "print(np.percentile(lengths_texts.counts, 95))\n",
    "print(np.percentile(lengths_texts.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.0\n",
      "9.0\n",
      "13.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of summaries\n",
    "print(np.percentile(lengths_summaries.counts, 90))\n",
    "print(np.percentile(lengths_summaries.counts, 95))\n",
    "print(np.percentile(lengths_summaries.counts, 99))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to counts the number of time `<UNK>` appears in a sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def unk_counter(sentence):\n",
    "    '''Counts the number of time UNK appears in a sentence.'''\n",
    "    unk_count = 0\n",
    "    for word in sentence:\n",
    "        if word == vocab_to_int[\"<UNK>\"]:\n",
    "            unk_count += 1\n",
    "    return unk_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filter** for length limit and number of `<UNK>`s\n",
    "\n",
    "**Sort** the summaries and texts by the length of the element in **texts** from shortest to longest\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "428278\n",
      "428278\n"
     ]
    }
   ],
   "source": [
    "max_text_length = 83 # This will cover up to 89.5% lengthes\n",
    "max_summary_length = 13 # This will cover up to 99% lengthes\n",
    "min_length = 2\n",
    "unk_text_limit = 1 # text can contain up to 1 UNK word\n",
    "unk_summary_limit = 0 # Summary should not contain any UNK word\n",
    "\n",
    "def filter_condition(item):\n",
    "    int_summary = item[0]\n",
    "    int_text = item[1]\n",
    "    if(len(int_summary) >= min_length and \n",
    "       len(int_summary) <= max_summary_length and \n",
    "       len(int_text) >= min_length and \n",
    "       len(int_text) <= max_text_length and \n",
    "       unk_counter(int_summary) <= unk_summary_limit and \n",
    "       unk_counter(int_text) <= unk_text_limit):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "int_text_summaries = list(zip(int_summaries , int_texts))\n",
    "int_text_summaries_filtered = list(filter(filter_condition, int_text_summaries))\n",
    "sorted_int_text_summaries = sorted(int_text_summaries_filtered, key=lambda item: len(item[1]))\n",
    "sorted_int_text_summaries = list(zip(*sorted_int_text_summaries))\n",
    "sorted_summaries = list(sorted_int_text_summaries[0])\n",
    "sorted_texts = list(sorted_int_text_summaries[1])\n",
    "# Delete those temporary varaibles\n",
    "del int_text_summaries, sorted_int_text_summaries, int_text_summaries_filtered\n",
    "# Compare lengths to ensure they match\n",
    "print(len(sorted_summaries))\n",
    "print(len(sorted_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect the length of text in sorted_texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lengths_texts = [len(text) for text in sorted_texts]\n",
    "lengths_texts[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save data for later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "__pickleStuff(\"./data/clean_summaries.p\",clean_summaries)\n",
    "__pickleStuff(\"./data/clean_texts.p\",clean_texts)\n",
    "\n",
    "__pickleStuff(\"./data/sorted_summaries.p\",sorted_summaries)\n",
    "__pickleStuff(\"./data/sorted_texts.p\",sorted_texts)\n",
    "__pickleStuff(\"./data/word_embedding_matrix.p\",word_embedding_matrix)\n",
    "\n",
    "__pickleStuff(\"./data/vocab_to_int.p\",vocab_to_int)\n",
    "__pickleStuff(\"./data/int_to_vocab.p\",int_to_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Building the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create palceholders for inputs to the model\n",
    "\n",
    "**summary_length** and **text_length** are the sentence lengths in a batch, and **max_summary_length** is the maximum length of a summary in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    summary_length = tf.placeholder(tf.int32, (None,), name='summary_length')\n",
    "    max_summary_length = tf.reduce_max(summary_length, name='max_dec_len')\n",
    "    text_length = tf.placeholder(tf.int32, (None,), name='text_length')\n",
    "\n",
    "    return input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the last word id from each batch and concatenate the id of `<GO>` to the begining of each batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(target_data, vocab_to_int, batch_size):  \n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1]) # slice it to target_data[0:batch_size, 0: -1]\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the encoding layers\n",
    "\n",
    "bidirectional_dynamic_rnn\n",
    "use **tf.variable_scope** so that variables are reused with each layer\n",
    "\n",
    "parameters\n",
    "- **rnn_size**: The number of units in the LSTM cell\n",
    "- **sequence_length**: size [batch_size], containing the actual lengths for each of the sequences in the batch\n",
    "- **num_layers**: number of bidirectional RNN layer\n",
    "- **rnn_inputs**: number of bidirectional RNN layer\n",
    "- **keep_prob**: RNN dropout input keep probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):\n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                    cell_bw, \n",
    "                                                                    rnn_inputs,\n",
    "                                                                    sequence_length,\n",
    "                                                                    dtype=tf.float32)\n",
    "            enc_output = tf.concat(enc_output,2)\n",
    "            # original code is missing this line below, that is how we connect layers \n",
    "            # by feeding the current layer's output to next layer's input\n",
    "            rnn_inputs = enc_output\n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the training decoding layer\n",
    "parameters\n",
    "- **dec_embed_input**: output of embedding_lookup for a batch of inputs\n",
    "- **summary_length**: length of each padded summary sequences in batch, since padded, all lengths should be same number \n",
    "- **dec_cell**: the decoder RNN cells' output with attention wapper\n",
    "- **output_layer**: fully connected layer to apply to the RNN output\n",
    "- **vocab_size**: vocabulary size i.e. len(vocab_to_int)+1\n",
    "- **max_summary_length**: the maximum length of a summary in a batch\n",
    "- **batch_size**: number of input sequences in a batch\n",
    "\n",
    "Three components\n",
    "\n",
    "- **TraingHelper** reads a sequence of integers from the encoding layer.\n",
    "- **BasicDecoder** processes the sequence with the decoding cell, and an output layer, which is a fully connected layer. **initial_state** set to zero state.\n",
    "- **dynamic_decode** creates our outputs that will be used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def training_decoding_layer(dec_embed_input, summary_length, dec_cell, output_layer,\n",
    "                            vocab_size, max_summary_length,batch_size):\n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                        sequence_length=summary_length,\n",
    "                                                        time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(cell=dec_cell,\n",
    "                                                       helper=training_helper,\n",
    "                                                       initial_state=dec_cell.zero_state(dtype=tf.float32, batch_size=batch_size),\n",
    "                                                       output_layer = output_layer)\n",
    "\n",
    "    training_logits = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                           output_time_major=False,\n",
    "                                                           impute_finished=True,\n",
    "                                                           maximum_iterations=max_summary_length)\n",
    "    return training_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create infer decoding layer\n",
    "\n",
    "parameters\n",
    "- **embeddings**: the CN's word_embedding_matrix\n",
    "- **start_token**: the id of `<GO>`\n",
    "- **end_token**: the id of `<EOS>`\n",
    "- **dec_cell**: the decoder RNN cells' output with attention wapper\n",
    "- **output_layer**: fully connected layer to apply to the RNN output\n",
    "- **max_summary_length**: the maximum length of a summary in a batch\n",
    "- **batch_size**: number of input sequences in a batch\n",
    "\n",
    "**GreedyEmbeddingHelper** argument **start_tokens**: int32 vector shaped [batch_size], the start tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, output_layer,\n",
    "                             max_summary_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    \n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                end_token)\n",
    "                \n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        dec_cell.zero_state(dtype=tf.float32, batch_size=batch_size),\n",
    "                                                        output_layer)\n",
    "                \n",
    "    inference_logits = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            output_time_major=False,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return inference_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Decoding layer\n",
    "3 parts: decoding cell, attention, and getting our logits.\n",
    "#### Decoding Cell: \n",
    "Just a two layer LSTM with dropout.\n",
    "#### Attention: \n",
    "Using Bhadanau, since trains faster than Luong. \n",
    "\n",
    "**AttentionWrapper** applies the attention mechanism to our decoding cell.\n",
    "\n",
    "parameters\n",
    "- **dec_embed_input**: output of embedding_lookup for a batch of inputs\n",
    "- **embeddings**: the CN's word_embedding_matrix\n",
    "- **enc_output**: encoder layer output, containing the forward and the backward rnn output\n",
    "- **enc_state**: encoder layer state, a tuple containing the forward and the backward final states of bidirectional rnn.\n",
    "- **vocab_size**: vocabulary size i.e. len(vocab_to_int)+1\n",
    "- **text_length**: the actual lengths for each of the input text sequences in the batch\n",
    "- **summary_length**: the actual lengths for each of the input summary sequences in the batch\n",
    "- **max_summary_length**: the maximum length of a summary in a batch\n",
    "- **rnn_size**: The number of units in the LSTM cell\n",
    "- **vocab_to_int**: vocab_to_int the dictionary\n",
    "- **keep_prob**: RNN dropout input keep probability\n",
    "- **batch_size**: number of input sequences in a batch\n",
    "- **num_layers**: number of decoder RNN layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lstm_cell(lstm_size, keep_prob):\n",
    "    cell = tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "    return tf.contrib.rnn.DropoutWrapper(cell, input_keep_prob = keep_prob)\n",
    "\n",
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, text_length, summary_length,\n",
    "                   max_summary_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    dec_cell = tf.contrib.rnn.MultiRNNCell([lstm_cell(rnn_size, keep_prob) for _ in range(num_layers)])\n",
    "    output_layer = Dense(vocab_size,kernel_initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.1))\n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                     enc_output,\n",
    "                                                     text_length,\n",
    "                                                     normalize=False,\n",
    "                                                     name='BahdanauAttention')\n",
    "    dec_cell = tf.contrib.seq2seq.AttentionWrapper(dec_cell,attn_mech,rnn_size)\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_logits = training_decoding_layer(dec_embed_input,summary_length,dec_cell,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size,\n",
    "                                                  max_summary_length,\n",
    "                                                  batch_size)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_logits = inference_decoding_layer(embeddings,\n",
    "                                                    vocab_to_int['<GO>'],\n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell,\n",
    "                                                    output_layer,\n",
    "                                                    max_summary_length,\n",
    "                                                    batch_size)\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, text_length, summary_length, max_summary_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    # Use Numberbatch's embeddings and the newly created ones as our embeddings\n",
    "    embeddings = word_embedding_matrix\n",
    "    enc_embed_input = tf.nn.embedding_lookup(embeddings, input_data)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, text_length, num_layers, enc_embed_input, keep_prob)\n",
    "    dec_input = process_encoding_input(target_data, vocab_to_int, batch_size) #shape=(batch_size, senquence length) each seq start with index of<GO>\n",
    "    dec_embed_input = tf.nn.embedding_lookup(embeddings, dec_input)\n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        text_length, \n",
    "                                                        summary_length, \n",
    "                                                        max_summary_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers)\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pad sentences for batch\n",
    "Pad so the actual lengths for each of the sequences in the batch have the same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to generate batch data for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(summaries, texts, batch_size):\n",
    "    \"\"\"Batch summaries, texts, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(texts)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        summaries_batch = summaries[start_i:start_i + batch_size]\n",
    "        texts_batch = texts[start_i:start_i + batch_size]\n",
    "        pad_summaries_batch = np.array(pad_sentence_batch(summaries_batch))\n",
    "        pad_texts_batch = np.array(pad_sentence_batch(texts_batch))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_summaries_lengths = []\n",
    "        for summary in pad_summaries_batch:\n",
    "            pad_summaries_lengths.append(len(summary))\n",
    "        \n",
    "        pad_texts_lengths = []\n",
    "        for text in pad_texts_batch:\n",
    "            pad_texts_lengths.append(len(text))\n",
    "        \n",
    "        yield pad_summaries_batch, pad_texts_batch, pad_summaries_lengths, pad_texts_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Just to test \"get_batches\" function\n",
    "Here we generate a batch with size of 5\n",
    "\n",
    "Checkout those \"59069\" they are `<PAD>`s, also all sequences' lengths are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'<PAD>' has id: 59069\n",
      "pad summaries batch samples:\n",
      "\r",
      " [[ 9218 18733 13131 39434 39434  4082  2454 29838 26219 33088 26752     4]\n",
      " [ 1417 42487  4397 22892 20719 59069 59069 59069 59069 59069 59069 59069]\n",
      " [ 2229 54986 19050 54986 44366 56008  8293 46449  6045 20974  4269 41958]\n",
      " [39205 16127  2875 26752 33799 58931 58335  5156 12490 59069 59069 59069]\n",
      " [54984 47044 12490 43359 46111 59069 59069 59069 59069 59069 59069 59069]]\n"
     ]
    }
   ],
   "source": [
    "print(\"'<PAD>' has id: {}\".format(vocab_to_int['<PAD>']))\n",
    "sorted_summaries_samples = sorted_summaries[7:50]\n",
    "sorted_texts_samples = sorted_texts[7:50]\n",
    "pad_summaries_batch_samples, pad_texts_batch_samples, pad_summaries_lengths_samples, pad_texts_lengths_samples = next(get_batches(\n",
    "    sorted_summaries_samples, sorted_texts_samples, 5))\n",
    "print(\"pad summaries batch samples:\\n\\r {}\".format(pad_summaries_batch_samples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the Hyperparameters\n",
    "epochs = 100\n",
    "batch_size = 64\n",
    "rnn_size = 256\n",
    "num_layers = 2\n",
    "learning_rate = 0.005\n",
    "keep_probability = 0.95"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph is built.\n",
      "./graph\n"
     ]
    }
   ],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      text_length,\n",
    "                                                      summary_length,\n",
    "                                                      max_summary_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size)\n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits[0].rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_logits[0].sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss, the sould be all True across since each batch is padded\n",
    "    masks = tf.sequence_mask(summary_length, max_summary_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "print(\"Graph is built.\")\n",
    "graph_location = \"./graph\"\n",
    "print(graph_location)\n",
    "train_writer = tf.summary.FileWriter(graph_location)\n",
    "train_writer.add_graph(train_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Training the Model\n",
    "\n",
    "Only going to use a subset of the data to reduce the traing time for this demo.\n",
    "\n",
    "We chose not use use the start of the subset because because those are shorter sequences and we don't want to make it too easy for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shortest text length: 25\n",
      "The longest text length: 31\n"
     ]
    }
   ],
   "source": [
    "# Subset the data for training\n",
    "start = 200000\n",
    "end = start + 50000\n",
    "sorted_summaries_short = sorted_summaries[start:end]\n",
    "sorted_texts_short = sorted_texts[start:end]\n",
    "print(\"The shortest text length:\", len(sorted_texts_short[0]))\n",
    "print(\"The longest text length:\",len(sorted_texts_short[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   1/100 Batch   20/781 - Loss:  5.205, Seconds: 4.33\n",
      "Epoch   1/100 Batch   40/781 - Loss:  2.856, Seconds: 3.35\n",
      "Epoch   1/100 Batch   60/781 - Loss:  2.914, Seconds: 4.61\n",
      "Epoch   1/100 Batch   80/781 - Loss:  2.825, Seconds: 4.13\n",
      "Epoch   1/100 Batch  100/781 - Loss:  2.698, Seconds: 4.21\n",
      "Epoch   1/100 Batch  120/781 - Loss:  2.711, Seconds: 3.79\n",
      "Epoch   1/100 Batch  140/781 - Loss:  2.587, Seconds: 3.97\n",
      "Epoch   1/100 Batch  160/781 - Loss:  2.844, Seconds: 3.27\n",
      "Epoch   1/100 Batch  180/781 - Loss:  2.685, Seconds: 3.65\n",
      "Epoch   1/100 Batch  200/781 - Loss:  2.676, Seconds: 4.35\n",
      "Epoch   1/100 Batch  220/781 - Loss:  2.608, Seconds: 4.05\n",
      "Epoch   1/100 Batch  240/781 - Loss:  2.471, Seconds: 4.09\n",
      "Average loss for this update: 2.896\n",
      "New Record!\n",
      "Epoch   1/100 Batch  260/781 - Loss:  2.540, Seconds: 4.59\n",
      "Epoch   1/100 Batch  280/781 - Loss:  2.611, Seconds: 4.21\n",
      "Epoch   1/100 Batch  300/781 - Loss:  2.674, Seconds: 4.13\n",
      "Epoch   1/100 Batch  320/781 - Loss:  2.685, Seconds: 4.13\n",
      "Epoch   1/100 Batch  340/781 - Loss:  2.483, Seconds: 4.67\n",
      "Epoch   1/100 Batch  360/781 - Loss:  2.591, Seconds: 4.37\n",
      "Epoch   1/100 Batch  380/781 - Loss:  2.430, Seconds: 4.67\n",
      "Epoch   1/100 Batch  400/781 - Loss:  2.550, Seconds: 4.61\n",
      "Epoch   1/100 Batch  420/781 - Loss:  2.512, Seconds: 3.83\n",
      "Epoch   1/100 Batch  440/781 - Loss:  2.619, Seconds: 4.63\n",
      "Epoch   1/100 Batch  460/781 - Loss:  2.652, Seconds: 4.51\n",
      "Epoch   1/100 Batch  480/781 - Loss:  2.440, Seconds: 3.95\n",
      "Epoch   1/100 Batch  500/781 - Loss:  2.464, Seconds: 4.23\n",
      "Average loss for this update: 2.55\n",
      "New Record!\n",
      "Epoch   1/100 Batch  520/781 - Loss:  2.462, Seconds: 3.95\n",
      "Epoch   1/100 Batch  540/781 - Loss:  2.477, Seconds: 4.53\n",
      "Epoch   1/100 Batch  560/781 - Loss:  2.419, Seconds: 4.05\n",
      "Epoch   1/100 Batch  580/781 - Loss:  2.470, Seconds: 4.31\n",
      "Epoch   1/100 Batch  600/781 - Loss:  2.654, Seconds: 4.25\n",
      "Epoch   1/100 Batch  620/781 - Loss:  2.529, Seconds: 4.21\n",
      "Epoch   1/100 Batch  640/781 - Loss:  2.432, Seconds: 4.29\n",
      "Epoch   1/100 Batch  660/781 - Loss:  2.368, Seconds: 4.69\n",
      "Epoch   1/100 Batch  680/781 - Loss:  2.284, Seconds: 4.27\n",
      "Epoch   1/100 Batch  700/781 - Loss:  2.440, Seconds: 4.29\n",
      "Epoch   1/100 Batch  720/781 - Loss:  2.582, Seconds: 4.53\n",
      "Epoch   1/100 Batch  740/781 - Loss:  2.421, Seconds: 4.62\n",
      "Epoch   1/100 Batch  760/781 - Loss:  2.419, Seconds: 4.23\n",
      "Average loss for this update: 2.442\n",
      "New Record!\n",
      "Epoch   1/100 Batch  780/781 - Loss:  2.229, Seconds: 3.87\n",
      "Epoch   2/100 Batch   20/781 - Loss:  2.374, Seconds: 4.19\n",
      "Epoch   2/100 Batch   40/781 - Loss:  2.232, Seconds: 3.49\n",
      "Epoch   2/100 Batch   60/781 - Loss:  2.322, Seconds: 4.31\n",
      "Epoch   2/100 Batch   80/781 - Loss:  2.287, Seconds: 4.25\n",
      "Epoch   2/100 Batch  100/781 - Loss:  2.184, Seconds: 4.35\n",
      "Epoch   2/100 Batch  120/781 - Loss:  2.215, Seconds: 3.87\n",
      "Epoch   2/100 Batch  140/781 - Loss:  2.079, Seconds: 4.07\n",
      "Epoch   2/100 Batch  160/781 - Loss:  2.356, Seconds: 3.31\n",
      "Epoch   2/100 Batch  180/781 - Loss:  2.215, Seconds: 3.65\n",
      "Epoch   2/100 Batch  200/781 - Loss:  2.221, Seconds: 4.41\n",
      "Epoch   2/100 Batch  220/781 - Loss:  2.154, Seconds: 4.36\n",
      "Epoch   2/100 Batch  240/781 - Loss:  2.013, Seconds: 4.11\n",
      "Average loss for this update: 2.211\n",
      "New Record!\n",
      "Epoch   2/100 Batch  260/781 - Loss:  2.074, Seconds: 4.61\n",
      "Epoch   2/100 Batch  280/781 - Loss:  2.124, Seconds: 4.41\n",
      "Epoch   2/100 Batch  300/781 - Loss:  2.238, Seconds: 4.11\n",
      "Epoch   2/100 Batch  320/781 - Loss:  2.276, Seconds: 4.33\n",
      "Epoch   2/100 Batch  340/781 - Loss:  2.091, Seconds: 4.67\n",
      "Epoch   2/100 Batch  360/781 - Loss:  2.186, Seconds: 4.27\n",
      "Epoch   2/100 Batch  380/781 - Loss:  2.033, Seconds: 4.63\n",
      "Epoch   2/100 Batch  400/781 - Loss:  2.148, Seconds: 4.62\n",
      "Epoch   2/100 Batch  420/781 - Loss:  2.130, Seconds: 4.09\n",
      "Epoch   2/100 Batch  440/781 - Loss:  2.233, Seconds: 4.75\n",
      "Epoch   2/100 Batch  460/781 - Loss:  2.285, Seconds: 4.33\n",
      "Epoch   2/100 Batch  480/781 - Loss:  2.098, Seconds: 4.21\n",
      "Epoch   2/100 Batch  500/781 - Loss:  2.132, Seconds: 4.57\n",
      "Average loss for this update: 2.157\n",
      "New Record!\n",
      "Epoch   2/100 Batch  520/781 - Loss:  2.093, Seconds: 4.24\n",
      "Epoch   2/100 Batch  540/781 - Loss:  2.097, Seconds: 4.29\n",
      "Epoch   2/100 Batch  560/781 - Loss:  2.055, Seconds: 3.91\n",
      "Epoch   2/100 Batch  580/781 - Loss:  2.177, Seconds: 4.27\n",
      "Epoch   2/100 Batch  600/781 - Loss:  2.329, Seconds: 4.23\n",
      "Epoch   2/100 Batch  620/781 - Loss:  2.219, Seconds: 3.99\n",
      "Epoch   2/100 Batch  640/781 - Loss:  2.115, Seconds: 4.27\n",
      "Epoch   2/100 Batch  660/781 - Loss:  2.047, Seconds: 4.50\n",
      "Epoch   2/100 Batch  680/781 - Loss:  1.999, Seconds: 4.35\n",
      "Epoch   2/100 Batch  700/781 - Loss:  2.156, Seconds: 4.31\n",
      "Epoch   2/100 Batch  720/781 - Loss:  2.297, Seconds: 4.51\n",
      "Epoch   2/100 Batch  740/781 - Loss:  2.160, Seconds: 4.79\n",
      "Epoch   2/100 Batch  760/781 - Loss:  2.161, Seconds: 4.41\n",
      "Average loss for this update: 2.138\n",
      "New Record!\n",
      "Epoch   2/100 Batch  780/781 - Loss:  1.973, Seconds: 3.99\n",
      "Epoch   3/100 Batch   20/781 - Loss:  2.163, Seconds: 4.33\n",
      "Epoch   3/100 Batch   40/781 - Loss:  2.015, Seconds: 3.43\n",
      "Epoch   3/100 Batch   60/781 - Loss:  2.076, Seconds: 4.43\n",
      "Epoch   3/100 Batch   80/781 - Loss:  2.057, Seconds: 4.21\n",
      "Epoch   3/100 Batch  100/781 - Loss:  1.947, Seconds: 4.15\n",
      "Epoch   3/100 Batch  120/781 - Loss:  1.995, Seconds: 3.79\n",
      "Epoch   3/100 Batch  140/781 - Loss:  1.837, Seconds: 3.99\n",
      "Epoch   3/100 Batch  160/781 - Loss:  2.138, Seconds: 3.31\n",
      "Epoch   3/100 Batch  180/781 - Loss:  2.005, Seconds: 3.89\n",
      "Epoch   3/100 Batch  200/781 - Loss:  2.011, Seconds: 4.29\n",
      "Epoch   3/100 Batch  220/781 - Loss:  1.942, Seconds: 4.05\n",
      "Epoch   3/100 Batch  240/781 - Loss:  1.796, Seconds: 4.19\n",
      "Average loss for this update: 1.99\n",
      "New Record!\n",
      "Epoch   3/100 Batch  260/781 - Loss:  1.877, Seconds: 4.49\n",
      "Epoch   3/100 Batch  280/781 - Loss:  1.905, Seconds: 4.19\n",
      "Epoch   3/100 Batch  300/781 - Loss:  2.033, Seconds: 4.19\n",
      "Epoch   3/100 Batch  320/781 - Loss:  2.075, Seconds: 4.27\n",
      "Epoch   3/100 Batch  340/781 - Loss:  1.902, Seconds: 4.75\n",
      "Epoch   3/100 Batch  360/781 - Loss:  1.997, Seconds: 4.39\n",
      "Epoch   3/100 Batch  380/781 - Loss:  1.816, Seconds: 4.47\n",
      "Epoch   3/100 Batch  400/781 - Loss:  1.941, Seconds: 4.63\n",
      "Epoch   3/100 Batch  420/781 - Loss:  1.911, Seconds: 4.03\n",
      "Epoch   3/100 Batch  440/781 - Loss:  2.010, Seconds: 4.53\n",
      "Epoch   3/100 Batch  460/781 - Loss:  2.071, Seconds: 4.39\n",
      "Epoch   3/100 Batch  480/781 - Loss:  1.883, Seconds: 3.99\n",
      "Epoch   3/100 Batch  500/781 - Loss:  1.921, Seconds: 4.25\n",
      "Average loss for this update: 1.947\n",
      "New Record!\n",
      "Epoch   3/100 Batch  520/781 - Loss:  1.861, Seconds: 3.91\n",
      "Epoch   3/100 Batch  540/781 - Loss:  1.889, Seconds: 4.41\n",
      "Epoch   3/100 Batch  560/781 - Loss:  1.842, Seconds: 4.03\n",
      "Epoch   3/100 Batch  580/781 - Loss:  1.997, Seconds: 4.35\n",
      "Epoch   3/100 Batch  600/781 - Loss:  2.124, Seconds: 4.35\n",
      "Epoch   3/100 Batch  620/781 - Loss:  2.016, Seconds: 4.03\n",
      "Epoch   3/100 Batch  640/781 - Loss:  1.915, Seconds: 4.49\n",
      "Epoch   3/100 Batch  660/781 - Loss:  1.837, Seconds: 4.45\n",
      "Epoch   3/100 Batch  680/781 - Loss:  1.817, Seconds: 4.27\n",
      "Epoch   3/100 Batch  700/781 - Loss:  1.956, Seconds: 4.45\n",
      "Epoch   3/100 Batch  720/781 - Loss:  2.116, Seconds: 4.51\n",
      "Epoch   3/100 Batch  740/781 - Loss:  1.975, Seconds: 4.79\n",
      "Epoch   3/100 Batch  760/781 - Loss:  1.988, Seconds: 4.33\n",
      "Average loss for this update: 1.944\n",
      "New Record!\n",
      "Epoch   3/100 Batch  780/781 - Loss:  1.783, Seconds: 3.99\n",
      "Epoch   4/100 Batch   20/781 - Loss:  2.024, Seconds: 4.29\n",
      "Epoch   4/100 Batch   40/781 - Loss:  1.855, Seconds: 3.57\n",
      "Epoch   4/100 Batch   60/781 - Loss:  1.903, Seconds: 4.35\n",
      "Epoch   4/100 Batch   80/781 - Loss:  1.899, Seconds: 4.19\n",
      "Epoch   4/100 Batch  100/781 - Loss:  1.763, Seconds: 4.21\n",
      "Epoch   4/100 Batch  120/781 - Loss:  1.821, Seconds: 3.85\n",
      "Epoch   4/100 Batch  140/781 - Loss:  1.684, Seconds: 4.01\n",
      "Epoch   4/100 Batch  160/781 - Loss:  1.981, Seconds: 3.29\n",
      "Epoch   4/100 Batch  180/781 - Loss:  1.853, Seconds: 3.67\n",
      "Epoch   4/100 Batch  200/781 - Loss:  1.856, Seconds: 4.27\n",
      "Epoch   4/100 Batch  220/781 - Loss:  1.793, Seconds: 4.09\n",
      "Epoch   4/100 Batch  240/781 - Loss:  1.635, Seconds: 4.19\n",
      "Average loss for this update: 1.831\n",
      "New Record!\n",
      "Epoch   4/100 Batch  260/781 - Loss:  1.724, Seconds: 4.55\n",
      "Epoch   4/100 Batch  280/781 - Loss:  1.746, Seconds: 4.33\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   4/100 Batch  300/781 - Loss:  1.882, Seconds: 4.23\n",
      "Epoch   4/100 Batch  320/781 - Loss:  1.929, Seconds: 4.11\n",
      "Epoch   4/100 Batch  340/781 - Loss:  1.754, Seconds: 4.65\n",
      "Epoch   4/100 Batch  360/781 - Loss:  1.842, Seconds: 4.25\n",
      "Epoch   4/100 Batch  380/781 - Loss:  1.654, Seconds: 4.67\n",
      "Epoch   4/100 Batch  400/781 - Loss:  1.781, Seconds: 4.69\n",
      "Epoch   4/100 Batch  420/781 - Loss:  1.760, Seconds: 4.03\n",
      "Epoch   4/100 Batch  440/781 - Loss:  1.869, Seconds: 4.53\n",
      "Epoch   4/100 Batch  460/781 - Loss:  1.923, Seconds: 4.55\n",
      "Epoch   4/100 Batch  480/781 - Loss:  1.737, Seconds: 4.01\n",
      "Epoch   4/100 Batch  500/781 - Loss:  1.772, Seconds: 4.09\n",
      "Average loss for this update: 1.793\n",
      "New Record!\n",
      "Epoch   4/100 Batch  520/781 - Loss:  1.681, Seconds: 3.91\n",
      "Epoch   4/100 Batch  540/781 - Loss:  1.740, Seconds: 4.49\n",
      "Epoch   4/100 Batch  560/781 - Loss:  1.693, Seconds: 4.33\n",
      "Epoch   4/100 Batch  580/781 - Loss:  1.859, Seconds: 4.25\n",
      "Epoch   4/100 Batch  600/781 - Loss:  1.982, Seconds: 4.47\n",
      "Epoch   4/100 Batch  620/781 - Loss:  1.871, Seconds: 4.01\n",
      "Epoch   4/100 Batch  640/781 - Loss:  1.775, Seconds: 4.25\n",
      "Epoch   4/100 Batch  660/781 - Loss:  1.680, Seconds: 4.67\n",
      "Epoch   4/100 Batch  680/781 - Loss:  1.678, Seconds: 4.45\n",
      "Epoch   4/100 Batch  700/781 - Loss:  1.821, Seconds: 4.35\n",
      "Epoch   4/100 Batch  720/781 - Loss:  1.983, Seconds: 4.53\n",
      "Epoch   4/100 Batch  740/781 - Loss:  1.857, Seconds: 4.91\n",
      "Epoch   4/100 Batch  760/781 - Loss:  1.840, Seconds: 4.41\n",
      "Average loss for this update: 1.803\n",
      "No Improvement.\n",
      "Epoch   4/100 Batch  780/781 - Loss:  1.640, Seconds: 4.05\n",
      "Epoch   5/100 Batch   20/781 - Loss:  1.914, Seconds: 4.47\n",
      "Epoch   5/100 Batch   40/781 - Loss:  1.728, Seconds: 3.60\n",
      "Epoch   5/100 Batch   60/781 - Loss:  1.778, Seconds: 4.61\n",
      "Epoch   5/100 Batch   80/781 - Loss:  1.771, Seconds: 4.45\n",
      "Epoch   5/100 Batch  100/781 - Loss:  1.626, Seconds: 4.25\n",
      "Epoch   5/100 Batch  120/781 - Loss:  1.683, Seconds: 3.89\n",
      "Epoch   5/100 Batch  140/781 - Loss:  1.561, Seconds: 4.01\n",
      "Epoch   5/100 Batch  160/781 - Loss:  1.854, Seconds: 3.41\n",
      "Epoch   5/100 Batch  180/781 - Loss:  1.734, Seconds: 3.83\n",
      "Epoch   5/100 Batch  200/781 - Loss:  1.722, Seconds: 4.33\n",
      "Epoch   5/100 Batch  220/781 - Loss:  1.658, Seconds: 4.25\n",
      "Epoch   5/100 Batch  240/781 - Loss:  1.502, Seconds: 4.13\n",
      "Average loss for this update: 1.702\n",
      "New Record!\n",
      "Epoch   5/100 Batch  260/781 - Loss:  1.586, Seconds: 4.65\n",
      "Epoch   5/100 Batch  280/781 - Loss:  1.648, Seconds: 4.23\n",
      "Epoch   5/100 Batch  300/781 - Loss:  1.751, Seconds: 4.27\n",
      "Epoch   5/100 Batch  320/781 - Loss:  1.813, Seconds: 4.11\n",
      "Epoch   5/100 Batch  340/781 - Loss:  1.650, Seconds: 4.67\n",
      "Epoch   5/100 Batch  360/781 - Loss:  1.718, Seconds: 4.49\n",
      "Epoch   5/100 Batch  380/781 - Loss:  1.521, Seconds: 4.45\n",
      "Epoch   5/100 Batch  400/781 - Loss:  1.666, Seconds: 4.57\n",
      "Epoch   5/100 Batch  420/781 - Loss:  1.639, Seconds: 3.97\n",
      "Epoch   5/100 Batch  440/781 - Loss:  1.749, Seconds: 4.57\n",
      "Epoch   5/100 Batch  460/781 - Loss:  1.810, Seconds: 4.36\n",
      "Epoch   5/100 Batch  480/781 - Loss:  1.624, Seconds: 4.17\n",
      "Epoch   5/100 Batch  500/781 - Loss:  1.654, Seconds: 4.31\n",
      "Average loss for this update: 1.676\n",
      "New Record!\n",
      "Epoch   5/100 Batch  520/781 - Loss:  1.565, Seconds: 3.97\n",
      "Epoch   5/100 Batch  540/781 - Loss:  1.616, Seconds: 4.43\n",
      "Epoch   5/100 Batch  560/781 - Loss:  1.576, Seconds: 3.95\n",
      "Epoch   5/100 Batch  580/781 - Loss:  1.736, Seconds: 4.37\n",
      "Epoch   5/100 Batch  600/781 - Loss:  1.860, Seconds: 4.23\n",
      "Epoch   5/100 Batch  620/781 - Loss:  1.747, Seconds: 4.33\n",
      "Epoch   5/100 Batch  640/781 - Loss:  1.660, Seconds: 4.45\n",
      "Epoch   5/100 Batch  660/781 - Loss:  1.549, Seconds: 4.45\n",
      "Epoch   5/100 Batch  680/781 - Loss:  1.560, Seconds: 4.17\n",
      "Epoch   5/100 Batch  700/781 - Loss:  1.698, Seconds: 4.29\n",
      "Epoch   5/100 Batch  720/781 - Loss:  1.861, Seconds: 4.61\n",
      "Epoch   5/100 Batch  740/781 - Loss:  1.749, Seconds: 4.89\n",
      "Epoch   5/100 Batch  760/781 - Loss:  1.736, Seconds: 4.47\n",
      "Average loss for this update: 1.683\n",
      "No Improvement.\n",
      "Epoch   5/100 Batch  780/781 - Loss:  1.523, Seconds: 4.29\n",
      "Epoch   6/100 Batch   20/781 - Loss:  1.793, Seconds: 4.27\n",
      "Epoch   6/100 Batch   40/781 - Loss:  1.642, Seconds: 3.55\n",
      "Epoch   6/100 Batch   60/781 - Loss:  1.658, Seconds: 4.39\n",
      "Epoch   6/100 Batch   80/781 - Loss:  1.662, Seconds: 4.23\n",
      "Epoch   6/100 Batch  100/781 - Loss:  1.504, Seconds: 4.37\n",
      "Epoch   6/100 Batch  120/781 - Loss:  1.577, Seconds: 4.01\n",
      "Epoch   6/100 Batch  140/781 - Loss:  1.461, Seconds: 3.99\n",
      "Epoch   6/100 Batch  160/781 - Loss:  1.759, Seconds: 3.33\n",
      "Epoch   6/100 Batch  180/781 - Loss:  1.633, Seconds: 3.73\n",
      "Epoch   6/100 Batch  200/781 - Loss:  1.612, Seconds: 4.19\n",
      "Epoch   6/100 Batch  220/781 - Loss:  1.565, Seconds: 4.07\n",
      "Epoch   6/100 Batch  240/781 - Loss:  1.395, Seconds: 4.07\n",
      "Average loss for this update: 1.597\n",
      "New Record!\n",
      "Epoch   6/100 Batch  260/781 - Loss:  1.490, Seconds: 4.61\n",
      "Epoch   6/100 Batch  280/781 - Loss:  1.541, Seconds: 4.25\n",
      "Epoch   6/100 Batch  300/781 - Loss:  1.661, Seconds: 4.39\n",
      "Epoch   6/100 Batch  320/781 - Loss:  1.706, Seconds: 4.23\n",
      "Epoch   6/100 Batch  340/781 - Loss:  1.544, Seconds: 4.51\n",
      "Epoch   6/100 Batch  360/781 - Loss:  1.623, Seconds: 4.45\n",
      "Epoch   6/100 Batch  380/781 - Loss:  1.430, Seconds: 4.61\n",
      "Epoch   6/100 Batch  400/781 - Loss:  1.567, Seconds: 4.49\n",
      "Epoch   6/100 Batch  420/781 - Loss:  1.535, Seconds: 3.97\n",
      "Epoch   6/100 Batch  440/781 - Loss:  1.642, Seconds: 4.67\n",
      "Epoch   6/100 Batch  460/781 - Loss:  1.701, Seconds: 4.35\n",
      "Epoch   6/100 Batch  480/781 - Loss:  1.529, Seconds: 4.07\n",
      "Epoch   6/100 Batch  500/781 - Loss:  1.554, Seconds: 4.33\n",
      "Average loss for this update: 1.575\n",
      "New Record!\n",
      "Epoch   6/100 Batch  520/781 - Loss:  1.462, Seconds: 3.91\n",
      "Epoch   6/100 Batch  540/781 - Loss:  1.530, Seconds: 4.59\n",
      "Epoch   6/100 Batch  560/781 - Loss:  1.482, Seconds: 4.11\n",
      "Epoch   6/100 Batch  580/781 - Loss:  1.658, Seconds: 4.07\n",
      "Epoch   6/100 Batch  600/781 - Loss:  1.717, Seconds: 4.17\n",
      "Epoch   6/100 Batch  620/781 - Loss:  1.624, Seconds: 4.05\n",
      "Epoch   6/100 Batch  640/781 - Loss:  1.551, Seconds: 4.29\n",
      "Epoch   6/100 Batch  660/781 - Loss:  1.433, Seconds: 4.39\n",
      "Epoch   6/100 Batch  680/781 - Loss:  1.468, Seconds: 4.35\n",
      "Epoch   6/100 Batch  700/781 - Loss:  1.611, Seconds: 4.63\n",
      "Epoch   6/100 Batch  720/781 - Loss:  1.763, Seconds: 4.45\n",
      "Epoch   6/100 Batch  740/781 - Loss:  1.646, Seconds: 4.79\n",
      "Epoch   6/100 Batch  760/781 - Loss:  1.631, Seconds: 4.33\n",
      "Average loss for this update: 1.582\n",
      "No Improvement.\n",
      "Epoch   6/100 Batch  780/781 - Loss:  1.434, Seconds: 3.85\n",
      "Epoch   7/100 Batch   20/781 - Loss:  1.688, Seconds: 4.19\n",
      "Epoch   7/100 Batch   40/781 - Loss:  1.547, Seconds: 3.43\n",
      "Epoch   7/100 Batch   60/781 - Loss:  1.570, Seconds: 4.53\n",
      "Epoch   7/100 Batch   80/781 - Loss:  1.564, Seconds: 4.23\n",
      "Epoch   7/100 Batch  100/781 - Loss:  1.413, Seconds: 4.31\n",
      "Epoch   7/100 Batch  120/781 - Loss:  1.472, Seconds: 3.81\n",
      "Epoch   7/100 Batch  140/781 - Loss:  1.395, Seconds: 4.07\n",
      "Epoch   7/100 Batch  160/781 - Loss:  1.645, Seconds: 3.47\n",
      "Epoch   7/100 Batch  180/781 - Loss:  1.555, Seconds: 3.79\n",
      "Epoch   7/100 Batch  200/781 - Loss:  1.534, Seconds: 4.39\n",
      "Epoch   7/100 Batch  220/781 - Loss:  1.498, Seconds: 4.09\n",
      "Epoch   7/100 Batch  240/781 - Loss:  1.307, Seconds: 4.27\n",
      "Average loss for this update: 1.508\n",
      "New Record!\n",
      "Epoch   7/100 Batch  260/781 - Loss:  1.407, Seconds: 4.59\n",
      "Epoch   7/100 Batch  280/781 - Loss:  1.459, Seconds: 4.33\n",
      "Epoch   7/100 Batch  300/781 - Loss:  1.557, Seconds: 4.29\n",
      "Epoch   7/100 Batch  320/781 - Loss:  1.616, Seconds: 4.31\n",
      "Epoch   7/100 Batch  340/781 - Loss:  1.462, Seconds: 4.47\n",
      "Epoch   7/100 Batch  360/781 - Loss:  1.534, Seconds: 4.47\n",
      "Epoch   7/100 Batch  380/781 - Loss:  1.341, Seconds: 4.47\n",
      "Epoch   7/100 Batch  400/781 - Loss:  1.497, Seconds: 4.53\n",
      "Epoch   7/100 Batch  420/781 - Loss:  1.442, Seconds: 4.05\n",
      "Epoch   7/100 Batch  440/781 - Loss:  1.561, Seconds: 4.61\n",
      "Epoch   7/100 Batch  460/781 - Loss:  1.603, Seconds: 4.35\n",
      "Epoch   7/100 Batch  480/781 - Loss:  1.442, Seconds: 3.99\n",
      "Epoch   7/100 Batch  500/781 - Loss:  1.476, Seconds: 4.15\n",
      "Average loss for this update: 1.488\n",
      "New Record!\n",
      "Epoch   7/100 Batch  520/781 - Loss:  1.375, Seconds: 3.99\n",
      "Epoch   7/100 Batch  540/781 - Loss:  1.457, Seconds: 4.47\n",
      "Epoch   7/100 Batch  560/781 - Loss:  1.406, Seconds: 3.95\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   7/100 Batch  580/781 - Loss:  1.564, Seconds: 4.35\n",
      "Epoch   7/100 Batch  600/781 - Loss:  1.615, Seconds: 4.39\n",
      "Epoch   7/100 Batch  620/781 - Loss:  1.533, Seconds: 3.99\n",
      "Epoch   7/100 Batch  640/781 - Loss:  1.448, Seconds: 4.19\n",
      "Epoch   7/100 Batch  660/781 - Loss:  1.343, Seconds: 4.67\n",
      "Epoch   7/100 Batch  680/781 - Loss:  1.388, Seconds: 4.23\n",
      "Epoch   7/100 Batch  700/781 - Loss:  1.504, Seconds: 4.37\n",
      "Epoch   7/100 Batch  720/781 - Loss:  1.675, Seconds: 4.79\n",
      "Epoch   7/100 Batch  740/781 - Loss:  1.553, Seconds: 4.89\n",
      "Epoch   7/100 Batch  760/781 - Loss:  1.534, Seconds: 4.39\n",
      "Average loss for this update: 1.491\n",
      "No Improvement.\n",
      "Epoch   7/100 Batch  780/781 - Loss:  1.354, Seconds: 4.03\n",
      "Epoch   8/100 Batch   20/781 - Loss:  1.612, Seconds: 4.39\n",
      "Epoch   8/100 Batch   40/781 - Loss:  1.480, Seconds: 3.47\n",
      "Epoch   8/100 Batch   60/781 - Loss:  1.498, Seconds: 4.51\n",
      "Epoch   8/100 Batch   80/781 - Loss:  1.474, Seconds: 4.25\n",
      "Epoch   8/100 Batch  100/781 - Loss:  1.334, Seconds: 4.35\n",
      "Epoch   8/100 Batch  120/781 - Loss:  1.410, Seconds: 3.91\n",
      "Epoch   8/100 Batch  140/781 - Loss:  1.320, Seconds: 4.25\n",
      "Epoch   8/100 Batch  160/781 - Loss:  1.559, Seconds: 3.19\n",
      "Epoch   8/100 Batch  180/781 - Loss:  1.465, Seconds: 3.67\n",
      "Epoch   8/100 Batch  200/781 - Loss:  1.444, Seconds: 4.35\n",
      "Epoch   8/100 Batch  220/781 - Loss:  1.420, Seconds: 4.05\n",
      "Epoch   8/100 Batch  240/781 - Loss:  1.253, Seconds: 4.05\n",
      "Average loss for this update: 1.432\n",
      "New Record!\n",
      "Epoch   8/100 Batch  260/781 - Loss:  1.336, Seconds: 4.77\n",
      "Epoch   8/100 Batch  280/781 - Loss:  1.372, Seconds: 4.29\n",
      "Epoch   8/100 Batch  300/781 - Loss:  1.496, Seconds: 4.19\n",
      "Epoch   8/100 Batch  320/781 - Loss:  1.536, Seconds: 4.11\n",
      "Epoch   8/100 Batch  340/781 - Loss:  1.385, Seconds: 4.57\n",
      "Epoch   8/100 Batch  360/781 - Loss:  1.452, Seconds: 4.25\n",
      "Epoch   8/100 Batch  380/781 - Loss:  1.284, Seconds: 4.59\n",
      "Epoch   8/100 Batch  400/781 - Loss:  1.421, Seconds: 4.84\n",
      "Epoch   8/100 Batch  420/781 - Loss:  1.361, Seconds: 4.09\n",
      "Epoch   8/100 Batch  440/781 - Loss:  1.476, Seconds: 4.67\n",
      "Epoch   8/100 Batch  460/781 - Loss:  1.515, Seconds: 4.53\n",
      "Epoch   8/100 Batch  480/781 - Loss:  1.371, Seconds: 4.11\n",
      "Epoch   8/100 Batch  500/781 - Loss:  1.401, Seconds: 4.27\n",
      "Average loss for this update: 1.413\n",
      "New Record!\n",
      "Epoch   8/100 Batch  520/781 - Loss:  1.309, Seconds: 3.93\n",
      "Epoch   8/100 Batch  540/781 - Loss:  1.379, Seconds: 4.45\n",
      "Epoch   8/100 Batch  560/781 - Loss:  1.333, Seconds: 3.97\n",
      "Epoch   8/100 Batch  580/781 - Loss:  1.476, Seconds: 4.23\n",
      "Epoch   8/100 Batch  600/781 - Loss:  1.532, Seconds: 4.17\n",
      "Epoch   8/100 Batch  620/781 - Loss:  1.465, Seconds: 4.19\n",
      "Epoch   8/100 Batch  640/781 - Loss:  1.369, Seconds: 4.27\n",
      "Epoch   8/100 Batch  660/781 - Loss:  1.268, Seconds: 4.65\n",
      "Epoch   8/100 Batch  680/781 - Loss:  1.312, Seconds: 4.53\n",
      "Epoch   8/100 Batch  700/781 - Loss:  1.419, Seconds: 4.55\n",
      "Epoch   8/100 Batch  720/781 - Loss:  1.588, Seconds: 4.61\n",
      "Epoch   8/100 Batch  740/781 - Loss:  1.482, Seconds: 4.87\n",
      "Epoch   8/100 Batch  760/781 - Loss:  1.471, Seconds: 4.53\n",
      "Average loss for this update: 1.414\n",
      "No Improvement.\n",
      "Epoch   8/100 Batch  780/781 - Loss:  1.285, Seconds: 4.13\n",
      "Epoch   9/100 Batch   20/781 - Loss:  1.542, Seconds: 4.19\n",
      "Epoch   9/100 Batch   40/781 - Loss:  1.404, Seconds: 3.47\n",
      "Epoch   9/100 Batch   60/781 - Loss:  1.421, Seconds: 4.47\n",
      "Epoch   9/100 Batch   80/781 - Loss:  1.407, Seconds: 4.31\n",
      "Epoch   9/100 Batch  100/781 - Loss:  1.268, Seconds: 4.25\n",
      "Epoch   9/100 Batch  120/781 - Loss:  1.336, Seconds: 3.81\n",
      "Epoch   9/100 Batch  140/781 - Loss:  1.259, Seconds: 4.01\n",
      "Epoch   9/100 Batch  160/781 - Loss:  1.500, Seconds: 3.33\n",
      "Epoch   9/100 Batch  180/781 - Loss:  1.395, Seconds: 3.79\n",
      "Epoch   9/100 Batch  200/781 - Loss:  1.392, Seconds: 4.47\n",
      "Epoch   9/100 Batch  220/781 - Loss:  1.341, Seconds: 4.05\n",
      "Epoch   9/100 Batch  240/781 - Loss:  1.192, Seconds: 4.09\n",
      "Average loss for this update: 1.364\n",
      "New Record!\n",
      "Epoch   9/100 Batch  260/781 - Loss:  1.268, Seconds: 4.59\n",
      "Epoch   9/100 Batch  280/781 - Loss:  1.320, Seconds: 4.33\n",
      "Epoch   9/100 Batch  300/781 - Loss:  1.412, Seconds: 4.11\n",
      "Epoch   9/100 Batch  320/781 - Loss:  1.479, Seconds: 4.25\n",
      "Epoch   9/100 Batch  340/781 - Loss:  1.317, Seconds: 4.67\n",
      "Epoch   9/100 Batch  360/781 - Loss:  1.379, Seconds: 4.53\n",
      "Epoch   9/100 Batch  380/781 - Loss:  1.222, Seconds: 4.81\n",
      "Epoch   9/100 Batch  400/781 - Loss:  1.358, Seconds: 4.67\n",
      "Epoch   9/100 Batch  420/781 - Loss:  1.299, Seconds: 4.03\n",
      "Epoch   9/100 Batch  440/781 - Loss:  1.388, Seconds: 4.67\n",
      "Epoch   9/100 Batch  460/781 - Loss:  1.451, Seconds: 4.45\n",
      "Epoch   9/100 Batch  480/781 - Loss:  1.303, Seconds: 4.01\n",
      "Epoch   9/100 Batch  500/781 - Loss:  1.344, Seconds: 4.35\n",
      "Average loss for this update: 1.346\n",
      "New Record!\n",
      "Epoch   9/100 Batch  520/781 - Loss:  1.245, Seconds: 3.93\n",
      "Epoch   9/100 Batch  540/781 - Loss:  1.300, Seconds: 4.61\n",
      "Epoch   9/100 Batch  560/781 - Loss:  1.257, Seconds: 4.07\n",
      "Epoch   9/100 Batch  580/781 - Loss:  1.409, Seconds: 4.23\n",
      "Epoch   9/100 Batch  600/781 - Loss:  1.465, Seconds: 4.37\n",
      "Epoch   9/100 Batch  620/781 - Loss:  1.373, Seconds: 3.99\n",
      "Epoch   9/100 Batch  640/781 - Loss:  1.316, Seconds: 4.29\n",
      "Epoch   9/100 Batch  660/781 - Loss:  1.198, Seconds: 4.61\n",
      "Epoch   9/100 Batch  680/781 - Loss:  1.263, Seconds: 4.31\n",
      "Epoch   9/100 Batch  700/781 - Loss:  1.369, Seconds: 4.55\n",
      "Epoch   9/100 Batch  720/781 - Loss:  1.511, Seconds: 4.77\n",
      "Epoch   9/100 Batch  740/781 - Loss:  1.425, Seconds: 4.73\n",
      "Epoch   9/100 Batch  760/781 - Loss:  1.413, Seconds: 4.19\n",
      "Average loss for this update: 1.349\n",
      "No Improvement.\n",
      "Epoch   9/100 Batch  780/781 - Loss:  1.234, Seconds: 3.89\n",
      "Epoch  10/100 Batch   20/781 - Loss:  1.470, Seconds: 4.47\n",
      "Epoch  10/100 Batch   40/781 - Loss:  1.351, Seconds: 3.53\n",
      "Epoch  10/100 Batch   60/781 - Loss:  1.343, Seconds: 4.73\n",
      "Epoch  10/100 Batch   80/781 - Loss:  1.350, Seconds: 4.45\n",
      "Epoch  10/100 Batch  100/781 - Loss:  1.202, Seconds: 4.25\n",
      "Epoch  10/100 Batch  120/781 - Loss:  1.274, Seconds: 3.79\n",
      "Epoch  10/100 Batch  140/781 - Loss:  1.189, Seconds: 4.07\n",
      "Epoch  10/100 Batch  160/781 - Loss:  1.432, Seconds: 3.25\n",
      "Epoch  10/100 Batch  180/781 - Loss:  1.339, Seconds: 3.65\n",
      "Epoch  10/100 Batch  200/781 - Loss:  1.334, Seconds: 4.65\n",
      "Epoch  10/100 Batch  220/781 - Loss:  1.299, Seconds: 4.01\n",
      "Epoch  10/100 Batch  240/781 - Loss:  1.140, Seconds: 4.07\n",
      "Average loss for this update: 1.304\n",
      "New Record!\n",
      "Epoch  10/100 Batch  260/781 - Loss:  1.210, Seconds: 4.53\n",
      "Epoch  10/100 Batch  280/781 - Loss:  1.272, Seconds: 4.45\n",
      "Epoch  10/100 Batch  300/781 - Loss:  1.351, Seconds: 4.19\n",
      "Epoch  10/100 Batch  320/781 - Loss:  1.399, Seconds: 4.25\n",
      "Epoch  10/100 Batch  340/781 - Loss:  1.273, Seconds: 4.63\n",
      "Epoch  10/100 Batch  360/781 - Loss:  1.301, Seconds: 4.59\n",
      "Epoch  10/100 Batch  380/781 - Loss:  1.164, Seconds: 4.69\n",
      "Epoch  10/100 Batch  400/781 - Loss:  1.284, Seconds: 4.75\n",
      "Epoch  10/100 Batch  420/781 - Loss:  1.221, Seconds: 3.95\n",
      "Epoch  10/100 Batch  440/781 - Loss:  1.333, Seconds: 4.67\n",
      "Epoch  10/100 Batch  460/781 - Loss:  1.383, Seconds: 4.61\n",
      "Epoch  10/100 Batch  480/781 - Loss:  1.240, Seconds: 4.19\n",
      "Epoch  10/100 Batch  500/781 - Loss:  1.298, Seconds: 4.15\n",
      "Average loss for this update: 1.282\n",
      "New Record!\n",
      "Epoch  10/100 Batch  520/781 - Loss:  1.167, Seconds: 4.07\n",
      "Epoch  10/100 Batch  540/781 - Loss:  1.242, Seconds: 4.47\n",
      "Epoch  10/100 Batch  560/781 - Loss:  1.201, Seconds: 4.03\n",
      "Epoch  10/100 Batch  580/781 - Loss:  1.343, Seconds: 4.33\n",
      "Epoch  10/100 Batch  600/781 - Loss:  1.399, Seconds: 4.63\n",
      "Epoch  10/100 Batch  620/781 - Loss:  1.308, Seconds: 4.37\n",
      "Epoch  10/100 Batch  640/781 - Loss:  1.260, Seconds: 4.37\n",
      "Epoch  10/100 Batch  660/781 - Loss:  1.139, Seconds: 4.35\n",
      "Epoch  10/100 Batch  680/781 - Loss:  1.203, Seconds: 4.41\n",
      "Epoch  10/100 Batch  700/781 - Loss:  1.299, Seconds: 4.39\n",
      "Epoch  10/100 Batch  720/781 - Loss:  1.437, Seconds: 4.59\n",
      "Epoch  10/100 Batch  740/781 - Loss:  1.346, Seconds: 4.73\n",
      "Epoch  10/100 Batch  760/781 - Loss:  1.340, Seconds: 4.27\n",
      "Average loss for this update: 1.284\n",
      "No Improvement.\n",
      "Epoch  10/100 Batch  780/781 - Loss:  1.176, Seconds: 4.33\n",
      "Epoch  11/100 Batch   20/781 - Loss:  1.408, Seconds: 4.49\n",
      "Epoch  11/100 Batch   40/781 - Loss:  1.280, Seconds: 3.41\n",
      "Epoch  11/100 Batch   60/781 - Loss:  1.287, Seconds: 4.65\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  11/100 Batch   80/781 - Loss:  1.300, Seconds: 4.27\n",
      "Epoch  11/100 Batch  100/781 - Loss:  1.147, Seconds: 4.39\n",
      "Epoch  11/100 Batch  120/781 - Loss:  1.242, Seconds: 3.79\n",
      "Epoch  11/100 Batch  140/781 - Loss:  1.150, Seconds: 4.17\n",
      "Epoch  11/100 Batch  160/781 - Loss:  1.364, Seconds: 3.25\n",
      "Epoch  11/100 Batch  180/781 - Loss:  1.293, Seconds: 3.67\n",
      "Epoch  11/100 Batch  200/781 - Loss:  1.263, Seconds: 4.33\n",
      "Epoch  11/100 Batch  220/781 - Loss:  1.238, Seconds: 4.09\n",
      "Epoch  11/100 Batch  240/781 - Loss:  1.088, Seconds: 4.01\n",
      "Average loss for this update: 1.249\n",
      "New Record!\n",
      "Epoch  11/100 Batch  260/781 - Loss:  1.169, Seconds: 4.55\n",
      "Epoch  11/100 Batch  280/781 - Loss:  1.206, Seconds: 4.41\n",
      "Epoch  11/100 Batch  300/781 - Loss:  1.294, Seconds: 4.35\n",
      "Epoch  11/100 Batch  320/781 - Loss:  1.366, Seconds: 4.17\n",
      "Epoch  11/100 Batch  340/781 - Loss:  1.212, Seconds: 4.67\n",
      "Epoch  11/100 Batch  360/781 - Loss:  1.248, Seconds: 4.39\n",
      "Epoch  11/100 Batch  380/781 - Loss:  1.105, Seconds: 4.61\n",
      "Epoch  11/100 Batch  400/781 - Loss:  1.236, Seconds: 4.71\n",
      "Epoch  11/100 Batch  420/781 - Loss:  1.175, Seconds: 3.97\n",
      "Epoch  11/100 Batch  440/781 - Loss:  1.276, Seconds: 4.83\n",
      "Epoch  11/100 Batch  460/781 - Loss:  1.346, Seconds: 4.59\n",
      "Epoch  11/100 Batch  480/781 - Loss:  1.198, Seconds: 4.01\n",
      "Epoch  11/100 Batch  500/781 - Loss:  1.244, Seconds: 4.39\n",
      "Average loss for this update: 1.233\n",
      "New Record!\n",
      "Epoch  11/100 Batch  520/781 - Loss:  1.132, Seconds: 3.83\n",
      "Epoch  11/100 Batch  540/781 - Loss:  1.182, Seconds: 4.65\n",
      "Epoch  11/100 Batch  560/781 - Loss:  1.153, Seconds: 3.89\n",
      "Epoch  11/100 Batch  580/781 - Loss:  1.290, Seconds: 4.23\n",
      "Epoch  11/100 Batch  600/781 - Loss:  1.335, Seconds: 4.17\n",
      "Epoch  11/100 Batch  620/781 - Loss:  1.250, Seconds: 4.09\n",
      "Epoch  11/100 Batch  640/781 - Loss:  1.202, Seconds: 4.25\n",
      "Epoch  11/100 Batch  660/781 - Loss:  1.085, Seconds: 4.45\n",
      "Epoch  11/100 Batch  680/781 - Loss:  1.150, Seconds: 4.27\n",
      "Epoch  11/100 Batch  700/781 - Loss:  1.248, Seconds: 4.35\n",
      "Epoch  11/100 Batch  720/781 - Loss:  1.394, Seconds: 4.65\n",
      "Epoch  11/100 Batch  740/781 - Loss:  1.287, Seconds: 4.75\n",
      "Epoch  11/100 Batch  760/781 - Loss:  1.294, Seconds: 4.61\n",
      "Average loss for this update: 1.231\n",
      "New Record!\n",
      "Epoch  11/100 Batch  780/781 - Loss:  1.121, Seconds: 3.97\n",
      "Epoch  12/100 Batch   20/781 - Loss:  1.349, Seconds: 4.37\n",
      "Epoch  12/100 Batch   40/781 - Loss:  1.242, Seconds: 3.45\n",
      "Epoch  12/100 Batch   60/781 - Loss:  1.216, Seconds: 4.39\n",
      "Epoch  12/100 Batch   80/781 - Loss:  1.235, Seconds: 4.41\n",
      "Epoch  12/100 Batch  100/781 - Loss:  1.096, Seconds: 4.31\n",
      "Epoch  12/100 Batch  120/781 - Loss:  1.184, Seconds: 3.99\n",
      "Epoch  12/100 Batch  140/781 - Loss:  1.113, Seconds: 4.25\n",
      "Epoch  12/100 Batch  160/781 - Loss:  1.303, Seconds: 3.27\n",
      "Epoch  12/100 Batch  180/781 - Loss:  1.239, Seconds: 3.71\n",
      "Epoch  12/100 Batch  200/781 - Loss:  1.202, Seconds: 4.39\n",
      "Epoch  12/100 Batch  220/781 - Loss:  1.175, Seconds: 4.11\n",
      "Epoch  12/100 Batch  240/781 - Loss:  1.029, Seconds: 4.22\n",
      "Average loss for this update: 1.193\n",
      "New Record!\n",
      "Epoch  12/100 Batch  260/781 - Loss:  1.113, Seconds: 4.53\n",
      "Epoch  12/100 Batch  280/781 - Loss:  1.160, Seconds: 4.39\n",
      "Epoch  12/100 Batch  300/781 - Loss:  1.231, Seconds: 4.23\n",
      "Epoch  12/100 Batch  320/781 - Loss:  1.298, Seconds: 4.17\n",
      "Epoch  12/100 Batch  340/781 - Loss:  1.161, Seconds: 4.51\n",
      "Epoch  12/100 Batch  360/781 - Loss:  1.190, Seconds: 4.43\n",
      "Epoch  12/100 Batch  380/781 - Loss:  1.045, Seconds: 4.59\n",
      "Epoch  12/100 Batch  400/781 - Loss:  1.182, Seconds: 4.51\n",
      "Epoch  12/100 Batch  420/781 - Loss:  1.127, Seconds: 3.95\n",
      "Epoch  12/100 Batch  440/781 - Loss:  1.213, Seconds: 4.75\n",
      "Epoch  12/100 Batch  460/781 - Loss:  1.280, Seconds: 4.53\n",
      "Epoch  12/100 Batch  480/781 - Loss:  1.149, Seconds: 4.19\n",
      "Epoch  12/100 Batch  500/781 - Loss:  1.196, Seconds: 4.19\n",
      "Average loss for this update: 1.177\n",
      "New Record!\n",
      "Epoch  12/100 Batch  520/781 - Loss:  1.083, Seconds: 3.99\n",
      "Epoch  12/100 Batch  540/781 - Loss:  1.148, Seconds: 4.39\n",
      "Epoch  12/100 Batch  560/781 - Loss:  1.103, Seconds: 3.95\n",
      "Epoch  12/100 Batch  580/781 - Loss:  1.230, Seconds: 4.19\n",
      "Epoch  12/100 Batch  600/781 - Loss:  1.295, Seconds: 4.21\n",
      "Epoch  12/100 Batch  620/781 - Loss:  1.208, Seconds: 4.09\n",
      "Epoch  12/100 Batch  640/781 - Loss:  1.152, Seconds: 4.41\n",
      "Epoch  12/100 Batch  660/781 - Loss:  1.040, Seconds: 4.51\n",
      "Epoch  12/100 Batch  680/781 - Loss:  1.108, Seconds: 4.37\n",
      "Epoch  12/100 Batch  700/781 - Loss:  1.192, Seconds: 4.31\n",
      "Epoch  12/100 Batch  720/781 - Loss:  1.338, Seconds: 4.39\n",
      "Epoch  12/100 Batch  740/781 - Loss:  1.234, Seconds: 4.79\n",
      "Epoch  12/100 Batch  760/781 - Loss:  1.241, Seconds: 4.33\n",
      "Average loss for this update: 1.184\n",
      "No Improvement.\n",
      "Epoch  12/100 Batch  780/781 - Loss:  1.095, Seconds: 3.89\n",
      "Epoch  13/100 Batch   20/781 - Loss:  1.297, Seconds: 4.33\n",
      "Epoch  13/100 Batch   40/781 - Loss:  1.190, Seconds: 3.41\n",
      "Epoch  13/100 Batch   60/781 - Loss:  1.192, Seconds: 4.51\n",
      "Epoch  13/100 Batch   80/781 - Loss:  1.197, Seconds: 4.49\n",
      "Epoch  13/100 Batch  100/781 - Loss:  1.058, Seconds: 4.13\n",
      "Epoch  13/100 Batch  120/781 - Loss:  1.144, Seconds: 3.99\n",
      "Epoch  13/100 Batch  140/781 - Loss:  1.065, Seconds: 4.09\n",
      "Epoch  13/100 Batch  160/781 - Loss:  1.246, Seconds: 3.27\n",
      "Epoch  13/100 Batch  180/781 - Loss:  1.194, Seconds: 3.65\n",
      "Epoch  13/100 Batch  200/781 - Loss:  1.164, Seconds: 4.35\n",
      "Epoch  13/100 Batch  220/781 - Loss:  1.142, Seconds: 4.25\n",
      "Epoch  13/100 Batch  240/781 - Loss:  0.991, Seconds: 3.99\n",
      "Average loss for this update: 1.151\n",
      "New Record!\n",
      "Epoch  13/100 Batch  260/781 - Loss:  1.075, Seconds: 4.61\n",
      "Epoch  13/100 Batch  280/781 - Loss:  1.125, Seconds: 4.33\n",
      "Epoch  13/100 Batch  300/781 - Loss:  1.210, Seconds: 4.13\n",
      "Epoch  13/100 Batch  320/781 - Loss:  1.260, Seconds: 4.29\n",
      "Epoch  13/100 Batch  340/781 - Loss:  1.127, Seconds: 4.67\n",
      "Epoch  13/100 Batch  360/781 - Loss:  1.152, Seconds: 4.37\n",
      "Epoch  13/100 Batch  380/781 - Loss:  1.007, Seconds: 4.73\n",
      "Epoch  13/100 Batch  400/781 - Loss:  1.149, Seconds: 4.59\n",
      "Epoch  13/100 Batch  420/781 - Loss:  1.071, Seconds: 3.97\n",
      "Epoch  13/100 Batch  440/781 - Loss:  1.166, Seconds: 4.61\n",
      "Epoch  13/100 Batch  460/781 - Loss:  1.235, Seconds: 4.59\n",
      "Epoch  13/100 Batch  480/781 - Loss:  1.106, Seconds: 4.33\n",
      "Epoch  13/100 Batch  500/781 - Loss:  1.142, Seconds: 4.33\n",
      "Average loss for this update: 1.137\n",
      "New Record!\n",
      "Epoch  13/100 Batch  520/781 - Loss:  1.041, Seconds: 3.83\n",
      "Epoch  13/100 Batch  540/781 - Loss:  1.099, Seconds: 4.39\n",
      "Epoch  13/100 Batch  560/781 - Loss:  1.055, Seconds: 3.87\n",
      "Epoch  13/100 Batch  580/781 - Loss:  1.182, Seconds: 4.33\n",
      "Epoch  13/100 Batch  600/781 - Loss:  1.233, Seconds: 4.35\n",
      "Epoch  13/100 Batch  620/781 - Loss:  1.164, Seconds: 4.01\n",
      "Epoch  13/100 Batch  640/781 - Loss:  1.122, Seconds: 4.29\n",
      "Epoch  13/100 Batch  660/781 - Loss:  0.998, Seconds: 4.57\n",
      "Epoch  13/100 Batch  680/781 - Loss:  1.072, Seconds: 4.31\n",
      "Epoch  13/100 Batch  700/781 - Loss:  1.151, Seconds: 4.43\n",
      "Epoch  13/100 Batch  720/781 - Loss:  1.302, Seconds: 4.71\n",
      "Epoch  13/100 Batch  740/781 - Loss:  1.185, Seconds: 4.77\n",
      "Epoch  13/100 Batch  760/781 - Loss:  1.195, Seconds: 4.29\n",
      "Average loss for this update: 1.139\n",
      "No Improvement.\n",
      "Epoch  13/100 Batch  780/781 - Loss:  1.048, Seconds: 3.97\n",
      "Epoch  14/100 Batch   20/781 - Loss:  1.255, Seconds: 4.41\n",
      "Epoch  14/100 Batch   40/781 - Loss:  1.164, Seconds: 3.59\n",
      "Epoch  14/100 Batch   60/781 - Loss:  1.130, Seconds: 4.61\n",
      "Epoch  14/100 Batch   80/781 - Loss:  1.142, Seconds: 4.33\n",
      "Epoch  14/100 Batch  100/781 - Loss:  1.009, Seconds: 4.31\n",
      "Epoch  14/100 Batch  120/781 - Loss:  1.100, Seconds: 4.03\n",
      "Epoch  14/100 Batch  140/781 - Loss:  1.025, Seconds: 4.05\n",
      "Epoch  14/100 Batch  160/781 - Loss:  1.202, Seconds: 3.21\n",
      "Epoch  14/100 Batch  180/781 - Loss:  1.192, Seconds: 3.63\n",
      "Epoch  14/100 Batch  200/781 - Loss:  1.128, Seconds: 4.41\n",
      "Epoch  14/100 Batch  220/781 - Loss:  1.109, Seconds: 4.23\n",
      "Epoch  14/100 Batch  240/781 - Loss:  0.962, Seconds: 4.15\n",
      "Average loss for this update: 1.113\n",
      "New Record!\n",
      "Epoch  14/100 Batch  260/781 - Loss:  1.041, Seconds: 4.73\n",
      "Epoch  14/100 Batch  280/781 - Loss:  1.082, Seconds: 4.46\n",
      "Epoch  14/100 Batch  300/781 - Loss:  1.156, Seconds: 4.21\n",
      "Epoch  14/100 Batch  320/781 - Loss:  1.214, Seconds: 4.25\n",
      "Epoch  14/100 Batch  340/781 - Loss:  1.083, Seconds: 4.58\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  14/100 Batch  360/781 - Loss:  1.113, Seconds: 4.67\n",
      "Epoch  14/100 Batch  380/781 - Loss:  0.979, Seconds: 4.51\n",
      "Epoch  14/100 Batch  400/781 - Loss:  1.119, Seconds: 4.61\n",
      "Epoch  14/100 Batch  420/781 - Loss:  1.048, Seconds: 3.91\n",
      "Epoch  14/100 Batch  440/781 - Loss:  1.154, Seconds: 4.63\n",
      "Epoch  14/100 Batch  460/781 - Loss:  1.194, Seconds: 4.39\n",
      "Epoch  14/100 Batch  480/781 - Loss:  1.065, Seconds: 4.07\n",
      "Epoch  14/100 Batch  500/781 - Loss:  1.117, Seconds: 4.23\n",
      "Average loss for this update: 1.101\n",
      "New Record!\n",
      "Epoch  14/100 Batch  520/781 - Loss:  1.005, Seconds: 3.85\n",
      "Epoch  14/100 Batch  540/781 - Loss:  1.053, Seconds: 4.37\n",
      "Epoch  14/100 Batch  560/781 - Loss:  1.018, Seconds: 3.95\n",
      "Epoch  14/100 Batch  580/781 - Loss:  1.157, Seconds: 4.49\n",
      "Epoch  14/100 Batch  600/781 - Loss:  1.189, Seconds: 4.21\n",
      "Epoch  14/100 Batch  620/781 - Loss:  1.123, Seconds: 4.21\n",
      "Epoch  14/100 Batch  640/781 - Loss:  1.087, Seconds: 4.49\n",
      "Epoch  14/100 Batch  660/781 - Loss:  0.954, Seconds: 4.53\n",
      "Epoch  14/100 Batch  680/781 - Loss:  1.025, Seconds: 4.35\n",
      "Epoch  14/100 Batch  700/781 - Loss:  1.118, Seconds: 4.49\n",
      "Epoch  14/100 Batch  720/781 - Loss:  1.248, Seconds: 4.71\n",
      "Epoch  14/100 Batch  740/781 - Loss:  1.166, Seconds: 4.79\n",
      "Epoch  14/100 Batch  760/781 - Loss:  1.166, Seconds: 4.27\n",
      "Average loss for this update: 1.102\n",
      "No Improvement.\n",
      "Epoch  14/100 Batch  780/781 - Loss:  1.011, Seconds: 3.85\n",
      "Epoch  15/100 Batch   20/781 - Loss:  1.214, Seconds: 4.19\n",
      "Epoch  15/100 Batch   40/781 - Loss:  1.116, Seconds: 3.43\n",
      "Epoch  15/100 Batch   60/781 - Loss:  1.103, Seconds: 4.61\n",
      "Epoch  15/100 Batch   80/781 - Loss:  1.109, Seconds: 4.43\n",
      "Epoch  15/100 Batch  100/781 - Loss:  0.976, Seconds: 4.29\n",
      "Epoch  15/100 Batch  120/781 - Loss:  1.064, Seconds: 3.89\n",
      "Epoch  15/100 Batch  140/781 - Loss:  1.001, Seconds: 3.99\n",
      "Epoch  15/100 Batch  160/781 - Loss:  1.159, Seconds: 3.27\n",
      "Epoch  15/100 Batch  180/781 - Loss:  1.123, Seconds: 3.79\n",
      "Epoch  15/100 Batch  200/781 - Loss:  1.082, Seconds: 4.39\n",
      "Epoch  15/100 Batch  220/781 - Loss:  1.071, Seconds: 4.23\n",
      "Epoch  15/100 Batch  240/781 - Loss:  0.923, Seconds: 4.15\n",
      "Average loss for this update: 1.074\n",
      "New Record!\n",
      "Epoch  15/100 Batch  260/781 - Loss:  1.008, Seconds: 4.67\n",
      "Epoch  15/100 Batch  280/781 - Loss:  1.035, Seconds: 4.19\n",
      "Epoch  15/100 Batch  300/781 - Loss:  1.130, Seconds: 4.13\n",
      "Epoch  15/100 Batch  320/781 - Loss:  1.169, Seconds: 4.25\n",
      "Epoch  15/100 Batch  340/781 - Loss:  1.041, Seconds: 4.71\n",
      "Epoch  15/100 Batch  360/781 - Loss:  1.092, Seconds: 4.57\n",
      "Epoch  15/100 Batch  380/781 - Loss:  0.944, Seconds: 4.49\n",
      "Epoch  15/100 Batch  400/781 - Loss:  1.076, Seconds: 4.73\n",
      "Epoch  15/100 Batch  420/781 - Loss:  1.013, Seconds: 3.97\n",
      "Epoch  15/100 Batch  440/781 - Loss:  1.091, Seconds: 4.75\n",
      "Epoch  15/100 Batch  460/781 - Loss:  1.146, Seconds: 4.49\n",
      "Epoch  15/100 Batch  480/781 - Loss:  1.036, Seconds: 4.03\n",
      "Epoch  15/100 Batch  500/781 - Loss:  1.069, Seconds: 4.27\n",
      "Average loss for this update: 1.062\n",
      "New Record!\n",
      "Epoch  15/100 Batch  520/781 - Loss:  0.970, Seconds: 4.00\n",
      "Epoch  15/100 Batch  540/781 - Loss:  1.020, Seconds: 4.40\n",
      "Epoch  15/100 Batch  560/781 - Loss:  0.996, Seconds: 3.93\n",
      "Epoch  15/100 Batch  580/781 - Loss:  1.105, Seconds: 4.41\n",
      "Epoch  15/100 Batch  600/781 - Loss:  1.139, Seconds: 4.35\n",
      "Epoch  15/100 Batch  620/781 - Loss:  1.069, Seconds: 4.37\n",
      "Epoch  15/100 Batch  640/781 - Loss:  1.056, Seconds: 4.35\n",
      "Epoch  15/100 Batch  660/781 - Loss:  0.931, Seconds: 4.43\n",
      "Epoch  15/100 Batch  680/781 - Loss:  0.996, Seconds: 4.57\n",
      "Epoch  15/100 Batch  700/781 - Loss:  1.086, Seconds: 4.29\n",
      "Epoch  15/100 Batch  720/781 - Loss:  1.216, Seconds: 4.43\n",
      "Epoch  15/100 Batch  740/781 - Loss:  1.116, Seconds: 4.77\n",
      "Epoch  15/100 Batch  760/781 - Loss:  1.131, Seconds: 4.51\n",
      "Average loss for this update: 1.065\n",
      "No Improvement.\n",
      "Epoch  15/100 Batch  780/781 - Loss:  0.985, Seconds: 3.83\n",
      "Epoch  16/100 Batch   20/781 - Loss:  1.174, Seconds: 4.33\n",
      "Epoch  16/100 Batch   40/781 - Loss:  1.077, Seconds: 3.51\n",
      "Epoch  16/100 Batch   60/781 - Loss:  1.082, Seconds: 4.41\n",
      "Epoch  16/100 Batch   80/781 - Loss:  1.067, Seconds: 4.49\n",
      "Epoch  16/100 Batch  100/781 - Loss:  0.946, Seconds: 4.49\n",
      "Epoch  16/100 Batch  120/781 - Loss:  1.019, Seconds: 3.83\n",
      "Epoch  16/100 Batch  140/781 - Loss:  0.968, Seconds: 4.01\n",
      "Epoch  16/100 Batch  160/781 - Loss:  1.127, Seconds: 3.33\n",
      "Epoch  16/100 Batch  180/781 - Loss:  1.093, Seconds: 3.91\n",
      "Epoch  16/100 Batch  200/781 - Loss:  1.045, Seconds: 4.43\n",
      "Epoch  16/100 Batch  220/781 - Loss:  1.044, Seconds: 4.09\n",
      "Epoch  16/100 Batch  240/781 - Loss:  0.892, Seconds: 4.35\n",
      "Average loss for this update: 1.04\n",
      "New Record!\n",
      "Epoch  16/100 Batch  260/781 - Loss:  0.973, Seconds: 4.65\n",
      "Epoch  16/100 Batch  280/781 - Loss:  1.016, Seconds: 4.39\n",
      "Epoch  16/100 Batch  300/781 - Loss:  1.095, Seconds: 4.27\n",
      "Epoch  16/100 Batch  320/781 - Loss:  1.139, Seconds: 4.05\n",
      "Epoch  16/100 Batch  340/781 - Loss:  1.002, Seconds: 4.85\n",
      "Epoch  16/100 Batch  360/781 - Loss:  1.058, Seconds: 4.49\n",
      "Epoch  16/100 Batch  380/781 - Loss:  0.909, Seconds: 4.65\n",
      "Epoch  16/100 Batch  400/781 - Loss:  1.042, Seconds: 4.63\n",
      "Epoch  16/100 Batch  420/781 - Loss:  0.985, Seconds: 4.01\n",
      "Epoch  16/100 Batch  440/781 - Loss:  1.053, Seconds: 4.59\n",
      "Epoch  16/100 Batch  460/781 - Loss:  1.115, Seconds: 4.41\n",
      "Epoch  16/100 Batch  480/781 - Loss:  1.001, Seconds: 4.27\n",
      "Epoch  16/100 Batch  500/781 - Loss:  1.052, Seconds: 4.31\n",
      "Average loss for this update: 1.031\n",
      "New Record!\n",
      "Epoch  16/100 Batch  520/781 - Loss:  0.940, Seconds: 3.91\n",
      "Epoch  16/100 Batch  540/781 - Loss:  1.000, Seconds: 4.61\n",
      "Epoch  16/100 Batch  560/781 - Loss:  0.961, Seconds: 4.21\n",
      "Epoch  16/100 Batch  580/781 - Loss:  1.082, Seconds: 4.31\n",
      "Epoch  16/100 Batch  600/781 - Loss:  1.110, Seconds: 4.33\n",
      "Epoch  16/100 Batch  620/781 - Loss:  1.035, Seconds: 4.13\n",
      "Epoch  16/100 Batch  640/781 - Loss:  1.026, Seconds: 4.25\n",
      "Epoch  16/100 Batch  660/781 - Loss:  0.902, Seconds: 4.53\n",
      "Epoch  16/100 Batch  680/781 - Loss:  0.970, Seconds: 4.41\n",
      "Epoch  16/100 Batch  700/781 - Loss:  1.062, Seconds: 4.60\n",
      "Epoch  16/100 Batch  720/781 - Loss:  1.192, Seconds: 4.63\n",
      "Epoch  16/100 Batch  740/781 - Loss:  1.094, Seconds: 4.85\n",
      "Epoch  16/100 Batch  760/781 - Loss:  1.111, Seconds: 4.49\n",
      "Average loss for this update: 1.04\n",
      "No Improvement.\n",
      "Epoch  16/100 Batch  780/781 - Loss:  0.965, Seconds: 3.83\n",
      "Epoch  17/100 Batch   20/781 - Loss:  1.147, Seconds: 4.21\n",
      "Epoch  17/100 Batch   40/781 - Loss:  1.055, Seconds: 3.43\n",
      "Epoch  17/100 Batch   60/781 - Loss:  1.053, Seconds: 4.41\n",
      "Epoch  17/100 Batch   80/781 - Loss:  1.040, Seconds: 4.51\n",
      "Epoch  17/100 Batch  100/781 - Loss:  0.918, Seconds: 4.23\n",
      "Epoch  17/100 Batch  120/781 - Loss:  0.988, Seconds: 3.89\n",
      "Epoch  17/100 Batch  140/781 - Loss:  0.944, Seconds: 4.09\n",
      "Epoch  17/100 Batch  160/781 - Loss:  1.078, Seconds: 3.36\n",
      "Epoch  17/100 Batch  180/781 - Loss:  1.053, Seconds: 3.69\n",
      "Epoch  17/100 Batch  200/781 - Loss:  1.022, Seconds: 4.40\n",
      "Epoch  17/100 Batch  220/781 - Loss:  1.007, Seconds: 4.20\n",
      "Epoch  17/100 Batch  240/781 - Loss:  0.854, Seconds: 4.22\n",
      "Average loss for this update: 1.009\n",
      "New Record!\n",
      "Epoch  17/100 Batch  260/781 - Loss:  0.950, Seconds: 4.57\n",
      "Epoch  17/100 Batch  280/781 - Loss:  0.971, Seconds: 4.33\n",
      "Epoch  17/100 Batch  300/781 - Loss:  1.070, Seconds: 4.21\n",
      "Epoch  17/100 Batch  320/781 - Loss:  1.125, Seconds: 4.45\n",
      "Epoch  17/100 Batch  340/781 - Loss:  0.983, Seconds: 4.49\n",
      "Epoch  17/100 Batch  360/781 - Loss:  1.026, Seconds: 4.33\n",
      "Epoch  17/100 Batch  380/781 - Loss:  0.881, Seconds: 4.97\n",
      "Epoch  17/100 Batch  400/781 - Loss:  1.019, Seconds: 4.67\n",
      "Epoch  17/100 Batch  420/781 - Loss:  0.962, Seconds: 4.03\n",
      "Epoch  17/100 Batch  440/781 - Loss:  1.011, Seconds: 4.61\n",
      "Epoch  17/100 Batch  460/781 - Loss:  1.076, Seconds: 4.45\n",
      "Epoch  17/100 Batch  480/781 - Loss:  0.975, Seconds: 4.05\n",
      "Epoch  17/100 Batch  500/781 - Loss:  1.024, Seconds: 4.23\n",
      "Average loss for this update: 1.002\n",
      "New Record!\n",
      "Epoch  17/100 Batch  520/781 - Loss:  0.915, Seconds: 4.03\n",
      "Epoch  17/100 Batch  540/781 - Loss:  0.963, Seconds: 4.41\n",
      "Epoch  17/100 Batch  560/781 - Loss:  0.913, Seconds: 4.09\n",
      "Epoch  17/100 Batch  580/781 - Loss:  1.042, Seconds: 4.29\n",
      "Epoch  17/100 Batch  600/781 - Loss:  1.078, Seconds: 4.29\n",
      "Epoch  17/100 Batch  620/781 - Loss:  1.010, Seconds: 4.01\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  17/100 Batch  640/781 - Loss:  1.001, Seconds: 4.23\n",
      "Epoch  17/100 Batch  660/781 - Loss:  0.882, Seconds: 4.47\n",
      "Epoch  17/100 Batch  680/781 - Loss:  0.947, Seconds: 4.37\n",
      "Epoch  17/100 Batch  700/781 - Loss:  1.028, Seconds: 4.31\n",
      "Epoch  17/100 Batch  720/781 - Loss:  1.147, Seconds: 4.53\n",
      "Epoch  17/100 Batch  740/781 - Loss:  1.047, Seconds: 4.77\n",
      "Epoch  17/100 Batch  760/781 - Loss:  1.062, Seconds: 4.31\n",
      "Average loss for this update: 1.003\n",
      "No Improvement.\n",
      "Epoch  17/100 Batch  780/781 - Loss:  0.922, Seconds: 3.91\n",
      "Epoch  18/100 Batch   20/781 - Loss:  1.115, Seconds: 4.41\n",
      "Epoch  18/100 Batch   40/781 - Loss:  1.022, Seconds: 3.31\n",
      "Epoch  18/100 Batch   60/781 - Loss:  1.016, Seconds: 4.41\n",
      "Epoch  18/100 Batch   80/781 - Loss:  1.003, Seconds: 4.19\n",
      "Epoch  18/100 Batch  100/781 - Loss:  0.893, Seconds: 4.23\n",
      "Epoch  18/100 Batch  120/781 - Loss:  0.970, Seconds: 3.77\n",
      "Epoch  18/100 Batch  140/781 - Loss:  0.917, Seconds: 4.09\n",
      "Epoch  18/100 Batch  160/781 - Loss:  1.062, Seconds: 3.33\n",
      "Epoch  18/100 Batch  180/781 - Loss:  1.011, Seconds: 3.73\n",
      "Epoch  18/100 Batch  200/781 - Loss:  0.982, Seconds: 4.37\n",
      "Epoch  18/100 Batch  220/781 - Loss:  0.982, Seconds: 4.03\n",
      "Epoch  18/100 Batch  240/781 - Loss:  0.839, Seconds: 4.09\n",
      "Average loss for this update: 0.98\n",
      "New Record!\n",
      "Epoch  18/100 Batch  260/781 - Loss:  0.918, Seconds: 4.45\n",
      "Epoch  18/100 Batch  280/781 - Loss:  0.930, Seconds: 4.29\n",
      "Epoch  18/100 Batch  300/781 - Loss:  1.033, Seconds: 4.07\n",
      "Epoch  18/100 Batch  320/781 - Loss:  1.063, Seconds: 4.23\n",
      "Epoch  18/100 Batch  340/781 - Loss:  0.951, Seconds: 4.59\n",
      "Epoch  18/100 Batch  360/781 - Loss:  1.000, Seconds: 4.23\n",
      "Epoch  18/100 Batch  380/781 - Loss:  0.851, Seconds: 4.55\n",
      "Epoch  18/100 Batch  400/781 - Loss:  0.980, Seconds: 4.45\n",
      "Epoch  18/100 Batch  420/781 - Loss:  0.916, Seconds: 3.95\n",
      "Epoch  18/100 Batch  440/781 - Loss:  0.991, Seconds: 4.53\n",
      "Epoch  18/100 Batch  460/781 - Loss:  1.049, Seconds: 4.35\n",
      "Epoch  18/100 Batch  480/781 - Loss:  0.943, Seconds: 3.99\n",
      "Epoch  18/100 Batch  500/781 - Loss:  0.993, Seconds: 4.13\n",
      "Average loss for this update: 0.968\n",
      "New Record!\n",
      "Epoch  18/100 Batch  520/781 - Loss:  0.890, Seconds: 3.87\n",
      "Epoch  18/100 Batch  540/781 - Loss:  0.939, Seconds: 4.31\n",
      "Epoch  18/100 Batch  560/781 - Loss:  0.906, Seconds: 3.95\n",
      "Epoch  18/100 Batch  580/781 - Loss:  1.027, Seconds: 4.31\n",
      "Epoch  18/100 Batch  600/781 - Loss:  1.034, Seconds: 4.33\n",
      "Epoch  18/100 Batch  620/781 - Loss:  0.987, Seconds: 4.03\n",
      "Epoch  18/100 Batch  640/781 - Loss:  0.987, Seconds: 4.27\n",
      "Epoch  18/100 Batch  660/781 - Loss:  0.855, Seconds: 4.53\n",
      "Epoch  18/100 Batch  680/781 - Loss:  0.926, Seconds: 4.25\n",
      "Epoch  18/100 Batch  700/781 - Loss:  1.014, Seconds: 4.33\n",
      "Epoch  18/100 Batch  720/781 - Loss:  1.130, Seconds: 4.47\n",
      "Epoch  18/100 Batch  740/781 - Loss:  1.033, Seconds: 4.69\n",
      "Epoch  18/100 Batch  760/781 - Loss:  1.041, Seconds: 4.29\n",
      "Average loss for this update: 0.984\n",
      "No Improvement.\n",
      "Epoch  18/100 Batch  780/781 - Loss:  0.910, Seconds: 3.87\n",
      "Epoch  19/100 Batch   20/781 - Loss:  1.078, Seconds: 4.23\n",
      "Epoch  19/100 Batch   40/781 - Loss:  1.007, Seconds: 3.37\n",
      "Epoch  19/100 Batch   60/781 - Loss:  0.987, Seconds: 4.41\n",
      "Epoch  19/100 Batch   80/781 - Loss:  0.983, Seconds: 4.21\n",
      "Epoch  19/100 Batch  100/781 - Loss:  0.880, Seconds: 4.15\n",
      "Epoch  19/100 Batch  120/781 - Loss:  0.949, Seconds: 3.79\n",
      "Epoch  19/100 Batch  140/781 - Loss:  0.886, Seconds: 3.99\n",
      "Epoch  19/100 Batch  160/781 - Loss:  1.047, Seconds: 3.33\n",
      "Epoch  19/100 Batch  180/781 - Loss:  1.013, Seconds: 3.65\n",
      "Epoch  19/100 Batch  200/781 - Loss:  0.972, Seconds: 4.27\n",
      "Epoch  19/100 Batch  220/781 - Loss:  0.967, Seconds: 4.31\n",
      "Epoch  19/100 Batch  240/781 - Loss:  0.824, Seconds: 4.12\n",
      "Average loss for this update: 0.962\n",
      "New Record!\n",
      "Epoch  19/100 Batch  260/781 - Loss:  0.912, Seconds: 4.67\n",
      "Epoch  19/100 Batch  280/781 - Loss:  0.927, Seconds: 4.35\n",
      "Epoch  19/100 Batch  300/781 - Loss:  1.008, Seconds: 4.09\n",
      "Epoch  19/100 Batch  320/781 - Loss:  1.041, Seconds: 4.11\n",
      "Epoch  19/100 Batch  340/781 - Loss:  0.929, Seconds: 4.63\n",
      "Epoch  19/100 Batch  360/781 - Loss:  1.007, Seconds: 4.51\n",
      "Epoch  19/100 Batch  380/781 - Loss:  0.860, Seconds: 4.61\n",
      "Epoch  19/100 Batch  400/781 - Loss:  0.965, Seconds: 4.67\n",
      "Epoch  19/100 Batch  420/781 - Loss:  0.903, Seconds: 3.85\n",
      "Epoch  19/100 Batch  440/781 - Loss:  0.976, Seconds: 4.73\n",
      "Epoch  19/100 Batch  460/781 - Loss:  1.033, Seconds: 4.45\n",
      "Epoch  19/100 Batch  480/781 - Loss:  0.912, Seconds: 4.11\n",
      "Epoch  19/100 Batch  500/781 - Loss:  0.976, Seconds: 4.31\n",
      "Average loss for this update: 0.954\n",
      "New Record!\n",
      "Epoch  19/100 Batch  520/781 - Loss:  0.868, Seconds: 4.03\n",
      "Epoch  19/100 Batch  540/781 - Loss:  0.922, Seconds: 4.59\n",
      "Epoch  19/100 Batch  560/781 - Loss:  0.893, Seconds: 4.07\n",
      "Epoch  19/100 Batch  580/781 - Loss:  1.004, Seconds: 4.57\n",
      "Epoch  19/100 Batch  600/781 - Loss:  1.024, Seconds: 4.43\n",
      "Epoch  19/100 Batch  620/781 - Loss:  0.947, Seconds: 4.23\n",
      "Epoch  19/100 Batch  640/781 - Loss:  0.949, Seconds: 4.55\n",
      "Epoch  19/100 Batch  660/781 - Loss:  0.840, Seconds: 4.59\n",
      "Epoch  19/100 Batch  680/781 - Loss:  0.917, Seconds: 4.31\n",
      "Epoch  19/100 Batch  700/781 - Loss:  0.994, Seconds: 4.47\n",
      "Epoch  19/100 Batch  720/781 - Loss:  1.095, Seconds: 4.61\n",
      "Epoch  19/100 Batch  740/781 - Loss:  0.991, Seconds: 4.73\n",
      "Epoch  19/100 Batch  760/781 - Loss:  1.018, Seconds: 4.45\n",
      "Average loss for this update: 0.96\n",
      "No Improvement.\n",
      "Epoch  19/100 Batch  780/781 - Loss:  0.882, Seconds: 4.09\n",
      "Epoch  20/100 Batch   20/781 - Loss:  1.101, Seconds: 4.27\n",
      "Epoch  20/100 Batch   40/781 - Loss:  0.987, Seconds: 3.55\n",
      "Epoch  20/100 Batch   60/781 - Loss:  0.966, Seconds: 4.51\n",
      "Epoch  20/100 Batch   80/781 - Loss:  0.978, Seconds: 4.27\n",
      "Epoch  20/100 Batch  100/781 - Loss:  0.863, Seconds: 4.31\n",
      "Epoch  20/100 Batch  120/781 - Loss:  0.927, Seconds: 3.91\n",
      "Epoch  20/100 Batch  140/781 - Loss:  0.872, Seconds: 4.03\n",
      "Epoch  20/100 Batch  160/781 - Loss:  1.023, Seconds: 3.25\n",
      "Epoch  20/100 Batch  180/781 - Loss:  0.974, Seconds: 3.89\n",
      "Epoch  20/100 Batch  200/781 - Loss:  0.935, Seconds: 4.39\n",
      "Epoch  20/100 Batch  220/781 - Loss:  0.957, Seconds: 4.11\n",
      "Epoch  20/100 Batch  240/781 - Loss:  0.805, Seconds: 4.15\n",
      "Average loss for this update: 0.946\n",
      "New Record!\n",
      "Epoch  20/100 Batch  260/781 - Loss:  0.898, Seconds: 4.69\n",
      "Epoch  20/100 Batch  280/781 - Loss:  0.909, Seconds: 4.50\n",
      "Epoch  20/100 Batch  300/781 - Loss:  0.989, Seconds: 4.23\n",
      "Epoch  20/100 Batch  320/781 - Loss:  1.021, Seconds: 4.23\n",
      "Epoch  20/100 Batch  340/781 - Loss:  0.911, Seconds: 4.71\n",
      "Epoch  20/100 Batch  360/781 - Loss:  0.960, Seconds: 4.53\n",
      "Epoch  20/100 Batch  380/781 - Loss:  0.824, Seconds: 4.61\n",
      "Epoch  20/100 Batch  400/781 - Loss:  0.934, Seconds: 4.73\n",
      "Epoch  20/100 Batch  420/781 - Loss:  0.874, Seconds: 4.03\n",
      "Epoch  20/100 Batch  440/781 - Loss:  0.921, Seconds: 4.77\n",
      "Epoch  20/100 Batch  460/781 - Loss:  0.990, Seconds: 4.51\n",
      "Epoch  20/100 Batch  480/781 - Loss:  0.893, Seconds: 4.03\n",
      "Epoch  20/100 Batch  500/781 - Loss:  0.938, Seconds: 4.17\n",
      "Average loss for this update: 0.923\n",
      "New Record!\n",
      "Epoch  20/100 Batch  520/781 - Loss:  0.842, Seconds: 4.01\n",
      "Epoch  20/100 Batch  540/781 - Loss:  0.898, Seconds: 4.63\n",
      "Epoch  20/100 Batch  560/781 - Loss:  0.860, Seconds: 4.07\n",
      "Epoch  20/100 Batch  580/781 - Loss:  0.974, Seconds: 4.37\n",
      "Epoch  20/100 Batch  600/781 - Loss:  0.980, Seconds: 4.47\n",
      "Epoch  20/100 Batch  620/781 - Loss:  0.929, Seconds: 4.17\n",
      "Epoch  20/100 Batch  640/781 - Loss:  0.922, Seconds: 4.29\n",
      "Epoch  20/100 Batch  660/781 - Loss:  0.827, Seconds: 4.47\n",
      "Epoch  20/100 Batch  680/781 - Loss:  0.882, Seconds: 4.27\n",
      "Epoch  20/100 Batch  700/781 - Loss:  0.960, Seconds: 4.31\n",
      "Epoch  20/100 Batch  720/781 - Loss:  1.073, Seconds: 4.61\n",
      "Epoch  20/100 Batch  740/781 - Loss:  0.974, Seconds: 4.65\n",
      "Epoch  20/100 Batch  760/781 - Loss:  0.989, Seconds: 4.23\n",
      "Average loss for this update: 0.933\n",
      "No Improvement.\n",
      "Epoch  20/100 Batch  780/781 - Loss:  0.858, Seconds: 3.91\n",
      "Epoch  21/100 Batch   20/781 - Loss:  1.041, Seconds: 4.27\n",
      "Epoch  21/100 Batch   40/781 - Loss:  0.954, Seconds: 3.39\n",
      "Epoch  21/100 Batch   60/781 - Loss:  0.946, Seconds: 4.37\n",
      "Epoch  21/100 Batch   80/781 - Loss:  0.942, Seconds: 4.33\n",
      "Epoch  21/100 Batch  100/781 - Loss:  0.827, Seconds: 4.53\n",
      "Epoch  21/100 Batch  120/781 - Loss:  0.897, Seconds: 3.81\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  21/100 Batch  140/781 - Loss:  0.851, Seconds: 4.17\n",
      "Epoch  21/100 Batch  160/781 - Loss:  0.983, Seconds: 3.27\n",
      "Epoch  21/100 Batch  180/781 - Loss:  0.949, Seconds: 3.59\n",
      "Epoch  21/100 Batch  200/781 - Loss:  0.904, Seconds: 4.35\n",
      "Epoch  21/100 Batch  220/781 - Loss:  0.918, Seconds: 4.11\n",
      "Epoch  21/100 Batch  240/781 - Loss:  0.773, Seconds: 4.01\n",
      "Average loss for this update: 0.913\n",
      "New Record!\n",
      "Epoch  21/100 Batch  260/781 - Loss:  0.874, Seconds: 4.55\n",
      "Epoch  21/100 Batch  280/781 - Loss:  0.875, Seconds: 4.27\n",
      "Epoch  21/100 Batch  300/781 - Loss:  0.943, Seconds: 4.15\n",
      "Epoch  21/100 Batch  320/781 - Loss:  0.996, Seconds: 4.25\n",
      "Epoch  21/100 Batch  340/781 - Loss:  0.879, Seconds: 4.65\n",
      "Epoch  21/100 Batch  360/781 - Loss:  0.925, Seconds: 4.41\n",
      "Epoch  21/100 Batch  380/781 - Loss:  0.787, Seconds: 4.57\n",
      "Epoch  21/100 Batch  400/781 - Loss:  0.915, Seconds: 4.57\n",
      "Epoch  21/100 Batch  420/781 - Loss:  0.850, Seconds: 4.07\n",
      "Epoch  21/100 Batch  440/781 - Loss:  0.908, Seconds: 4.73\n",
      "Epoch  21/100 Batch  460/781 - Loss:  0.966, Seconds: 4.61\n",
      "Epoch  21/100 Batch  480/781 - Loss:  0.866, Seconds: 3.97\n",
      "Epoch  21/100 Batch  500/781 - Loss:  0.915, Seconds: 4.51\n",
      "Average loss for this update: 0.894\n",
      "New Record!\n",
      "Epoch  21/100 Batch  520/781 - Loss:  0.812, Seconds: 3.93\n",
      "Epoch  21/100 Batch  540/781 - Loss:  0.880, Seconds: 4.59\n",
      "Epoch  21/100 Batch  560/781 - Loss:  0.843, Seconds: 4.11\n",
      "Epoch  21/100 Batch  580/781 - Loss:  0.945, Seconds: 4.31\n",
      "Epoch  21/100 Batch  600/781 - Loss:  0.969, Seconds: 4.56\n",
      "Epoch  21/100 Batch  620/781 - Loss:  0.906, Seconds: 4.19\n",
      "Epoch  21/100 Batch  640/781 - Loss:  0.896, Seconds: 4.25\n",
      "Epoch  21/100 Batch  660/781 - Loss:  0.793, Seconds: 4.51\n",
      "Epoch  21/100 Batch  680/781 - Loss:  0.849, Seconds: 4.21\n",
      "Epoch  21/100 Batch  700/781 - Loss:  0.944, Seconds: 4.25\n",
      "Epoch  21/100 Batch  720/781 - Loss:  1.048, Seconds: 4.51\n",
      "Epoch  21/100 Batch  740/781 - Loss:  0.944, Seconds: 4.71\n",
      "Epoch  21/100 Batch  760/781 - Loss:  0.972, Seconds: 4.45\n",
      "Average loss for this update: 0.911\n",
      "No Improvement.\n",
      "Epoch  21/100 Batch  780/781 - Loss:  0.850, Seconds: 4.11\n",
      "Epoch  22/100 Batch   20/781 - Loss:  1.015, Seconds: 4.25\n",
      "Epoch  22/100 Batch   40/781 - Loss:  0.936, Seconds: 3.51\n",
      "Epoch  22/100 Batch   60/781 - Loss:  0.925, Seconds: 4.27\n",
      "Epoch  22/100 Batch   80/781 - Loss:  0.913, Seconds: 4.15\n",
      "Epoch  22/100 Batch  100/781 - Loss:  0.818, Seconds: 4.25\n",
      "Epoch  22/100 Batch  120/781 - Loss:  0.871, Seconds: 3.81\n",
      "Epoch  22/100 Batch  140/781 - Loss:  0.843, Seconds: 4.05\n",
      "Epoch  22/100 Batch  160/781 - Loss:  0.968, Seconds: 3.33\n",
      "Epoch  22/100 Batch  180/781 - Loss:  0.926, Seconds: 3.69\n",
      "Epoch  22/100 Batch  200/781 - Loss:  0.896, Seconds: 4.29\n",
      "Epoch  22/100 Batch  220/781 - Loss:  0.891, Seconds: 4.09\n",
      "Epoch  22/100 Batch  240/781 - Loss:  0.756, Seconds: 4.11\n",
      "Average loss for this update: 0.893\n",
      "New Record!\n",
      "Epoch  22/100 Batch  260/781 - Loss:  0.849, Seconds: 4.61\n",
      "Epoch  22/100 Batch  280/781 - Loss:  0.846, Seconds: 4.27\n",
      "Epoch  22/100 Batch  300/781 - Loss:  0.935, Seconds: 4.09\n",
      "Epoch  22/100 Batch  320/781 - Loss:  0.980, Seconds: 4.25\n",
      "Epoch  22/100 Batch  340/781 - Loss:  0.857, Seconds: 4.55\n",
      "Epoch  22/100 Batch  360/781 - Loss:  0.914, Seconds: 4.67\n",
      "Epoch  22/100 Batch  380/781 - Loss:  0.786, Seconds: 4.63\n",
      "Epoch  22/100 Batch  400/781 - Loss:  0.895, Seconds: 4.61\n",
      "Epoch  22/100 Batch  420/781 - Loss:  0.837, Seconds: 3.91\n",
      "Epoch  22/100 Batch  440/781 - Loss:  0.901, Seconds: 4.63\n",
      "Epoch  22/100 Batch  460/781 - Loss:  0.944, Seconds: 4.39\n",
      "Epoch  22/100 Batch  480/781 - Loss:  0.842, Seconds: 3.97\n",
      "Epoch  22/100 Batch  500/781 - Loss:  0.908, Seconds: 4.15\n",
      "Average loss for this update: 0.88\n",
      "New Record!\n",
      "Epoch  22/100 Batch  520/781 - Loss:  0.796, Seconds: 3.85\n",
      "Epoch  22/100 Batch  540/781 - Loss:  0.856, Seconds: 4.37\n",
      "Epoch  22/100 Batch  560/781 - Loss:  0.817, Seconds: 4.01\n",
      "Epoch  22/100 Batch  580/781 - Loss:  0.931, Seconds: 4.31\n",
      "Epoch  22/100 Batch  600/781 - Loss:  0.962, Seconds: 4.27\n",
      "Epoch  22/100 Batch  620/781 - Loss:  0.903, Seconds: 4.03\n",
      "Epoch  22/100 Batch  640/781 - Loss:  0.872, Seconds: 4.13\n",
      "Epoch  22/100 Batch  660/781 - Loss:  0.788, Seconds: 4.43\n",
      "Epoch  22/100 Batch  680/781 - Loss:  0.839, Seconds: 4.37\n",
      "Epoch  22/100 Batch  700/781 - Loss:  0.928, Seconds: 4.31\n",
      "Epoch  22/100 Batch  720/781 - Loss:  1.030, Seconds: 4.49\n",
      "Epoch  22/100 Batch  740/781 - Loss:  0.934, Seconds: 4.85\n",
      "Epoch  22/100 Batch  760/781 - Loss:  0.955, Seconds: 4.33\n",
      "Average loss for this update: 0.895\n",
      "No Improvement.\n",
      "Epoch  22/100 Batch  780/781 - Loss:  0.830, Seconds: 3.89\n",
      "Epoch  23/100 Batch   20/781 - Loss:  0.994, Seconds: 4.33\n",
      "Epoch  23/100 Batch   40/781 - Loss:  0.901, Seconds: 3.47\n",
      "Epoch  23/100 Batch   60/781 - Loss:  0.897, Seconds: 4.35\n",
      "Epoch  23/100 Batch   80/781 - Loss:  0.893, Seconds: 4.27\n",
      "Epoch  23/100 Batch  100/781 - Loss:  0.796, Seconds: 4.31\n",
      "Epoch  23/100 Batch  120/781 - Loss:  0.861, Seconds: 3.71\n",
      "Epoch  23/100 Batch  140/781 - Loss:  0.816, Seconds: 4.05\n",
      "Epoch  23/100 Batch  160/781 - Loss:  0.952, Seconds: 3.41\n",
      "Epoch  23/100 Batch  180/781 - Loss:  0.905, Seconds: 3.65\n",
      "Epoch  23/100 Batch  200/781 - Loss:  0.885, Seconds: 4.23\n",
      "Epoch  23/100 Batch  220/781 - Loss:  0.880, Seconds: 4.15\n",
      "Epoch  23/100 Batch  240/781 - Loss:  0.738, Seconds: 4.13\n",
      "Average loss for this update: 0.874\n",
      "New Record!\n",
      "Epoch  23/100 Batch  260/781 - Loss:  0.833, Seconds: 4.49\n",
      "Epoch  23/100 Batch  280/781 - Loss:  0.838, Seconds: 4.29\n",
      "Epoch  23/100 Batch  300/781 - Loss:  0.915, Seconds: 4.25\n",
      "Epoch  23/100 Batch  320/781 - Loss:  0.953, Seconds: 4.13\n",
      "Epoch  23/100 Batch  340/781 - Loss:  0.847, Seconds: 4.49\n",
      "Epoch  23/100 Batch  360/781 - Loss:  0.889, Seconds: 4.33\n",
      "Epoch  23/100 Batch  380/781 - Loss:  0.775, Seconds: 4.55\n",
      "Epoch  23/100 Batch  400/781 - Loss:  0.904, Seconds: 4.59\n",
      "Epoch  23/100 Batch  420/781 - Loss:  0.824, Seconds: 3.95\n",
      "Epoch  23/100 Batch  440/781 - Loss:  0.883, Seconds: 4.57\n",
      "Epoch  23/100 Batch  460/781 - Loss:  0.936, Seconds: 4.47\n",
      "Epoch  23/100 Batch  480/781 - Loss:  0.862, Seconds: 3.85\n",
      "Epoch  23/100 Batch  500/781 - Loss:  0.913, Seconds: 4.29\n",
      "Average loss for this update: 0.871\n",
      "New Record!\n",
      "Epoch  23/100 Batch  520/781 - Loss:  0.793, Seconds: 3.91\n",
      "Epoch  23/100 Batch  540/781 - Loss:  0.833, Seconds: 4.41\n",
      "Epoch  23/100 Batch  560/781 - Loss:  0.812, Seconds: 3.97\n",
      "Epoch  23/100 Batch  580/781 - Loss:  0.905, Seconds: 4.25\n",
      "Epoch  23/100 Batch  600/781 - Loss:  0.934, Seconds: 4.19\n",
      "Epoch  23/100 Batch  620/781 - Loss:  0.874, Seconds: 4.07\n",
      "Epoch  23/100 Batch  640/781 - Loss:  0.867, Seconds: 4.37\n",
      "Epoch  23/100 Batch  660/781 - Loss:  0.762, Seconds: 4.41\n",
      "Epoch  23/100 Batch  680/781 - Loss:  0.821, Seconds: 4.25\n",
      "Epoch  23/100 Batch  700/781 - Loss:  0.917, Seconds: 4.37\n",
      "Epoch  23/100 Batch  720/781 - Loss:  1.014, Seconds: 4.47\n",
      "Epoch  23/100 Batch  740/781 - Loss:  0.926, Seconds: 4.61\n",
      "Epoch  23/100 Batch  760/781 - Loss:  0.963, Seconds: 4.35\n",
      "Average loss for this update: 0.881\n",
      "No Improvement.\n",
      "Epoch  23/100 Batch  780/781 - Loss:  0.827, Seconds: 3.87\n",
      "Epoch  24/100 Batch   20/781 - Loss:  0.965, Seconds: 4.19\n",
      "Epoch  24/100 Batch   40/781 - Loss:  0.886, Seconds: 3.41\n",
      "Epoch  24/100 Batch   60/781 - Loss:  0.874, Seconds: 4.35\n",
      "Epoch  24/100 Batch   80/781 - Loss:  0.885, Seconds: 4.19\n",
      "Epoch  24/100 Batch  100/781 - Loss:  0.778, Seconds: 4.18\n",
      "Epoch  24/100 Batch  120/781 - Loss:  0.851, Seconds: 3.73\n",
      "Epoch  24/100 Batch  140/781 - Loss:  0.797, Seconds: 4.05\n",
      "Epoch  24/100 Batch  160/781 - Loss:  0.932, Seconds: 3.29\n",
      "Epoch  24/100 Batch  180/781 - Loss:  0.893, Seconds: 3.63\n",
      "Epoch  24/100 Batch  200/781 - Loss:  0.867, Seconds: 4.21\n",
      "Epoch  24/100 Batch  220/781 - Loss:  0.865, Seconds: 4.05\n",
      "Epoch  24/100 Batch  240/781 - Loss:  0.719, Seconds: 4.03\n",
      "Average loss for this update: 0.856\n",
      "New Record!\n",
      "Epoch  24/100 Batch  260/781 - Loss:  0.811, Seconds: 4.65\n",
      "Epoch  24/100 Batch  280/781 - Loss:  0.830, Seconds: 4.31\n",
      "Epoch  24/100 Batch  300/781 - Loss:  0.902, Seconds: 4.23\n",
      "Epoch  24/100 Batch  320/781 - Loss:  0.935, Seconds: 4.23\n",
      "Epoch  24/100 Batch  340/781 - Loss:  0.819, Seconds: 4.49\n",
      "Epoch  24/100 Batch  360/781 - Loss:  0.871, Seconds: 4.31\n",
      "Epoch  24/100 Batch  380/781 - Loss:  0.738, Seconds: 4.67\n",
      "Epoch  24/100 Batch  400/781 - Loss:  0.857, Seconds: 4.53\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  24/100 Batch  420/781 - Loss:  0.796, Seconds: 3.85\n",
      "Epoch  24/100 Batch  440/781 - Loss:  0.865, Seconds: 4.51\n",
      "Epoch  24/100 Batch  460/781 - Loss:  0.914, Seconds: 4.45\n",
      "Epoch  24/100 Batch  480/781 - Loss:  0.823, Seconds: 3.89\n",
      "Epoch  24/100 Batch  500/781 - Loss:  0.879, Seconds: 4.27\n",
      "Average loss for this update: 0.846\n",
      "New Record!\n",
      "Epoch  24/100 Batch  520/781 - Loss:  0.766, Seconds: 3.89\n",
      "Epoch  24/100 Batch  540/781 - Loss:  0.818, Seconds: 4.47\n",
      "Epoch  24/100 Batch  560/781 - Loss:  0.784, Seconds: 4.13\n",
      "Epoch  24/100 Batch  580/781 - Loss:  0.897, Seconds: 4.19\n",
      "Epoch  24/100 Batch  600/781 - Loss:  0.902, Seconds: 4.43\n",
      "Epoch  24/100 Batch  620/781 - Loss:  0.859, Seconds: 4.15\n",
      "Epoch  24/100 Batch  640/781 - Loss:  0.847, Seconds: 4.17\n",
      "Epoch  24/100 Batch  660/781 - Loss:  0.744, Seconds: 4.47\n",
      "Epoch  24/100 Batch  680/781 - Loss:  0.788, Seconds: 4.31\n",
      "Epoch  24/100 Batch  700/781 - Loss:  0.886, Seconds: 4.31\n",
      "Epoch  24/100 Batch  720/781 - Loss:  0.987, Seconds: 4.43\n",
      "Epoch  24/100 Batch  740/781 - Loss:  0.898, Seconds: 4.79\n",
      "Epoch  24/100 Batch  760/781 - Loss:  0.924, Seconds: 4.33\n",
      "Average loss for this update: 0.857\n",
      "No Improvement.\n",
      "Epoch  24/100 Batch  780/781 - Loss:  0.808, Seconds: 3.99\n",
      "Epoch  25/100 Batch   20/781 - Loss:  0.941, Seconds: 4.21\n",
      "Epoch  25/100 Batch   40/781 - Loss:  0.873, Seconds: 3.41\n",
      "Epoch  25/100 Batch   60/781 - Loss:  0.868, Seconds: 4.49\n",
      "Epoch  25/100 Batch   80/781 - Loss:  0.858, Seconds: 4.25\n",
      "Epoch  25/100 Batch  100/781 - Loss:  0.761, Seconds: 4.25\n",
      "Epoch  25/100 Batch  120/781 - Loss:  0.809, Seconds: 3.85\n",
      "Epoch  25/100 Batch  140/781 - Loss:  0.784, Seconds: 3.99\n",
      "Epoch  25/100 Batch  160/781 - Loss:  0.925, Seconds: 3.25\n",
      "Epoch  25/100 Batch  180/781 - Loss:  0.886, Seconds: 3.77\n",
      "Epoch  25/100 Batch  200/781 - Loss:  0.860, Seconds: 4.17\n",
      "Epoch  25/100 Batch  220/781 - Loss:  0.846, Seconds: 4.03\n",
      "Epoch  25/100 Batch  240/781 - Loss:  0.709, Seconds: 4.19\n",
      "Average loss for this update: 0.84\n",
      "New Record!\n",
      "Epoch  25/100 Batch  260/781 - Loss:  0.790, Seconds: 4.61\n",
      "Epoch  25/100 Batch  280/781 - Loss:  0.822, Seconds: 4.27\n",
      "Epoch  25/100 Batch  300/781 - Loss:  0.880, Seconds: 4.09\n",
      "Epoch  25/100 Batch  320/781 - Loss:  0.922, Seconds: 4.09\n",
      "Epoch  25/100 Batch  340/781 - Loss:  0.816, Seconds: 4.59\n",
      "Epoch  25/100 Batch  360/781 - Loss:  0.845, Seconds: 4.31\n",
      "Epoch  25/100 Batch  380/781 - Loss:  0.732, Seconds: 4.51\n",
      "Epoch  25/100 Batch  400/781 - Loss:  0.851, Seconds: 4.47\n",
      "Epoch  25/100 Batch  420/781 - Loss:  0.787, Seconds: 3.91\n",
      "Epoch  25/100 Batch  440/781 - Loss:  0.848, Seconds: 4.51\n",
      "Epoch  25/100 Batch  460/781 - Loss:  0.897, Seconds: 4.39\n",
      "Epoch  25/100 Batch  480/781 - Loss:  0.808, Seconds: 3.97\n",
      "Epoch  25/100 Batch  500/781 - Loss:  0.867, Seconds: 4.23\n",
      "Average loss for this update: 0.832\n",
      "New Record!\n",
      "Epoch  25/100 Batch  520/781 - Loss:  0.746, Seconds: 3.95\n",
      "Epoch  25/100 Batch  540/781 - Loss:  0.806, Seconds: 4.33\n",
      "Epoch  25/100 Batch  560/781 - Loss:  0.777, Seconds: 4.09\n",
      "Epoch  25/100 Batch  580/781 - Loss:  0.870, Seconds: 4.15\n",
      "Epoch  25/100 Batch  600/781 - Loss:  0.891, Seconds: 4.23\n",
      "Epoch  25/100 Batch  620/781 - Loss:  0.844, Seconds: 4.15\n",
      "Epoch  25/100 Batch  640/781 - Loss:  0.840, Seconds: 4.23\n",
      "Epoch  25/100 Batch  660/781 - Loss:  0.743, Seconds: 4.45\n",
      "Epoch  25/100 Batch  680/781 - Loss:  0.783, Seconds: 4.17\n",
      "Epoch  25/100 Batch  700/781 - Loss:  0.871, Seconds: 4.39\n",
      "Epoch  25/100 Batch  720/781 - Loss:  0.966, Seconds: 4.43\n",
      "Epoch  25/100 Batch  740/781 - Loss:  0.879, Seconds: 4.67\n",
      "Epoch  25/100 Batch  760/781 - Loss:  0.910, Seconds: 4.31\n",
      "Average loss for this update: 0.843\n",
      "No Improvement.\n",
      "Epoch  25/100 Batch  780/781 - Loss:  0.786, Seconds: 3.91\n",
      "Epoch  26/100 Batch   20/781 - Loss:  0.923, Seconds: 4.27\n",
      "Epoch  26/100 Batch   40/781 - Loss:  0.857, Seconds: 3.35\n",
      "Epoch  26/100 Batch   60/781 - Loss:  0.850, Seconds: 4.35\n",
      "Epoch  26/100 Batch   80/781 - Loss:  0.843, Seconds: 4.37\n",
      "Epoch  26/100 Batch  100/781 - Loss:  0.743, Seconds: 4.27\n",
      "Epoch  26/100 Batch  120/781 - Loss:  0.816, Seconds: 3.77\n",
      "Epoch  26/100 Batch  140/781 - Loss:  0.775, Seconds: 3.97\n",
      "Epoch  26/100 Batch  160/781 - Loss:  0.898, Seconds: 3.25\n",
      "Epoch  26/100 Batch  180/781 - Loss:  0.877, Seconds: 3.71\n",
      "Epoch  26/100 Batch  200/781 - Loss:  0.820, Seconds: 4.27\n",
      "Epoch  26/100 Batch  220/781 - Loss:  0.836, Seconds: 3.97\n",
      "Epoch  26/100 Batch  240/781 - Loss:  0.703, Seconds: 4.03\n",
      "Average loss for this update: 0.825\n",
      "New Record!\n",
      "Epoch  26/100 Batch  260/781 - Loss:  0.785, Seconds: 4.65\n",
      "Epoch  26/100 Batch  280/781 - Loss:  0.797, Seconds: 4.33\n",
      "Epoch  26/100 Batch  300/781 - Loss:  0.868, Seconds: 4.09\n",
      "Epoch  26/100 Batch  320/781 - Loss:  0.898, Seconds: 4.13\n",
      "Epoch  26/100 Batch  340/781 - Loss:  0.794, Seconds: 4.71\n",
      "Epoch  26/100 Batch  360/781 - Loss:  0.831, Seconds: 4.35\n",
      "Epoch  26/100 Batch  380/781 - Loss:  0.710, Seconds: 4.59\n",
      "Epoch  26/100 Batch  400/781 - Loss:  0.826, Seconds: 4.51\n",
      "Epoch  26/100 Batch  420/781 - Loss:  0.773, Seconds: 3.85\n",
      "Epoch  26/100 Batch  440/781 - Loss:  0.824, Seconds: 4.55\n",
      "Epoch  26/100 Batch  460/781 - Loss:  0.861, Seconds: 4.31\n",
      "Epoch  26/100 Batch  480/781 - Loss:  0.785, Seconds: 3.93\n",
      "Epoch  26/100 Batch  500/781 - Loss:  0.842, Seconds: 4.21\n",
      "Average loss for this update: 0.811\n",
      "New Record!\n",
      "Epoch  26/100 Batch  520/781 - Loss:  0.744, Seconds: 3.99\n",
      "Epoch  26/100 Batch  540/781 - Loss:  0.791, Seconds: 4.33\n",
      "Epoch  26/100 Batch  560/781 - Loss:  0.762, Seconds: 4.01\n",
      "Epoch  26/100 Batch  580/781 - Loss:  0.857, Seconds: 4.23\n",
      "Epoch  26/100 Batch  600/781 - Loss:  0.875, Seconds: 4.25\n",
      "Epoch  26/100 Batch  620/781 - Loss:  0.818, Seconds: 4.09\n",
      "Epoch  26/100 Batch  640/781 - Loss:  0.813, Seconds: 4.33\n",
      "Epoch  26/100 Batch  660/781 - Loss:  0.710, Seconds: 4.51\n",
      "Epoch  26/100 Batch  680/781 - Loss:  0.775, Seconds: 4.37\n",
      "Epoch  26/100 Batch  700/781 - Loss:  0.865, Seconds: 4.19\n",
      "Epoch  26/100 Batch  720/781 - Loss:  0.955, Seconds: 4.51\n",
      "Epoch  26/100 Batch  740/781 - Loss:  0.865, Seconds: 4.77\n",
      "Epoch  26/100 Batch  760/781 - Loss:  0.887, Seconds: 4.33\n",
      "Average loss for this update: 0.827\n",
      "No Improvement.\n",
      "Epoch  26/100 Batch  780/781 - Loss:  0.771, Seconds: 3.85\n",
      "Epoch  27/100 Batch   20/781 - Loss:  0.913, Seconds: 4.17\n",
      "Epoch  27/100 Batch   40/781 - Loss:  0.848, Seconds: 3.47\n",
      "Epoch  27/100 Batch   60/781 - Loss:  0.840, Seconds: 4.37\n",
      "Epoch  27/100 Batch   80/781 - Loss:  0.842, Seconds: 4.23\n",
      "Epoch  27/100 Batch  100/781 - Loss:  0.741, Seconds: 4.23\n",
      "Epoch  27/100 Batch  120/781 - Loss:  0.789, Seconds: 3.77\n",
      "Epoch  27/100 Batch  140/781 - Loss:  0.757, Seconds: 4.07\n",
      "Epoch  27/100 Batch  160/781 - Loss:  0.889, Seconds: 3.21\n",
      "Epoch  27/100 Batch  180/781 - Loss:  0.848, Seconds: 3.71\n",
      "Epoch  27/100 Batch  200/781 - Loss:  0.809, Seconds: 4.29\n",
      "Epoch  27/100 Batch  220/781 - Loss:  0.824, Seconds: 4.25\n",
      "Epoch  27/100 Batch  240/781 - Loss:  0.706, Seconds: 4.11\n",
      "Average loss for this update: 0.815\n",
      "No Improvement.\n",
      "Epoch  27/100 Batch  260/781 - Loss:  0.789, Seconds: 4.33\n",
      "Epoch  27/100 Batch  280/781 - Loss:  0.785, Seconds: 4.31\n",
      "Epoch  27/100 Batch  300/781 - Loss:  0.856, Seconds: 4.23\n",
      "Epoch  27/100 Batch  320/781 - Loss:  0.884, Seconds: 4.15\n",
      "Epoch  27/100 Batch  340/781 - Loss:  0.783, Seconds: 4.53\n",
      "Epoch  27/100 Batch  360/781 - Loss:  0.825, Seconds: 4.31\n",
      "Epoch  27/100 Batch  380/781 - Loss:  0.704, Seconds: 4.66\n",
      "Epoch  27/100 Batch  400/781 - Loss:  0.812, Seconds: 4.61\n",
      "Epoch  27/100 Batch  420/781 - Loss:  0.760, Seconds: 3.87\n",
      "Epoch  27/100 Batch  440/781 - Loss:  0.813, Seconds: 4.59\n",
      "Epoch  27/100 Batch  460/781 - Loss:  0.856, Seconds: 4.43\n",
      "Epoch  27/100 Batch  480/781 - Loss:  0.803, Seconds: 4.21\n",
      "Epoch  27/100 Batch  500/781 - Loss:  0.867, Seconds: 4.11\n",
      "Average loss for this update: 0.807\n",
      "New Record!\n",
      "Epoch  27/100 Batch  520/781 - Loss:  0.753, Seconds: 3.87\n",
      "Epoch  27/100 Batch  540/781 - Loss:  0.791, Seconds: 4.47\n",
      "Epoch  27/100 Batch  560/781 - Loss:  0.760, Seconds: 3.95\n",
      "Epoch  27/100 Batch  580/781 - Loss:  0.853, Seconds: 4.25\n",
      "Epoch  27/100 Batch  600/781 - Loss:  0.880, Seconds: 4.31\n",
      "Epoch  27/100 Batch  620/781 - Loss:  0.826, Seconds: 3.99\n",
      "Epoch  27/100 Batch  640/781 - Loss:  0.817, Seconds: 4.29\n",
      "Epoch  27/100 Batch  660/781 - Loss:  0.726, Seconds: 4.45\n",
      "Epoch  27/100 Batch  680/781 - Loss:  0.761, Seconds: 4.25\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  27/100 Batch  700/781 - Loss:  0.853, Seconds: 4.31\n",
      "Epoch  27/100 Batch  720/781 - Loss:  0.944, Seconds: 4.51\n",
      "Epoch  27/100 Batch  740/781 - Loss:  0.856, Seconds: 4.76\n",
      "Epoch  27/100 Batch  760/781 - Loss:  0.891, Seconds: 4.27\n",
      "Average loss for this update: 0.826\n",
      "No Improvement.\n",
      "Epoch  27/100 Batch  780/781 - Loss:  0.777, Seconds: 3.85\n",
      "Epoch  28/100 Batch   20/781 - Loss:  0.902, Seconds: 4.25\n",
      "Epoch  28/100 Batch   40/781 - Loss:  0.824, Seconds: 3.43\n",
      "Epoch  28/100 Batch   60/781 - Loss:  0.823, Seconds: 4.31\n",
      "Epoch  28/100 Batch   80/781 - Loss:  0.837, Seconds: 4.45\n",
      "Epoch  28/100 Batch  100/781 - Loss:  0.743, Seconds: 4.27\n",
      "Epoch  28/100 Batch  120/781 - Loss:  0.792, Seconds: 3.81\n",
      "Epoch  28/100 Batch  140/781 - Loss:  0.759, Seconds: 4.01\n",
      "Epoch  28/100 Batch  160/781 - Loss:  0.870, Seconds: 3.25\n",
      "Epoch  28/100 Batch  180/781 - Loss:  0.830, Seconds: 3.61\n",
      "Epoch  28/100 Batch  200/781 - Loss:  0.792, Seconds: 4.23\n",
      "Epoch  28/100 Batch  220/781 - Loss:  0.810, Seconds: 4.13\n",
      "Epoch  28/100 Batch  240/781 - Loss:  0.684, Seconds: 4.15\n",
      "Average loss for this update: 0.802\n",
      "New Record!\n",
      "Epoch  28/100 Batch  260/781 - Loss:  0.763, Seconds: 4.75\n",
      "Epoch  28/100 Batch  280/781 - Loss:  0.780, Seconds: 4.21\n",
      "Epoch  28/100 Batch  300/781 - Loss:  0.837, Seconds: 4.11\n",
      "Epoch  28/100 Batch  320/781 - Loss:  0.875, Seconds: 4.03\n",
      "Epoch  28/100 Batch  340/781 - Loss:  0.781, Seconds: 4.57\n",
      "Epoch  28/100 Batch  360/781 - Loss:  0.808, Seconds: 4.31\n",
      "Epoch  28/100 Batch  380/781 - Loss:  0.704, Seconds: 4.49\n",
      "Epoch  28/100 Batch  400/781 - Loss:  0.797, Seconds: 4.59\n",
      "Epoch  28/100 Batch  420/781 - Loss:  0.742, Seconds: 3.97\n",
      "Epoch  28/100 Batch  440/781 - Loss:  0.800, Seconds: 4.61\n",
      "Epoch  28/100 Batch  460/781 - Loss:  0.844, Seconds: 4.31\n",
      "Epoch  28/100 Batch  480/781 - Loss:  0.752, Seconds: 3.93\n",
      "Epoch  28/100 Batch  500/781 - Loss:  0.810, Seconds: 4.29\n",
      "Average loss for this update: 0.788\n",
      "New Record!\n",
      "Epoch  28/100 Batch  520/781 - Loss:  0.717, Seconds: 3.89\n",
      "Epoch  28/100 Batch  540/781 - Loss:  0.779, Seconds: 4.41\n",
      "Epoch  28/100 Batch  560/781 - Loss:  0.740, Seconds: 3.93\n",
      "Epoch  28/100 Batch  580/781 - Loss:  0.830, Seconds: 4.27\n",
      "Epoch  28/100 Batch  600/781 - Loss:  0.835, Seconds: 4.17\n",
      "Epoch  28/100 Batch  620/781 - Loss:  0.788, Seconds: 4.15\n",
      "Epoch  28/100 Batch  640/781 - Loss:  0.792, Seconds: 4.35\n",
      "Epoch  28/100 Batch  660/781 - Loss:  0.702, Seconds: 4.47\n",
      "Epoch  28/100 Batch  680/781 - Loss:  0.764, Seconds: 4.25\n",
      "Epoch  28/100 Batch  700/781 - Loss:  0.842, Seconds: 4.61\n",
      "Epoch  28/100 Batch  720/781 - Loss:  0.926, Seconds: 4.67\n",
      "Epoch  28/100 Batch  740/781 - Loss:  0.836, Seconds: 4.65\n",
      "Epoch  28/100 Batch  760/781 - Loss:  0.866, Seconds: 4.19\n",
      "Average loss for this update: 0.804\n",
      "No Improvement.\n",
      "Epoch  28/100 Batch  780/781 - Loss:  0.761, Seconds: 3.81\n",
      "Epoch  29/100 Batch   20/781 - Loss:  0.882, Seconds: 4.27\n",
      "Epoch  29/100 Batch   40/781 - Loss:  0.814, Seconds: 3.37\n",
      "Epoch  29/100 Batch   60/781 - Loss:  0.814, Seconds: 4.31\n",
      "Epoch  29/100 Batch   80/781 - Loss:  0.804, Seconds: 4.17\n",
      "Epoch  29/100 Batch  100/781 - Loss:  0.712, Seconds: 4.37\n",
      "Epoch  29/100 Batch  120/781 - Loss:  0.774, Seconds: 3.93\n",
      "Epoch  29/100 Batch  140/781 - Loss:  0.743, Seconds: 3.89\n",
      "Epoch  29/100 Batch  160/781 - Loss:  0.861, Seconds: 3.21\n",
      "Epoch  29/100 Batch  180/781 - Loss:  0.816, Seconds: 3.63\n",
      "Epoch  29/100 Batch  200/781 - Loss:  0.773, Seconds: 4.29\n",
      "Epoch  29/100 Batch  220/781 - Loss:  0.793, Seconds: 4.17\n",
      "Epoch  29/100 Batch  240/781 - Loss:  0.657, Seconds: 4.03\n",
      "Average loss for this update: 0.785\n",
      "New Record!\n",
      "Epoch  29/100 Batch  260/781 - Loss:  0.748, Seconds: 4.57\n",
      "Epoch  29/100 Batch  280/781 - Loss:  0.745, Seconds: 4.29\n",
      "Epoch  29/100 Batch  300/781 - Loss:  0.803, Seconds: 4.21\n",
      "Epoch  29/100 Batch  320/781 - Loss:  0.860, Seconds: 4.19\n",
      "Epoch  29/100 Batch  340/781 - Loss:  0.759, Seconds: 4.69\n",
      "Epoch  29/100 Batch  360/781 - Loss:  0.794, Seconds: 4.37\n",
      "Epoch  29/100 Batch  380/781 - Loss:  0.679, Seconds: 4.61\n",
      "Epoch  29/100 Batch  400/781 - Loss:  0.794, Seconds: 4.55\n",
      "Epoch  29/100 Batch  420/781 - Loss:  0.739, Seconds: 3.99\n",
      "Epoch  29/100 Batch  440/781 - Loss:  0.791, Seconds: 4.69\n",
      "Epoch  29/100 Batch  460/781 - Loss:  0.820, Seconds: 4.59\n",
      "Epoch  29/100 Batch  480/781 - Loss:  0.747, Seconds: 4.05\n",
      "Epoch  29/100 Batch  500/781 - Loss:  0.804, Seconds: 4.33\n",
      "Average loss for this update: 0.771\n",
      "New Record!\n",
      "Epoch  29/100 Batch  520/781 - Loss:  0.701, Seconds: 4.03\n",
      "Epoch  29/100 Batch  540/781 - Loss:  0.757, Seconds: 4.37\n",
      "Epoch  29/100 Batch  560/781 - Loss:  0.724, Seconds: 3.91\n",
      "Epoch  29/100 Batch  580/781 - Loss:  0.813, Seconds: 4.25\n",
      "Epoch  29/100 Batch  600/781 - Loss:  0.825, Seconds: 4.19\n",
      "Epoch  29/100 Batch  620/781 - Loss:  0.785, Seconds: 4.03\n",
      "Epoch  29/100 Batch  640/781 - Loss:  0.786, Seconds: 4.13\n",
      "Epoch  29/100 Batch  660/781 - Loss:  0.690, Seconds: 4.35\n",
      "Epoch  29/100 Batch  680/781 - Loss:  0.734, Seconds: 4.23\n",
      "Epoch  29/100 Batch  700/781 - Loss:  0.821, Seconds: 4.27\n",
      "Epoch  29/100 Batch  720/781 - Loss:  0.905, Seconds: 4.53\n",
      "Epoch  29/100 Batch  740/781 - Loss:  0.836, Seconds: 4.73\n",
      "Epoch  29/100 Batch  760/781 - Loss:  0.854, Seconds: 4.33\n",
      "Average loss for this update: 0.79\n",
      "No Improvement.\n",
      "Epoch  29/100 Batch  780/781 - Loss:  0.741, Seconds: 3.85\n",
      "Epoch  30/100 Batch   20/781 - Loss:  0.855, Seconds: 4.25\n",
      "Epoch  30/100 Batch   40/781 - Loss:  0.793, Seconds: 3.69\n",
      "Epoch  30/100 Batch   60/781 - Loss:  0.793, Seconds: 4.39\n",
      "Epoch  30/100 Batch   80/781 - Loss:  0.791, Seconds: 4.11\n",
      "Epoch  30/100 Batch  100/781 - Loss:  0.690, Seconds: 4.23\n",
      "Epoch  30/100 Batch  120/781 - Loss:  0.766, Seconds: 3.85\n",
      "Epoch  30/100 Batch  140/781 - Loss:  0.733, Seconds: 3.99\n",
      "Epoch  30/100 Batch  160/781 - Loss:  0.845, Seconds: 3.33\n",
      "Epoch  30/100 Batch  180/781 - Loss:  0.793, Seconds: 3.71\n",
      "Epoch  30/100 Batch  200/781 - Loss:  0.761, Seconds: 4.27\n",
      "Epoch  30/100 Batch  220/781 - Loss:  0.771, Seconds: 4.13\n",
      "Epoch  30/100 Batch  240/781 - Loss:  0.646, Seconds: 4.03\n",
      "Average loss for this update: 0.768\n",
      "New Record!\n",
      "Epoch  30/100 Batch  260/781 - Loss:  0.744, Seconds: 4.63\n",
      "Epoch  30/100 Batch  280/781 - Loss:  0.734, Seconds: 4.27\n",
      "Epoch  30/100 Batch  300/781 - Loss:  0.788, Seconds: 4.09\n",
      "Epoch  30/100 Batch  320/781 - Loss:  0.839, Seconds: 4.11\n",
      "Epoch  30/100 Batch  340/781 - Loss:  0.742, Seconds: 4.53\n",
      "Epoch  30/100 Batch  360/781 - Loss:  0.792, Seconds: 4.33\n",
      "Epoch  30/100 Batch  380/781 - Loss:  0.669, Seconds: 4.61\n",
      "Epoch  30/100 Batch  400/781 - Loss:  0.777, Seconds: 4.55\n",
      "Epoch  30/100 Batch  420/781 - Loss:  0.738, Seconds: 3.91\n",
      "Epoch  30/100 Batch  440/781 - Loss:  0.824, Seconds: 4.59\n",
      "Epoch  30/100 Batch  460/781 - Loss:  0.821, Seconds: 4.41\n",
      "Epoch  30/100 Batch  480/781 - Loss:  0.727, Seconds: 4.03\n",
      "Epoch  30/100 Batch  500/781 - Loss:  0.807, Seconds: 4.23\n",
      "Average loss for this update: 0.765\n",
      "New Record!\n",
      "Epoch  30/100 Batch  520/781 - Loss:  0.693, Seconds: 4.07\n",
      "Epoch  30/100 Batch  540/781 - Loss:  0.748, Seconds: 4.49\n",
      "Epoch  30/100 Batch  560/781 - Loss:  0.724, Seconds: 4.01\n",
      "Epoch  30/100 Batch  580/781 - Loss:  0.797, Seconds: 4.53\n",
      "Epoch  30/100 Batch  600/781 - Loss:  0.810, Seconds: 4.31\n",
      "Epoch  30/100 Batch  620/781 - Loss:  0.770, Seconds: 4.25\n",
      "Epoch  30/100 Batch  640/781 - Loss:  0.767, Seconds: 4.21\n",
      "Epoch  30/100 Batch  660/781 - Loss:  0.670, Seconds: 4.39\n",
      "Epoch  30/100 Batch  680/781 - Loss:  0.722, Seconds: 4.33\n",
      "Epoch  30/100 Batch  700/781 - Loss:  0.789, Seconds: 4.25\n",
      "Epoch  30/100 Batch  720/781 - Loss:  0.894, Seconds: 4.77\n",
      "Epoch  30/100 Batch  740/781 - Loss:  0.818, Seconds: 4.69\n",
      "Epoch  30/100 Batch  760/781 - Loss:  0.856, Seconds: 4.43\n",
      "Average loss for this update: 0.777\n",
      "No Improvement.\n",
      "Epoch  30/100 Batch  780/781 - Loss:  0.735, Seconds: 3.85\n",
      "Epoch  31/100 Batch   20/781 - Loss:  0.863, Seconds: 4.15\n",
      "Epoch  31/100 Batch   40/781 - Loss:  0.789, Seconds: 3.47\n",
      "Epoch  31/100 Batch   60/781 - Loss:  0.772, Seconds: 4.39\n",
      "Epoch  31/100 Batch   80/781 - Loss:  0.780, Seconds: 4.25\n",
      "Epoch  31/100 Batch  100/781 - Loss:  0.694, Seconds: 4.17\n",
      "Epoch  31/100 Batch  120/781 - Loss:  0.744, Seconds: 4.01\n",
      "Epoch  31/100 Batch  140/781 - Loss:  0.725, Seconds: 4.09\n",
      "Epoch  31/100 Batch  160/781 - Loss:  0.822, Seconds: 3.27\n",
      "Epoch  31/100 Batch  180/781 - Loss:  0.778, Seconds: 3.63\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  31/100 Batch  200/781 - Loss:  0.736, Seconds: 4.51\n",
      "Epoch  31/100 Batch  220/781 - Loss:  0.775, Seconds: 4.03\n",
      "Epoch  31/100 Batch  240/781 - Loss:  0.647, Seconds: 3.99\n",
      "Average loss for this update: 0.758\n",
      "New Record!\n",
      "Epoch  31/100 Batch  260/781 - Loss:  0.730, Seconds: 4.55\n",
      "Epoch  31/100 Batch  280/781 - Loss:  0.752, Seconds: 4.39\n",
      "Epoch  31/100 Batch  300/781 - Loss:  0.792, Seconds: 4.21\n",
      "Epoch  31/100 Batch  320/781 - Loss:  0.832, Seconds: 4.11\n",
      "Epoch  31/100 Batch  340/781 - Loss:  0.749, Seconds: 4.51\n",
      "Epoch  31/100 Batch  360/781 - Loss:  0.782, Seconds: 4.31\n",
      "Epoch  31/100 Batch  380/781 - Loss:  0.669, Seconds: 4.55\n",
      "Epoch  31/100 Batch  400/781 - Loss:  0.775, Seconds: 4.59\n",
      "Epoch  31/100 Batch  420/781 - Loss:  0.716, Seconds: 3.89\n",
      "Epoch  31/100 Batch  440/781 - Loss:  0.769, Seconds: 4.59\n",
      "Epoch  31/100 Batch  460/781 - Loss:  0.786, Seconds: 4.33\n",
      "Epoch  31/100 Batch  480/781 - Loss:  0.713, Seconds: 3.97\n",
      "Epoch  31/100 Batch  500/781 - Loss:  0.787, Seconds: 4.19\n",
      "Average loss for this update: 0.753\n",
      "New Record!\n",
      "Epoch  31/100 Batch  520/781 - Loss:  0.672, Seconds: 3.99\n",
      "Epoch  31/100 Batch  540/781 - Loss:  0.739, Seconds: 4.31\n",
      "Epoch  31/100 Batch  560/781 - Loss:  0.700, Seconds: 3.99\n",
      "Epoch  31/100 Batch  580/781 - Loss:  0.783, Seconds: 4.29\n",
      "Epoch  31/100 Batch  600/781 - Loss:  0.796, Seconds: 4.29\n",
      "Epoch  31/100 Batch  620/781 - Loss:  0.750, Seconds: 3.97\n",
      "Epoch  31/100 Batch  640/781 - Loss:  0.757, Seconds: 4.31\n",
      "Epoch  31/100 Batch  660/781 - Loss:  0.676, Seconds: 4.59\n",
      "Epoch  31/100 Batch  680/781 - Loss:  0.748, Seconds: 4.21\n",
      "Epoch  31/100 Batch  700/781 - Loss:  0.798, Seconds: 4.37\n",
      "Epoch  31/100 Batch  720/781 - Loss:  0.883, Seconds: 4.55\n",
      "Epoch  31/100 Batch  740/781 - Loss:  0.805, Seconds: 4.73\n",
      "Epoch  31/100 Batch  760/781 - Loss:  0.840, Seconds: 4.29\n",
      "Average loss for this update: 0.768\n",
      "No Improvement.\n",
      "Epoch  31/100 Batch  780/781 - Loss:  0.722, Seconds: 4.03\n",
      "Epoch  32/100 Batch   20/781 - Loss:  0.836, Seconds: 4.23\n",
      "Epoch  32/100 Batch   40/781 - Loss:  0.777, Seconds: 3.33\n",
      "Epoch  32/100 Batch   60/781 - Loss:  0.766, Seconds: 4.51\n",
      "Epoch  32/100 Batch   80/781 - Loss:  0.770, Seconds: 4.21\n",
      "Epoch  32/100 Batch  100/781 - Loss:  0.688, Seconds: 4.27\n",
      "Epoch  32/100 Batch  120/781 - Loss:  0.738, Seconds: 3.95\n",
      "Epoch  32/100 Batch  140/781 - Loss:  0.711, Seconds: 3.93\n",
      "Epoch  32/100 Batch  160/781 - Loss:  0.825, Seconds: 3.23\n",
      "Epoch  32/100 Batch  180/781 - Loss:  0.780, Seconds: 3.75\n",
      "Epoch  32/100 Batch  200/781 - Loss:  0.765, Seconds: 4.25\n",
      "Epoch  32/100 Batch  220/781 - Loss:  0.762, Seconds: 4.15\n",
      "Epoch  32/100 Batch  240/781 - Loss:  0.636, Seconds: 4.09\n",
      "Average loss for this update: 0.752\n",
      "New Record!\n",
      "Epoch  32/100 Batch  260/781 - Loss:  0.721, Seconds: 4.59\n",
      "Epoch  32/100 Batch  280/781 - Loss:  0.724, Seconds: 4.35\n",
      "Epoch  32/100 Batch  300/781 - Loss:  0.788, Seconds: 4.19\n",
      "Epoch  32/100 Batch  320/781 - Loss:  0.835, Seconds: 4.13\n",
      "Epoch  32/100 Batch  340/781 - Loss:  0.720, Seconds: 4.55\n",
      "Epoch  32/100 Batch  360/781 - Loss:  0.752, Seconds: 4.41\n",
      "Epoch  32/100 Batch  380/781 - Loss:  0.658, Seconds: 4.57\n",
      "Epoch  32/100 Batch  400/781 - Loss:  0.763, Seconds: 4.61\n",
      "Epoch  32/100 Batch  420/781 - Loss:  0.711, Seconds: 3.93\n",
      "Epoch  32/100 Batch  440/781 - Loss:  0.760, Seconds: 4.47\n",
      "Epoch  32/100 Batch  460/781 - Loss:  0.793, Seconds: 4.33\n",
      "Epoch  32/100 Batch  480/781 - Loss:  0.696, Seconds: 4.09\n",
      "Epoch  32/100 Batch  500/781 - Loss:  0.772, Seconds: 4.15\n",
      "Average loss for this update: 0.741\n",
      "New Record!\n",
      "Epoch  32/100 Batch  520/781 - Loss:  0.661, Seconds: 4.13\n",
      "Epoch  32/100 Batch  540/781 - Loss:  0.740, Seconds: 4.45\n",
      "Epoch  32/100 Batch  560/781 - Loss:  0.702, Seconds: 3.97\n",
      "Epoch  32/100 Batch  580/781 - Loss:  0.796, Seconds: 4.25\n",
      "Epoch  32/100 Batch  600/781 - Loss:  0.806, Seconds: 4.33\n",
      "Epoch  32/100 Batch  620/781 - Loss:  0.738, Seconds: 4.03\n",
      "Epoch  32/100 Batch  640/781 - Loss:  0.746, Seconds: 4.23\n",
      "Epoch  32/100 Batch  660/781 - Loss:  0.654, Seconds: 4.43\n",
      "Epoch  32/100 Batch  680/781 - Loss:  0.701, Seconds: 4.27\n",
      "Epoch  32/100 Batch  700/781 - Loss:  0.779, Seconds: 4.29\n",
      "Epoch  32/100 Batch  720/781 - Loss:  0.862, Seconds: 4.59\n",
      "Epoch  32/100 Batch  740/781 - Loss:  0.777, Seconds: 4.63\n",
      "Epoch  32/100 Batch  760/781 - Loss:  0.833, Seconds: 4.29\n",
      "Average loss for this update: 0.757\n",
      "No Improvement.\n",
      "Epoch  32/100 Batch  780/781 - Loss:  0.708, Seconds: 3.89\n",
      "Epoch  33/100 Batch   20/781 - Loss:  0.820, Seconds: 4.23\n",
      "Epoch  33/100 Batch   40/781 - Loss:  0.761, Seconds: 3.45\n",
      "Epoch  33/100 Batch   60/781 - Loss:  0.758, Seconds: 4.45\n",
      "Epoch  33/100 Batch   80/781 - Loss:  0.744, Seconds: 4.23\n",
      "Epoch  33/100 Batch  100/781 - Loss:  0.672, Seconds: 4.23\n",
      "Epoch  33/100 Batch  120/781 - Loss:  0.730, Seconds: 3.89\n",
      "Epoch  33/100 Batch  140/781 - Loss:  0.700, Seconds: 3.97\n",
      "Epoch  33/100 Batch  160/781 - Loss:  0.809, Seconds: 3.25\n",
      "Epoch  33/100 Batch  180/781 - Loss:  0.758, Seconds: 3.65\n",
      "Epoch  33/100 Batch  200/781 - Loss:  0.734, Seconds: 4.29\n",
      "Epoch  33/100 Batch  220/781 - Loss:  0.748, Seconds: 4.01\n",
      "Epoch  33/100 Batch  240/781 - Loss:  0.630, Seconds: 4.07\n",
      "Average loss for this update: 0.737\n",
      "New Record!\n",
      "Epoch  33/100 Batch  260/781 - Loss:  0.714, Seconds: 4.61\n",
      "Epoch  33/100 Batch  280/781 - Loss:  0.712, Seconds: 4.23\n",
      "Epoch  33/100 Batch  300/781 - Loss:  0.764, Seconds: 4.13\n",
      "Epoch  33/100 Batch  320/781 - Loss:  0.814, Seconds: 4.13\n",
      "Epoch  33/100 Batch  340/781 - Loss:  0.717, Seconds: 5.39\n",
      "Epoch  33/100 Batch  360/781 - Loss:  0.753, Seconds: 4.31\n",
      "Epoch  33/100 Batch  380/781 - Loss:  0.661, Seconds: 5.16\n",
      "Epoch  33/100 Batch  400/781 - Loss:  0.742, Seconds: 4.57\n",
      "Epoch  33/100 Batch  420/781 - Loss:  0.698, Seconds: 3.89\n",
      "Epoch  33/100 Batch  440/781 - Loss:  0.740, Seconds: 4.59\n",
      "Epoch  33/100 Batch  460/781 - Loss:  0.768, Seconds: 4.37\n",
      "Epoch  33/100 Batch  480/781 - Loss:  0.690, Seconds: 3.99\n",
      "Epoch  33/100 Batch  500/781 - Loss:  0.765, Seconds: 4.37\n",
      "Average loss for this update: 0.73\n",
      "New Record!\n",
      "Epoch  33/100 Batch  520/781 - Loss:  0.673, Seconds: 3.91\n",
      "Epoch  33/100 Batch  540/781 - Loss:  0.712, Seconds: 4.45\n",
      "Epoch  33/100 Batch  560/781 - Loss:  0.678, Seconds: 3.93\n",
      "Epoch  33/100 Batch  580/781 - Loss:  0.765, Seconds: 4.19\n",
      "Epoch  33/100 Batch  600/781 - Loss:  0.771, Seconds: 4.29\n",
      "Epoch  33/100 Batch  620/781 - Loss:  0.731, Seconds: 4.09\n",
      "Epoch  33/100 Batch  640/781 - Loss:  0.730, Seconds: 4.31\n",
      "Epoch  33/100 Batch  660/781 - Loss:  0.645, Seconds: 4.53\n",
      "Epoch  33/100 Batch  680/781 - Loss:  0.696, Seconds: 4.19\n",
      "Epoch  33/100 Batch  700/781 - Loss:  0.773, Seconds: 4.37\n",
      "Epoch  33/100 Batch  720/781 - Loss:  0.841, Seconds: 4.59\n",
      "Epoch  33/100 Batch  740/781 - Loss:  0.766, Seconds: 4.55\n",
      "Epoch  33/100 Batch  760/781 - Loss:  0.817, Seconds: 4.29\n",
      "Average loss for this update: 0.741\n",
      "No Improvement.\n",
      "Epoch  33/100 Batch  780/781 - Loss:  0.702, Seconds: 3.91\n",
      "Epoch  34/100 Batch   20/781 - Loss:  0.809, Seconds: 4.27\n",
      "Epoch  34/100 Batch   40/781 - Loss:  0.749, Seconds: 3.41\n",
      "Epoch  34/100 Batch   60/781 - Loss:  0.742, Seconds: 4.49\n",
      "Epoch  34/100 Batch   80/781 - Loss:  0.743, Seconds: 4.19\n",
      "Epoch  34/100 Batch  100/781 - Loss:  0.664, Seconds: 4.21\n",
      "Epoch  34/100 Batch  120/781 - Loss:  0.713, Seconds: 3.93\n",
      "Epoch  34/100 Batch  140/781 - Loss:  0.698, Seconds: 3.93\n",
      "Epoch  34/100 Batch  160/781 - Loss:  0.784, Seconds: 3.29\n",
      "Epoch  34/100 Batch  180/781 - Loss:  0.742, Seconds: 3.79\n",
      "Epoch  34/100 Batch  200/781 - Loss:  0.713, Seconds: 4.31\n",
      "Epoch  34/100 Batch  220/781 - Loss:  0.746, Seconds: 4.01\n",
      "Epoch  34/100 Batch  240/781 - Loss:  0.616, Seconds: 4.05\n",
      "Average loss for this update: 0.724\n",
      "New Record!\n",
      "Epoch  34/100 Batch  260/781 - Loss:  0.692, Seconds: 4.61\n",
      "Epoch  34/100 Batch  280/781 - Loss:  0.710, Seconds: 4.17\n",
      "Epoch  34/100 Batch  300/781 - Loss:  0.768, Seconds: 4.07\n",
      "Epoch  34/100 Batch  320/781 - Loss:  0.811, Seconds: 4.11\n",
      "Epoch  34/100 Batch  340/781 - Loss:  0.706, Seconds: 4.73\n",
      "Epoch  34/100 Batch  360/781 - Loss:  0.750, Seconds: 4.41\n",
      "Epoch  34/100 Batch  380/781 - Loss:  0.641, Seconds: 4.51\n",
      "Epoch  34/100 Batch  400/781 - Loss:  0.742, Seconds: 4.47\n",
      "Epoch  34/100 Batch  420/781 - Loss:  0.695, Seconds: 3.91\n",
      "Epoch  34/100 Batch  440/781 - Loss:  0.730, Seconds: 4.65\n",
      "Epoch  34/100 Batch  460/781 - Loss:  0.763, Seconds: 4.47\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  34/100 Batch  480/781 - Loss:  0.696, Seconds: 4.09\n",
      "Epoch  34/100 Batch  500/781 - Loss:  0.751, Seconds: 4.17\n",
      "Average loss for this update: 0.725\n",
      "No Improvement.\n",
      "Epoch  34/100 Batch  520/781 - Loss:  0.662, Seconds: 3.85\n",
      "Epoch  34/100 Batch  540/781 - Loss:  0.696, Seconds: 4.43\n",
      "Epoch  34/100 Batch  560/781 - Loss:  0.672, Seconds: 4.03\n",
      "Epoch  34/100 Batch  580/781 - Loss:  0.756, Seconds: 4.29\n",
      "Epoch  34/100 Batch  600/781 - Loss:  0.766, Seconds: 4.31\n",
      "Epoch  34/100 Batch  620/781 - Loss:  0.728, Seconds: 4.07\n",
      "Epoch  34/100 Batch  640/781 - Loss:  0.726, Seconds: 4.33\n",
      "Epoch  34/100 Batch  660/781 - Loss:  0.633, Seconds: 4.39\n",
      "Epoch  34/100 Batch  680/781 - Loss:  0.698, Seconds: 4.21\n",
      "Epoch  34/100 Batch  700/781 - Loss:  0.761, Seconds: 4.35\n",
      "Epoch  34/100 Batch  720/781 - Loss:  0.833, Seconds: 4.47\n",
      "Epoch  34/100 Batch  740/781 - Loss:  0.756, Seconds: 4.73\n",
      "Epoch  34/100 Batch  760/781 - Loss:  0.788, Seconds: 4.35\n",
      "Average loss for this update: 0.731\n",
      "No Improvement.\n",
      "Epoch  34/100 Batch  780/781 - Loss:  0.689, Seconds: 3.93\n",
      "Epoch  35/100 Batch   20/781 - Loss:  0.788, Seconds: 4.21\n",
      "Epoch  35/100 Batch   40/781 - Loss:  0.730, Seconds: 3.47\n",
      "Epoch  35/100 Batch   60/781 - Loss:  0.731, Seconds: 4.37\n",
      "Epoch  35/100 Batch   80/781 - Loss:  0.729, Seconds: 4.25\n",
      "Epoch  35/100 Batch  100/781 - Loss:  0.648, Seconds: 4.25\n",
      "Epoch  35/100 Batch  120/781 - Loss:  0.700, Seconds: 3.91\n",
      "Epoch  35/100 Batch  140/781 - Loss:  0.668, Seconds: 4.23\n",
      "Epoch  35/100 Batch  160/781 - Loss:  0.766, Seconds: 3.33\n",
      "Epoch  35/100 Batch  180/781 - Loss:  0.726, Seconds: 3.75\n",
      "Epoch  35/100 Batch  200/781 - Loss:  0.691, Seconds: 4.19\n",
      "Epoch  35/100 Batch  220/781 - Loss:  0.733, Seconds: 4.11\n",
      "Epoch  35/100 Batch  240/781 - Loss:  0.605, Seconds: 4.21\n",
      "Average loss for this update: 0.708\n",
      "New Record!\n",
      "Epoch  35/100 Batch  260/781 - Loss:  0.689, Seconds: 4.51\n",
      "Epoch  35/100 Batch  280/781 - Loss:  0.693, Seconds: 4.31\n",
      "Epoch  35/100 Batch  300/781 - Loss:  0.746, Seconds: 4.17\n",
      "Epoch  35/100 Batch  320/781 - Loss:  0.790, Seconds: 4.27\n",
      "Epoch  35/100 Batch  340/781 - Loss:  0.690, Seconds: 4.55\n",
      "Epoch  35/100 Batch  360/781 - Loss:  0.729, Seconds: 4.79\n",
      "Epoch  35/100 Batch  380/781 - Loss:  0.638, Seconds: 4.61\n",
      "Epoch  35/100 Batch  400/781 - Loss:  0.719, Seconds: 4.89\n",
      "Epoch  35/100 Batch  420/781 - Loss:  0.684, Seconds: 3.93\n",
      "Epoch  35/100 Batch  440/781 - Loss:  0.722, Seconds: 4.57\n",
      "Epoch  35/100 Batch  460/781 - Loss:  0.738, Seconds: 4.35\n",
      "Epoch  35/100 Batch  480/781 - Loss:  0.678, Seconds: 3.95\n",
      "Epoch  35/100 Batch  500/781 - Loss:  0.744, Seconds: 4.37\n",
      "Average loss for this update: 0.708\n",
      "New Record!\n",
      "Epoch  35/100 Batch  520/781 - Loss:  0.632, Seconds: 3.77\n",
      "Epoch  35/100 Batch  540/781 - Loss:  0.686, Seconds: 4.45\n",
      "Epoch  35/100 Batch  560/781 - Loss:  0.671, Seconds: 4.17\n",
      "Epoch  35/100 Batch  580/781 - Loss:  0.750, Seconds: 4.29\n",
      "Epoch  35/100 Batch  600/781 - Loss:  0.756, Seconds: 4.27\n",
      "Epoch  35/100 Batch  620/781 - Loss:  0.696, Seconds: 4.01\n",
      "Epoch  35/100 Batch  640/781 - Loss:  0.709, Seconds: 4.29\n",
      "Epoch  35/100 Batch  660/781 - Loss:  0.626, Seconds: 4.55\n",
      "Epoch  35/100 Batch  680/781 - Loss:  0.685, Seconds: 4.33\n",
      "Epoch  35/100 Batch  700/781 - Loss:  0.753, Seconds: 4.55\n",
      "Epoch  35/100 Batch  720/781 - Loss:  0.834, Seconds: 4.57\n",
      "Epoch  35/100 Batch  740/781 - Loss:  0.744, Seconds: 4.71\n",
      "Epoch  35/100 Batch  760/781 - Loss:  0.787, Seconds: 4.33\n",
      "Average loss for this update: 0.721\n",
      "No Improvement.\n",
      "Epoch  35/100 Batch  780/781 - Loss:  0.681, Seconds: 4.13\n",
      "Epoch  36/100 Batch   20/781 - Loss:  0.797, Seconds: 4.27\n",
      "Epoch  36/100 Batch   40/781 - Loss:  0.727, Seconds: 3.61\n",
      "Epoch  36/100 Batch   60/781 - Loss:  0.722, Seconds: 4.31\n",
      "Epoch  36/100 Batch   80/781 - Loss:  0.723, Seconds: 4.27\n",
      "Epoch  36/100 Batch  100/781 - Loss:  0.639, Seconds: 4.25\n",
      "Epoch  36/100 Batch  120/781 - Loss:  0.690, Seconds: 3.85\n",
      "Epoch  36/100 Batch  140/781 - Loss:  0.663, Seconds: 4.07\n",
      "Epoch  36/100 Batch  160/781 - Loss:  0.760, Seconds: 3.31\n",
      "Epoch  36/100 Batch  180/781 - Loss:  0.718, Seconds: 3.89\n",
      "Epoch  36/100 Batch  200/781 - Loss:  0.713, Seconds: 4.31\n",
      "Epoch  36/100 Batch  220/781 - Loss:  0.738, Seconds: 4.17\n",
      "Epoch  36/100 Batch  240/781 - Loss:  0.602, Seconds: 4.05\n",
      "Average loss for this update: 0.706\n",
      "New Record!\n",
      "Epoch  36/100 Batch  260/781 - Loss:  0.684, Seconds: 4.53\n",
      "Epoch  36/100 Batch  280/781 - Loss:  0.704, Seconds: 4.49\n",
      "Epoch  36/100 Batch  300/781 - Loss:  0.725, Seconds: 4.27\n",
      "Epoch  36/100 Batch  320/781 - Loss:  0.785, Seconds: 4.07\n",
      "Epoch  36/100 Batch  340/781 - Loss:  0.683, Seconds: 4.47\n",
      "Epoch  36/100 Batch  360/781 - Loss:  0.704, Seconds: 4.53\n",
      "Epoch  36/100 Batch  380/781 - Loss:  0.626, Seconds: 4.57\n",
      "Epoch  36/100 Batch  400/781 - Loss:  0.699, Seconds: 4.45\n",
      "Epoch  36/100 Batch  420/781 - Loss:  0.668, Seconds: 3.95\n",
      "Epoch  36/100 Batch  440/781 - Loss:  0.704, Seconds: 4.77\n",
      "Epoch  36/100 Batch  460/781 - Loss:  0.732, Seconds: 4.63\n",
      "Epoch  36/100 Batch  480/781 - Loss:  0.707, Seconds: 4.01\n",
      "Epoch  36/100 Batch  500/781 - Loss:  0.790, Seconds: 4.22\n",
      "Average loss for this update: 0.706\n",
      "No Improvement.\n",
      "Epoch  36/100 Batch  520/781 - Loss:  0.663, Seconds: 3.85\n",
      "Epoch  36/100 Batch  540/781 - Loss:  0.703, Seconds: 4.35\n",
      "Epoch  36/100 Batch  560/781 - Loss:  0.681, Seconds: 4.11\n",
      "Epoch  36/100 Batch  580/781 - Loss:  0.759, Seconds: 4.37\n",
      "Epoch  36/100 Batch  600/781 - Loss:  0.743, Seconds: 4.29\n",
      "Epoch  36/100 Batch  620/781 - Loss:  0.711, Seconds: 4.05\n",
      "Epoch  36/100 Batch  640/781 - Loss:  0.722, Seconds: 4.21\n",
      "Epoch  36/100 Batch  660/781 - Loss:  0.619, Seconds: 4.43\n",
      "Epoch  36/100 Batch  680/781 - Loss:  0.677, Seconds: 4.29\n",
      "Epoch  36/100 Batch  700/781 - Loss:  0.740, Seconds: 4.31\n",
      "Epoch  36/100 Batch  720/781 - Loss:  0.825, Seconds: 4.43\n",
      "Epoch  36/100 Batch  740/781 - Loss:  0.738, Seconds: 5.05\n",
      "Epoch  36/100 Batch  760/781 - Loss:  0.772, Seconds: 4.27\n",
      "Average loss for this update: 0.721\n",
      "No Improvement.\n",
      "Epoch  36/100 Batch  780/781 - Loss:  0.689, Seconds: 4.35\n",
      "Epoch  37/100 Batch   20/781 - Loss:  0.849, Seconds: 4.29\n",
      "Epoch  37/100 Batch   40/781 - Loss:  0.742, Seconds: 3.49\n",
      "Epoch  37/100 Batch   60/781 - Loss:  0.720, Seconds: 4.41\n",
      "Epoch  37/100 Batch   80/781 - Loss:  0.724, Seconds: 4.39\n",
      "Epoch  37/100 Batch  100/781 - Loss:  0.643, Seconds: 4.33\n",
      "Epoch  37/100 Batch  120/781 - Loss:  0.690, Seconds: 3.81\n",
      "Epoch  37/100 Batch  140/781 - Loss:  0.673, Seconds: 4.03\n",
      "Epoch  37/100 Batch  160/781 - Loss:  0.757, Seconds: 3.33\n",
      "Epoch  37/100 Batch  180/781 - Loss:  0.716, Seconds: 3.83\n",
      "Epoch  37/100 Batch  200/781 - Loss:  0.690, Seconds: 4.33\n",
      "Epoch  37/100 Batch  220/781 - Loss:  0.715, Seconds: 4.09\n",
      "Epoch  37/100 Batch  240/781 - Loss:  0.603, Seconds: 4.23\n",
      "Average loss for this update: 0.707\n",
      "No Improvement.\n",
      "Stopping Training.\n"
     ]
    }
   ],
   "source": [
    "# Train the Model\n",
    "learning_rate_decay = 0.95\n",
    "min_learning_rate = 0.0005\n",
    "display_step = 20 # Check training loss after every 20 batches\n",
    "stop_early = 0 \n",
    "stop = 3 # If the update loss does not decrease in 3 consecutive update checks, stop training\n",
    "per_epoch = 3 # Make 3 update checks per epoch\n",
    "update_check = (len(sorted_texts_short)//batch_size//per_epoch)-1\n",
    "\n",
    "update_loss = 0 \n",
    "batch_loss = 0\n",
    "summary_update_loss = [] # Record the update losses for saving improvements in the model\n",
    "\n",
    "checkpoint = \"./best_model.ckpt\" \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # If we want to continue training a previous session\n",
    "    #loader = tf.train.import_meta_graph(\"./\" + checkpoint + '.meta')\n",
    "    #loader.restore(sess, checkpoint)\n",
    "    \n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        update_loss = 0\n",
    "        batch_loss = 0\n",
    "        for batch_i, (summaries_batch, texts_batch, summaries_lengths, texts_lengths) in enumerate(\n",
    "                get_batches(sorted_summaries_short, sorted_texts_short, batch_size)):\n",
    "            start_time = time.time()\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: texts_batch,\n",
    "                 targets: summaries_batch,\n",
    "                 lr: learning_rate,\n",
    "                 summary_length: summaries_lengths,\n",
    "                 text_length: texts_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "            batch_loss += loss\n",
    "            update_loss += loss\n",
    "            end_time = time.time()\n",
    "            batch_time = end_time - start_time\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(sorted_texts_short) // batch_size, \n",
    "                              batch_loss / display_step, \n",
    "                              batch_time*display_step))\n",
    "                batch_loss = 0\n",
    "\n",
    "            if batch_i % update_check == 0 and batch_i > 0:\n",
    "                print(\"Average loss for this update:\", round(update_loss/update_check,3))\n",
    "                summary_update_loss.append(update_loss)\n",
    "                \n",
    "                # If the update loss is at a new minimum, save the model\n",
    "                if update_loss <= min(summary_update_loss):\n",
    "                    print('New Record!') \n",
    "                    stop_early = 0\n",
    "                    saver = tf.train.Saver() \n",
    "                    saver.save(sess, checkpoint)\n",
    "\n",
    "                else:\n",
    "                    print(\"No Improvement.\")\n",
    "                    stop_early += 1\n",
    "                    if stop_early == stop:\n",
    "                        break\n",
    "                update_loss = 0\n",
    "            \n",
    "                    \n",
    "        # Reduce learning rate, but not below its minimum value\n",
    "        learning_rate *= learning_rate_decay\n",
    "        if learning_rate < min_learning_rate:\n",
    "            learning_rate = min_learning_rate\n",
    "        \n",
    "        if stop_early == stop:\n",
    "            print(\"Stopping Training.\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the quality of the summaries that this model can generate, you can either create your own review, or use a review from the dataset. You can set the length of the summary to a fixed value, or use a random value like I have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in text.split()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- **input_sentences**: a list of reviews strings we are going to summarize\n",
    "- **generagte_summary_length**: a int or list, if a list must be same length as input_sentences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./best_model.ckpt\n",
      "- Review:\n",
      " The coffee tasted great and was at such a good price! I highly recommend this to everyone!\n",
      "- Summary:\n",
      " great great coffee\n",
      "\n",
      "\n",
      "- Review:\n",
      " love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
      "- Summary:\n",
      " great taste\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "input_sentences=[\"The coffee tasted great and was at such a good price! I highly recommend this to everyone!\",\n",
    "               \"love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\"]\n",
    "generagte_summary_length =  [3,2]\n",
    "\n",
    "texts = [text_to_seq(input_sentence) for input_sentence in input_sentences]\n",
    "checkpoint = \"./best_model.ckpt\"\n",
    "if type(generagte_summary_length) is list:\n",
    "    if len(input_sentences)!=len(generagte_summary_length):\n",
    "        raise Exception(\"[Error] makeSummaries parameter generagte_summary_length must be same length as input_sentences or an integer\")\n",
    "    generagte_summary_length_list = generagte_summary_length\n",
    "else:\n",
    "    generagte_summary_length_list = [generagte_summary_length] * len(texts)\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    text_length = loaded_graph.get_tensor_by_name('text_length:0')\n",
    "    summary_length = loaded_graph.get_tensor_by_name('summary_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    for i, text in enumerate(texts):\n",
    "        generagte_summary_length = generagte_summary_length_list[i]\n",
    "        answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                          summary_length: [generagte_summary_length], #summary_length: [np.random.randint(5,8)], \n",
    "                                          text_length: [len(text)]*batch_size,\n",
    "                                          keep_prob: 1.0})[0] \n",
    "        # Remove the padding from the summaries\n",
    "        pad = vocab_to_int[\"<PAD>\"] \n",
    "        print('- Review:\\n\\r {}'.format(input_sentences[i]))\n",
    "        print('- Summary:\\n\\r {}\\n\\r\\n\\r'.format(\" \".join([int_to_vocab[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope that you found this project to be rather interesting and informative. One of my main recommendations for working with this dataset and model is either use a GPU, a subset of the dataset, or plenty of time to train your model. As you might be able to expect, the model will not be able to make good predictions just by seeing many reviews, it needs so see the reviews many times to be able to understand the relationship between words and between descriptions & summaries. \n",
    "\n",
    "In short, I'm pleased with how well this model performs. After creating numerous reviews and checking those from the dataset, I can happily say that most of the generated summaries are appropriate, some of them are great, and some of them make mistakes. I'll try to improve this model and if it gets better, I'll update my GitHub.\n",
    "\n",
    "Thanks for reading!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}