How_to_build_own_text_summarizer_using_deep_learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "How to build own text summarizer using deep learning.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "language_info": {
      "name": "python",
      "version": "3.6.4",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFuL-RBgXqgU",
        "colab_type": "text"
      },
      "source": [
        "In this notebook, we will build an abstractive based text summarizer using deep learning from the scratch in python using keras\n",
        "\n",
        "I recommend you to go through the article over [here](https://www.analyticsvidhya.com/blog/2019/06/comprehensive-guide-text-summarization-using-deep-learning-python/) to cover all the concepts which is required to build our own summarizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5dSoP8lGMZi",
        "colab_type": "text"
      },
      "source": [
        "#Understanding the Problem Statement\n",
        "\n",
        "Customer reviews can often be long and descriptive. Analyzing these reviews manually, as you can imagine, is really time-consuming. This is where the brilliance of Natural Language Processing can be applied to generate a summary for long reviews.\n",
        "\n",
        "We will be working on a really cool dataset. Our objective here is to generate a summary for the Amazon Fine Food reviews using the abstraction-based approach we learned about above. You can download the dataset from[ here ](https://www.kaggle.com/snap/amazon-fine-food-reviews)\n",
        "\n",
        "It’s time to fire up our Jupyter notebooks! Let’s dive into the implementation details right away.\n",
        "\n",
        "#Custom Attention Layer\n",
        "\n",
        "Keras does not officially support attention layer. So, we can either implement our own attention layer or use a third-party implementation. We will go with the latter option for this article. You can download the attention layer from [here](https://github.com/thushv89/attention_keras/blob/master/layers/attention.py) and copy it in a different file called attention.py.\n",
        "\n",
        "Let’s import it into our environment:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Fi64aA0FFxcS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from attention import AttentionLayer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUValOzcHtEK",
        "colab_type": "text"
      },
      "source": [
        "#Import the Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "id": "_Jpu8qLEFxcY",
        "colab_type": "code",
        "colab": {},
        "outputId": "95968e01-faac-4911-c802-9c008a4e62cf"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd \n",
        "import re\n",
        "from bs4 import BeautifulSoup\n",
        "from keras.preprocessing.text import Tokenizer \n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from nltk.corpus import stopwords\n",
        "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "import warnings\n",
        "pd.set_option(\"display.max_colwidth\", 200)\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVakjZ3oICgx",
        "colab_type": "text"
      },
      "source": [
        "#Read the dataset\n",
        "\n",
        "This dataset consists of reviews of fine foods from Amazon. The data spans a period of more than 10 years, including all ~500,000 reviews up to October 2012. These reviews include product and user information, ratings, plain text review, and summary. It also includes reviews from all other Amazon categories.\n",
        "\n",
        "We’ll take a sample of 100,000 reviews to reduce the training time of our model. Feel free to use the entire dataset for training your model if your machine has that kind of computational power."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "wnK5o4Z1Fxcj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data=pd.read_csv(\"../input/amazon-fine-food-reviews/Reviews.csv\",nrows=100000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGNQKvCaISIn",
        "colab_type": "text"
      },
      "source": [
        "# Drop Duplicates and NA values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Cjul88oOFxcr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.drop_duplicates(subset=['Text'],inplace=True)#dropping duplicates\n",
        "data.dropna(axis=0,inplace=True)#dropping na"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qi0xD6BkIWAm",
        "colab_type": "text"
      },
      "source": [
        "# Information about dataset\n",
        "\n",
        "Let us look at datatypes and shape of the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "__fy-JxTFxc9",
        "colab_type": "code",
        "colab": {},
        "outputId": "d42c6e36-bbc8-43c2-de0e-d3effe3e8c4c"
      },
      "source": [
        "data.info()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 88421 entries, 0 to 99999\n",
            "Data columns (total 10 columns):\n",
            "Id                        88421 non-null int64\n",
            "ProductId                 88421 non-null object\n",
            "UserId                    88421 non-null object\n",
            "ProfileName               88421 non-null object\n",
            "HelpfulnessNumerator      88421 non-null int64\n",
            "HelpfulnessDenominator    88421 non-null int64\n",
            "Score                     88421 non-null int64\n",
            "Time                      88421 non-null int64\n",
            "Summary                   88421 non-null object\n",
            "Text                      88421 non-null object\n",
            "dtypes: int64(5), object(5)\n",
            "memory usage: 7.4+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0xLYACiFxdJ",
        "colab_type": "text"
      },
      "source": [
        "#Preprocessing\n",
        "\n",
        "Performing basic preprocessing steps is very important before we get to the model building part. Using messy and uncleaned text data is a potentially disastrous move. So in this step, we will drop all the unwanted symbols, characters, etc. from the text that do not affect the objective of our problem.\n",
        "\n",
        "Here is the dictionary that we will use for expanding the contractions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "0s6IY-x2FxdL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "contraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
        "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
        "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
        "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
        "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
        "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
        "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
        "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
        "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
        "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
        "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
        "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
        "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
        "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
        "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
        "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
        "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
        "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
        "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
        "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
        "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
        "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
        "                           \"you're\": \"you are\", \"you've\": \"you have\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2JFRXFHmI7Mj",
        "colab_type": "text"
      },
      "source": [
        "We will perform the below preprocessing tasks for our data:\n",
        "\n",
        "1.Convert everything to lowercase\n",
        "\n",
        "2.Remove HTML tags\n",
        "\n",
        "3.Contraction mapping\n",
        "\n",
        "4.Remove (‘s)\n",
        "\n",
        "5.Remove any text inside the parenthesis ( )\n",
        "\n",
        "6.Eliminate punctuations and special characters\n",
        "\n",
        "7.Remove stopwords\n",
        "\n",
        "8.Remove short words\n",
        "\n",
        "Let’s define the function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "XZr-u3OEFxdT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "stop_words = set(stopwords.words('english')) \n",
        "\n",
        "def text_cleaner(text,num):\n",
        "    newString = text.lower()\n",
        "    newString = BeautifulSoup(newString, \"lxml\").text\n",
        "    newString = re.sub(r'\\([^)]*\\)', '', newString)\n",
        "    newString = re.sub('\"','', newString)\n",
        "    newString = ' '.join([contraction_mapping[t] if t in contraction_mapping else t for t in newString.split(\" \")])    \n",
        "    newString = re.sub(r\"'s\\b\",\"\",newString)\n",
        "    newString = re.sub(\"[^a-zA-Z]\", \" \", newString) \n",
        "    newString = re.sub('[m]{2,}', 'mm', newString)\n",
        "    if(num==0):\n",
        "        tokens = [w for w in newString.split() if not w in stop_words]\n",
        "    else:\n",
        "        tokens=newString.split()\n",
        "    long_words=[]\n",
        "    for i in tokens:\n",
        "        if len(i)>1:                                                 #removing short word\n",
        "            long_words.append(i)   \n",
        "    return (\" \".join(long_words)).strip()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "A2QAeCHWFxdY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#call the function\n",
        "cleaned_text = []\n",
        "for t in data['Text']:\n",
        "    cleaned_text.append(text_cleaner(t,0)) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "snRZY8wjLao2",
        "colab_type": "text"
      },
      "source": [
        "Let us look at the first five preprocessed reviews"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "NCAIkhWbFxdh",
        "colab_type": "code",
        "colab": {},
        "outputId": "c2da1a36-4488-4e32-ef9e-fcfe496e374d"
      },
      "source": [
        "cleaned_text[:5]  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['bought several vitality canned dog food products found good quality product looks like stew processed meat smells better labrador finicky appreciates product better',\n",
              " 'product arrived labeled jumbo salted peanuts peanuts actually small sized unsalted sure error vendor intended represent product jumbo',\n",
              " 'confection around centuries light pillowy citrus gelatin nuts case filberts cut tiny squares liberally coated powdered sugar tiny mouthful heaven chewy flavorful highly recommend yummy treat familiar story lewis lion witch wardrobe treat seduces edmund selling brother sisters witch',\n",
              " 'looking secret ingredient robitussin believe found got addition root beer extract ordered made cherry soda flavor medicinal',\n",
              " 'great taffy great price wide assortment yummy taffy delivery quick taffy lover deal']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "GsRXocxoFxd-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#call the function\n",
        "cleaned_summary = []\n",
        "for t in data['Summary']:\n",
        "    cleaned_summary.append(text_cleaner(t,1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZeD0gs6Lnb-",
        "colab_type": "text"
      },
      "source": [
        "Let us look at the first 10 preprocessed summaries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "jQJdZcAzFxee",
        "colab_type": "code",
        "colab": {},
        "outputId": "a1fbe683-c03f-4afb-addf-e075021c121b"
      },
      "source": [
        "cleaned_summary[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['good quality dog food',\n",
              " 'not as advertised',\n",
              " 'delight says it all',\n",
              " 'cough medicine',\n",
              " 'great taffy',\n",
              " 'nice taffy',\n",
              " 'great just as good as the expensive brands',\n",
              " 'wonderful tasty taffy',\n",
              " 'yay barley',\n",
              " 'healthy dog food']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "L1zLpnqsFxey",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data['cleaned_text']=cleaned_text\n",
        "data['cleaned_summary']=cleaned_summary"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KT_D2cLiLy77",
        "colab_type": "text"
      },
      "source": [
        "#Drop empty rows"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "sYK390unFxfA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.replace('', np.nan, inplace=True)\n",
        "data.dropna(axis=0,inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vm8Fk2TCL7Sp",
        "colab_type": "text"
      },
      "source": [
        "#Understanding the distribution of the sequences\n",
        "\n",
        "Here, we will analyze the length of the reviews and the summary to get an overall idea about the distribution of length of the text. This will help us fix the maximum length of the sequence:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "MdF76AHHFxgw",
        "colab_type": "code",
        "colab": {},
        "outputId": "e3bbe165-4235-482f-bfd4-36a3f1d95290"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "text_word_count = []\n",
        "summary_word_count = []\n",
        "\n",
        "# populate the lists with sentence lengths\n",
        "for i in data['cleaned_text']:\n",
        "      text_word_count.append(len(i.split()))\n",
        "\n",
        "for i in data['cleaned_summary']:\n",
        "      summary_word_count.append(len(i.split()))\n",
        "\n",
        "length_df = pd.DataFrame({'text':text_word_count, 'summary':summary_word_count})\n",
        "\n",
        "length_df.hist(bins = 30)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEICAYAAAC0+DhzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X+01XWd7/HnK03zWgZonRBssImaZTKRcJVZdZtTJiI1YXeVQd5AY0kttbF1WRU2rUWjOZfupI3OeC1KrtCY6NVMpjA6kXuZ6w4KJImgDkfC62EhFKB0qCzoff/4fnZ82d999tlwfuwfvB5r7bX3fn8/3+/+fs767vPe38/38/18FBGYmZnlvaLRO2BmZs3HycHMzAqcHMzMrMDJwczMCpwczMyswMnBzMwKnBzMzKzAycHMmpqkbZLeNwjbuUPSlwdjn44FTg5WN0nHN3ofzGx4ODkMM0mfl7Rd0q8lPSPp/MpfNJI6JfXk3m+T9FlJT0jaL+l2SR2SHkzb+bGkkansOEkh6XJJz0vaK+lTkv5zWv9FSf+S2/afS/qJpN2SfiXpTkkjKj7785KeAPan/bivok63SLp5SP9wdkyS9G3gjcC/SeqV9DlJUyT933Qs/1xSZyo7SlKPpL9J718tqVvSbEnzgEuBz6Xt/FvDKtUqIsKPYXoAbwWeB05P78cBfw7cAXw5V64T6Mm93wasATqAMcAu4GfAO4BXAT8BFua2GcDX07KpwO+A7wGvz63/16n8m4ELgBOB1wEPA/9U8dkbgDOAk4DRwH5gRFp+fNrepEb/ff1oz0c6Bt+XXo8BdgPTyX7cXpDevy4tnwq8kI71bwL35rZz2PfMj9oPnzkMr4Nk/4TPkvTKiNgWEc/Wue4/R8TOiNgO/BR4NCIej4jfAfeTJYq86yPidxHxI7J/5ndFxK7c+u8AiIjuiOiKiJcj4pfATcBfV2zrloh4PiJ+GxE7yBLIR9KyacCvImL9Ef0lzI7OfwNWRsTKiPhjRHQB68iSBel4/z/A6hT7ZMP2tMU5OQyjiOgGPgN8Cdglabmk0+tcfWfu9W+rvH/10ZRPzVPLU1PXPuBfgdMqtvV8xfulZF9S0vO366yD2UD9GfCR1KT0oqQXgXeRndGWLQbOBu6IiN2N2Ml24OQwzCLiOxHxLrKDPICvkP2y/0+5Ym8Yxl36h7QfEyLiFLJ/9qooUzl07/eAv5R0NvAB4M4h30s7luWPv+eBb0fEiNzj5IhYBCDpOLLksAy4UtKb+9iO9cPJYRhJequk90o6kew6wG+BP5K16U9PF9TeQHZ2MVxeA/QCL0kaA3y2vxVSU9a9wHeAxyLi/w3tLtoxbifwpvT6X4G/kXShpOMkvSp14Bibln+BLAl8AvhHYFlKGJXbsX44OQyvE4FFwK84dNHsWrJmmZ+TXXj7EXD3MO7T3wPnAC8BPwC+W+d6S4EJuEnJht7/AL6YmpA+CswgSwK/JDuT+CzwCkmTgP8OzI6Ig2Rn5QEsSNu5nex634uSvjfMdWg5SlfxzY6IpDcCTwNviIh9jd4fMxtcPnOwIybpFWS/0JY7MZi1J9/xakdE0slkbbfPkXVjNbM25GYlMzMr6LdZSdIZkh6StFnSJknXpPgoSV2StqTn8vANSsMpdKfhGs7JbWtOKr9F0pxcfJKkjWmdWyRVdqU0M7Nh1O+Zg6TRwOiI+Jmk1wDrgYuBy4A9EbFI0gJgZER8XtJ04NNkdyeeB9wcEedJGkV2J+Nksh4E68mGXNgr6THgb4FHgZVkd+Q+WGu/TjvttBg3bhz79+/n5JNPPuo/QDNwHRpj/fr1v4qI1zV6P+pVPuYrteLfvh6u19Co+7g/0vE2gAfIxjN5hixpQHZ34jPp9TeAWbnyz6Tls4Bv5OLfSLHRwNO5+GHl+npMmjQpIiIeeuihaHWuQ2MA66IJxrCp91E+5iu14t++Hq7X0Kj3uD+iC9KSxpGNyfMo0BHZODuQ9dnvSK/HcPhwCz0pViveUyVe7fPnAfMAOjo6KJVK9Pb2UiqVjqQaTcd1MLNmU3dykPRq4D7gMxGxL39ZICJC0pBf2Y6IxWS3xjN58uTo7OykVCrR2dk51B89pFwHM2s2dd3nIOmVZInhzogo30G7M12PKF+X2JXi28mGdy4bm2K14mOrxM3MrEHq6a0kstvOn4qIm3KLVgDlHkdzyK5FlOOzU6+lKcBLqflpFTBV0sjUs2kqsCot25cm8BAwO7ctMzNrgHqald4JfBzYKGlDin2BbIygeyTNJbsh6pK0bCVZT6Vu4DfA5QARsUfS9cDaVO66iNiTXl9JNhHHScCD6WFmZg3Sb3KIiEcoDuFcdn6V8gFc1ce2lgBLqsTXkY2/bmZmTcBjK5mZWYGTg5mZFTg5mJlZwTExKuu4BT847P22Re9v0J6YDQ0f4zbYfOZgZmYFTg5mZlbg5GBmZgVODmZmVuDkYGZmBU4OZmZW4ORgVoWkEZLulfS0pKck/ZWnxrVjiZODWXU3Az+MiL8A3g48BSwAVkfEeGB1eg9wETA+PeYBt0E2zzqwkGy63HOBheWEkspckVtv2jDUyaxuTg5mFSS9Fng32VD1RMTvI+JFYAawNBVbSjaXOim+LM3CuAYYkeY4uRDoiog9EbEX6AKmpWWnRMSaNFDlsty2zJrCMXGHtNkROhP4JfC/Jb0dWA9cQ5NMjVupt7eX+RMOHhZrhylb23Xq2Vapl5ODWdHxwDnApyPiUUk3c6gJCWjs1LiVSqUSNz6y/7DYtkuL5VpNu0492yr1crOSWVEP0BMRj6b395IlC0+Na8cMJwezChHxAvC8pLem0PnAZjw1rh1D3KxkVt2ngTslnQBsJZvu9hV4alw7RvSbHCQtAT4A7IqIs1PsbqD8q2oE8GJETJQ0jqzL3zNp2ZqI+FRaZxKHvgwrgWtSu+0o4G5gHLANuCT17DBrmIjYAEyusshT49oxoZ5mpTuo6IMdER+NiIkRMRG4D/hubvGz5WXlxJD01a+7r77jZmbWIP0mh4h4GNhTbVlqL70EuKvWNvrp191X33EzM2uQgV5z+C/AzojYkoudKelxYB/wxYj4KbX7dffVd7ygWp/vevoMz59w4LD3zdbHuFX6PdfSDnUws0MGmhxmcfhZww7gjRGxO11j+J6kt9W7sf76jlfr811Pn+HLKqdQbLI+4K3S77mWdqiDmR1y1MlB0vHAfwUmlWMR8TLwcnq9XtKzwFuo3a97p6TREbGjou+4mZk1yEDuc3gf8HRE/Km5SNLrJB2XXr+J7MLz1n76dffVd9zMzBqk3+Qg6S7g34G3SupJfbwBZlK8EP1u4AlJG8juKv1URb/ub5H1BX+WQ/26FwEXSNpClnAWDaA+ZmY2CPptVoqIWX3EL6sSu4+sa2u18lX7dUfEbqr0HTczs8bx8BlmZlbg5GBmZgVODmZmVnBMDrw3ruK+B4Bti97fgD0xM2tOPnMwM7MCJwczMytwcjAzswInBzMzK3ByMDOzAicHMzMrcHIwM7MCJwczMytwcjAzswInBzMzK3ByMDOzAicHMzMrcHIwq0LSNkkbJW2QtC7FRknqkrQlPY9McUm6RVK3pCcknZPbzpxUfoukObn4pLT97rSuhr+WZn1zcjDr23siYmJETE7vFwCrI2I8sDq9B7iIbL708cA84DbIkgmwEDgPOBdYWE4oqcwVufWmDX11zOpXzxzSSyTtkvRkLvYlSdvTr6oNkqbnll2bfg09I+nCXHxainVLWpCLnynp0RS/W9IJg1lBs0E0A1iaXi8FLs7Fl0VmDTBC0mjgQqArIvZExF6gC5iWlp0SEWsiIoBluW2ZNYV65nO4A/gXsgM472sR8dV8QNJZwEzgbcDpwI8lvSUtvhW4AOgB1kpaERGbga+kbS2X9HVgLumXl1kDBfAjSQF8IyIWAx0RsSMtfwHoSK/HAM/n1u1JsVrxnirxAknzyM5G6OjooFQqFcr09vYyf8LBw2LVyrWa3t7etqhHpVapV7/JISIeljSuzu3NAJZHxMvALyR1k51OA3RHxFYAScuBGZKeAt4LfCyVWQp8CScHa7x3RcR2Sa8HuiQ9nV8YEZESx5BKSWkxwOTJk6Ozs7NQplQqceMj+w+Lbbu0WK7VlEolqtW31bVKvQYyE9zVkmYD64D56bR5DLAmVyb/i6jyF9R5wKnAixFxoEr5gmq/ourJwvMnHKi5HBr7S6tVfknU0g51yIuI7el5l6T7yX7k7JQ0OiJ2pKahXan4duCM3OpjU2w70FkRL6X42CrlzZrG0SaH24DryU69rwduBD4xWDvVl2q/ourJwpdVmRa0UiN/abXKL4la2qEOZZJOBl4REb9Or6cC1wErgDnAovT8QFplBdmPpeVkP3peSglkFfAPuYvQU4FrI2KPpH2SpgCPArOBfx6u+pnV46iSQ0TsLL+W9E3g++ltX7+g6CO+m+zi3fHp7MG/oKwZdAD3p96lxwPfiYgfSloL3CNpLvAccEkqvxKYDnQDvwEuB0hJ4HpgbSp3XUTsSa+vJLuedxLwYHqYNY2jSg7lU+v09kNAuSfTCuA7km4iuyA9HngMEDBe0plk//xnAh9L7bYPAR8GlnP4rzGzhkjXxt5eJb4bOL9KPICr+tjWEmBJlfg64OwB76zZEOk3OUi6i6zd9DRJPWT9tjslTSRrVtoGfBIgIjZJugfYDBwAroqIg2k7VwOrgOOAJRGxKX3E54Hlkr4MPA7cPmi1MzOzo1JPb6VZVcJ9/gOPiBuAG6rEV5KdflfGt3KoR5OZmTUB3yFtZmYFTg5mZlbg5GBmZgVODmZmVuDkYGZmBU4OZmZWMJCxldrKuIohNrYten+D9sTMrPF85mBmZgVODmZmVuDkYGZmBU4OZmZW4ORgZmYFTg5mZlbg5GBmZgVODmZmVuDkYGZmBU4OZmZW4ORgZmYF/SYHSUsk7ZL0ZC72j5KelvSEpPsljUjxcZJ+K2lDenw9t84kSRsldUu6RZJSfJSkLklb0vPIoaiomZnVr54zhzuAaRWxLuDsiPhL4D+Aa3PLno2IienxqVz8NuAKYHx6lLe5AFgdEeOB1em9mZk1UL/JISIeBvZUxH4UEQfS2zXA2FrbkDQaOCUi1kREAMuAi9PiGcDS9HppLm5mZg0yGEN2fwK4O/f+TEmPA/uAL0bET4ExQE+uTE+KAXRExI70+gWgo68PkjQPmAfQ0dFBqVSit7eXUqlUcwfnTzhQc3k1/W1zMNVTh2bXDnWoJOk4YB2wPSI+IOlMYDlwKrAe+HhE/F7SiWQ/eCYBu4GPRsS2tI1rgbnAQeBvI2JVik8DbgaOA74VEYuGtXJm/RhQcpD0d8AB4M4U2gG8MSJ2S5oEfE/S2+rdXkSEpKixfDGwGGDy5MnR2dlJqVSis7Oz5nYvq5iroR7bLq29zcFUTx2aXTvUoYprgKeAU9L7rwBfi4jl6XraXLLm0rnA3oh4s6SZqdxHJZ0FzATeBpwO/FjSW9K2bgUuIPuhtFbSiojYPFwVM+vPUfdWknQZ8AHg0tRURES8HBG70+v1wLPAW4DtHN70NDbFAHamZqdy89Ouo90ns8EiaSzwfuBb6b2A9wL3piL5JtB80+i9wPmp/Axgefpe/ALoBs5Nj+6I2BoRvyc7G5kx9LUyq99RJYd0Svw54IMR8Ztc/HXpVBxJbyK78Lw1NRvtkzQlfWlmAw+k1VYAc9LrObm4WSP9E9kx/sf0/lTgxdy1tnzT6BjgeYC0/KVU/k/xinX6ips1jX6blSTdBXQCp0nqARaS9U46EehKPVLXpJ5J7wauk/QHsi/VpyKifDH7SrKeTycBD6YHwCLgHklzgeeASwalZmZHSdIHgF0RsV5SZ4P3pXCdrVJvby/zJxw8LNYO13/a8ToWtE69+k0OETGrSvj2PsreB9zXx7J1wNlV4ruB8/vbD7Nh9E7gg5KmA68iu+ZwMzBC0vHp7CDfNLodOAPokXQ88FqyC9PleFl+nb7ih6l2na1SqVTixkf2HxYbzmtmQ6VNr2O1TL18h7RZhYi4NiLGRsQ4sgvKP4mIS4GHgA+nYvkm0HzT6IdT+UjxmZJOTD2dxgOPAWuB8ZLOlHRC+owVw1A1s7oNRldWs2PF54Hlkr4MPM6hM+jbgW9L6ia7J2gmQERsknQPsJmsV99VEXEQQNLVwCqyrqxLImLTsNbErB9ODmY1REQJKKXXW8l6GlWW+R3wkT7WvwG4oUp8JbByEHfVbFC5WcnMzAqcHMzMrMDJwczMCpwczMyswMnBzMwKnBzMzKzAycHMzAqcHMzMrMDJwczMCpwczMyswMnBzMwKnBzMzKzAycHMzAqcHMzMrMDJwczMCupKDpKWSNol6clcbJSkLklb0vPIFJekWyR1S3pC0jm5deak8lskzcnFJ0namNa5RWliajMza4x6zxzuAKZVxBYAqyNiPLA6vQe4iGw6xPFkE6PfBlkyARYC55FNmLKwnFBSmSty61V+1rAbt+AHhz3MzI4ldSWHiHiYbPrDvBnA0vR6KXBxLr4sMmvIJmUfDVwIdEXEnojYC3QB09KyUyJiTZp3d1luW2Zm1gADmSa0IyJ2pNcvAB3p9Rjg+Vy5nhSrFe+pEi+QNI/sbISOjg5KpRK9vb2USqWaOzp/woE6qlNbf58xEPXUodm1Qx3M7JBBmUM6IkJSDMa2+vmcxcBigMmTJ0dnZyelUonOzs6a6102CM1C2y6t/RkDUU8dml071MHMDhlIb6WdqUmI9LwrxbcDZ+TKjU2xWvGxVeJmZtYgA0kOK4Byj6M5wAO5+OzUa2kK8FJqfloFTJU0Ml2IngqsSsv2SZqSeinNzm3LzMwaoK5mJUl3AZ3AaZJ6yHodLQLukTQXeA64JBVfCUwHuoHfAJcDRMQeSdcDa1O56yKifJH7SrIeUScBD6aHmZk1SF3JISJm9bHo/CplA7iqj+0sAZZUia8Dzq5nX8zMbOj5DmmzKiS9StJjkn4uaZOkv0/xMyU9mm7YvFvSCSl+YnrfnZaPy23r2hR/RtKFufi0FOuWtKByH8waycnBrLqXgfdGxNuBiWT35EwBvgJ8LSLeDOwF5qbyc4G9Kf61VA5JZwEzgbeR3dz5vyQdJ+k44Faym0bPAmalsmZNwcnBrIp0E2dvevvK9AjgvcC9KV5582f5ptB7gfNTB4sZwPKIeDkifkF2Le7c9OiOiK0R8XtgeSpr1hQG5T4Hs3aUft2vB95M9iv/WeDFiCjfVZm/YfNPN3lGxAFJLwGnpvia3Gbz61TeFHpelX0o3PhZqbe3l/kTDh4Wa4cbEtv1xspWqZeTg1kfIuIgMFHSCOB+4C8asA+FGz8rlUolbnxk/2Gxobxpc7i0642VrVIvNyuZ9SMiXgQeAv6KbKyw8o+q/A2bf7rJMy1/LbCbI78p1KwptN2Zg0dQtcEg6XXAHyLiRUknAReQXWR+CPgw2TWCyps/5wD/npb/JA0rswL4jqSbgNPJRh1+DBAwXtKZZElhJvCx4aqfWX/aLjmYDZLRwNJ03eEVwD0R8X1Jm4Hlkr4MPA7cnsrfDnxbUjfZCMYzASJik6R7gM3AAeCq1FyFpKvJRg44DlgSEZuGr3pmtTk5mFUREU8A76gS30rW06gy/jvgI31s6wbghirxlWQjCpg1HV9zMDOzAicHMzMrcHIwM7MCJwczMytwcjAzswInBzMzK3ByMDOzAicHMzMrcHIwM7OCo04Okt4qaUPusU/SZyR9SdL2XHx6bh3PiGVm1gKOeviMiHiGbIas8rj328mGNb6cbKasr+bLV8yIdTrwY0lvSYtvJRvYrAdYK2lFRGw+2n0zM7OBGayxlc4Hno2I57LJr6r604xYwC/SAGXlMWq605g1SCrPiOXkYGbWIIOVHGYCd+XeXy1pNrAOmB8RexngjFhQfVasylmV5k84UG3VARvKmZtaZWaoWtqhDmZ2yICTg6QTgA8C16bQbcD1ZPPtXg/cCHxioJ8D1WfFqpxV6bIhms9hKGfWapWZoWpphzqY2SGDceZwEfCziNgJUH4GkPRN4Pvpba2ZrzwjlplZExmMrqyzyDUpSRqdW/Yh4Mn0egUwU9KJafar8oxYa0kzYqWzkJmprJmZNciAzhwknUzWy+iTufD/lDSRrFlpW3mZZ8QyM2sdA0oOEbEfOLUi9vEa5Vt2Rqxqc1NvW/T+BuyJmdnQ8x3SZmZW4ORgZmYFTg5mZlbg5GBmZgVODmZmVuDkYGZmBU4OZhUknSHpIUmbJW2SdE2Kj5LUJWlLeh6Z4pJ0Sxpy/glJ5+S2NSeV3yJpTi4+SdLGtM4tqjFipVkjODmYFR0gGzDyLGAKcFUacn4BsDoixgOr03vIhpAZnx7zyMYXQ9IoYCHZQJLnAgvLCSWVuSK33rRhqJdZ3ZwczCpExI6I+Fl6/WvgKbIRhGcAS1OxpcDF6fUMYFlk1gAj0jAyFwJdEbEnjUzcBUxLy06JiDUREcCy3LbMmsJgDdlt1pYkjQPeATwKdETEjrToBaAjvR5Dcdj5Mf3Ee6rEq31+YZj6Sr29vcyfcPCwWDsMn96uw8C3Sr2cHMz6IOnVwH3AZyJiX/6yQESEpBjqfag2TH2lUqnEjY/sPyw2lEPMD5d2HQa+VerlZiWzKiS9kiwx3BkR303hneVRh9PzrhTvazj6WvGxVeJmTcPJwaxC6jl0O/BURNyUW7QCKPc4mgM8kIvPTr2WpgAvpeanVcBUSSPTheipwKq0bJ+kKemzZue2ZdYU3KxkVvRO4OPARkkbUuwLwCLgHklzgeeAS9KylcB0oBv4DXA5QETskXQ92ZwlANdFxJ70+krgDuAk4MH0MGsaTg5mFSLiEaCv+w7Or1I+gKv62NYSYEmV+Drg7AHsptmQcrOSmZkV+MzBrA1VTk7liansSPnMwczMCgacHCRtS2PEbJC0LsUGbQwaMzMbfoN15vCeiJgYEZPT+8Ecg8bMzIbZUDUrDcoYNEO0b2Zm1o/BuCAdwI/SUALfSLf7D9YYNIepNs5M5Tgl8yccGIQq1WewxkdplbFWammHOpjZIYORHN4VEdslvR7okvR0fuFgjkFTbZyZynFKLqvopTGUBmv8mlYZa6WWdqiDmR0y4GaliNienncB95NdMxisMWjMzKwBBpQcJJ0s6TXl12RjxzzJII1BM5B9MzOzozfQZqUO4P40lPHxwHci4oeS1jJ4Y9CYmdkwG1ByiIitwNurxHczSGPQmJnZ8PPwGQPgIQrMrF15+AwzMytwcjAzswInBzMzK3ByMDOzAicHMzMrcHIwM7MCJwczMytwcjAzswInBzMzK3ByMDOzAicHMzMrcHIwq0LSEkm7JD2Zi42S1CVpS3oemeKSdIukbklPSDont86cVH6LpDm5+CRJG9M6tygNbWzWLJwczKq7g+I85guA1RExHlid3gNcBIxPj3nAbZAlE2AhcB7ZJFgLywkllbkit57nTLem4uRgVkVEPAxUzikyA1iaXi8FLs7Fl0VmDTAizYB4IdAVEXsiYi/QBUxLy06JiDVpGPtluW2ZNQUP2W1Wv440cyHAC2STXQGMAZ7PletJsVrxnirxAknzyM5G6OjooFQqFcr09vYyf8LBmjtebb1m19vb25L73Z9WqZeTwyCqnN8BPMdDu4qIkBTD8DmLgcUAkydPjs7OzkKZUqnEjY/sr7mdbZcW12t2pVKJavVtda1SLzcrmdVvZ2oSIj3vSvHtwBm5cmNTrFZ8bJW4WdM46uQg6QxJD0naLGmTpGtS/EuStkvakB7Tc+tcm3pnPCPpwlx8Wop1S1pQ7fPMmsAKoNzjaA7wQC4+O/VamgK8lJqfVgFTJY1MF6KnAqvSsn2SpqReSrNz2zJrCgNpVjoAzI+In0l6DbBeUlda9rWI+Gq+sKSzgJnA24DTgR9LektafCtwAVnb61pJKyJi8wD2zWxAJN0FdAKnSeoh63W0CLhH0lzgOeCSVHwlMB3oBn4DXA4QEXskXQ+sTeWui4jyRe4ryXpEnQQ8mB5mTeOok0P69bMjvf61pKfo46JaMgNYHhEvA7+Q1E3WvQ+gOyK2Akhanso6OVjDRMSsPhadX6VsAFf1sZ0lwJIq8XXA2QPZR7OhNCgXpCWNA94BPAq8E7ha0mxgHdnZxV6yxLEmt1q+h0Zlj47z+vicQs+Nyiv/8yccGHiFBlE9vRJapfdCLe1QBzM7ZMDJQdKrgfuAz0TEPkm3AdcDkZ5vBD4x0M+B6j03Kq/8X1alx1Aj1dNLpFV6L9TSDnUws0MGlBwkvZIsMdwZEd8FiIidueXfBL6f3vbVc4MacTMza4CB9FYScDvwVETclIuPzhX7EFAem2YFMFPSiZLOJBsy4DGyi3XjJZ0p6QSyi9Yrjna/zMxs4AZy5vBO4OPARkkbUuwLwCxJE8malbYBnwSIiE2S7iG70HwAuCoiDgJIupqs299xwJKI2DSA/TIzswEaSG+lR4BqI0murLHODcANVeIra63XyirvmvYd02bWCnyHtJmZFTg5mJlZgZODmZkVODmYmVmBk4OZmRU4OZiZWYEn+zE7BngiKjtSPnMwM7MCnzk0gY3bXzpswED/ojOzRvOZg5mZFTg5mJlZgZODmZkVODmYmVmBL0g3IY/kamaN5jMHMzMrcHIwM7MCNyu1AN/dakPBzZdWi88czMysoGnOHCRNA24mm0f6WxGxqMG71NT8q6/1+Zi3ZtYUyUHSccCtwAVAD7BW0oqI2NzYPWsdbnpqLc14zPsYsrymSA7AuUB3RGwFkLQcmAE4OQxAtS/7kfI/hyHTEsd8PceQj5H21CzJYQzwfO59D3BeZSFJ84B56W2vpGeA04BfDfkeDhJ9pWq4aevQx/5W07R1qOHPGvjZAznmKzX0b38Ex8iRasVjqh6Nrlddx32zJIe6RMRiYHE+JmldRExu0C4NCtfB+lLtmK/Urn9716uxmqW30nbgjNz7sSlm1q58zFtTa5bksBYYL+lMSScAM4EVDd4ns6HkY96aWlM0K0XEAUlXA6vIuvUtiYhNda5e85S7RbgOx5gBHvOV2vVv73o1kCKi0ftw05ypAAACzElEQVRgZmZNplmalczMrIk4OZiZWUHLJgdJ0yQ9I6lb0oJG7089JC2RtEvSk7nYKEldkrak55GN3Mf+SDpD0kOSNkvaJOmaFG+perSLVvwelEnaJmmjpA2S1qVY1eNImVtSPZ+QdE5j9/6QI/le16qHpDmp/BZJcxpRl7yWTA65oQcuAs4CZkk6q7F7VZc7gGkVsQXA6ogYD6xO75vZAWB+RJwFTAGuSn/7VqtHy2vh70HeeyJiYq7ff1/H0UXA+PSYB9w27Hvatzuo/3tdtR6SRgELyW6EPBdY2OgfWC2ZHMgNPRARvwfKQw80tYh4GNhTEZ4BLE2vlwIXD+tOHaGI2BERP0uvfw08RXa3b0vVo0205PegH30dRzOAZZFZA4yQNLoRO1jpCL/XfdXjQqArIvZExF6gi2LCGVatmhyqDT0wpkH7MlAdEbEjvX4B6GjkzhwJSeOAdwCP0sL1aGGt/j0I4EeS1qdhQqDv46jV6nqk9Wi6+jXFfQ6WiYiQ1BJ9iyW9GrgP+ExE7JP0p2WtVA9rqHdFxHZJrwe6JD2dX9gux1Gr1qNVzxzaaeiBneXT4/S8q8H70y9JryRLDHdGxHdTuOXq0QZa+nsQEdvT8y7gfrJmsr6Oo1ar65HWo+nq16rJoZ2GHlgBlHsmzAEeaOC+9EvZKcLtwFMRcVNuUUvVo0207PdA0smSXlN+DUwFnqTv42gFMDv19pkCvJRrtmlGR1qPVcBUSSPTheipKdY4EdGSD2A68B/As8DfNXp/6tznu4AdwB/I2hTnAqeS9WbYAvwYGNXo/eynDu8iayt+AtiQHtNbrR7t8mjF70Ha7zcBP0+PTeV97+s4AkTWM+tZYCMwudF1yNWl7u91rXoAnwC60+PyRtfLw2eYmVlBqzYrmZnZEHJyMDOzAicHMzMrcHIwM7MCJwczMytwcjAzswInBzMzK/j/GVUGM5tv3DMAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QwdSGIhGMEbz",
        "colab_type": "text"
      },
      "source": [
        "Interesting. We can fix the maximum length of the summary to 8 since that seems to be the majority summary length.\n",
        "\n",
        "Let us understand the proportion of the length of summaries below 8"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "7JRjwdIOFxg3",
        "colab_type": "code",
        "colab": {},
        "outputId": "f968be82-c539-471d-ce23-16f18b059ea0"
      },
      "source": [
        "cnt=0\n",
        "for i in data['cleaned_summary']:\n",
        "    if(len(i.split())<=8):\n",
        "        cnt=cnt+1\n",
        "print(cnt/len(data['cleaned_summary']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.9424907471335922\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYB4Ga9KMjEu",
        "colab_type": "text"
      },
      "source": [
        "We observe that 94% of the summaries have length below 8. So, we can fix maximum length of summary to 8.\n",
        "\n",
        "Let us fix the maximum length of review to 30"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "ZKD5VOWqFxhC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "max_text_len=30\n",
        "max_summary_len=8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6d48E-8M4VO",
        "colab_type": "text"
      },
      "source": [
        "Let us select the reviews and summaries whose length falls below or equal to **max_text_len** and **max_summary_len**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "yY0tEJP0FxhI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cleaned_text =np.array(data['cleaned_text'])\n",
        "cleaned_summary=np.array(data['cleaned_summary'])\n",
        "\n",
        "short_text=[]\n",
        "short_summary=[]\n",
        "\n",
        "for i in range(len(cleaned_text)):\n",
        "    if(len(cleaned_summary[i].split())<=max_summary_len and len(cleaned_text[i].split())<=max_text_len):\n",
        "        short_text.append(cleaned_text[i])\n",
        "        short_summary.append(cleaned_summary[i])\n",
        "        \n",
        "df=pd.DataFrame({'text':short_text,'summary':short_summary})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tR1uh8xSNUma",
        "colab_type": "text"
      },
      "source": [
        "Remember to add the **START** and **END** special tokens at the beginning and end of the summary. Here, I have chosen **sostok** and **eostok** as START and END tokens\n",
        "\n",
        "**Note:** Be sure that the chosen special tokens never appear in the summary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "EwLUH78CFxhg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df['summary'] = df['summary'].apply(lambda x : 'sostok '+ x + ' eostok')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1GlcX4RFOh13",
        "colab_type": "text"
      },
      "source": [
        "We are getting closer to the model building part. Before that, we need to split our dataset into a training and validation set. We’ll use 90% of the dataset as the training data and evaluate the performance on the remaining 10% (holdout set):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "RakakKHcFxhl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "x_tr,x_val,y_tr,y_val=train_test_split(np.array(df['text']),np.array(df['summary']),test_size=0.1,random_state=0,shuffle=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vq1mqyOHOtIl",
        "colab_type": "text"
      },
      "source": [
        "#Preparing the Tokenizer\n",
        "\n",
        "A tokenizer builds the vocabulary and converts a word sequence to an integer sequence. Go ahead and build tokenizers for text and summary:\n",
        "\n",
        "#Text Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "oRHTgX6hFxhq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer \n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "#prepare a tokenizer for reviews on training data\n",
        "x_tokenizer = Tokenizer() \n",
        "x_tokenizer.fit_on_texts(list(x_tr))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzvLwYL_PDcx",
        "colab_type": "text"
      },
      "source": [
        "#Rarewords and its Coverage\n",
        "\n",
        "Let us look at the proportion rare words and its total coverage in the entire text\n",
        "\n",
        "Here, I am defining the threshold to be 4 which means word whose count is below 4 is considered as a rare word"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "y8KronV2Fxhx",
        "colab_type": "code",
        "colab": {},
        "outputId": "d2eb2f27-fbbc-4e61-9556-3c3ff5e4327b"
      },
      "source": [
        "thresh=4\n",
        "\n",
        "cnt=0\n",
        "tot_cnt=0\n",
        "freq=0\n",
        "tot_freq=0\n",
        "\n",
        "for key,value in x_tokenizer.word_counts.items():\n",
        "    tot_cnt=tot_cnt+1\n",
        "    tot_freq=tot_freq+value\n",
        "    if(value<thresh):\n",
        "        cnt=cnt+1\n",
        "        freq=freq+value\n",
        "    \n",
        "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
        "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "% of rare words in vocabulary: 66.12339930151339\n",
            "Total Coverage of rare words: 2.953684513790566\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "So-J-5kzQIeO",
        "colab_type": "text"
      },
      "source": [
        "**Remember**:\n",
        "\n",
        "\n",
        "* **tot_cnt** gives the size of vocabulary (which means every unique words in the text)\n",
        " \n",
        "*   **cnt** gives me the no. of rare words whose count falls below threshold\n",
        "\n",
        "*  **tot_cnt - cnt** gives me the top most common words \n",
        "\n",
        "Let us define the tokenizer with top most common words for reviews."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "J2giEsF3Fxh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "x_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
        "x_tokenizer.fit_on_texts(list(x_tr))\n",
        "\n",
        "#convert text sequences into integer sequences\n",
        "x_tr_seq    =   x_tokenizer.texts_to_sequences(x_tr) \n",
        "x_val_seq   =   x_tokenizer.texts_to_sequences(x_val)\n",
        "\n",
        "#padding zero upto maximum length\n",
        "x_tr    =   pad_sequences(x_tr_seq,  maxlen=max_text_len, padding='post')\n",
        "x_val   =   pad_sequences(x_val_seq, maxlen=max_text_len, padding='post')\n",
        "\n",
        "#size of vocabulary ( +1 for padding token)\n",
        "x_voc   =  x_tokenizer.num_words + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "DCbGMsm4FxiA",
        "colab_type": "code",
        "colab": {},
        "outputId": "2d9165f0-e542-4114-91f3-e070d483fce9"
      },
      "source": [
        "x_voc"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8440"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQfKP3sqRxi9",
        "colab_type": "text"
      },
      "source": [
        "#Summary Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "eRHqyBkBFxiJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "y_tokenizer = Tokenizer()   \n",
        "y_tokenizer.fit_on_texts(list(y_tr))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KInA6O6ZSkJz",
        "colab_type": "text"
      },
      "source": [
        "#Rarewords and its Coverage\n",
        "\n",
        "Let us look at the proportion rare words and its total coverage in the entire summary\n",
        "\n",
        "Here, I am defining the threshold to be 6 which means word whose count is below 6 is considered as a rare word"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "yzE5OiRLFxiM",
        "colab_type": "code",
        "colab": {},
        "outputId": "7f7a4f89-b088-4847-8172-09e5a2383d0e"
      },
      "source": [
        "thresh=6\n",
        "\n",
        "cnt=0\n",
        "tot_cnt=0\n",
        "freq=0\n",
        "tot_freq=0\n",
        "\n",
        "for key,value in y_tokenizer.word_counts.items():\n",
        "    tot_cnt=tot_cnt+1\n",
        "    tot_freq=tot_freq+value\n",
        "    if(value<thresh):\n",
        "        cnt=cnt+1\n",
        "        freq=freq+value\n",
        "    \n",
        "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
        "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "% of rare words in vocabulary: 78.12740675541863\n",
            "Total Coverage of rare words: 5.3921899389571895\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PBhzKuRSw_9",
        "colab_type": "text"
      },
      "source": [
        "Let us define the tokenizer with top most common words for summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "-fswLvIgFxiR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "y_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
        "y_tokenizer.fit_on_texts(list(y_tr))\n",
        "\n",
        "#convert text sequences into integer sequences\n",
        "y_tr_seq    =   y_tokenizer.texts_to_sequences(y_tr) \n",
        "y_val_seq   =   y_tokenizer.texts_to_sequences(y_val) \n",
        "\n",
        "#padding zero upto maximum length\n",
        "y_tr    =   pad_sequences(y_tr_seq, maxlen=max_summary_len, padding='post')\n",
        "y_val   =   pad_sequences(y_val_seq, maxlen=max_summary_len, padding='post')\n",
        "\n",
        "#size of vocabulary\n",
        "y_voc  =   y_tokenizer.num_words +1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqwDUT5oTFmn",
        "colab_type": "text"
      },
      "source": [
        "Let us check whether word count of start token is equal to length of the training data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "pR8IX9FRFxiY",
        "colab_type": "code",
        "colab": {},
        "outputId": "b116cdbd-42c4-4ede-9f6d-46284115393e"
      },
      "source": [
        "y_tokenizer.word_counts['sostok'],len(y_tr)   "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42453, 42453)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LVFhFVguTTtw",
        "colab_type": "text"
      },
      "source": [
        "Here, I am deleting the rows that contain only **START** and **END** tokens"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "kZ-vW82sFxih",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ind=[]\n",
        "for i in range(len(y_tr)):\n",
        "    cnt=0\n",
        "    for j in y_tr[i]:\n",
        "        if j!=0:\n",
        "            cnt=cnt+1\n",
        "    if(cnt==2):\n",
        "        ind.append(i)\n",
        "\n",
        "y_tr=np.delete(y_tr,ind, axis=0)\n",
        "x_tr=np.delete(x_tr,ind, axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "cx5NISuMFxik",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ind=[]\n",
        "for i in range(len(y_val)):\n",
        "    cnt=0\n",
        "    for j in y_val[i]:\n",
        "        if j!=0:\n",
        "            cnt=cnt+1\n",
        "    if(cnt==2):\n",
        "        ind.append(i)\n",
        "\n",
        "y_val=np.delete(y_val,ind, axis=0)\n",
        "x_val=np.delete(x_val,ind, axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOtlDcthFxip",
        "colab_type": "text"
      },
      "source": [
        "# Model building\n",
        "\n",
        "We are finally at the model building part. But before we do that, we need to familiarize ourselves with a few terms which are required prior to building the model.\n",
        "\n",
        "**Return Sequences = True**: When the return sequences parameter is set to True, LSTM produces the hidden state and cell state for every timestep\n",
        "\n",
        "**Return State = True**: When return state = True, LSTM produces the hidden state and cell state of the last timestep only\n",
        "\n",
        "**Initial State**: This is used to initialize the internal states of the LSTM for the first timestep\n",
        "\n",
        "**Stacked LSTM**: Stacked LSTM has multiple layers of LSTM stacked on top of each other. \n",
        "This leads to a better representation of the sequence. I encourage you to experiment with the multiple layers of the LSTM stacked on top of each other (it’s a great way to learn this)\n",
        "\n",
        "Here, we are building a 3 stacked LSTM for the encoder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "zXef38nBFxir",
        "colab_type": "code",
        "colab": {},
        "outputId": "7ae99521-46f8-4c6f-9cba-4979deffeee8"
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "\n",
        "latent_dim = 300\n",
        "embedding_dim=100\n",
        "\n",
        "# Encoder\n",
        "encoder_inputs = Input(shape=(max_text_len,))\n",
        "\n",
        "#embedding layer\n",
        "enc_emb =  Embedding(x_voc, embedding_dim,trainable=True)(encoder_inputs)\n",
        "\n",
        "#encoder lstm 1\n",
        "encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
        "\n",
        "#encoder lstm 2\n",
        "encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
        "\n",
        "#encoder lstm 3\n",
        "encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)\n",
        "\n",
        "# Set up the decoder, using `encoder_states` as initial state.\n",
        "decoder_inputs = Input(shape=(None,))\n",
        "\n",
        "#embedding layer\n",
        "dec_emb_layer = Embedding(y_voc, embedding_dim,trainable=True)\n",
        "dec_emb = dec_emb_layer(decoder_inputs)\n",
        "\n",
        "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,dropout=0.4,recurrent_dropout=0.2)\n",
        "decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])\n",
        "\n",
        "# Attention layer\n",
        "attn_layer = AttentionLayer(name='attention_layer')\n",
        "attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])\n",
        "\n",
        "# Concat attention input and decoder LSTM output\n",
        "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
        "\n",
        "#dense layer\n",
        "decoder_dense =  TimeDistributed(Dense(y_voc, activation='softmax'))\n",
        "decoder_outputs = decoder_dense(decoder_concat_input)\n",
        "\n",
        "# Define the model \n",
        "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
        "\n",
        "model.summary() "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/backend.py:4010: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            (None, 30)           0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding (Embedding)           (None, 30, 100)      844000      input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm (LSTM)                     [(None, 30, 300), (N 481200      embedding[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "input_2 (InputLayer)            (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "lstm_1 (LSTM)                   [(None, 30, 300), (N 721200      lstm[0][0]                       \n",
            "__________________________________________________________________________________________________\n",
            "embedding_1 (Embedding)         (None, None, 100)    198900      input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm_2 (LSTM)                   [(None, 30, 300), (N 721200      lstm_1[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "lstm_3 (LSTM)                   [(None, None, 300),  481200      embedding_1[0][0]                \n",
            "                                                                 lstm_2[0][1]                     \n",
            "                                                                 lstm_2[0][2]                     \n",
            "__________________________________________________________________________________________________\n",
            "attention_layer (AttentionLayer [(None, None, 300),  180300      lstm_2[0][0]                     \n",
            "                                                                 lstm_3[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "concat_layer (Concatenate)      (None, None, 600)    0           lstm_3[0][0]                     \n",
            "                                                                 attention_layer[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "time_distributed (TimeDistribut (None, None, 1989)   1195389     concat_layer[0][0]               \n",
            "==================================================================================================\n",
            "Total params: 4,823,389\n",
            "Trainable params: 4,823,389\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZVlfRuMUcoP",
        "colab_type": "text"
      },
      "source": [
        "I am using sparse categorical cross-entropy as the loss function since it converts the integer sequence to a one-hot vector on the fly. This overcomes any memory issues."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Lwfi1Fm8Fxiz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p0ykDbxfUhyw",
        "colab_type": "text"
      },
      "source": [
        "Remember the concept of early stopping? It is used to stop training the neural network at the right time by monitoring a user-specified metric. Here, I am monitoring the validation loss (val_loss). Our model will stop training once the validation loss increases:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s-A3J92MUljB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mw6CVECaUq5b",
        "colab_type": "text"
      },
      "source": [
        "We’ll train the model on a batch size of 128 and validate it on the holdout set (which is 10% of our dataset):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "ETnPzA4OFxi3",
        "colab_type": "code",
        "colab": {},
        "outputId": "477e374f-7cf2-4d60-f86e-2c49c9cebedb"
      },
      "source": [
        "history=model.fit([x_tr,y_tr[:,:-1]], y_tr.reshape(y_tr.shape[0],y_tr.shape[1], 1)[:,1:] ,epochs=50,callbacks=[es],batch_size=128, validation_data=([x_val,y_val[:,:-1]], y_val.reshape(y_val.shape[0],y_val.shape[1], 1)[:,1:]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 41346 samples, validate on 4588 samples\n",
            "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "Epoch 1/50\n",
            "41346/41346 [==============================] - 85s 2ms/sample - loss: 2.8152 - val_loss: 2.5780\n",
            "Epoch 2/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 2.4859 - val_loss: 2.4072\n",
            "Epoch 3/50\n",
            "41346/41346 [==============================] - 81s 2ms/sample - loss: 2.3259 - val_loss: 2.3232\n",
            "Epoch 4/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 2.2281 - val_loss: 2.2534\n",
            "Epoch 5/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 2.1604 - val_loss: 2.1862\n",
            "Epoch 6/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 2.1065 - val_loss: 2.1549\n",
            "Epoch 7/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 2.0616 - val_loss: 2.1177\n",
            "Epoch 8/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 2.0202 - val_loss: 2.0992\n",
            "Epoch 9/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 1.9835 - val_loss: 2.0822\n",
            "Epoch 10/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.9476 - val_loss: 2.0636\n",
            "Epoch 11/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.9145 - val_loss: 2.0606\n",
            "Epoch 12/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 1.8826 - val_loss: 2.0672\n",
            "Epoch 13/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 1.8553 - val_loss: 2.0444\n",
            "Epoch 14/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.8267 - val_loss: 2.0422\n",
            "Epoch 15/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.7980 - val_loss: 2.0456\n",
            "Epoch 16/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 1.7745 - val_loss: 2.0409\n",
            "Epoch 17/50\n",
            "41346/41346 [==============================] - 79s 2ms/sample - loss: 1.7518 - val_loss: 2.0374\n",
            "Epoch 18/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.7299 - val_loss: 2.0434\n",
            "Epoch 19/50\n",
            "41346/41346 [==============================] - 80s 2ms/sample - loss: 1.7070 - val_loss: 2.0398\n",
            "Epoch 00019: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ezKYOp2UxG5",
        "colab_type": "text"
      },
      "source": [
        "#Understanding the Diagnostic plot\n",
        "\n",
        "Now, we will plot a few diagnostic plots to understand the behavior of the model over time:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "tDTNLAURFxjE",
        "colab_type": "code",
        "colab": {},
        "outputId": "e2ea6e44-3931-4014-97a1-03fa2a441228"
      },
      "source": [
        "from matplotlib import pyplot\n",
        "pyplot.plot(history.history['loss'], label='train')\n",
        "pyplot.plot(history.history['val_loss'], label='test')\n",
        "pyplot.legend()\n",
        "pyplot.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VFX+//HXSU9ISEglIaTQQoIQSmiCGkSKoKhfFdfeVlZd96H7df3qupZ11993i7t+d3VXWQu2dW3YEBvFICg1IBFIIZBCQkI6qaSf3x93AiGkTGAyLZ/n4zGPmcw9M/PJMLzn5Nxzz1Vaa4QQQjgXF1sXIIQQwvIk3IUQwglJuAshhBOScBdCCCck4S6EEE5Iwl0IIZyQhLsQQjghCXchhHBCEu5CCOGE3Gz1wsHBwTomJsZWLy+EEA5p9+7d5VrrkL7a2SzcY2JiSE1NtdXLCyGEQ1JK5ZvTToZlhBDCCUm4CyGEE5JwF0IIJ2SzMXchhDgbLS0tFBYW0tjYaOtSBpSXlxeRkZG4u7uf1eMl3IUQDqWwsBA/Pz9iYmJQStm6nAGhtaaiooLCwkJiY2PP6jlkWEYI4VAaGxsJCgpy2mAHUEoRFBR0Tn+dSLgLIRyOMwd7h3P9HfsMd6XUSKVUilIqXSl1QCl1fzdt/JVSnyml0kxtbj+nqnpxsKSW369Np6m1baBeQgghHJ45PfdW4EGtdQIwC/i5UiqhS5ufA+la60QgGfirUsrDopWaFFY18Op3uezMrRyIpxdCiF4dP36cF154od+PW7JkCcePHx+AirrXZ7hrrYu11ntMt2uBDGBE12aAnzL+jvAFKjG+FCxu9qhgPNxcSMksG4inF0KIXvUU7q2tvUfeF198QUBAwECVdYZ+jbkrpWKAKcCOLpv+AcQDRcA+4H6tdbsF6juDt4crs0YFselg6UA8vRBC9OqRRx7h8OHDTJ48menTp3PBBRewbNkyEhKMAY0rr7ySadOmMWHCBF566aWTj4uJiaG8vJy8vDzi4+O56667mDBhAgsXLuTEiRMWr9PsqZBKKV/gQ+ABrXVNl82LgL3AxcBoYL1SakvXdkqpFcAKgKioqLMuel5cCE99lk5+RT3RQUPO+nmEEI7tqc8OkF7UNY7OTULEUJ68fEKP2//4xz+yf/9+9u7dy6ZNm1i6dCn79+8/OWVx1apVBAYGcuLECaZPn87VV19NUFDQac+RnZ3NO++8w8svv8zy5cv58MMPuemmmyz6e5jVc1dKuWME+9ta64+6aXI78JE2HAJygfFdG2mtX9JaJ2mtk0JC+lzUrEfz4kIB2JQlQzNCCNuaMWPGaXPRn3vuORITE5k1axYFBQVkZ2ef8ZjY2FgmT54MwLRp08jLy7N4XX323E3j6K8CGVrrZ3todgSYD2xRSoUBcUCOxarsIiZ4CLHBQ9iUVcqt58cM1MsIIexcbz1saxky5NTowaZNm9iwYQPbtm3Dx8eH5OTkbueqe3p6nrzt6upqs2GZOcDNwD6l1F7TfY8CUQBa65XA74HXlVL7AAU8rLUut3i1nVw0LoR3dh6hsaUNL3fXgXwpIYQ4yc/Pj9ra2m63VVdXM2zYMHx8fMjMzGT79u1Wru6UPsNda/0dRmD31qYIWGiposwxb3wor2/NY1tOxclhGiGEGGhBQUHMmTOH8847D29vb8LCwk5uW7x4MStXriQ+Pp64uDhmzZplszoddm2ZmbGBeLm78G1WmYS7EMKq/vOf/3R7v6enJ19++WW32zrG1YODg9m/f//J+3/1q19ZvD5w4OUHvNxdOX90MN9klqK1tnU5QghhVxw23MGYEnmksoHc8npblyKEEHbFocM9WaZECiFEtxw63EcG+jA6ZAgpWXK0qhBCdObQ4Q7GAU07cippaB6QpWyEEMIhOX64jw+lua2dbYcrbF2KEELYDYcP96SYYfh4uMrQjBDCKs52yV+Av/3tbzQ0NFi4ou45fLh7urkyZ0wwKZllMiVSCDHgHCXcHfYgps7mxYWyPr2EQ6V1jA3zs3U5Qggn1nnJ3wULFhAaGsr7779PU1MTV111FU899RT19fUsX76cwsJC2traePzxxykpKaGoqIh58+YRHBxMSkrKgNbpFOGeHGesMLkpq0zCXYjB5MtH4Ng+yz7n8Ilw6R973Nx5yd9169axevVqdu7cidaaZcuWsXnzZsrKyoiIiODzzz8HjDVn/P39efbZZ0lJSSE4ONiyNXfD4YdlACICvIkL85NxdyGEVa1bt45169YxZcoUpk6dSmZmJtnZ2UycOJH169fz8MMPs2XLFvz9/a1em1P03AGSx4ew6rtc6ppa8fV0ml9LCNGbXnrY1qC15te//jU/+9nPzti2Z88evvjiCx577DHmz5/PE088YdXanKLnDsa4e0ub5vtDA7rSsBBikOu85O+iRYtYtWoVdXV1ABw9epTS0lKKiorw8fHhpptu4qGHHmLPnj1nPHagOU0Xd1r0MPw83diUVcqiCcNtXY4Qwkl1XvL30ksv5YYbbmD27NkA+Pr68u9//5tDhw7x0EMP4eLigru7Oy+++CIAK1asYPHixURERAz4DlVlq+mDSUlJOjU11aLPec+/d/PDkeNs+/XFGCeQEkI4m4yMDOLj421dhlV097sqpXZrrZP6eqzTDMuAMTRzrKaRrBLr/NkjhBD2yqnC/SLTlMiUTFklUggxuDlVuIcN9SIhfKhMiRTCyQ2Go9HP9Xd0qnAHmDc+hN35VdQ0tti6FCHEAPDy8qKiosKpA15rTUVFBV5eXmf9HE4zW6ZDclwo/0w5zHfZ5SyZGG7rcoQQFhYZGUlhYSFlZc49/Orl5UVkZORZP97pwn3KyACGermRklkq4S6EE3J3dyc2NtbWZdg9pxuWcXN14cJxIWw6KKtECiEGL6cLdzCGZspqmzhQVGPrUoQQwiacMtwvGtexSqTMmhFCDE5OGe4hfp5MivRnU5Zz73ARQoieOGW4gzE0s+dIFccbmm1dihBCWF2f4a6UGqmUSlFKpSulDiil7u+hXbJSaq+pzbeWL7V/kuNCaNewOVtWiRRCDD7m9NxbgQe11gnALODnSqmEzg2UUgHAC8AyrfUE4FqLV9pPiZEBDPNxZ1OmjLsLIQafPsNda12std5jul0LZAAjujS7AfhIa33E1M7mierqorhoXAjfHiyjvV2mRAohBpd+jbkrpWKAKcCOLpvGAcOUUpuUUruVUrdYprxzkxwXSkV9M/uOVtu6FCGEsCqzw10p5Qt8CDygte46gdwNmAYsBRYBjyulxnXzHCuUUqlKqVRrHDp84bgQlEIWEhNCDDpmhbtSyh0j2N/WWn/UTZNC4Gutdb3WuhzYDCR2baS1fklrnaS1TgoJCTmXus0SOMSDySMDZEqkEGLQMWe2jAJeBTK01s/20OxTYK5Syk0p5QPMxBibt7z2dsjdbHbz5HGhpBUep6KuaUDKEUIIe2ROz30OcDNwsWmq416l1BKl1N1KqbsBtNYZwFfAj8BO4BWt9f4BqfiHt+CNyyF3i1nN540PQWvYnC29dyHE4NHnqpBa6++APk9IqrV+BnjGEkX1atJy+PZPsO4xuCsFXHr/fjovwp9gXw82ZZVx1ZSzXz5TCCEcieMdoeruDfOfgOK9sH91n81dXBQXmqZEtsmUSCHEIOF44Q4wcTmEJ8LG30HLiT6bz4sL5XhDC3sLjluhOCGEsD3HDHcXF1j4NFQXwI6VfTa/cGwILgq+lSmRQohBwjHDHSD2Qhh3KWx5Fup7Xz/G38edadHDSJEpkUKIQcJxwx1gwVPQXG/sYO1Dclwo+45WU1rbaIXChBDCthw73EPiYNptkLoKyrN7bZocZxw0tfmgrBIphHB+jh3uAMm/Bjdv2PDbXpslhA8l1M9TliIQQgwKjh/uviEw9wHIXAt53/fYTClFclwIWw6W0drWbsUChRDC+hw/3AFm3QtDRxgHNrX3HNzz4kKpaWzlB5kSKYRwcs4R7h4+cPHjULQHDnS3rplhzthg3FwUKXICDyGEk3OOcAeYdB0MnwgbnoKW7mfEDPWSKZFCiMHBecLdxQUW/j+oPgI7/9Vjs3njQ8koruFYtUyJFEI4L+cJd4BRF8HYRbD5r1Bf0W2TeXGhAHx7UIZmhBDOy7nCHWDB76C5Fjb/udvN48J8Cff3IiVThmaEEM7L+cI9dDxMvRV2vQLlh87YbEyJDOW7Q+U0trTZoEAhhBh4zhfuYDqwyQs2PNnt5qunjqCuqZWV3x62cmFCCGEdzhnufmEwx3RgU/7WMzYnxQRyeWIEL246TEFlgw0KFEKIgeWc4Q4w++fgFw5f/6bbA5seXTIeVxfF79em26A4IYQYWM4b7n0c2BTu780vLh7LuvQSvj0oO1eFEM7FecMdIPEnENbzgU13zI0hNngIT605QHOrrDcjhHAezh3uLq6w6GnTgU0vnbHZ082VJy9PIKe8nlXf59qgQCGEGBjOHe4Ao5Jh7ELY/BdoqDxjc3JcKAsSwnh+Y7YctSqEcBrOH+5w6sCmb7s/sOnxpQm0tGv+8GWGlQsTQoiBMTjCPTQept4Cu16GijPntkcF+XD3RaP5dG8RO3K6X7ZACCEcyeAId4DkR8HVs8czNt1z0WhGBHjz5JoDcjIPIYTDGzzh7hdmnLEpYw0c2X7GZm8PVx6/LJ7MY7W8veOIDQoUQgjLGTzhDqcf2KT1GZsXTRjOBWOD+eu6LCrqmmxQoBBCWEaf4a6UGqmUSlFKpSulDiil7u+l7XSlVKtS6hrLlmkhHkPg4sfgaCoc+PiMzUopnrx8Ag3NbTzzdZYNChRCCMswp+feCjyotU4AZgE/V0oldG2klHIF/gSss2yJFpZ4PYSdZywqVl9+xuYxob7cMTeW91IL2CvnWhVCOKg+w11rXay13mO6XQtkACO6afoL4EPAvs+C4eIKS/4CdaXw6gKozDmjyS8uHkOIrydPfrqf9vYzh2+EEMLe9WvMXSkVA0wBdnS5fwRwFfBiH49foZRKVUqllpXZcD2X6Nlw62dw4ji8sgAKd5+22c/LnV8vGU9aYTWrdxfaqEghhDh7Zoe7UsoXo2f+gNa6psvmvwEPa617nUOotX5Ja52ktU4KCQnpf7WWNHIG3LneGId/fSlkfXXa5isnj2B6zDD+9FUm1Q0tNipSCCHOjlnhrpRyxwj2t7XWZy6xCEnAu0qpPOAa4AWl1JUWq3KgBI+Bn26AkDh493pIfe3kJqUUv102gaqGZv5vw0EbFimEEP1nzmwZBbwKZGitn+2ujdY6Vmsdo7WOAVYD92qtP7FopQPFNxRu+xxGz4e1D8A3T5+cJjkhwp8bZ0bz5rY8Moq7/rEihBD2y5ye+xzgZuBipdRe02WJUupupdTdA1yfdXj6wvXvwpSbYfMz8Mm90GYMxTy4cBz+3u48ueYAupu58UIIYY/c+mqgtf4OUOY+odb6tnMpyGZc3WDZ8+AfCZv+AHXHYPmbBPj48dCi8Tz68T7WpBVxxeTuJgoJIYR9GVxHqPZFKUh+BJb9A3K+hdeWQO0xrps+kokj/PnfLzKob2q1dZVCCNEnCffuTL0ZbnjPWEHylQW4VmTz1BUTKKlp4vlvDtm6OiGE6JOEe0/GLoDb1kLrCVi1kKlkcc20SF79LofDZXW2rk4IIXol4d6bEVONufDegfDGMh4ffQgvN1ee+ixddq4KIeyahHtfAmONgA9PxH/NnbwUt5vNB8tYn15i68qEEKJHEu7mGBIEt66B8UuZffBPPDP0A37/2X4aW9psXZkQQnRLwt1c7t6w/E2YfhfXNn/MQ/V/4eUUOeeqEMI+Sbj3h4srLHkGLvkty1y3MfO7n7I1Ld3WVQkhxBkk3PtLKZj7SxouW8kklxzGfbSYA1vOPPGHEELYkoT7WfJJup7G2zdS6xrAhI23Ubz64ZNLFgghhK1JuJ+DgOiJ+N63mc/cFxG+fyX1KxdAVb6tyxJCCAn3cxUSGMD0+97kt56/or0sk7YX50L6p7YuSwgxyEm4W8Bwfy/uuudX3OHxLBnNIfD+LbD2v6Gl0dalCSEGKQl3CxkR4M1ffnYFd7v/L2+qZZD6KrwyH8rkRB9CCOuTcLeg6KAhvLFiLs+53sr9rr+hrboIXroIfnj75AlAhBDCGiTcLWx0iC//uWsmW5jCf+lnaAybAp/eCx+tgKZaW5cnhBgkJNwHwLgwP966cwa5TX4sqvgltbP/B/avhn9dCEV7bV2eEGIQkHAfIBMi/HnrzplUNLSzbN8cqq792NjB+uoC2L5ShmmEEANKwn0AJY4M4I07plNS08jyrxSVt3xjnIj7q4fh3RugodLWJQohnJSE+wCbFh3IqtumU1DVwI3/OcTxK96AxX+E7PWwci7kb7V1iUIIJyThbgWzRgXx8i1JHC6r45bXdlEz+afw0/Xg5gmvL4VvnpalC4QQFiXhbiUXjA3hxRunklFcw22rdlIXNBF+thkSr4fNz8CqRcY5W4UQwgIk3K1ofnwYz18/hbTCau58fRcnlA9c+QJc+zpUHIKVF8Cet2RnqxDinEm4W9ni88L5v+smsyuvkrveTDXO5jThKrhnq3HO1jX3GcsXyM5WIcQ5kHC3gWWJEfz5mkS+O1TOXW+mUlnfDP6RcMsaWPA7yPoSXjwfDqfYulQhhIOScLeRa6ZF8udrJrEjp5JFf9vMluwycHGBOffDXRvB0w/euhK+/g20Ntm6XCGEg+kz3JVSI5VSKUqpdKXUAaXU/d20uVEp9aNSap9SaqtSKnFgynUuy5NG8ul9cwjwdufmV3fy9Np0mlrbIDwRVnwL038K2/4BL8+HUjlfqxDCfOb03FuBB7XWCcAs4OdKqYQubXKBi7TWE4HfAy9ZtkznFR8+lM9+MZdbZkfzyne5XPnPrWSX1IKHDyz9K1z/HtQWw0vJsOMl2dkqhDBLn+GutS7WWu8x3a4FMoARXdps1VpXmX7cDkRaulBn5uXuyu+uOI9Xb02itKaRy57/jre256O1hrjFcO82iLkAvnwI/rMc6kptXbIQws71a8xdKRUDTAF29NLsTuDLsy9p8JofH8aXD1zArFFBPP7Jfu56M5WKuibwDYUbP4BLn4HczfDCbMj6ytblCiHsmNnhrpTyBT4EHtBa1/TQZh5GuD/cw/YVSqlUpVRqWVnZ2dTr9EL9vHjttuk8cVkCmw+Ws/jvW9h8sAyUgpkrYMUm8BsO71xnnO2pucHWJQsh7JDSZozhKqXcgbXA11rrZ3toMwn4GLhUa93n6YeSkpJ0ampqP8sdXDKKa7j/3R84WFLHHXNi+Z/FcXi5uxqzZzb+ztjZGjwOLv87RM02vgCEEE5NKbVba53UZ7u+wl0ppYA3gEqt9QM9tIkCvgFu0VqbtRKWhLt5Glva+MMXGbyxLZ/xw/14/vopjA3zMzYeToFP7jF2uPpHQcIy44CoEdMk6IVwUpYM97nAFmAf0G66+1EgCkBrvVIp9QpwNZBv2t7a14tLuPfPN5klPPTBj9Q1tfLY0nhumhWNUgoaqyHjM0j/1Aj79hbwHwkJVxiXEUnG/HkhhFOwWLgPFAn3/iurbeKh1Wlsyipj/vhQ/nTNJIJ9PU81OFFlHN164BM4/I0R9ENHmIL+SoicLkEvhIOTcHdSWmte35rHH77MZKiXO39dnshF40LObHjiOBz8yhT0G6GtGfwijKGbhCth5EwJeiEckIS7k8s8VsP97+wlq6SWm2dF8+DCcQT4eHTfuLEaDn5tBP2hDdDWBH7hEL8MJnQEvat1fwEhxFmRcB8EGlva+PNXWby+NRc/L3d+eclYbpwVjbtrLz3yxhrIXgcHPjaCvrURfMPgvKth6i0QGm+9X0AI0W8S7oNI1rFanv48nS3Z5YwOGcJjSxNIjgsxdrj2pqnW1KP/2Lhub4HIGUbIT7gKPH2t8wsIIcwm4T7IaK1JySrl6bUZ5JTXc+G4EB5bGs+4jmmTfakvh7R3Yc8bUH4QPPxgoqk3HzFVplYKYSck3AeplrZ23tqWz982HKS+uY0bZkTxywXjCBzSw3h8V1pDwQ7Y/YbRo289AWETjZCfdC14DxvYX0AI0SsJ90Guqr6Zv2/M5q3t+fh4uHL//LHcMjsGD7d+zJBprIZ9q43efHEauHkZ0yqn3grR50tvXggbkHAXABwqreXpzzPYlFVGbPAQHl0SzyXxoX2Px3dVtBf2vAn7PoCmGggaY/TmE683FjYTQliFhLs4zaasUp7+PINDpXXMGRPEY0sTiA8f2v8nam6A9E+MoD+yDVzcIG6J0ZsflQyubpYuXQjRiYS7OENLWzvv7DzCs+sPUnOiheumR/HgwnGnH+XaH2VZRsinvQMNFeDmDcPPM84k1XEJiQc3M8f7hRB9knAXPapuaOG5b7J5Y2se3u6u3HfxGG6bE4On21keyNTaZEylPLLdGJs/9qMxdAPg4g5hCafCfngihE0wzjQlhOg3CXfRp5yyOv73iww2ZJQS7u/FXReM4iczRuLjcY5DK+3tUJVrBH3ny4lKY7tygeC403v4wyeC11kMEwkxyEi4C7N9f6ic5zZmsyO3kmE+7tw+J5ZbZ8fg7+NuuRfRGqoLzwz8umOn2gSONoZ1ws4zevehCRAQLWvgCNGJhLvot935lbyQcpiNmaUM8XDlplnR3Dk3ltChXgP3orUlxjBO8V5jRk7JAaPX38HD1wj5sAmnLqEJ4B0wcDUJYcck3MVZyzxWw4ubDvNZWhFuri5cMy2Suy8cTVSQlcbJm+qgLBNK9hthX5Ju3G48fqqN/8hTQR82wejtB42R2TrC6Um4i3OWX1HPvzbnsDq1kNb2di5PjOCe5NGMH26DsXGtoabICPvSA6bQP2AsldDearRx9YSQccZO2/BEiJhs2nk7xPr1CjFAJNyFxZTWNPLqd7n8e3s+9c1tzB8fyr3zRjMtOtDWpUFrsxHwJQdO9fSL06Ch3NiuXIzzzIYnQvhk2XkrHJ6Eu7C46oYW3tiWx2vf51LV0MLM2EDunTeGC8cG9/+I14HU0cs/beftXuNcsx0CRxs9+5OzdSaBjx18WXVoazGmmMrKnKILCXcxYBqaW3lnZwGvbMmhuLqR80YM5Z6LxrD4vOG4uthRyHfVeedtcRoUpUH1kVPbA6JPBb13gLGWjpsXuHmaee11+sweraG5zjj94YnjpusqY99Bx+0z7jf93FxnPEfwOGMdn6jzIXo2BERZ9z0TdkfCXQy45tZ2PvnhKCu/PUxOeT3RQT7cdn4M1yaNxNfTQXZsNlSe3rsvToPKnLN/Phd3I+Rd3Yz18jv2B/TU1ntYp0vAqdteAcbCbIW7jIPDOg4K8x9pCvvZED0HgsfKAm6DjIS7sJq2ds3XB46x6rtcUvOr8PN049qkkdx2foz1ZthYUnMDNNcbZ6lqbep0faKb+7q5bmk0TnziOfTMwO4c5O4+5gVze5uxLyF/KxzZCvnboL7U2OYTbPToo843Qn/4RDllopOTcBc2kVZwnNe+z2Xtj8W0ac2C+DDumBvLzNhA+xqXd2RaQ8VhU9CbLsfzjW0efjByhhH00ecbJ1pxH8DjFLrW1dYMLaYvwc7XaOOYBQ9fY/aSh69MWz1LEu7CpkpqGnlrWz5v78inqqGFhPCh3DE3lssTw89+DRvRs+qjxiqdHWFflnFqm3IBVw/j4uJ26rare5frjttd7lcupwd1d+HdOcTN5eZ1Kuw9/U6Ffk8/u/uAu3eni4/xHF3vd/M276jmjn0ijdXGvo7G6m4uXe83HWtx2mt7dfrZu5v7Ov3cUd/QCBga3q9/4pP/nBLuwh40trTxyQ9Hee37PLJKagn29eDGmdHcNCuaEL+zXI1S9K2h0gj7kgPGcFF7izEDp63ZdGnpdN3D/e2m2+1tplDqFFRu3l2uvTq16eYaZQRpc71x3VRn+tl0X7c/1xrXuq3/v7+r55mB7+5t7APpHNi6vffn8fA1htO8/E2Xocbv0nri1BfdyS+3BmNIrqWBPr/k5twPC37X/98LCXdhZ7TWbD1cwarvctmYWYqHqwuXJ0Zw+5wYzhvhb+vyhL3S2vhy6vhS6PzXQ0tDl59PdArdTkF7sk0DKFdjf8dpgW26eHe+L8DYZ3I2Q0cnh6caev4CCIiC0PFn9ZZIuAu7lVNWxxtb8/hgdyENzW3MjA3k9jmxLEgIs++plELYAQl3YfeqT7Tw/q4CXt+ax9HjJxgZ6M0NM6K5etoIQv2stBNQCAdjsXBXSo0E3gTCMAaSXtJa/71LGwX8HVgCNAC3aa339Pa8Eu6iQ2tbO+vTS3htax47cytxdVHMiwvluukjmRcXgpurLPkrRAdzw92cAaVW4EGt9R6llB+wWym1Xmud3qnNpcBY02Um8KLpWog+ubm6cOnEcC6dGM7hsjreTy3gw91H2ZBRQoifJ1dPjWR5UiSjQuRQfCHM1e9hGaXUp8A/tNbrO933L2CT1vod089ZQLLWuriHp5Geu+hVS1s7KZmlvJ9aSEpWKW3tmhkxgSyfPpIlE4ef+9mihHBQluy5d37SGGAKsKPLphFAQaefC0339RjuQvTG3dWFhROGs3DCcEprGvlwz1HeTy3gVx+k8ds1B7g8MYLrpo8kMdJfDo4Sohtmh7tSyhf4EHhAa11zNi+mlFoBrACIipIFkIR5Qod6cU/yaO6+aBS78qp4b1cBn/xwlHd2HiEuzI/l00dy1ZQRBA7xsHWpQtgNs4ZllFLuwFrga631s91sl2EZYVW1jS18llbMe6kFpBUcx91VsTBhONcmRXLB2BCZUimclsWGZUwzYV4FMroLdpM1wH1KqXcxdqRW9xbsQpwrPy93bpgZxQ0zo8g6Vst7uwr4+IdCPt9XTLi/F1dPjeSaaZHEBMtZmMTgZM5UyLnAFmAf0HGs7qNAFIDWeqXpC+AfwGKMqZC3a6177ZZLz11YWlNrGxvSS/lgdwGbD5bRrmFGTCDXJEWydGI4QxxlGWIheiEHMYlB7Vh1Ix/9UMjq1EJyyuvx8XBlycRwrp0WyQxZoVI4MAl3ITDWtNlzpIoPUgv5LK2I+uY2ooN8uGbs0LwPAAANzElEQVRqJFdPiyQiwNvWJQrRLxLuQnTR0NzKl/uO8cHuArbnVKIUzB0TzDXTIlk0YThe7rIUsbB/Eu5C9OJIRQOr9xTy4e5Cjh4/wVAvN5ZNjuDaaSOZJHPnhR2TcBfCDO3tmm05FXyQWsCX+4/R1NrOuDBfrpwygssnRTAy0AFPEyicmoS7EP1U09jC2rRiVu8uYM8R44w7U6MCuGLyCJZMDJeTiwi7IOEuxDkoqGxgTVoRn6UVkXmsFhcFc8YEsywxgkXnDWeol7utSxSDlIS7EBZysKSWNXuL+DTtKAWVJ/Bwc2FeXAjLEkcwPz5UdsQKq5JwF8LCtNbsLTjOmrQi1v5YTFltE76ebixMCOPyyRHMHROMu6w9LwaYhLsQA6itXbM9p4I1e4v4cn8xNY2tBA7xYMnE4SxLHEFS9DBcZH0bMQAk3IWwkqbWNr7NKmNNWhEbMkpobGknwt+LpZPCWTopQpYlFhYl4S6EDdQ3tbI+vYTP0orYnF1GS5tmRIA3l00KZ+mkcCaOkKAX50bCXQgbqz7Rwvr0Ej7/sYgt2eW0tmuiAn2MHv3EcCZEDJWgF/0m4S6EHTne0My69BI+/7GY7w8ZQR8T1BH0EcSH+0nQC7NIuAthp6rqm1mXfoy1Pxaz9XAFbe2aUcFDTGP04cSFSdCLnkm4C+EAKuub+frAMdb+WMS2wxW0axgdMoSlkyJYOjGccWG+EvTiNBLuQjiY8romvtp/jM9/LGZHrhH0kcO8mRcXSnJcCLNHB+HjISccGewk3IVwYKW1jaxPL2FTVhnfHyqnobkNDzcXZo0KYl5cCPPiQuUUgoOUhLsQTqKptY1duVWkZJWyKauUw2X1AMQE+ZAcF8q88aHMjA2UZRAGCQl3IZzUkYoGNh0sJSWzlK2HK2hqbcfL3YU5o4NJjgshOS5Ulip2YhLuQgwCjS1tbMup4NusMr7JLOVIZQMAY0J9Tw7fTI8NlDVvnIiEuxCDjNaa3PJ6UrLK2JRVyo6cSprb2vHzcuOicSFcEh9GclwIAT4eti5VnAMJdyEGufqmVr4/VM7GjFI2ZpZQXteMq4siKXoYl8SHcUlCGLGyU9bhSLgLIU5qb9ekFR5nY0YpGzJKyDxWC8CokCFG0MeHMTUqADcZvrF7Eu5CiB4VVDbwTaYR9NtzKmhp0wT4uDMvLpRL4sO4cFwwfnK2Kbsk4S6EMEttYwtbssvZkF5CSlYpVQ0tuLsqZsYGMT/eCHuZfWM/JNyFEP3W2tbOniPH2ZhRwoaMkpNz6scP92N+fCjz48OYHBkgJyKxIQl3IcQ5yy2vZ0O6EfSp+VW0tWuCfT25eLwx+2bu2GBZEsHKLBbuSqlVwGVAqdb6vG62+wP/BqIAN+AvWuvX+nphCXchHMvxhmY2ZZWxIaOEb7PKqG1qxcPNhTmjg7gkIYz548MY7u9l6zKdniXD/UKgDnizh3B/FPDXWj+slAoBsoDhWuvm3p5Xwl0Ix9Xc2s6uvEo2mIZvCipPADBxhP/JcXo5GcnAMDfc+/x7Smu9WSkV01sTwE8Z/4q+QCXQamadQggH5OHmwpwxwcwZE8wTlyWQXVrH+vQSNmaU8PeN2fxtQzbh/l4nx+lnjwqStW+szKwxd1O4r+2h5+4HrAHGA37AdVrrz3t4nhXACoCoqKhp+fn5Z124EMI+ldc1GdMs00vYkl3OiZY2PN1cmBo1jJmjApk1KojJIwMk7M+SRXeo9hHu1wBzgP8GRgPrgUStdU1vzynDMkI4v8aWNrYdrmBLdjk7citIL65Ba6PnP2VkADNHBTFrVCBTo4ZJ2JvJYsMyZrgd+KM2viUOKaVyMXrxOy3w3EIIB+bl7sq88cayxADVDS3syqtke04FO3Ir+cc32Ty3ETxcXZg8MuBkz35q1DC8PSTsz4Ulwv0IMB/YopQKA+KAHAs8rxDCyfj7uHNJgrGuDUBNYwupeZVsz6lkR04F/0w5xPPfHMLdVZEYeSrsp0UPkymX/WTObJl3gGQgGCgBngTcAbTWK5VSEcDrQDigMHrx/+7rhWVYRgjRVW1jC6n5VezIMXr3+45W09aucXNRTIz0Z0ZMINNNF3+fwbk8ghzEJIRweHVNrezOr2JHTgU7cytJKzxOS5tGKYgL82NGrBH0M2IDCRs6OObYS7gLIZxOY0sbewuOsyu3kp15lezOr6KhuQ2A6CAfI+hjApkeG0hMkI9TzrO35g5VIYSwCi93V2aNCmLWqCDAWAsnvbiGnbmV7MytZGNGCat3FwIQ4ufJDFOvfnpMIHHD/XAdRGviSM9dCOE02ts1h8vq2JlXafTucyspqm4EwM/LjaToYSTFBDItehiJkQEOOSNHeu5CiEHHxUUxNsyPsWF+3DgzGoDCqgZ25layK6+SXXlVpGRlAeDmopgwwt8I/OhhTIsZRqif84zbS89dCDGoVNU3s+dIFan5VezOqyKt8DhNre0ARAX6nAz66TGBjAnxtbvljWWHqhBCmKG5tZ39RdXszqsiNd/YSVteZ6x7ONTLjWl2NpQjwzJCCGEGD9O6N1OjhnEXo9Bak1/RYPTs8ytJ7TKU0zHffkZsIEnR9jvfXnruQgjRh+MNxlDOrrwqUvMqSSuoprmt/eR8+5mxgcyIDWJ67MCP28uwjBBCDJDe5tvHBg85Odd+ZmwgkcO8LTrfXoZlhBBigHSdb9/S1k56kTHffkduJV+nH+O91AIAwv29Th5FOzM2kDGhvlY5uEp67kIIYWHt7Zrs0jp25lawM89YPqG0tgmAYT7u3Js8hrsuHHVWzy09dyGEsBEXF0XccD/ihvtx8+wYtNYcqWw4eSRtmBXONSvhLoQQA0wpRXTQEKKDhnBt0kirvKaLVV5FCCGEVUm4CyGEE5JwF0IIJyThLoQQTkjCXQghnJCEuxBCOCEJdyGEcEIS7kII4YRstvyAUqoMyD/LhwcD5RYsZyA5Sq1Sp+U5Sq1Sp2UNdJ3RWuuQvhrZLNzPhVIq1Zy1FeyBo9QqdVqeo9QqdVqWvdQpwzJCCOGEJNyFEMIJOWq4v2TrAvrBUWqVOi3PUWqVOi3LLup0yDF3IYQQvXPUnrsQQohe2HW4K6UWK6WylFKHlFKPdLPdUyn1nmn7DqVUjA1qHKmUSlFKpSulDiil7u+mTbJSqloptdd0ecLadXaqJU8ptc9UxxmnwlKG50zv6Y9Kqak2qDGu03u1VylVo5R6oEsbm72nSqlVSqlSpdT+TvcFKqXWK6WyTdfDenjsraY22UqpW21Q5zNKqUzTv+3HSqmAHh7b6+fECnX+Vil1tNO/75IeHttrRlihzvc61ZinlNrbw2Ot9n6epLW2ywvgChwGRgEeQBqQ0KXNvcBK0+2fAO/ZoM5wYKrpth9wsJs6k4G1tn5PTbXkAcG9bF8CfAkoYBawww4+B8cw5vbaxXsKXAhMBfZ3uu/PwCOm248Af+rmcYFAjul6mOn2MCvXuRBwM93+U3d1mvM5sUKdvwV+ZcZno9eMGOg6u2z/K/CErd/Pjos999xnAIe01jla62bgXeCKLm2uAN4w3V4NzFfWOPNsJ1rrYq31HtPtWiADGGHNGizsCuBNbdgOBCilwm1Yz3zgsNb6bA94szit9WagssvdnT+LbwBXdvPQRcB6rXWl1roKWA8stmadWut1WutW04/bgciBen1z9fB+msOcjLCY3uo05c5y4J2Bev3+sudwHwEUdPq5kDND82Qb0we2GgiySnXdMA0LTQF2dLN5tlIqTSn1pVJqglULO50G1imldiulVnSz3Zz33Zp+Qs//YezlPQUI01oXm24fA8K6aWNv7+0dGH+ldaevz4k13GcaPlrVwzCXPb2fFwAlWuvsHrZb/f2053B3KEopX+BD4AGtdU2XzXswhhUSgeeBT6xdXydztdZTgUuBnyulLrRhLb1SSnkAy4APutlsT+/pabTxd7hdT0NTSv0GaAXe7qGJrT8nLwKjgclAMcaQhz27nt577VZ/P+053I8Cnc8kG2m6r9s2Sik3wB+osEp1nSil3DGC/W2t9Uddt2uta7TWdabbXwDuSqlgK5fZUctR03Up8DHGn7admfO+W8ulwB6tdUnXDfb0npqUdAxfma5Lu2ljF++tUuo24DLgRtMX0RnM+JwMKK11ida6TWvdDrzcw+vby/vpBvwX8F5PbWzxftpzuO8CxiqlYk09uJ8Aa7q0WQN0zDi4Bvimpw/rQDGNtb0KZGitn+2hzfCOfQFKqRkY77stvoSGKKX8Om5j7Fzb36XZGuAW06yZWUB1p+EGa+uxN2Qv72knnT+LtwKfdtPma2ChUmqYaZhhoek+q1FKLQb+B1imtW7ooY05n5MB1WU/z1U9vL45GWENlwCZWuvC7jba7P205t7b/l4wZm4cxNgj/hvTfb/D+GACeGH8yX4I2AmMskGNczH+BP8R2Gu6LAHuBu42tbkPOICxN387cL6N3s9RphrSTPV0vKeda1XAP03v+T4gyUa1DsEIa/9O99nFe4rxhVMMtGCM896Jsa9nI5ANbAACTW2TgFc6PfYO0+f1EHC7Deo8hDFO3fFZ7ZhtFgF80dvnxMp1vmX6/P2IEdjhXes0/XxGRlizTtP9r3d8Lju1tdn72XGRI1SFEMIJ2fOwjBBCiLMk4S6EEE5Iwl0IIZyQhLsQQjghCXchhHBCEu5CCOGEJNyFEMIJSbgLIYQT+v9RbSbbr55Y9gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HSyx-HvpUz2o",
        "colab_type": "text"
      },
      "source": [
        "From the plot, we can infer that validation loss has increased after epoch 17 for 2 successive epochs. Hence, training is stopped at epoch 19.\n",
        "\n",
        "Next, let’s build the dictionary to convert the index to word for target and source vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "sBX0zZnOFxjW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reverse_target_word_index=y_tokenizer.index_word\n",
        "reverse_source_word_index=x_tokenizer.index_word\n",
        "target_word_index=y_tokenizer.word_index"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eM_nU_VvFxjq",
        "colab_type": "text"
      },
      "source": [
        "# Inference\n",
        "\n",
        "Set up the inference for the encoder and decoder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "9QkrNV-4Fxjt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Encode the input sequence to get the feature vector\n",
        "encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])\n",
        "\n",
        "# Decoder setup\n",
        "# Below tensors will hold the states of the previous time step\n",
        "decoder_state_input_h = Input(shape=(latent_dim,))\n",
        "decoder_state_input_c = Input(shape=(latent_dim,))\n",
        "decoder_hidden_state_input = Input(shape=(max_text_len,latent_dim))\n",
        "\n",
        "# Get the embeddings of the decoder sequence\n",
        "dec_emb2= dec_emb_layer(decoder_inputs) \n",
        "# To predict the next word in the sequence, set the initial states to the states from the previous time step\n",
        "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])\n",
        "\n",
        "#attention inference\n",
        "attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])\n",
        "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
        "\n",
        "# A dense softmax layer to generate prob dist. over the target vocabulary\n",
        "decoder_outputs2 = decoder_dense(decoder_inf_concat) \n",
        "\n",
        "# Final decoder model\n",
        "decoder_model = Model(\n",
        "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
        "    [decoder_outputs2] + [state_h2, state_c2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zOiyk4ToWe74",
        "colab_type": "text"
      },
      "source": [
        "We are defining a function below which is the implementation of the inference process (which we covered [here](https://www.analyticsvidhya.com/blog/2019/06/comprehensive-guide-text-summarization-using-deep-learning-python/)):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "6f6TTFnBFxj6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def decode_sequence(input_seq):\n",
        "    # Encode the input as state vectors.\n",
        "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
        "    \n",
        "    # Generate empty target sequence of length 1.\n",
        "    target_seq = np.zeros((1,1))\n",
        "    \n",
        "    # Populate the first word of target sequence with the start word.\n",
        "    target_seq[0, 0] = target_word_index['sostok']\n",
        "\n",
        "    stop_condition = False\n",
        "    decoded_sentence = ''\n",
        "    while not stop_condition:\n",
        "      \n",
        "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
        "\n",
        "        # Sample a token\n",
        "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
        "        sampled_token = reverse_target_word_index[sampled_token_index]\n",
        "        \n",
        "        if(sampled_token!='eostok'):\n",
        "            decoded_sentence += ' '+sampled_token\n",
        "\n",
        "        # Exit condition: either hit max length or find stop word.\n",
        "        if (sampled_token == 'eostok'  or len(decoded_sentence.split()) >= (max_summary_len-1)):\n",
        "            stop_condition = True\n",
        "\n",
        "        # Update the target sequence (of length 1).\n",
        "        target_seq = np.zeros((1,1))\n",
        "        target_seq[0, 0] = sampled_token_index\n",
        "\n",
        "        # Update internal states\n",
        "        e_h, e_c = h, c\n",
        "\n",
        "    return decoded_sentence"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6GuDf4TPWt6_",
        "colab_type": "text"
      },
      "source": [
        "Let us define the functions to convert an integer sequence to a word sequence for summary as well as the reviews:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "aAUntznIFxj9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def seq2summary(input_seq):\n",
        "    newString=''\n",
        "    for i in input_seq:\n",
        "        if((i!=0 and i!=target_word_index['sostok']) and i!=target_word_index['eostok']):\n",
        "            newString=newString+reverse_target_word_index[i]+' '\n",
        "    return newString\n",
        "\n",
        "def seq2text(input_seq):\n",
        "    newString=''\n",
        "    for i in input_seq:\n",
        "        if(i!=0):\n",
        "            newString=newString+reverse_source_word_index[i]+' '\n",
        "    return newString"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gM4ALyfWwA9",
        "colab_type": "text"
      },
      "source": [
        "Here are a few summaries generated by the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "BUtQmQTmFxkI",
        "colab_type": "code",
        "colab": {},
        "outputId": "f407d9fc-e0cd-4082-98f5-bd1f562dc26f"
      },
      "source": [
        "for i in range(0,100):\n",
        "    print(\"Review:\",seq2text(x_tr[i]))\n",
        "    print(\"Original summary:\",seq2summary(y_tr[i]))\n",
        "    print(\"Predicted summary:\",decode_sequence(x_tr[i].reshape(1,max_text_len)))\n",
        "    print(\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Review: gave caffeine shakes heart anxiety attack plus tastes unbelievably bad stick coffee tea soda thanks \n",
            "Original summary: hour \n",
            "Predicted summary:  not worth the money\n",
            "\n",
            "\n",
            "Review: got great course good belgian chocolates better \n",
            "Original summary: would like to give it stars but \n",
            "Predicted summary:  good\n",
            "\n",
            "\n",
            "Review: one best flavored coffees tried usually like flavored coffees one great serve company love \n",
            "Original summary: delicious \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: salt separate area pain makes hard regulate salt putting like salt go ahead get product \n",
            "Original summary: tastes ok packaging \n",
            "Predicted summary:  salt\n",
            "\n",
            "\n",
            "Review: really like product super easy order online delivered much cheaper buying gas station stocking good long drives \n",
            "Original summary: turkey jerky is great \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: best salad dressing delivered promptly quantities last vidalia onion dressing compares made oak hill farms sometimes find costco order front door want even orders cut shipping costs \n",
            "Original summary: my favorite salad dressing \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: think sitting around warehouse long time took long time send got tea tasted like cardboard red rasberry leaf tea know supposed taste like \n",
            "Original summary: stale \n",
            "Predicted summary:  not as good as\n",
            "\n",
            "\n",
            "Review: year old cat special diet digestive problems also diabetes stopped eating usual special formula food tried different kinds catfood one liked easy digestion diabetes thank newman \n",
            "Original summary: wonderful \n",
            "Predicted summary:  cat food\n",
            "\n",
            "\n",
            "Review: always perfect snack dog loves knows exactly starts ask time evening gets greenie snack thank excellent product fast delivery \n",
            "Original summary: greenies buddy treat \n",
            "Predicted summary:  great treat\n",
            "\n",
            "\n",
            "Review: dog loves tiny treats keep one car one house \n",
            "Original summary: dog loves them \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: liked coffee much subscribing dark rich smooth \n",
            "Original summary: makes great cup of java \n",
            "Predicted summary:  good coffee\n",
            "\n",
            "\n",
            "Review: far dog tried chicken peanut butter flavor absolutely loves love natural makes happy giving dog something healthy treats small soft big plus calories \n",
            "Original summary: love zuke mini naturals \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: absolutely delicious satisfy something sweet really filling great early morning time make breakfast great afternoon snack work feeling sluggish \n",
            "Original summary: protein bar \n",
            "Predicted summary:  yummy\n",
            "\n",
            "\n",
            "Review: aware decaf coffee although showed search decaf cups intended purchase gift kept recipient drink caffeine favorite means \n",
            "Original summary: not decaf \n",
            "Predicted summary:  decaf decaf\n",
            "\n",
            "\n",
            "Review: wonderful wrote perfect iced cookie one pen writing cookies names happy ca \n",
            "Original summary: cookie \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: truffle oil quite good prefer brand france urbani italy expensive oh delicious tried black white good black bit stronger pungent event healthy alternative butter enjoy \n",
            "Original summary: delicious but not the best \n",
            "Predicted summary:  great flavor\n",
            "\n",
            "\n",
            "Review: enjoy coffee office split right middle loving think worth try order regularly \n",
            "Original summary: hit or miss \n",
            "Predicted summary:  good coffee\n",
            "\n",
            "\n",
            "Review: husband gluten free food several years tried several different bread mixes first actually enjoys buying amazon saves loaf \n",
            "Original summary: really good gluten free bread \n",
            "Predicted summary:  great gluten free bread\n",
            "\n",
            "\n",
            "Review: hubby eats says good snacks morning done apple flavor \n",
            "Original summary: really good nice snack \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: waste money disgusting product chocolate taste tastes like plastic lining paper carton using milk treated ultra high temperatures like fresh milk go get fresh milk hershey syrup want chocolate milk \n",
            "Original summary: please do not waste your money \n",
            "Predicted summary:  yuck\n",
            "\n",
            "\n",
            "Review: absolutely loves apple chicken happy hips looks forward one morning one night gets soooo excited would eat allowed \n",
            "Original summary: healthy treats \n",
            "Predicted summary:  great for training\n",
            "\n",
            "\n",
            "Review: strong much flavor little aroma tried purchase another time similiar brands met standards expected \n",
            "Original summary: no flavor \n",
            "Predicted summary:  san francisco bay coffee\n",
            "\n",
            "\n",
            "Review: company wanted chose order anyway \n",
            "Original summary: water \n",
            "Predicted summary:  not good\n",
            "\n",
            "\n",
            "Review: introduced number people hooked best sour gummy ever great flavors got great price \n",
            "Original summary: new favorite \n",
            "Predicted summary:  best licorice\n",
            "\n",
            "\n",
            "Review: new price attractive however tastes horrible maybe old zico coconut water brands might find acceptable \n",
            "Original summary: do not be by the price \n",
            "Predicted summary:  terrible\n",
            "\n",
            "\n",
            "Review: sure ever going buy product way expensive market price \n",
            "Original summary: too expensive \n",
            "Predicted summary:  good value\n",
            "\n",
            "\n",
            "Review: flavor normally find local stores plus buy bulk things take savings add veggies even stir egg noodles cook add nutrition quick meals lot extra \n",
            "Original summary: good value \n",
            "Predicted summary:  great deal\n",
            "\n",
            "\n",
            "Review: order tea labeled decaff must caffeine residue levels tested tea caffeine decaff non decaff tea anywhere caffeine caffeine caffeinated tea caffeine slightly less naturally present tea leaf \n",
            "Original summary: caffeine is not \n",
            "Predicted summary:  not as good as\n",
            "\n",
            "\n",
            "Review: excellent babies toddler really best offer little one delicious rich vitamins calcium protein low fat sorry products available website \n",
            "Original summary: excellent product for babies and toddler \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: purchased item dented would bet run dented product clearing ship ones \n",
            "Original summary: sometimes dented \n",
            "Predicted summary:  dented cans\n",
            "\n",
            "\n",
            "Review: almost tastes like mini blueberry pie love one favorite thoroughly fallen love \n",
            "Original summary: excellent love the blueberry pecan \n",
            "Predicted summary:  yummy\n",
            "\n",
            "\n",
            "Review: dog loves keeps busy minutes long time chew hound \n",
            "Original summary: chew away \n",
            "Predicted summary:  dog loves it\n",
            "\n",
            "\n",
            "Review: plant came quickly looks great office nice pot plant thriving well \n",
            "Original summary: very nice office plant \n",
            "Predicted summary:  plant\n",
            "\n",
            "\n",
            "Review: dog loves lickety stik bacon flavor since likes much plan getting flavors great liquid treat dog highly recommend lickety stik \n",
            "Original summary: great dog treat \n",
            "Predicted summary:  dog loves them\n",
            "\n",
            "\n",
            "Review: great toy dogs chew everything else little literally eats toys one toys yet destroy loves carries around everywhere got rex cutest thing \n",
            "Original summary: good for chewers \n",
            "Predicted summary:  dogs love it\n",
            "\n",
            "\n",
            "Review: really search good deals tea tea great price tea amazon almost cup price cup coffee herbal varieties low caffine good option wife used dinner coffe \n",
            "Original summary: great price for great tea \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: pricey essentially small bag hard crumbs maybe dog spoiled treats like third class treats definitely bottom doggie treat often simply walk away glad people like buying \n",
            "Original summary: waste of money \n",
            "Predicted summary:  not for dogs\n",
            "\n",
            "\n",
            "Review: little pricey consider sugar low cal caffine really rich flavor best chai ever found \n",
            "Original summary: fabulous product \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: loves taste beef freeze dried dog treats use training really works \n",
            "Original summary: dog lover \n",
            "Predicted summary:  great training treat\n",
            "\n",
            "\n",
            "Review: three dogs cairn terriers year old border collie proud greenies like taste helps keep gums teeth good shape \n",
            "Original summary: our dogs love greenies \n",
            "Predicted summary:  greenies\n",
            "\n",
            "\n",
            "Review: good soft drink smooth strawberry cream soda tasty \n",
            "Original summary: good stuff \n",
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: item arrived sugar free shipped regular version caramel syrup small internal sticker bottle stated sugar free although company label bottle stated regular version \n",
            "Original summary: wrong item \n",
            "Predicted summary:  not as good as\n",
            "\n",
            "\n",
            "Review: like strong coffee coffee rated found weak sickening taste \n",
            "Original summary: disapointed \n",
            "Predicted summary:  not bad\n",
            "\n",
            "\n",
            "Review: saw peanut butter chocolate cereal knew try pleased eat chocolate breakfast feel guilty two kids love cereal well great eat alone favorite milk product yogurt mix homemade granola well \n",
            "Original summary: the yummy \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted summary:  yummy\n",
            "\n",
            "\n",
            "Review: begging time loves used buy small bottle buying every weeks since saw oz buying last lot longer gas money cheaper buy online \n",
            "Original summary: my dog loves it \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: true also need decent scale tried caviar recipe everything worked perfectly first try fun easy make kit comes large enough samples looks like good uses \n",
            "Original summary: great to \n",
            "Predicted summary:  works great\n",
            "\n",
            "\n",
            "Review: dog really likes treats like buy run mill treats loaded fat fillers continue buy \n",
            "Original summary: buddy biscuits \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: tulsi green tea great good iced tea well \n",
            "Original summary: green tea \n",
            "Predicted summary:  green tea\n",
            "\n",
            "\n",
            "Review: always put something market couple poof gone best tasting product pepsi \n",
            "Original summary: best taste \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: like tomatoes fresh flavorful also come carton welcome alternative metal cans impart flavor sometimes lined plastic containing \n",
            "Original summary: yummy tomatoes good packaging \n",
            "Predicted summary:  good product\n",
            "\n",
            "\n",
            "Review: great get habit forming careful bought whole case save overall versus going supermarket rich dark chocolate crisp cookie worth every penny oreo eat heart \n",
            "Original summary: delicious \n",
            "Predicted summary:  love these\n",
            "\n",
            "\n",
            "Review: else say arrived promptly perhaps time expected expiration date like next day good go \n",
            "Original summary: baby loves it \n",
            "Predicted summary:  not what expected\n",
            "\n",
            "\n",
            "Review: bought local recently advertised cheesy flavor detectable product even salt flavor avoid product \n",
            "Original summary: no cheese flavor \n",
            "Predicted summary:  good but not great\n",
            "\n",
            "\n",
            "Review: big volume coffee morning one great \n",
            "Original summary: great morning coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: drank try keep awake fell asleep minutes drinking feel anything \n",
            "Original summary: it made me fall \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: drink cups day verona italian french roast coffee wanted try lower acid version brand coffee smells tastes like vinegar totally unpalatable better drinking water acid coffee bothers \n",
            "Original summary: single worst coffee ever \n",
            "Predicted summary:  not very good\n",
            "\n",
            "\n",
            "Review: getting price however afraid stocking anymore reduced price think one trying eat crackers low calorie string cheese breakfast every total calories put breakfast baggie go \n",
            "Original summary: am addicted to these \n",
            "Predicted summary:  not as good as expected\n",
            "\n",
            "\n",
            "Review: first time using fondarific fondant general one really easy use baby shower cake worked indicated also colored made two tier cake final product looked great greasy \n",
            "Original summary: easy to use \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: work home drink cups cup coffee day good tasting coffee lowest price cup market \n",
            "Original summary: great coffee great price \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: guys say natural really tastes great pleasantly surprised stand flavor carbonated think would even better product time come fed sweet juices aftertaste make obvious really natural switch really gets vote \n",
            "Original summary: great taste all natural \n",
            "Predicted summary:  not very good\n",
            "\n",
            "\n",
            "Review: product good goes long way quite good one dd good product less \n",
            "Original summary: very good \n",
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: tea wonderful soothing even soothing get shipped house found hard find decaffeinated tea grocery store much easier \n",
            "Original summary: decaffeinated french vanilla tea yummy \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: wow little calorie espresso sugar serve cold delicious little shot espresso sugar overly sweet sugar helps offset taste espresso caffe bitter sweet tastes good really gave afternoon kick pants \n",
            "Original summary: nice little pick me up \n",
            "Predicted summary:  best water ever\n",
            "\n",
            "\n",
            "Review: mayonnaise delicious side side taste test would give hellman edge hellman richer taste \n",
            "Original summary: excellent but \n",
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: love medium full flavored roast smooth taste bitter acidic taste excellent coffee good value also try timothy kona good also \n",
            "Original summary: wonderful coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: nice item chunks meat good gravy cat fond varieties nice little treat nonetheless think item bit pricy per ounce \n",
            "Original summary: nice but pricey \n",
            "Predicted summary:  good cat food\n",
            "\n",
            "\n",
            "Review: bought cookies gifts open last long good make great gifts would definitely buy \n",
            "Original summary: mouth watery cookies \n",
            "Predicted summary:  cookies\n",
            "\n",
            "\n",
            "Review: great price fast shipping best chips better ingredients less calories snack foods plus taste like real chips \n",
            "Original summary: pop chips are the best \n",
            "Predicted summary:  great chips\n",
            "\n",
            "\n",
            "Review: taco bell chipotle sauce bold flavorful tried chicken wings tacos salad made dish extremely tasty glad sampled new sauce staple condiment \n",
            "Original summary: bold flavor \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: bought seeds make centerpieces really surprised fast grow planted seeds potting soil without ny preparation anything kept watering days super tall ready displayed centerpieces perfect \n",
            "Original summary: perfect for in days \n",
            "Predicted summary:  great seeds\n",
            "\n",
            "\n",
            "Review: every time need sun dried tomatoes local grocery stores conveniently small pouches ensure always hand called recipe \n",
            "Original summary: sun dried tomato bliss \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love soup eat plain use recipe cannot find area glad amazon \n",
            "Original summary: soup chicken cheese \n",
            "Predicted summary:  soup\n",
            "\n",
            "\n",
            "Review: size quite good dog training smell strong cannot put open bag must seal everytime gave treat otherwise dog stand trying fetch believe taste great puppy purchase sure \n",
            "Original summary: strong smell and my puppy loves it \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: love chips auto order every months taste great whole bag calories bag every day sure helped weight loss little bags eat huge amount \n",
            "Original summary: great purchase \n",
            "Predicted summary:  great chips\n",
            "\n",
            "\n",
            "Review: many kit wines cost three four times made many kits find fine table wine recommend adding water five gallon mark flavor \n",
            "Original summary: good wine \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: sooo much pepper heavy salt reminds adams trick food cannot eat seriously fresh nuts seasoned \n",
            "Original summary: over the top seasoning \n",
            "Predicted summary:  great salt\n",
            "\n",
            "\n",
            "Review: loved brand best vanilla flavor others tried would buy better price \n",
            "Original summary: wolfgang puck coffee vanilla \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: another brand cinammon carried amazon much better tasting brand maybe packaging part problem simple plastic bag tie amazon brand comes carefully set plastic box \n",
            "Original summary: edible have had much better \n",
            "Predicted summary:  not as good as expected\n",
            "\n",
            "\n",
            "Review: throw pack one actually taste bad especially compared orange tangerine like carbonation adds juice flavors need work switch drinks best worst watermelon strawberry kiwi berry black cherry orange tangerine \n",
            "Original summary: my favorite of the four tried \n",
            "Predicted summary:  tastes like\n",
            "\n",
            "\n",
            "Review: daughter drinking since months old months old still loves snack time healthy delicious great addition menu \n",
            "Original summary: great snack \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: live guinea africa order products delivered boat every months sometimes disappointed time zero calories zero carbs taste great price zero delivery costs prime ordered different flavors one favorite love \n",
            "Original summary: love it \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: purchased larger size love size perfect keep purse snack especially times others dessert snack cannot eat must gluten free spouse touch diet food loves \n",
            "Original summary: cannot get enough \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: always house drink favorite mix sprite oh good every day mind larger bottles use much bring \n",
            "Original summary: am an adult still love this \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: ginger snaps overpowering ginger go great milk really enjoyed house great buy affordable compared alternative diet foods last least week store well \n",
            "Original summary: you can eat ginger again \n",
            "Predicted summary:  ginger\n",
            "\n",
            "\n",
            "Review: give squid one star use might thoroughly disappointed quite possibly call crazy \n",
            "Original summary: can for your \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: quality seeds excellent begin germinate hours days ready use never sprouted seeds results good easily recommend sprouter whether human consumption four legged friends \n",
            "Original summary: wheat grass seeds \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love stuff great store bought homemade baked goods kicking things professional level works colored dark light frosting also used dusting powdered sugar pretty fine texture \n",
            "Original summary: fun like dust \n",
            "Predicted summary:  perfect\n",
            "\n",
            "\n",
            "Review: bought jumbo greenies black lab loved way expensive regular use notice difference breath primary reason buying \n",
            "Original summary: jumbo greenies good but very expensive \n",
            "Predicted summary:  greenies\n",
            "\n",
            "\n",
            "Review: also bought costco per box included bags oz kids fighting remaining bags good buying due price high price prevent product reaching mass distribution \n",
            "Original summary: very good but too pricey \n",
            "Predicted summary:  good but not the best\n",
            "\n",
            "\n",
            "Review: originally found mints whole foods taste superb get lot money plus comes cute little tin uses dog loves go organic \n",
            "Original summary: wonderful \n",
            "Predicted summary:  my dog loves this\n",
            "\n",
            "\n",
            "Review: regular spam awful almost inedible would give tastes like animal know mean fellow spam turkey spam pretty good great would give worth try \n",
            "Original summary: better than regular \n",
            "Predicted summary:  not bad\n",
            "\n",
            "\n",
            "Review: really need know many cans also whitefish tuna buffet canned cat food thanks \n",
            "Original summary: need to know how many in case \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: great tasting rich flavor perfect making nice hot cup mocha bought test hershey syrup mocha incredible distinct taste difference noticeable much richer tastes like chocolate less sugary hershey syrup \n",
            "Original summary: great taste \n",
            "Predicted summary:  great flavor\n",
            "\n",
            "\n",
            "Review: number one japan number one great save get shipped automatically every month lugging car \n",
            "Original summary: great tea \n",
            "Predicted summary:  great price\n",
            "\n",
            "\n",
            "Review: bought item read best mayo sold yes even better worlds favorite hellman well review good bit better hellman fact put empty hellman jar said nothing family never knew difference \n",
            "Original summary: blue mayo \n",
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: gum great makes car smell good leave refreshing sweet tart smooth \n",
            "Original summary: love the gum and the price \n",
            "Predicted summary:  gum\n",
            "\n",
            "\n",
            "Review: flavorful smells like heaven great price compared stores arrived fast \n",
            "Original summary: divine \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love low calorie organic doctors recommend grams fiber daily smart bran grams per serving fruits veggies set day eat dry vanilla frozen yogurt cinnamon \n",
            "Original summary: yes to smart bran \n",
            "Predicted summary:  love these\n",
            "\n",
            "\n",
            "Review: found spice blend dallas years back tell restaurant using grilled shrimp like cajun spice grilling fish recommend store dry place replace every year least lose flavor \n",
            "Original summary: good stuff \n",
            "Predicted summary:  good stuff\n",
            "\n",
            "\n",
            "Review: plain riceselect couscous delicious easy quick prepare great side item base main course far found bad product riceselect \n",
            "Original summary: yummy \n",
            "Predicted summary:  good\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OTkaYNjHW4lC",
        "colab_type": "text"
      },
      "source": [
        "This is really cool stuff. Even though the actual summary and the summary generated by our model do not match in terms of words, both of them are conveying the same meaning. Our model is able to generate a legible summary based on the context present in the text.\n",
        "\n",
        "This is how we can perform text summarization using deep learning concepts in Python.\n",
        "\n",
        "#How can we Improve the Model’s Performance Even Further?\n",
        "\n",
        "Your learning doesn’t stop here! There’s a lot more you can do to play around and experiment with the model:\n",
        "\n",
        "I recommend you to **increase the training dataset** size and build the model. The generalization capability of a deep learning model enhances with an increase in the training dataset size\n",
        "\n",
        "Try implementing **Bi-Directional LSTM** which is capable of capturing the context from both the directions and results in a better context vector\n",
        "\n",
        "Use the **beam search strategy** for decoding the test sequence instead of using the greedy approach (argmax)\n",
        "\n",
        "Evaluate the performance of your model based on the **BLEU score**\n",
        "\n",
        "Implement **pointer-generator networks** and **coverage mechanisms**\n",
        " \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R_qIecuvY5GT",
        "colab_type": "text"
      },
      "source": [
        "#End Notes\n",
        "\n",
        "If you have any feedback on this article or any doubts/queries, kindly share them in the comments section over [here](https://www.analyticsvidhya.com/blog/2019/06/comprehensive-guide-text-summarization-using-deep-learning-python/) and I will get back to you. And make sure you experiment with the model we built here and share your results with me!"
      ]
    }
  ]
}