Untitled.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader, TensorDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualising the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('reviews.txt') as f:\n",
    "    reviews = f.read()\n",
    "\n",
    "with open('labels.txt') as f:\n",
    "    labels = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers  . the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students . when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled . . . . . . . . . at . . . . . . . . . . high . a classic line inspector i  m here to sack one of your teachers . student welcome to bromwell high . i expect that many adults of my age think that bromwell high is far fetched . what a pity that it isn  t   \n",
      "story of a man who has unnatural feelings for a pig . starts out with a opening scene that is a terrific example of absurd comedy . a formal orchestra audience is turned into an insane  violent mob by the crazy chantings of it  s singers . unfortunately it stays absurd the whole time with no general narrative eventually making it just too off putting . even those from the era should be turned off . the cryptic dialogue would make shakespeare seem easy to a third grader . on a technical level it  s better than you might think with some good cinematography by future great vilmos zsigmond . future stars sally kirkland and frederic forrest can be seen briefly .  \n",
      "homelessness  or houselessness as george carlin stated  has been an issue for years but never a plan to help those on the street that were once considered human who did everything from going to school  work  or vote for the matter . most people think of the homeless as just a lost cause while worrying about things such as racism  the war on iraq  pressuring kids to succeed  technology  the elections  inflation  or worrying if they  ll be next to end up on the streets .  br    br   but what if y\n",
      "\n",
      "positive\n",
      "negative\n",
      "po\n"
     ]
    }
   ],
   "source": [
    "print(reviews[:2000])\n",
    "print()\n",
    "print(labels[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the reviews are saperated by new lines so are labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "We need data to be in proper format to feed into the neural network. So encode the words into a unique number and we will also clean a bit\n",
    "1. First we remove the punctuations\n",
    "2. Next we remove new line characters and combine everything into one\n",
    "3. Next we will create a list of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Punctuations:  !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "#removing punctuations\n",
    "from string import punctuation\n",
    "print('Punctuations: ', punctuation)\n",
    "\n",
    "reviews = reviews.lower()\n",
    "all_text = ''.join([ch for ch in reviews if ch not in punctuation])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#joining by new lines, combining and splitting by words\n",
    "reviews_split = all_text.split('\\n')\n",
    "all_text = ' '.join(reviews_split)\n",
    "all_text = all_text.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bromwell', 'high', 'is', 'a', 'cartoon', 'comedy', 'it', 'ran', 'at', 'the']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_text[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encode the data\n",
    "Here we convert each unique word into an integer so that we can feed into the neural network. We use `Counter` so that the most frequent word will be given the lowest integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "vocab2int = {}\n",
    "i = 1\n",
    "for x, y in Counter(all_text).most_common():\n",
    "    vocab2int[x] = i\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews_ints = []\n",
    "for sent in reviews_split:\n",
    "    a = []\n",
    "    for word in sent.split():\n",
    "        a.append(vocab2int[word])\n",
    "        \n",
    "    reviews_ints.append(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test code for above implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words:  74072\n",
      "\n",
      "Tokenized view:\n",
      " [[21025, 308, 6, 3, 1050, 207, 8, 2138, 32, 1, 171, 57, 15, 49, 81, 5785, 44, 382, 110, 140, 15, 5194, 60, 154, 9, 1, 4975, 5852, 475, 71, 5, 260, 12, 21025, 308, 13, 1978, 6, 74, 2395, 5, 613, 73, 6, 5194, 1, 24103, 5, 1983, 10166, 1, 5786, 1499, 36, 51, 66, 204, 145, 67, 1199, 5194, 19869, 1, 37442, 4, 1, 221, 883, 31, 2988, 71, 4, 1, 5787, 10, 686, 2, 67, 1499, 54, 10, 216, 1, 383, 9, 62, 3, 1406, 3686, 783, 5, 3483, 180, 1, 382, 10, 1212, 13583, 32, 308, 3, 349, 341, 2913, 10, 143, 127, 5, 7690, 30, 4, 129, 5194, 1406, 2326, 5, 21025, 308, 10, 528, 12, 109, 1448, 4, 60, 543, 102, 12, 21025, 308, 6, 227, 4146, 48, 3, 2211, 12, 8, 215, 23]]\n"
     ]
    }
   ],
   "source": [
    "print('Unique words: ', len(vocab2int))\n",
    "print()\n",
    "\n",
    "print('Tokenized view:\\n', reviews_ints[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding the labels\n",
    "We have only two labels. Those must be encoded. We will convert positive label to 1 and negative label to 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 0, 1]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_encoded = []\n",
    "labels_split = labels.split('\\n')\n",
    "\n",
    "for x in labels_split:\n",
    "    if x == 'positive':\n",
    "        labels_encoded.append(1)\n",
    "    else:\n",
    "        labels_encoded.append(0)\n",
    "\n",
    "labels_encoded[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can verify these outputs with the starting outputted values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing the outliers\n",
    "We want to make sure that our model will not get some outliers i.e., values that are extremely big or too small.\n",
    "\n",
    "For other values, we want to make sure that all the reviews should be of same lenth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reviews with zero length:  1\n",
      "Reviews with max length:  2514\n"
     ]
    }
   ],
   "source": [
    "review_lengths = Counter([len(x) for x in reviews_ints])\n",
    "print('Reviews with zero length: ', review_lengths[0])\n",
    "print('Reviews with max length: ', max(review_lengths))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see that we have one review that has zero length and also we see that the maximum length is very large. So we want to remove the reviews with shorter length and truncate the reviews with longer length so that our model will learn more effectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of reviews before removing the outliers:  25001\n",
      "Number of reviews after removing the outliers:  25000\n"
     ]
    }
   ],
   "source": [
    "print('Number of reviews before removing the outliers: ', len(reviews_ints))\n",
    "\n",
    "#removing the reviews with zero length\n",
    "non_zero_idx = [ii for ii, rev in enumerate(reviews_ints) if len(rev)!=0]\n",
    "\n",
    "reviews_ints = [reviews_ints[ii] for ii in non_zero_idx]\n",
    "labels_encoded = np.array([labels_encoded[ii] for ii in non_zero_idx])\n",
    "\n",
    "print('Number of reviews after removing the outliers: ', len(reviews_ints))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Padding and truncating reviews\n",
    "Now we want to truncate the reviews that are too long to some specified length and the reviews that are shorter than specified length then we will pad them with 0s. The good sequence length will be 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_features(reviews_ints, seq_length):\n",
    "    '''\n",
    "    This function will return an array of reviews that are either truncated to seq_length\n",
    "    or padded with 0s\n",
    "    '''\n",
    "    features = []\n",
    "    \n",
    "    for rev in reviews_ints:\n",
    "        if len(rev) >= seq_length:\n",
    "            features.append(rev[:seq_length])\n",
    "        else:\n",
    "            features.append([0]*(seq_length-len(rev)) + rev)\n",
    "    \n",
    "    return np.array(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test code to test the above function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows in features: 25000 -- Rows in reviews_ints: 25000\n",
      "Columns in features: 200 -- Sequence length: 200\n"
     ]
    }
   ],
   "source": [
    "seq_length = 200\n",
    "\n",
    "features = pad_features(reviews_ints, seq_length)\n",
    "\n",
    "#features should have same rows as reviews_ints\n",
    "print('Rows in features: {} -- Rows in reviews_ints: {}'.format(len(features), len(reviews_ints)))\n",
    "\n",
    "#features should have same number of columns as sequence length\n",
    "print('Columns in features: {} -- Sequence length: {}'.format(len(features[0]), seq_length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [22382    42 46418    15   706 17139  3389    47    77    35]\n",
      " [ 4505   505    15     3  3342   162  8312  1652     6  4819]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [   54    10    14   116    60   798   552    71   364     5]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    1   330   578    34     3   162   748  2731     9   325]\n",
      " [    9    11 10171  5305  1946   689   444    22   280   673]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    1   307 10399  2069  1565  6202  6528  3288 17946 10628]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [   21   122  2069  1565   515  8181    88     6  1325  1182]\n",
      " [    1    20     6    76    40     6    58    81    95     5]\n",
      " [   54    10    84   329 26230 46427    63    10    14   614]\n",
      " [   11    20     6    30  1436 32317  3769   690 15100     6]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [   40    26   109 17952  1422     9     1   327     4   125]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [   10   499     1   307 10399    55    74     8    13    30]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0]]\n"
     ]
    }
   ],
   "source": [
    "print(features[:30, :10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting into training, validation and test sets\n",
    "We will split the data into three parts. We will declare a `split_frac` that will tell how much of the data should be in training data and the remaining data is split into two halves, one for validation and other for testing\n",
    "\n",
    "First I have generated random indexes of size 80% of length of features. Then I have used these indexes to extract traaining samples and out of remaining samples, I have taken 50% to validation and other 50% test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_frac = 0.8\n",
    "\n",
    "train_idx = np.random.choice(np.arange(len(features)), int(len(features)*split_frac), replace=False)\n",
    "\n",
    "train_x = np.array([features[ii] for ii in train_idx])\n",
    "train_y = np.array([labels_encoded[ii] for ii in train_idx])\n",
    "\n",
    "valid_reviews = np.array([features[ii] for ii in range(len(features)) if ii not in train_idx])\n",
    "valid_labels  = np.array([labels_encoded[ii] for ii in range(len(features)) if ii not in train_idx])\n",
    "\n",
    "valid_x = valid_reviews[:int(valid_reviews.shape[0]/2)]\n",
    "test_x = valid_reviews[int(valid_reviews.shape[0]/2):]\n",
    "\n",
    "valid_y = valid_labels[:int(valid_reviews.shape[0]/2)]\n",
    "test_y = valid_labels[int(valid_reviews.shape[0]/2):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Shapes\n",
      " Train set: (20000, 200) \n",
      "Valid set: (2500, 200) \n",
      "Test set: (2500, 200)\n"
     ]
    }
   ],
   "source": [
    "#printing the shape of splitted data\n",
    "print('Feature Shapes\\n',\n",
    "      'Train set:', train_x.shape,\n",
    "      '\\nValid set:', valid_x.shape,\n",
    "      '\\nTest set:', test_x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataLoaders and Batching\n",
    "Now we have x,y that is inputs and targets, we convert them to dataset `TensorDataset` class, next we can use this dataset to pass to `DataLoader` class and get batches of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "valid_data = TensorDataset(torch.from_numpy(valid_x), torch.from_numpy(valid_y))\n",
    "test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))\n",
    "\n",
    "#defining the batch size\n",
    "batch_size = 50\n",
    "\n",
    "trainloader = DataLoader(train_data, batch_size=batch_size, shuffle=True)\n",
    "validloader = DataLoader(valid_data, batch_size=batch_size, shuffle=True)\n",
    "testloader = DataLoader(test_data, batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of sample_x:  torch.Size([50, 200])\n",
      "sample_x: \n",
      " tensor([[    0,     0,     0,  ...,   288,  3278,    45],\n",
      "        [    0,     0,     0,  ...,  1169,  3526,  5876],\n",
      "        [    0,     0,     0,  ...,    55,    55,   799],\n",
      "        ...,\n",
      "        [   11,   392,    20,  ...,   276,   752,    17],\n",
      "        [    0,     0,     0,  ...,    58,   541,    48],\n",
      "        [    0,     0,     0,  ...,   194, 10895,   130]], dtype=torch.int32)\n",
      "\n",
      "Shape of sample_y:  torch.Size([50])\n",
      "sample_y: \n",
      " tensor([1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1,\n",
      "        0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1,\n",
      "        0, 0], dtype=torch.int32)\n"
     ]
    }
   ],
   "source": [
    "#obtaining one batch of training data\n",
    "sample_x, sample_y = next(iter(trainloader))\n",
    "\n",
    "print('Shape of sample_x: ', sample_x.shape)\n",
    "print('sample_x: \\n', sample_x)\n",
    "print()\n",
    "print('Shape of sample_y: ', sample_y.shape)\n",
    "print('sample_y: \\n', sample_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the sentiment network\n",
    "We have to define the network. Here we will use the `embedding layer` as input to the `lstm` because we have a lot of words and using one-hot encoding to these would be inefficient.\n",
    "1. We defined the embedding layer to convert the words tokens(integers) to embeddings of some specific size.\n",
    "2. Defined a lstm layers that takes inputs from the embedding layer\n",
    "3. Define a fully connected layer that maps lstm outputs to our desired output size\n",
    "4. Use sigmoid activation function, which turns outputs in the value of range 0-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on CPU\n"
     ]
    }
   ],
   "source": [
    "#checking availability of gpu\n",
    "cuda_available = torch.cuda.is_available()\n",
    "if cuda_available:\n",
    "    print('Training on GPU')\n",
    "else:\n",
    "    print('Training on CPU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SentimentRNN(nn.Module):\n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_p=0.2):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_dim = hidden_dim\n",
    "        \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, batch_first=True, dropout=drop_p)\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=drop_p)\n",
    "        \n",
    "    \n",
    "    def forward(self, x, hidden):\n",
    "        \n",
    "        batch_size = x.shape[0]\n",
    "        \n",
    "        x = self.embedding(x)\n",
    "        output, hidden = self.lstm(x,hidden)\n",
    "        \n",
    "        #stack up lstm outputs\n",
    "        output = output.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        output = self.dropout(output)\n",
    "        \n",
    "        \n",
    "        output = self.fc(output)\n",
    "        output = F.sigmoid(output)\n",
    "        \n",
    "        #reshape to be batch first\n",
    "        output = output.view(batch_size, -1)\n",
    "        output = output[:, -1] #get the last batch\n",
    "        \n",
    "        \n",
    "        return output, hidden\n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        weight = next(self.parameters()).data\n",
    "        \n",
    "        if cuda_available:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())\n",
    "        else:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())\n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiate the model\n",
    "Here, we'll instantiate the network. First up, defining the hyperparameters.\n",
    "1. vocab_size: Size of our vocabulary or the range of values for our input, word tokens\n",
    "2. output_size: Size of our desired output; the number of class scores we want to output (pos/neg)\n",
    "3. embedding_dim: Number of columns in the embedding lookup table; size of our embeddings.\n",
    "4. hidden_dim: Number of units in the hidden layers of our LSTM cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.\n",
    "5. n_layers: Number of LSTM layers in the network. Typically between 1-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SentimentRNN(\n",
      "  (embedding): Embedding(74073, 400)\n",
      "  (lstm): LSTM(400, 256, num_layers=2, batch_first=True, dropout=0.2)\n",
      "  (fc): Linear(in_features=256, out_features=1, bias=True)\n",
      "  (dropout): Dropout(p=0.2)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "#instantiate the model with hyper parameters\n",
    "vocab_size = len(vocab2int)+1 #+1 for the padding 0\n",
    "output_size = 1\n",
    "embedding_dim = 400\n",
    "hidden_dim = 256\n",
    "n_layers = 2\n",
    "\n",
    "model = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "Now it's time to train our model. \n",
    "This time we use different loss function for only one output called `Binary Cross Entropy Loss`.\n",
    "\n",
    "The hyper parameters for training are:\n",
    "1. lr: learning rate\n",
    "2. epochs: the number of epochs\n",
    "3. clip: the maximum gradient value to clip at"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model,trainloader, validloader, batch_size,clip=5, print_every=50, epochs=1, lr=0.01):\n",
    "    \n",
    "    if cuda_available:\n",
    "        model = model.cuda()\n",
    "    \n",
    "    criterion = nn.BCELoss()\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "    \n",
    "    counter = 0 \n",
    "    for e in range(epochs):\n",
    "        \n",
    "        hidden = model.init_hidden(batch_size)\n",
    "        for x, y in trainloader:\n",
    "            counter += 1\n",
    "            \n",
    "            if cuda_available:\n",
    "                x, y = x.cuda(), y.cuda()\n",
    "            \n",
    "            hidden = tuple([each.data for each in hidden])\n",
    "            output, hidden = model(x.type(torch.LongTensor), hidden)\n",
    "            loss = criterion(output.squeeze(), y.float())\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "            optimizer.step()\n",
    "            \n",
    "            if counter%print_every==0:\n",
    "                \n",
    "                val_h = model.init_hidden(batch_size)\n",
    "                \n",
    "                model.eval()\n",
    "                \n",
    "                for x, y in validloader:\n",
    "                    \n",
    "                    val_h = tuple([each.data for each in val_h])\n",
    "                    \n",
    "                    if cuda_available:\n",
    "                        x, y = x.cuda(), y.cuda()\n",
    "                        \n",
    "                    output, hidden = model(x.type(torch.LongTensor), val_h)\n",
    "                    loss_valid = criterion(output.squeeze(), y.float())\n",
    "                    \n",
    "                print('Epoch: {}/{}'.format(e+1, epochs),\n",
    "                        'Training Loss: {}'.format(loss.item()),\n",
    "                        'Validation loss: {}'.format(loss_valid.item()))\n",
    "                model.train()\n",
    "                break\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Surya Prakash Reddy\\Anaconda3\\lib\\site-packages\\torch\\nn\\functional.py:1006: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
      "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1 Training Loss: 0.6673436164855957 Validation loss: 0.6653851866722107\n"
     ]
    }
   ],
   "source": [
    "train(model, trainloader, validloader, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}