Explanation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Intro:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use:\n",
    "\n",
    "Main file is **'siamese.py'**, the configuration file is **'siamese-config.yaml'**.  \n",
    "\n",
    "#### 1. Initialization:\n",
    "    \n",
    "    First run with the original dataset, you should input parameters **'--config siamese-config.yaml'**, while configuring **make_dict** and **data_preprocessing** as **True**, it will make the embedding and preprocess the original dataset, saved them for future usage.  \n",
    "\n",
    "\n",
    "#### 2. Model:\n",
    "\n",
    "**Tuning Parameters**:\n",
    "    \n",
    "    a. Classifier\n",
    "    \n",
    "       fc_dim: classifier fully connected layer size, \n",
    "    \n",
    "    b. Encoder\n",
    "           \n",
    "       hidden_size: lstm hidden size\n",
    "       num_layers: lstm layer \n",
    "       bidirectional: bidirectional lstm can get more info\n",
    "       dropout: avoid overfitting\n",
    "        \n",
    "    c. Embedding\n",
    "           \n",
    "       embedding_freeze: Set it to false, then the embedding will participate backpropogation. Not so good from my experience, especially small training dataset.\n",
    "    \n",
    "    d. Other\n",
    "    \n",
    "**Structure**:\n",
    "    \n",
    "    a. Classifier\n",
    "        \n",
    "        fc layers, non-linear fc layers(add ReLU)\n",
    "    \n",
    "    b. Encoder \n",
    "        \n",
    "        Features generating method, current method is (v1, v2, abs(v1-v2), v1*v2), more features with different vector distance measurement?\n",
    "        \n",
    "    \n",
    "#### 3. Training\n",
    "\n",
    "    a. Optimizer\n",
    "    \n",
    "       Default SGD, lots of optimizer in torch.\n",
    "\n",
    "    b. Learning rate\n",
    "    \n",
    "       It should be small enough to avoid oscillation. Furthur exploration can be dynamic lr clipping.\n",
    "        \n",
    "    c. Other"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To do list:\n",
    "\n",
    "    1. Early stopping\n",
    "    \n",
    "    2. Dynamic learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuration "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    'experiment_name': 'siamese-baseline',\n",
    "    'task': 'train',\n",
    "    'make_dict': True,\n",
    "    'data_preprocessing': True,\n",
    "\n",
    "    'ckpt_dir': 'ckpt/',\n",
    "\n",
    "    'training':{\n",
    "        'num_epochs': 20,\n",
    "        'learning_rate': 0.01,\n",
    "        'optimizer': 'sgd'\n",
    "    },\n",
    "    \n",
    "    'embedding':{\n",
    "        'full_embedding_path': 'input/wiki.es.vec',\n",
    "        'cur_embedding_path': 'input/embedding.pkl',\n",
    "    },\n",
    "        \n",
    "    'model':{\n",
    "        'fc_dim': 100,\n",
    "        'name': 'siamese',\n",
    "        'embed_size': 300,\n",
    "        'batch_size': 1,\n",
    "        'embedding_freeze': False,\n",
    "        'encoder':{\n",
    "            'hidden_size': 150,\n",
    "            'num_layers': 1,\n",
    "            'bidirectional': False,\n",
    "            'dropout': 0.0,\n",
    "        },  \n",
    "    },   \n",
    "    \n",
    "    'result':{\n",
    "        'filename':'result.txt',\n",
    "        'filepath':'res/',\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#coding=utf-8\n",
    "import os\n",
    "import yaml\n",
    "import argparse\n",
    "from datetime import datetime\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# utils\n",
    "from utils import get_embedding, load_embed, save_embed, data_preprocessing\n",
    "# data\n",
    "from data import myDS, mytestDS\n",
    "# model\n",
    "from model import Siamese_lstm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preprocess the original data:** \n",
    "\n",
    "    1. Remove spanish punctuations, convert unique spanish uppercase letters into lower case since .lower() can not be applied to spanish uppercase letters.      \n",
    "        E.g. Remove ¡,¿; Convert Á to á, Ó to ó, Ú to ú, É to é, Í to í.  \n",
    "        \n",
    "    2. Remove spanish stopwords.  \n",
    "    \n",
    "    3. After stopwords removal, there are some empty sentences, drop those sentences pairs.\n",
    "\n",
    "Details in **utils.py** function: **data_preprocessing**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pre-processing Original Data ...\n",
      "dirty sample count: 73\n",
      "positive dirty training sample: 5\n",
      "negative dirty training sample: 68\n",
      "Train sample count: 21327 Test sample count: 4998\n",
      "Data Pre-processing Done!\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Data Preprocessing \"\"\"\n",
    "\n",
    "if config['data_preprocessing']:\n",
    "    print 'Pre-processing Original Data ...'\n",
    "    data_preprocessing()\n",
    "    print 'Data Pre-processing Done!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Spliting and Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Split train_data into train and valid, load them into myDS**:\n",
    "    \n",
    "    1. Split train_data into train, valid by 0.8\n",
    "    \n",
    "    2. Load them into myDS, with vocabulary of all train_data and test_data.\n",
    "    \n",
    "Details in **data.py** class **myDS** and **Vocab**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data size: 21327 4998\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Read Data \"\"\"\n",
    "\n",
    "train_data = pd.read_csv('input/cleaned_train.csv')\n",
    "test_data = pd.read_csv('input/cleaned_test.csv')\n",
    "\n",
    "# split dataset\n",
    "msk = np.random.rand(len(train_data)) < 0.8\n",
    "train = train_data[msk]\n",
    "valid = train_data[~msk]\n",
    "all_sents = train_data['s1'].tolist() + train_data['s2'].tolist() + test_data['s1'].tolist() + test_data['s2'].tolist()\n",
    "\n",
    "# dataset\n",
    "trainDS = myDS(train, all_sents)\n",
    "validDS = myDS(valid, all_sents)\n",
    "\n",
    "print 'Data size:',train_data.shape[0], test_data.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Embedding\n",
    "\n",
    "**Generating or loading saved embeddings**\n",
    "\n",
    "    1. Retrieving known words' embeddings in vocabulary and initialize unknown words' embeddings including <sos>,<eos>,<unk>,<pad>\n",
    "    \n",
    "    2. Loading existing embeddings if there is one.\n",
    "    \n",
    "    3. Get embedding weight_matrix and convert to nn.Embeddings, add to config.\n",
    "    \n",
    "Details in **utils.py** function **get_embedding**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Making embedding...\n",
      "Found 5142/5774 words with embedding vectors\n",
      "Missing Ratio: 10.95%\n",
      "Filled missing words' embeddings.\n",
      "Embedding Matrix Size:  5774\n",
      "Embedding saved\n",
      "Saved generated embedding.\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Get Embedding \"\"\"\n",
    "\n",
    "full_embed_path = config['embedding']['full_embedding_path']\n",
    "cur_embed_path = config['embedding']['cur_embedding_path']\n",
    "\n",
    "if os.path.exists(cur_embed_path) and not config['make_dict']:\n",
    "    embed_dict = load_embed(cur_embed_path)\n",
    "    print 'Loaded existing embedding.'\n",
    "else:\n",
    "    print 'Making embedding...'\n",
    "    embed_dict = get_embedding(trainDS.vocab._id2word, full_embed_path)\n",
    "    save_embed(embed_dict,cur_embed_path)\n",
    "    print 'Saved generated embedding.'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make embedding weight_matrix in the same order of embed_dict, which can be refered to words embeddings when the sentences are converted into ids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = len(embed_dict)\n",
    "# initialize nn embedding\n",
    "embedding = nn.Embedding(vocab_size, config['model']['embed_size'])\n",
    "embed_list = []\n",
    "for word in trainDS.vocab._id2word:\n",
    "    embed_list.append(embed_dict[word])\n",
    "weight_matrix = np.array(embed_list)\n",
    "# pass weights to nn embedding\n",
    "embedding.weight = nn.Parameter(torch.from_numpy(weight_matrix).type(torch.FloatTensor), requires_grad = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add to config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# embedding\n",
    "config['embedding_matrix'] = embedding\n",
    "config['vocab_size'] = len(embed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model\n",
    "siamese = Siamese_lstm(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss function\n",
    "Considering the amounts of Pos and Neg samples are roughly 1:3, thus conpensate the pos samples on loss weights, set the loss weight as 1:3 (Neg:Pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loss func\n",
    "loss_weights = Variable(torch.FloatTensor([1, 3]))\n",
    "if torch.cuda.is_available():\n",
    "    loss_weights = loss_weights.cuda()\n",
    "criterion = torch.nn.CrossEntropyLoss(loss_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer\n",
    "\n",
    "Using SGD as default, learning rate 0.01.  \n",
    "\n",
    "Possible options: adam, adadelta, rmsprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# optimizer\n",
    "learning_rate = config['training']['learning_rate']\n",
    "if config['training']['optimizer'] == 'sgd':\n",
    "    optimizer = torch.optim.SGD(filter(lambda x: x.requires_grad, siamese.parameters()), lr=learning_rate)\n",
    "elif config['training']['optimizer'] == 'adam':\n",
    "    optimizer = torch.optim.Adam(filter(lambda x: x.requires_grad, siamese.parameters()), lr=learning_rate)\n",
    "elif config['training']['optimizer'] == 'adadelta':\n",
    "    optimizer = torch.optim.Adadelta(filter(lambda x: x.requires_grad, siamese.parameters()), lr=learning_rate)\n",
    "elif config['training']['optimizer'] == 'rmsprop':\n",
    "    optimizer = torch.optim.RMSprop(filter(lambda x: x.requires_grad, siamese.parameters()), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Log info to be printed during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log info\n",
    "train_log_string = '%s :: Epoch %i :: Iter %i / %i :: train loss: %0.4f'\n",
    "valid_log_string = '%s :: Epoch %i :: valid loss: %0.4f\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Restoring saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fresh start!\n"
     ]
    }
   ],
   "source": [
    "# Restore saved model (if one exists).\n",
    "ckpt_path = os.path.join(config['ckpt_dir'], config['experiment_name']+'.pt')\n",
    "\n",
    "if os.path.exists(ckpt_path):\n",
    "    print('Loading checkpoint: %s' % ckpt_path)\n",
    "    ckpt = torch.load(ckpt_path)\n",
    "    epoch = ckpt['epoch']\n",
    "    siamese.load_state_dict(ckpt['siamese'])\n",
    "    optimizer.load_state_dict(ckpt['optimizer'])\n",
    "else:\n",
    "    epoch = 0\n",
    "    print 'Fresh start!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment:siamese-baseline\n",
      "Start Training...\n",
      "2018-07-23 16:34:41.168817 :: Epoch 0 :: Iter 5000 / 17147 :: train loss: 0.5557\n",
      "2018-07-23 16:36:07.004283 :: Epoch 0 :: Iter 10000 / 17147 :: train loss: 0.5542\n",
      "2018-07-23 16:37:32.792845 :: Epoch 0 :: Iter 15000 / 17147 :: train loss: 0.4959\n",
      "Train Loss at epoch0: 0.531489431858\n",
      "Validating...\n",
      "\n",
      "2018-07-23 16:38:43.673260 :: Epoch 0 :: valid loss: 0.4599\n",
      "Model saved!\n",
      "2018-07-23 16:40:12.886336 :: Epoch 1 :: Iter 5000 / 17147 :: train loss: 0.4715\n",
      "2018-07-23 16:41:41.563174 :: Epoch 1 :: Iter 10000 / 17147 :: train loss: 0.4456\n",
      "2018-07-23 16:43:10.717947 :: Epoch 1 :: Iter 15000 / 17147 :: train loss: 0.4368\n",
      "Train Loss at epoch1: 0.449176549911\n",
      "Validating...\n",
      "\n",
      "2018-07-23 16:44:20.584676 :: Epoch 1 :: valid loss: 0.4146\n",
      "Model saved!\n",
      "2018-07-23 16:45:50.638466 :: Epoch 2 :: Iter 5000 / 17147 :: train loss: 0.3952\n",
      "2018-07-23 16:47:18.924652 :: Epoch 2 :: Iter 10000 / 17147 :: train loss: 0.4057\n",
      "2018-07-23 16:48:51.142279 :: Epoch 2 :: Iter 15000 / 17147 :: train loss: 0.3965\n",
      "Train Loss at epoch2: 0.395285815001\n",
      "Validating...\n",
      "\n",
      "2018-07-23 16:50:02.225188 :: Epoch 2 :: valid loss: 0.3757\n",
      "Model saved!\n",
      "2018-07-23 16:51:29.301383 :: Epoch 3 :: Iter 5000 / 17147 :: train loss: 0.3412\n",
      "2018-07-23 16:52:59.841305 :: Epoch 3 :: Iter 10000 / 17147 :: train loss: 0.3536\n",
      "2018-07-23 16:54:29.552551 :: Epoch 3 :: Iter 15000 / 17147 :: train loss: 0.3475\n",
      "Train Loss at epoch3: 0.346245110035\n",
      "Validating...\n",
      "\n",
      "2018-07-23 16:55:39.220740 :: Epoch 3 :: valid loss: 0.3986\n",
      "Model saved!\n",
      "2018-07-23 16:57:08.430296 :: Epoch 4 :: Iter 5000 / 17147 :: train loss: 0.3041\n",
      "2018-07-23 16:58:38.196314 :: Epoch 4 :: Iter 10000 / 17147 :: train loss: 0.2994\n",
      "2018-07-23 17:00:06.848947 :: Epoch 4 :: Iter 15000 / 17147 :: train loss: 0.3049\n",
      "Train Loss at epoch4: 0.300772458315\n",
      "Validating...\n",
      "\n",
      "2018-07-23 17:01:15.158128 :: Epoch 4 :: valid loss: 0.3627\n",
      "Model saved!\n",
      "2018-07-23 17:02:49.306927 :: Epoch 5 :: Iter 5000 / 17147 :: train loss: 0.2488\n",
      "2018-07-23 17:04:19.772324 :: Epoch 5 :: Iter 10000 / 17147 :: train loss: 0.2634\n",
      "2018-07-23 17:05:51.507928 :: Epoch 5 :: Iter 15000 / 17147 :: train loss: 0.2676\n",
      "Train Loss at epoch5: 0.260998487473\n",
      "Validating...\n",
      "\n",
      "2018-07-23 17:07:01.983421 :: Epoch 5 :: valid loss: 0.3652\n",
      "Model saved!\n",
      "2018-07-23 17:08:29.362507 :: Epoch 6 :: Iter 5000 / 17147 :: train loss: 0.2186\n",
      "2018-07-23 17:09:54.561620 :: Epoch 6 :: Iter 10000 / 17147 :: train loss: 0.2257\n",
      "2018-07-23 17:11:19.173175 :: Epoch 6 :: Iter 15000 / 17147 :: train loss: 0.2382\n",
      "Train Loss at epoch6: 0.229992002249\n",
      "Validating...\n",
      "\n",
      "2018-07-23 17:12:27.213352 :: Epoch 6 :: valid loss: 0.3514\n",
      "Model saved!\n",
      "2018-07-23 17:13:54.693773 :: Epoch 7 :: Iter 5000 / 17147 :: train loss: 0.1823\n",
      "2018-07-23 17:15:21.823692 :: Epoch 7 :: Iter 10000 / 17147 :: train loss: 0.1933\n",
      "2018-07-23 17:16:50.493504 :: Epoch 7 :: Iter 15000 / 17147 :: train loss: 0.2051\n",
      "Train Loss at epoch7: 0.19348089397\n",
      "Validating...\n",
      "\n",
      "2018-07-23 17:18:03.089794 :: Epoch 7 :: valid loss: 0.3858\n",
      "Model saved!\n",
      "2018-07-23 17:19:32.379577 :: Epoch 8 :: Iter 5000 / 17147 :: train loss: 0.1498\n",
      "2018-07-23 17:20:59.453695 :: Epoch 8 :: Iter 10000 / 17147 :: train loss: 0.1562\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process Process-53:\n",
      "Process Process-54:\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/process.py\", line 267, in _bootstrap\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/process.py\", line 267, in _bootstrap\n",
      "    self.run()\n",
      "    self.run()\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/process.py\", line 114, in run\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/process.py\", line 114, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/utils/data/dataloader.py\", line 52, in _worker_loop\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/utils/data/dataloader.py\", line 61, in _worker_loop\n",
      "    data_queue.put((idx, samples))\n",
      "    r = index_queue.get()\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/queues.py\", line 376, in get\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/multiprocessing/queues.py\", line 390, in put\n",
      "    return recv()\n",
      "    return send(obj)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/multiprocessing/queue.py\", line 21, in recv\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/multiprocessing/queue.py\", line 17, in send\n",
      "    ForkingPickler(buf, pickle.HIGHEST_PROTOCOL).dump(obj)\n",
      "    buf = self.recv_bytes()\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 224, in dump\n",
      "KeyboardInterrupt\n",
      "    self.save(obj)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 286, in save\n",
      "    f(self, obj) # Call unbound method with explicit self\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 554, in save_tuple\n",
      "    save(element)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 286, in save\n",
      "    f(self, obj) # Call unbound method with explicit self\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 606, in save_list\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-7864b05347bc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     28\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m             \u001b[0;31m# input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msiamese\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ms2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     31\u001b[0m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    489\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    490\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 491\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    492\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/PycharmProjects/lstm_siamese/model.pyc\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, s1, s2)\u001b[0m\n\u001b[1;32m     53\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0mv1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mh1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mc1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mh1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mc1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    489\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    490\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 491\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    492\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/PycharmProjects/lstm_siamese/model.pyc\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input, hidden, cell)\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m         \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0membedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m         \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlstm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    489\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    490\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 491\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    492\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input, hx)\u001b[0m\n\u001b[1;32m    176\u001b[0m             \u001b[0mflat_weight\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    177\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 178\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_forward_args\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_sizes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    179\u001b[0m         func = self._backend.RNN(\n\u001b[1;32m    180\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc\u001b[0m in \u001b[0;36mcheck_forward_args\u001b[0;34m(self, input, hidden, batch_sizes)\u001b[0m\n\u001b[1;32m    147\u001b[0m                               'Expected hidden[0] size {}, got {}')\n\u001b[1;32m    148\u001b[0m             check_hidden_size(hidden[1], expected_hidden_size,\n\u001b[0;32m--> 149\u001b[0;31m                               'Expected hidden[1] size {}, got {}')\n\u001b[0m\u001b[1;32m    150\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    151\u001b[0m             \u001b[0mcheck_hidden_size\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexpected_hidden_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/liushijing/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc\u001b[0m in \u001b[0;36mcheck_hidden_size\u001b[0;34m(hx, expected_hidden_size, msg)\u001b[0m\n\u001b[1;32m    140\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mcheck_hidden_size\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexpected_hidden_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Expected hidden size {}, got {}'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m             \u001b[0;32mif\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mexpected_hidden_size\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    143\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexpected_hidden_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "    self._batch_appends(iter(obj))\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 639, in _batch_appends\n",
      "    save(x)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 286, in save\n",
      "    f(self, obj) # Call unbound method with explicit self\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 606, in save_list\n",
      "    self._batch_appends(iter(obj))\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 639, in _batch_appends\n",
      "    save(x)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 331, in save\n",
      "    self.save_reduce(obj=obj, *rv)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 401, in save_reduce\n",
      "    save(args)\n",
      "  File \"/Users/liushijing/anaconda3/envs/python27/lib/python2.7/pickle.py\", line 284, in save\n",
      "    f = self.dispatch.get(t)\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Train \"\"\"\n",
    "\n",
    "if config['task'] == 'train':\n",
    "    \n",
    "    # save every epoch for visualization\n",
    "    train_loss_record = []\n",
    "    valid_loss_record = []\n",
    "    best_record = 10.0\n",
    "    \n",
    "    # training\n",
    "    print 'Experiment:{}\\n'.format(config['experiment_name'])\n",
    "\n",
    "    \n",
    "    while epoch < config['training']['num_epochs']:\n",
    "        \n",
    "        print 'Start Epoch{} Training...'.format(epoch)\n",
    "        \n",
    "        # loss\n",
    "        train_loss = []\n",
    "        train_loss_sum = []\n",
    "        # dataloader\n",
    "        train_dataloader = DataLoader(dataset=trainDS, shuffle=True, num_workers=2, batch_size=1)\n",
    "\n",
    "        for idx, data in enumerate(train_dataloader, 0):\n",
    "\n",
    "            # get data\n",
    "            s1, s2, label = data\n",
    "\n",
    "            # clear gradients\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # input\n",
    "            output = siamese(s1, s2)\n",
    "            output = output.squeeze(0)\n",
    "\n",
    "            # loss backward\n",
    "            loss = criterion(output, Variable(label))\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            train_loss.append(loss.data.cpu())\n",
    "            train_loss_sum.append(loss.data.cpu())\n",
    "            \n",
    "            # Every once and a while check on the loss\n",
    "            if ((idx + 1) % 5000) == 0:\n",
    "                print(train_log_string % (datetime.now(), epoch, idx + 1, len(train), np.mean(train_loss)))\n",
    "                train_loss = []\n",
    "        \n",
    "        # Record at every epoch\n",
    "        print 'Train Loss at epoch{}: {}\\n'.format(epoch, np.mean(train_loss_sum))\n",
    "        train_loss_record.append(np.mean(train_loss_sum))\n",
    "\n",
    "        # Valid\n",
    "        print 'Epoch{} Validating...'.format(epoch)\n",
    "\n",
    "        # loss\n",
    "        valid_loss = []\n",
    "        # dataloader\n",
    "        valid_dataloader = DataLoader(dataset=validDS, shuffle=True, num_workers=2, batch_size=1)\n",
    "\n",
    "        for idx, data in enumerate(valid_dataloader, 0):\n",
    "            # get data\n",
    "            s1, s2, label = data\n",
    "\n",
    "            # input\n",
    "            output = siamese(s1, s2)\n",
    "            output = output.squeeze(0)\n",
    "\n",
    "            # loss\n",
    "            loss = criterion(output, Variable(label))\n",
    "            valid_loss.append(loss.data.cpu())\n",
    "\n",
    "        print(valid_log_string % (datetime.now(), epoch, np.mean(valid_loss)))\n",
    "        # Record\n",
    "        valid_loss_record.append(np.mean(valid_loss))\n",
    "\n",
    "        epoch += 1\n",
    "\n",
    "        # Keep track of best record\n",
    "        if np.mean(valid_loss) < best_record:\n",
    "            best_record = np.mean(valid_loss)\n",
    "            # save the best model\n",
    "            state_dict = {\n",
    "                'epoch': epoch,\n",
    "                'siamese': siamese.state_dict(),\n",
    "                'optimizer': optimizer.state_dict(),\n",
    "            }\n",
    "            torch.save(state_dict, ckpt_path)\n",
    "            print 'Model saved!\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1233f8d90>]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VGXax/HvnQ4h1ISeEEoA6SX0KoIGUdBFFFTqKqIg2Mu677qL667KWkCxsIhYFhGxLKsiC0ivCZ1QE2qooQUIpD/vH2eQyAKZhEnOlPtzXXOROXPOnDsovznztCPGGJRSSvkGP7sLUEopVXI09JVSyodo6CullA/R0FdKKR+ioa+UUj5EQ18ppXyIhr5SSvkQDX2llPIhGvpKKeVDAuwu4Erh4eEmOjra7jKUUsqjrFu37oQxJqKg/dwu9KOjo0lISLC7DKWU8igist+Z/bR5RymlfIiGvlJK+RANfaWU8iEa+kop5UM09JVSyodo6CullA/R0FdKKR/iNaGfk5vH33/azqEzF+0uRSml3JbXhH7K6YvMWHuAIR+v4XR6lt3lKKWUW/Ka0I8OD+WfQ2I5ePoiIz6N50JWjt0lKaWU2/Ga0AdoX6cSkwa2YNPBM4yZsYHs3Dy7S1JKKbfiVaEPENekGuP7NeGXHcf5w7dbMMbYXZJSSrkNt1twzRUebF+L4+cymbRwNxFhwTwX19DukpRSyi14ZegDPNkzhtRzmby/OJmIsGCGd6ptd0lKKWU7rw19EeGvdzXh5PlMxv+wjfAywdzZvLrdZSmllK28rk0/P38/YdKglrSpVZGnZm1kRdIJu0tSSilbeXXoA4QE+vPPIbHUCS/DI5+vY+uhNLtLUkop23h96AOUKx3IpyPaUq5UIMM+iWf/yXS7S1JKKVv4ROgDVC0Xwqcj2pKTl8eQaWtJPZdpd0lKKVXifCb0AepVLsO0YW04djaD4dPXcj5TZ+0qpXyLT4U+QKuoCrz/QCu2HznHqM/XkZWjs3aVUr7D50IfoEfDKrz2u6YsTzrBM19vIi9PZ+0qpXyD147TL8iA2EhSz2fyxs87CS8TzP/dcRMiYndZSilVrJy60heROBHZKSJJIvLCVV4fJiKpIrLR8Xgo32tDRWS34zHUlcXfqEe71WVYx2imrdjLR0v32F2OUkoVuwKv9EXEH5gM9AJSgHgRmWOM2XbFrl8ZY8ZccWxF4GUgFjDAOsexp11S/Q0SEf50RyNOnM/ktbk7iCgTTP/WNe0uSymlio0zV/ptgSRjzB5jTBYwE+jn5PvfBsw3xpxyBP18IK5opRYPPz/hzXub06leJZ77ZjOLdh63uySllCo2zoR+DeBgvucpjm1X6i8im0VktohEFvJYWwUH+PPhg61pWDWMx75Yz4YDbvFFRCmlXM5Vo3f+A0QbY5phXc1/WpiDRWSkiCSISEJqaqqLSiqcsJBApg9vS0RYMCOmx5Ocet6WOpRSqjg5E/qHgMh8z2s6tv3KGHPSGHNpiutUoLWzxzqOn2KMiTXGxEZERDhbu8tFhAXz2Yi2+PsJQz5ey7GzGbbVopRSxcGZ0I8HYkSktogEAQOBOfl3EJFq+Z72BbY7fp4H3CoiFUSkAnCrY5vbig4PZfrwtpy5kMXQaWtJu5htd0lKKeUyBYa+MSYHGIMV1tuBWcaYRBEZLyJ9HbuNFZFEEdkEjAWGOY49BbyC9cERD4x3bHNrTWqU46PBsSSnnufhzxLIyM61uySllHIJcbd7yMbGxpqEhAS7ywBgzqbDjP1yA3GNqzL5gVb4++nkLaWUexKRdcaY2IL288llGJzVt3l1/nRHI35OPMqf/r1Vb7KulPJ4PrsMg7NGdK7N8XOZfLgkmcphIYzrGWN3SUopVWQa+k54Pq4BqecyeXvBLiLCgrm/XZTdJSmlVJFo6DtBRHitf1NOpWfyx++3UKlMELc1rmp3WUopVWjapu+kQH8/Jj/QimY1y/P4lxtYu9ftByEppdT/0NAvhNJBAUwb1oaaFUrx0Kfx7Dx6zu6SlFKqUDT0C6liaBCfjWhLqSB/hk5by6EzF+0uSSmlnKahXwQ1K5Tm0xFtSc/KYcjHazidnmV3SUop5RQN/SJqWLUsU4fEcvD0RUZ8Gs+FLL3JulLK/Wno34B2dSoxaWBLNh08w5gZG8jO1ZusK6Xcm4b+DYprUpVX7mrCLzuO84dvt+isXaWUW9Nx+i7wQLtaHD+bycSFu4kIC+a5uIZ2l6SUUleloe8iT/SMIfV8Ju8vTiYiLJjhnWrbXZJSSv0PDX0XERFe6deEk+czGf/DNsLLBHNn8+p2l6WUUr+hbfou5O8nTBzYkja1KvLUrI2sSDphd0lKKfUbGvouFhLozz+HxlInvAyPfL6OrYfS7C5JKaV+paFfDMqVCuTTEW0pVyqQYZ/Ec+DkBbtLUkopwJtCPzcbvhwESQvsrgSAquVC+HREW3Lz8hg8bQ2p5zILPkgppYqZ94R+Wgqc2AVf9IdZQyDtkN0VUa9yGT4e1oZjZzMYPn0t5zN11q5Syl7eE/oVa8OjK6HHH2HXPHivDax81/oGYKNWURV4/4FWbD9yjlGfryMrR2ftKqXs41Toi0iciOwUkSQReeE6+/UXESMisY7n0SJyUUQ2Oh4fuqrwqwoIhq7Pwug1EN0Z/vtH+Kgr7F9VrKctSI+GVXi9fzOWJ53gma83kZens3aVUvYoMPRFxB+YDPQGGgGDRKTRVfYLA8YBa654KdkY08LxGOWCmgtWIRru/woGzoDMc/BJHHz/GKTbN4TyntY1eT6uIXM2HeavP27X5RqUUrZw5kq/LZBkjNljjMkCZgL9rrLfK8DrQIYL6ys6EWjYx7rq7/wkbP4K3m0NCdMgL9eWkkZ1q8PwTtFMW7GXj5busaUGpZRvcyb0awAH8z1PcWz7lYi0AiKNMT9e5fjaIrJBRJaISJeil1pEQaHQ888wagVUbQo/PAkf94LDG0q8FBHh//o04s7m1Xlt7g5mJRws+CCllHKhG+7IFRE/4C3g6au8fASIMsa0BJ4CZohI2au8x0gRSRCRhNTU1Bst6eoqN4Sh/4HfTYUzB+GfPeCnZ+HimeI53zX4+Qn/GNCMzvXCeW72Zv48J5GMbHu+eSilfI8zoX8IiMz3vKZj2yVhQBNgsYjsA9oDc0Qk1hiTaYw5CWCMWQckA/WvPIExZooxJtYYExsREVG038QZItBsAIyJhzYPQ/xUa5TPpq+gBNvYgwP8mTo0luGdopm+ch93v7+SpON6v12lVPFzJvTjgRgRqS0iQcBAYM6lF40xacaYcGNMtDEmGlgN9DXGJIhIhKMjGBGpA8QA9jdmlyoPt78BDy+C8pHw3UiYfgcc31FiJYQE+vPynY2ZNiyWY2czuOPd5cxce0A7eJVSxarA0DfG5ABjgHnAdmCWMSZRRMaLSN8CDu8KbBaRjcBsYJQx5tSNFu0y1VvA7xfAHe/Asa3wYSeY/zJkpZdYCT0aVuHncV1oXasCL3y7hdEz1pN2wd65BUop7yXudmUZGxtrEhISSv7E6SeswN/4BZSLhLjXrNE/IiVy+rw8w0dL9/Dmf3dSpWwIEwe2IDa6YomcWynl+URknTEmtqD9vGdG7o0KDYe7JsOIeRBcFr56AGbcB6f2lsjp/fyER7vXZfajHfH3E+79aBUTF+wmVydyKaVcSEP/SlHt4ZElcOursH8FvN8elkyAnJJZMK1FZHl+HNuZfi1q8PaCXQz652oOn7lYIudWSnk/Df2r8Q+EjmOsUT4NesOiv8L7HSD5lxI5fVhIIG/f14K37m1O4qE0ek9cxs9bj5TIuZVS3k1D/3rKVocB0+HBbwEDn98NXw+Ds4dL5PS/a1WTH8d2oVal0oz6Yj1/+G4LF7N0TL9Squg09J1R7xZ4dBXc/BLs+Mka279qMuQW/1LJ0eGhzB7VkUe61WHGmgP0fW85O46eLfbzKqW8k4a+swJDoNtzMHo11OoI8/4AU7rBgdXFfuqgAD9e7H0Tn41oy+kL2fR9bwWfrdqnY/qVUoWmoV9YFevA/bPgvi+sJRym3Qb/Hg3pJ4v91F3rR/DzE13oVLcSf/p3Ig9/to5T6VnFfl6llPfQ0C8KEbjpThizFjo9AZtmwnutYd10yCvem6SElwlm2rA2/OmORizdlUrviUtZmWzfktFKKc+ioX8jgkKh11+sFTwrN4b/jLNW8DyyqVhPKyKM6Fybbx/rSGhwAA9MXcOEeTvIztW7cimlrk9D3xUqN4RhP8DdU+DMfpjSHX56DjLSivW0TWqU44fHO3Nv60gmL0rm3o9WcfDUhWI9p1LKs2nou4oINL8PxiRA7O9h7RRrlM/mr4t1Bc/SQQG8fk8z3h3UkqRj57l94jL+vdH+m8IrpdyThr6rlSoPff4BIxdB2Rrw7UPw6Z2QurNYT3tn8+r8NK4LMVXKMG7mRp75ehPpmcU/pFQp5Vk09ItL9Zbw0ALo8xYc3QwfdIIFf4Gs4mt+iaxYmlmPdGBsj3p8sz6FO95dztZDxdvEpJTyLBr6xcnPH9r8Hsasg6YDYPlbMLmdNcGrmAT4+/HUrQ2Y8VB7Lmblcvf7K5i6bA95unCbUgoN/ZJRJgLu/gCGz4XgMjBzEMwYCKf3FdspO9StxNxxXbi5QWX++uN2hk+PJ/VcySwap5RyXxr6JalWR3hkKdz6V9i71OronfsCnC+e+wJXCA3io8GteeWuJqzec5LeE5exdFcx3YNYKeURNPRLmn8gdHzcWsGz+UBrlM/E5rDwlWK5SbuIMLh9LeaM6UzF0ECGTFvLqz9uIytHx/Qr5Yv0zll2O5EEi/8GW7+BkHLWDN92j1gTv1wsIzuXV3/czuer99O0RjkmDWpJ7XDXn0cpVfKcvXOWhr67OLIZFr0Ku36G0MrQ9VloPRQCgl1+qnmJR3n+m81k5eQxvl8T+reqgZTQbSGVUsVDb5foaao1g/u/ghH/hfD6MPdZeDcWNvzL5Us439a4KnPHdaFpjXI88/UmnvhqI+cy9GbsSvkCp0JfROJEZKeIJInIC9fZr7+IGBGJzbftRcdxO0XkNlcU7dWi2llLOgz+DkIrwb8fgw86QOL3Ll3MrVq5Usx4uD1P96rPD5uPcPukZWw4cNpl76+Uck8Fhr6I+AOTgd5AI2CQiDS6yn5hwDhgTb5tjYCBQGMgDnjf8X7qekSgbg94eJG1hLP4wddD4Z/dYfcCly3r4O8nPH5LDLMeaU9eHgz4cBXvL07SMf1KeTFnrvTbAknGmD3GmCxgJtDvKvu9ArwOZOTb1g+YaYzJNMbsBZIc76eccWkJ50dXwt0fWaN7/tUfPrkd9q902Wla16rIT+O6ENekKm/8vJMHP17DsbMZBR+olPI4zoR+DeBgvucpjm2/EpFWQKQx5sfCHquc4OdvDe8ckwB93oRTe+CT3vDFPXB4o0tOUa5UIO8Oaskb/Zux4cAZ4t5ZyoJtx1zy3kop93HDHbki4ge8BTx9A+8xUkQSRCQhNVUnD11TQBC0eQjGboBe4+FQgnXLxllDIXXXDb+9iHBvm0h+GNuZauVK8dBnCfx5TiIZ2R5yM/aLZ2DLbJg9wvo7ORhvd0VKuZ0Ch2yKSAfgz8aY2xzPXwQwxvzd8bwckAycdxxSFTgF9AV6XbHvPMd7rbrW+Xx2yGZRZKRZN2hfNRmyL0Dz+6H781A+6obfOjMnl9fn7mTair00rBrGu4NaElMlzAVFu9iZA7BzLuz4EfavgLwca8hrXjZcPA21u1nDX6M7W81lSnkpl43TF5EAYBdwC3AIiAfuN8YkXmP/xcAzxpgEEWkMzMBqx68OLARijDHXvHTU0C+C9BOw/G1Y+08weRA7Aro8DWFVbvitF+04bi3TnJXDC3ENGdIhGj8/G8PTGOvOZDvnws4f4egWa3t4A2h4OzS4HWrEWh+C6z6Ble/C+WMQ2R66PgP1emr4K6/k0slZInI78A7gD0wzxrwqIuOBBGPMnCv2XYwj9B3PXwJGADnAE8aYudc7l4b+DUg7BEvfgPWfW5O62o2CTmOhVIUbetvjZzN47pvNLN6ZSpvoCrxxT/OSncmbkwX7l1urk+6cC2dTrBFNke2skG/YByrVvfqx2Rdhwxew/B3ruGrNrSv/Bn3AT6epKO+hM3J92clkWPx3q307uKwV/O1GWSt8FpExhm/WH2L8fxLJzMnjmVsbMKJzbfyL66o/Iw12z4edP1l/Zp6FgFLWUNaGt0P9OAgNd/79crJg81fW8tan9kDETdaVf+O7rY5ypTychr6Co1utpR12/gShEdDlGYgdfkNLOxw7m8FL321lwfZjtIwqz4R7mlGvsova+s8cvNxss2+5o30+wgr4hn2gTncILHVj58jNgcTvYNmbkLodKtaFzk9Cs/usjnKlPJSGvrrsYDws/AvsWwZla0L3F6D5IPAPKNLbGWOYs+kwL89J5EJWLk/0jGFklzoE+BeyucQY665iO36yPpiObra2V4pxtM/3gZqxxXMlnpdnfbgsnWD1EZSLhE7joOVgCAxx/fmUKmYa+up/7VkMC8fDoXVQqR7c/BI0uqvIbdup5zL507+3MnfrUZrWKMeEAc1oWLXs9Q/Kzbau4nc62ufTDgJitc9f6ogNjylSPUViDCQtgCVvQMpaKFMFOo61vhEVw0qnSl3TyWQ4exhqdynS4Rr66uqMsQJ34StW80bVptDjTxDTq8ijWn7acoT/+34rZzOyGXNzDI/dXJfA/Ff9GWlWsO641D6f5mifv9kK+fpx1t3F7GSM9U1o6QTrBjelKkKHx6DtSGvJa6WKy6F1sGIibJtjXfCMXlukf4sa+ur68nKtNfwXvWrdtjGyPdzyJ4juVKS3O5WexZ/nJDJn02FuqlaWd+LCaZC23Bo/v2+5NW6+dDg0iLOabep0h6DSrvyNXOfgWlj6D9g9D4LLQbuR0O5RawE8pVzBGEhaCCvesS42gstZ99NuN6rIQ6019JVzcrNhw+dW88a5I1D3Frjl/6B6y8K9jzFwdAtJy2eRnfgjN7EHgLyK9fBr6BhWWbONZ42UObLJCv/tcyAwFNqMgA6Pu2T+g/JRuTmQ+K11ZX9sK4RVt75RthoKIQU0jRZAQ18VTvZFiJ8Ky96Ci6fgpr5Wm3/lhtc+JjfbmgV7afx82gFAyKkRy9zsVrx9MIaAyvWZcE9zmkeWL7FfxeWOb7f+XrbOBr9AaDXE6vQtH2l3ZcpTZKVb82dWTbb+nYQ3sIZSN73XZaPGNPRV0WSchdXvw8r3IDsdmg20lnaoEH359aT5Vsjv/q/VXh8QAnVuvjx+vkxlwJrN++K3Wzh+LoOHu9bhyZ71CQn0oCv9K51Mtr6Ob/wSMNYIqM5PXntimFLpJ2HtR9a9sC+etppRO42z/p24eHKghr66MeknYYVjaYe8XGsc+7nDsHeZo32+EtTvbQV9ne7XHOlyNiObv/24nZnxB6kTEcqEe5rRulbFEv1VXO7MQVg5CdZ9av1dNLkHujwFlW+yuzLlLk7vsy6cNnwBORetAQudxkFU+2I7pYa+co2zh60RLes/g/K1Lo+fj2xbqPb5ZbtTeeGbLRxOu8iITrV55tYGlAry4Kt+gHPHYNV7EP+x9a3opjutCXDVW9hdmbLLkU1We33idyD+1sVSp7EQ0aDYT62hr1wrNxv8Am5osbLzmTm8Nnc7X6w+QHSl0rzevxnt6njBiJgLp2D1B7DmI2s4asyt1vo+kXq/IJ9gDOxdYq3vtGcRBIVB7DBo/xiUrV5iZWjoK7e1MvkEL3yzhQOnLjC0Qy2ei2tIaHDRZge7lYw0qzls1WSrMzy6ixX+tbvqyp7eKDfHGtm1YiIc2Wgt6d3+UWuV21IlP3BBQ1+5tQtZOUyYt5PpK/dRo3wpXu/fjE71CrGAmjvLSoeET6x2//PHoGZbK/xvYAKcciPZF2Hjv6xlu0/vs9Zv6jTWGvRg4xIeGvrKI8TvO8Vzszez90Q6g9pG8YfbGxIWEmh3Wa6RnQEbHcs6px2Eqs2s8G94hy7r7IkunLL6b9Z8CBdOQI3W0OkJaw6KG8w/0dBXHiMjO5e35u9i6rI9VC0bwt/7N6NbfZuXZXCl3GxrWedljvsbRzS0bnLT+HdFXvSuWOXlWSNOcjKtq9qcDMefmdb2oDLW7+ArC9OdOWgNY173qdVhH3OrFfa1OrrVNzcNfeVxNhw4zbOzN5N0/DwDWtfkj30aUa60l1z1gzX0NfE7a5Zv6naoUNsa6tls4LUn6OTmXCOAMxw/Z1ivZ2fk21bU1x2hnptV8O8i/taIlCpNrPWbLj0Kc48Dd3csEVZMsiblgTU0t9NYqNLY3rquQUNfeaSM7Fze/WU3Hy7ZQ6XQIP52d1N6NvKyZQ/y8qxF75ZOsDoAy1S1FpzLzvjfq+q8nKKfxy/Quv9AQIh1VR7geFza9uv2Upf/DAjOd4zjef7XA0OsZo5jW61bVR7dAmcPXT5nWLX//SCoWMctmj+cYow1y3zFRGvyYWAotB5qjcRx8xnYGvrKo21JSePZ2ZvYcfQcd7Wozst3NqZCqJfd5OTSolvrp1tX9FcL5qsF7/8E89UCPKTkgvbCqcsfAJc+DFJ3XP7ACixtXR3/+mHQDKo0cq+lq/NyrcUBV0yEQwnW4oDtRlmLoJX2jMmEGvrK42Xl5DF5URKTFyVRvnQQf72rMXFNqtldlnJGTiak7rziw2CzNawVALGWr8j/QVC1ifVNoSTbybMzYPNMayTOySRruZGOj0OLB278Lm0lTENfeY1th8/y7OxNJB4+S59m1RjftzGVyhT9lo/KJsZYo5iOXmoa2mx9GJzed3mf0pWsD4EqTS5/EITXB38X9+1cPAMJ06yROOePQbXmVudso36e0xR1BQ195VWyc/P4aEkykxYmUSYkgL/0bcwdzaohbjR6QhVRRprVaXrU8W3g6BZrZdPcTOt1/yBrXaMq+foJqjQu2gSos4etkTgJ0yHrHNTtYa2JU7ubW43EKQqXhr6IxAETAX9gqjHmtSteHwWMBnKB88BIY8w2EYkGtgM7HbuuNsaMut65NPTV9ew6do5nv97EppQ0bmtchVfuakLlMB8ZOuhLcnPg5O7ffhAc3WKNj7+kfJSjfyBfx3H5qKuHd+pOayTO5q/A5FrDZTuNta7wvYTLQl9E/IFdQC8gBYgHBhljtuXbp6wx5qzj577AY8aYOEfo/2CMaeJs4Rr6qiA5uXl8vHwvb87fRalAf16+sxF3t6yhV/3ezhirKebSB8Clx8kkwJFjweWsJqFLHwRlKsO66dZoqYBS0GowdBh9ealwL+Js6DszM6QtkGSM2eN445lAP+DX0L8U+A6h/PpfQCnXC/D345FudenZqArPzd7MU7M28ePmI7x6d1OqltOrfq8lAmFVrUdMr8vbs9Kt5qCjmy/3F2z4wppIBVCqAnR7wbrfsd7y0qkr/XuAOGPMQ47ng4F2xpgxV+w3GngKCAJ6GGN2O670E7G+KZwF/miMWXaVc4wERgJERUW13r9//w3+WspX5OYZpq/cx4R5Owj09+P/+jRiQGxNver3dXl5cHqv9Yjq4F7DQ4uJs1f6LlsAxBgz2RhTF3ge+KNj8xEgyhjTEusDYYaI/M+NII0xU4wxscaY2IgIL5p+r4qdv5/w+861+XlcVxpVK8tz32xmyLS1pJy+YHdpyk5+ftaQ0Ho9fSLwC8OZ0D8E5J+KVtOx7VpmAncBGGMyjTEnHT+vA5KB+kUrValriw4P5cuH2/NKv8as23+aW99eypSlyWTn5tldmlJuxZnQjwdiRKS2iAQBA4E5+XcQkZh8T/sAux3bIxwdwYhIHSAG2OOKwpW6kp+fMLhDNPOe6ErHupX42087uGPScuL3nbK7NKXcRoGhb4zJAcYA87CGX84yxiSKyHjHSB2AMSKSKCIbsZpxhjq2dwU2O7bPBkYZY/RfoCpWkRVLM3VoG6YMbs35zBwGfLiKZ7/exMnzmXaXppTtdHKW8moXsnKYtDCJqcv2UCYkgOfjGnJfbCR+ftrRq7xLiXfkKuWOSgcF8ELvhswd14UGVcJ48dst9P9wJYmH0wo+WCkvpKGvfEJMlTBmjmzPmwOac+DkBe58dznj/7ONcxnZdpemVInS0Fc+Q0To37omvzzdnUFto/hk5V56vrWEHzYfxt2aOZUqLhr6yueUKx3Iq3c35bvHOhERFsyYGRsYMm0te0+k212aUsVOQ1/5rBaR5fn36M78pW9jNh44w23vLOXt+bvIyM61uzSlio2GvvJp/n7C0I7RLHy6G72bVGXiwt3c9s5SluxKtbs0pYqFhr5SQOWyIUwc2JJ/PdQOfxGGTlvLY/9ax9G0DLtLU8qlNPSVyqdTvXDmPtGFZ26tz8Ltx7nlzcVMXbaHHF3OQXkJDX2lrhAc4M+YHjHMf7IbbWtX5K8/bueOd5ezbr9OJleeT0NfqWuIqlSaacPa8OGDrUm7mE3/D1bx/OzNnE7Psrs0pYpMQ1+p6xAR4ppUZcFT3Xikax2+WZ9CjzcX81X8AfLydGy/8jwa+ko5ITQ4gBdvv4kfx3YhpnIYz3+zhQEfrWL7kbMFH6yUG9HQV6oQGlQN46tH2jPhnmbsPZHOHe8u568/bON8Zo7dpSnlFA19pQpJRBgQG8kvT3fjvjaRfLxiLz3fXMJPW47ocg7K7WnoK1VE5UsH8be7m/LNox2pGBrEY/9az7BP4tmnyzkoN6ahr9QNahVVgTljOvHynY2sWzW+s5SJC3brcg7KLWnoK+UCAf5+DO9Um4VPd+PWRlV4e8Euek9cxrLdupyDci8a+kq5UJWyIbx3fys+/31bAAZ/vJbRM9Zz7Kwu56Dcg4a+UsWgS0wEc8d14ale9Zm/7Ri3vLmEj5fv1eUclO009JUqJiGB/oy9JYb5T3alda0KvPLDNvq+t4L1B07bXZryYU6FvojEichOEUkSkReu8vooEdkiIhtFZLmINMr32ouO43aKyG2uLF4pT1CrUijTh7fhgwdacSo9i9+9v5IXv93MmQu6nIMqeVLQuGIR8Qd2Ab2AFCAeGGSM2ZZvn7LGmLOq1CySAAAQqElEQVSOn/sCjxlj4hzh/yXQFqgOLADqG2OuOawhNjbWJCQk3NhvpZSbOp+Zw8QFu5i2Yh/lSgXyQu+G3NOqJn5+YndpysOJyDpjTGxB+zlzpd8WSDLG7DHGZAEzgX75d7gU+A6hwKVPkn7ATGNMpjFmL5DkeD+lfFKZ4ABe6tOIHx7vTO3wUJ6bvZn7pqxic8oZu0tTPsKZ0K8BHMz3PMWx7TdEZLSIJANvAGMLc6xSvuamamX5+pEOvHFPM5JT0+n73gpGz1iv9+lVxc5lHbnGmMnGmLrA88AfC3OsiIwUkQQRSUhN1XHNyjf4+Qn3xkay5NnujO1Rj0U7jtPzrSW89N0WjusQT1VMnAn9Q0Bkvuc1HduuZSZwV2GONcZMMcbEGmNiIyIinChJKe8RFhLIU7c2YMmzN/NAuyi+ij9ItwmLmTBvB2kXs+0uT3kZZ0I/HogRkdoiEgQMBObk30FEYvI97QPsdvw8BxgoIsEiUhuIAdbeeNlKeZ+IsGDG92vCwqe70atRFSYvSqbbhEVMWZqsSzoolykw9I0xOcAYYB6wHZhljEkUkfGOkToAY0QkUUQ2Ak8BQx3HJgKzgG3Az8Do643cUUpZQzwnDWrJD493plnN8vztpx3c/I/FzIo/qJO71A0rcMhmSdMhm0r91srkE7z+8042HTxDvcplePa2BtzaqAoiOsxTXebKIZtKKRt1rBvO94915MMHW5FnDI98vo7+H6xkzZ6TdpemPJCGvlIewLpXbzX++0RXXvtdUw6fyeC+KasZ/slavWWjKhRt3lHKA2Vk5zJ95T7eX5TEucwc7mpRg6d61SeyYmm7S1M2cbZ5R0NfKQ+WdiGbD5Yk88mKveQZwwPtajGmRz3CywTbXZoqYRr6SvmQo2kZTFy4i1kJKYQE+PFw1zo81KUOZYID7C5NlRANfaV8UNLx87z5353M3XqUSqFBPN6jHoPaRREc4G93aaqY6egdpXxQvcpl+ODB1nw/uhP1q4Tx5/9s45Y3l/D9hkPk5bnXBZ6yh4a+Ul6oRWR5Zjzcjk9HtKVsSCBPfLWR2yctY9GO47jbt3tVsjT0lfJSIkK3+hH88HhnJg5swYWsXIZPj+e+Kav17l0+TENfKS/n5yf0a1GDBU91Y3y/xuxJPc/v3l/JyM8SSDp+zu7yVAnTjlylfEx6Zg4fL9/LlKV7uJCVwz2ta/JEz/pUL1/K7tLUDdDRO0qp6zp5PpPJi5L5YvV+EBjWMZrHutelfOkgu0tTRaChr5RySsrpC7w1fxffbThEmeAARnWry4hOtSkVpMM8PYmGvlKqUHYcPcs/5u1kwfbjVA4LZlzPGO6NjSTQX7v+PIGO01dKFUrDqmWZOrQNX4/qQGTF0rz03VZufXspP24+osM8vYiGvlLqN9pEV2T2qA5MHRJLoL8wesZ6+k1ewYqkE3aXplxAQ18p9T9EhJ6NqjB3XFf+MaA5J89n8cDUNQz+eA0bdIy/R9M2faVUgTKyc/li9X4mL0ri9IVsmtcsx+AO0dzRrBohgdrh6w60I1cp5XLnMrL5Zl0Kn6/eT3JqOhVKB3Jvm0gebFdL1/K3mYa+UqrYGGNYlXySz1btZ/72Y+QZw80NKjO4Qy26xUTg56f37y1pLg19EYkDJgL+wFRjzGtXvP4U8BCQA6QCI4wx+x2v5QJbHLseMMb0vd65NPSV8ixH0i7y5ZoDzFh7kBPnM4mqWJoH20dxb2ykTvQqQS4LfRHxB3YBvYAUIB4YZIzZlm+fm4E1xpgLIvIo0N0Yc5/jtfPGmDLOFq6hr5RnysrJ4+fEo3y+ah/x+04THOBH3+bVGdIhmqY1y9ldntdzNvSdua1OWyDJGLPH8cYzgX7Ar6FvjFmUb//VwIOFK1cp5emCHCHft3l1th85y+er9/P9hkN8vS6F5pHlGdK+Fn2049d2zgzZrAEczPc8xbHtWn4PzM33PEREEkRktYjcVYQalVIe5qZqZfnb3U1Z/YdbePnORpzLyObprzfR8bVfeG3uDg6eumB3iT7LpTfQFJEHgVigW77NtYwxh0SkDvCLiGwxxiRfcdxIYCRAVFSUK0tSStmobEggwzvVZljHaFYmn+SzVfuYsjSZj5Ym08PR8dtVO35LlDOhfwiIzPe8pmPbb4hIT+AloJsxJvPSdmPMIcefe0RkMdAS+E3oG2OmAFPAatMv3K+glHJ3IkKneuF0qhfO4TMX+XLtAb5ce5CFn8RTq1JpHmxXiwGxNbXjtwQ405EbgNWRewtW2McD9xtjEvPt0xKYDcQZY3bn214BuGCMyRSRcGAV0C9/J/CVtCNXKd+QlZPH3K1H+GL1/l87fvu1sDp+m9TQjt/CcllHrjEmR0TGAPOwhmxOM8Ykish4IMEYMweYAJQBvhYRuDw08ybgIxHJw+o/eO16ga+U8h1BAX70a1GDfi1qsO3w5Y7fWQkptIgsz5AOtbi9qXb8uppOzlJKuY20i9l8u96a8bsnNZ2KoUHc1yaS+9tG6YzfAuiMXKWUxzLGsCLJ6vhdsP0YAD0aVmZwh2i61AvXjt+rcOU4faWUKlEiQueYcDrHWB2/M9YcYGb8ARZsX0t0pdI82L4WA1pHUq50oN2lehy90ldKeYTMnFx+3nqUz1btZ93+04QE+tGveQ0Gd6ilHb9o845SyoslHk7ji9X7+X7DYS5m59IqqjyDHR2/wQG+2fGroa+U8nppF7OZvS6FL1bvZ++JdCpd6vhtF0XNCr7V8auhr5TyGXl5hhXJJ/hs1X4W/trxW4UhHWrRJSYcx1Byr6YduUopn+HnJ3SJiaBLTASHzlxkxpr9zFx7kAXbj9G4elke7xHDrY2q6Kgf9EpfKeWlMnNymbPxMJMXJbHv5AUaVg1jTI969G5SDX8vDH9t3lFKKSAnN48fNh/h3V92k5yaTr3KZXi8Rz3uaFbdq8JfQ18ppfLJzTPM3XqEdxcmsfPYOeqEh/LYzfW4q0V1AvydWWXevWnoK6XUVeTlGf677SiTFiax7chZoiqWZvTNdbm7ZU2CAjw3/DX0lVLqOowxLNx+nEm/7GZzSho1ypfi0e51GRBb0yPH+mvoK6WUE4wxLN6VyqSFu9lw4AxVy4bwaPe63Ncm0qNW+NTQV0qpQri0yNukhbtZu+8UEWHBPNK1Dg+0q0WpIPcPfw19pZQqotV7rPBfmXyS8DJBPNSlDoPb1yI02H2nNmnoK6XUDYrfd4pJC3ezbPcJKpQO5KEudRjSoRZhIe63uqeGvlJKuciGA6d595ckftlxnLIhAYzoXJvhHWu71dLOGvpKKeViW1LSmPTLbuZvO0ZYcADDOkUzolNtKoTaf0N3DX2llCom2w6f5b1Fu/lpy1FCg/wZ3CGah7vUplKZYNtq0tBXSqlitvPoOd5blMQPmw8TEuDPg+2jeLhrHSqHhZR4Lc6GvlPTz0QkTkR2ikiSiLxwldefEpFtIrJZRBaKSK18rw0Vkd2Ox9DC/RpKKeW+GlQN491BLZn/ZDd6N6nKx8v30uX1Rfx5TiJH0zLsLu+qCrzSFxF/YBfQC0gB4oFBxpht+fa5GVhjjLkgIo8C3Y0x94lIRSABiAUMsA5obYw5fa3z6ZW+UspT7TuRzuRFSXy34RB+ItzXJpJR3etSo3ypYj+3K6/02wJJxpg9xpgsYCbQL/8OxphFxpgLjqergZqOn28D5htjTjmCfj4Q5+wvoZRSniQ6PJQJA5qz6Jnu9G9dk5nxB+g+YREvfruZg6cuFPwGJcCZ0K8BHMz3PMWx7Vp+D8wt4rFKKeXxIiuW5u+/a8riZ29mUNsovll3iO7/WMyzX29i34l0W2tz6fQyEXkQqymnWyGPGwmMBIiKinJlSUopZZsa5Usxvl8THutej4+WJjNjzQG+WZ9CvxY1GH1zPepVLlPiNTlzpX8IiMz3vKZj22+ISE/gJaCvMSazMMcaY6YYY2KNMbERERHO1q6UUh6harkQXr6zMcuev5mHutTh561H6fX2EsbMWM/Oo+dKtBZnOnIDsDpyb8EK7HjgfmNMYr59WgKzgThjzO582ytidd62cmxaj9WRe+pa59OOXKWUtzt5PpOpy/fy2cp9pGfl0rtJVcb0qEfj6uWK/J4uHacvIrcD7wD+wDRjzKsiMh5IMMbMEZEFQFPgiOOQA8aYvo5jRwB/cGx/1RjzyfXOpaGvlPIVp9Oz+GTFXj5ZsY9zmTn0aVaN9wa1RKTwt3HUyVlKKeUh0i5mM33FPrJyc3n2toZFeg9nQ9991wlVSikfUa5UION6xpTIuTz3hpBKKaUKTUNfKaV8iIa+Ukr5EA19pZTyIRr6SinlQzT0lVLKh2joK6WUD9HQV0opH+J2M3JFJBXYfwNvEQ6ccFE5xc2TagXPqteTagXPqteTagXPqvdGaq1ljClwxUq3C/0bJSIJzkxFdgeeVCt4Vr2eVCt4Vr2eVCt4Vr0lUas27yillA/R0FdKKR/ijaE/xe4CCsGTagXPqteTagXPqteTagXPqrfYa/W6Nn2llFLX5o1X+koppa7Ba0JfROJEZKeIJInIC3bXcz0iMk1EjovIVrtrKYiIRIrIIhHZJiKJIjLO7pquR0RCRGStiGxy1PsXu2sqiIj4i8gGEfnB7loKIiL7RGSLiGwUEbe+25GIlBeR2SKyQ0S2i0gHu2u6FhFp4Pg7vfQ4KyJPFMu5vKF5R0T8se7j2wtIwbqP7yBjzDZbC7sGEekKnAc+M8Y0sbue6xGRakA1Y8x6EQnDuufxXW78dytAqDHmvIgEAsuBccaY1TaXdk0i8hQQC5Q1xtxhdz3XIyL7gFhjjNuPexeRT4FlxpipIhIElDbGnLG7roI48uwQ0M4YcyNzlq7KW6702wJJxpg9xpgsYCbQz+aarskYsxS45s3h3Ykx5ogxZr3j53PAdqCGvVVdm7GcdzwNdDzc9spGRGoCfYCpdtfiTUSkHNAV+BjAGJPlCYHvcAuQXByBD94T+jWAg/mep+DGweSpRCQaaAmssbeS63M0l2wEjgPzjTHuXO87wHNAnt2FOMkA/xWRdSIy0u5irqM2kAp84mg6myoioXYX5aSBwJfF9ebeEvqqmIlIGeAb4AljzFm767keY0yuMaYFUBNoKyJu2YQmIncAx40x6+yupRA6G2NaAb2B0Y6mSncUALQCPjDGtATSAbfu6wNwNEP1Bb4urnN4S+gfAiLzPa/p2KZcwNE2/g3wL2PMt3bX4yzH1/lFQJzdtVxDJ6Cvo518JtBDRL6wt6TrM8Yccvx5HPgOq2nVHaUAKfm+5c3G+hBwd72B9caYY8V1Am8J/XggRkRqOz4pBwJzbK7JKzg6Rj8Gthtj3rK7noKISISIlHf8XAqrc3+HvVVdnTHmRWNMTWNMNNb/s78YYx60uaxrEpFQR2c+jqaSWwG3HIFmjDkKHBSRBo5NtwBuOfjgCoMoxqYdsL4CeTxjTI6IjAHmAf7ANGNMos1lXZOIfAl0B8JFJAV42Rjzsb1VXVMnYDCwxdFODvAHY8xPNtZ0PdWATx0jIPyAWcYYtx8K6SGqAN9Z1wEEADOMMT/bW9J1PQ78y3EhuAcYbnM91+X4IO0FPFKs5/GGIZtKKaWc4y3NO0oppZygoa+UUj5EQ18ppXyIhr5SSvkQDX2llPIhGvpKKeVDNPSVUsqHaOgrpZQP+X/4eVB3nmR7swAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_loss_record)\n",
    "plt.plot(valid_loss_record)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test set Loading  \n",
    "\n",
    "    1. Test set is a little bit different since it has no labels, load test_data into mytestDS.\n",
    "    \n",
    "    2. No shuffling at dataloader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Inference \"\"\"\n",
    "# if config['taks'] == 'inference':\n",
    "testDS = mytestDS(test_data, all_sents)\n",
    "# Do not shuffle here\n",
    "test_dataloader = DataLoader(dataset=testDS, num_workers=2, batch_size=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Done.\n"
     ]
    }
   ],
   "source": [
    "result = []\n",
    "for idx, data in enumerate(test_dataloader, 0):\n",
    "\n",
    "    # get data\n",
    "    s1, s2 = data\n",
    "\n",
    "    # input\n",
    "    output = siamese(s1,s2)\n",
    "    output = output.squeeze(0)\n",
    "\n",
    "    # feed output into softmax to get prob prediction\n",
    "    sm = nn.Softmax(dim=1)\n",
    "    res = sm(output.data)[:,1]\n",
    "    result += res.data.tolist()\n",
    "\n",
    "result = pd.DataFrame(result)\n",
    "print 'Inference Done.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write down the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result has writtn to res/result.txt , Good Luck!\n"
     ]
    }
   ],
   "source": [
    "res_path = os.path.join(config['result']['filepath'], config['result']['filename'])\n",
    "result.to_csv(res_path,header=False,index=False)\n",
    "print 'Result has writtn to', res_path, ', Good Luck!'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}