word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hX4n9TsbGw-f"
   },
   "source": [
    "##### Copyright 2020 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:17.622698Z",
     "iopub.status.busy": "2021-01-21T03:00:17.622097Z",
     "iopub.status.idle": "2021-01-21T03:00:17.623988Z",
     "shell.execute_reply": "2021-01-21T03:00:17.624355Z"
    },
    "id": "0nbI5DtDGw-i"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AOpGoE2T-YXS"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/word2vec\">\n",
    "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
    "    View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/word2vec.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
    "    Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/word2vec.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
    "    View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/word2vec.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "haJUNjSB60Kh"
   },
   "source": [
    "# Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "99d4ky2lWFvn"
   },
   "source": [
    "Word2Vec is not a singular algorithm, rather, it is a family of model architectures and optimizations that can be used to learn word embeddings from large datasets. Embeddings learned through Word2Vec have proven to be successful on a variety of downstream natural language processing tasks.\n",
    "\n",
    "Note: This tutorial is based on [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf) and\n",
    "[Distributed\n",
    "Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf). It is not an exact implementation of the papers. Rather, it is intended to illustrate the key ideas.\n",
    "\n",
    "These papers proposed two methods for learning representations of words: \n",
    "\n",
    "*   **Continuous Bag-of-Words Model** which predicts the middle word based on surrounding context words. The context consists of a few words before and after the current (middle) word. This architecture is called a bag-of-words model as the order of words in the context is not important.\n",
    "*   **Continuous Skip-gram Model** which predict words within a certain range before and after the current word in the same sentence. A worked example of this is given below.\n",
    "\n",
    "\n",
    "You'll use the skip-gram approach in this tutorial. First, you'll explore skip-grams and other concepts using a single sentence for illustration. Next, you'll train your own Word2Vec model on a small dataset. This tutorial also contains code to export the trained embeddings and visualize them in the [TensorFlow Embedding Projector](http://projector.tensorflow.org/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xP00WlaMWBZC"
   },
   "source": [
    "## Skip-gram and Negative Sampling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zr2wjv0bW236"
   },
   "source": [
    "While a bag-of-words model predicts a word given the neighboring context, a skip-gram model predicts the context (or neighbors) of a word, given the word itself. The model is trained on skip-grams, which are n-grams that allow tokens to be skipped (see the diagram below for an example). The context of a word can be represented through a set of skip-gram pairs of `(target_word, context_word)` where `context_word` appears in the neighboring context of `target_word`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ICjc-McbaVTd"
   },
   "source": [
    "Consider the following sentence of 8 words.\n",
    "> The wide road shimmered in the hot sun. \n",
    "\n",
    "The context words for each of the 8 words of this sentence are defined by a window size. The window size determines the span of words on either side of a `target_word` that can be considered `context word`. Take a look at this table of skip-grams for target words based on different window sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YKE87IKT_YT8"
   },
   "source": [
    "Note: For this tutorial, a window size of *n* implies n words on each side with a total window span of 2*n+1 words across a word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RsCwQ07E8mqU"
   },
   "source": [
    "![word2vec_skipgrams](images/word2vec_skipgram.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gK1gN1jwkMpU"
   },
   "source": [
    "The training objective of the skip-gram model is to maximize the probability of predicting context words given the target word. For a sequence of words *w<sub>1</sub>, w<sub>2</sub>, ... w<sub>T</sub>*, the objective can be written as the average log probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pILO_iAc84e-"
   },
   "source": [
    "![word2vec_skipgram_objective](images/word2vec_skipgram_objective.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Gsy6TUbtnz_K"
   },
   "source": [
    "where `c` is the size of the training context. The basic skip-gram formulation defines this probability using the softmax function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P81Qavbb9APd"
   },
   "source": [
    "![word2vec_full_softmax](images/word2vec_full_softmax.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "axZvd-hhotVB"
   },
   "source": [
    "where *v* and *v<sup>'<sup>* are target and context vector representations of words and *W* is vocabulary size. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SoLzxbqSpT6_"
   },
   "source": [
    "Computing the denominator of this formulation involves performing a full softmax over the entire vocabulary words which is often large (10<sup>5</sup>-10<sup>7</sup>) terms. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y5VWYtmFzHkU"
   },
   "source": [
    "The [Noise Contrastive Estimation](https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss) loss function is an efficient approximation for a full softmax. With an objective to learn word embeddings instead of modelling the word distribution, NCE loss can be [simplified](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) to use negative sampling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WTZBPf1RsOsg"
   },
   "source": [
    "The simplified negative sampling objective for a target word is to distinguish  the context word from *num_ns* negative samples drawn from noise distribution *P<sub>n</sub>(w)* of words. More precisely, an efficient approximation of full softmax over the vocabulary is, for a skip-gram pair, to pose the loss for a target word as a classification problem between the context word and *num_ns* negative samples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cl0rSfHjt6Mf"
   },
   "source": [
    "A negative sample is defined as a (target_word, context_word) pair such that the context_word does not appear in the `window_size` neighborhood of the target_word. For the example sentence, these are few potential negative samples (when `window_size` is 2).\n",
    "\n",
    "```\n",
    "(hot, shimmered)\n",
    "(wide, hot)\n",
    "(wide, sun)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kq0q2uqbucFg"
   },
   "source": [
    "In the next section, you'll generate skip-grams and negative samples for a single sentence. You'll also learn about subsampling techniques and train a classification model for positive and negative training examples later in the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mk4-Hpe1CH16"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:17.637454Z",
     "iopub.status.busy": "2021-01-21T03:00:17.633831Z",
     "iopub.status.idle": "2021-01-21T03:00:18.983028Z",
     "shell.execute_reply": "2021-01-21T03:00:18.982359Z"
    },
    "id": "OSUalgiCchft"
   },
   "outputs": [],
   "source": [
    "!pip install -q tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:18.988694Z",
     "iopub.status.busy": "2021-01-21T03:00:18.987959Z",
     "iopub.status.idle": "2021-01-21T03:00:25.234405Z",
     "shell.execute_reply": "2021-01-21T03:00:25.234859Z"
    },
    "id": "RutaI-Tpev3T"
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import itertools\n",
    "import numpy as np\n",
    "import os\n",
    "import re\n",
    "import string\n",
    "import tensorflow as tf\n",
    "import tqdm\n",
    "\n",
    "from tensorflow.keras import Model, Sequential\n",
    "from tensorflow.keras.layers import Activation, Dense, Dot, Embedding, Flatten, GlobalAveragePooling1D, Reshape\n",
    "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.239217Z",
     "iopub.status.busy": "2021-01-21T03:00:25.238545Z",
     "iopub.status.idle": "2021-01-21T03:00:25.240981Z",
     "shell.execute_reply": "2021-01-21T03:00:25.240501Z"
    },
    "id": "XkJ5299Tek6B"
   },
   "outputs": [],
   "source": [
    "SEED = 42 \n",
    "AUTOTUNE = tf.data.AUTOTUNE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RW-g5buCHwh3"
   },
   "source": [
    "### Vectorize an example sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y8TfZIgoQrcP"
   },
   "source": [
    "Consider the following sentence:    \n",
    "`The wide road shimmered in the hot sun.`\n",
    "\n",
    "Tokenize the sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.245707Z",
     "iopub.status.busy": "2021-01-21T03:00:25.245019Z",
     "iopub.status.idle": "2021-01-21T03:00:25.249082Z",
     "shell.execute_reply": "2021-01-21T03:00:25.249520Z"
    },
    "id": "bsl7jBzV6_KK"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    }
   ],
   "source": [
    "sentence = \"The wide road shimmered in the hot sun\"\n",
    "tokens = list(sentence.lower().split())\n",
    "print(len(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PU-bs1XtThEw"
   },
   "source": [
    "Create a vocabulary to save mappings from tokens to integer indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.254232Z",
     "iopub.status.busy": "2021-01-21T03:00:25.253611Z",
     "iopub.status.idle": "2021-01-21T03:00:25.255627Z",
     "shell.execute_reply": "2021-01-21T03:00:25.256085Z"
    },
    "id": "UdYv1HJUQ8XA"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'<pad>': 0, 'the': 1, 'wide': 2, 'road': 3, 'shimmered': 4, 'in': 5, 'hot': 6, 'sun': 7}\n"
     ]
    }
   ],
   "source": [
    "vocab, index = {}, 1 # start indexing from 1\n",
    "vocab['<pad>'] = 0 # add a padding token \n",
    "for token in tokens:\n",
    "  if token not in vocab: \n",
    "    vocab[token] = index\n",
    "    index += 1\n",
    "vocab_size = len(vocab)\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZpuP43Dddasr"
   },
   "source": [
    "Create an inverse vocabulary to save mappings from integer indices to tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.260004Z",
     "iopub.status.busy": "2021-01-21T03:00:25.259391Z",
     "iopub.status.idle": "2021-01-21T03:00:25.261324Z",
     "shell.execute_reply": "2021-01-21T03:00:25.261687Z"
    },
    "id": "o9ULAJYtEvKl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: '<pad>', 1: 'the', 2: 'wide', 3: 'road', 4: 'shimmered', 5: 'in', 6: 'hot', 7: 'sun'}\n"
     ]
    }
   ],
   "source": [
    "inverse_vocab = {index: token for token, index in vocab.items()}\n",
    "print(inverse_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n3qtuyxIRyii"
   },
   "source": [
    "Vectorize your sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.265064Z",
     "iopub.status.busy": "2021-01-21T03:00:25.264505Z",
     "iopub.status.idle": "2021-01-21T03:00:25.266446Z",
     "shell.execute_reply": "2021-01-21T03:00:25.266783Z"
    },
    "id": "CsB3-9uQQYyl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 1, 6, 7]\n"
     ]
    }
   ],
   "source": [
    "example_sequence = [vocab[word] for word in tokens]\n",
    "print(example_sequence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ox1I28JRIOdM"
   },
   "source": [
    "### Generate skip-grams from one sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t7NNKAmSiHvy"
   },
   "source": [
    "The `tf.keras.preprocessing.sequence` module provides useful functions that simplify data preparation for Word2Vec. You can use the `tf.keras.preprocessing.sequence.skipgrams` to generate skip-gram pairs from the `example_sequence` with a given `window_size` from tokens in the range `[0, vocab_size)`.\n",
    "\n",
    "Note: `negative_samples` is set to `0` here as batching negative samples generated by this function requires a bit of code. You will use another function to perform negative sampling in the next section.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.271094Z",
     "iopub.status.busy": "2021-01-21T03:00:25.270506Z",
     "iopub.status.idle": "2021-01-21T03:00:25.272960Z",
     "shell.execute_reply": "2021-01-21T03:00:25.272521Z"
    },
    "id": "USAJxW4RD7pn"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26\n"
     ]
    }
   ],
   "source": [
    "window_size = 2\n",
    "positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n",
    "      example_sequence, \n",
    "      vocabulary_size=vocab_size,\n",
    "      window_size=window_size,\n",
    "      negative_samples=0)\n",
    "print(len(positive_skip_grams))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uc9uhiMwY-AQ"
   },
   "source": [
    "Take a look at few positive skip-grams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:25.277382Z",
     "iopub.status.busy": "2021-01-21T03:00:25.276766Z",
     "iopub.status.idle": "2021-01-21T03:00:25.279506Z",
     "shell.execute_reply": "2021-01-21T03:00:25.279019Z"
    },
    "id": "SCnqEukIE9pt"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 3): (the, road)\n",
      "(4, 1): (shimmered, the)\n",
      "(5, 6): (in, hot)\n",
      "(4, 2): (shimmered, wide)\n",
      "(3, 2): (road, wide)\n"
     ]
    }
   ],
   "source": [
    "for target, context in positive_skip_grams[:5]:\n",
    "  print(f\"({target}, {context}): ({inverse_vocab[target]}, {inverse_vocab[context]})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ua9PkMTISF0"
   },
   "source": [
    "### Negative sampling for one skip-gram "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Esqn8WBfZnEK"
   },
   "source": [
    "The `skipgrams` function returns all positive skip-gram pairs by sliding over a given window span. To produce additional skip-gram pairs that would serve as negative samples for training, you need to sample random words from the vocabulary. Use the `tf.random.log_uniform_candidate_sampler` function to sample `num_ns` number of negative samples for a given target word in a window. You can call the funtion on one skip-grams's target word and pass the context word as true class to exclude it from being sampled.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AgH3aSvw3xTD"
   },
   "source": [
    "Key point: *num_ns* (number of negative samples per positive context word) between [5, 20] is [shown to work](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) best for smaller datasets, while *num_ns* between [2,5] suffices for larger datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:26.683966Z",
     "iopub.status.busy": "2021-01-21T03:00:26.683244Z",
     "iopub.status.idle": "2021-01-21T03:00:26.964417Z",
     "shell.execute_reply": "2021-01-21T03:00:26.963972Z"
    },
    "id": "m_LmdzqIGr5L"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([2 1 4 3], shape=(4,), dtype=int64)\n",
      "['wide', 'the', 'shimmered', 'road']\n"
     ]
    }
   ],
   "source": [
    "# Get target and context words for one positive skip-gram.\n",
    "target_word, context_word = positive_skip_grams[0]\n",
    "\n",
    "# Set the number of negative samples per positive context. \n",
    "num_ns = 4\n",
    "\n",
    "context_class = tf.reshape(tf.constant(context_word, dtype=\"int64\"), (1, 1))\n",
    "negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(\n",
    "    true_classes=context_class, # class that should be sampled as 'positive'\n",
    "    num_true=1, # each positive skip-gram has 1 positive context class\n",
    "    num_sampled=num_ns, # number of negative context words to sample\n",
    "    unique=True, # all the negative samples should be unique\n",
    "    range_max=vocab_size, # pick index of the samples from [0, vocab_size]\n",
    "    seed=SEED, # seed for reproducibility\n",
    "    name=\"negative_sampling\" # name of this operation\n",
    ")\n",
    "print(negative_sampling_candidates)\n",
    "print([inverse_vocab[index.numpy()] for index in negative_sampling_candidates])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8MSxWCrLIalp"
   },
   "source": [
    "### Construct one training example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q6uEWdj8vKKv"
   },
   "source": [
    "For a given positive `(target_word, context_word)` skip-gram, you now also have `num_ns` negative sampled context words that do not appear in the window size neighborhood of `target_word`. Batch the `1` positive `context_word` and `num_ns` negative context words into one tensor. This produces a set of positive skip-grams (labelled as `1`) and negative samples (labelled as `0`) for each target word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:26.969368Z",
     "iopub.status.busy": "2021-01-21T03:00:26.968744Z",
     "iopub.status.idle": "2021-01-21T03:00:26.971488Z",
     "shell.execute_reply": "2021-01-21T03:00:26.971917Z"
    },
    "id": "zSiZwifuLvHf"
   },
   "outputs": [],
   "source": [
    "# Add a dimension so you can use concatenation (on the next step).\n",
    "negative_sampling_candidates = tf.expand_dims(negative_sampling_candidates, 1)\n",
    "\n",
    "# Concat positive context word with negative sampled words.\n",
    "context = tf.concat([context_class, negative_sampling_candidates], 0)\n",
    "\n",
    "# Label first context word as 1 (positive) followed by num_ns 0s (negative).\n",
    "label = tf.constant([1] + [0]*num_ns, dtype=\"int64\") \n",
    "\n",
    "# Reshape target to shape (1,) and context and label to (num_ns+1,).\n",
    "target = tf.squeeze(target_word)\n",
    "context = tf.squeeze(context)\n",
    "label =  tf.squeeze(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OIJeoFCAwtXJ"
   },
   "source": [
    "Take a look at the context and the corresponding labels for the target word from the skip-gram example above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:26.975824Z",
     "iopub.status.busy": "2021-01-21T03:00:26.975217Z",
     "iopub.status.idle": "2021-01-21T03:00:26.979930Z",
     "shell.execute_reply": "2021-01-21T03:00:26.980290Z"
    },
    "id": "tzyCPCuZwmdL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target_index    : 1\n",
      "target_word     : the\n",
      "context_indices : [3 2 1 4 3]\n",
      "context_words   : ['road', 'wide', 'the', 'shimmered', 'road']\n",
      "label           : [1 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print(f\"target_index    : {target}\")\n",
    "print(f\"target_word     : {inverse_vocab[target_word]}\")\n",
    "print(f\"context_indices : {context}\")\n",
    "print(f\"context_words   : {[inverse_vocab[c.numpy()] for c in context]}\")\n",
    "print(f\"label           : {label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gBtTcUVQr8EO"
   },
   "source": [
    "A tuple of `(target, context, label)` tensors constitutes one training example for training your skip-gram negative sampling Word2Vec model. Notice that the target is of shape `(1,)` while the context and label are of shape `(1+num_ns,)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:26.984327Z",
     "iopub.status.busy": "2021-01-21T03:00:26.983717Z",
     "iopub.status.idle": "2021-01-21T03:00:26.986322Z",
     "shell.execute_reply": "2021-01-21T03:00:26.986728Z"
    },
    "id": "x-FwkR8jx9-Z"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target  : tf.Tensor(1, shape=(), dtype=int32)\n",
      "context : tf.Tensor([3 2 1 4 3], shape=(5,), dtype=int64)\n",
      "label   : tf.Tensor([1 0 0 0 0], shape=(5,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "print(f\"target  :\", target)\n",
    "print(f\"context :\", context )\n",
    "print(f\"label   :\", label )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4bRJIlow4Dlv"
   },
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pWkuha0oykG5"
   },
   "source": [
    "This picture summarizes the procedure of generating training example from a sentence. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_KlwdiAa9crJ"
   },
   "source": [
    "![word2vec_negative_sampling](images/word2vec_negative_sampling.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9wmdO_MEIpaM"
   },
   "source": [
    "## Compile all steps into one function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iLKwNAczHsKg"
   },
   "source": [
    "### Skip-gram Sampling table "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TUUK3uDtFNFE"
   },
   "source": [
    "A large dataset means larger vocabulary with higher number of more frequent words such as stopwords. Training examples obtained from sampling commonly occuring words (such as `the`, `is`, `on`) don't add much useful information  for the model to learn from. [Mikolov et al.](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) suggest subsampling of frequent words as a helpful practice to improve embedding quality. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bPtbv7zNP7Dx"
   },
   "source": [
    "The `tf.keras.preprocessing.sequence.skipgrams` function accepts a sampling table argument to encode probabilities of sampling any token. You can use the `tf.keras.preprocessing.sequence.make_sampling_table` to  generate a word-frequency rank based probabilistic sampling table and pass it to `skipgrams` function. Take a look at the sampling probabilities for a `vocab_size` of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:26.991491Z",
     "iopub.status.busy": "2021-01-21T03:00:26.990898Z",
     "iopub.status.idle": "2021-01-21T03:00:26.993349Z",
     "shell.execute_reply": "2021-01-21T03:00:26.992949Z"
    },
    "id": "Rn9zAnDccyRg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.00315225 0.00315225 0.00547597 0.00741556 0.00912817 0.01068435\n",
      " 0.01212381 0.01347162 0.01474487 0.0159558 ]\n"
     ]
    }
   ],
   "source": [
    "sampling_table = tf.keras.preprocessing.sequence.make_sampling_table(size=10)\n",
    "print(sampling_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EHvSptcPk5fp"
   },
   "source": [
    "`sampling_table[i]` denotes the probability of sampling the i-th most common word in a dataset. The function assumes a [Zipf's distribution](https://en.wikipedia.org/wiki/Zipf%27s_law) of the word frequencies for sampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mRHMssMmHgH-"
   },
   "source": [
    "Key point: The `tf.random.log_uniform_candidate_sampler` already assumes that the vocabulary frequency follows a log-uniform (Zipf's) distribution. Using these distribution weighted sampling also helps approximate the Noise Contrastive Estimation (NCE) loss with simpler loss functions for training a negative sampling objective."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aj--8RFK6fgW"
   },
   "source": [
    "### Generate training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dy5hl4lQ0B2M"
   },
   "source": [
    "Compile all the steps described above into a function that can be called on a list of vectorized sentences obtained from any text dataset. Notice that the sampling table is built before sampling skip-gram word pairs. You will use this function in the later sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.001289Z",
     "iopub.status.busy": "2021-01-21T03:00:27.000700Z",
     "iopub.status.idle": "2021-01-21T03:00:27.002547Z",
     "shell.execute_reply": "2021-01-21T03:00:27.002130Z"
    },
    "id": "63INISDEX1Hu"
   },
   "outputs": [],
   "source": [
    "# Generates skip-gram pairs with negative sampling for a list of sequences\n",
    "# (int-encoded sentences) based on window size, number of negative samples\n",
    "# and vocabulary size.\n",
    "def generate_training_data(sequences, window_size, num_ns, vocab_size, seed):\n",
    "  # Elements of each training example are appended to these lists.\n",
    "  targets, contexts, labels = [], [], []\n",
    "\n",
    "  # Build the sampling table for vocab_size tokens.\n",
    "  sampling_table = tf.keras.preprocessing.sequence.make_sampling_table(vocab_size)\n",
    "\n",
    "  # Iterate over all sequences (sentences) in dataset.\n",
    "  for sequence in tqdm.tqdm(sequences):\n",
    "\n",
    "    # Generate positive skip-gram pairs for a sequence (sentence).\n",
    "    positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n",
    "          sequence, \n",
    "          vocabulary_size=vocab_size,\n",
    "          sampling_table=sampling_table,\n",
    "          window_size=window_size,\n",
    "          negative_samples=0)\n",
    "    \n",
    "    # Iterate over each positive skip-gram pair to produce training examples \n",
    "    # with positive context word and negative samples.\n",
    "    for target_word, context_word in positive_skip_grams:\n",
    "      context_class = tf.expand_dims(\n",
    "          tf.constant([context_word], dtype=\"int64\"), 1)\n",
    "      negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(\n",
    "          true_classes=context_class,\n",
    "          num_true=1, \n",
    "          num_sampled=num_ns, \n",
    "          unique=True, \n",
    "          range_max=vocab_size, \n",
    "          seed=SEED, \n",
    "          name=\"negative_sampling\")\n",
    "      \n",
    "      # Build context and label vectors (for one target word)\n",
    "      negative_sampling_candidates = tf.expand_dims(\n",
    "          negative_sampling_candidates, 1)\n",
    "\n",
    "      context = tf.concat([context_class, negative_sampling_candidates], 0)\n",
    "      label = tf.constant([1] + [0]*num_ns, dtype=\"int64\")\n",
    "\n",
    "      # Append each element from the training example to global lists.\n",
    "      targets.append(target_word)\n",
    "      contexts.append(context)\n",
    "      labels.append(label)\n",
    "\n",
    "  return targets, contexts, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "shvPC8Ji2cMK"
   },
   "source": [
    "## Prepare training data for Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j5mbZsZu6uKg"
   },
   "source": [
    "With an understanding of how to work with one sentence for a skip-gram negative sampling based Word2Vec model, you can proceed to generate training examples from a larger list of sentences!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OFlikI6L26nh"
   },
   "source": [
    "### Download text corpus\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rEFavOgN98al"
   },
   "source": [
    "You will use a text file of Shakespeare's writing for this tutorial. Change the following line to run this code on your own data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.007710Z",
     "iopub.status.busy": "2021-01-21T03:00:27.007122Z",
     "iopub.status.idle": "2021-01-21T03:00:27.209026Z",
     "shell.execute_reply": "2021-01-21T03:00:27.208344Z"
    },
    "id": "QFkitxzVVaAi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "   8192/1115394 [..............................] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "1122304/1115394 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sOsbLq8a37dr"
   },
   "source": [
    "Read text from the file and take a look at the first few lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.213900Z",
     "iopub.status.busy": "2021-01-21T03:00:27.213203Z",
     "iopub.status.idle": "2021-01-21T03:00:27.223892Z",
     "shell.execute_reply": "2021-01-21T03:00:27.223373Z"
    },
    "id": "lfgnsUw3ofMD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us kill him, and we'll have corn at our own price.\n"
     ]
    }
   ],
   "source": [
    "with open(path_to_file) as f: \n",
    "  lines = f.read().splitlines()\n",
    "for line in lines[:20]:\n",
    "  print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gTNZYqUs5C2V"
   },
   "source": [
    "Use the non empty lines to construct a `tf.data.TextLineDataset` object for next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.229363Z",
     "iopub.status.busy": "2021-01-21T03:00:27.228703Z",
     "iopub.status.idle": "2021-01-21T03:00:27.275948Z",
     "shell.execute_reply": "2021-01-21T03:00:27.275227Z"
    },
    "id": "ViDrwy-HjAs9"
   },
   "outputs": [],
   "source": [
    "text_ds = tf.data.TextLineDataset(path_to_file).filter(lambda x: tf.cast(tf.strings.length(x), bool))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vfsc88zE9upk"
   },
   "source": [
    "### Vectorize sentences from the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XfgZo8zR94KK"
   },
   "source": [
    "You can use the `TextVectorization` layer to vectorize sentences from the corpus. Learn more about using this layer in this [Text Classification](https://www.tensorflow.org/tutorials/keras/text_classification) tutorial. Notice from the first few sentences above that the text needs to be in one case and punctuation needs to be removed. To do this, define a `custom_standardization function` that can be used in the TextVectorization layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.286198Z",
     "iopub.status.busy": "2021-01-21T03:00:27.280936Z",
     "iopub.status.idle": "2021-01-21T03:00:27.308776Z",
     "shell.execute_reply": "2021-01-21T03:00:27.309189Z"
    },
    "id": "2MlsXzo-ZlfK"
   },
   "outputs": [],
   "source": [
    "# We create a custom standardization function to lowercase the text and \n",
    "# remove punctuation.\n",
    "def custom_standardization(input_data):\n",
    "  lowercase = tf.strings.lower(input_data)\n",
    "  return tf.strings.regex_replace(lowercase,\n",
    "                                  '[%s]' % re.escape(string.punctuation), '')\n",
    "\n",
    "# Define the vocabulary size and number of words in a sequence.\n",
    "vocab_size = 4096\n",
    "sequence_length = 10\n",
    "\n",
    "# Use the text vectorization layer to normalize, split, and map strings to\n",
    "# integers. Set output_sequence_length length to pad all samples to same length.\n",
    "vectorize_layer = TextVectorization(\n",
    "    standardize=custom_standardization,\n",
    "    max_tokens=vocab_size,\n",
    "    output_mode='int',\n",
    "    output_sequence_length=sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g92LuvnyBmz1"
   },
   "source": [
    "Call `adapt` on the text dataset to create vocabulary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:27.313372Z",
     "iopub.status.busy": "2021-01-21T03:00:27.312717Z",
     "iopub.status.idle": "2021-01-21T03:00:28.558323Z",
     "shell.execute_reply": "2021-01-21T03:00:28.557711Z"
    },
    "id": "seZau_iYMPFT"
   },
   "outputs": [],
   "source": [
    "vectorize_layer.adapt(text_ds.batch(1024))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jg2z7eeHMnH-"
   },
   "source": [
    "Once the state of the layer has been adapted to represent the text corpus, the vocabulary can be accessed with `get_vocabulary()`. This function returns a list of all vocabulary tokens sorted (descending) by their frequency. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:28.562873Z",
     "iopub.status.busy": "2021-01-21T03:00:28.562273Z",
     "iopub.status.idle": "2021-01-21T03:00:28.569259Z",
     "shell.execute_reply": "2021-01-21T03:00:28.569653Z"
    },
    "id": "jgw9pTA7MRaU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['', '[UNK]', 'the', 'and', 'to', 'i', 'of', 'you', 'my', 'a', 'that', 'in', 'is', 'not', 'for', 'with', 'me', 'it', 'be', 'your']\n"
     ]
    }
   ],
   "source": [
    "# Save the created vocabulary for reference.\n",
    "inverse_vocab = vectorize_layer.get_vocabulary()\n",
    "print(inverse_vocab[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DOQ30Tx6KA2G"
   },
   "source": [
    "The vectorize_layer can now be used to generate vectors for each element in the `text_ds`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:28.574518Z",
     "iopub.status.busy": "2021-01-21T03:00:28.573933Z",
     "iopub.status.idle": "2021-01-21T03:00:28.621475Z",
     "shell.execute_reply": "2021-01-21T03:00:28.620922Z"
    },
    "id": "yUVYrDp0araQ"
   },
   "outputs": [],
   "source": [
    "def vectorize_text(text):\n",
    "  text = tf.expand_dims(text, -1)\n",
    "  return tf.squeeze(vectorize_layer(text))\n",
    "\n",
    "# Vectorize the data in text_ds.\n",
    "text_vector_ds = text_ds.batch(1024).prefetch(AUTOTUNE).map(vectorize_layer).unbatch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7YyH_SYzB72p"
   },
   "source": [
    "### Obtain sequences from the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NFUQLX0_KaRC"
   },
   "source": [
    "You now have a `tf.data.Dataset` of integer encoded sentences. To prepare the dataset for training a Word2Vec model, flatten the dataset into a list of sentence vector sequences. This step is required as you would iterate over each sentence in the dataset to produce positive and negative examples. \n",
    "\n",
    "Note: Since the `generate_training_data()` defined earlier uses non-TF python/numpy functions, you could also use a `tf.py_function` or `tf.numpy_function` with `tf.data.Dataset.map()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:28.627815Z",
     "iopub.status.busy": "2021-01-21T03:00:28.627206Z",
     "iopub.status.idle": "2021-01-21T03:00:30.982078Z",
     "shell.execute_reply": "2021-01-21T03:00:30.981446Z"
    },
    "id": "sGXoOh9y11pM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32777\n"
     ]
    }
   ],
   "source": [
    "sequences = list(text_vector_ds.as_numpy_iterator())\n",
    "print(len(sequences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tDc4riukLTqg"
   },
   "source": [
    "Take a look at few examples from `sequences`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:30.987147Z",
     "iopub.status.busy": "2021-01-21T03:00:30.986485Z",
     "iopub.status.idle": "2021-01-21T03:00:30.989065Z",
     "shell.execute_reply": "2021-01-21T03:00:30.988596Z"
    },
    "id": "WZf1RIbB2Dfb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 89 270   0   0   0   0   0   0   0   0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n",
      "[138  36 982 144 673 125  16 106   0   0] => ['before', 'we', 'proceed', 'any', 'further', 'hear', 'me', 'speak', '', '']\n",
      "[34  0  0  0  0  0  0  0  0  0] => ['all', '', '', '', '', '', '', '', '', '']\n",
      "[106 106   0   0   0   0   0   0   0   0] => ['speak', 'speak', '', '', '', '', '', '', '', '']\n",
      "[ 89 270   0   0   0   0   0   0   0   0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n"
     ]
    }
   ],
   "source": [
    "for seq in sequences[:5]:\n",
    "  print(f\"{seq} => {[inverse_vocab[i] for i in seq]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yDzSOjNwCWNh"
   },
   "source": [
    "### Generate training examples from sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BehvYr-nEKyY"
   },
   "source": [
    "`sequences` is now a list of int encoded sentences. Just call the `generate_training_data()` function defined earlier to generate training examples for the Word2Vec model. To recap, the function iterates over each word from each sequence to collect positive and negative context words. Length of target, contexts and labels should be same, representing the total number of training examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:30.993704Z",
     "iopub.status.busy": "2021-01-21T03:00:30.993116Z",
     "iopub.status.idle": "2021-01-21T03:00:58.391465Z",
     "shell.execute_reply": "2021-01-21T03:00:58.391946Z"
    },
    "id": "44DJ22M6nX5o"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/32777 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 130/32777 [00:00<00:25, 1290.67it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  1%|          | 260/32777 [00:00<00:26, 1204.74it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  1%|          | 403/32777 [00:00<00:24, 1302.79it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  2%|▏         | 534/32777 [00:00<00:26, 1220.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  2%|▏         | 704/32777 [00:00<00:23, 1374.92it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 843/32777 [00:00<00:24, 1314.59it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 985/32777 [00:00<00:23, 1343.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 1141/32777 [00:00<00:22, 1400.11it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  4%|▍         | 1282/32777 [00:00<00:22, 1395.62it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  4%|▍         | 1425/32777 [00:01<00:22, 1398.25it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|▍         | 1566/32777 [00:01<00:25, 1227.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|▌         | 1693/32777 [00:01<00:26, 1161.30it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  6%|▌         | 1822/32777 [00:01<00:25, 1193.64it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  6%|▌         | 1944/32777 [00:01<00:25, 1189.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  7%|▋         | 2139/32777 [00:01<00:21, 1396.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  7%|▋         | 2313/32777 [00:01<00:20, 1489.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  8%|▊         | 2537/32777 [00:01<00:17, 1705.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  8%|▊         | 2710/32777 [00:01<00:17, 1687.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  9%|▉         | 2881/32777 [00:02<00:18, 1657.44it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  9%|▉         | 3048/32777 [00:02<00:18, 1642.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 10%|▉         | 3214/32777 [00:02<00:18, 1618.60it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 10%|█         | 3377/32777 [00:02<00:19, 1539.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|█         | 3532/32777 [00:02<00:19, 1501.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|█         | 3683/32777 [00:02<00:20, 1419.09it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 12%|█▏        | 3880/32777 [00:02<00:18, 1563.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 12%|█▏        | 4039/32777 [00:02<00:18, 1554.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 13%|█▎        | 4196/32777 [00:02<00:18, 1529.13it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 13%|█▎        | 4350/32777 [00:03<00:20, 1359.67it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 14%|█▎        | 4490/32777 [00:03<00:22, 1245.60it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 14%|█▍        | 4631/32777 [00:03<00:21, 1286.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 15%|█▍        | 4794/32777 [00:03<00:20, 1373.31it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 15%|█▌        | 4935/32777 [00:03<00:21, 1293.66it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 15%|█▌        | 5068/32777 [00:03<00:21, 1261.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|█▌        | 5209/32777 [00:03<00:21, 1300.66it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|█▋        | 5372/32777 [00:03<00:19, 1391.87it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 17%|█▋        | 5514/32777 [00:03<00:19, 1395.77it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 17%|█▋        | 5655/32777 [00:04<00:21, 1289.41it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 18%|█▊        | 5787/32777 [00:04<00:22, 1187.64it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 18%|█▊        | 5913/32777 [00:04<00:22, 1206.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 18%|█▊        | 6042/32777 [00:04<00:21, 1224.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 19%|█▉        | 6173/32777 [00:04<00:21, 1247.39it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 19%|█▉        | 6303/32777 [00:04<00:21, 1258.49it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|█▉        | 6444/32777 [00:04<00:20, 1296.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 6575/32777 [00:04<00:20, 1252.69it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 6703/32777 [00:04<00:20, 1259.99it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|██        | 6830/32777 [00:05<00:21, 1231.77it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|██        | 6954/32777 [00:05<00:21, 1175.20it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 22%|██▏       | 7073/32777 [00:05<00:22, 1131.32it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 22%|██▏       | 7206/32777 [00:05<00:21, 1183.55it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 22%|██▏       | 7339/32777 [00:05<00:20, 1223.83it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 23%|██▎       | 7463/32777 [00:05<00:21, 1177.79it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 23%|██▎       | 7582/32777 [00:05<00:21, 1176.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 24%|██▎       | 7742/32777 [00:05<00:19, 1290.61it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 24%|██▍       | 7872/32777 [00:05<00:19, 1274.76it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 24%|██▍       | 8000/32777 [00:06<00:20, 1196.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 25%|██▍       | 8121/32777 [00:06<00:26, 919.60it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 25%|██▌       | 8223/32777 [00:06<00:26, 936.05it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 25%|██▌       | 8327/32777 [00:06<00:25, 960.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|██▌       | 8429/32777 [00:06<00:26, 914.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|██▌       | 8573/32777 [00:06<00:23, 1050.69it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|██▋       | 8684/32777 [00:06<00:22, 1048.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 27%|██▋       | 8818/32777 [00:06<00:21, 1125.35it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 27%|██▋       | 8988/32777 [00:06<00:18, 1285.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 28%|██▊       | 9120/32777 [00:07<00:18, 1250.11it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 28%|██▊       | 9248/32777 [00:07<00:20, 1160.04it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 29%|██▊       | 9376/32777 [00:07<00:19, 1183.71it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 29%|██▉       | 9498/32777 [00:07<00:19, 1190.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 29%|██▉       | 9621/32777 [00:07<00:19, 1193.34it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|██▉       | 9742/32777 [00:07<00:19, 1177.71it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 9861/32777 [00:07<00:19, 1166.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 9979/32777 [00:07<00:21, 1075.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 31%|███       | 10089/32777 [00:07<00:21, 1037.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 31%|███       | 10198/32777 [00:08<00:21, 1048.57it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 31%|███▏      | 10304/32777 [00:08<00:23, 943.35it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 10401/32777 [00:08<00:23, 947.13it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 10509/32777 [00:08<00:22, 981.23it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 10611/32777 [00:08<00:22, 987.06it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 33%|███▎      | 10739/32777 [00:08<00:20, 1065.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 33%|███▎      | 10860/32777 [00:08<00:19, 1103.49it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 34%|███▎      | 11000/32777 [00:08<00:18, 1183.49it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 34%|███▍      | 11123/32777 [00:08<00:18, 1189.01it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 34%|███▍      | 11243/32777 [00:09<00:19, 1112.40it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 35%|███▍      | 11356/32777 [00:09<00:20, 1065.67it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 35%|███▍      | 11464/32777 [00:09<00:20, 1023.70it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 35%|███▌      | 11576/32777 [00:09<00:20, 1050.07it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 36%|███▌      | 11685/32777 [00:09<00:19, 1060.09it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 36%|███▌      | 11792/32777 [00:09<00:20, 1040.53it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 36%|███▋      | 11897/32777 [00:09<00:22, 927.83it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|███▋      | 11993/32777 [00:09<00:23, 899.47it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|███▋      | 12087/32777 [00:09<00:22, 908.80it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|███▋      | 12180/32777 [00:10<00:22, 907.81it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|███▋      | 12285/32777 [00:10<00:21, 942.30it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 38%|███▊      | 12415/32777 [00:10<00:19, 1040.59it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 38%|███▊      | 12534/32777 [00:10<00:18, 1082.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 39%|███▊      | 12658/32777 [00:10<00:17, 1126.25it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 39%|███▉      | 12772/32777 [00:10<00:17, 1114.55it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 39%|███▉      | 12884/32777 [00:10<00:18, 1102.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|███▉      | 13030/32777 [00:10<00:16, 1207.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 13172/32777 [00:10<00:15, 1260.32it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 41%|████      | 13299/32777 [00:11<00:17, 1126.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 41%|████      | 13415/32777 [00:11<00:17, 1129.64it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 41%|████▏     | 13532/32777 [00:11<00:16, 1137.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|████▏     | 13660/32777 [00:11<00:16, 1168.83it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|████▏     | 13806/32777 [00:11<00:15, 1251.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 43%|████▎     | 13933/32777 [00:11<00:15, 1205.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 43%|████▎     | 14055/32777 [00:11<00:16, 1151.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 43%|████▎     | 14189/32777 [00:11<00:15, 1198.47it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 44%|████▎     | 14310/32777 [00:11<00:16, 1148.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 44%|████▍     | 14463/32777 [00:11<00:14, 1250.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 45%|████▍     | 14590/32777 [00:12<00:15, 1170.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 45%|████▍     | 14709/32777 [00:12<00:15, 1157.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 45%|████▌     | 14848/32777 [00:12<00:14, 1214.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 46%|████▌     | 14971/32777 [00:12<00:15, 1172.55it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 46%|████▌     | 15090/32777 [00:12<00:16, 1104.12it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 46%|████▋     | 15211/32777 [00:12<00:15, 1130.11it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|████▋     | 15329/32777 [00:12<00:15, 1140.80it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|████▋     | 15460/32777 [00:12<00:14, 1184.63it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 48%|████▊     | 15600/32777 [00:12<00:13, 1241.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 48%|████▊     | 15762/32777 [00:13<00:12, 1345.56it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 49%|████▊     | 15898/32777 [00:13<00:13, 1253.70it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 49%|████▉     | 16035/32777 [00:13<00:13, 1284.49it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 49%|████▉     | 16165/32777 [00:13<00:13, 1227.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|████▉     | 16290/32777 [00:13<00:13, 1225.49it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 16414/32777 [00:13<00:14, 1144.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 16547/32777 [00:13<00:13, 1192.57it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 51%|█████     | 16673/32777 [00:13<00:13, 1206.39it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 51%|█████     | 16795/32777 [00:13<00:13, 1147.37it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 52%|█████▏    | 16940/32777 [00:14<00:12, 1231.27it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 52%|█████▏    | 17097/32777 [00:14<00:11, 1326.58it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 17232/32777 [00:14<00:11, 1319.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 17365/32777 [00:14<00:12, 1247.83it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 17492/32777 [00:14<00:12, 1183.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 54%|█████▎    | 17612/32777 [00:14<00:12, 1167.53it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 54%|█████▍    | 17730/32777 [00:14<00:14, 1068.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 55%|█████▍    | 17864/32777 [00:14<00:13, 1133.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 55%|█████▍    | 17980/32777 [00:14<00:13, 1113.10it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 55%|█████▌    | 18093/32777 [00:15<00:14, 1016.62it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▌    | 18197/32777 [00:15<00:15, 927.30it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▌    | 18293/32777 [00:15<00:15, 916.76it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▌    | 18396/32777 [00:15<00:15, 944.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▋    | 18502/32777 [00:15<00:14, 976.30it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 57%|█████▋    | 18643/32777 [00:15<00:12, 1097.76it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 57%|█████▋    | 18755/32777 [00:15<00:13, 1066.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|█████▊    | 18864/32777 [00:15<00:13, 1047.34it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|█████▊    | 18998/32777 [00:15<00:12, 1123.54it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|█████▊    | 19120/32777 [00:16<00:11, 1146.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 59%|█████▊    | 19239/32777 [00:16<00:11, 1155.74it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 59%|█████▉    | 19388/32777 [00:16<00:10, 1239.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|█████▉    | 19513/32777 [00:16<00:12, 1074.10it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|█████▉    | 19625/32777 [00:16<00:13, 967.20it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 19735/32777 [00:16<00:13, 999.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 61%|██████    | 19839/32777 [00:16<00:13, 960.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 61%|██████    | 19977/32777 [00:16<00:12, 1063.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 61%|██████▏   | 20087/32777 [00:16<00:12, 998.98it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 20190/32777 [00:17<00:14, 879.51it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 20310/32777 [00:17<00:13, 957.27it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 20410/32777 [00:17<00:12, 959.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|██████▎   | 20509/32777 [00:17<00:12, 952.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|██████▎   | 20607/32777 [00:17<00:12, 951.56it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|██████▎   | 20712/32777 [00:17<00:12, 976.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 64%|██████▎   | 20834/32777 [00:17<00:11, 1043.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 64%|██████▍   | 20961/32777 [00:17<00:10, 1102.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 64%|██████▍   | 21103/32777 [00:17<00:09, 1187.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 65%|██████▍   | 21223/32777 [00:18<00:09, 1158.79it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 65%|██████▌   | 21359/32777 [00:18<00:09, 1212.72it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 66%|██████▌   | 21492/32777 [00:18<00:09, 1246.48it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 66%|██████▌   | 21645/32777 [00:18<00:08, 1328.81it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 66%|██████▋   | 21779/32777 [00:18<00:08, 1244.05it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 21905/32777 [00:18<00:09, 1172.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 22032/32777 [00:18<00:08, 1194.44it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████▊   | 22164/32777 [00:18<00:08, 1220.60it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████▊   | 22288/32777 [00:18<00:09, 1090.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████▊   | 22401/32777 [00:19<00:11, 907.78it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 69%|██████▊   | 22499/32777 [00:19<00:11, 890.76it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 69%|██████▉   | 22629/32777 [00:19<00:10, 990.92it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 69%|██████▉   | 22743/32777 [00:19<00:09, 1029.57it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|██████▉   | 22851/32777 [00:19<00:09, 1022.50it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 22957/32777 [00:19<00:10, 954.09it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 23056/32777 [00:19<00:10, 915.06it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 71%|███████   | 23162/32777 [00:19<00:10, 951.84it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 71%|███████   | 23260/32777 [00:20<00:09, 952.61it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 71%|███████▏  | 23412/32777 [00:20<00:08, 1109.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 72%|███████▏  | 23539/32777 [00:20<00:07, 1155.53it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 72%|███████▏  | 23677/32777 [00:20<00:07, 1213.40it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 73%|███████▎  | 23818/32777 [00:20<00:07, 1268.89it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 73%|███████▎  | 23971/32777 [00:20<00:06, 1342.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|███████▎  | 24114/32777 [00:20<00:06, 1363.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|███████▍  | 24251/32777 [00:20<00:07, 1211.59it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|███████▍  | 24376/32777 [00:20<00:07, 1179.99it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▍  | 24513/32777 [00:20<00:06, 1230.80it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▌  | 24653/32777 [00:21<00:06, 1277.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 76%|███████▌  | 24815/32777 [00:21<00:05, 1370.16it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 76%|███████▌  | 24954/32777 [00:21<00:05, 1329.11it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 77%|███████▋  | 25094/32777 [00:21<00:05, 1335.46it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 77%|███████▋  | 25304/32777 [00:21<00:04, 1550.06it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 78%|███████▊  | 25461/32777 [00:21<00:04, 1484.39it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 78%|███████▊  | 25636/32777 [00:21<00:04, 1554.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|███████▊  | 25793/32777 [00:21<00:04, 1555.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|███████▉  | 25960/32777 [00:21<00:04, 1587.37it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|███████▉  | 26120/32777 [00:22<00:04, 1473.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 26270/32777 [00:22<00:04, 1431.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 81%|████████  | 26415/32777 [00:22<00:05, 1134.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 81%|████████  | 26539/32777 [00:22<00:05, 1139.16it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 81%|████████▏ | 26691/32777 [00:22<00:04, 1229.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 82%|████████▏ | 26821/32777 [00:22<00:04, 1230.82it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 82%|████████▏ | 26962/32777 [00:22<00:04, 1275.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 83%|████████▎ | 27094/32777 [00:22<00:04, 1214.54it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 83%|████████▎ | 27232/32777 [00:23<00:04, 1257.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▎ | 27372/32777 [00:23<00:04, 1291.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▍ | 27512/32777 [00:23<00:03, 1320.56it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▍ | 27647/32777 [00:23<00:03, 1326.09it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 85%|████████▍ | 27781/32777 [00:23<00:03, 1285.54it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 85%|████████▌ | 27922/32777 [00:23<00:03, 1312.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 86%|████████▌ | 28054/32777 [00:23<00:03, 1222.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 86%|████████▌ | 28178/32777 [00:23<00:03, 1195.10it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 86%|████████▋ | 28299/32777 [00:23<00:04, 1076.82it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 28410/32777 [00:24<00:04, 1045.85it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 28541/32777 [00:24<00:03, 1114.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 28668/32777 [00:24<00:03, 1153.79it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 88%|████████▊ | 28786/32777 [00:24<00:03, 1074.73it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 88%|████████▊ | 28896/32777 [00:24<00:03, 1054.76it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|████████▊ | 29044/32777 [00:24<00:03, 1169.72it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|████████▉ | 29225/32777 [00:24<00:02, 1349.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|████████▉ | 29363/32777 [00:24<00:02, 1333.65it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 29522/32777 [00:24<00:02, 1401.68it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 91%|█████████ | 29664/32777 [00:24<00:02, 1285.23it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 91%|█████████ | 29796/32777 [00:25<00:02, 1261.55it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 91%|█████████▏| 29925/32777 [00:25<00:02, 1185.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 92%|█████████▏| 30071/32777 [00:25<00:02, 1254.46it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 92%|█████████▏| 30199/32777 [00:25<00:02, 1184.73it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 93%|█████████▎| 30353/32777 [00:25<00:01, 1280.64it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 93%|█████████▎| 30492/32777 [00:25<00:01, 1307.37it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 93%|█████████▎| 30635/32777 [00:25<00:01, 1340.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 94%|█████████▍| 30771/32777 [00:25<00:01, 1312.51it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 94%|█████████▍| 30904/32777 [00:26<00:01, 973.49it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|█████████▍| 31043/32777 [00:26<00:01, 1068.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|█████████▌| 31208/32777 [00:26<00:01, 1207.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 96%|█████████▌| 31374/32777 [00:26<00:01, 1325.21it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 96%|█████████▌| 31547/32777 [00:26<00:00, 1430.27it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 97%|█████████▋| 31698/32777 [00:26<00:00, 1358.48it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 97%|█████████▋| 31840/32777 [00:26<00:00, 1343.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 31979/32777 [00:26<00:00, 1283.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 32111/32777 [00:26<00:00, 1272.53it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 32241/32777 [00:27<00:00, 1242.12it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 99%|█████████▉| 32382/32777 [00:27<00:00, 1286.56it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 99%|█████████▉| 32571/32777 [00:27<00:00, 1457.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|█████████▉| 32727/32777 [00:27<00:00, 1485.36it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 32777/32777 [00:27<00:00, 1196.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64362 64362 64362\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "targets, contexts, labels = generate_training_data(\n",
    "    sequences=sequences, \n",
    "    window_size=2, \n",
    "    num_ns=4, \n",
    "    vocab_size=vocab_size, \n",
    "    seed=SEED)\n",
    "print(len(targets), len(contexts), len(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "97PqsusOFEpc"
   },
   "source": [
    "### Configure the dataset for performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7jnFVySViQTj"
   },
   "source": [
    "To perform efficient batching for the potentially large number of training examples, use the `tf.data.Dataset` API. After this step, you would have a `tf.data.Dataset` object of `(target_word, context_word), (label)` elements to train your Word2Vec model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:00:58.446610Z",
     "iopub.status.busy": "2021-01-21T03:00:58.430902Z",
     "iopub.status.idle": "2021-01-21T03:01:03.282771Z",
     "shell.execute_reply": "2021-01-21T03:01:03.283225Z"
    },
    "id": "nbu8PxPSnVY2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<BatchDataset shapes: (((1024,), (1024, 5, 1)), (1024, 5)), types: ((tf.int32, tf.int64), tf.int64)>\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 1024\n",
    "BUFFER_SIZE = 10000\n",
    "dataset = tf.data.Dataset.from_tensor_slices(((targets, contexts), labels))\n",
    "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tyrNX6Fs6K3F"
   },
   "source": [
    "Add `cache()` and `prefetch()` to improve performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:03.288682Z",
     "iopub.status.busy": "2021-01-21T03:01:03.287892Z",
     "iopub.status.idle": "2021-01-21T03:01:03.290788Z",
     "shell.execute_reply": "2021-01-21T03:01:03.291164Z"
    },
    "id": "Y5Ueg6bcFPVL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<PrefetchDataset shapes: (((1024,), (1024, 5, 1)), (1024, 5)), types: ((tf.int32, tf.int64), tf.int64)>\n"
     ]
    }
   ],
   "source": [
    "dataset = dataset.cache().prefetch(buffer_size=AUTOTUNE)\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1S-CmUMszyEf"
   },
   "source": [
    "## Model and Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sQFqaBMPwBqC"
   },
   "source": [
    "The Word2Vec model can be implemented as a classifier to distinguish between true context words from skip-grams and false context words obtained through negative sampling. You can perform a dot product between the embeddings of target and context words to obtain predictions for labels and compute loss against true labels in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oc7kTbiwD9sy"
   },
   "source": [
    "### Subclassed Word2Vec Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jvr9pM1G1sQN"
   },
   "source": [
    "Use the [Keras Subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) to define your Word2Vec model with the following layers:\n",
    "\n",
    "\n",
    "* `target_embedding`: A `tf.keras.layers.Embedding` layer which looks up the embedding of a word when it appears as a target word. The number of parameters in this layer are `(vocab_size * embedding_dim)`.\n",
    "* `context_embedding`: Another `tf.keras.layers.Embedding` layer which looks up the embedding of a word when it appears as a context word. The number of parameters in this layer are the same as those in `target_embedding`, i.e. `(vocab_size * embedding_dim)`.\n",
    "* `dots`: A `tf.keras.layers.Dot` layer that computes the dot product of target and context embeddings from a training pair.\n",
    "* `flatten`: A `tf.keras.layers.Flatten` layer to flatten the results of `dots` layer into logits.\n",
    "\n",
    "With the sublassed model, you can define the `call()` function that accepts `(target, context)` pairs which can then be passed into their corresponding embedding layer. Reshape the `context_embedding` to perform a dot product with `target_embedding` and return the flattened result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KiAwuIqqw7-7"
   },
   "source": [
    "Key point: The `target_embedding` and `context_embedding` layers can be shared as well. You could also use a concatenation of both embeddings as the final Word2Vec embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:03.298311Z",
     "iopub.status.busy": "2021-01-21T03:01:03.297643Z",
     "iopub.status.idle": "2021-01-21T03:01:03.299432Z",
     "shell.execute_reply": "2021-01-21T03:01:03.299791Z"
    },
    "id": "i9ec-sS6xd8Z"
   },
   "outputs": [],
   "source": [
    "class Word2Vec(Model):\n",
    "  def __init__(self, vocab_size, embedding_dim):\n",
    "    super(Word2Vec, self).__init__()\n",
    "    self.target_embedding = Embedding(vocab_size, \n",
    "                                      embedding_dim,\n",
    "                                      input_length=1,\n",
    "                                      name=\"w2v_embedding\", )\n",
    "    self.context_embedding = Embedding(vocab_size, \n",
    "                                       embedding_dim, \n",
    "                                       input_length=num_ns+1)\n",
    "    self.dots = Dot(axes=(3,2))\n",
    "    self.flatten = Flatten()\n",
    "\n",
    "  def call(self, pair):\n",
    "    target, context = pair\n",
    "    we = self.target_embedding(target)\n",
    "    ce = self.context_embedding(context)\n",
    "    dots = self.dots([ce, we])\n",
    "    return self.flatten(dots)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-RLKz9LFECXu"
   },
   "source": [
    "### Define loss function and compile model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I3Md-9QanqBM"
   },
   "source": [
    "For simplicity, you can use `tf.keras.losses.CategoricalCrossEntropy` as an alternative to the negative sampling loss. If you would like to write your own custom loss function, you can also do so as follows:\n",
    "\n",
    "``` python\n",
    "def custom_loss(x_logit, y_true):\n",
    "      return tf.nn.sigmoid_cross_entropy_with_logits(logits=x_logit, labels=y_true)\n",
    "```\n",
    "\n",
    "It's time to build your model! Instantiate your Word2Vec class with an embedding dimension of 128 (you could experiment with different values). Compile the model with the `tf.keras.optimizers.Adam` optimizer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:03.306680Z",
     "iopub.status.busy": "2021-01-21T03:01:03.306015Z",
     "iopub.status.idle": "2021-01-21T03:01:03.338993Z",
     "shell.execute_reply": "2021-01-21T03:01:03.339416Z"
    },
    "id": "ekQg_KbWnnmQ"
   },
   "outputs": [],
   "source": [
    "embedding_dim = 128\n",
    "word2vec = Word2Vec(vocab_size, embedding_dim)\n",
    "word2vec.compile(optimizer='adam',\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P3MUMrluqNX2"
   },
   "source": [
    "Also define a callback to log training statistics for tensorboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:03.344614Z",
     "iopub.status.busy": "2021-01-21T03:01:03.343968Z",
     "iopub.status.idle": "2021-01-21T03:01:03.346083Z",
     "shell.execute_reply": "2021-01-21T03:01:03.345649Z"
    },
    "id": "9d-ftBCeEZIR"
   },
   "outputs": [],
   "source": [
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h5wEBotlGZ7B"
   },
   "source": [
    "Train the model with `dataset` prepared above for some number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:03.350012Z",
     "iopub.status.busy": "2021-01-21T03:01:03.349464Z",
     "iopub.status.idle": "2021-01-21T03:01:11.787650Z",
     "shell.execute_reply": "2021-01-21T03:01:11.788032Z"
    },
    "id": "gmC1BJalEZIY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 57s - loss: 1.6086 - accuracy: 0.2236"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 3/62 [>.............................] - ETA: 2s - loss: 1.6090 - accuracy: 0.2168 "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 1.6093 - accuracy: 0.2127"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "20/62 [========>.....................] - ETA: 0s - loss: 1.6093 - accuracy: 0.2135"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 1.6092 - accuracy: 0.2141"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "37/62 [================>.............] - ETA: 0s - loss: 1.6092 - accuracy: 0.2154"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "47/62 [=====================>........] - ETA: 0s - loss: 1.6091 - accuracy: 0.2170"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "55/62 [=========================>....] - ETA: 0s - loss: 1.6090 - accuracy: 0.2183"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 1s 7ms/step - loss: 1.6089 - accuracy: 0.2197\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.5899 - accuracy: 0.7832"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 1.5930 - accuracy: 0.6796"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "18/62 [=======>......................] - ETA: 0s - loss: 1.5937 - accuracy: 0.6478"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 1.5939 - accuracy: 0.6270"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "38/62 [=================>............] - ETA: 0s - loss: 1.5937 - accuracy: 0.6143"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 1.5935 - accuracy: 0.6064"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 1.5931 - accuracy: 0.5984"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.5928 - accuracy: 0.5937\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.5601 - accuracy: 0.7656"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 9/62 [===>..........................] - ETA: 0s - loss: 1.5630 - accuracy: 0.7238"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "18/62 [=======>......................] - ETA: 0s - loss: 1.5627 - accuracy: 0.7008"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 1.5616 - accuracy: 0.6862"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 1.5604 - accuracy: 0.6776"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 1.5588 - accuracy: 0.6681"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 1.5571 - accuracy: 0.6596"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.5558 - accuracy: 0.6543\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.4925 - accuracy: 0.6670"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 1.4947 - accuracy: 0.6363"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "21/62 [=========>....................] - ETA: 0s - loss: 1.4927 - accuracy: 0.6251"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "30/62 [=============>................] - ETA: 0s - loss: 1.4903 - accuracy: 0.6187"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 1.4878 - accuracy: 0.6135"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "49/62 [======================>.......] - ETA: 0s - loss: 1.4850 - accuracy: 0.6085"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "58/62 [===========================>..] - ETA: 0s - loss: 1.4824 - accuracy: 0.6045"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.4810 - accuracy: 0.6025\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.3973 - accuracy: 0.6113"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 1.3988 - accuracy: 0.6039"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "20/62 [========>.....................] - ETA: 0s - loss: 1.3964 - accuracy: 0.6010"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 1.3936 - accuracy: 0.5986"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 1.3905 - accuracy: 0.5963"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "47/62 [=====================>........] - ETA: 0s - loss: 1.3881 - accuracy: 0.5946"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 1.3855 - accuracy: 0.5928"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.3835 - accuracy: 0.5918\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.2968 - accuracy: 0.6182"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 1.2982 - accuracy: 0.6144"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 1.2959 - accuracy: 0.6141"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 1.2929 - accuracy: 0.6134"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "38/62 [=================>............] - ETA: 0s - loss: 1.2903 - accuracy: 0.6129"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "47/62 [=====================>........] - ETA: 0s - loss: 1.2880 - accuracy: 0.6121"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "57/62 [==========================>...] - ETA: 0s - loss: 1.2855 - accuracy: 0.6113"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.2840 - accuracy: 0.6110\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.2016 - accuracy: 0.6445"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 1.2031 - accuracy: 0.6404"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 1.2010 - accuracy: 0.6409"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 1.1985 - accuracy: 0.6415"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 1.1965 - accuracy: 0.6417"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "45/62 [====================>.........] - ETA: 0s - loss: 1.1945 - accuracy: 0.6416"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "54/62 [=========================>....] - ETA: 0s - loss: 1.1927 - accuracy: 0.6413"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - ETA: 0s - loss: 1.1911 - accuracy: 0.6412"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.1909 - accuracy: 0.6412\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.1132 - accuracy: 0.6758"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 1.1151 - accuracy: 0.6757"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 1.1134 - accuracy: 0.6760"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "27/62 [============>.................] - ETA: 0s - loss: 1.1114 - accuracy: 0.6762"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 1.1094 - accuracy: 0.6766"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "45/62 [====================>.........] - ETA: 0s - loss: 1.1077 - accuracy: 0.6766"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "53/62 [========================>.....] - ETA: 0s - loss: 1.1064 - accuracy: 0.6764"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - ETA: 0s - loss: 1.1049 - accuracy: 0.6763"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.1047 - accuracy: 0.6763\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 1.0313 - accuracy: 0.7100"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 1.0339 - accuracy: 0.7087"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 1.0325 - accuracy: 0.7093"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 1.0301 - accuracy: 0.7093"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "37/62 [================>.............] - ETA: 0s - loss: 1.0286 - accuracy: 0.7095"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 1.0272 - accuracy: 0.7093"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 1.0259 - accuracy: 0.7090"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 1.0249 - accuracy: 0.7090\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.9556 - accuracy: 0.7344"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 0.9586 - accuracy: 0.7377"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 0.9576 - accuracy: 0.7383"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 0.9555 - accuracy: 0.7384"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 0.9541 - accuracy: 0.7387"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "45/62 [====================>.........] - ETA: 0s - loss: 0.9529 - accuracy: 0.7386"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "54/62 [=========================>....] - ETA: 0s - loss: 0.9519 - accuracy: 0.7382"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.9508 - accuracy: 0.7382\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.8857 - accuracy: 0.7627"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.8891 - accuracy: 0.7651"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 0.8883 - accuracy: 0.7651"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 0.8863 - accuracy: 0.7650"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "37/62 [================>.............] - ETA: 0s - loss: 0.8848 - accuracy: 0.7652"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 0.8838 - accuracy: 0.7651"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "55/62 [=========================>....] - ETA: 0s - loss: 0.8830 - accuracy: 0.7647"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.8822 - accuracy: 0.7645\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.8213 - accuracy: 0.7881"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "10/62 [===>..........................] - ETA: 0s - loss: 0.8248 - accuracy: 0.7873"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 0.8243 - accuracy: 0.7871"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 0.8223 - accuracy: 0.7872"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "37/62 [================>.............] - ETA: 0s - loss: 0.8210 - accuracy: 0.7876"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 0.8202 - accuracy: 0.7875"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 0.8195 - accuracy: 0.7872"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.8188 - accuracy: 0.7871\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.7622 - accuracy: 0.8057"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.7659 - accuracy: 0.8057"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "21/62 [=========>....................] - ETA: 0s - loss: 0.7652 - accuracy: 0.8063"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "30/62 [=============>................] - ETA: 0s - loss: 0.7634 - accuracy: 0.8067"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 0.7621 - accuracy: 0.8072"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "49/62 [======================>.......] - ETA: 0s - loss: 0.7614 - accuracy: 0.8070"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "58/62 [===========================>..] - ETA: 0s - loss: 0.7609 - accuracy: 0.8067"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.7605 - accuracy: 0.8067\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.7082 - accuracy: 0.8145"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.7119 - accuracy: 0.8214"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "20/62 [========>.....................] - ETA: 0s - loss: 0.7114 - accuracy: 0.8226"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 0.7097 - accuracy: 0.8233"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 0.7084 - accuracy: 0.8240"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "48/62 [======================>.......] - ETA: 0s - loss: 0.7078 - accuracy: 0.8240"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "57/62 [==========================>...] - ETA: 0s - loss: 0.7074 - accuracy: 0.8238"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.7070 - accuracy: 0.8238\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.6588 - accuracy: 0.8320"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.6624 - accuracy: 0.8367"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 0.6623 - accuracy: 0.8376"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 0.6605 - accuracy: 0.8385"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "38/62 [=================>............] - ETA: 0s - loss: 0.6593 - accuracy: 0.8393"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "46/62 [=====================>........] - ETA: 0s - loss: 0.6588 - accuracy: 0.8394"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "55/62 [=========================>....] - ETA: 0s - loss: 0.6585 - accuracy: 0.8393"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.6580 - accuracy: 0.8393\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.6139 - accuracy: 0.8525"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 9/62 [===>..........................] - ETA: 0s - loss: 0.6170 - accuracy: 0.8525"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "18/62 [=======>......................] - ETA: 0s - loss: 0.6175 - accuracy: 0.8527"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 0.6157 - accuracy: 0.8537"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 0.6146 - accuracy: 0.8543"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "45/62 [====================>.........] - ETA: 0s - loss: 0.6140 - accuracy: 0.8545"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "55/62 [=========================>....] - ETA: 0s - loss: 0.6137 - accuracy: 0.8543"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.6134 - accuracy: 0.8542\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.5730 - accuracy: 0.8672"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.5764 - accuracy: 0.8659"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "21/62 [=========>....................] - ETA: 0s - loss: 0.5761 - accuracy: 0.8660"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 0.5747 - accuracy: 0.8666"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 0.5735 - accuracy: 0.8673"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "49/62 [======================>.......] - ETA: 0s - loss: 0.5731 - accuracy: 0.8673"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "57/62 [==========================>...] - ETA: 0s - loss: 0.5730 - accuracy: 0.8671"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.5727 - accuracy: 0.8670\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.5359 - accuracy: 0.8760"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.5391 - accuracy: 0.8761"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "19/62 [========>.....................] - ETA: 0s - loss: 0.5391 - accuracy: 0.8760"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 0.5377 - accuracy: 0.8766"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "38/62 [=================>............] - ETA: 0s - loss: 0.5365 - accuracy: 0.8773"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "47/62 [=====================>........] - ETA: 0s - loss: 0.5361 - accuracy: 0.8775"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "56/62 [==========================>...] - ETA: 0s - loss: 0.5359 - accuracy: 0.8774"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.5357 - accuracy: 0.8773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.5022 - accuracy: 0.8828"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 9/62 [===>..........................] - ETA: 0s - loss: 0.5049 - accuracy: 0.8834"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "18/62 [=======>......................] - ETA: 0s - loss: 0.5054 - accuracy: 0.8836"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "28/62 [============>.................] - ETA: 0s - loss: 0.5039 - accuracy: 0.8847"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "36/62 [================>.............] - ETA: 0s - loss: 0.5029 - accuracy: 0.8855"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "45/62 [====================>.........] - ETA: 0s - loss: 0.5024 - accuracy: 0.8859"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "55/62 [=========================>....] - ETA: 0s - loss: 0.5023 - accuracy: 0.8860"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.5021 - accuracy: 0.8860\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20/20\n",
      "\r",
      " 1/62 [..............................] - ETA: 0s - loss: 0.4716 - accuracy: 0.8965"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11/62 [====>.........................] - ETA: 0s - loss: 0.4745 - accuracy: 0.8938"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "21/62 [=========>....................] - ETA: 0s - loss: 0.4743 - accuracy: 0.8938"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "29/62 [=============>................] - ETA: 0s - loss: 0.4730 - accuracy: 0.8944"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "39/62 [=================>............] - ETA: 0s - loss: 0.4719 - accuracy: 0.8950"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "49/62 [======================>.......] - ETA: 0s - loss: 0.4717 - accuracy: 0.8952"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "57/62 [==========================>...] - ETA: 0s - loss: 0.4716 - accuracy: 0.8951"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "62/62 [==============================] - 0s 6ms/step - loss: 0.4715 - accuracy: 0.8951\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f65706f69b0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2vec.fit(dataset, epochs=20, callbacks=[tensorboard_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wze38jG57XvZ"
   },
   "source": [
    "Tensorboard now shows the Word2Vec model's accuracy and loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i0VuZcvm8zaI"
   },
   "source": [
    "```python\n",
    "%tensorboard --logdir logs\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "awF3iRQCZOLj"
   },
   "source": [
    "![word2vec_tensorboard](images/word2vec_tensorboard.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TaDW2tIIz8fL"
   },
   "source": [
    "## Embedding lookup and analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zp5rv01WG2YA"
   },
   "source": [
    "Obtain the weights from the model using `get_layer()` and `get_weights()`. The `get_vocabulary()` function provides the vocabulary to build a metadata file with one token per line. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:11.793637Z",
     "iopub.status.busy": "2021-01-21T03:01:11.792261Z",
     "iopub.status.idle": "2021-01-21T03:01:11.801164Z",
     "shell.execute_reply": "2021-01-21T03:01:11.800655Z"
    },
    "id": "_Uamp1YH8RzU"
   },
   "outputs": [],
   "source": [
    "weights = word2vec.get_layer('w2v_embedding').get_weights()[0]\n",
    "vocab = vectorize_layer.get_vocabulary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gWzdmUzS8Sl4"
   },
   "source": [
    "Create and save the vectors and metadata file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:11.806879Z",
     "iopub.status.busy": "2021-01-21T03:01:11.805849Z",
     "iopub.status.idle": "2021-01-21T03:01:12.149051Z",
     "shell.execute_reply": "2021-01-21T03:01:12.148328Z"
    },
    "id": "VLIahl9s53XT"
   },
   "outputs": [],
   "source": [
    "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
    "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
    "\n",
    "for index, word in enumerate(vocab):\n",
    "  if  index == 0: continue # skip 0, it's padding.\n",
    "  vec = weights[index] \n",
    "  out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
    "  out_m.write(word + \"\\n\")\n",
    "out_v.close()\n",
    "out_m.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1T8KcThhIU8-"
   },
   "source": [
    "Download the `vectors.tsv` and `metadata.tsv` to analyze the obtained embeddings in the [Embedding Projector](https://projector.tensorflow.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T03:01:12.155357Z",
     "iopub.status.busy": "2021-01-21T03:01:12.154229Z",
     "iopub.status.idle": "2021-01-21T03:01:12.156768Z",
     "shell.execute_reply": "2021-01-21T03:01:12.156132Z"
    },
    "id": "lUsjQOKMIV2z"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  from google.colab import files\n",
    "  files.download('vectors.tsv')\n",
    "  files.download('metadata.tsv')\n",
    "except Exception as e:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iS_uMeMw3Xpj"
   },
   "source": [
    "## Next steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BSgAZpwF5xF_"
   },
   "source": [
    "This tutorial has shown you how to implement a skip-gram Word2Vec model with negative sampling from scratch and visualize the obtained word embeddings.\n",
    "\n",
    "* To learn more about word vectors and their mathematical representations, refer to these [notes](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf).\n",
    "\n",
    "* To learn more about advanced text processing, read the [Transformer model for language understanding](https://www.tensorflow.org/tutorials/text/transformer) tutorial.\n",
    "\n",
    "* If you’re interested in pre-trained embedding models, you may also be interested in [Exploring the TF-Hub CORD-19 Swivel Embeddings](https://www.tensorflow.org/hub/tutorials/cord_19_embeddings_keras), or the [Multilingual Universal Sentence Encoder](https://www.tensorflow.org/hub/tutorials/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder)\n",
    "\n",
    "* You may also like to train the model on a new dataset (there are many available in [TensorFlow Datasets](https://www.tensorflow.org/datasets)).\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "word2vec.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}