doctors_nlp4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZUfspgOulSVP"
   },
   "source": [
    "+ title: Fine-tune a German GPT-2 Model with Tensorflow in Transformers for text generation (NLP part 4)\n",
    "+ date: 2021-03-19\n",
    "+ tags: python, NLP, text generation, GPT-2, tensorflow, transformers, transfer learning\n",
    "+ Slug: finetune-german-gpt2-on-tpu-transformers-tensorflow-for-text-generation-of-reviews\n",
    "+ Category: Python\n",
    "+ Authors: MC\n",
    "+ Summary: We build a model that can be prompted to generate human like positive and negative medical reviews in German. For that, we fine-tune GPT-2 on an unique data set using Tensorflow in Transformers. For fast results, we use TPUs on Google Colab. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MfP-g75VlSVV"
   },
   "source": [
    "## Motivation \n",
    "\n",
    "This is part four of our NLP series. In this series, we use a unique data set of German reviews of physicians written by their patients. So far, we've explored several methods for doing sentiment analysis / binary text classification. (Check out: [part 1]({filename}/doctors_nlp1.ipynb), [part 2]({filename}/doctors_nlp2.ipynb) and [part 3]({filename}/doctors_nlp3.ipynb))  \n",
    "Here, we move to an exciting new area: text generation with neural networks. Because our data set is not only extensive but also of high quality, it lends itself perfectly for training a neural network. As before, we make use of the power of transfer learning: We use a model that has been pre trained on German texts and fine-tune it using our review data. One of the currently available, best models for this task is GPT-2. It achieves amazing results in writing human like texts. A [German version of GPT-2](https://huggingface.co/dbmdz/german-gpt2) has been recently (Nov. 2020) released for the transformers library. Shout out to the [Bavarian State Library](https://github.com/dbmdz) for publishing it!  \n",
    "So far, there are few learning resources dealing with this model. A notable exception is this [blog post](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface) by Philipp Schmid. It shows how to use the model in transformers using PyTorch for creating novel recipes.  \n",
    "We add to this in several ways and also approach things a bit differently:\n",
    " * We use an extensive data set comprised of >400k German medical text reviews to fine-tune the model using the transformers library\n",
    " * We use the Tensorflow instead of PyTorch implementation and deal with some undocumented quirks\n",
    " * Using TPUs on Google Colab we reduce training time to a reasonable amount\n",
    " * Our data set has text comments and their corresponding rating. Hence, we're able to teach our model to generate positive and negative reviews, as requested\n",
    " \n",
    "You can download this notebook or follow along in an free interactive version of it on Google Colab: <a href=\"https://colab.research.google.com/github/mc51/blog_posts/blob/master/doctors_nlp4.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\" style=\"display:inline;\"/></a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IqpjhTBSlSVX"
   },
   "source": [
    "### Setup / Data set / cleaning / pre processing\n",
    "\n",
    "As this is not the focus of this post, we will go through these steps rather quickly. If you need a refresher on cleaning and pre processing, check out the [first]({filename}/doctors_nlp1.ipynb) and [second post]({filename}/doctors_nlp2.ipynb) again. \n",
    "\n",
    "We'll be using an updated version of the data that is more recent and contains even more reviews compared to the previous posts. You can take a look at it on [data.world](https://data.world/mc51/german-language-reviews-of-doctors-by-patients-2021) or directly download it from [here](https://query.data.world/s/wssnwo4qomcjbketegzxtort7pnaxk).  \n",
    "We will need to use a [TPU](https://cloud.google.com/tpu/docs/colabs) because of the high computational demand of GPT-2 and the size of the data. While you can get away with using GPUs as well, you won't stand any chance to run this on a CPU. Luckily, the notebooks on Google Colab offer free TPU usage. If you want to replicate this post, your best bet is to start there.  \n",
    "Now, let's get rolling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_7Fmtt1m8_GI",
    "outputId": "43f4e9e8-6691-4ccd-a948-c4610141ff0f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /drive\n",
      "2.4.1\n",
      "4.4.2\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# Do this only Google Colab\n",
    "if os.environ.get(\"COLAB_GPU\", False):\n",
    "    !pip install -U datasets transformers\n",
    "    from google.colab import drive\n",
    "\n",
    "    drive.mount(\"/drive\")\n",
    "\n",
    "import warnings\n",
    "import re\n",
    "import random\n",
    "import datasets\n",
    "import transformers\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from datetime import datetime\n",
    "from transformers import AutoTokenizer, TFGPT2LMHeadModel\n",
    "from datasets import Dataset, load_dataset\n",
    "\n",
    "pd.options.display.max_colwidth = 6000\n",
    "pd.options.display.max_rows = 400\n",
    "np.set_printoptions(suppress=True)\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "# Log Level and suppress extensive tf warnings\n",
    "os.environ[\"TRANSFORMERS_VERBOSITY\"] = \"info\"\n",
    "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\n",
    "print(tf.__version__)\n",
    "print(transformers.__version__)\n",
    "\n",
    "PATH_BASE = \"/drive/My Drive\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mi-5WjQjv-gH",
    "outputId": "a7b978a0-4af9-4982-d82c-33856ec9b944"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Entering into master device scope: /job:worker/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on TPU  ['10.92.53.250:8470']\n",
      "REPLICAS:  8\n"
     ]
    }
   ],
   "source": [
    "# Try to run on TPU if available\n",
    "# Detect hardware, return appropriate distribution strategy\n",
    "try:\n",
    "    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection\n",
    "    print(\"Running on TPU \", tpu.cluster_spec().as_dict()[\"worker\"])\n",
    "except ValueError:\n",
    "    tpu = None\n",
    "if tpu:\n",
    "    tf.config.experimental_connect_to_cluster(tpu)\n",
    "    tf.tpu.experimental.initialize_tpu_system(tpu)\n",
    "    strategy = tf.distribute.TPUStrategy(tpu)\n",
    "else:\n",
    "    strategy = tf.distribute.get_strategy()\n",
    "print(\"REPLICAS: \", strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Slu66k0UlSVa"
   },
   "source": [
    "We are running on 8 TPUs. For free. Thanks Google! Next, let's download the data and take a first look at it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "73lVmPORWkKG"
   },
   "outputs": [],
   "source": [
    "# Download and extract data\n",
    "!wget -O 2021_reviews.zip https://query.data.world/s/wssnwo4qomcjbketegzxtort7pnaxk\n",
    "!unzip 2021_reviews.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 155
    },
    "id": "VQ30F8BDXIeQ",
    "outputId": "38819b8e-6033-4691-a0c7-5a275c098e74"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rating</th>\n",
       "      <th>comment</th>\n",
       "      <th>bad_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>Ich bin franzose und bin seit ein paar Wochen in muenchen. Ich hatte Zahn Schmerzen und mein Kollegue hat mir Dr mainka empfohlen. Ich habe schnell ein Termin bekommen, das Team war nett und meine schmerzen sind weg!! Ich bin als Angst Patient sehr zurieden!!</td>\n",
       "      <td>&lt;|review_pos|&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.0</td>\n",
       "      <td>Dieser Arzt ist das unmöglichste was mir in meinem Leben je begegnet ist,er ist unfreundlich ,sehr herablassend und medizinisch unkompetent.Nach seiner Diagnose bin ich zu einem anderen Hautarzt gegangen der mich ordentlich behandelt hat und mir auch half.Meine Beschweerden hatten einen völlig anderen Grund.&lt;br /&gt;\\nNach seiner \" Behandlung \" und Diagnose ,waren seine letzten Worte .....und tschüss.Alles inerhalb von ca 5 Minuten.</td>\n",
       "      <td>&lt;|review_neg|&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>Hatte akute Beschwerden am Rücken. Herr Magura war der erste Arzt der sich wirklich Zeit für einen Therapieplan genommen hat um nachhaltig meine Schmerzen zu beseitigen</td>\n",
       "      <td>&lt;|review_pos|&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   rating  ...      bad_rating\n",
       "0     2.0  ...  <|review_pos|>\n",
       "1     6.0  ...  <|review_neg|>\n",
       "2     1.0  ...  <|review_pos|>\n",
       "\n",
       "[3 rows x 3 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read data\n",
    "reviews = pd.read_csv(\"2021_german_doctor_reviews.csv\")\n",
    "# mark bad ratings\n",
    "reviews[\"bad_rating\"] = np.where(\n",
    "    reviews[\"rating\"] > 3, \"<|review_neg|>\", \"<|review_pos|>\"\n",
    ")\n",
    "reviews.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PBlue03SlSVb"
   },
   "source": [
    "Here's an important first trick we use. As we have the review's comment and rating (1 best, 6 worst), we are able to classify negative and positive comments. In the previous posts, we've use this to create a classification model. This time, we will use this to teach our model to generate positive as well as negative reviews. For that, we create the special tokens `<|review_neg|>` and `<|review_pos|>` according to the rating. These will allow GPT-2 to learn the differences between good and bad reviews during training. Here's quick glimpse at the composition of our reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m-H0w7CCpzAR",
    "outputId": "c0c22949-96bc-479c-8386-1061992d7960"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<|review_pos|>    390134\n",
       "<|review_neg|>     49109\n",
       "Name: bad_rating, dtype: int64"
      ]
     },
     "execution_count": 62,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[\"bad_rating\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AVTvFBLflSVc"
   },
   "source": [
    "In total we have 439243 reviews. Most of them are positive, but still we have more than 49k negative ones as well.  \n",
    "Following, we pre process the comments. We do so by removing irregular characters from the text and making sure we use consistent spacing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xz9bGO2AXpZf",
    "outputId": "661acd21-b261-4356-fe39-4821d693c653"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 23.4 s, sys: 173 ms, total: 23.6 s\n",
      "Wall time: 23.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "def clean_text(text):\n",
    "    \"\"\"\n",
    "    - remove any html tags (< /br> often found)\n",
    "    - Keep only ASCII + Latin chars, digits and whitespaces\n",
    "    - pad punctuation chars with whitespace\n",
    "    - convert all whitespaces (tabs etc.) to single wspace\n",
    "    \"\"\"\n",
    "    RE_PUNCTUATION = re.compile(\"([!?.,;-])\")\n",
    "    RE_TAGS = re.compile(r\"<[^>]+>\")\n",
    "    RE_ASCII = re.compile(r\"[^A-Za-zÀ-ž,.!?0-9 ]\", re.IGNORECASE)\n",
    "    RE_WSPACE = re.compile(r\"\\s+\", re.IGNORECASE)\n",
    "    text = re.sub(RE_TAGS, \" \", text)\n",
    "    text = re.sub(RE_ASCII, \" \", text)\n",
    "    text = re.sub(RE_PUNCTUATION, r\" \\1 \", text)\n",
    "    text = re.sub(RE_WSPACE, \" \", text)\n",
    "    return text\n",
    "\n",
    "\n",
    "# Clean Comments. Only keep long enough\n",
    "reviews[\"comment_clean\"] = reviews.loc[reviews[\"comment\"].str.len() > 10, \"comment\"]\n",
    "reviews[\"comment_clean\"] = reviews[\"comment_clean\"].map(\n",
    "    lambda x: clean_text(x) if isinstance(x, str) else x\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "id": "rQh4Q-_QfZpW",
    "outputId": "49afc2e4-2a7a-4e55-9d1d-b32197b3a2e7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;|review_pos|&gt; Ich bin franzose und bin seit ein paar Wochen in muenchen . Ich hatte Zahn Schmerzen und mein Kollegue hat mir Dr mainka empfohlen . Ich habe schnell ein Termin bekommen , das Team war nett und meine schmerzen sind weg ! ! Ich bin als Angst Patient sehr zurieden ! !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;|review_neg|&gt; Dieser Arzt ist das unmöglichste was mir in meinem Leben je begegnet ist , er ist unfreundlich , sehr herablassend und medizinisch unkompetent . Nach seiner Diagnose bin ich zu einem anderen Hautarzt gegangen der mich ordentlich behandelt hat und mir auch half . Meine Beschweerden hatten einen völlig anderen Grund . Nach seiner Behandlung und Diagnose , waren seine letzten Worte . . . . . und tschüss . Alles inerhalb von ca 5 Minuten .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n",
       "0                                                                                                                                                                               <|review_pos|> Ich bin franzose und bin seit ein paar Wochen in muenchen . Ich hatte Zahn Schmerzen und mein Kollegue hat mir Dr mainka empfohlen . Ich habe schnell ein Termin bekommen , das Team war nett und meine schmerzen sind weg ! ! Ich bin als Angst Patient sehr zurieden ! ! \n",
       "1  <|review_neg|> Dieser Arzt ist das unmöglichste was mir in meinem Leben je begegnet ist , er ist unfreundlich , sehr herablassend und medizinisch unkompetent . Nach seiner Diagnose bin ich zu einem anderen Hautarzt gegangen der mich ordentlich behandelt hat und mir auch half . Meine Beschweerden hatten einen völlig anderen Grund . Nach seiner Behandlung und Diagnose , waren seine letzten Worte . . . . . und tschüss . Alles inerhalb von ca 5 Minuten . "
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop Missing and save to file\n",
    "reviews = reviews.dropna(axis=\"index\", subset=[\"comment_clean\"]).reset_index(drop=True)\n",
    "# add rating indicator as first word of comment\n",
    "reviews[\"comment_clean\"] = reviews[\"bad_rating\"] + \" \" + reviews[\"comment_clean\"]\n",
    "data = reviews[[\"comment_clean\"]]\n",
    "data.columns = [\"text\"]\n",
    "# data.to_csv(PATH_BASE + \"/data/reviews_clean_rating.csv\", index=False)\n",
    "data.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sEl0kgVClSVe"
   },
   "source": [
    "That'll be the foundation for creating the input to our GPT-2 model. We didn't have to do too much with our text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jnyc_LDNlSVe"
   },
   "source": [
    "### Create Model Inputs\n",
    "Following, we use the [datasets](https://huggingface.co/docs/datasets/) library to convert the pandas dataframe to a `Dataset` object. This object has some helpful properties when working with the transformers library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CeGF76Oc9FWv",
    "outputId": "798cf1bf-84f9-49f6-841c-febc21a4d10f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text'],\n",
       "    num_rows: 439243\n",
       "})"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read data from file and load as dataset\n",
    "data = pd.read_csv(PATH_BASE + \"/data/reviews_clean_rating.csv\")\n",
    "data = Dataset.from_pandas(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the pre trained tokenizer for German that comes with the model. As we've introduced tokens for negative and positive reviews, we add them to the tokenizer's vocabulary using `add_tokens`. We also add a new token for padding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272,
     "referenced_widgets": [
      "f002271f2aed4631b7ac0162179cc837",
      "5f689a5215194f00b0129938f402a9c1",
      "a16dec1bc165466aa94ef1ef0f1a561b",
      "0cd67ffa0c7845b994ad84e18a5fcf5c",
      "e244ee03e9d84eaa8a7ae94f6a66475f",
      "40f6b58f52f34a81a54f3fd65abb9a35",
      "926511e11f9f4b20b294ea108c78fc2c",
      "47ae582e4a6042aa8336bad34faeccae",
      "5cc509987c134a4d85b105b19deb3412",
      "e3848d5b7446473892f6f9728cf8f479",
      "7763aa44107f479e836c5f61bcb19f32",
      "0460335740884e439962b8fe92ffd541",
      "9dd6c4b71dcc45b98298663b8c488484",
      "fc1bbc17406946a899448b0db95ae594",
      "0592ffd34ba040f29e912d4c61859057",
      "91e9a5d5902c496d998ef0e05ac15589",
      "521fd9329f3248cf8067bc78a0363bba",
      "640ae95cb2644ee6bc64e3e389c8390c",
      "6d0e484ff31c41509c7e45efd7dc2111",
      "d8b566d7ebf74234b958267d68135dda",
      "2f4fddb4432746f6b0cf9b0529c0b9f2",
      "b0597e8ca8854a7d96678af48e957e98",
      "b8c313048e804381a938b9d59f922ab9",
      "7c36fba5084845e19a794e274752aba7",
      "ce7d793e883b4a7996c0aedec943be70",
      "4eff621547da4cc0ba301f6d106de34a",
      "fa189c1c5f544778adb8e9b5e47f73e1",
      "a662696e4427436db59c5a4b60fd1a84",
      "4e1757f336d14bdba0eb5dbf96765cdd",
      "46625128fd8f4c40affbf1451a6f7a0b",
      "3c19950b0c8a4cc5808f57a4ae9eeddd",
      "9d8895c7ba8c49dd9876828ecfde277c"
     ]
    },
    "id": "sUxlfURTDwaN",
    "outputId": "f88b1ff5-3e1e-4ecd-a919-bcb5d04353a3"
   },
   "outputs": [],
   "source": [
    "MAX_TOKENS = 128\n",
    "POS_TOKEN = \"<|review_pos|>\"\n",
    "NEG_TOKEN = \"<|review_neg|>\"\n",
    "BOS_TOKENS = [NEG_TOKEN, POS_TOKEN]\n",
    "EOS_TOKEN = \"<|endoftext|>\"\n",
    "PAD_TOKEN = \"<|pad|>\"\n",
    "\n",
    "# this will download and initialize the pre trained tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    \"dbmdz/german-gpt2\",\n",
    "    eos_token=EOS_TOKEN,\n",
    "    pad_token=PAD_TOKEN,\n",
    "    max_length=MAX_TOKENS,\n",
    "    is_split_into_words=True,\n",
    ")\n",
    "tokenizer.add_tokens(BOS_TOKENS, special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vPvflTAXlSVf"
   },
   "source": [
    "Coming up, we prepare the model input. For that, we have to do several tasks:\n",
    "1. We add our end of sentence (eos) token, `<|endoftext|>` at the end of each comment\n",
    "2. We tokenize each comment using the pre trained German [GPT-2 tokenizer](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast) we've initialized before\n",
    "3. While doing so, we add padding (using `<|pad|>` as a token) / or truncate each comment so that is has exactly `MAX_TOKENS` tokens\n",
    "4. The tokenizer then also takes care of converting the tokens to a numeric representation `input_ids` (each token corresponds to a specific numeric id)\n",
    "5. Next, we need to create `labels` as an input as well. These are actually the same as the shifted `input_ids`, but we replace ids corresponding to our padding token with `-100`\n",
    "6. Finally, the tokenizer also creates the `attention_mask`. This is just a vector consisting of `1` for all relevant elements in `input_ids` and `0` for all padding tokens  \n",
    "\n",
    "I'll explain a bite more in the following. But first, here's the corresponding code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f2RlcIYotPeY"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "output = {}\n",
    "# texts to numeric vectors of MAX_TOKENS\n",
    "def tokenize_function(examples, tokenizer=tokenizer):\n",
    "    # Add start and end token to each comment\n",
    "    examples = [ex + EOS_TOKEN for ex in examples[\"text\"]]\n",
    "    # tokenizer created input_ids and attention_mask as output\n",
    "    output = tokenizer(\n",
    "        examples,\n",
    "        add_special_tokens=True,  # Only adds pad not eos and bos\n",
    "        max_length=MAX_TOKENS,\n",
    "        truncation=True,\n",
    "        pad_to_max_length=True,\n",
    "    )\n",
    "    # shift labels for next token prediction\n",
    "    # set padding token labels to -100 which is ignored in loss computation\n",
    "    output[\"labels\"] = [x[1:] for x in output[\"input_ids\"]]\n",
    "    output[\"labels\"] = [\n",
    "        [-100 if x == tokenizer.pad_token_id else x for x in y]\n",
    "        for y in output[\"labels\"]\n",
    "    ]\n",
    "    # truncate input ids and attention mask to account for label shift\n",
    "    output[\"input_ids\"] = [x[:-1] for x in output[\"input_ids\"]]\n",
    "    output[\"attention_mask\"] = [x[:-1] for x in output[\"attention_mask\"]]\n",
    "    return output\n",
    "\n",
    "\n",
    "data = data.map(\n",
    "    tokenize_function,\n",
    "    batched=True,\n",
    "    num_proc=strategy.num_replicas_in_sync,\n",
    "    remove_columns=[\"text\"],\n",
    "    load_from_cache_file=True,\n",
    ")\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ipT0YwgdlSVg"
   },
   "source": [
    "The `map` method applies `tokenize_function` to our data set. By default, `batched=True` applies the function to a batch of 10k rows and using `num_proc` we can do this in parallel. We use a `MAX_TOKENS` value of 128. This covers about 80% of all reviews in our data set without truncation. Increasing this is costly in terms of additional computation time, so this is a good compromise.  \n",
    "We need to talk a bit more about what we're doing after we've applied `tokenizer` to our batches. This is because those quirks are currently **not properly documented** and **they differ in the Tensorflow transformers implementation of GPT-2** compared to the PyTorch version. In particular, the shifting of `input_ids` and `labels` happens inside the model during training when using the PyTorch model but not in the Tensorflow version. Thus, we need to explicitly do it beforehand. But why do we need the shifting at all? Well, we train GPT-2 on the task of causal language modeling. Basically, given a sequence of words the model learns to guess the next word. However, the model uses ids not words. So, given a sequence of `input_ids` what will be the next `input_id`? For computing the loss on this task, the model compares its predicted output with a label. Hence, `labels` must be the shifted `input_ids`.  \n",
    "Moreover, we don't want the model to train on or predict padding tokens. This is where the [attention mask](https://huggingface.co/transformers/glossary.html#attention-mask) comes into play. But this must be also be taken into account during loss computation. The transformer implementation of GPT-2 does this internally by ignoring all labels that are `-100`. Hence, we adapt our `labels` accordingly.  \n",
    "*Notice*: Currently this behavior is documented for the [PyTorch implementation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel) but not for the [Tensorflow one](https://huggingface.co/transformers/model_doc/gpt2.html#tfgpt2lmheadmodel).    \n",
    "Now, we're almost done preparing our data. We just need to create a split for training and testing and convert the data to the appropriate format for Tensorflow before we can start building our model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tnOz6LJTtSAY",
    "outputId": "e7645a3b-f7eb-41c3-bc99-d6d800bf2a0a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:datasets.arrow_dataset:Loading cached split indices for dataset at /drive/My Drive/data/reviews_tokenized_128_ratings/cache-1f0eaf0c3021a7e0.arrow and /drive/My Drive/data/reviews_tokenized_128_ratings/cache-9a58798e33d68773.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['attention_mask', 'input_ids', 'labels'],\n",
      "        num_rows: 351394\n",
      "    })\n",
      "    test: Dataset({\n",
      "        features: ['attention_mask', 'input_ids', 'labels'],\n",
      "        num_rows: 87849\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "# Load Inputs and create test and train split\n",
    "# data.save_to_disk(PATH_BASE + \"/data/reviews_tokenized_128_ratings\")\n",
    "data = datasets.load_from_disk(PATH_BASE + \"/data/reviews_tokenized_128_ratings\")\n",
    "data.set_format(type=\"python\", columns=[\"input_ids\", \"attention_mask\", \"labels\"])\n",
    "data = data.train_test_split(\n",
    "    test_size=0.20, shuffle=True, seed=1, load_from_cache_file=True\n",
    ")\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7JXLryO8-Sam",
    "outputId": "326e3d52-3b10-4e01-c343-ba70d76969a1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 23s, sys: 12.8 s, total: 2min 35s\n",
      "Wall time: 2min 36s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# prepare for use in tensorflow\n",
    "train_tensor_inputs = tf.convert_to_tensor(data[\"train\"][\"input_ids\"])\n",
    "train_tensor_labels = tf.convert_to_tensor(data[\"train\"][\"labels\"])\n",
    "train_tensor_mask = tf.convert_to_tensor(data[\"train\"][\"attention_mask\"])\n",
    "train = tf.data.Dataset.from_tensor_slices(\n",
    "    (\n",
    "        {\"input_ids\": train_tensor_inputs, \"attention_mask\": train_tensor_mask},\n",
    "        train_tensor_labels,\n",
    "    )\n",
    ")\n",
    "\n",
    "test_tensor_inputs = tf.convert_to_tensor(data[\"test\"][\"input_ids\"])\n",
    "test_tensor_labels = tf.convert_to_tensor(data[\"test\"][\"labels\"])\n",
    "test_tensor_mask = tf.convert_to_tensor(data[\"test\"][\"attention_mask\"])\n",
    "test = tf.data.Dataset.from_tensor_slices(\n",
    "    (\n",
    "        {\"input_ids\": test_tensor_inputs, \"attention_mask\": test_tensor_mask},\n",
    "        test_tensor_labels,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y7nvk5ENlSVh"
   },
   "source": [
    "This concludes the data preparation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hcm_3P1glSVh"
   },
   "source": [
    "### Build and train GPT-2 Model\n",
    "\n",
    "Next, we can start defining our model architecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "fq_m8UVc8sbZ"
   },
   "outputs": [],
   "source": [
    "# Model params\n",
    "BATCH_SIZE_PER_REPLICA = 28\n",
    "EPOCHS = 6\n",
    "INITAL_LEARNING_RATE = 0.001\n",
    "try:\n",
    "    BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\n",
    "except NameError as e:\n",
    "    BATCH_SIZE = BATCH_SIZE_PER_REPLICA\n",
    "BUFFER_SIZE = len(train)\n",
    "\n",
    "# prepare data for consumption\n",
    "train_ds = (\n",
    "    train.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n",
    ")\n",
    "test_ds = test.batch(BATCH_SIZE, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iaGjG_V6lSVi"
   },
   "source": [
    "The maximal batch size we can use on Colab in this case is about 28 per TPU. In total that will sum to 224. We set an initial learning rate that is probably higher than what is usually used for fine tuning. However, we will use a learning rate scheduler that decreases this rate rather quickly in the next step. This is by no means a well tested strategy, so feel free to play around with the learning rate yourself! The same goes for the number of epochs, which we keep on the lower side here to decrease training time.  \n",
    "Following, we set up our model from the pre trained version.  \n",
    "*Please note:* There is a [(TF)Trainer](https://huggingface.co/transformers/main_classes/trainer.html) class in transformers which provides a feature complete and simple training API. However, the Tensorflow implementation (`TFTrainer`) [does not currently work on TPUs](https://github.com/huggingface/transformers/issues/6672). Hence, we use the \"traditional\" way of defining and training our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1oqFlHCXnT9q",
    "outputId": "8a718d14-2684-465d-ba24-9b563d12c11d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint layers were used when initializing TFGPT2LMHeadModel.\n",
      "\n",
      "All the layers of TFGPT2LMHeadModel were initialized from the model checkpoint at dbmdz/german-gpt2.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"tfgp_t2lm_head_model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "transformer (TFGPT2MainLayer multiple                  125781504 \n",
      "=================================================================\n",
      "Total params: 125,781,504\n",
      "Trainable params: 125,781,504\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Drecreasing learning rate scheduler\n",
    "lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(\n",
    "    INITAL_LEARNING_RATE,\n",
    "    decay_steps=500,\n",
    "    decay_rate=0.7,\n",
    "    staircase=True)\n",
    "\n",
    "# initialize model, use_cache=False important! else wrong shape at loss calc\n",
    "with strategy.scope():\n",
    "    model = TFGPT2LMHeadModel.from_pretrained(\n",
    "        \"dbmdz/german-gpt2\",\n",
    "        use_cache=False,\n",
    "        pad_token_id=tokenizer.pad_token_id,\n",
    "        eos_token_id=tokenizer.eos_token_id,\n",
    "    )\n",
    "    model.resize_token_embeddings(len(tokenizer))\n",
    "    optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)\n",
    "    model.compile(optimizer=optimizer, loss=model.compute_loss)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6DNQOzTilSVj"
   },
   "source": [
    "Again, there are some pitfalls here. Most important, training will fail when we don't set `use_cache=False`! Also, we need to let the model know about the padding and eos tokens we are using. Moreover, we've added some tokens in addition to the ones that were included in the pre trained tokenizer and used to train the pre trained model. Consequently, we need to adapt the model to deal with that. We do this with the `resize_token_embeddings` method. Finally, the GPT-2 model implementation uses a custom function for computing the loss (for example it ignores labels containing -100, as mentioned before). So, instead of using one of the regular loss functions, we need to refer to the model's own `compute_loss` method.  \n",
    "Before we start training, let's define some callbacks that will be used during training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "JP5DJdA-0ywe"
   },
   "outputs": [],
   "source": [
    "# Stop training when validation acc starts dropping\n",
    "# Save checkpoint of model after each period\n",
    "now = datetime.now().strftime(\"%Y-%m-%d_%H%M\")\n",
    "# Create callbacks\n",
    "callbacks = [\n",
    "    tf.keras.callbacks.EarlyStopping(\n",
    "        monitor=\"val_loss\", verbose=1, patience=1, restore_best_weights=True\n",
    "    ),\n",
    "    tf.keras.callbacks.ModelCheckpoint(\n",
    "        PATH_BASE + \"/data/models/\" + now + \"_GPT2-Model_{epoch:02d}_{val_loss:.4f}.h5\",\n",
    "        monitor=\"val_loss\",\n",
    "        save_best_only=True,\n",
    "        verbose=1,\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6U1LSa23lSVj"
   },
   "source": [
    "We will stop model training prematurely, if the the validation loss does not improve after an epoch. Also, we make sure to save model checkpoints after each epoch so that we can resume training later on.  \n",
    "Now, we're all set for training. Let's start fine-tuning our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PgeHxshi-JQS",
    "outputId": "ace863a0-52b3-4568-9b0e-8894d116a2a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Params:\n",
      "batch_size: 224\n",
      "Epochs: 6\n",
      "Step p. Epoch: 1568\n",
      "Initial Learning rate: 0.001\n",
      "Epoch 1/6\n",
      "1568/1568 [==============================] - 737s 433ms/step - loss: 3.3801 - val_loss: 2.9180\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 2.91796, saving model to /drive/My Drive/data/models/2021-03-21_1925_GPT2-Model_01_2.9180.h5\n",
      "Epoch 2/6\n",
      "1568/1568 [==============================] - 673s 429ms/step - loss: 2.7971 - val_loss: 2.8363\n",
      "\n",
      "Epoch 00002: val_loss improved from 2.91796 to 2.83635, saving model to /drive/My Drive/data/models/2021-03-21_1925_GPT2-Model_02_2.8363.h5\n",
      "Epoch 3/6\n",
      "1568/1568 [==============================] - 673s 429ms/step - loss: 2.6261 - val_loss: 2.8273\n",
      "\n",
      "Epoch 00003: val_loss improved from 2.83635 to 2.82731, saving model to /drive/My Drive/data/models/2021-03-21_1925_GPT2-Model_03_2.8273.h5\n",
      "Epoch 4/6\n",
      "1568/1568 [==============================] - 673s 429ms/step - loss: 2.5464 - val_loss: 2.8364\n",
      "Restoring model weights from the end of the best epoch.\n",
      "\n",
      "Epoch 00004: val_loss did not improve from 2.82731\n",
      "Epoch 00004: early stopping\n",
      "CPU times: user 1min 33s, sys: 19.6 s, total: 1min 53s\n",
      "Wall time: 46min 52s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Train Model\n",
    "steps_per_epoch = int(BUFFER_SIZE // BATCH_SIZE)\n",
    "print(\n",
    "    f\"Model Params:\\nbatch_size: {BATCH_SIZE}\\nEpochs: {EPOCHS}\\n\"\n",
    "    f\"Step p. Epoch: {steps_per_epoch}\\n\"\n",
    "    f\"Initial Learning rate: {INITAL_LEARNING_RATE}\"\n",
    ")\n",
    "hist = model.fit(\n",
    "    train_ds,\n",
    "    validation_data=test_ds,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    epochs=EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "5GiDHBeCPuPp",
    "outputId": "2c103b4d-1f34-4821-ee1e-910f7fcca57b"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9dn//9eVhYQECCQE2UlYZN8EAUUFRCxVW23VuqFiqdbW1q1fb22/1tb7136r1apVsUqVuqDiUpfb1tutIqggSygKshRM2EFCwhYgZLt+f8wkhJiEkIVZ8n4+HvNgcs6ZM9cQeM9nPnPOdczdERGRyBcT6gJERKRxKNBFRKKEAl1EJEoo0EVEooQCXUQkSijQRUSihAJdmhUzyzAzN7O4Omw71cw+aeh+RI4XBbqELTNbb2ZFZta+yvJ/B8M0IzSViYQnBbqEuxzgsvIfzGwwkBS6ckTClwJdwt1zwFWVfr4aeLbyBmaWYmbPmlmumW0wszvNLCa4LtbM7jeznWaWDZxbzWOfMrNtZrbFzH5nZrHHWqSZdTaz/zGzfDNbZ2bXVlo3ysyWmNleM/vazB4ILk80s1lmlmdmu81ssZmdcKzPLVJOgS7h7jOgjZn1DwbtpcCsKts8AqQAPYFxBN4ArgmuuxY4DxgOjAQuqvLYp4ESoHdwm7OBH9WjztnAZqBz8Dn+n5mdGVz3Z+DP7t4G6AW8HFx+dbDubkAacD1wsB7PLQIo0CUylI/SJwGrgC3lKyqF/C/dfZ+7rwf+BFwZ3OQHwEPuvsnd84E/VHrsCcA5wM3uvt/ddwAPBvdXZ2bWDRgL3O7uhe6+DHiSw58sioHeZtbe3Qvc/bNKy9OA3u5e6u5Z7r73WJ5bpDIFukSC54DLgalUmW4B2gPxwIZKyzYAXYL3OwObqqwr1yP42G3BKY/dwBNAh2OsrzOQ7+77aqhhGnAisDo4rXJepdf1LjDbzLaa2R/NLP4Yn1ukggJdwp67byDw5eg5wGtVVu8kMNLtUWlZdw6P4rcRmNKovK7cJuAQ0N7d2wZvbdx94DGWuBVINbPW1dXg7mvd/TICbxT3Aq+aWbK7F7v73e4+ADiVwNTQVYjUkwJdIsU04Ex33195obuXEpiT/r2ZtTazHsCtHJ5nfxm40cy6mlk74I5Kj90GvAf8yczamFmMmfUys3HHUpi7bwLmA38IftE5JFjvLAAzm2Jm6e5eBuwOPqzMzCaY2eDgtNFeAm9MZcfy3CKVKdAlIrj7V+6+pIbVPwf2A9nAJ8ALwMzgur8SmNb4HFjKN0f4VwEtgJXALuBVoFM9SrwMyCAwWn8d+I27fxBcNxn40swKCHxBeqm7HwQ6Bp9vL4HvBuYSmIYRqRfTBS5ERKKDRugiIlFCgS4iEiUU6CIiUUKBLiISJULW+rN9+/aekZERqqcXEYlIWVlZO909vbp1IQv0jIwMliyp6Sg0ERGpjpltqGmdplxERKKEAl1EJEoo0EVEooSuhygija64uJjNmzdTWFgY6lIiVmJiIl27diU+vu4NOBXoItLoNm/eTOvWrcnIyMDMQl1OxHF38vLy2Lx5M5mZmXV+nKZcRKTRFRYWkpaWpjCvJzMjLS3tmD/hKNBFpEkozBumPn9/ERfoufsOcfdbX1JUorbRIiKVRVygL8rJ52+fruf/vr4ctf4Vkbo655xz2L17d63btGrVqtrlU6dO5dVXX22KshpVxH0peu6QTqzZ3puHP1xHz/RW/GR8r1CXJCJhzN1xd95+++1Ql9LkIm6EDnDLpBM5b0gn7n1nNe+s2BbqckTkOLjjjjuYPn16xc+//e1v+d3vfsfEiRM56aSTGDx4MG+++SYA69evp2/fvlx11VUMGjSITZs2kZGRwc6dOwG44IILGDFiBAMHDmTGjBlHPM8tt9zCwIEDmThxIrm5ud+oIysri3HjxjFixAi+9a1vsW1bGGVQ+bvX8b6NGDHCG+JgUYmf/+gn3vfOt/2LTbsbtC8RaVwrV65s9H0uXbrUzzjjjIqf+/fv7xs3bvQ9e/a4u3tubq736tXLy8rKPCcnx83MFyxYULF9jx49PDc3193d8/Ly3N39wIEDPnDgQN+5c6e7uwM+a9Ysd3e/++67/YYbbnB396uvvtpfeeUVLyoq8lNOOcV37Njh7u6zZ8/2a665ptFfa7nq/h6BJV5DrkbclEu5xPhY/nrVSC6Y/inTnlnMmz8bS6eUlqEuS0SayPDhw9mxYwdbt24lNzeXdu3a0bFjR2655RbmzZtHTEwMW7Zs4euvvwagR48ejBkzptp9Pfzww7z++usAbNq0ibVr15KWlkZMTAyXXHIJAFOmTOH73//+EY9bs2YNK1asYNKkSQCUlpbSqVN9LkHbNCI20AHSWyfw1NSRXPjYfKY9vYRXrj+F5ISIfkkiUouLL76YV199le3bt3PJJZfw/PPPk5ubS1ZWFvHx8WRkZFQcu52cnFztPj766CM++OADFixYQFJSEuPHj6/xeO+qhw66OwMHDmTBggWN+8IaSUTOoVfWr2MbHr38JFZv38vNLy2jtExHvohEq0suuYTZs2fz6quvcvHFF7Nnzx46dOhAfHw8c+bMYcOGGjvLVtizZw/t2rUjKSmJ1atX89lnn1WsKysrqzia5YUXXuC000474rF9+/YlNze3ItCLi4v58ssvG/EVNkzEBzrAhH4d+PV5A3h/5df88Z3VoS5HRJrIwIED2bdvH126dKFTp05cccUVLFmyhMGDB/Pss8/Sr1+/o+5j8uTJlJSU0L9/f+64444jpmWSk5NZtGgRgwYN4sMPP+Suu+464rEtWrTg1Vdf5fbbb2fo0KEMGzaM+fPnN/rrrC/zEB3LPXLkSG/MC1y4O3e9+SXPfbaBe74/mEtHdW+0fYvIsVm1ahX9+/cPdRkRr7q/RzPLcveR1W0fFSN0CMx1/eY7Azi9T3vufGMF89ftDHVJIiLHVdQEOkBcbAzTrziJzPbJXD8ri+zcglCXJCJy3ERVoAO0SYxn5tSTiYuN4YdPL2bX/qJQlyQiclxEXaADdEtNYsaVI9i6u5DrZ2WpkZeINAtHDXQzSzSzRWb2uZl9aWZ3V7PNGWa21MxKzOyipin12IzMSOWPFw1hYU6+GnmJSLNQl7NwDgFnunuBmcUDn5jZ/7r7Z5W22QhMBf5PE9RYbxcM70J2boEaeYlIs3DUEXqwfUD5t4vxwZtX2Wa9u38BhN3chhp5iTQ/u3fv5rHHHqvXY+vSZrey3/72t9x///31eq7GVqc5dDOLNbNlwA7gfXdf2LRlNR4z4/6LhzKsW1tufmkZyzfvCXVJItLEagv0kpKSWh/79ttv07Zt26Yoq8nVKdDdvdTdhwFdgVFmNqg+T2Zm15nZEjNbUl1byqZS3sgrLTmBac8sZtueg8ftuUXk+Lvjjjv46quvGDZsGLfddhsfffQRp59+Ot/97ncZMGAAUHML3fI2u+vXr6d///5ce+21DBw4kLPPPpuDB2vPjmXLljFmzBiGDBnC9773PXbt2gUEmoENGDCAIUOGcOmllwIwd+5chg0bxrBhwxg+fDj79u1r8Os+pk5W7r7bzOYAk4EVx/pk7j4DmAGBM0WP9fENUd7I66K/LFAjL5Hj6O63vmTl1r2Nus8Bndvwm+8MrHH9Pffcw4oVK1i2bBkQaMi1dOlSVqxYQWZmJgAzZ84kNTWVgwcPcvLJJ3PhhReSlpZ2xH7Wrl3Liy++yF//+ld+8IMf8Pe//50pU6bU+LxXXXUVjzzyCOPGjeOuu+7i7rvv5qGHHuKee+4hJyeHhISEiumc+++/n+nTpzN27FgKCgpITExs6F9LnY5ySTeztsH7LYFJQEQ2TOnXsQ2PXDZcjbxEmqFRo0ZVhDkERs1Dhw5lzJgxFS10q8rMzGTYsGEAjBgxgvXr19e4/z179rB7927GjRsHwNVXX828efMAGDJkCFdccQWzZs0iLi4wkBw7diy33norDz/8MLt3765Y3hB12UMn4BkziyXwBvCyu//DzP6bQKP1/zGzk4HXgXbAd8zsbnev+e0zhMobed391kr++M5qfnmO+k2INKXaRtLHU+V2unVtoZuQkFBxPzY29qhTLjX55z//ybx583jrrbf4/e9/z/Lly7njjjs499xzefvttxk7dizvvvtunZqL1eaogR48emV4NcvvqnR/MYH59Ygw9dQMsnP388S8bDLbJ6uRl0iUad26da1z0rW10K2vlJQU2rVrx8cff8zpp5/Oc889x7hx4ygrK2PTpk1MmDCB0047jdmzZ1NQUEBeXh6DBw9m8ODBLF68mNWrVzd9oEej8kZe6/P2c+cbK+iemsSpvduHuiwRaSRpaWmMHTuWQYMG8e1vf5tzzz33iPWTJ0/m8ccfp3///vTt27fGKxsdq2eeeYbrr7+eAwcO0LNnT/72t79RWlrKlClT2LNnD+7OjTfeSNu2bfn1r3/NnDlziImJYeDAgXz7299u8PNHTfvc+thbWMyFj83n672FvHHDWHqmtwppPSLRQu1zG0ezbZ9bH2rkJSLRpFkHOqiRl4hEj2Yf6KBGXiISHZrll6LVuWB4F7J37ufhf61VIy8RiUgK9EpuOasP2bkF3PvOajLbJzF5UKdQlyQiUmeacqmkvJHX8O5q5CUikUeBXkVifCwzrlQjL5FI1pD2uQAPPfQQBw4cqHbd+PHjCfUh1zVRoFejvJHXgaJSpj29hP2Ham+3KSLhpSkDPZwp0GugRl4ikatq+1yA++67j5NPPpkhQ4bwm9/8BoD9+/dz7rnnMnToUAYNGsRLL73Eww8/zNatW5kwYQITJkyo9XlefPFFBg8ezKBBg7j99tsBKC0tZerUqQwaNIjBgwfz4IMPAtW30G1s+lK0FpUbed37zmp+pUZeIsfuf++A7csbd58dB8O376lxddX2ue+99x5r165l0aJFuDvf/e53mTdvHrm5uXTu3Jl//vOfQKDHS0pKCg888ABz5syhffuaW4Js3bqV22+/naysLNq1a8fZZ5/NG2+8Qbdu3diyZQsrVgQ6jJe3y62uhW5j0wj9KKaemsGVY3owY142sxdtDHU5IlIP7733Hu+99x7Dhw/npJNOYvXq1axdu5bBgwfz/vvvc/vtt/Pxxx+TkpJS530uXryY8ePHk56eTlxcHFdccQXz5s2jZ8+eZGdn8/Of/5x33nmHNm3aANW30G1sGqEfhRp5iTRQLSPp48Xd+eUvf8mPf/zjb6xbunQpb7/9NnfeeScTJ07krrvuqmYPddeuXTs+//xz3n33XR5//HFefvllZs6cWW0L3cYOdo3Q6yAuNobpV5xEZvtkrp+VRXZuwdEfJCIhU7V97re+9S1mzpxJQUHg/+6WLVvYsWMHW7duJSkpiSlTpnDbbbexdOnSah9fnVGjRjF37lx27txJaWkpL774IuPGjWPnzp2UlZVx4YUX8rvf/Y6lS5ce0UL33nvvZc+ePRW1NCaN0OuovJHXBdM/5YdPL+b1n46lXXKLUJclItWo2j73vvvuY9WqVZxyyikAtGrVilmzZrFu3Tpuu+02YmJiiI+P5y9/+QsA1113HZMnT6Zz587MmTOn2ufo1KkT99xzDxMmTMDdOffcczn//PP5/PPPueaaaygrC/SF+sMf/lBjC93G1qzb59ZH1oZ8LpuxkOHd2/LctNG0iNOHHJGq1D63cah9bhMb0UONvEQkPGnKpR7UyEtEwpECvZ7UyEukdu6OmYW6jIhVn0//mnKpJzXyEqlZYmIieXl5mpKsJ3cnLy+PxMTEY3qcvhRtoNx9h7hg+qcUl5bx5s/G0imlZahLEgm54uJiNm/eTGFhYahLiViJiYl07dqV+Pj4I5bX9qWoAr0RrN6+l4v+soDuqUm8cv0pJCdoJktEmoaOcmli/Tq24ZHL1chLRELrqIFuZolmtsjMPjezL83s7mq2STCzl8xsnZktNLOMpig2nE3o24G7zhvA+yu/5t53Voe6HBFphuoyQj8EnOnuQ4FhwGQzG1Nlm2nALnfvDTwI3Nu4ZUaGq9XIS0RC6KiB7gHlTQfig7eqcwrnA88E778KTLRmeLxSeSOv0/u05843VjB/3c5QlyQizUid5tDNLNbMlgE7gPfdfWGVTboAmwDcvQTYA6Q1ZqGRQo28RCRU6hTo7l7q7sOArsAoMxtUnyczs+vMbImZLcnNza3PLiJCeSOv+NgYfvj0YnbtLwp1SSLSDBzTUS7uvhuYA0yusmoL0A3AzOKAFCCvmsfPcPeR7j4yPT29fhVHiG6pScy4agRbdxdy/awsikrKQl2SiES5uhzlkm5mbYP3WwKTgKqHcfwPcHXw/kXAh65TxNTIS0SOq7qcAdMJeMbMYgm8Abzs7v8ws/8Glrj7/wBPAc+Z2TogH2iaK6BGIDXyEpHj5aiB7u5fAMOrWX5XpfuFwMWNW1r0uOWsPuTs3K9GXiLSpHSm6HFgZtx30RA18hKRJqVAP04S42OZceVI0pITmPbMYrbtORjqkkQkyijQj6P01gk8NXUkB4pKmfb0EvYfKgl1SSISRRTox5kaeYlIU1Ggh4AaeYlIU1Dj7hC5+tQMvsrdz4x52fRsn8ylo7qHuiQRiXAK9BApb+S1If8Ad76xgu6pSZzau32oyxKRCKYplxCKi43h0cuHq5GXiDQKBXqIqZGXiDQWBXoYqGjktUeNvESk/hToYWJEj1TuUyMvEWkAfSkaRs4f1oWvctXIS0TqR4EeZtTIS0TqS1MuYUaNvESkvhToYUiNvESkPhToYSq9dQIzp56sRl4iUmcK9DDWt2NrNfISkTpToIc5NfISkbrSUS4RYOrYTLJ3qpGXiNROgR4h7jpvAOvz1MhLRGoWeVMuh/bBzrXQzM6kVCMvETmayAv0dR/AoyPhwYHwxk/hi5dh39ehruq4UCMvEamNhapnyMiRI33JkiXH/sC92+A/70D2R5AzFw7uCixP7w89xwduGWMhoXWj1Rpusjbkc9lfFzK8W1uemzaaFnGR974sIvVjZlnuPrLadREX6JWVlcH2LwLhnv0RbFwAJYUQEwddRkLPcYGA7zIS4lo0vOgw8uayLdw0exkXj+jKHy8agpmFuiQROQ4aFOhm1g14FjgBcGCGu/+5yjbtgJlAL6AQ+KG7r6htv40S6FUVF8LmRYcDfuu/wcsgPjkwau85PnDrMACiIAAfeP8/PPyvtdw+uZ8aeYk0Ew0N9E5AJ3dfamatgSzgAndfWWmb+4ACd7/bzPoB0919Ym37bZJAr+rgLlj/STDg50Le2sDy5HTIHHc44Nt2a9o6moi7c+PsZbz1+VYen3KSGnmJNAO1BfpRD1t0923AtuD9fWa2CugCrKy02QDgnuA2q80sw8xOcPfQflvZsh30/07gBrBncyDYy0fwK14NLE/tFQz3cZBxOiSlhqbeY1TeyGvzrgPc/NIyXmmbxOCuKaEuS0RC5Jjm0M0sA5gHDHL3vZWW/z+gpbvfYmajgPnAaHfPqmlfx2WEXht32LEq8MVq9keBkXxRAWDQedjh0Xu3MRCfGLo66yB33yEumP4pxaVlvPmzsXRKaRnqkkSkiTTKl6Jm1gqYC/ze3V+rsq4N8GdgOLAc6Adc6+7Lqmx3HXAdQPfu3Uds2LDhGF9KEyothi1Zh6dnNi+CshKITYDuYw4HfKehEBMb0lKrs2b7Pi78y3y6pybxyvWnkJygc8ZEolGDA93M4oF/AO+6+wNH2daAHGBI5VF8VSEfoR/NoQLYMP/w9MyOLwPLE9tC5hmHAz61Z9h8wTpnzQ6mPb2Yif1P4PEpI4iNCY+6RKTxNPRLUQOeAfLd/eYatmkLHHD3IjO7Fjjd3a+qbb9hH+hVFeyAnHmQPQe++gj2bg4sT+kWPDxyQiDoW3UIaZlPf5rDb99ayXVn9ORX5/QPaS0i0vga9KUoMBa4ElhuZuVTKL8CugO4++NAf+AZM3PgS2Bag6sON606wOCLAjd3yM8OhHv2R7DqH/DvWYHtThh0+AiaHqdCQqvjWqYaeYk0X5F9YlG4KCuFbZ9XOsHpMyg9FDjBqeuow0fQdBkBsfFNXk5JaRk/fGYJ89ft5NkfjlIjL5EoEr1nioar4oOBUC9vT7B1GeDQohVknHZ4/j29X5PNv+8tLObCx+bz9d5C3rhhLD3Tj+8nBRFpGgr0UDuQD+s/PjyCz88OLG91wpEnOKV0adSn3ZR/gAumf0rrxDhe/+lY2iVHV/sDkeZIgR5udm888gSnAzsDy9P6VGowdhq0bNvgp1IjL5HookAPZ2VlsGPl4emZ9Z9C8X6wGOg8vNIJTqMhLqFeT6FGXiLRo6FHuUhTiomBjoMCt1N/BiVFsGXJ4dH7Jw/Bx3+CuJbQ45TDUzQdhwQeWwfnD+tCdu5+/vyvtfRMb6VGXiJRSiP0cFe498gTnHJXBZa3TK1yglNmrbtRIy+R6KAReiRLbAN9JwduAPu2Hzn/vvKNwPK23Q+He+Y4SD7yUEU18hKJfhqhRzJ3yFt3ONxzPoZDewLrOg4Ohvv4wFRNi2RAjbxEIp2+FG0uSktg27LDAb9pIZQWQUx84EvVnuOh53jWxPbmwicWqZGXSARSoDdXRQcCl+UrP4Jm2xeAQ0IbctufzPT1XSnNHMdvr/kesbE6nFEkEijQJWB/Hqyfd3gEv2s9APvi02k94KzgETTjoE3nUFYpIrVQoEv18nN48/UXiVk/l0kt15BYtCuwvH3fSic4jYVEfXkqEi4U6FKj8kZeC9bt4JUL2jCsODgHv2E+lBwEi4UuJx0O+K4n1/sEJxFpOAW61KpyI6/XbxhLr/RWUHIINi8+PD2zJQu8DOKTAm2Be44P3DoMrPMJTiLScAp0OaqjNvI6uBs2fHr4En071wSWJ6UdPns1tSfEJUJci8CfscE/4xICt9gEiNURNRLFysoCrbNLgrfK90sOQUlhYFlKN2jfp15PoUCXOjmmRl57tx55glPB9ro9icUeGfBxCVV+TqxmWXXbVH3TKP+5uu3Lb5W21xtLdHGvFJhFgT9Lig4HaPm6imXl29QWvEfZprr9lhXXrd6xN8Oku+v1UhXoUmf1auTlDjvXBkL9G/+JqvvPUN1/umP4T+hlDX+hFnP4zeOY3zSq+eRRp8dH4RtLeZBW+/uu9Ls7pn8Dx/DvpHy/pUWN83piaxlMHNMA4ijbpHSFtt3qVaJO/Zc6q1cjLzNIPzFwOx5KS45hBFWXN40aAqRwL5Tm1jwqa8w3lmP6pFHLJ4+4Sm8esQmBq2aVHqrjG+2hum1TWuXvqzHE1vTmV+m1Jrap4xtmHd6Ua3qjjvBOpAp0+Yabz+pD9s793PvOajLbJ4VfI6/YOIgNgyswlZbUYdR5lDeNugTooX1QklvzG5OX1v81xLaoYVRZKewS29T+KaOun1xqeqOKbaEv1huJAl2+QY286ig2LnAL9skJmapvLJVH0GUlNY9WFaRRR3PoUiM18hIJP7XNoevtWWqU3jqBmVNP5kBRKdOeXsL+QyWhLklEaqFAl1r17diaRy4fzurte7n5pWWUloXmE52IHJ0CXY5qQt8O3HXeAN5f+TX3vrM61OWISA2OGuhm1s3M5pjZSjP70sxuqmabFDN7y8w+D25zTdOUK6EydWwmV53Sgxnzspm9aGOoyxGRatTlKJcS4BfuvtTMWgNZZva+u6+stM0NwEp3/46ZpQNrzOx5d2+kg1QlHNx13gDW5x3gzjdW0D01iVN7tz/6g0TkuDnqCN3dt7n70uD9fcAqoEvVzYDWFjitsBWQT+CNQKJIXGwMj14+nMz2yVw/K4uvcgtCXZKIVHJMc+hmlgEMBxZWWfUo0B/YCiwHbnL/5ml0ZnadmS0xsyW5ubn1KlhCq01iPDOnnkx8bAzTnl7Mrv36ECYSLuoc6GbWCvg7cLO7762y+lvAMqAzMAx41MzaVN2Hu89w95HuPjI9Pb0BZUsodUtNYsZVI9i6p5DrZ2VRVNIIp8CLSIPVKdDNLJ5AmD/v7q9Vs8k1wGsesA7IAfo1XpkSbkb0SOW+i4awMCef//v6ckJ1gpqIHFaXo1wMeApY5e4P1LDZRmBicPsTgL5AdmMVKeHp/GFduGliH17J2szjc/XrFgm1uhzlMha4ElhuZsuCy34FdAdw98eB/w942syWAwbc7u47m6BeCTOVG3klJ8Ry+ajuxMXq9AaRUFAvF2mwwuJSpj2zmE/X5dErPZlbJ/Xl24M6EhMT2a1IRcKRerlIk0qMj2XWtNE8PmUEsTHGDS8s5TuPfsKcNTs0ty5yHCnQpVGYGZMHdeR/bzqDBy8Zyt7CYq7522J+8MQCFuXkh7o8kWZBUy7SJIpKynh5ySYe/tdaduw7xLgT0/k/Z/dVX3WRBtI1RSVkDhaV8txn63nso6/YfaCYcwZ35NZJJ9K7Q+tQlyYSkRToEnL7Cot58uMcnvw4m4PFpXxveFduPqsP3VKTQl2aSERRoEvYyN9fxF8+WsezCzZQ5s5lo7rzswm96dAmMdSliUQEBbqEne17Cnnkw7W8tHgTcbHG1FMzuX5cT9omtQh1aSJhTYEuYWtD3n4e+mAtbyzbQqsWcVx3Rk+uOS2TVgm6frlIdRToEvbWbN/Hn95bw3srvyYtuQU/ndCbK0Z3JzE+NtSliYQVBbpEjGWbdnP/u2v4ZN1OOqUkcuPEPlw0oivxaicgAuhMUYkgw7q1ZdaPRvPCtaPpmJLIL19bzqQH5vLmsi2U6QLVIrVSoEtYOrVXe177yak8dfVIEuNjuWn2Ms55+GM+WPm12gmI1ECBLmHLzJjY/wTevvF0Hr5sOIXFpfzo2SV8/y/zmf+VmnmKVKVAl7AXE2N8d2hn3r91HPd8fzDb9xRy+V8XMuXJhSzbtDvU5YmEDX0pKhGnsLiU5xdu5LE568jbX8SkASfwi7NPpF/Hb1z1UCTq6CgXiUoFh0r42yc5zJiXTUFRCecP7cwtk06kR1pyqEsTaTIKdIlquw8U8cS8bP72aQ4lpc4PTu7GjWf2oWOK2glI9FGgS7OwY28h0+es44VFG4kx46pTevCT8b1JTVY7AYkeCt7o3YQAAA7HSURBVHRpVjblH+DP/1rLa0s30zI+lh+d3pMfnZ5J68T4UJcm0mAKdGmW1u3YxwPv/4e3l2+nbVI8PxnXi6tOyaBlC7UTkMilQJdmbfnmPdz/3hrm/ieXDq0T+PnEPlwyshst4nTUrkQeBboIsDA7j/vfW8Pi9bvoltqSW846kfOHdSE2xkJdmkidqZeLCDC6Zxov//gUnr7mZNokxnPry58z+aF5vLNiu9oJSFRQoEuzYmaM79uBt352Go9dcRJl7lw/K4vzp3/KvP/kKtgloh010M2sm5nNMbOVZvalmd1UzTa3mdmy4G2FmZWaWWrTlCzScDExxjmDO/HuzWdw30VDyCso4qqZi7h0xmdkbcgPdXki9XLUOXQz6wR0cvelZtYayAIucPeVNWz/HeAWdz+ztv1qDl3CyaGSUmYv2sQjH65jZ8EhzuzXgV+cfSIDO6eEujSRIzRoDt3dt7n70uD9fcAqoEstD7kMeLE+hYqESkJcLFefmsG8/xrP7ZP7kbVhF+c+/Ak/e2Ep2bkFoS5PpE6O6SgXM8sA5gGD3H1vNeuTgM1Ab3f/xudWM7sOuA6ge/fuIzZs2FC/qkWa2J6DxTz5cTZPfZLDoZIyLjqpKzee1YcubVuGujRp5hrlsEUzawXMBX7v7q/VsM0lwBR3/87R9qcpF4kEOwsO8dicr5j1WWDwccWY7vx0fG/SWyeEuDJprhp82KKZxQN/B56vKcyDLkXTLRJF2rdK4K7vDGDObeP5/kldeHbBBsbdN4f7313DnoPFoS5P5Ah1+VLUgGeAfHe/uZbtUoAcoJu77z/aE2uELpEoO7eABz9Yy1ufb6VNYhw/HteLa8ZmkNQiLtSlSTPRoCkXMzsN+BhYDpQFF/8K6A7g7o8Ht5sKTHb3S+tSlAJdItnKrXv503tr+NfqHbRvlcDPJvTistHdSYhTnxhpWjr1X6SJZG3YxX3vruaz7Hy6tG3JTWf14fvDuxAXq3P2pGno1H+RJjKiRztevHYMs6aNpn2rFvzXq19w9kPz+OcX2ygr01mncnwp0EUayMw4rU973rhhLE9cOYK4GOOGF5Zy3iOfMGf1DrUTkONGgS7SSMyMbw3syP/edAYPXjKUgkMlXPP0Yi5+fAELs/NCXZ40Awp0kUYWG2N8b3hXPrh1HL+7YBAb8w9wyYzPuGrmIpZv3hPq8iSK6UtRkSZWWFzKswvW85ePvmLXgWK+Pagjt046kT4ntA51aRKBdJSLSBjYV1jMU5/k8OTHORwoKuF7w7ty81l96JaaFOrSJIIo0EXCSP7+Ih6f+xXPzF9PmTuXjerOzyb0pkObxFCXJhFAgS4ShrbvKeSRD9fy0uJNxMUaV5+awfVn9KJdcotQlyZhTIEuEsY25O3noQ/W8sayLbRqEce1Z/Tkh6dl0ipB7QTkmxToIhFgzfZ9PPD+Gt798mtSk1vw0/G9mDKmB4nxaicghynQRSLIsk27+dN7a/h47U46pSRy48Q+XDSiK/FqJyDo1H+RiDKsW1uemzaaF64dTaeURH752nLOemAuby7bonYCUisFukiYOrVXe/7+k1N56uqRtIyP5abZyzjn4Y95f+XXaicg1VKgi4QxM2Ni/xN4+8bTefiy4RwqKePaZ5fwvcfmM3/dzlCXJ2FGgS4SAWJijO8O7cz7t5zBPd8fzNd7C7n8yYVc8eRn/HvjrlCXJ2FCX4qKRKDC4lJeWLiR6XPWkbe/iEkDTuAXZ59Iv45tQl2aNDEd5SISpfYfKuFvn+bwxLxsCg6VcP7Qztx81olktE8OdWnSRBToIlFu94EinpiXzd8+zaGk1Ll4ZDdunNibTiktQ12aNDIFukgzsWNfIdM/XMcLizZiZlx2cjcm9OvAiB7taJ0YH+rypBEo0EWamU35B/jzv9byxr+3UFLmxBgM6pLCqIxURmUGbm2T1DMmEinQRZqp/YdK+PfG3SzKyeOznHyWbdpNUUkZAP06tmZUZiqjM9M4ObMdHVqr22MkUKCLCBA4OubzTbtZlJPPovX5LFm/i4PFpQD0TE9mdGb5CD6NLm01/x6Oagt0tXMTaUYS42MZ3TON0T3TACguLWPFlj2BgM/J5x9fbOPFRZsA6NquZXAEHxjF90hLwsxCWb4chUboIlKhtMxZvX0vC7PzK0bx+fuLAOjQOiEQ8D3TGJ2ZSp8OrRTwIdCgKRcz6wY8C5wAODDD3f9czXbjgYeAeGCnu4+rbb8KdJHw5+6s21HAwuAIfmFOHl/vPQRAanILTs5ox6jMQMD379SG2BgFfFNraKB3Ajq5+1Izaw1kARe4+8pK27QF5gOT3X2jmXVw9x217VeBLhJ53J2N+QdYmJMfGMWvz2NT/kEAWifEMTKjHaN7pjEqM5XBXVLU8rcJNGgO3d23AduC9/eZ2SqgC7Cy0maXA6+5+8bgdrWGuYhEJjOjR1oyPdKS+cHIbgBs3X0wOHrPZ1FOHnPW5ALQMj6WET3aVRwmOaxbW12so4kd0xy6mWUA84BB7r630vLyqZaBQGvgz+7+bDWPvw64DqB79+4jNmzY0JDaRSQM5e47xOL1+SzMzmNhTj5rvt6HO7SIjWFYt7bBefhUTurejmRdZu+YNcphi2bWCpgL/N7dX6uy7lFgJDARaAksAM519//UtD9NuYg0D7sPFLF4/S4W5eSxKCefFVv3UlrmxMYYg7qkMCY4gh+ZkUpKS53NejQNPmzRzOKBvwPPVw3zoM1AnrvvB/ab2TxgKFBjoItI89A2qQWTBpzApAEnAFBwqISsDYGAX5idz8xgczEz6N+xTcWhkqMyU0lrlRDi6iNLXb4UNeAZIN/db65hm/7Ao8C3gBbAIuBSd19R0341QhcRCJzs9O+Nu1kYHMEv3biLwuLA2ay9O7SqCPfRmWl0TNHZrA0doY8FrgSWm9my4LJfAd0B3P1xd19lZu8AXwBlwJO1hbmISLnE+FhO6ZXGKb0CJzsVlZSxfMvuikMl31y2lecXbgSgR1pSRT+aMT3T6NqupY6Fr0QnFolIWCspLWPVtn0szAl8ybp4fT67DxQD0CklsWL0PiozlV7pyVEf8OrlIiJRo6zMWbujoCLgF2bns7MgcLJT+1YtAodJZgT60fTr2JqYKDvZSYEuIlHL3cnZub/iWPiF2Xls3VMIQJvEuIrj4EdnpjGwcxviIvxkJzXnEpGoZWb0TG9Fz/RWXDqqOwCbdx04oh/NB6sC5zomt4jlpB7tGBM8m3VI1xQS4qLnZCeN0EUk6n29t7Cio+TCnDz+83UBAAlxMQzv3pZRmWmMyUxlePd2tGwR3gGvKRcRkUry9xdVBPyi9Xms3LqXMof4WGNwl5SKfjQjw/DSfQp0EZFa7C0sJmv9rop+NF9s3lNx6b4BndtUHEUzKiOVdsmhvXSfAl1E5BgcKApcuq+8H82/K126r+8JrSv60YzKSKVDm+N7spMCXUSkAQ6VlPL5pj2BdgU5+WRt2MWBosCl+zLbV750Xypd2yU1aS0KdBGRRlRcWsaXW/dW9KNZtD6ffYUlAHRp2/Jwu4KeaWQ08qX7FOgiIk2otMxZs31fRT+aRTn55AUv3ZcevHTfmODFt/t0aNWgk50U6CIix5G781VuQcWZrJUv3dcuKZ6fju/NtWf0rNe+dWKRiMhxZGb07tCa3h1ac8XoHrg7m/IP8llwBH9CE3WNVKCLiDQxM6N7WhLd05IqLt3XFCK7qYGIiFRQoIuIRAkFuohIlFCgi4hECQW6iEiUUKCLiEQJBbqISJRQoIuIRImQnfpvZrnAhno+vD2wsxHLkcah30v40e8kPDXk99LD3dOrWxGyQG8IM1tSUy8DCR39XsKPfifhqal+L5pyERGJEgp0EZEoEamBPiPUBUi19HsJP/qdhKcm+b1E5By6iIh8U6SO0EVEpAoFuohIlIioQDezmWa2w8xWhLoWCTCzbmY2x8xWmtmXZnZTqGsSMLNEM1tkZp8Hfy93h7omCTCzWDP7t5n9o7H3HVGBDjwNTA51EXKEEuAX7j4AGAPcYGYDQlyTwCHgTHcfCgwDJpvZmBDXJAE3AauaYscRFejuPg/ID3Udcpi7b3P3pcH7+wj8Q+0S2qrEAwqCP8YHbzoCIsTMrCtwLvBkU+w/ogJdwpuZZQDDgYWhrUSg4qP9MmAH8L676/cSeg8B/wWUNcXOFejSKMysFfB34GZ33xvqegTcvdTdhwFdgVFmNijUNTVnZnYesMPds5rqORTo0mBmFk8gzJ9399dCXY8cyd13A3PQ90+hNhb4rpmtB2YDZ5rZrMZ8AgW6NIiZGfAUsMrdHwh1PRJgZulm1jZ4vyUwCVgd2qqaN3f/pbt3dfcM4FLgQ3ef0pjPEVGBbmYvAguAvma22cymhbomYSxwJYHRxrLg7ZxQFyV0AuaY2RfAYgJz6I1+mJyEF536LyISJSJqhC4iIjVToIuIRAkFuohIlFCgi4hECQW6iEiUUKCL1IOZjW+KbnkiDaFAFxGJEgp0iWpmNiXYF3yZmT0RbFhVYGYPBvuE/8vM0oPbDjOzz8zsCzN73czaBZf3NrMPgr3Fl5pZr+DuW5nZq2a22syeD541KxIyCnSJWmbWH7gEGBtsUlUKXAEkA0vcfSAwF/hN8CHPAre7+xBgeaXlzwPTg73FTwW2BZcPB24GBgA9CZw1KxIycaEuQKQJTQRGAIuDg+eWBFrJlgEvBbeZBbxmZilAW3efG1z+DPCKmbUGurj76wDuXggQ3N8id98c/HkZkAF80vQvS6R6CnSJZgY84+6/PGKh2a+rbFff/heHKt0vRf+fJMQ05SLR7F/ARWbWAcDMUs2sB4F/9xcFt7kc+MTd9wC7zOz04PIrgbnBqzBtNrMLgvtIMLOk4/oqROpIIwqJWu6+0szuBN4zsxigGLgB2E/ggg93EpiCuST4kKuBx4OBnQ1cE1x+JfCEmf13cB8XH8eXIVJn6rYozY6ZFbh7q1DXIdLYNOUiIhIlNEIXEYkSGqGLiEQJBbqISJRQoIuIRAkFuohIlFCgi4hEif8feXrhqXQPv5sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = pd.DataFrame(\n",
    "    {\"train loss\": hist.history[\"loss\"], \"test loss\": hist.history[\"val_loss\"]}\n",
    ").melt()\n",
    "loss[\"epoch\"] = loss.groupby(\"variable\").cumcount() + 1\n",
    "sns.lineplot(x=\"epoch\", y=\"value\", hue=\"variable\", data=loss).set(\n",
    "    title=\"Model loss\",\n",
    "    ylabel=\"\",\n",
    "    xticks=range(1, loss[\"epoch\"].max() + 1),\n",
    "    xticklabels=loss[\"epoch\"].unique(),\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wIt01nSRlSVk"
   },
   "source": [
    "After training for three epochs and about 45 minutes, the validation loss is around 2.8273 and seems to be flattening out. I'm sure, with some longer training times and adaptation of the hyperparameters we could improve this further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v9F9gICKlSVk"
   },
   "source": [
    "### Text generation\n",
    "\n",
    "Let's see how our training results translate to the quality of text generation. Using the [pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) class in transformers is straight forward for text generation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "we_RLH0NSlrR"
   },
   "outputs": [],
   "source": [
    "# Restored Trained Model weights\n",
    "# model.load_weights(PATH_BASE + \"/data/models/2021-03-21_1925_GPT2-Model_03_2.8273.h5\")\n",
    "\n",
    "from transformers import pipeline\n",
    "\n",
    "review = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x-QA5X7LlSVl"
   },
   "source": [
    "We call our `review` object using the token we defined for positive reviews as a prompt. Remember that we have trained our model to implicitly differentiate between positive and negative reviews using the appropriate tokens in the inputs. Consequently, the model hopefully learned to create reviews with different sentiments depending on the prompt we feed it. We wish to create outputs that are at most 150 tokens long, which should allow for mostly complete reviews (the model stops either when it generates an `<|endoftext|>` token or after `max_length` tokens) . Lastly, we ask for six different example reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 280
    },
    "id": "T65CJ00_DVyV",
    "outputId": "bceedcd9-0d94-4488-c6ae-d0f296f08529"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>generated_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;|review_pos|&gt; Habe mir bei meiner letzten Laserbehandlung mit dem Kryolipolyseverfahren die Augen lasern lassen . Es hat alles super geklappt ich hatte nach der Behandlung weder blaue Flecken , noch Pickelanfälle oder sonst irgendwelche Beschwerden !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;|review_pos|&gt; Ich habe leider sehr schlechte Zähne in der Familie , das kann ich aus eigener Erfahrung nicht behaupten</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;|review_pos|&gt; Ich kann mich über die Praxis und die Behandlung sehr gut und kompetent informieren . Man bekommt auch am Wochenende Zeit für Notfälle , das Wartezimmer ist immer ansprechend und auch die Wartezeit ist sehr gering .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;|review_pos|&gt; Ich war mit meiner Tochter in der Behandlung von Frau Dr . Brummel . Nach einer sehr ausführlichen Voruntersuchung wurde uns auf unsere Ängste und Sorgen sehr verständnis und kompetent und freundlich eingegangen . In einer Sitzung wurde mein Sohn zur Kontrolle gebeten . Hier fühlten wir uns mit allem gut aufgehoben . Vielen Dank dafür an Frau Dr . Brummel und ihr nettes Team , wir kommen gern wieder !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;|review_pos|&gt; Ich war mit meinem Sohn 5 bei Dr . Zerhusen . Er hat sich Zeit genommen und ist auf all meine Fragen und Wünsche eingegangen . Meine Kinder waren noch sehr jung und hatten Schwierigkeiten , die Behandlung von Dr . Zerhusen zu beginnen . Wir fühlen uns mit unserem Sohn bei ihm sehr gut aufgehoben und können ihn aus vollem Herzen weiterempfehlen .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;|review_pos|&gt; Bei Dr . Thiesing bin ich seit vielen Jahren in Behandlung , und bin absolut zufrieden , so wie von mir gewünscht . Sehr sehr sympathisch und kompetent , dabei immer freundlich und fürsorglich . Vorher hat alles bestens geklappt , und die neuen Ergebnisse freuen sich immer noch über jeden Besuch . Ich würde auch als junge Patientin in der Praxis weiter bleiben .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                         generated_text\n",
       "0                                                                                                                                                                          <|review_pos|> Habe mir bei meiner letzten Laserbehandlung mit dem Kryolipolyseverfahren die Augen lasern lassen . Es hat alles super geklappt ich hatte nach der Behandlung weder blaue Flecken , noch Pickelanfälle oder sonst irgendwelche Beschwerden ! \n",
       "1                                                                                                                                                                                                                                                                                                              <|review_pos|> Ich habe leider sehr schlechte Zähne in der Familie , das kann ich aus eigener Erfahrung nicht behaupten \n",
       "2                                                                                                                                                                                               <|review_pos|> Ich kann mich über die Praxis und die Behandlung sehr gut und kompetent informieren . Man bekommt auch am Wochenende Zeit für Notfälle , das Wartezimmer ist immer ansprechend und auch die Wartezeit ist sehr gering . \n",
       "3  <|review_pos|> Ich war mit meiner Tochter in der Behandlung von Frau Dr . Brummel . Nach einer sehr ausführlichen Voruntersuchung wurde uns auf unsere Ängste und Sorgen sehr verständnis und kompetent und freundlich eingegangen . In einer Sitzung wurde mein Sohn zur Kontrolle gebeten . Hier fühlten wir uns mit allem gut aufgehoben . Vielen Dank dafür an Frau Dr . Brummel und ihr nettes Team , wir kommen gern wieder ! \n",
       "4                                                           <|review_pos|> Ich war mit meinem Sohn 5 bei Dr . Zerhusen . Er hat sich Zeit genommen und ist auf all meine Fragen und Wünsche eingegangen . Meine Kinder waren noch sehr jung und hatten Schwierigkeiten , die Behandlung von Dr . Zerhusen zu beginnen . Wir fühlen uns mit unserem Sohn bei ihm sehr gut aufgehoben und können ihn aus vollem Herzen weiterempfehlen . \n",
       "5                                          <|review_pos|> Bei Dr . Thiesing bin ich seit vielen Jahren in Behandlung , und bin absolut zufrieden , so wie von mir gewünscht . Sehr sehr sympathisch und kompetent , dabei immer freundlich und fürsorglich . Vorher hat alles bestens geklappt , und die neuen Ergebnisse freuen sich immer noch über jeden Besuch . Ich würde auch als junge Patientin in der Praxis weiter bleiben . "
      ]
     },
     "execution_count": 47,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_pos = review(\"<|review_pos|>\", max_length=150, num_return_sequences=6)\n",
    "pd.DataFrame(gen_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3g2Ln3VGlSVl"
   },
   "source": [
    "I have to admit, I am super excited about the quality of those comments. Everything from grammar, to punctuation to spelling and even coherence seems natural with only few exceptions. I'm sure, most of these would easily pass as credible reviews! This really goes back to the fact that our data set is fantastic and we have a very well pre trained German GPT-2 to build upon.  \n",
    "But what about negative reviews? We had way fewer of those in our training set. Was it enough to fine-tune our model for creating negative reviews as well? Let's check it out by creating reviews using a different prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "Na_rumQpdVsP",
    "outputId": "b1455718-3b22-47b6-9f5d-dad62dbb9443"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>generated_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;|review_neg|&gt; Frau Dr . Wrensch wurde mir empfohlen . Sie wirkte auf mich ziemlich arrogant und unfreundlich . Ich habe mich nicht wohl bei ihr gefühlt . Als ich ihr meine Symptome schilderte , gab sie mir ein Rezept in die Hand und fragte mich daraufhin ob ich das Medikament nicht doch lieber nehme . Sie hat sich keine Zeit genommen und mich auch nicht weiter über mögliche Nebenwirkungen aufgeklärt . Auf meine Frage wurde ich unterbrochen ob sie mir nicht mal sagt , das wir das Medikament nehmen und nicht mit einer Packung ein Rezept aus der Apotheke zugeschickt werden . Ich werde nicht noch einmal hingehen .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;|review_neg|&gt; Ich hatte heute einen Termin und stand vor der verschlossenen Tür . Frau Dr . war pampig und unfreundlich .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;|review_neg|&gt; Dr . Heitkamp ist schon etwas älter . Er spricht auch nur mit einem über ein Kamm . Ich bin 20 Jahre alt und er hat mir in dem kurzen Zeitraum sehr geholfen . Es reicht ihm alles zu sagen , er ist ja jetzt da .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;|review_neg|&gt; Ich hatte schon mehrmals den Eindruck , dass ich von Dr . Thum untersucht wurde und dass er auch mit mir und meinem Problem nicht ausreichend kommuniziert hat . Ich musste immer wieder Nachfragen zu Beschwerden und auch zu Unverträglichkeiten und was der Grund für meine Beschwerden sein könnte , bekam ich immer wieder die Antwort ich soll doch ins nächste Krankenhaus gehen , da mein Problem nicht besser wird . Ich bin mir sicher , dass man dies auch auf die Art und Weise erklären kann und trotzdem nimmt er sich keine Zeit und geht mit seinem Latein am Ende aus der Tür . Die Arzthelferinnen wirken schon manchmal gestresst und genervt . Allerdings gibt es eine sehr nette Kollegin mit was auch immer sie der Praxis etwas eigen ist .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;|review_neg|&gt; Kann mich nur den vielen positiven Meinungen anschließen . Man bekommt relativ schnell einen Termin . Ist nicht gründlich , zieht einen auch vor wenn es um die Beschwerden geht . Die Untersuchung erfolgt unter Kurz und Überweisung ins Krankenhaus . Ich war lange Jahre Patientin in der Praxis und leider nicht zufrieden . Jetzt bin ich schwanger und fühle mich unwohl .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;|review_neg|&gt; Ich verstehe nur das es hier sehr viel zu wenig gute Ärzte gibt ! Man wird nicht Ernst genommen und auf die Beschwerden der Patienten wird gar nicht eingegangen ! Es war verlorene Zeit und Geldmacherei ! Die Arzthelferinnen sind nicht hilfsbereit !</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       generated_text\n",
       "0                                                                                                                                         <|review_neg|> Frau Dr . Wrensch wurde mir empfohlen . Sie wirkte auf mich ziemlich arrogant und unfreundlich . Ich habe mich nicht wohl bei ihr gefühlt . Als ich ihr meine Symptome schilderte , gab sie mir ein Rezept in die Hand und fragte mich daraufhin ob ich das Medikament nicht doch lieber nehme . Sie hat sich keine Zeit genommen und mich auch nicht weiter über mögliche Nebenwirkungen aufgeklärt . Auf meine Frage wurde ich unterbrochen ob sie mir nicht mal sagt , das wir das Medikament nehmen und nicht mit einer Packung ein Rezept aus der Apotheke zugeschickt werden . Ich werde nicht noch einmal hingehen . \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         <|review_neg|> Ich hatte heute einen Termin und stand vor der verschlossenen Tür . Frau Dr . war pampig und unfreundlich . \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  <|review_neg|> Dr . Heitkamp ist schon etwas älter . Er spricht auch nur mit einem über ein Kamm . Ich bin 20 Jahre alt und er hat mir in dem kurzen Zeitraum sehr geholfen . Es reicht ihm alles zu sagen , er ist ja jetzt da . \n",
       "3  <|review_neg|> Ich hatte schon mehrmals den Eindruck , dass ich von Dr . Thum untersucht wurde und dass er auch mit mir und meinem Problem nicht ausreichend kommuniziert hat . Ich musste immer wieder Nachfragen zu Beschwerden und auch zu Unverträglichkeiten und was der Grund für meine Beschwerden sein könnte , bekam ich immer wieder die Antwort ich soll doch ins nächste Krankenhaus gehen , da mein Problem nicht besser wird . Ich bin mir sicher , dass man dies auch auf die Art und Weise erklären kann und trotzdem nimmt er sich keine Zeit und geht mit seinem Latein am Ende aus der Tür . Die Arzthelferinnen wirken schon manchmal gestresst und genervt . Allerdings gibt es eine sehr nette Kollegin mit was auch immer sie der Praxis etwas eigen ist . \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                   <|review_neg|> Kann mich nur den vielen positiven Meinungen anschließen . Man bekommt relativ schnell einen Termin . Ist nicht gründlich , zieht einen auch vor wenn es um die Beschwerden geht . Die Untersuchung erfolgt unter Kurz und Überweisung ins Krankenhaus . Ich war lange Jahre Patientin in der Praxis und leider nicht zufrieden . Jetzt bin ich schwanger und fühle mich unwohl . \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <|review_neg|> Ich verstehe nur das es hier sehr viel zu wenig gute Ärzte gibt ! Man wird nicht Ernst genommen und auf die Beschwerden der Patienten wird gar nicht eingegangen ! Es war verlorene Zeit und Geldmacherei ! Die Arzthelferinnen sind nicht hilfsbereit ! "
      ]
     },
     "execution_count": 48,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_neg = review(\"<|review_neg|>\", max_length=150, num_return_sequences=6)\n",
    "pd.DataFrame(gen_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hwBjIEBSlSVm"
   },
   "source": [
    "First of all, each comment contains at least partly negative sentiments. So, training our model to differentiate between sentiments has been a success. However, my subjective feeling is that the quality is somewhat lower than before. While most of those are still pretty believable, coherence seems to have suffered a bit. Taking into account that we have only 49k negative examples in our data set, the result is still super fascinating!  \n",
    "*Please note*: I have not redacted the outputs at all. Those were all first take results! Often times, when people post incredible results they are at least somewhat redacted. Usually, you let the model output multiple examples and manually choose the most appropriate ones. Thus, what we see here really is just a lower bound in terms of quality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ttfTa1w8lSVm"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "We've learned how to fine-tune a pre trained German language GPT-2 model using the Transformers library and Tensorflow. For that, we've used a rich data set of German reviews of patients on their physicians. Leveraging the power of TPUs on Google Colab, we've trained the model on the task of causal language modeling. To do so, we provided work arounds to some quirks and undocumented pitfalls of the GPT-2 implementation. As a result, our model is able to generate human like, high quality text reviews. On top of that, it can be prompted to generate either positive or negative reviews.\n",
    "<br>"
   ]
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "collapsed_sections": [],
   "name": "doctors_nlp4.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0460335740884e439962b8fe92ffd541": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_91e9a5d5902c496d998ef0e05ac15589",
      "placeholder": "​",
      "style": "IPY_MODEL_0592ffd34ba040f29e912d4c61859057",
      "value": " 971k/971k [00:00&lt;00:00, 2.29MB/s]"
     }
    },
    "0592ffd34ba040f29e912d4c61859057": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0cd67ffa0c7845b994ad84e18a5fcf5c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_47ae582e4a6042aa8336bad34faeccae",
      "placeholder": "​",
      "style": "IPY_MODEL_926511e11f9f4b20b294ea108c78fc2c",
      "value": " 666/666 [00:01&lt;00:00, 458B/s]"
     }
    },
    "2f4fddb4432746f6b0cf9b0529c0b9f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "3c19950b0c8a4cc5808f57a4ae9eeddd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "40f6b58f52f34a81a54f3fd65abb9a35": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "46625128fd8f4c40affbf1451a6f7a0b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "47ae582e4a6042aa8336bad34faeccae": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4e1757f336d14bdba0eb5dbf96765cdd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "4eff621547da4cc0ba301f6d106de34a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "521fd9329f3248cf8067bc78a0363bba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6d0e484ff31c41509c7e45efd7dc2111",
       "IPY_MODEL_d8b566d7ebf74234b958267d68135dda"
      ],
      "layout": "IPY_MODEL_640ae95cb2644ee6bc64e3e389c8390c"
     }
    },
    "5cc509987c134a4d85b105b19deb3412": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7763aa44107f479e836c5f61bcb19f32",
       "IPY_MODEL_0460335740884e439962b8fe92ffd541"
      ],
      "layout": "IPY_MODEL_e3848d5b7446473892f6f9728cf8f479"
     }
    },
    "5f689a5215194f00b0129938f402a9c1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "640ae95cb2644ee6bc64e3e389c8390c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6d0e484ff31c41509c7e45efd7dc2111": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b0597e8ca8854a7d96678af48e957e98",
      "max": 512918,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2f4fddb4432746f6b0cf9b0529c0b9f2",
      "value": 512918
     }
    },
    "7763aa44107f479e836c5f61bcb19f32": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fc1bbc17406946a899448b0db95ae594",
      "max": 970833,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9dd6c4b71dcc45b98298663b8c488484",
      "value": 970833
     }
    },
    "7c36fba5084845e19a794e274752aba7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "91e9a5d5902c496d998ef0e05ac15589": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "926511e11f9f4b20b294ea108c78fc2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9d8895c7ba8c49dd9876828ecfde277c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9dd6c4b71dcc45b98298663b8c488484": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "a16dec1bc165466aa94ef1ef0f1a561b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_40f6b58f52f34a81a54f3fd65abb9a35",
      "max": 666,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e244ee03e9d84eaa8a7ae94f6a66475f",
      "value": 666
     }
    },
    "a662696e4427436db59c5a4b60fd1a84": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9d8895c7ba8c49dd9876828ecfde277c",
      "placeholder": "​",
      "style": "IPY_MODEL_3c19950b0c8a4cc5808f57a4ae9eeddd",
      "value": " 62.0/62.0 [00:00&lt;00:00, 489B/s]"
     }
    },
    "b0597e8ca8854a7d96678af48e957e98": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b8c313048e804381a938b9d59f922ab9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ce7d793e883b4a7996c0aedec943be70": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_fa189c1c5f544778adb8e9b5e47f73e1",
       "IPY_MODEL_a662696e4427436db59c5a4b60fd1a84"
      ],
      "layout": "IPY_MODEL_4eff621547da4cc0ba301f6d106de34a"
     }
    },
    "d8b566d7ebf74234b958267d68135dda": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c36fba5084845e19a794e274752aba7",
      "placeholder": "​",
      "style": "IPY_MODEL_b8c313048e804381a938b9d59f922ab9",
      "value": " 513k/513k [00:00&lt;00:00, 650kB/s]"
     }
    },
    "e244ee03e9d84eaa8a7ae94f6a66475f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "e3848d5b7446473892f6f9728cf8f479": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f002271f2aed4631b7ac0162179cc837": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a16dec1bc165466aa94ef1ef0f1a561b",
       "IPY_MODEL_0cd67ffa0c7845b994ad84e18a5fcf5c"
      ],
      "layout": "IPY_MODEL_5f689a5215194f00b0129938f402a9c1"
     }
    },
    "fa189c1c5f544778adb8e9b5e47f73e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_46625128fd8f4c40affbf1451a6f7a0b",
      "max": 62,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4e1757f336d14bdba0eb5dbf96765cdd",
      "value": 62
     }
    },
    "fc1bbc17406946a899448b0db95ae594": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}