language_model/nn-czech.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Czech ULMFiT from scratch\n",
    "\n",
    "This notebook is based on [Turkish ULMFiT from scratch](https://github.com/fastai/course-nlp/blob/master/nn-turkish.ipynb) notebook from [fast.ai](http://fast.ai) [NLP course](https://github.com/fastai/course-nlp/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Tesla P4'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GPU test\n",
    "#import torch\n",
    "#torch.cuda.current_device()\n",
    "#torch.cuda.device(0)\n",
    "#torch.cuda.device_count()\n",
    "#torch.cuda.get_device_name(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=128\n",
    "data_path = Config.data_path()\n",
    "\n",
    "lang = 'cs'\n",
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "lm_fns = [mdl_path/f'{lang}_wt', mdl_path/f'{lang}_wt_vocab']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Czech wikipedia model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nlputils import split_wiki,get_wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unzipping...\n",
      "extracting...\n",
      "<doc id=\"10\" url=\"https://cs.wikipedia.org/wiki?curid=10\" title=\"Astronomie\">\n",
      "Astronomie\n",
      "\n",
      "Astronomie, řecky αστρονομία z άστρον (astron) hvězda a νόμος (nomos) zákon, česky též hvězdářství, je věda, která se zabývá jevy za hranicemi zemské atmosféry. Zvláště tedy výzkumem vesmírných těles, jejich soustav, různých dějů ve vesmíru i vesmírem jako celkem.\n"
     ]
    }
   ],
   "source": [
    "get_wiki(path, lang)\n",
    "!head -n4 {path}/{name}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.fastai/data/cswiki/docs already exists; not splitting\n"
     ]
    }
   ],
   "source": [
    "dest = split_wiki(path, lang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Czech language](https://en.wikipedia.org/wiki/Czech_language) has declinations, so it needs special care!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(30000, 97506)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz = 30000)])\n",
    "        .split_by_rand_pct(0.1, seed=42)\n",
    "        .label_for_lm()\n",
    "        .databunch(bs=bs, num_workers=1))\n",
    "\n",
    "data.save(f'{lang}_databunch')\n",
    "len(data.vocab.itos),len(data.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(dest, f'{lang}_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>▁v ▁severní ▁části ▁xxmaj ▁polska , ▁ve ▁vojvodství ▁xxmaj ▁po mor ski e , ▁poblíž ▁obcí ▁xxmaj ▁ ch oj nice , ▁xxmaj ▁brus y , ▁xxmaj ▁char zy kow y ▁v ▁srdci ▁xxmaj ▁tu ch ol ských ▁lesů , ▁největším ▁zalesněné m ▁území ▁v ▁xxmaj ▁polsku . ▁xxmaj ▁správa ▁parku ▁je ▁v ▁obci , ▁která ▁se ▁nachází ▁3 ▁km ▁od ▁xxmaj ▁ ch oj nic . ▁xxmaj ▁nachází ▁se</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>lon ▁( st . ▁xxmaj ▁peter sburg , ▁xxup ▁usa ). ▁xxmaj ▁je ▁deseti násobný m ▁mistrem ▁xxup ▁čr ▁v ▁olympijské m ▁tri at lon u . ▁xxmaj ▁při ▁svém ▁prvním ▁startu ▁na ▁dlouhé m ▁tri at lon u ▁- ▁xxmaj ▁iron man ▁xxmaj ▁florida ▁2013 ▁obsadil ▁3. ▁místo ▁a ▁časem ▁7 : 58 : 44 ▁se ▁zařadil ▁mezi ▁ne ce lou ▁dva cí tku ▁atlet ů ▁světa , ▁kteří</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>▁páni ▁z ▁xxmaj ▁ne č tin . ▁xxmaj ▁nový ▁majitel ▁pro ▁město ▁roku ▁14 16 ▁získal ▁právo ▁pořád at ▁výroční ▁trh . ▁xxmaj ▁jindřich ▁nepře síd lil ▁na ▁ žlut ický ▁hrad . ▁xxmaj ▁jako ▁katolík ▁žil ▁až ▁do ▁konce ▁života ▁v ▁neu stál ých ▁bojích ▁s ▁ husit y . ▁xxmaj ▁roku ▁1421 ▁byly ▁xxmaj ▁ žlut ice ▁obsazen y ▁křižáck ým ▁vojskem . ▁xxmaj ▁praž ané ▁město ▁o</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>▁festivalu , ▁přibyl o ▁oblíbený ch ▁xxmaj ▁ bus ▁xxmaj ▁partie s , ▁na bit ý ▁program ▁nabídl ▁již ▁podruhé ▁v ▁řadě ▁xxup ▁ bec ▁stan . ▁xxmaj ▁festival ▁pak ▁dále ▁pokračuje ▁ve ▁využívání ▁bez ho t ovost ní ▁systému ▁plat eb ▁a ▁rozšiřuje ▁ekologické ▁aktivity ▁v ▁rámci ▁programu ▁xxmaj ▁ roll ▁xxmaj ▁in ▁xxmaj ▁green . ▁xxmaj ▁oficiální ▁web ▁festivalu ▁xxmaj ▁oficiální ▁xxmaj ▁facebook ▁festivalu ▁xxmaj ▁oficiální ▁youtube ▁festivalu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>▁jsou ▁garant ovány ▁po ▁dobu ▁20 ▁let ▁provozování ▁konkrétní ho ▁zařízení . ▁xxmaj ▁tzv . ▁zelený ▁bonus ▁se ▁však ▁mění ▁každý ▁rok ▁a ▁pro ▁konkrétní ▁výpočet ▁návrat nosti ▁je ▁potřeba ▁z o hled nit ▁cen ový ▁výnos ▁xxup ▁ eru ▁v ▁době ▁po řízení ▁foto vol ta ické ▁elektrárny . ▁xxmaj ▁pro ▁provozovatel e ▁je ▁jistě ▁zajímavá ▁i ▁skutečnost , ▁že ▁je ▁po ▁dobu ▁pěti ▁let ▁osvobozen ▁od ▁daní ▁z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "learn = language_model_learner(data, AWD_LSTM, drop_mult=0.1, wd=0.1, pretrained=False).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='2' class='' max='4', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      50.00% [2/4 4:35:52<4:35:52]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.261763</td>\n",
       "      <td>4.478383</td>\n",
       "      <td>0.271987</td>\n",
       "      <td>2:17:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.132481</td>\n",
       "      <td>4.315895</td>\n",
       "      <td>0.286108</td>\n",
       "      <td>2:18:05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='7511' class='' max='13238', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      56.74% [7511/13238 1:15:18<57:25 3.9775]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# takes ~28 hours, therefore rather run this in terminal\n",
    "# >>> lr = 3e-3\n",
    "# >>> lr *= bs/48\n",
    "# >>> learn.unfreeze()\n",
    "# >>> learn.fit_one_cycle(12, lr, moms=(0.8,0.7))\n",
    "# epoch     train_loss  valid_loss  accuracy  time\n",
    "# 0         4.220986    4.310223    0.289666  2:20:18\n",
    "# 1         4.138725    4.280440    0.289980  2:20:52\n",
    "# 2         4.187871    4.383055    0.280299  2:21:02\n",
    "# 3         4.170604    4.376703    0.281176  2:21:01\n",
    "# 4         4.160329    4.325862    0.285679  2:21:03\n",
    "# 5         4.060669    4.256678    0.292394  2:21:02\n",
    "# 6         4.003880    4.159348    0.300838  2:21:05\n",
    "# 7         3.907038    4.030453    0.313692  2:20:51\n",
    "# 8         3.787868    3.886877    0.329324  2:20:52\n",
    "# 9         3.651094    3.736353    0.346885  2:20:45\n",
    "# 10        3.611639    3.603158    0.364177  2:20:39\n",
    "# 11        3.561713    3.551184    0.371601  2:20:40\n",
    "\n",
    "# learn.to_fp32().save(lm_fns[0], with_opt=False)\n",
    "# learn.data.vocab.save(lm_fns[1].with_suffix('.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Czech sentiment analysis resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://liks.fav.zcu.cz/sentiment/\n",
    "\n",
    "https://www.aclweb.org/anthology/W13-1609"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/positive.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/neutral.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/licence.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/cs_clas_databunch'),\n",
       " PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/models'),\n",
       " PosixPath('/home/jupyter/.fastai/data/cswiki/csfd/negative.txt')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_clas = path/'csfd'\n",
    "path_clas.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jako se mě líbila stejně jako jednička. Má sic...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sice se rika ze dvojky pokulhavaji za jednicko...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tenhle film můžu opravdu kdykoliv. A to nejen ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>velice povedená krimikomedie s plejádou skvělý...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Rok 1969. Při výročí invaze střílejí do lidí č...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  pos\n",
       "0  Jako se mě líbila stejně jako jednička. Má sic...    1\n",
       "1  Sice se rika ze dvojky pokulhavaji za jednicko...    1\n",
       "2  Tenhle film můžu opravdu kdykoliv. A to nejen ...    1\n",
       "3  velice povedená krimikomedie s plejádou skvělý...    1\n",
       "4  Rok 1969. Při výročí invaze střílejí do lidí č...    1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos = (path_clas/'positive.txt').open().readlines()\n",
    "pos_df = pd.DataFrame({'text':pos})\n",
    "pos_df['pos'] = 1\n",
    "pos_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Tommy Lee Jonesovi jsou role jako je tato šity...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>se stejnou upřímností a váhou jako je Pravdivá...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fakt blbost. 1. díl byl ucházející a dalo se n...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jako mladíkovi uprostřed normalizačních sedmde...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>...aneb proč neudělat ze Tří mušketyrů sci-fi....</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  pos\n",
       "0  Tommy Lee Jonesovi jsou role jako je tato šity...    0\n",
       "1  se stejnou upřímností a váhou jako je Pravdivá...    0\n",
       "2  Fakt blbost. 1. díl byl ucházející a dalo se n...    0\n",
       "3  Jako mladíkovi uprostřed normalizačních sedmde...    0\n",
       "4  ...aneb proč neudělat ze Tří mušketyrů sci-fi....    0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neg = (path_clas/'negative.txt').open().readlines()\n",
    "neg_df = pd.DataFrame({'text':neg})\n",
    "neg_df['pos'] = 0\n",
    "neg_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([pos_df,neg_df], sort=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_df(df, path_clas, cols='text', processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))\n",
    "\n",
    "data_lm.save(f'{lang}_clas_databunch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path_clas, f'{lang}_clas_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>o . ▁a ▁o pet ▁nez klam ala ▁ perfekt ni ▁ muzik a ! ! ! ▁xxbos ▁xxmaj ▁ten hle ▁film ▁ m ů ž u ▁opravdu ▁kdykoliv . ▁a ▁to ▁nejen ▁díky ▁par fek tní mu , ▁ hlá ška mi ▁na bit ému ▁scénář i ▁a ▁herecký m ▁výkonů m ▁všech ▁z ů čas t ně ných ▁( hlavně ▁pak ▁trojice ▁xxmaj ▁troj an , ▁xxmaj ▁mit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>▁před ▁sebou ▁2 ▁prázdné ▁krabic e ▁od ▁velkého ▁pop corn u , ▁ prázdn á ▁dvou lit rov ka ▁a ▁na ▁rukou ▁oko us ané ▁ne h ty . ▁xxmaj ▁to ▁co ▁jsem ▁ten krát ▁v ▁k ině ▁zažil , ▁byl ▁oprav dob ý ▁filmový ▁zážitek , ▁jaký ▁se ▁u ▁mě ▁zatím ▁ne opa koval . ▁xxmaj ▁celou ▁dobu ▁jsem ▁hl tal ▁ty ▁nádherné ▁záběry , ▁s trh ující ▁bitevní</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>. ▁xxbos ▁ špi čkový ▁komerční ▁ válečný ▁seriál , ▁který ▁mě ▁do ▁druhé ▁světové ▁války ▁v táhl ▁snad ▁nejlépe ▁ze ▁všech ▁filmů ▁i ▁seriál ů ▁vůbec . ▁xxmaj ▁osudy ▁dotyčný ch ▁vojáků ▁se ▁nevy poč it atel ně ▁kříží , ▁až ▁to ▁ve ▁ m ne ▁mnohdy ▁vyvolá valo ▁deprese ▁a ▁pocity ▁úzkost i . ▁xxmaj ▁všechny ▁příběhy , ▁zřejmě ▁měly ▁vz bud it ▁v ▁di vá cích ▁odraz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>a ▁xxmaj ▁me lich erová ), ▁které ▁už ▁propadl ▁ne jeden ▁muž ▁( mus ím ▁při zna t , ▁že ▁včetně ▁mě ), ▁ale ▁žádný ▁jí ▁zatím ▁nebyl ▁dost ▁dobrý . ▁xxmaj ▁ja šek ▁se ▁jí ▁ líb í , ▁ovšem ▁protože ▁je ▁příliš ▁mě k ký , ▁tak ▁nakonec ▁podléhá ▁pořád nému ▁chlap ovi , ▁který ▁se ▁nez d ráh á ▁ji ▁u ho dit , ▁muži ▁činu ▁a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>. . ▁xxbos ▁xxmaj ▁za ▁socialistické ho ▁xxmaj ▁československa ▁u ▁nás ▁běžel ▁tento ▁film ▁s ▁názvem ▁\" smrt ▁xxmaj ▁jo a ▁xxmaj ▁indián a \" ▁a ▁jako ▁deseti leté mu ▁kluk ovi ▁se ▁ mi ▁to ▁moc ▁líbil o . ▁xxmaj ▁je ▁to ▁vlastně ▁3. ▁díl ▁čtyř dílné ▁televizní ▁série ▁http : ▁/ ▁/ ▁www . ama zon . de ▁/ ▁ saw y ers - hu c kle berry</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, pretrained_fnames=lm_fns, drop_mult=1.0, wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.585116</td>\n",
       "      <td>4.181531</td>\n",
       "      <td>0.291311</td>\n",
       "      <td>05:05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(1, lr*10, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.390386</td>\n",
       "      <td>4.141454</td>\n",
       "      <td>0.297032</td>\n",
       "      <td>06:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.292165</td>\n",
       "      <td>4.066234</td>\n",
       "      <td>0.307350</td>\n",
       "      <td>06:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.131751</td>\n",
       "      <td>3.927355</td>\n",
       "      <td>0.321946</td>\n",
       "      <td>06:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.919436</td>\n",
       "      <td>3.777414</td>\n",
       "      <td>0.338433</td>\n",
       "      <td>06:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.735788</td>\n",
       "      <td>3.732159</td>\n",
       "      <td>0.343680</td>\n",
       "      <td>06:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(5, slice(lr/10,lr*10), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}_csfd_2classes_fine_tuned')\n",
    "learn_lm.save_encoder(f'{lang}_csfd_2classes_fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = (TextList.from_df(df, path_clas, cols='text', processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols='pos')\n",
    "    .databunch(bs=bs // 2, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, pretrained=False, wd=0.1).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}_csfd_2classes_fine_tuned_enc')\n",
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.307579</td>\n",
       "      <td>0.258988</td>\n",
       "      <td>0.892262</td>\n",
       "      <td>01:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.269991</td>\n",
       "      <td>0.248872</td>\n",
       "      <td>0.898532</td>\n",
       "      <td>01:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.274198</td>\n",
       "      <td>0.216400</td>\n",
       "      <td>0.914701</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(3, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.247607</td>\n",
       "      <td>0.206075</td>\n",
       "      <td>0.921135</td>\n",
       "      <td>02:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.215242</td>\n",
       "      <td>0.177407</td>\n",
       "      <td>0.928725</td>\n",
       "      <td>01:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.170189</td>\n",
       "      <td>0.169069</td>\n",
       "      <td>0.933674</td>\n",
       "      <td>01:56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(3, slice(lr/2/(2.6**4),lr/2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.220555</td>\n",
       "      <td>0.194989</td>\n",
       "      <td>0.919320</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.179532</td>\n",
       "      <td>0.158899</td>\n",
       "      <td>0.936809</td>\n",
       "      <td>03:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.107521</td>\n",
       "      <td>0.148248</td>\n",
       "      <td>0.944564</td>\n",
       "      <td>04:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(3, slice(lr/4/(2.6**4),lr/4), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}_csfd_2classes');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#learn_c.unfreeze()\n",
    "#learn_c.fit_one_cycle(5, slice(lr/10/(2.6**4),lr/20), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#learn_c.save(f'{lang}_csfd_2classes_overfitted')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}