tutorials/flowcap_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Cross Validation of GateNet\n",
    "In this example, we will cross-validate GateNet using the publicly available ND dataset consisting of 30 expert gated samples which were published during the [FlowCAP I challenge](https://www.nature.com/articles/nmeth.2365).\n",
    "\n",
    "To rerun this notebook, the [FlowCAP data](http://flowcap.flowsite.org/codeanddata/FlowCAP-I.zip) has to be downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import sys; sys.path.append('..')\n",
    "import numpy as np\n",
    "\n",
    "from src.data.files import get_files_df\n",
    "from src.data.utils import unique_labels\n",
    "from src.train import HPARAMS\n",
    "from src.cv import cv\n",
    "from src.utils import set_seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "For deterministic results we can fix all seeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "set_seed(seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To rerun this notebook, you have to unzip the downloaded [FlowCAP data](http://flowcap.flowsite.org/codeanddata/FlowCAP-I.zip). Then you have to set 'flowcap_dir' to the path of your unzipped directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "flowcap_dir = '/mnt/data-ssd/cyto/flowcap1'\n",
    "fcs_dir = f'{flowcap_dir}/Data/FCM/fcs/NDD/FCS'\n",
    "label_dir = f'{flowcap_dir}/Data/Labels/NDD'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The FCS filepaths (.fcs, .lmd or .LMD) and respective gating labels (.csv or .ftr) are loaded into a pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                        fcs  \\\n",
      "0  /mnt/data-ssd/cyto/flowcap1/Data/FCM/fcs/NDD/FCS/001.fcs   \n",
      "1  /mnt/data-ssd/cyto/flowcap1/Data/FCM/fcs/NDD/FCS/002.fcs   \n",
      "2  /mnt/data-ssd/cyto/flowcap1/Data/FCM/fcs/NDD/FCS/003.fcs   \n",
      "3  /mnt/data-ssd/cyto/flowcap1/Data/FCM/fcs/NDD/FCS/004.fcs   \n",
      "4  /mnt/data-ssd/cyto/flowcap1/Data/FCM/fcs/NDD/FCS/005.fcs   \n",
      "\n",
      "                                                labels  \n",
      "0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/001.csv  \n",
      "1  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/002.csv  \n",
      "2  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv  \n",
      "3  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/004.csv  \n",
      "4  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/005.csv  \n"
     ]
    }
   ],
   "source": [
    "df = get_files_df(fcs_dir, label_dir)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Before the data can be loaded, we first have to specify a complete vocabulary of labels (i.e. cell types) which occur in the dataset (should print ['0', '1', '2', '3', '4', '5', '6', '7'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['0', '1', '2', '3', '4', '5', '6', '7']\n"
     ]
    }
   ],
   "source": [
    "vocab = unique_labels(label_dir, label_col := 0)\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ds_kwargs = {'vocab': vocab, 'multiclass_col': label_col}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "On top of that, the marker intensities have to be rescaled and normalised such that they resemble a normal distribution (mean 0 and standard deviation 1) as closely as possible.\n",
    "\n",
    "Color marker intensities are typically normally distributed while forward and side scatter intensities often are distributed exponentially. Therefore, the scatter intensities (here 'FSC-A' and 'SSC-A') have to be rescaled using $log(x + 1)$ to yield normal distributions. Alongside the 'forward' transformation (here $log(x + 1)$) the respective, inverse 'backward' transformation (here $e^x - 1$) has to be specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fcs_scaling = {'forward': {'FSC-A': lambda x: np.log(x + 1), 'SSC-A': lambda x: np.log(x + 1)},\n",
    "               'backward': {'FSC-A': lambda x: np.exp(x) - 1, 'SSC-A': lambda x: np.exp(x) - 1}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The distributions have to be normalized such that the mean is 0 and a standard deviation is 1. This is simply done by subtracting the current mean and dividing by the current standard deviation. For the ND dataset, (mean, standard deviation) of the rescaled markers are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fcs_norm_stats = {'FSC-A': (12., .43), 'SSC-A': (11., .79), 'FITC-A': (1.8, 0.44), 'PerCP-Cy5-5-A': (0.96, 0.34),\n",
    "                  'Pacific Blue-A': (1.6, 0.36), 'Pacifc Orange-A': (1.3, 0.25), 'QDot 605-A': (0.72, 0.34),\n",
    "                  'APC-A': (1.1, 0.41), 'Alexa 700-A': (2.4, 0.5), 'PE-A': (1.4, 0.26),\n",
    "                  'PE-Cy5-A': (2.4, 0.36), 'PE-Cy7-A': (2.2, 0.47)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dl_kwargs = {'fcs_norm_stats': fcs_norm_stats, 'fcs_scaling': fcs_scaling}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally, the hyperparameters for the GateNet training are set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'iters': 5000, 'iters_min': 50, 'epochs_max': 10, 'lr': 0.002, 'bs': 1024, 'gamma': 5, 'beta': 0.99, 'n_filter': (1024, 512, 256), 'n_filter_context': (64, 48), 'n_head_nodes': (32,), 'balance_ids': False, 'oversample_beta': 0.999, 'n_context_events': 1000}\n"
     ]
    }
   ],
   "source": [
    "hparams = HPARAMS\n",
    "print(hparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "For a quick test run (with lower performance), the number of iterations can be reduced. By default it is set to 5000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# hparams.update({'iters': 100})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Five-fold cross-validation is done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>balanced_accuracy_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.331502</td>\n",
       "      <td>0.349790</td>\n",
       "      <td>0.920772</td>\n",
       "      <td>0.839955</td>\n",
       "      <td>0.921666</td>\n",
       "      <td>0.767254</td>\n",
       "      <td>00:54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.287854</td>\n",
       "      <td>0.315396</td>\n",
       "      <td>0.934621</td>\n",
       "      <td>0.835345</td>\n",
       "      <td>0.934053</td>\n",
       "      <td>0.799593</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.276954</td>\n",
       "      <td>0.308518</td>\n",
       "      <td>0.940455</td>\n",
       "      <td>0.816973</td>\n",
       "      <td>0.940226</td>\n",
       "      <td>0.821060</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.309970</td>\n",
       "      <td>0.939438</td>\n",
       "      <td>0.791703</td>\n",
       "      <td>0.939121</td>\n",
       "      <td>0.813853</td>\n",
       "      <td>00:34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>balanced_accuracy_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.328945</td>\n",
       "      <td>0.374404</td>\n",
       "      <td>0.905191</td>\n",
       "      <td>0.650804</td>\n",
       "      <td>0.903985</td>\n",
       "      <td>0.698999</td>\n",
       "      <td>00:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.288371</td>\n",
       "      <td>0.320582</td>\n",
       "      <td>0.928531</td>\n",
       "      <td>0.840802</td>\n",
       "      <td>0.928217</td>\n",
       "      <td>0.777078</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.277280</td>\n",
       "      <td>0.309787</td>\n",
       "      <td>0.939966</td>\n",
       "      <td>0.782693</td>\n",
       "      <td>0.939864</td>\n",
       "      <td>0.803522</td>\n",
       "      <td>00:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.307630</td>\n",
       "      <td>0.941781</td>\n",
       "      <td>0.795386</td>\n",
       "      <td>0.941699</td>\n",
       "      <td>0.805400</td>\n",
       "      <td>00:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>balanced_accuracy_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.335025</td>\n",
       "      <td>0.378757</td>\n",
       "      <td>0.895719</td>\n",
       "      <td>0.683681</td>\n",
       "      <td>0.890608</td>\n",
       "      <td>0.706837</td>\n",
       "      <td>00:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.288947</td>\n",
       "      <td>0.334425</td>\n",
       "      <td>0.915612</td>\n",
       "      <td>0.820928</td>\n",
       "      <td>0.915470</td>\n",
       "      <td>0.769504</td>\n",
       "      <td>00:54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.277297</td>\n",
       "      <td>0.332003</td>\n",
       "      <td>0.918387</td>\n",
       "      <td>0.766567</td>\n",
       "      <td>0.917122</td>\n",
       "      <td>0.762992</td>\n",
       "      <td>00:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.333893</td>\n",
       "      <td>0.916977</td>\n",
       "      <td>0.801941</td>\n",
       "      <td>0.916964</td>\n",
       "      <td>0.762374</td>\n",
       "      <td>00:37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>balanced_accuracy_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.332336</td>\n",
       "      <td>0.355914</td>\n",
       "      <td>0.912118</td>\n",
       "      <td>0.737885</td>\n",
       "      <td>0.910918</td>\n",
       "      <td>0.719314</td>\n",
       "      <td>00:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.287965</td>\n",
       "      <td>0.324425</td>\n",
       "      <td>0.924736</td>\n",
       "      <td>0.815850</td>\n",
       "      <td>0.924411</td>\n",
       "      <td>0.800123</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.276375</td>\n",
       "      <td>0.309509</td>\n",
       "      <td>0.939645</td>\n",
       "      <td>0.801907</td>\n",
       "      <td>0.939173</td>\n",
       "      <td>0.825537</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.308429</td>\n",
       "      <td>0.941033</td>\n",
       "      <td>0.804786</td>\n",
       "      <td>0.940343</td>\n",
       "      <td>0.830924</td>\n",
       "      <td>00:35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>balanced_accuracy_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.329188</td>\n",
       "      <td>0.367625</td>\n",
       "      <td>0.909043</td>\n",
       "      <td>0.760958</td>\n",
       "      <td>0.909384</td>\n",
       "      <td>0.721276</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.285419</td>\n",
       "      <td>0.331609</td>\n",
       "      <td>0.918785</td>\n",
       "      <td>0.823468</td>\n",
       "      <td>0.918248</td>\n",
       "      <td>0.764566</td>\n",
       "      <td>00:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.276099</td>\n",
       "      <td>0.307568</td>\n",
       "      <td>0.941815</td>\n",
       "      <td>0.830266</td>\n",
       "      <td>0.941609</td>\n",
       "      <td>0.820574</td>\n",
       "      <td>00:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.307045</td>\n",
       "      <td>0.942259</td>\n",
       "      <td>0.832155</td>\n",
       "      <td>0.942094</td>\n",
       "      <td>0.821786</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       FSC-A      SSC-A    FITC-A  PerCP-Cy5-5-A  Pacific Blue-A  \\\n",
      "0  11.540473  10.790753  2.157689       0.579083        1.930591   \n",
      "1  10.871103   9.989284  1.962930       0.401545        1.797101   \n",
      "2  11.093181  10.688510  2.064609       0.281483        1.785131   \n",
      "3  11.567129  12.020021  2.266574       1.119162        1.928758   \n",
      "4  10.843714  10.350043  2.076122       0.328035        1.740867   \n",
      "\n",
      "   Pacifc Orange-A  QDot 605-A     APC-A  Alexa 700-A      PE-A  ...  0  1  2  \\\n",
      "0         1.253792    0.842704  0.887293     1.839019  1.305078  ...  1  0  0   \n",
      "1         1.227518    0.871632  1.043129     1.782810  1.151766  ...  1  0  0   \n",
      "2         1.213261    0.635844  0.288231     1.856309  1.181292  ...  1  0  0   \n",
      "3         1.275034    0.840032  1.035842     2.085839  1.334119  ...  0  0  1   \n",
      "4         1.627157    0.956006  0.725581     1.928788  1.099105  ...  1  0  0   \n",
      "\n",
      "   3  4  5  6  7                                                   id  fold  \n",
      "0  0  0  0  0  0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "1  0  0  0  0  0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "2  0  0  0  0  0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "3  0  0  0  0  0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "4  0  0  0  0  0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "\n",
      "[5 rows x 30 columns]\n"
     ]
    }
   ],
   "source": [
    "result = cv(df, hparams, ds_kwargs, dl_kwargs, folds=5, with_targs=True)\n",
    "print(result.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The 'result' of the cross validation is a DataFrame with marker intensities + true labels + predicted labels + id (filename) + fold across all validation folds. In this case, this are too many columns to display. So we split it up into the marker intensities..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       FSC-A      SSC-A    FITC-A  PerCP-Cy5-5-A  Pacific Blue-A  \\\n",
      "0  11.540473  10.790753  2.157689       0.579083        1.930591   \n",
      "1  10.871103   9.989284  1.962930       0.401545        1.797101   \n",
      "2  11.093181  10.688510  2.064609       0.281483        1.785131   \n",
      "3  11.567129  12.020021  2.266574       1.119162        1.928758   \n",
      "4  10.843714  10.350043  2.076122       0.328035        1.740867   \n",
      "\n",
      "   Pacifc Orange-A  QDot 605-A     APC-A  Alexa 700-A      PE-A  PE-Cy5-A  \\\n",
      "0         1.253792    0.842704  0.887293     1.839019  1.305078  2.300368   \n",
      "1         1.227518    0.871632  1.043129     1.782810  1.151766  2.834283   \n",
      "2         1.213261    0.635844  0.288231     1.856309  1.181292  3.147278   \n",
      "3         1.275034    0.840032  1.035842     2.085839  1.334119  2.058220   \n",
      "4         1.627157    0.956006  0.725581     1.928788  1.099105  2.853909   \n",
      "\n",
      "   PE-Cy7-A  \n",
      "0  1.809408  \n",
      "1  2.883132  \n",
      "2  2.747585  \n",
      "3  1.577168  \n",
      "4  3.004694  \n"
     ]
    }
   ],
   "source": [
    "print(result.iloc[:, :12].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "...and the predicted labels, true labels, id (filename) and fold across all validation folds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     0_pred    1_pred    2_pred    3_pred    4_pred    5_pred    6_pred  \\\n",
      "0  0.000527  0.000037  0.000563  0.994218  0.000398  0.000058  0.000234   \n",
      "1  0.000428  0.000088  0.000003  0.000118  0.000062  0.000055  0.997056   \n",
      "2  0.999785  0.000050  0.000088  0.000004  0.000058  0.000007  0.000001   \n",
      "3  0.040541  0.000211  0.955730  0.000080  0.000080  0.000241  0.001102   \n",
      "4  0.000064  0.000260  0.000002  0.000505  0.000069  0.000253  0.994033   \n",
      "\n",
      "     7_pred  0  1  2  3  4  5  6  7  \\\n",
      "0  0.003965  1  0  0  0  0  0  0  0   \n",
      "1  0.002190  1  0  0  0  0  0  0  0   \n",
      "2  0.000008  1  0  0  0  0  0  0  0   \n",
      "3  0.002014  0  0  1  0  0  0  0  0   \n",
      "4  0.004815  1  0  0  0  0  0  0  0   \n",
      "\n",
      "                                                    id  fold  \n",
      "0  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "1  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "2  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "3  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n",
      "4  /mnt/data-ssd/cyto/flowcap1/Data/Labels/NDD/003.csv     0  \n"
     ]
    }
   ],
   "source": [
    "print(result.iloc[:, 12:].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Visualize results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAERCAYAAAC9/4bfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABjfklEQVR4nO2dd1wUx9vAv8PRLahBRYpdjBpj771jwRZrouk/E3ti7CYaE02MRqNJjL1EYzexI2IviTQBC4iKFRDEbsRCuXn/uOMEpRzKwZF3vn72Izc7M8+zu7PPPvvsFCGlRKFQKBTmjUVeK6BQKBSKrFHGWqFQKPIBylgrFApFPkAZa4VCocgHKGOtUCgU+QBlrBUKhSIfoIy1QqH4f4MQYrkQIk4IccbI/H2EEGFCiFAhxFpT65epLqqftUKh+P+CEKI58BBYJaV8I4u8lYCNQGsp5V0hRAkpZVxu6JkeyrNWKBT/b5BSHgHupE4TQlQQQngLIU4IIY4KIV7X7/ofMF9KeVdfNs8MNShjrVAoFIuB4VLKOsBo4Dd9ujvgLoT4WwjhK4TwyDMNAcu8FK5QKBR5iRCiINAY2CSESEm20f9vCVQCWgKuwFEhxBtSynu5rKZBGYVCofj/igVwT0pZM519UYCvlDIRuCyEOIfOeAfkon4GVBhEoVD8v0VK+QCdIe4NIHTU0O/eCrTSpzuiC4tcygs9QRlrhULx/wghxDrgOFBZCBElhPgIeAf4SAhxEggFuumz7wFuCyHCgIPAGCnl7bzQG1TXPYVCocgXKM9aoVAo8gFm+4FxpcsAs3D5/3fzYF6rgKXGPC6TVmrzWgWStXmvg7lg8az3Qp6iNYO386SE6Fc+GYm3Lhl9IFaO5XP95JuHFVAoFIq8Rpuc1xpkijLWCoVCAWAGb46ZoYy1QqFQAJh5iE0Za4VCoQCk8qwVCoUiH5CclNcaZIoy1gqFQgHqA6NCoVDkC1QYRKFQKPIBZv6BUY1gVCgUCnQfGI3dsiKr5cP0E0b9LISIEEKcEkLUzqpOszXWwkLguWcabX7/AoC6X/anx+GZdN37Ha2WfoZ1YXtD3urDPOl5bDY9jszCuUV1Q7qFlYZGP3xIj6Oz6HF4JmU61QOgZIPKeHpP492rv1Omc700ctv9MZa3wxYZ5KbHyBH/IyTkAMHB+1m9ej42NjYULVqE3V7rCAs9xm6vdRQp4vBMv+pVOHpku65M0D5sbHTT5e7bu4kzZ44QGOBDYIAPxYu/luk5cXUthbf3eoKD93PixF6GDv0AgJ49O3HixF7i4y9Tu/az4y9WrAje3uu5eTOMn376Jk1dvXp1wd/fmxMn9jJ9+gRD+oABvbh2LQhfXy98fb14//1+L+ixaNGPRF4LJujEPkPalCmjCQzwwd/Pm10711CqVEkA2rRpxvF/dnEicC/H/9lFy5aNAbCzs2XrlpWcOnmQ4KB9TPt2vKGupk0b4Hvci/iHl+nRo1Om5yQ9Ro74HydDDhASvJ8/9Ndn7ZoFhvMccd6XwAAfAOrVrWlIPxG4l27dcmZ++SWLZ3M96iQhwftf2Dfq809ISojmtdeKAtC2TTP8fHcTHLQPP9/dtGrZJFuyFi/6kajIEIKDnl2PokWL4OW1ltDQo3h5rU3THgHc3Jy5c/scn3/+iSGtVq3qBJ3YR1jYMebM+SZNXp89G/H38+ZE4F48PFpnSz939wqGcxwY4MOdW+GMGP4xb75ZlWNHthMctI+tW1ZSqFBBAPr375Emf8KTSGrUqJYtmS+FVmv8ljUrgcwaU0d0061WAgYBC7Kq0GwncvKfukY6vlkOq0J27H9vNs7N3yDm7zBkspY6E/sCcOK7DThUcqbFb0PZ2XkK9iWL0n79eLY0G43USmp+0ROhsSB45mYQApsiBXh69yEFXR2xKmRHtU87EekTxNVdz6anLdW0Gho7ayoPaM3+92a/MNzc2dmJQwe38GaNVjx58oS1axfivfsAVapW4s6de8yaNZ8xY4ZStKgDEyd+h0ajIcDfm/c/GMmpU2EUK1aUe/fuo9Vq2bd3E+PGfcuJoFOZnouU4eZOTiVwcipBSMgZChYswD//7KRPn0FIKdFqtfz663dMmDCdoKDTANjb21GzZjWqVq1MtWqV+fzzyYDOiPv6etG4cRdu3brDkiWzWbPmLw4d+psBA3pRp86bhrypSRlu3rRpAx4+jGf5srnUrtMWgEKFCvLvvw8BGDrkA6pUqcSw4ROpUaMacXG3iIm5QdWqldm54w/KV6iHnZ0t9evX4vDh41hZWeHtvZ6ZP/zCHp9DlCnjSqFCBfn880/YuXMvW7Z4GXTIari5s7MThw9uobr++qxbu5Dduw+wavVGQ55ZP0zm/oMHTJs+Fzs7WxISEklOTsbJqQRBgXtxK1Ob5ORX+9jUTH+OVqyYR81abQzprq7OLF44i8qVK1K/oQe3b9+lZs1q3LihO0fVqlXGa+caypSrm6WMlOHmKddjxfK51Kqtux7ffzdJ1x5/nM+Y0fr2OOk7Q9kN6xej1WrxDwjmp58WAfD3sZ2M+mIyfn5BbN++mvnzl7Nnz0F+++0HQkLOsHjxaqq8Xolt21bhXrmRoa7sDDe3sLDg2pUTNG7ahQ3rFzNu3LccOerL++/1pVy50kz5elaa/G+88Tp/bV6O++uNM603J4abPw0/bPSB2LzeIkt5QoiywM701noUQiwCDkkp1+l/nwNaSiljMqrPXD1rV9c2NTm/7pAh4fqRM8hk3Y16M+gi9qWKAVC6Qx0ub/NFm5DEw8ib/HvlBo61KgBQqV8LTv+yQ1eBlDy9qzMmD6NucfdsJGhfvDYxx0JJevgkU+UsLS2xs7NFo9Fgb2fH9ZhYPD07sHr1JgBWr95E1666h2q7di04ffosp06FAXDnzl20Lxkbi42NIyRE91b18GE84eERODuX5Ny5CC5ceHGa3UePHvPPP4E8efI0TXq5cqW5cOEyt27plqI7cOAY3bt3NFqPY8f8uHv3Xpq0FEMNYF/AnpT79+TJUGJibgAQFnYOW1sbrK2tefz4CYcPHwcgMTGRkODTuLiWAuDq1SjOnAlHm871MYbnr09MTGya/b16ebJ+wzYAHj9+YjDMtrY25JTzcvSYH3eeO0cAs3/8mvETp6eRExLy7ByFhp7D1tYWa2tro2Wldz08Pduz+g99e/xjE127djDs69q1A5cuXyMs7LwhzcmpBIULF8TPLwiANX9sNpSRUlK4UCEACjsUMuj6MrRp3ZRLl65y7Vo0ld0rcOSoLwD79h9N9y2qX9/ubNi47aXlZQupNXoTQgwSQgSm2gZlU5oLEJnqd5Q+LUNMZqyFEK8LIcbp4zLz9H9XMbL43BPT1qVrTAEq9WtO9EGdN2rvVJT468/Wv4yPuYO9U1FDmKTW2F54ek+j5aLh2DoWfqVjArh+PZafflrIpYv+RF4L5sGDB+zbd4SSJRyJjdWtpxkbG0cJfUjDvVJ5pIRdO9fg7+fNF18MTlPf0qVzCAzwYeLEz7KlR+nSrtSsWY2AgJBsH8PFi1eoXLkCpUu7otFo6Nq1A656QwnQrVtH/P29Wbt2QZr0rJg6dSwREX7079eDqd/8+ML+Hj06cfLkGRISEtKkOzgUpnPnthw8+He2j+V5rl+PZc5PC7l80Z+oa8Hcf/CAvfuOGPY3a9qAG3E3iYi4bEirX6+WLmwStJ8hw8a/sledEV26tCM6Osbw4E6Pnj07ExLy4jnKLiWea48pITZ7eztGfzGEadPmpMnv7OxEVPQzpy4qOgZnZycAvv12Dm+/3ZNLFwPYvm0Vn33+1Uvr1adPN9Zv2AroHkyenu0B6PVWF9xcnV/I37uXpyG/yclGGERKuVhKWTfVtjib0tLzzDP1FExirIUQ44D1eoX80S2DI4B1QojxmZQb1L9//wt//PFH8z9996ab580RXdEmabn0198phV7MJEFoLCjg/BpxAefZ4fElcSciqDf57Vc8MihSxAFPzw5Ucm9I6TK1sS9gz9tv98wwv8ZSQ+PG9Xj3vWG0aNmd7t060qpVUwDefW84tWq3pWWrHjRtUp8BA3oZpUOBAvasW7eQMWO+SePRGsu9ew8YMWISf/zxK/v3b+bq1SiS9QMCvLz28frrTahf34MDB46xZMmcLGp7xpQpM6lYsQHr1m9h8OD30+yrUsWd76ZPZOiwCWnSNRoNq1f9yvz5K7h8+Vq2j+V5ihRxoKtnByq6N8StTG0KPHd9+vbtzoYNaT01/4BgatRsTcPGnRg/dpjhm0JOYmdny8TxI/h66osPsRSqVnXn++kTGTx0XI7LT2Hy5C/4+eclxMc/SpMu0r2PdLajb99urFq9kfIV6tG127usXDEv/fxZYGVlhWeX9mz+cycAHw8axZBP38fPdzeFChUgISExTf769Wrx6PFjQkPPZVvWS5ENzzoHiALcUv12Ba5nVsBUXfc+Aqrp1y4zIISYg24lhhnpFdI/ncoBA7u36IDGxgqrQnY0+3kwR0csoELvZri2rcWePt8byjyKuUMB52KG3wVKFePRjbs8vfuQxEdPuLo7EIArO/2o1K/FKx9YmzbNuHLlmiGEsHXrbho1rMuNuFs4OZUgNjYOJ6cSxN3ULSgRHR3D0aO+3L59F4Dd3geoVesNDh48xvXrutfzhw/jWb9+K/Xq1uSPPzZnKt/S0pJ16xayYcNWtm3zfunj8PLaj5eX7uPXhx/2N3iTd+7cM+RZvnwd06Zl+GzNkA0btrJ1y+98+63O0Lu4OLFp4xI+/OgzLl26mibvb7/9QETEZX75ddlLHkla2rRpxuVU12eL/vqsXfsXGo2GHt07Ur9h+iGf8PAI4uMf80a1yll+R8guFSqUpWzZ0gQF6pwQV9dSBPjtoVGTzty4cRMXl1Js3rSMDz4c+cI5ehninmuPN/XtsX69WvTs0ZnvvptEkSKF0WolT548ZcsWL1xdnr1FubqU4ro+3PHB+/3o4jkAAD+/IGxsbXB0LGao01g8PFoRHHyauLhbAJw7d5GOnXUOVKVK5enUsU2a/H37dHvhwWpScrfr3nZgmBBiPdAAuJ9ZvBpMFwbRAi++00Ap/b7MmAC4bm74OYeHzCfm7zCOjliAS8s3qT6kC/vfn0Pyk2eviJE+QZTr1hALa0sKuhWncDknbgVfBCBqbzBOjXWRF+em1bh/IfqVDyzyWjT1G9TGzs4WgNatmhIefoGdO3wYOLA3AAMH9mbHjj0A+Pgcpnr1KoYYavNmDTl79gIajcbQG8DS0pJOndsa5UEsXDiTc+ci+Pnnpa90HCmvxUWKFGbQoIGsWLEe0MUuU+jSpR3nzkUYVV/FCmWflev8rJyDQ2G2bvmdL7+awfHjgWnKfP31GBwKF+KL0V+/wpGkJfJaNA3SuT6g63Vx7lwE0ale98uWdUOj0QBQurQL7u7luXI18sWKX5EzZ8Jxdq1BRfeGVHRvSFRUDPUadODGjZs4OBRm+7ZVTPrye/557hy9LDt27mXgAH17HNCbHTt0vV9at3kL98qNcK/ciF9+WcYPM39hwYKVxMbG8e+/D6lfX9eD7J0BvQxlrkVeN7wNvv56RWxtbLJtqEEXf04d0khpg0IIJk4YyaLFqw37hBC89VaX3ItXA1KbaPSWFektHyaE+FQI8ak+ixe69RwjgCXAkCzrNEVvECGEB/ArcIFnQfTSQEVgmJQyS5dwpcsA6dSoCtU+7cT+92bT89hsNDaWho+EN4MiOD5+BaALjVTs2wKZrMV/ympDPLuAy2s0+3kw1oXteXLnX/7+fDHx12/zWo3ytF72GdYO9iQ/TeRx3H22tdZ5kB3/+gqHiqWwtLfl6b2H9P54GHv3Hk6j2+TJX9C7d1eSkpI4GRLKoE9GU7BgAdatXYibmwuRkdH06/+J4aPP22/3ZOzYYUgp8fY+wIQJ07G3t+Pggb+wsrLEQqPhwP6jjB4zNd2Pjym9QRo3rsv+/X9y+vRZQ74pU2ZhY2PNnDlTcXQsxr17Dzh1KoyuXd8FIDz8GIUKFcLa2or79x/QpctAwsMv8PvvP1O9elUAvv9+Hps26T7EfvPNWDp3bkdSUhJ3795nxIhJnD+ve/il9AZZtepXmjdriKNjMW7cuMW302bj0aE17u4V0Gq1XLsWxbDhE7l+PZbx40cwdszQNDHizl3ewdraiksXAwgPv8DTp7qH74KFK1mxYj116tRg44YlFC3qwJMnT7lxI87Qy8GYxQempLo+Ifrrk5CQwLKlP+HnF8TiJc+MwjvvvMXYMUNJTExCq9UybfpPbN++J0sZWfHH6vm0aN7IcI6mfvMjK1auN+yPOO9Lg0YduX37LhMnjGTc2GFcSHWOOnbqn6VBTOkNsnrVrzRPJeubb2ezfbs3a1O1x/79P33hI+RXX47iYXy8oTdI7dpvsmzpHGztbNmz5xCfffYlAFVer8SCBTMpWLAAUkomTJzOvlTfAYzpDWJnZ8uVS4FUqtyIBw/+BWD4sI8M4bKtW72YOOnZG3OL5o34bvpEmjTzzLJuyJneIE+CthttDG1rd831xQdM1nVPCGEB1Ef3hVOgi9EESCmN+nqjVop5hlop5hlqpZhnqJVinpEjxvrEVuONdZ3u/52VYqRumI+vqepXKBSKHEVN5KRQKBT5ADN4c8wMZawVCoUCzH4iJ2WsFQqFAtTiAwqFQpEvUJ61QqFQmD9GdlTLM5SxVigUClCetUKhUOQLVG8QhUKhyAcoz/rlGHTrUF6rAICNpfHzCpuKJ0mvNl2m4r+JOYwc/E+heoMoFApFPkCFQRQKhSIfoMIgCoVCkQ9QxlqhUCjyASoMolAoFPkA9YFRoVAo8gEqDKJQKBT5ABUGUSgUinyAmXvWplow12SMGPExIcH7CQ7ax+pVv2JjY8NXX47i8qVAAvz3EOC/Bw+P1mnKuLk5c+f2OT7//BNDmpWVFb/99gOhZ45w+tQhenTvlKFMGxtrDh3ZynFfLwIC9zDpy88AKFrUge07VhNy6gDbd6ymSJHCALRq3ZSjf2/Hz383R//eTosWjQx19e7tiZ//bnz9drNl20rDorkzfviSf3x38Y/vLoJPHiDq+kmjz4m7ewUCA3wM251b4YwY/jE1alTj76M7CAzwwfe4F/Xq1jSUGTd2GOFhxwg9c4T27V591ffMsLGx4fjfOzkRuJeTIQeYMvkLk8rLiCWLZ3M96iQhwfvzXG7RokXw9lrH2dBjeHuto0gRhxyX6+rqzD6fTZw+dYiTIQcYPuwjACZ/NYqrlwMN7aVjqvvF1O0ivXPx5ptVOXZkO8FB+9i6ZSWFChXMcblGodUav+UBJluD8VWxtnF9QTFnZycOHvyLGjVa8+TJE9auWcBu7wOULeOWZuHP59mwfjFarRb/gGBDnslffYFGY8GUr2chhKBYsSLcvn33RT00VgAUKGBPfPwjLC0t2bt/E2NHT6VrNw/u3r3HnNkLGfXFpxQp4sDkr37gzRpViYu7RWxMHFWrurN1+++4V2yERqMh4qIvdeu05/btu3w7bTyPHz/mu+nz0sj89NP3eLNmVYZ8Og7I3ghGCwsLrl05QeOmXVi0YBbzfl6C956DdPRozegvBtOmXW+qVKnEH6t/o1Hjzjg7l2TP7vVUqdYs3cV6c4rU5+/IoS18PmoKfv5BJpOXHs2aNuDhw3hWrJhHzVpt8lTujO8ncefOPWbOms/YMUMpWtSBCRO/y1G5Tk4lKOVUguCQMxQsWAB/P2/e6vUhvXt58vBhPHOeu19yo12kdy6O/7OLceO+5chRX95/ry/lypVmytezslVvTqzB+HjDVKONoV3fKbm+BmO+86wtNZbY2dmi0Wiws7cjJuZGpvm7du3ApcvXCAs7nyb9vff68sPMXwGQUqZrqFMTH/8IACsrS6ysLJFA5y7tWLPmTwDWrPmTLp7tATh1MozYmDgAwsLOY2Njg7W1NUIIhBDY29sDULhwQWL0+VLTq48nmzbuyOJMpE+b1k25dOkq165FI6WkUOFCOlkOhbiuP1ddPTuwceM2EhISuHIlkosXr1C/Xq2Xkmcsqc+fpZUVeeEkHD3mx53nVvjOK7menh1YtXoTAKtWb6JrV48clxsbG0dwyBkAHj6MJzz8Ai7OThnmz412kd65qOxegSNHdcu17tt/lB49Mn7LNSlJScZveUC+MtbXr8fy09xFXIzw49rVIB7c/5d9+44AMPjT9zkRuJfFi340vFLa29sx+oshTJs2J009Dg66cMXXX4/Bz3c369YupEQJx0xlW1hY8I/vLi5fDeTA/mMEBoRQooQjN2JvAnAj9ibFi7/2Qrnu3Tty6mQoCQkJJCUl8dnIr/AL2E3EJT9ef70Sv6/ckCa/m5sLZcu6cfjQPy91jvr06cb6DVsBGDV6Cj98/yWXLwYwc8ZXTPrye0D3hhIZdd1QJio6BmeXjG/inMDCwoLAAB9iok+xf/8R/AOCTSrP3ClZwpHYWN2DOjY2jhLptJ2cpEwZV2rWeAM/f915HzL4A4JO7GXJ4tmG+yUv2gVAaOg5PPWOTq+3uuDm6mxymekitcZveUCuG2shxAeZ7BskhAgUQgRqk+Nf2F+kiAOeXdrjXrkRZcrWoUABO97u35NFi1fxepUm1K3XntjYOGb+8BUAkyd/wc8/LzF4dSlYWmpwc3Pm+D+BNGjYEV+/E/ww46tM9dZqtTRu2JnKlRpRt24NqlZ1z/JYq1SpxDfTxjFi+CS9XEs+/t87NGnUhYrlG3DmTDijxwxJU6ZX7y5s3bL7pV49rays8OzSns1/7gTgk0Hv8sWYrylXoR5fjJnKkkWzARDixTc4U3u6Wq2WuvXaU6ZcXerVrUW1apVNKk/xjAIF7Nm4YQmjRk/h338fsnDRKtxfb0ydurr7ZdbMyUDetAuAjweNYsin7+Pnu5tChQqQkJBocpnpYuYx67zwrKdmtENKuVhKWVdKWddCU+CF/W1aN+XKlUhu3bpDUlISW7fupmGjOsTF3UKr1SKlZNnytdSrVxOA+vVq8d13kzh/7jjDh3/EuLHDGTz4fW7fvkt8/CO2btsNwJ9/7qRWrTeMUv7+/X85etSXtu1aEBd3i5JOxQEo6VScmzdvG/I5uzixdv0iBn38BZcvXwPgzRpVAQy///pzFw0a1k5Tf6/enmzauN0oXZ7Hw6MVwcGniYu7BcC7A3uzZYsXAJs37zCcl+jomDTei6tLKWKuZx5Oyinu33/A4SP/0KF9y1yRZ67ciLuFk1MJQBdbjkvVdnISS0tLNm1Ywrp1W9i6VdfeU98vS5etyfN2ce7cRTp2fpsGDTuyfsM2Ll26YnKZ6SKl8VseYBJjLYQ4lcF2Gij5svVei7xOgwa1sLOzBaBVq6aEh0cYGj1At24ehIaeA6B1m7dwr9wI98qN+OWXZfww8xcWLFgJwK5dew29NFq1asrZsxcylOvoWAwHB13s19bWhlatmnL+/EW8du3jnXfeAuCdd95i1869ADg4FOLPP5fz9eSZ+PqeMNRz/Xosr1ephKNjMb1+TTkXftGwv1Kl8hQp4oCf38t9eOvXt7shBAJwPeYGLZrrjrF1q6ZciLgMwI6dPvTp0w1ra2vKlnWjYsVyJg1L6M6fLvRka2tLm9bNOHfuYhal/tvs3OHDuwN7A7qH6o4de0wiZ8ni2ZwNj2DuvMWGtNT3S/duHQ33S263ixRSwodCCCZOGMmixatNLjNdzNyzNlU/65JAB+D5r3YCeLlgLBAQEMxff3nh7+dNUlISISGhLF26hkULZ1GjRjWklFy9GsmQoeOzrGvipO9YsXwes3+cys1bt/nf/0ZlfDBOJVi85Ec0FhosLAR//bUL790H8PcLYtXqX3n3vT5ERV5n4IChAHzy6XuUr1CGcROGM27CcAC6eb5LbEwc3383jz0+G0hMTOJaZDSfDhptkNO7jyebN73ch0U7O1vatmnO4CHjDGmffjqGOXO+wdLSkqdPnjB48FhA99Fz8+YdnD55kKTkZEaMnGTSniClSpVk+bK5aDQWWFhYsHnzDnZ57TOZvIz4Y/V8WjRvhKNjMa5cCmTqNz+yYuX6PJH7w6z5rF+7kA/e709kZDR9+3+SdUXZpEnjegwc0ItTp8MIDPAB4KuvZtC3b3dq1Kiqv1+iDG0mN9pFeueiYMECDB78PgBbt3qx8vcNmVdiKsy8n7VJuu4JIZYBK6SUx9LZt1ZK+XZWdaTXdS8vSOm6l5eoxQcUiszJia57jxZ/brTNsR/0U6533TOJZy2l/CiTfVkaaoVCoch1zNyzVsPNFQqFAtTcIAqFQpEv0JpF5DVDlLFWKBQKMPswSL4awahQKBQmIznZ+C0LhBAeQohzQogIIcQL3dOEEA5CiB1CiJNCiNDMBgumoDxrhUKhgBzzrIUQGmA+0A6IAgKEENullGGpsg0FwqSUnkKI4sA5IcQaKWWGXb+UZ61QKBSgi1kbu2VOfSBCSnlJb3zXA92eyyOBQkI3xr8gcAfIdIYoZawVCoUCsjWRU+p5jPTboFQ1uQCRqX5H6dNS8ytQBbgOnAZGSpl5dxQVBlEoFArIVm8QKeViYHEGu9MbMPN85R2AEKA1UAHYK4Q4KqV8kJFMszXWFsI8nH5zGD34+PrRvFYBADvnZnmtgkJhMmTO9QaJAtxS/XZF50Gn5gNghtQNIY8QQlwGXgf8M6rUPCyiQqFQ5DU51xskAKgkhCgnhLAG+gHPT6V5DWgDIIQoCVQGLmVWqdl61gqFQpGr5NCgGCllkhBiGLAH0ADLpZShQohP9fsXAt8CK/UzkQpgnJTyVmb1KmOtUCgUkKODYqSUXoDXc2kLU/19HWifnTqVsVYoFApQw80VCoUiX6AmclIoFIp8gPKsFQqFwvyRSVnP+ZGXKGOtUCgUYPaetdn3s160aBbXrgVx4sReQ9rq1fPx89uNn99uzp37Gz+/3WnKuLk5c+vWWT77TDcC1M7Oli1bVnDy5AGCgvbx7bfPJsGaOXOyoa7Tpw8RG3s6W/otWTyb61EnCQne/8K+UZ9/QlJCNK4uhShb2h43FzuDfvfunGfU58/W3XutmDWlXe1xdbHD2vrZZSntak+PgYN5672h9PlwhCE9/PxF3v7fZ4b002Hnnum0agMd+3xIl34f87ffswV75y1aSZseA6nXtke2jjEz7Ow0uLnYU9rVniIOaZdAK1zYCjcXe+rXLc+hg5s5feoQJ0MOMHxYhgsJmZQO7VsSeuYI4WHHGDtmaJ7oYC56mIMO5qQHkK3h5nmB2XvWq1dvYsGC31m27CdD2sCBzy7qjBlf8uDBv2nKzJw5mT17DqVJmzt3MYcPH8fKygpv73W0b98SH59DjB37jSHP4MHvU7NmtWzpt2rVRn77bQUrVsxLk+7q6kzbNs25ejWKhEQL4m7GU7K4DQCzf/wa7z0HDXnt7TRYW1lwLeoRNjYWFH/NhuiYx4b9y3+ZQdEiDmnqn/3bMgZ/+A7NGtXjyD/+zP5tGSt/ncnFy1fZvf8w2/5YSNytO3w8cgK71i9Fo9HQskkD3n6rK536Zd9YTpo2G1tbDU+epH1VLP6aDddjH5OUJHF1tiP+URKJiRJbWw0F7DVERj8iMfkhX3wxlRNBpylYsAD+ft7s238k0xXlcxoLCwt+njcdj079iYqKwfe4Fzt2+uSqDuaihznoYE56GFCe9atx7Jg/d+/ey3B/r15d2LBhm+G3p2d7Ll++xtmz5w1pjx8/4fDh4wAkJiYSHHwGV9dSL9TVp09XNm58fqBR5hw95seddPSb/ePXjJ84HSklyckWaPUNoWvXDly+dI2wVJ6wvb0l/z7UTbj19KkWCwuBRpP5epxCCB7GPwLgYfwjSji+BsCBo750bNMCa2trXJ2dKO3qzGn9uajxRhWKOxZ7oa47d+/x2cRp9P1oBH0/GkHQqVCjjt3GxoLERC1JSVKvRxIF7HXPf4dClty7lwhAbGwcJ4J0bywPH8YTHn4BF2cno2TkFPXr1eLixStcvnyNxMRENm7cRlfPDrmqg7noYQ46mJMeKUitNHrLC0xmrIUQrwsh2gghCj6X7pFTMpo2rc+NG7e4ePEKAPb2dnzxxWCmT5+bYRkHh8J07tyWgwf/TpNeurQLZcuWfiH9ZejSpR3R0TGcOhWWJl1YWDB29FC+mTYnTbqlpSAp6dmrVVKyFstUxnrQ55Po8+FwNm171sd+3MhPmP3bMtr0GMiPvy7ls0/fByDu5m2cShY35CtZwpG4m5kOjGLGvEW827c7G5b9zNzpXzJlxrxM8xv01giSkp813KRkiaWlTm8rKwtsbTW4lLLD2ckOG31op0wZV2rWeAM//2CjZOQUzi5OREY9m54hKjoG51x+YJiLHuaggznpYSAp2fgtDzBJGEQIMQLd5NpngWVCiJFSyhT39zvAO4Nyg4BBAJaWRdFoCqaXzUCfPt3YuPGZV/3VV6P45ZdlxOs9zufRaDSsWvUL8+ev4PLla2n29e7dlS1bdqF9xVFMdna2TBw/Ao9OLy7iXqK4I3N/XpKhfqlJMYHRMY855/8rt+/e43+fTaRcGTfq1qzOhi27GDd8EO1aNcV7/xEmfz+XpfO+R74wuReIdCcBe4ZvQDAXU52P+PhHxMc/IuTMWeb8thyAmBtxlHC0QauVSEmaME16igsBFha6fDbWFpQsYcvtu7BxwxJGjZ7Cv/8+zPIc5CS6aYOfU1XmvodkDnqYgw7mpIcBMw+DmCpm/T+gjpTyoRCiLLBZCFFWSjmP9KcPBNJOO2hrWzrTM6fRaOjWzYPGjTsb0urXr0XPnp347rsJODgURquVPHnylIULfwfgt99mEBFxhV9/XfZCfb17e/LZZ19l/0ifo0KFspQtW5qgQN0HUVfXUgT47aFx047Y2dky47tJzPhuEkWKFEar1fLkyVM2blyHpaUFPNU9KCw1FiTrPdaU/18rWoQ2zRtzOuwcdWtWZ/vufUz47FMAOrRuxpQZcwEoWdyR2Bs3DfrciLtF8eKvZaqzVkrWLJ6DrY1NmvQmDerQpEEdQBez/m35jjQx66RkmeYNILWnnZQkiX+ky/s0QYulpSWbNi5g3botbN2a9oNwbhAdFYObq7Pht6tLKWJibvy/1MMcdDAnPQyYubE2VRhEI6V8CCClvAK0BDoKIeaQibHODq1bN+X8+YtER8ca0tq06UXlyk2oXLkJv/66nJkzfzUY6q+/Hk3hwoUYPfrrF+qqVKk8RYs64Ot74oV92eXMmXCcXWtQ0b0hFd0bEhUVQ70GHbh16w5XrlwzpP/8y1Jm/PALvy1YyaNHSRQqqHtu2thYoJWS5GSJEDoPFeDR4yf84x9EpfJlASju+BoBwbo4sN+JEMq46eY2b9W0Ibv3HyYhIYGo67Fci7pO9SrumercuH5t1m7eYfgdfv6iUcf69KkWKysLQ+ijYAFLg4GOf5SEna0GACtLwQ8/fMfZsxHMnZfRFMCmJSAwhIoVy1G2rBtWVlb06dONHTt9/l/qYQ46mJMeKUgpjd7yAlN51rFCiJpSyhAAvYfdBVgOVM9ORatW/UKzZo1wdCxKRIQf06bNYeXKDfTp05UNG4z7GOji4sT48SMID7+Ar68u7rtw4e+sWLEegL59u7Fx447MqsiQP1bPp0XzRjg6FuPKpUCmfvMjK1auT5PH3i6JgvZ2aDSCMm723LmbwPnzYTx9+hSAR4+TsbfXUNrVHq2U3LypS9doBE4lbOn53hCSk5Lp1L4lTRvWBWDquBHMmLeIpORkbKytmTJW162vYvkydGjdjK7vfIKlRsOkUUPQaHRGc/b8ZXjtPciTJ09p030APT09GPrRACZ89inTZv9Gj3cHk5ycTJ0a1ZkydrhRx3/r9lNKOdkhgAf/JpKYqHs7ePBvEiWK2+DmYkedOnXo0aMHp06HERiguxm/+moGu70PvNQ5fxmSk5MZ+dmXeO1ai8bCgpW/byAs7HzWBf+DepiDDuakhwEz96yFKZ4SQghXIElKGZvOviZSyiy/4mUVBsktkrR5P6pJLT6gUGROUkL0K7+xP/iondE2p/CyvTkSIcgOJvGspZRRmex79e4WCoVCkcPIJDWRk0KhUJg/5m2rlbFWKBQKIM8GuxiLMtYKhUIBZv+BURlrhUKhABUGUSgUivyACoMoFApFPkAmKWOtUCgU5o8KgygUCoX5Y+br5Zqvsdaa+5nLRcxl5GDJAkXyWgVuxN/LaxUU/1XM3OSYrbFWKBSK3MTc/UNlrBUKhQKQSXmtQeYoY61QKBQoz1qhUCjyBcpYKxQKRX5A5vqsp9lCGWuFQqFAedYKhUKRL5Ba5VkrFAqF2aNNNm9jbaoFc3OMxYt+JCoyhOCgfYa0okWL4OW1ltDQo3h5raVIEQcA6tatSYD/HgL89xAY4EO3rh6GMr17eXIicC8hwfv5/rtJhvSmTRvg57ubR/FX6Nnj2UrpOUGH9i0JPXOE8LBjjB0zNEfrzgh39woEBvgYtju3whkx/GMmfzWKq5cDDekdPVq/VP0VKpbF58ifhi38qh8ffzqQLt3ac+CfbUTePs2bNasZ8ru6ORNx/YQh/4w5kw37uvbwYO+xvzjwzzYmTf3CkN6gcR28D23i6s2TdO7a/uVPhh4LCwsC/PewbcvvadJHff4JSQnRvPZa0VeWkR1yq10sWTyb61EnCQne/8K+54+9WLGi7PPZxL0755k3d5rJdLKxseH43zs5EbiXkyEHmDJZd91/+P5Lzpw+TNCJvWzetBQHh8Im0yEjpNb4LSuEEB5CiHNCiAghxPgM8rQUQoQIIUKFEIezqtPsjfWq1Zvo4jkgTdrYMUM5eOBvqlVrxsEDfxsafGhoOA0bdaJe/Q508RzA/Pkz0Gg0FCtWhO+//5IOHn2pWasNJUo40qpVEwAiI6P5+ONRrF+/NUf1trCw4Od50+niOYDqNVrRt293qlSplKMy0uP8+YvUrdeeuvXaU7+BB48ePWbrtt0AzPt5iWHfyy5WezHiCu2bv0X75m/h0bI3jx8/YfeufYSfjeB/747E95/AF8pcvRJpKDN+1DcAFC3qwJffjKZvt49o3bgbxYu/RtPmDQCIjozh86GT2Lp510uehbSMGP4x4eEX0qS5ujrTtk1zrl7NcAU6k5Cb7WLVqo107vLOC+npHfuTJ0+Y8vVMxo771iS6pPD06VPatu9DnbrtqFO3PR3at6RB/drs23+EGjVbU7tOOy5cuMT4ccNMqkd6SK0wessMIYQGmA90BKoC/YUQVZ/LUwT4DegqpawG9M5KP7M31seO+XH37r00aZ6e7Vn9xyYAVv+xia5dOwDw+PETkpN1C9za2toYlowvV64MFyIucevWHQAOHDhGjx6dALh6NYrTZ86i1ebs14X69Wpx8eIVLl++RmJiIhs3bqOrZ4cclZEVbVo35dKlq1y7Fm2S+pu2aMjVK5FER8YQcf4SFyOuGF22dFk3LkVc4c7tuwAcPXycTnovOiryOmdDz6PNgSkrXVxK0aljG5YvX5cmffaPXzN+4nRMsWB0ZuRmuzh6zI87z907kP6xP3r0mL//CeDJk6cm0SU18fGPALCyssTSygopJXv3HTHcu75+Qbi4lDK5Hs8jpfFbFtQHIqSUl6SUCcB6oNtzed4G/pJSXtPJlnFZVWr2xjo9SpRwJDZWd2yxsXEUL/6aYV+9erUICd5P0Il9DBs2geTkZC5evEJl94qUKeOKRqOha9cOuLk6m1RHZxcnIqOuG35HRcfg7OxkUpnP06dPN9Zv2Gr4PWTwBwSd2MuSxbMNoaNXoVvPjmz90yvLfKVLu7Dn8GY271xJ/Ua1Abhy6RoVK5XD1c0ZjUZDh05tcHbJ+fMzZ/ZUxk+YluZh3KVLO6KjYzh1KizH5WVFXreLvDz2FCwsLAgM8CEm+hT79x/BPyA4zf4P3u+H956Dua5XdjxrIcQgIURgqm1QqqpcgMhUv6P0aalxB4oKIQ4JIU4IId7NSj+TGWshRH0hRD3931WFEKOEEJ2yKGM4Adrk+JeSGxAQTM1abWjcpDNjxw7DxsaGe/fuM3zEBNb8sYCDB/7iytVIkpKSX6p+YxHixVel3PTirKys8OzSns1/7gRg4aJVuL/emDp12xMbG8esmZOzqCHr+tt3bMXOrXsyzRd34yb1q7elQ4teTJ00k/lLZlKwUAHu33/AhNHfsmD5bLZ4rSLqWjRJSTk73rdzp7bExd0iKPi0Ic3OzpaJ40fw9dQfc1SWseRlu8jrY09Bq9VSt157ypSrS726tahWrbJh34TxI0hKSmLt2r9yX69kYfQmpVwspaybalucqqr04iTPX2RLoA7QGegAfCWEcM9MP5P0BhFCTEEXr7EUQuwFGgCHgPFCiFpSyunpldMf8GIAaxvXDFtwXNwtnJxKEBsbh5NTCW7evP1CnvDwCOLjH1GtWmWCgk6xa9c+du3SfaT86KN30CabtlNldFRMGu/d1aUUMTE3TCozNR4erQgOPk1c3C0Aw/8AS5etYdvW3zMqahSt2jbl9MkwbqVz7lOTkJBIQsJ9AE6fDOPK5UjKVyjLqZBQ9nofYq/3IQDeea83yTkcimrcuC6eXdrT0aM1trY2FC5ciN9X/kzZsqUJCtwLgKtrKQL89tCoSWdu3LiZo/LTIy/bRYUKZfP02J/n/v0HHD7yj+6Da+g5Bg7sTedObWnXoU+u6wI52nUvCnBL9dsVuJ5OnltSynggXghxBKgBnM+o0gw9ayHEL0KInzPaslC2F9AEaA4MBbpLKb9B9wTpm0XZLNmxcy8DB+ji8QMH9GbHDh8AypZ1Q6PRALpXb3f38ly9qnsbSQmVFCniwKefvMvyFWtfVY1MCQgMoWLFcpQt64aVlRV9+nRjx04fk8pMTb++3dOEQJycShj+7t6tI6Gh516p/u69OhkVAin2WlEsLHTNrHQZV8qVL8O1K7oPW685FgPAwaEw733Uj3WrNr+STs8z6csZlC1fl4ruDXlnwBAOHvybPn0H4exag4ruDano3pCoqBjqNeiQa8YqL9vFmTPheXrsAI6OxQw9PWxtbWnTuhnnzl2kQ/uWjBk9hO493+fx4ye5pk9qpBRGb1kQAFQSQpQTQlgD/YDtz+XZBjQTQlgKIezRObRnM6s0M8/6xc/6xpMkpUwGHgkhLkopHwBIKR8LIbLlPq1e9SvNmzfC0bEYly4G8M23s5k161fWrl3I+x/0IzIymv79PwWgSeP6jBkzhMTEJLRaLSNGTuK2/gPWnNlTefNN3QfZ6dPncuHCZQDq1KnBpo1LKVrUgc6d2zF58ihq1mrzCoeuIzk5mZGffYnXrrVoLCxY+fsGwsIyfGjmKHZ2trRt05zBQ8YZ0mZ8/yU1alRFSsnVq1Fp9mUXWztbmrdszLjPpxrSPDq3YdoPEynmWIxVG34j9PQ53uk1iIaN6zJ6wjCSk5NJTk5mwhffcO+eztP+ZsYEqupfgX+atYBLF68CUKPWGyxbPQ+HIoVp59GSL8YPpXXj57/P5E9ys138sXo+LfT3zpVLgUz95kdWrFyfYf6I874ULlwQa2trunX1oGPn/pw9eyHD/C9DqVIlWb5sLhqNBRYWFmzevINdXvsIDzuGjY0N3rt1+vn5BTF0WLo93kxGTo1glFImCSGGAXsADbBcShkqhPhUv3+hlPKsEMIbOIVuJu2lUsozmdUrTBEvE0L4Aa2klI+EEBZS6k6DEMIBOCilrJ1VHZmFQXITbS73FjBn1OIDCnMlKSH6lWMY56t4GH2zu5/1zvURNFnGrIUQxYFx6PoL2qakSykzG1XRXEr5VJ8v9fPKCnjv5VRVKBQK02FEeCNPMaY3yBp0sZRywFTgCrqYTIakGOp00m9JKU+nt0+hUCjykuz0BskLjDHWr0kplwGJUsrDUsoPgYYm1kuhUChylZwawWgqjOm6l6j/P0YI0RldFxRX06mkUCgUuY/WzMMgxhjrafoPg18AvwCFgc9NqpVCoVDkMuYes87SWEspd+r/vA+0Mq06CoVCkTeYe8cvY3qDrODFoZLoY9cKhULxn+C/EAbZmepvW6AHLw6dVCgUinyNNr+vFCOl/DP1byHEOmBfBtkVCoUiX/Jf8KyfpxJQOqcVeR41ctD8UKMHFf9l8v0HRiHEv6SNWceiG9GoUCgU/xnyvWctpSyUG4ooFApFXmLu7/JZjmAUQryw2mZ6aQqFQpGfSdZaGL3lBRl61kIIW8AecBRCFOXZ6geFAdOuiaVQKBS5jGmXI3l1MguDfAJ8hs4wn+CZsX6AbuVehUKh+M8g012Ny3zI0FhLKecB84QQw6WUv+SiTgqFQpHraM08aG1M8EUrhCiS8kMIUVQIMcR0KikUCkXuo0UYveUFxhjr/0kp76X8kFLeBf5nMo0UCoUiD5AIo7e8wJhBMRZCCCH1638JITSAtWnVUigUitwl2cxj1sZ41nuAjUKINkKI1sA6YLdp1TIOB4fCbFi/mDOnD3P61CEaNqjD2jULCAzwITDAh4jzvgQG6FaOLlasKPt8NnHvznnmzZ2WK/p1aN+S0DNHCA87xtgxQ3NFZnpYWFgQ4L+HbVt+N6mcJYtncz3qJCHBz3p2Fi1aBG+vdZwNPYa31zqKFHEAwMrKiqVL5hActI8TgXtp0bzRS8u1sbHh+N87ORG4l5MhB5gy+QsA3nqrCydDDpDwJJI6td805M9M9q4dfxjqmf/rDMPK7DnJ8GEfERK8n5MhBxgx/OMcrz8rXF2d2eezidOnDnEy5ADDh32U6zqkYC73COh6gxi75QXGtMRxwH5gMDAU3Wq8dqZUylh+mvMNe/Yc5I3qLahdpx1nwy/w9juDqVuvPXXrtWfLFi+2bvUC4MmTJ0z5eiZjx32bK7pZWFjw87zpdPEcQPUarejbtztVqlTKFdnPM2L4x4SH5+wq1emxatVGOnd5J03auLFDOXDwGFWqNeXAwWOMG6u7IT/+6G0AatVui0fHfsycORkhXs6zefr0KW3b96FO3XbUqdueDu1b0qB+bUJDw+nd538cPeqbJn9msvu9/Sl16rajRs3WFC9ejF69uryUThlRrVplPvrobRo17kztOu3o3KktFSuWy1EZWZGUlMSYsVOp/mZLmjT1ZPDg9/OkbZrTPQL/AWOtX/DWF7gE1AXaoFuTMU8pVKggzZo2YPmKdQAkJiZy//6DNHl69fJk/YZtADx69Ji//wngyZN0l4fMcerXq8XFi1e4fPkaiYmJbNy4ja6eHXJFdmpcXErRqWMbli9fZ3JZR4/5cefuvTRpnp4dWLV6EwCrVm+ia1cPAKpUcefAwWMA3Lx5m/v3HlC3To2Xlh0f/wgAKytLLK2skFISHh7B+fMXX8ibmex//30IgKWlJdbW1jk+x/Hrr1fCzy+Ix4+fkJyczJGjvnTv5pGzQrIgNjaO4JAzADx8GE94+AVcnJ1yVQcwn3skBXOPWWdorIUQ7kKIyUKIs8CvQCSAlLKVlPLX7AoSQqx6eTVfpHz5Mty6dZtlS38iwH8PixbOwt7+mcPfrGkDbsTdJCLick6KNRpnFycio57NJBsVHYNzHtwQc2ZPZfyEaWi1eeMPlCzhSGxsHKAzEiWKvwbAqVNhdPXsgEajoWxZN2rXro6r28uPtbKwsCAwwIeY6FPs338E/4DgDPNmJdtr5xpiok/y778P+fPPnRnW8zKEhobTrFlDihUrip2dLR09WuPqmndjzMqUcaVmjTfw88/4fJkKc7lHUtAK47e8IDPPOhydF+0ppWyq72udbEylQojtz207gJ4pvzMpN0gIESiECNRq4zOVYanRUKtWdRYtWkW9+h2Ij3/EuLHDDPv79u3OBr1XnRek90ovc3kmwc6d2hIXd4ugYPNbUH7FyvVER8Xg57ubObOncvx4IElJSS9dn1arpW699pQpV5d6dWtRrVrll5bdqcs7uJaujY2NNa1bNXlpndIjPDyCWbPm4717HV4713DyVBjJSUbdVjlOgQL2bNywhFGjpxjeKHITc7hHUmPuXfcy6w3yFtAPOCiE8AbWg9FaugJhwFJ086MIdCGU2ZkVklIuBhYDWFq7ZHrVoqJjiIqKMXhQf/21i7FjdMZao9HQo3tH6jfsaKS6OU90VAxuqTwmV5dSxMTcyFUdGjeui2eX9nT0aI2trQ2FCxfi95U/8977I3JNhxtxt3ByKkFsbBxOTiWIu3kbgOTkZL4Y87Uh39HD23LkLej+/QccPvKP7sNV6Ll08xgj++nTp+zYuRdPzw7s23/0lfVKzYqV61mxcj0A074dT1RUTI7WbwyWlpZs2rCEdeu2sHVr3vQXMId7JDV588g0ngw9aynlFillX+B14BC6RXJLCiEWCCHaZ1FvXXRD1CcB96WUh4DHUsrDUsrDOaH4jRs3iYq6jrt7BQBat27K2bPnAWjbphnnzkUQHZ37N0EKAYEhVKxYjrJl3bCysqJPn27s2OmTqzpM+nIGZcvXpaJ7Q94ZMISDB//OVUMNsHOHD+8O7A3AuwN7s2PHHgDs7GwNYau2bZqRlJTE2bMv9xHU0bEYDg6FAbC1taVN62acO/dirDqFjGQXKGCPk1MJQPfA7+jRmnPnIl5Kp8worg8Fubk50717R9Zv2JrjMrJiyeLZnA2PYO68xbkuOwVzuEdSoxXC6C0vMGaK1HhgDbBGCFEM6A2MBzI8q/qPkj8JITbp/79hjKzsMvLzr1j1+y9YW1tx+fI1Pvp4FAB9+nQzfFhMTcR5XwoXLoi1tTXdunrQsXP/lzYQWZGcnMzIz77Ea9daNBYWrPx9A2Fh500iy1z4Y/V8WjRvhKNjMa5cCmTqNz/yw6z5rF+7kA/e709kZDR9+38CQIkSjnjtWotWq+V6dCzvffDyD5FSpUqyfNlcNBoLLCws2Lx5B7u89tGtmwfzfppG8eLF2L5tFSdPhtKpyzsZyi5QwJ4tf63AxsYajUbDwYN/s2jx6hw5N6nZtGEJxV4rSmJiEiNGTOLevfs5LiMzmjSux8ABvTh1OszQtfWrr2aw2/tAruphbveImY82R+RGjEgI0RloIqWcaGyZrMIgCoVCkUJSQvQru7sbSr1jtM3pG7Mm193rHPd200NKuQvYlRuyFAqF4mUw8/Vyc8dYKxQKhblj7sPNlbFWKBQKlGetUCgU+QJzXykmbxYTUygUCjNDZmPLCiGEhxDinBAiQggxPpN89YQQyUKIXlnVqTxrhUKhIOfCIPpppOcD7YAoIEAIsV1KGZZOvh/QzWyaJcqzVigUCnJ01r36QISU8pKUMgHd6O9u6eQbDvwJxBmjnzLWCoVCASQL47cscEE/8Z2eKH2aASGEC9ADWGisfspYKxQKBdnzrFNPOqffBqWqKj1z/nyoey4wTkpp9JQkKmatUCgUZK83SOpJ59IhCnBL9dsVuP5cnrrAev3Mg45AJyFEkpRya0YylbFWKBQKcnRukACgkhCiHBCNbvbSt9PIktKwPJAQYiWwMzNDDcpYKxQKBZBzvUGklElCiGHoenlogOVSylAhxKf6/UbHqVOjjLVCoVCQs4NipJRegNdzaekaaSnl+8bUqYy1QqFQYP6LDyhjrVAoFKi5QRQKhSJfYO5zgyhjrVAoFJj/SjHKWCsUCgWgNXNz/Z8YwejuXoHAAB/DdudWOCOGf8wP33/JmdOHCTqxl82blhoWVc0tIs77Ehy0j8AAH3yPe2VdwAR0aN+S0DNHCA87xtgxQ/NEh7zUI71rYKp2sWTxbK5HnSQkeL8h7a23unAy5AAJTyKpU/vNF8q4uTlz7855Rn2uW5uyYMECadpy7PXTzP5x6ivr5urqzD6fTZw+dYiTIQcYPuwjACZ/NYqrlwMN8jp6tH5lWdnBXNon6D4wGrvlBbmyBuPL8LJrMFpYWHDtygkaN+1CZfcKHDj4N8nJyXz/nW75xwkTv8tRPTMj4rwvDRp15Pbtu7kmMzUWFhacDT2KR6f+REXF4HvciwEDh5hskWBz1CO9a9CubXOTtItmTRvw8GE8K1bMo2atNgC8/npFtFrJgvkzGDvuW04EnUpTZuOGxWi1En//IOb8tOiFOv18dzN69NccPeb3Sro5OZWglFMJgkPOULBgAfz9vHmr14f07uXJw4fx6co2NTnZLnJiDcavyxi/BuPXV3N/Dcb/hGedmjatm3Lp0lWuXYtm774jJCfrnoO+fkG4uJTKY+1yl/r1anHx4hUuX75GYmIiGzduo6tnh/+3eqRgqnZx9Jgfd+7eS5MWHh7B+fMX083ftWsHLl+6RljYuXT3V6xYjhLFHV/ZUAPExsYRHHIGgIcP4wkPv4CLs9Mr1/sqmFu70Arjt7wgV4y1EKKpEGKUEKK9qWX16dON9Ru2vpD+wfv98N5z0NTi0yClZLfXOvx8d/PxR+/kqmwAZxcnIqOeTUkQFR2Dcx7coHmpR1bXIC/aBYC9vR1jRw/lm2lzMszTr283Nm3anuOyy5RxpWaNN/DzDwZgyOAPCDqxlyWLZ1OkiEOOy8sIc2mfKWiRRm95gUmMtRDCP9Xf/wN+BQoBU7JYNcEwk5VWG59tuVZWVnh2ac/mP3emSZ8wfgRJSUmsXftXtut8FZq37E79Bh508RzA4MHv06xpg1yVr58kJg15EfbKSz0yuwZ51S4Avp48mrk/LyE+/lGGeTJyPF6FAgXs2bhhCaNGT+Hffx+ycNEq3F9vTJ267YmNjWPWzMk5Ki8zzKV9GmRnY8sLTNUbxCrV34OAdlLKm0KIHwFfYEZ6hVLPZPUyMWsPj1YEB58mLu6WIW3gwN507tSWdh36ZLe6VyYm5gYAN2/eZtu23dSrVzNHXmmNJToqBjdXZ8NvV5dSBp1yk7zUI6NrkJftAqB+/Vr07NmZGd9NokiRwmi1Wp48ecpvC1YC8OabVbG0tCQo+HSOybS0tGTThiWsW7eFrVt3A6S5V5YuW8O2rb/nmLysMJf2mYK597M2VRjEQghRVAjxGrqPmDcBpJTxQJKJZNKvb/c0nkiH9i0ZM3oI3Xu+z+PHT0wlNl3s7e0oWLCA4e92bVsQGpp+bNJUBASGULFiOcqWdcPKyoo+fbqxY6dPruqQl3pkdA3ysl2k0LJ1Tyq6N6Sie0N+/mUpM374xWCoQRcC2ZDDXvWSxbM5Gx7B3HnPZvZ0ciph+Lt7t4652kbNpX2mkIw0essLTOVZOwAn0E3CLYUQTlLKWCFEQdKfmPuVsbOzpW2b5gweMs6QNm/uNGxsbPDevR4AP78ghg7LMAqTo5QsWZzNm5YBYGmpYf36rezxOZQrslNITk5m5Gdf4rVrLRoLC1b+voGwsPO5qkNe6pHRNQgPO2aSdvHH6vm0aN4IR8diXLkUyNRvfuTO3XvM+2kaxYsXY/u2VZw8GUqnLll/v+j1liee3Qa+sk4pNGlcj4EDenHqdBiBATqD+NVXM+jbtzs1alRFSsnVq1Fp7h9TYy7tMwVz96xzteueEMIeKCmlvJxV3pftuqdQKP7/kRNd90aV7We0zZlzZX2u9wnJ1RGMUspHQJaGWqFQKHIbc/cO1XBzhUKhwPzDIMpYKxQKBeTZh0NjUcZaoVAoMP+JnJSxVigUClTMWqFQKPIFyrNWKBSKfID6wKhQKBT5AKk865dDY2Ees7cma839eav4/4pTwaJ5rQIAsQ/zZr72nEb1BlEoFIp8gLm7ZcpYKxQKBaA101WzUlDGWqFQKFBd9xQKhSJfoLruKRQKRT5A9QZRKBSKfECSMtYKhUJh/ijPWqFQKPIBquueQqFQ5APycmV1YzCPYYKZsGjRj0ReCyboxD5D2pQpowkM8MHfz5tdO9dQqlRJw74xY4YSFnqU06cO0a5tC0N6nz7dOBG4l8AAH3ZsX81rr+lGf5Uu7YL37nUEBvjg47MRFxenbOnn6urMPp9NnD51iJMhBxg+7CMA3nqrCydDDpDwJJI6td805G/bphl+vrsJDtqHn+9uWrVs8lLnxVg6tG9J6JkjhIcdY+yYoTlad0bHDjB0yAeEnjnCyZADzPh+kiG9evUqHDuynZMhBwgO2oeNjQ0AvXt3JejE3hfyvypLFs/metRJQoL3G9LefLMqx45sJzhoH1u3rKRQoYI5Ji8jHBwKs2H9Ys6cPszpU4do2KBOjulRvmJZ9hzebNjOXvXlo08H8NuyHw1px0P2sOfwZkC3yvlP86ez79hfHPTdztDPPjbUtWn7Cg777TCUe82xGABTpo81pB3x30no5X8y1Sm98160aBG8vdZxNvQY3l7rKFLEIU0ZNzdn7t05z6jPPzGkmapdpIcWafSWF+TqGozZwcbWTQI0bdqAhw/jWb5sLrXrtAWgUKGC/PvvQ0BnFKpUqcSw4RN5/fVKrF71K02aeuLsXJLdXuuo9kZzhBBcuRxIzVqtuX37Lt9Nn8ijx4+ZNu0n1q5ZgNfu/fzxx2ZatmzMu+/24cMPPzPokdVwcyenEpRyKkFwyBkKFiyAv583b/X6ECklWq1kwfwZjB33LSeCTgFQs2Y1bty4RUzMDapVq4zXzjWUKVfXFKcQCwsLzoYexaNTf6KiYvA97sWAgUM4e/ZCjtSf0bGXLFGcCeNH4NntXRISEihe/DVu3ryNRqMhwN+b9z8YyalTYRQrVpR79+5TpIgDgf57qN/Qg1u37rB82Vz++GMzBw4ee2Udm+nbz4oV86hZqw0Ax//Zxbhx33LkqC/vv9eXcuVKM+XrWa8sKzOWL5vLsWN+LF+xDisrK+zt7fDeve6V9EhvuLmFhQWBoQfwbNef6KgYQ/pX347m3wcPmTtrId3f6kS7jq0Y+vEYbO1sOXh8G709PyAq8jqbtq/g28k/ciokNEO5H/zvbaq9WYXRw78C0h9unt55n/H9JO7cucfMWfMZO2YoRYs6MGHid4YyGzcsRquV+PsHMeenRRQrVtTodpETazB2Kd3ZaGO489quTOUJITyAeYAGWCqlnPHc/neAlNWJHwKDpZQnM6vT7D3rY8f8uHv3Xpq0FEMNYF/AnpTnjadnezZu2k5CQgJXrkRy8eIV6tWriRACIQQFCtgDULhwQWJibgBQpUolDuov/qFD/+DZpX229IuNjSM45AwADx/GEx5+ARdnJ8LDIzh//uIL+UNCQg2yQ0PPYWtri7W1dbZkGkv9erW4ePEKly9fIzExkY0bt9HVs0OO1Z/RsX/yybvMnDWfhIQEAG7evA1A+3YtOH36LKdOhQFw585dtFot5cuV5sKFS9y6dQeA/QeO0qNHpxzR8egxP+48134qu1fgyFFfAPbtzzlZGVGoUEGaNW3A8hXrAEhMTOT+/Qcm0aNpi4ZcvRKZxlADeHb3YNufXoDudd/e3g6NRoOtrQ2JCYk8THVPZUW3tzoZ6sqI9M67p2cHVq3eBMCq1Zvo2tXDsK9r1w5cvnSNsLBzhjRTtov0yCnPWgihAeYDHYGqQH8hRNXnsl0GWkgp3wS+BRZnpZ9JjLUQooEQorD+bzshxFQhxA4hxA9CCIesyhvD1KljiYjwo3+/Hkz95kcAXJydiIq6bsgTFR2Ds7MTSUlJDB8xkROBe7lyOZDXq7izYsV6AE6dPkuP7roG0K2bB4ULF6JYsSIvpVOZMq7UrPEGfv7BRuXv2bMzISFnDEYtp3F2cSIynfNhClIfe6VK5WnatD7/HNvBgX2bqVunBgCVKpVHSvDauQZ/P29GfzEYgIiLV6hcuSJlyrii0Wjo1rUDbm7OJtETdA9JT0/dQ7nXW11wczWdLIDy5ctw69Ztli39iQD/PSxaOAt7ezuT6NG1Z8cXDGmDRnW4GXeby5euAbBr+14ePXpM0NmD+J/ay6L5K7l374Eh/5xfv2XP4c2MHP0Jz+PiWgq30i78fcQv27qVLOFIbGwcoHvQlyj+GgD29naMHT2Ub6bNSZM/t9uFlNLoLQvqAxFSyktSygRgPdDtOVn/SClTXkl8AdesKjWVZ70ceKT/ex7gAPygT1uRUSEhxCAhRKAQIjA5OfMn/ZQpM6lYsQHr1m9h8OD3U8q/kE9KiaWlJZ8MGkiDhh0pW64uZ06fZezYYQCMHz+NZs0a4ue7m+bNGhIVFUNSUnK2D7hAAXs2bljCqNFT0nj+GVG1qjvfT5/I4KHjssz7smR0PnKa54/d0lJDkSIONG7qybjx01i3diEAlpYamjSux8D3htGiZXe6d+tI61ZNuXfvPsOGT2DdmgUcPriFq1eiSEpKynE9U/h40CiGfPo+fr67KVSoAAkJiSaTBWCp0VCrVnUWLVpFvfodiI9/xLixw3JcDysrS9p7tGTnNp806d3e6sS2v54Z8Jp1qqNNTqZO1dY0quXBoCHvUbqMzlYM/2QcbZv2pGfnd6nfqA5v9e2atq6eHfHa7oM2B2ej/HryaOb+vIT4+Edp0nO7XWizsaW2VfptUKqqXIDIVL+j9GkZ8RGwOyv9TNUbxEJKmXJW60opa+v/PiaECMmokJRyMfrXgZSYdVZs2LCVrVt+59tv5xAVHYNrKu/E1aUUMTE3qFGjGgCXLl0FYPOfOxkzeggAMTE36NtPd54LFLCne/dOPHjwr9EHCroPNps2LGHdui1s3ZrlOcfFpRSbNy3jgw9HGnQyBdFRMWm8tZTzkZOkd+zRUTGGvwMCQ9BqtTg6FiMqOoYjR325fVvnUOz2PkCtWm9w4OAxdu7ay85dewH4+KN3SNZm/4FpLOfOXaRj57cBnbffqWMbk8kC3RtNVFQM/gG6N66//trF2DHDmPL1rBzVo1XbZpw+dZZb+rATgEajoWOXtnRq3ceQ1v2tThza/zdJSUncvnWHAP8Q3qxVjWtXo4iN0Xm+8Q8fsXXzLmrVfoM/N2w3lO3asyOTxk5/Kf1uxN3CyakEsbFxODmVIE6vZ/36tejZszMzvptEkSKF0Wq1PHnylN8WrMzVdpGdftapbVU6pBfPTrdyIUQrdMa6aVYyTeVZnxFCfKD/+6QQoq5eMXfgld2YihXKGv7u0rkd585FALBz51769O6KtbU1Zcu6UbFiWQICQrh+PZbXX6+Eo/7Ldps2zQgP15V57bWiBg907Nhh/L5qQ7b1WbJ4NmfDI5g7L8uwEw4Ohdm+bRWTvvyef44HZltWdggIDKFixXKULeuGlZUVffp0Y8dOn6wLZoP0jn3b9j20aqXr5VKpUnmsra25desOPj6HqV69CnZ2tmg0Gpo3a2j42Flc/0pcpIgDn376HsuWr8tRPVOTIksIwcQJI1m0eLXJZAHcuHGTqKjruLtXAKB166acPXs+x/VIL5bcrGVDLl64RMz1Zw/p61ExNG5eHwA7eztq132Ti+cvo9FoKKoPAVpaWtK2QwvCz0YYypWvWBaHIoU54R/yUvrt3OHDuwN7A/DuwN7s2LEHgJate1LRvSEV3Rvy8y9LmfHDL/y2YCWQu+0iB3uDRAFuqX67AtefzySEeBNYCnSTUt5+fv/zmMqz/hiYJ4T4ErgFHBdCRKJ7Nfg405LPsWrVrzRv1hBHx2JcjPDn22mz8ejQGnf3Cmi1Wq5di2LY8IkAnD17ns1/7uRkyAGSkpIYOfJLtFotMTE3mD59Lvv3bSYxMYlr16L4+H+jAGjevBHTvh2PlJKjx/wYOfLLbB1ok8b1GDigF6dOhxEYoDOEX301A2sba+b9NI3ixYuxfdsqTp4MpVOXdxg65AMqVijLpImfMWniZwB07NTf8BEuJ0lOTmbkZ1/itWstGgsLVv6+gbCw8zlWf0bHvmLlepYumU1I8H4SEhL58KPPAN1r7dx5i/E97oWUEm/vA3jt1nXt+mnON7z5pu4bzLTpP3HhwqUc0fGP1fNp0bwRjo7FuHIpkKnf/EjBggUMobOtW71Y+Xv2H9DZZeTnX7Hq91+wtrbi8uVrfPTxKAYO6JVjetja2dK8ZSPGfz41TXrXHh3Z+mfat72Vy9Yx59dp7P9nK0IINq7dytmw89jZ27Fm8yKsrKyw0Fhw7LAva1dtNpTr/lYntv+V9ZsjpH/ef5g1n/VrF/LB+/2JjIymb/8XY+LPY6p2kR7JMsdCOwFAJSFEOSAa6Ae8nTqDEKI08BcwUEpp1E1p0q57QohCQHl0D4UoKaXR7+DGhkFMjVopRmGuqJVinpETXfdaurY12uYcitqXVde9TsBcdF33lksppwshPgWQUi4UQiwF3gJS4qBJUspM+/CafT/rvEYZa4W5ooz1M3LCWDd3aWO0zTkSvf+V5WUXNdxcoVAoUIsPKBQKRb5ALT6gUCgU+QBlrBUKhSIfkIO9QUyCMtYKhUKBWnxAoVAo8gXm2jMuBWWsFQqFAhWzVigUinyB8qxfEjUYRaHIHHMYjPJfItnMV2E0W2OtUCgUuYlWedYKhUJh/qjeIAqFQpEPUJ61QqFQ5AOUZ61QKBT5AOVZKxQKRT5ADTdXKBSKfIAKgygUCkU+QJq5Z22qBXNNwpLFs7kedZKQ4P2GtKJFi+DttY6zocfw9lpHkSIOANSrW5PAAB8CA3w4EbiXbt08DGWsrKxY8NsPhIUe5czpw/To0clk+qUw6vNPSEqI5rXXcnd1jw7tWxJ65gjhYccYO2aoSWXZ2Nhw/O+dnAjcy8mQA0yZ/AWQ8TUCGDd2GOFhxwg9c4T27VqYVD8Hh8JsWL+YM6cPc/rUIRo2qGNSeenh7l7B0C4DA3y4cyucEcOztSxppqTXBteuWWCQF3He17BepqWlJcuXzSU4aB+nTx1i3NhhhjL7924i9MwRQ7mUhWuNwdXVmX0+mzh96hAnQw4wfNhHafY/fy+0bdMMP9/dBAftw893N61aNjHkrV2rOsFB+wgPO8ZPc755qXNiLDm4YK5JMNtlvSytXV5QrFnTBjx8GM+KFfOoWasNADO+n8SdO/eYOWs+Y8cMpWhRByZM/A47O1sSEhJJTk7GyakEQYF7cStTm+TkZKZM/gKNRsPkKTMRQlCsWBFu33710WDp6Qe6xrt44SwqV65I/YYeOSLLGCwsLDgbehSPTv2JiorB97gXAwYOMawobgoKFLAnPv4RlpaWHDm0hc9HTaFHj47pXqMqVSrxx+rfaNS4M87OJdmzez1VqjVDa6LRq8uXzeXYMT+Wr1iHlZUV9vZ23L//wCSyjMHCwoJrV07QuGkXrl2LzpE6M2qDKcz6YTL3Hzxg2vS59OvXHc8u7XlnwBDs7Gw5ffIQbdr14urVKPbv3cTYcd9yIuhUtnVwcipBKacSBIecoWDBAvj7efNWrw85e/ZCuvdCzZrVuHHjFjExN6hWrTJeO9dQppxuOcLjf+/k81FT8PU7wc7tq/l1/nK89xx8QWZOLOtVulh1o43htTunc31Zr3zlWR895sedu/fSpHl6dmDV6k0ArFq9ia5ddR7048dPSE5OBsDW1ibNuP/33+vHjB9+AXTzAeSU8UxPP4DZP37N+InTc33ugfr1anHx4hUuX75GYmIiGzduo6tnB5PKjI9/BICVlSWWVlZIKTO8Rl09O7Bx4zYSEhK4ciWSixevUL9eLZPoVahQQZo1bcDyFesASExMzFNDDdCmdVMuXbqaY4YaMm6DKfTq5cn6DdsAXdsvUMAejUaDnZ0dCYmJPHjw8JV1iI2NIzjkDAAPH8YTHn4BF2cnIP17ISQklJgY3VraoaHnsLW1xdraGienEhQqXAhfvxMArF6z2dB2TIG5e9YmMdZCiBFCCDdT1P08JUs4EhsbB+gaSYlUr2v169XiZMgBQoL2M2TYeJKTk3FwKAzAN1+Pxd/Pm/XrFlGihKPJ9OvSpR3R0TGcOhVmMhkZ4eziRGTUdcPvqOgYnPU3jamwsLAgMMCHmOhT7N9/BP+A4AyvkbNzOvq5mEa/8uXLcOvWbZYt/YkA/z0sWjgLe3s7k8gylj59urF+w9Zck9esaQNuxN0kIuIyAH/+uYv4+EdEXQvm8kV/5sxZyN1Uhn7p0jkEBvgwaeJnLy2zTBlXatZ4Az//YKPuhZ49OxMScoaEhARcnJ2Ijoox7IuOijEYfVOQrNUaveUFpvKsvwX8hBBHhRBDhBDFjSkkhBgkhAgUQgRqtfGvrIR/QDA1aramYeNOjB87DBsbGywtNbi5OfP38QDqN/DA1/cEM3+Y/Mqy0sPOzpaJ40fw9dQfTVJ/Vgjx4puaqb17rVZL3XrtKVOuLvXq1qJatcoZ5s1N/Sw1GmrVqs6iRauoV78D8fGP0sRocxsrKys8u7Rn8587c01m377d2aD3qgHq16tJcnIybmVqU9G9IZ9//gnlypUGYOB7w6lVuy0tW/WgaZP6DBjQK9vyChSwZ+OGJYwaPYWkpKQs74WqVd35fvpEBg8dB2TQPkzo1cps/MsLTGWsLwGu6Ix2HSBMCOEthHhPCFEoo0JSysVSyrpSyroWFgWMEnQj7hZOTiUAXaws7ubtF/KEh0cQH/+YN6pV5vbtu8THP2Lr1t0AbP5zJ7VqvZHd4zOKChXKUrZsaYIC9xJx3hdX11IE+O2hZEmjnl2vTHRUDG6uzobfri6lDK+bpub+/QccPvIPHdq3zPAaRUeno9910+gXFR1DVFQM/gHBAPz11y5q1axuElnG4OHRiuDg08TF3coVeRqNhh7dO7Jx03ZDWr9+Pdjjc4ikpCRu3rzNP/8EUKdODQCuX48FdGGMdeu3Uq9uzWzJs7S0ZNOGJaxbt4WtW3dneS+4uJRi86ZlfPDhSC5dugrorpmLaylDnS6upbhuovYBOkfB2C0vMJWxllJKrZTSR0r5EeAM/AZ4oDPkOcbOHT68O7A3AO8O7M2OHXsAKFvWDY1GA0Dp0i64u5fnytVIXZlde2nZojEArVs1NdkHtzNnwnF2rUFF94ZUdG9IVFQM9Rp04MaNmyaR9zwBgSFUrFiOsmXdsLKyok+fbuzY6WMyeY6OxQxhJltbW9q0bsa5cxczvEY7dvrQp083rK2tKVvWjYoVyxmMaU5z48ZNoqKu4+5eAYDWrZty9ux5k8gyhn59u+dqCKRtm2acOxdBdPSzsEJkZLSh54W9vR0NGtTm3LkINBqNoaeGpaUlnTu3JTT0XLbkLVk8m7PhEcydtxjI/F5wcCjM9m2rmPTl9/xzPNBQR2xsHP/++5AG9WsDMPCdXoa2YwrMPWZtkt4gQohgKWW6X4qEEHZSysdZ1ZFeb5A/Vs+nRfNGODoW48aNW0z95ke2bd/D+rULcXNzITIymr79P+Hu3Xu8885bjB0zlMTEJLRaLdOm/8T27boLXbq0C7+v+BmHIoW5dfMOH/3vcyIjr7+gQ3ZJT78VK9cb9kec96VBo4651hsEoKNHa2bPnorGwoKVv2/g+xk/m0xW9epVWL5sLhqNBRYWFmzevINp0+dSrFjRdK8RwITxI3j/vb4kJSfzxRdT0v3Sn1PUqFGNRQtnYW1txeXL1/jo41Hcu3ffZPIyws7OliuXAqlUuREPHvybo3Vn1AaXLf0JP78gFi9ZbchboIA9y5b+RJUqlRBC8PvvG5g9ZyH29nYcPPAXVlaWaDQa9u8/yugxU43updOkcT0OH9rKqdNhaLW62/irr2aw2/uAIU/qe2HihJGMGzuMC/pYOkDHTv25efM2dWq/ybJlP2Fna4v3noOM/OzLdGXmRG8Qx8LuRhvDWw/O53pvEFMZa3cp5Su5LekZa4VCoUiPnDDWRQtWNNrm3H0YkevG2iQjGF/VUCsUCkVuo9ZgVCgUinyAuQ4QTEEZa4VCoUBNkapQKBT5AjXrnkKhUOQDlGetUCgU+QCtmiJVoVAozJ+cHMEohPAQQpwTQkQIIcans18IIX7W7z8lhKidVZ3KWCsUCgU5Z6yFEBpgPtARqAr0F0JUfS5bR6CSfhsELMhKP2WsFQqFApDZ2LKgPhAhpbwkpUwA1gPdnsvTDVgldfgCRYQQpZ6vKDVmG7POiRFJQohBUsrFOaFPftbBXPQwBx3MRQ9z0MFc9DAHHSB7NkcIMQidR5zC4lTH4AJEptoXBTR4ror08rgAMWTAf92zHpR1FpNjDjqAeehhDjqAeehhDjqAeehhDjpki9QzhOq31A+b9Iz+8w65MXnS8F831gqFQpHbRAGpF19xBZ6fKc6YPGlQxlqhUChylgCgkhCinBDCGugHbH8uz3bgXX2vkIbAfSllhiEQMOOYdQ6R53EwzEMHMA89zEEHMA89zEEHMA89zEGHHENKmSSEGAbsATTAcillqBDiU/3+hYAX0AmIAB4BH2RVr9mubq5QKBSKZ6gwiEKhUOQDlLFWKBSKfMB/0lhnNdQzl3RYLoSIE0KcyQv5eh3chBAHhRBnhRChQoiReaSHrRDCXwhxUq/H1LzQQ6+LRggRLITIvWXFX9ThihDitBAiRAgRmHUJk+hQRAixWQgRrm8fjfJAh8r6c5CyPRBCfJbbeuQX/nMxa/1Qz/NAO3TdYwKA/lLKsFzWoznwEN0oJdMsn561DqWAUlLKIP2q8ieA7nlwLgRQQEr5UAhhBRwDRupHbuUqQohRQF2gsJSyS27L1+twBagrpcydpc3T1+F34KiUcqm+x4K9lPJeHuqjAaKBBlLKq3mlhznzX/SsjRnqaXKklEeAO7kt9zkdYqSUQfq//wXOohslldt6SCnlQ/1PK/2W616CEMIV6AwszW3Z5oQQojDQHFgGIKVMyEtDracNcFEZ6oz5LxrrjIZx/r9GCFEWqAX45ZF8jRAiBIgD9kop80KPucBYIK/nwpSAjxDihH7Ycm5THrgJrNCHhJYKIQrkgR6p6Qesy2MdzJr/orHO9jDO/zpCiILAn8BnUsoHeaGDlDJZSlkT3Uit+kKIXA0NCSG6AHFSyhO5KTcDmkgpa6ObeW2oPmSWm1gCtYEFUspaQDyQJ992APRhmK7AprzSIT/wXzTW2R7G+V9GHyP+E1gjpfwrr/XRv24fAjxyWXQToKs+XrweaC2E+COXdQBASnld/38csAVd6C43iQKiUr3dbEZnvPOKjkCQlPJGHupg9vwXjbUxQz3/X6D/sLcMOCulnJOHehQXQhTR/20HtAXCc1MHKeUEKaWrlLIsujZxQEo5IDd1ABBCFNB/7EUfemgP5GqPISllLBAphKisT2oD5OpH5+fojwqBZMl/brh5RkM9c1sPIcQ6oCXgKISIAqZIKZflshpNgIHAaX28GGCilNIrl/UoBfyu/+JvAWyUUuZZ17k8piSwRfccxRJYK6X0zgM9hgNr9A7NJYwY7mwKhBD26HpufZIX8vMT/7muewqFQvFf5L8YBlEoFIr/HMpYKxQKRT5AGWuFQqHIByhjrVAoFPkAZawVCoUiH6CMtcIkCCGS9TOpnRFCbNJ30XrZulYKIXrp/14qhKiaSd6WQojGLyHjihDC8WV1VChMjTLWClPxWEpZUz/jYALwaeqd+j7X2UZK+XEWswa2BLJtrBUKc0cZa0VucBSoqPd6Dwoh1qIbqKMRQswSQgQIIU4JIT4B3chLIcSvQogwIcQuoERKRUKIQ0KIuvq/PYQQQfp5svfrJ6v6FPhc79U304+e/FMvI0AI0URf9jUhhI9+IqNFpD+njEJhNvznRjAqzAshhCW6uR9SRunVB96QUl7Wzzh3X0pZTwhhA/wthPBBNztgZaA6uhF/YcDy5+otDiwBmuvrKialvCOEWAg8lFL+qM+3FvhJSnlMCFEa3cjWKsAU4JiU8hshRGcgL2a/UyiMRhlrhamwSzXE/Si6OUoaA/5Sysv69PbAmynxaMABqIRuruV1Uspk4LoQ4kA69TcEjqTUJaXMaO7wtkBV/fBugML6uTmaAz31ZXcJIe6+3GEqFLmDMtYKU/FYPyWqAb3BjE+dBAyXUu55Ll8nsp7WVhiRB3ShvkZSysfp6KLmWlDkG1TMWpGX7AEG66dxRQjhrp+J7gjQTx/TLgW0SqfscaCFEKKcvmwxffq/QKFU+XyAYSk/hBA19X8eAd7Rp3UEiubUQSkUpkAZa0VeshRdPDpI6BYWXoTubW8LcAE4DSwADj9fUEp5E12c+S8hxElgg37XDqBHygdGYARQV/8BM4xnvVKmAs2FEEHowjHXTHSMCkWOoGbdUygUinyA8qwVCoUiH6CMtUKhUOQDlLFWKBSKfIAy1gqFQpEPUMZaoVAo8gHKWCsUCkU+QBlrhUKhyAf8HxBwvwcrxkQwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from src.eval import plot_confusion_matrix, cv_metrics_table\n",
    "\n",
    "plot_confusion_matrix(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With 'cv_metrics_table' the metrics accuracy, balanced accuracy, weighted and unweighted F1 score of the results can be calculated. The rows \"mean\" and \"std\" show the average and standard deviation of the metrics across folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ACC</th>\n",
       "      <th>BACC</th>\n",
       "      <th>F1_weighted</th>\n",
       "      <th>F1_macro</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.939407</td>\n",
       "      <td>0.790963</td>\n",
       "      <td>0.939076</td>\n",
       "      <td>0.813443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.941879</td>\n",
       "      <td>0.795431</td>\n",
       "      <td>0.941803</td>\n",
       "      <td>0.804426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.917142</td>\n",
       "      <td>0.801626</td>\n",
       "      <td>0.917121</td>\n",
       "      <td>0.762276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.941038</td>\n",
       "      <td>0.804906</td>\n",
       "      <td>0.940335</td>\n",
       "      <td>0.831395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.942321</td>\n",
       "      <td>0.831302</td>\n",
       "      <td>0.942154</td>\n",
       "      <td>0.821575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.936358</td>\n",
       "      <td>0.804845</td>\n",
       "      <td>0.936098</td>\n",
       "      <td>0.806623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.009659</td>\n",
       "      <td>0.014083</td>\n",
       "      <td>0.009552</td>\n",
       "      <td>0.023897</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           ACC      BACC  F1_weighted  F1_macro\n",
       "0     0.939407  0.790963     0.939076  0.813443\n",
       "1     0.941879  0.795431     0.941803  0.804426\n",
       "2     0.917142  0.801626     0.917121  0.762276\n",
       "3     0.941038  0.804906     0.940335  0.831395\n",
       "4     0.942321  0.831302     0.942154  0.821575\n",
       "mean  0.936358  0.804845     0.936098  0.806623\n",
       "std   0.009659  0.014083     0.009552  0.023897"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_metrics_table(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}