lesson1-DogBreed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kaggle | Dog Breed Identification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use convolutional neural networks (CNNs) to allow our computer to see - something that is only possible thanks to deep learning. More, we are going to reuse a pre-trained model.\n",
    "\n",
    "Link : https://www.kaggle.com/c/dog-breed-identification/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video from the fastai course"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watch video : https://youtu.be/JNxcznsrRb8?t=1h39m50s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Review steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 1) All steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "1. Enable data augmentation, and precompute=True (it is just to save time but you can start with precompute=False)\n",
    "1. Use `lr_find()` to find highest learning rate where loss is still clearly improving\n",
    "1. Train last layer from precomputed activations for 1-2 epochs\n",
    "1. Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1\n",
    "1. Unfreeze all layers\n",
    "1. Set earlier layers to 3x-10x lower learning rate than next higher layer\n",
    "1. Use `lr_find()` again\n",
    "1. Train full network with cycle_mult=2 until over-fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 2) Main steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "1. Get best LR\n",
    "1. Train frozen firts layers\n",
    "1. Train unfrozen model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "1. Use `lr_find()` to find highest learning rate where loss is still clearly improving\n",
    "1. Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1\n",
    "1. Unfreeze all layers\n",
    "1. Set earlier layers to 3x-10x lower learning rate than next higher layer\n",
    "1. Train full network with cycle_mult=2 until over-fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1) Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put these at the top of every notebook, to get automatic reloading and inline plotting\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we import the libraries we need. We'll learn about what each does during the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This file contains all the main external libs we'll use\n",
    "from fastai.imports import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.torch_imports import *\n",
    "from fastai.transforms import *\n",
    "from fastai.conv_learner import *\n",
    "from fastai.model import *\n",
    "from fastai.dataset import *\n",
    "from fastai.sgdr import *\n",
    "from fastai.plots import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you have multiple GPU, choose one \n",
    "# torch.cuda.set_device(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = \"data/dogbreed/\"\n",
    "arch=resnet34\n",
    "#arch=resnext50\n",
    "#arch=resnext101_64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sz=299\n",
    "bs=64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2) Train and valid images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_csv = f'{PATH}labels.csv'\n",
    "n = len(list(open(label_csv)))-1\n",
    "val_idxs = get_cv_idxs(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{n} training images')\n",
    "print(f'{len(val_idxs)} validation images')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3) Data analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls {PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_df = pd.read_csv(label_csv)\n",
    "label_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_df.pivot_table(index='breed', aggfunc=len).sort_values('id', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\n",
    "data = ImageClassifierData.from_csv(PATH, 'train', label_csv, test_name='test',\n",
    "                                    val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = PATH+data.trn_ds.fnames[0]; fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = PIL.Image.open(fn); img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size_d = {k: PIL.Image.open(PATH+k).size for k in data.trn_ds.fnames}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sz, col_sz = zip(*size_d.values())\n",
    "row_sz = np.array(row_sz)\n",
    "col_sz = np.array(col_sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sz[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(row_sz);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(row_sz[row_sz<1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(col_sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(col_sz);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(col_sz[col_sz<1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(data.trn_ds),len(data.test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(data.classes),data.classes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3) Initial model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz,bs):\n",
    "    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\n",
    "    data = ImageClassifierData.from_csv(PATH, 'train', label_csv, test_name='test', num_workers=4,\n",
    "                                    val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)\n",
    "    # without test files\n",
    "    #data = ImageClassifierData.from_csv(PATH, 'train', f'{label_csv}', num_workers=4,\n",
    "    #                                val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)\n",
    "    return data if sz>300 else data.resize(340, 'tmp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1) Step : pre - Pre-compute (check everything works well)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can set sz a small value like 64 to check if everything works well before to raise it \n",
    "data = get_data(sz,bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.pretrained(arch, data, precompute=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all details about layers\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the layers structure of model\n",
    "learn.get_layer_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time learn.fit(lr, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('weights1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2) Step1 - Data augmentation and LR restart (run a new model learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_data(sz,bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.pretrained(arch, data, precompute=True, ps=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.summaryetails about layers\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the layers structure of model\n",
    "learn.get_layer_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time learn.fit(lr, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.precompute = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all details about layers\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the layers structure of model\n",
    "learn.get_layer_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time learn.fit(lr, 5, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3) Step 2 - Increase size images (improve your model and avoid overfitting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Increase size of images to 299 x 299 (224 x 224 before)\n",
    "learn.set_data(get_data(299,bs))\n",
    "\n",
    "# Call freeze method to be sure that all layers but last layer are frozen\n",
    "learn.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all details about layers\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the layers structure of model\n",
    "learn.get_layer_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit(lr, 3, cycle_len=1)\n",
    "# This will show that our model is underfitting as Training loss >> Validation loss\n",
    "# it means that my restart occurs too soon : we move to another hill without reaching the low of the valley"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights4')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We set up cycle_mult to 2 in order to avoid to restart too soon\n",
    "# Then, Traing Loss and Validation Loss are about the same\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this gives prediction for validation set. Predictions are in log scale\n",
    "log_preds,y = learn.TTA()\n",
    "probs = np.exp(log_preds)\n",
    "accuracy(log_preds,y), metrics.log_loss(y,probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One more fit with cycle_len = 2 could improve a little bit the accuracy\n",
    "learn.fit(lr, 1, cycle_len=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('weights6')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this gives prediction for validation set. Predictions are in log scale\n",
    "log_preds,y = learn.TTA()\n",
    "probs = np.exp(log_preds)\n",
    "accuracy(log_preds,y), metrics.log_loss(y,probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 3.4) Unfreeze ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "No need to try to unfreeze the first layers because the dataset of this kaggle is very similar to the one of ImageNet (it is a subset ?)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 3.5) Take the average of weights ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Use cycle_save_name in learn.fit() if you want to save weights after each cycle_len and get at the end the average of weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 3.6) Analyzing results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Confusion matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "preds = np.argmax(log_preds, axis=1)\n",
    "probs = np.exp(log_preds[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "A common way to analyze the result of a classification model is to use a [confusion matrix](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/). Scikit-learn has a convenient function we can use for this purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(y, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can just print out the confusion matrix, or we can show a graphical view (which is mainly useful for dependents with a larger number of categories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plot_confusion_matrix(cm, data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Images analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "to be done through image display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4) Final model (use the whole training set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just follow the same steps but with the full training step (ie, with 20% more images) before to make predictions on the test set through the following code (see lesson1-DogBreed-resnet34-FullTrainingSet.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_idxs=[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5) Create a submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data.test_ds.fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this gives prediction for test set. Predictions are in log scale\n",
    "log_preds,y = learn.TTA(is_test=True)\n",
    "probs = np.exp(log_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(probs)\n",
    "df.columns = data.classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.insert(0, 'id', [o[5:-4] for o in data.test_ds.fnames])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SUBM = f'{PATH}subm/'\n",
    "os.makedirs(SUBM, exist_ok=True)\n",
    "df.to_csv(f'{SUBM}subm.gz', compression='gzip', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FileLink(f'{SUBM}subm.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6) Save predictions on test file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save predictions\n",
    "def save_array(fname, arr):\n",
    "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
    "    c.flush()\n",
    "\n",
    "save_array(f'{SUBM}probs.bc', probs)\n",
    "\n",
    "# If many probs from diferent models, you can take the average\n",
    "# avg = (prob1 + prob2 + prob3) / 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# 7) Individual prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## 7.1) All steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fn = data.val_ds.fname[0]; fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "Image.open(PATH.fn).resize((150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "trn_tfms,val_tfms = tfms_from_model(arch, sz) # get transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "ds = FilesIndexArrayDataset([fn], np.array([0]), val_tfms, PATH)\n",
    "dl = DataLoader(ds)\n",
    "preds = learn_predict_dl(dl)\n",
    "np.argmax(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# short version\n",
    "im = val_tfms(Image.open(PATH+fn))\n",
    "preds = learn.predict_array(im[None])\n",
    "np.argmax(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# create a bar graph of the predictions\n",
    "plt.barh(np.arange(2), preds[0])\n",
    "_ = plt.yticks(np.arange(2), data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## 7.2) Short version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Get image url\n",
    "fn = data.val_ds.fname[0]\n",
    "Image.open(PATH.fn).resize((150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# get transformations\n",
    "trn_tfms,val_tfms = tfms_from_model(arch, sz) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# get prediction\n",
    "im = val_tfms(Image.open(PATH+fn))\n",
    "learn.precompute=False # We'll pass in a raw image, not activations\n",
    "preds = learn.predict_array(im[None])\n",
    "np.argmax(preds) # preds are log probabilities of classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# create a bar graph of the predictions\n",
    "plt.barh(np.arange(2), preds[0])\n",
    "_ = plt.yticks(np.arange(2), data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Annex : get automatically the best learning rate number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def plot_loss_change(sched, sma=1, n_skip=20, y_lim=(-0.01,0.01)):\n",
    "    \"\"\"\n",
    "    Plots rate of change of the loss function.\n",
    "    Parameters:\n",
    "        sched - learning rate scheduler, an instance of LR_Finder class.\n",
    "        sma - number of batches for simple moving average to smooth out the curve.\n",
    "        n_skip - number of batches to skip on the left.\n",
    "        y_lim - limits for the y axis.\n",
    "    \"\"\"\n",
    "    derivatives = [0] * (sma + 1)\n",
    "    for i in range(1 + sma, len(learn.sched.lrs)):\n",
    "        derivative = (learn.sched.losses[i] - learn.sched.losses[i - sma]) / sma\n",
    "        derivatives.append(derivative)\n",
    "        \n",
    "    plt.ylabel(\"d/loss\")\n",
    "    plt.xlabel(\"learning rate (log scale)\")\n",
    "    plt.plot(learn.sched.lrs[n_skip:], derivatives[n_skip:])\n",
    "    plt.xscale('log')\n",
    "    plt.ylim(y_lim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plot_loss_change(learn.sched, sma=20)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "266px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}