01_intro_dldiy_hti_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_intro_dldiy_hti_colab.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/abursuc/practicals-tsp-hti/blob/master/01_intro_dldiy_hti_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-SvfKxAGCdhF"
      },
      "source": [
        "# Using CNN for dogs vs cats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VI1VQlwECdhG"
      },
      "source": [
        "We are going to create a model to enter the [Dogs vs Cats](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition) competition at Kaggle."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyhW08piCdhH"
      },
      "source": [
        "There are 25,000 labelled dog and cat photos available for training, and 12,500 in the test set that we have to try to label for this competition. According to the Kaggle web-site, when this competition was launched (end of 2013): *\"**State of the art**: The current literature suggests machine classifiers can score above 80% accuracy on this task\"*. So if you can beat 80%, then you will be at the cutting edge as of 2013!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h4dAhx8-CdhM"
      },
      "source": [
        "##  Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6oU4Z_DPCdhM"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import os\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision\n",
        "from torchvision import models,transforms,datasets\n",
        "import time\n",
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A66_r51xCdhS"
      },
      "source": [
        "torch.__version__"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KN3FFTFhHQyi"
      },
      "source": [
        "import sys\n",
        "sys.version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tuej9DPjCdhX"
      },
      "source": [
        "Check if GPU is available and if not change the [runtime](https://jovianlin.io/pytorch-with-gpu-in-google-colab/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t56d0zbFCdhY"
      },
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "print('Using gpu: %s ' % torch.cuda.is_available())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDLlOjT5Et4p"
      },
      "source": [
        "## Downloading the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSmB5oKYCdhc"
      },
      "source": [
        "You can download the full dataset from Kaggle directly.\n",
        "\n",
        "Alternatively, Jeremy Howard provides a direct link to the catvsdogs [dataset](http://files.fast.ai/data/dogscats.zip). He's separated the cats and dogs into separate folders and created a validation folder as well. You'll need this folder structure to run VGG.\n",
        "\n",
        "For test purpose (or if you run on cpu), you should use the (small) sample directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rnn5pLK6EyJK"
      },
      "source": [
        "%mkdir data\n",
        "%cd /content/data/\n",
        "!wget http://files.fast.ai/data/examples/dogscats.tgz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BTobJ9vTE37J"
      },
      "source": [
        "!tar -zxvf dogscats.tgz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XsMtmnCbCdhd"
      },
      "source": [
        "%ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5OW01WzCdhu"
      },
      "source": [
        "%cd dogscats/\n",
        "%ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WetB1w-dZ1sr"
      },
      "source": [
        "The structure of the sub-folders inside the folder dogscats will be important for what follows\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JP2DNyUqZ7bN"
      },
      "source": [
        "```bash\n",
        ".\n",
        "├── test1 # contains 12500 images of cats and dogs\n",
        "├── train\n",
        "|   └── cats # contains 11500 images of cats\n",
        "|   └── dogs # contains 11500 images of dogs\n",
        "├── valid\n",
        "|   └── cats # contains 1000 images of cats\n",
        "|   └── dogs # contains 1000 images of dogs\n",
        "├── sample\n",
        "|   └── train\n",
        "|       └── cats # contains 8 images of cats\n",
        "|       └── dogs # contains 8 images of dogs    \n",
        "|   └── valid \n",
        "|       └── cats # contains 4 images of cats\n",
        "|       └── dogs # contains 4 images of dogs    \n",
        "├── models # empty folder\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TFx4dvOUaX-_"
      },
      "source": [
        "You see that the 12 500 images of the test are in the test1 sub-folder; the dataset of 25 000 labelled images has been split into a train set and a validation set.\n",
        "\n",
        "The sub-folder sample is here only to make sure the code is running properly on a very small dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mkj3DZjpCdha"
      },
      "source": [
        "## Data processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iasXk_FKCdhy"
      },
      "source": [
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MJRnJgGOCdh4"
      },
      "source": [
        "data_dir = '/content/data/dogscats'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U3lE0cvyCdh8"
      },
      "source": [
        "```datasets``` is a class of the ```torchvision``` package (see [torchvision.datasets](http://pytorch.org/docs/master/torchvision/datasets.html)) and deals with data loading. It integrates a multi-threaded loader that fetches images from the disk, groups them in mini-batches and serves them \n",
        "continously to the GPU right after each _forward_/_backward_ pass through the network.\n",
        "\n",
        "Images needs a bit of preparation before passing them throught the network. They need to have all the same size $224\\times 224 \\times 3$ plus some extra formatting done below by the normalize transform (explained later)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8t4vokNrF19p"
      },
      "source": [
        "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
        "\n",
        "imagenet_format = transforms.Compose([\n",
        "                transforms.CenterCrop(224),\n",
        "                transforms.ToTensor(),\n",
        "                normalize,\n",
        "            ])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8LMReVECdh-"
      },
      "source": [
        "dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), imagenet_format)\n",
        "         for x in ['train', 'valid']}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pMh7kjEBCdiC"
      },
      "source": [
        "os.path.join(data_dir,'train')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VNyS0TyeKwdO"
      },
      "source": [
        "Interactive help on jupyter notebook thanks to `?`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WX5pkMX4CdiF"
      },
      "source": [
        "?datasets.ImageFolder"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYmIGNEjKwdR"
      },
      "source": [
        "We see that `datasets.ImageFolder` has attributes: classes, class_to_idx, imgs.\n",
        "\n",
        "Let see what they are?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m9ifn_R7CdiH"
      },
      "source": [
        "dsets['train'].classes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JzCEqtD3anli"
      },
      "source": [
        "The name of the classes are directly inferred from the structure of the folder:\n",
        "\n",
        "\n",
        "```bash\n",
        "├── train\n",
        "|   └── cats\n",
        "|   └── dogs\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TB6sTwFuCdiK"
      },
      "source": [
        "dsets['train'].class_to_idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmmiRQVva5iv"
      },
      "source": [
        "\n",
        "The label 0 will correspond to cats and 1 to dogs.\n",
        "\n",
        "Below, you see that the first 5 imgs are pairs (location_of_the_image, label):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y9ECrD2ACdiO"
      },
      "source": [
        "dsets['train'].imgs[:5]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WefhjZb2CdiQ"
      },
      "source": [
        "dset_sizes = {x: len(dsets[x]) for x in ['train', 'valid']}\n",
        "dset_sizes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5GGfdBjLbKEd"
      },
      "source": [
        "\n",
        "As expected we have 23 000 images in the training set and 2 000 in the validation set.\n",
        "\n",
        "Below, we store the classes in the variable dset_classes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SCLV1YgaCdiT"
      },
      "source": [
        "dset_classes = dsets['train'].classes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zy52-XhACdiX"
      },
      "source": [
        "The ```torchvision``` packages allows complex pre-processing/transforms of the input data (_e.g._ normalization, cropping, flipping, jittering). A sequence of transforms can be grouped in a pipeline with the help of the ```torchvision.transforms.Compose``` function, see [torchvision.transforms](http://pytorch.org/docs/master/torchvision/transforms.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iphL57PhKwdh"
      },
      "source": [
        "The magic help `?` allows you to retrieve function you defined and forgot!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X_ARYwraCdiY"
      },
      "source": [
        "?imagenet_format"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WX9-DhJRbc78"
      },
      "source": [
        "Where is this normalization coming from?\n",
        "\n",
        "As explained in the [PyTorch doc](https://pytorch.org/docs/stable/torchvision/models.html), you will use a pretrained model. All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape `(3 x H x W)`, where `H` and `W` are expected to be at least `224`. The images have to be loaded in to a range of `[0, 1]` and then normalized using `mean = [0.485, 0.456, 0.406]` and `std = [0.229, 0.224, 0.225]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tWnJQiWgGP_R"
      },
      "source": [
        "loader_train = torch.utils.data.DataLoader(dsets['train'], batch_size=64, shuffle=True, num_workers=6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iern_6GNCdie"
      },
      "source": [
        "?torch.utils.data.DataLoader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wN1BKHfDCdig"
      },
      "source": [
        "loader_valid = torch.utils.data.DataLoader(dsets['valid'], batch_size=5, shuffle=False, num_workers=6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpoBZb-HbtUE"
      },
      "source": [
        "\n",
        "Try to understand what the following cell is doing?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z4Be7lSLCdik"
      },
      "source": [
        "count = 1\n",
        "for data in loader_valid:\n",
        "    print(count, end=',')\n",
        "    if count == 1:\n",
        "        inputs_try,labels_try = data\n",
        "    count +=1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KLgH--ImNi0G"
      },
      "source": [
        "len(loader_valid.dataset.imgs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3BvxQqfzCdiq"
      },
      "source": [
        "labels_try"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MLNsfqc8Cdis"
      },
      "source": [
        "inputs_try.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-rK_8Urob0X7"
      },
      "source": [
        "Got it: the validation dataset contains 2 000 images, hence this is 400 batches of size 5. labels_try contains the labels of the first batch and inputs_try the images of the first batch.\n",
        "\n",
        "What is an image for your computer?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BQtvLJhqb6aJ"
      },
      "source": [
        "inputs_try[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vsaL21ouKwd9"
      },
      "source": [
        "A small function to display images:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DreJeLsCb87J"
      },
      "source": [
        "A 3-channel RGB image is of shape (3 x H x W). Note that entries can be negative because of the normalization.\n",
        "\n",
        "A small function to display images:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "346yl-gbcLLm"
      },
      "source": [
        "def imshow(inp, title=None):\n",
        "#   Imshow for Tensor.\n",
        "    inp = inp.numpy().transpose((1, 2, 0))\n",
        "    mean = np.array([0.485, 0.456, 0.406])\n",
        "    std = np.array([0.229, 0.224, 0.225])\n",
        "    inp = np.clip(std * inp + mean, 0,1)\n",
        "    plt.imshow(inp)\n",
        "    if title is not None:\n",
        "        plt.title(title)\n",
        "    plt.pause(0.001)  # pause a bit so that plots are updated"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJbW-QzGCdiv"
      },
      "source": [
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs_try)\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2HvFNWJICdiz"
      },
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(loader_train))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iUrTGlTpCdi5"
      },
      "source": [
        "# Get a batch of validation data\n",
        "inputs, classes = next(iter(loader_valid))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOvkYiROCdi7"
      },
      "source": [
        "## Creating VGG Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BO5LAG4bCdi7"
      },
      "source": [
        "The torchvision module comes with a zoo of popular CNN architectures which are already trained on [ImageNet](http://www.image-net.org/) (1.2M training images). When called the first time, if ```pretrained=True``` the model is fetched over the internet and downloaded to ```~/.torch/models```.\n",
        "For next calls, the model will be directly read from there."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K9PsHjXgCdi9"
      },
      "source": [
        "model_vgg = models.vgg16(pretrained=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VBzGe2mlCdi-"
      },
      "source": [
        "We will first use VGG Model without any modification. In order to interpret the results, we need to import the 1000 ImageNet categories, available at: [https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json](https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qhvs0ki_Cdi_"
      },
      "source": [
        "!wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hppljqo5CdjC"
      },
      "source": [
        "import json\n",
        "\n",
        "fpath = '/content/data/imagenet_class_index.json'\n",
        "\n",
        "with open(fpath) as f:\n",
        "    class_dict = json.load(f)\n",
        "dic_imagenet = [class_dict[str(i)][1] for i in range(len(class_dict))]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QATvAFILCdjF"
      },
      "source": [
        "dic_imagenet[:4]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p6QQhwruCdjI"
      },
      "source": [
        "inputs_try , labels_try = inputs_try.to(device), labels_try.to(device)\n",
        "\n",
        "model_vgg = model_vgg.to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "epMB0UF9CdjM"
      },
      "source": [
        "outputs_try = model_vgg(inputs_try)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dOlx7YcPCdjO"
      },
      "source": [
        "outputs_try"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OrweFfK9k0TW"
      },
      "source": [
        "outputs_try.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nk538XVpHc0y"
      },
      "source": [
        "To translate the outputs of the network into 'probabilities', we pass it through a [Softmax function](https://en.wikipedia.org/wiki/Softmax_function)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCIxHN2QCdjT"
      },
      "source": [
        "m_softm = nn.Softmax(dim=1)\n",
        "probs = m_softm(outputs_try)\n",
        "vals_try,preds_try = torch.max(probs,dim=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pNuZhT3WcUBC"
      },
      "source": [
        "Let's check that we obtain a probability!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGTVkH2pk0Tb"
      },
      "source": [
        "torch.sum(probs,1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CvGbb2bQCdjZ"
      },
      "source": [
        "vals_try"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0TUdNKCJCdjc"
      },
      "source": [
        "print([dic_imagenet[i] for i in preds_try.data])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jtzuHob9Cdjf"
      },
      "source": [
        "out = torchvision.utils.make_grid(inputs_try.data.cpu())\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try.data.cpu()])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3Bktt5PCdjh"
      },
      "source": [
        "### Modifying the last layer and setting the gradient false to all layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L8kr3-tjCdji"
      },
      "source": [
        "print(model_vgg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXoMW73MCdjl"
      },
      "source": [
        "We'll learn about what these different blocks do later in the course. For now, it's enough to know that:\n",
        "\n",
        "- Convolution layers are for finding small to medium size patterns in images -- analyzing the images locally\n",
        "- Dense (fully connected) layers are for combining patterns across an image -- analyzing the images globally\n",
        "- Pooling layers downsample -- in order to reduce image size and to improve invariance of learned features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hA2f5FRuCdjm"
      },
      "source": [
        "![vgg16](https://raw.githubusercontent.com/abursuc/practicals-tsp-hti/master/images/vgg16.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SK6lfAzfCdjn"
      },
      "source": [
        "In this practical example, our goal is to use the already trained model and just change the number of output classes. To this end we replace the last ```nn.Linear``` layer trained for 1000 classes to ones with 2 classes. In order to freeze the weights of the other layers during training, we set the field ```required_grad=False```. In this manner no gradient will be computed for them during backprop and hence no update in the weights. Only the weights for the 2 class layer will be updated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rQwRKKC-Cdjo"
      },
      "source": [
        "for param in model_vgg.parameters():\n",
        "    param.requires_grad = False\n",
        "model_vgg.classifier._modules['6'] = nn.Linear(4096, 2)\n",
        "model_vgg.classifier._modules['7'] = torch.nn.LogSoftmax(dim = 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qAr5tj2NKwfF"
      },
      "source": [
        "PyTorch documentation for [LogSoftmax](https://pytorch.org/docs/stable/nn.html#logsoftmax)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jJ3OenJpCdjp"
      },
      "source": [
        "print(model_vgg.classifier)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ju4VWgFrchUb"
      },
      "source": [
        "\n",
        "We load the model on GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ITZFX2MCdju"
      },
      "source": [
        "model_vgg = model_vgg.to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fU9vWWT2CdkN"
      },
      "source": [
        "## Training fully connected module"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqp2u3IXCdkO"
      },
      "source": [
        "### Creating loss function and optimizer\n",
        "\n",
        "PyTorch documentation for [NLLLoss](https://pytorch.org/docs/stable/nn.html#nllloss) and the [torch.optim module](https://pytorch.org/docs/stable/optim.html#module-torch.optim)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oP1F4yb8CdkO"
      },
      "source": [
        "criterion = nn.NLLLoss()\n",
        "lr = 0.001\n",
        "optimizer_vgg = torch.optim.SGD(model_vgg.classifier[6].parameters(),lr = lr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tenuLj67CdkS"
      },
      "source": [
        "### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7nNAUibjCdkS"
      },
      "source": [
        "\n",
        "def train_model(model,dataloader,size,epochs=1,optimizer=None):\n",
        "    model.train()\n",
        "    \n",
        "    for epoch in range(epochs):\n",
        "        running_loss = 0.0\n",
        "        running_corrects = 0\n",
        "        for inputs,classes in dataloader:\n",
        "            inputs = inputs.to(device)\n",
        "            classes = classes.to(device)\n",
        "            outputs = model(inputs)\n",
        "            loss = criterion(outputs,classes)\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "            running_loss += loss.data.item()\n",
        "            running_corrects += torch.sum(preds == classes.data)\n",
        "        epoch_loss = running_loss / size\n",
        "        epoch_acc = running_corrects.data.item() / size\n",
        "        print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Jts2jK1CdkV",
        "scrolled": false
      },
      "source": [
        "%%time\n",
        "train_model(model_vgg,loader_train,size=dset_sizes['train'],epochs=1,optimizer=optimizer_vgg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xkMYIX21Hc1k"
      },
      "source": [
        "def test_model(model,dataloader,size):\n",
        "    model.eval()\n",
        "    predictions = np.zeros(size)\n",
        "    all_classes = np.zeros(size)\n",
        "    all_proba = np.zeros((size,2))\n",
        "    i = 0\n",
        "    running_loss = 0.0\n",
        "    running_corrects = 0\n",
        "    for inputs,classes in dataloader:\n",
        "        inputs = inputs.to(device)\n",
        "        classes = classes.to(device)\n",
        "        outputs = model(inputs)\n",
        "        loss = criterion(outputs,classes)           \n",
        "        _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "        running_loss += loss.data.item()\n",
        "        running_corrects += torch.sum(preds == classes.data)\n",
        "        predictions[i:i+len(classes)] = preds.to('cpu').numpy()\n",
        "        all_classes[i:i+len(classes)] = classes.to('cpu').numpy()\n",
        "        all_proba[i:i+len(classes),:] = outputs.data.to('cpu').numpy()\n",
        "        i += len(classes)\n",
        "    epoch_loss = running_loss / size\n",
        "    epoch_acc = running_corrects.data.item() / size\n",
        "    print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))\n",
        "    return predictions, all_proba, all_classes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y3JjYayKCdkW"
      },
      "source": [
        "predictions, all_proba, all_classes = test_model(model_vgg,loader_valid,size=dset_sizes['valid'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cn6cIwZNCdkY"
      },
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(loader_valid))\n",
        "\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ghkbm7ByCdke"
      },
      "source": [
        "outputs = model_vgg(inputs[:n_images].to(device))\n",
        "print(torch.exp(outputs))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F1qdUtvjCdkg"
      },
      "source": [
        "classes[:n_images]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0n0hLxTHc1w"
      },
      "source": [
        "## Speeding up the learning by precomputing features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUe_3nDMecsO"
      },
      "source": [
        "Here we are wasting a lot of time computing over and over the same quantities. Indeed, the first part of the VGG model (called features and made of convolutional layers) is frozen and never updated. Hence, we can precompute for each image in the dataset, the output of these convolutional layers as these outputs will always be the same during your training process.\n",
        "\n",
        "This is what is done below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "39AALjfgHc1y"
      },
      "source": [
        "x_try = model_vgg.features(inputs_try)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bwjaBq8lHc11"
      },
      "source": [
        "x_try.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tqn0vTmJehy7"
      },
      "source": [
        "You see that the features computed for an image is of shape `512x7x7` (above we have a batch corresponding to 5 images)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mZegyLN3Hc13"
      },
      "source": [
        "def preconvfeat(dataloader):\n",
        "    conv_features = []\n",
        "    labels_list = []\n",
        "    for data in dataloader:\n",
        "        inputs,labels = data\n",
        "        inputs = inputs.to(device)\n",
        "        labels = labels.to(device)\n",
        "        \n",
        "        x = model_vgg.features(inputs)\n",
        "        conv_features.extend(x.data.cpu().numpy())\n",
        "        labels_list.extend(labels.data.cpu().numpy())\n",
        "    conv_features = np.concatenate([[feat] for feat in conv_features])\n",
        "    return (conv_features,labels_list)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UQiycgsqHc1_"
      },
      "source": [
        "%%time\n",
        "conv_feat_train,labels_train = preconvfeat(loader_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imSeW6-7Hc2D"
      },
      "source": [
        "conv_feat_train.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q3e5bjsuHc2F"
      },
      "source": [
        "%%time\n",
        "conv_feat_valid,labels_valid = preconvfeat(loader_valid)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0fpZByOHc2H"
      },
      "source": [
        "### Creating a new data generator\n",
        "\n",
        "We will not load images anymore, so we need to build our own data loader."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hpx-uU90Hc2H"
      },
      "source": [
        "\n",
        "dtype=torch.float\n",
        "datasetfeat_train = [[torch.from_numpy(f).type(dtype),torch.tensor(l).type(torch.long)] for (f,l) in zip(conv_feat_train,labels_train)]\n",
        "datasetfeat_train = [(inputs.reshape(-1), classes) for [inputs,classes] in datasetfeat_train]\n",
        "loaderfeat_train = torch.utils.data.DataLoader(datasetfeat_train, batch_size=128, shuffle=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x3EvEnzue0mN"
      },
      "source": [
        "%%time\n",
        "train_model(model_vgg.classifier,dataloader=loaderfeat_train,size=dset_sizes['train'],epochs=50,optimizer=optimizer_vgg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lK875TWzHc2J"
      },
      "source": [
        "\n",
        "datasetfeat_valid = [[torch.from_numpy(f).type(dtype),torch.tensor(l).type(torch.long)] for (f,l) in zip(conv_feat_valid,labels_valid)]\n",
        "datasetfeat_valid = [(inputs.reshape(-1), classes) for [inputs,classes] in datasetfeat_valid]\n",
        "loaderfeat_valid = torch.utils.data.DataLoader(datasetfeat_valid, batch_size=128, shuffle=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5N6rbg-Hc2T"
      },
      "source": [
        "predictions, all_proba, all_classes = test_model(model_vgg.classifier,dataloader=loaderfeat_valid,size=dset_sizes['valid'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nlBOnslWHc2Y"
      },
      "source": [
        "## 4. Viewing model prediction (qualitative analysis)\n",
        "\n",
        "The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.\n",
        "\n",
        "With our first model we should try to overfit before we start worrying about how to handle that - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques after the 2 weeks break...)\n",
        "\n",
        "\n",
        "As well as looking at the overall metrics, it's also a good idea to look at examples of each of:\n",
        "\n",
        "   1. A few correct labels at random\n",
        "   2. A few incorrect labels at random\n",
        "   3. The most correct labels of each class (ie those with highest probability that are correct)\n",
        "   4. The most incorrect labels of each class (ie those with highest probability that are incorrect)\n",
        "   5. The most uncertain labels (ie those with probability closest to 0.5).\n",
        "\n",
        "In general, these are particularly useful for debugging problems in the model. Since our model is very simple, there may not be too much to learn at this stage..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EJr5EysXHc2a"
      },
      "source": [
        "# Number of images to view for each visualization task\n",
        "n_view = 8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_hokobPHc2i"
      },
      "source": [
        "correct = np.where(predictions==all_classes)[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4Y8kRv-Hc2o"
      },
      "source": [
        "len(correct)/dset_sizes['valid']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "72LZk1rXHc2v"
      },
      "source": [
        "from numpy.random import random, permutation\n",
        "idx = permutation(correct)[:n_view]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wrel6SmsHc2z"
      },
      "source": [
        "idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rQ2pQ7p_Hc22"
      },
      "source": [
        "loader_correct = torch.utils.data.DataLoader([dsets['valid'][x] for x in idx],batch_size = n_view,shuffle=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j-Phdh-tHc25"
      },
      "source": [
        "for data in loader_correct:\n",
        "    inputs_cor,labels_cor = data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3PIV_a2XHc27"
      },
      "source": [
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs_cor)\n",
        "\n",
        "imshow(out, title=[l.item() for l in labels_cor])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p8hWSmjJHc2-"
      },
      "source": [
        "from IPython.display import Image, display\n",
        "for x in idx:\n",
        "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hXx3REbpHc3B"
      },
      "source": [
        "incorrect = np.where(predictions!=all_classes)[0]\n",
        "for x in permutation(incorrect)[:n_view]:\n",
        "    #print(dsets['valid'].imgs[x][1])\n",
        "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kh7QdEVZHc3E"
      },
      "source": [
        "#3. The images we most confident were cats, and are actually cats\n",
        "correct_cats = np.where((predictions==0) & (predictions==all_classes))[0]\n",
        "most_correct_cats = np.argsort(all_proba[correct_cats,1])[:n_view]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ocYE1fvHHc3I"
      },
      "source": [
        "for x in most_correct_cats:\n",
        "    display(Image(filename=dsets['valid'].imgs[correct_cats[x]][0], retina=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SmZkwPEsHc3O"
      },
      "source": [
        "#3. The images we most confident were dogs, and are actually dogs\n",
        "correct_dogs = np.where((predictions==1) & (predictions==all_classes))[0]\n",
        "most_correct_dogs = np.argsort(all_proba[correct_dogs,0])[:n_view]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8jUtIj4Hc3R"
      },
      "source": [
        "for x in most_correct_dogs:\n",
        "    display(Image(filename=dsets['valid'].imgs[correct_dogs[x]][0], retina=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hjwIo3gbsGMx"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "What did we do in the end? A simple logistic regression! If the connection is unclear, we'll explain it on a much simpler example in the next course.\n",
        "\n",
        "We probably killed a fly with a sledge hammer!\n",
        "\n",
        "In our case, the sledge hammer is VGG pretrained on Imagenet, a dataset containing a lot of pictures of cats and dogs. Indeed, we saw that without modification the network was able to predict dog and cat breeds. Hence it is not very surprising that the features computed by VGG are very accurate for our classification task. In the end, we need to learn only the parameters of the last linear layer, i.e. `8194` parameters (do not forget the bias `2\\times 4096+2`). Indeed, this can be done on CPU without any problem.\n",
        "\n",
        "Nevertheless, this example is still instructive as it shows all the necessary steps in a deep learning project. Here we did not struggle with the learning process of a deep network, but we did all the preliminary engineering tasks: dowloading a dataset, setting up the environment to use a GPU, preparing the data, computing the features with a pretrained VGG, saving them on your drive so that you can use them for a later experiment... These steps are essential in any deep learning project and a necessary requirement before having fun playing with network architectures and understanding the learning process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_NhP9XMbuK1A"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hMAwUx7QCdjv"
      },
      "source": [
        "### Saving preconvoluted features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4DQHnbD2H6BJ"
      },
      "source": [
        "!pip install -U bcolz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3-APvzkACdkD"
      },
      "source": [
        "import bcolz\n",
        "\n",
        "def save_array(fname, arr):\n",
        "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
        "    c.flush()\n",
        "def load_array(fname):\n",
        "    return bcolz.open(fname)[:]\n",
        "\n",
        "\n",
        "%mkdir /content/data/dogscats/vgg16\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AWadFbvtCdkF"
      },
      "source": [
        "save_array(os.path.join(data_dir,'vgg16','feat_train.bc'),conv_feat_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_train.bc'),labels_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','feat_val.bc'),conv_feat_valid)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_val.bc'),labels_valid)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pyjlrH4wCdkH"
      },
      "source": [
        "### Uploading Precomputed features\n",
        "\n",
        "This section will allow you to store the precomputed features on your Google drive for later use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "27gRV8KiJw6H"
      },
      "source": [
        "%cd /content/data/dogscats/\n",
        "!zip -r vgg16 vgg16/*"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "de4y59inJx_p"
      },
      "source": [
        "!pip install -U -q PyDrive\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "# 1. Authenticate and create the PyDrive client.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JIaZyCpcJ1Hd"
      },
      "source": [
        "upload = drive.CreateFile({'title': 'vgg16_drive.zip'})\n",
        "upload.SetContentFile('vgg16.zip')\n",
        "upload.Upload()\n",
        "print('Uploaded file with ID {}'.format(upload.get('id')))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}