4-Modules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# < [Optimization](3-Optimization.ipynb) | Modules | [CNN & LSTM](5-CNN-LSTM.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is this notebook about?\n",
    "\n",
    "In this notebook, we will learning about PyTorch modules and the great functionalities they provide. Later on, we'll create a small a multilayer perceptron to perform image classification on MNIST."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Colab only!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# execute only if you're using Google Colab\n",
    "!wget -q https://raw.githubusercontent.com/ahug/amld-pytorch-workshop/master/binder/requirements.txt -O requirements.txt\n",
    "!pip install -qr requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "print(\"Torch version:\", torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In PyTorch, there are many predefined layer like Convolutions, RNN, Pooling, Linear, etc.\n",
    "\n",
    "These functions are wrapped in **modules** and inherit from the **torch.nn.Module** base class.\n",
    "\n",
    "When designing a custom model in PyTorch, you should follow this strategy and derive your class from **torch.nn.Module**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.nn.Module.__doc__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modules are doing a lot of \"magic\" under the hood.\n",
    "\n",
    "- It registers all the parameters of your model.\n",
    "- It simplifies the saving/loading of your model.\n",
    "- It provides helper functions to reset/freeze/update the gradients.\n",
    "- It provides helper functions to put all parameters on a device (GPU)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a torch.nn.Parameter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Parameter is a Tensor with `requires_grad` to `True` by default, and which is automatically added to the list of parameters when used within a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the documentation ([torch.nn.Paramter](https://pytorch.org/docs/stable/_modules/torch/nn/parameter.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.nn.Parameter.__doc__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod = nn.Conv1d(10, 2, 3)\n",
    "print(mod.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Very simple example of a module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A module has to implement two functions:\n",
    "\n",
    "- the `__init__` function, where you define all the layers that have learnable parameters. In the `__init__` function, you are just specifying each layer and not how it is connected to others, so it does not need to be in order of execution. Since your model's submodules and parameters are instantiated in the `__init__` function, PyTorch knows that they exist and registers them.  \n",
    "Also, don't forget to always call the `super()` method.  \n",
    "\n",
    "\n",
    "- the `forward` function, which is the method that defines what has to be executed during the forward pass and especially how the layers are connected. This is where you call the layers that you defined inside the `__init__` function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A simple module\n",
    "class MySuperSimpleModule(nn.Module):\n",
    "    def __init__(self, input_size, num_classes):\n",
    "        super(MySuperSimpleModule, self).__init__()  # Mandatory call to super\n",
    "        self.linear = nn.Linear(input_size, num_classes)  # Define one Linear layer\n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = self.linear(x)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the print function to list a model's submodules and parameters defined inside `init`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MySuperSimpleModule(input_size=20, num_classes=5)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use **`model.parameters()`** to get the list of parameters of your model automatically inferred by PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, p in model.named_parameters():  # Here we use a sligtly different version of the parameters() function\n",
    "    print(name, \":\\n\", p)                 # which also returns the parameter name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple network for image classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![We need to go depper](figures/deeper.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create a more complicated model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement a simple multilayer perceptron with two hidden layers and the following structure:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://raw.githubusercontent.com/ledell/sldm4-h2o/master/mlp_network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Input-size: *input_size*\n",
    "- 1st hidden layer: 75\n",
    "- 2nd hidden layer: 50\n",
    "- Output layer: *num_classes*\n",
    "\n",
    "Additionally, we use `ReLU`s as activation functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need some PyTorch NN modules - Find them in the [PyTorch doc](https://pytorch.org/docs/master/nn.html) (especially nn.Linear)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import Parameter\n",
    "import torch.nn.functional as F  # provides some helper functions like Relu's, Sigmoids, Tanh, etc.\n",
    "\n",
    "\n",
    "class MyMultilayerPerceptron(nn.Module):\n",
    "    def __init__(self, input_size, num_classes):\n",
    "        super(MyMultilayerPerceptron, self).__init__()\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.num_classes = num_classes\n",
    "        \n",
    "        self.linear_1 = nn.Linear(input_size, 75)\n",
    "        self.linear_2 = # <YOUR CODE>\n",
    "        self.linear_3 = nn.Linear(50, num_classes)\n",
    "        \n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = F.relu(self.linear_1(x))\n",
    "        out = # <YOUR CODE>\n",
    "        out = # <YOUR CODE>\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print your network's parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MyMultilayerPerceptron(784, 10)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feed an input to your network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "x = torch.rand(16, 784)  # the first dimension is reserved for the 'batch_size'\n",
    "out = model(x)  # equivalent to model.forward(x)\n",
    "out[0, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the functions to train a model follow a similar pattern in PyTorch.\n",
    "In most of the cases in consists of the following steps:\n",
    "- Loop over data (in batches)\n",
    "- Forward pass\n",
    "- Zero gradients!\n",
    "- Backward pass\n",
    "- Parameter update (Optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, num_epochs, data_loader, device):\n",
    "    model = model.to(device)\n",
    "    \n",
    "    # Define the Loss function and Optimizer that you want to use\n",
    "    criterion = nn.CrossEntropyLoss()  \n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)  # NOTE: model.parameters()\n",
    "    \n",
    "    # Outter training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        # Inner training loop\n",
    "        cum_loss = 0\n",
    "        for (inputs, labels) in data_loader:\n",
    "            # Prepare inputs and labels for processing by the model (e.g. reshape, move to device, ...)\n",
    "            inputs = inputs.to(device)\n",
    "            labels = labels.to(device)\n",
    "            \n",
    "            # original shape is [batch_size, 28, 28] because it's an image of size 28x28\n",
    "            inputs = inputs.view(-1, 28*28)\n",
    "\n",
    "            # Do Forward -> Loss Computation -> Backward -> Optimization\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(inputs)\n",
    "            loss = criterion(outputs, labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            cum_loss += loss.item()\n",
    "        print(\"Epoch %d, Loss=%.4f\" % (epoch+1, cum_loss/len(train_loader)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "- we can use the `.to` function on the model directly. Indeed, since PyTorch knows all the model parameters, it can put all the parameters on the correct device.\n",
    "- we use `model.parameters()` to get all the parameters of the model and we can instantiate an optimizer that will optimize these parameters `torch.optim.SGD(model.parameters())`.\n",
    "- to apply the forward function of the module, we write `model(input)`. In most cases, `model.forward(inputs)` would also work, but there is a slight difference : PyTorch allows you to register hook functions for a model that are automatically called when you do a forward pass on your model. Using `model(input)` will call these hooks and then call the forward function, while using `model.forward(inputs)` will just silently ignore them.\n",
    "\n",
    "Do you feel the power of Modules ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch comes with a lot of predefined loss functions :\n",
    "- L1Loss\n",
    "- MSELoss\n",
    "- CrossEntropyLoss\n",
    "- NLLLoss\n",
    "- PoissonNLLLoss\n",
    "- KLDivLoss\n",
    "- BCELoss\n",
    "- MarginRankingLoss\n",
    "- HingeEmbeddingLoss\n",
    "- MultiLabelMarginLoss\n",
    "- CosineEmbeddingLoss\n",
    "- TripletMarginLoss\n",
    "- ...\n",
    "\n",
    "Check out the [PyTorch Documentation](https://pytorch.org/docs/master/nn.html#loss-functions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's train our model on the MNIST digit classification task\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![MNIST](figures/mnist.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we have to load the training and test images. MNIST is a widely used dataset, therefore the torchvision package provides simple functionalities to load images from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.datasets as datasets\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "batch_size = 64\n",
    "\n",
    "# MNIST Dataset (Images and Labels)\n",
    "train_dataset = datasets.MNIST(root='./data', train=True, transform=transforms.ToTensor(), download=True)\n",
    "test_dataset = datasets.MNIST(root='./data', train=False, transform=transforms.ToTensor())\n",
    "\n",
    "# Dataset Loader (Input Batcher)\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In PyTorch, `Dataset` and `Dataloaders` are classes that can help to quickly define how to access and iterate over your data. This is specially interesting when your data is distributed over several files (for instance, if you have several images in some directory structure)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Call the actual training function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model = MyMultilayerPerceptron(input_size=784, num_classes=10)\n",
    "num_epochs = 5\n",
    "\n",
    "train(model, num_epochs, train_loader, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How can we now assess the model's performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function loops over another `data_loader` (usually containing test/validation data) and computes the model's accuracy on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(model, data_loader, device):\n",
    "    with torch.no_grad(): # during model evaluation, we don't need the autograd mechanism (speeds things up)\n",
    "        correct = 0\n",
    "        total = 0\n",
    "        for inputs, labels in data_loader:\n",
    "            inputs = inputs.to(device)     \n",
    "            inputs = inputs.view(-1, 28*28)\n",
    "            \n",
    "            outputs = model(inputs)\n",
    "            _, predicted = outputs.max(1)\n",
    "            \n",
    "            correct += (predicted.cpu() == labels).sum().item()\n",
    "            total += labels.size(0)\n",
    "            \n",
    "    acc = correct / total\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy(model, test_loader, device)  # look at: accuracy(model, train_loader, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We get an accuracy of ~97.9%, can we do better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How can we now store our trained model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model, \"my_model.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_model_loaded = torch.load(\"my_model.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.linear_3.bias, my_model_loaded.linear_3.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This intro to modules used [this medium post](https://medium.com/deeplearningbrasilia/deep-learning-introduction-to-pytorch-5bd39421c84) as a resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Don't forget to download the notebook, otherwise your changes may be lost!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Download the notebook](figures/notebook-download.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# < [Optimization](3-Optimization.ipynb) | Modules | [CNN & LSTM](5-CNN-LSTM.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}