PyTorch_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch for Fast and Easy Machine Learning\n",
    "\n",
    "https://github.com/compdyn/pytorch_tutorial\n",
    "\n",
    "**Matthew West**  \n",
    "mwest@illinois.edu  \n",
    "http://lagrange.mechse.illinois.edu/  \n",
    "https://github.com/mwest1066\n",
    "\n",
    "_\"Hands-On with CSE\" tutorial series_\n",
    "\n",
    "April 26, 2019\n",
    "\n",
    "**Description:** PyTorch allows you to easily train and run machine learning models. It uses standard Python methods for writing code, so it's both simple and powerful. We will cover the core automatic differentiation capabilities of PyTorch, training deep neural networks, using GPUs and parallel computation, managing training and test data, saving and loading models, and visualizing training progress. We will assume a good knowledge of Python and NumPy, and basic knowledge of machine learning with neural nets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List of resources\n",
    "\n",
    "- PyTorch tutorials: https://pytorch.org/tutorials/\n",
    "- PyTorch manual: https://pytorch.org/docs/stable/index.html\n",
    "- PyTorch paper: https://openreview.net/forum?id=BJJsrmfCZ\n",
    "- Calculus on computational graphs: http://colah.github.io/posts/2015-08-Backprop/\n",
    "- Einstein summation in PyTorch: https://rockt.github.io/2018/04/30/einsum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch citation\n",
    "\n",
    "```\n",
    "@inproceedings{paszke2017automatic,\n",
    "  title={Automatic differentiation in PyTorch},\n",
    "  author={Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam},\n",
    "  booktitle={NIPS-W},\n",
    "  year={2017},\n",
    "  url={https://openreview.net/forum?id=BJJsrmfCZ},\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is PyTorch?\n",
    "\n",
    "- Like NumPy with automatic differentiation via dynamic computation graphs\n",
    "- Transparent ability to compute on GPUs and in parallel\n",
    "- Library of neural net functions and constructors\n",
    "- Library of gradient-based optimizers\n",
    "- Various other useful functions (e.g., data management)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing PyTorch\n",
    "\n",
    "- Use Anaconda\n",
    "- https://pytorch.org/get-started/locally/\n",
    "- At the moment we need Python 3.6 to support TensorBoard\n",
    "```\n",
    "conda create --name torch python=3.6\n",
    "conda activate torch\n",
    "conda install scipy matplotlib jupyter\n",
    "conda install pytorch torchvision -c pytorch\n",
    "pip install tensorflow # for TensorBoard\n",
    "pip install tensorboardX # bridge from PyTorch\n",
    "jupyter-notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import random, datetime\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch is similar to NumPy\n",
    "\n",
    "PyTorch works in immediate mode, which is different to the default TensorFlow model.\n",
    "\n",
    "PyTorch is very picky about datatypes, and defaults to single precision (NumPy defaults to double)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1.0, 2.0, 3.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify datatypes\n",
    "\n",
    "Use `dtype=torch.float64` and `.double()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.double()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1, 2, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.log(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1, 2, 3], dtype=torch.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.log(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Annoying differences from NumPy\n",
    "\n",
    "`np.sum(x, axis=1)` versus `torch.sum(x, dim=1)`\n",
    "\n",
    "`np.clip()` versus `torch.clamp()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log([1, 2, 3]).clip(0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.log(x).clip(0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.log(x).clamp(0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert to/from NumPy arrays\n",
    "\n",
    "`.numpy()` and `torch.from_numpy()`\n",
    "\n",
    "Memory is shared!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "y = torch.from_numpy(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[0] = 7\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `torch.autograd`: Computing derivatives\n",
    "\n",
    "PyTorch constructs the computation graph as you do operations (dynamic graphs) unlike TensorFlow (static graphs)\n",
    "\n",
    "Using the computation graph, the chain rule (back propagation) can compute derivatives\n",
    "\n",
    "Derivatives are available in the leaf nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(5.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = torch.tensor(3.0, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = x * y**2\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'x.grad = {x.grad}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$z = x y^2$\n",
    "\n",
    "$\\frac{\\partial z}{\\partial y} = 2 x y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2*x*y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Control what we differentiate with respect to\n",
    "\n",
    "`requires_grad=True`\n",
    "\n",
    "`with no_grad():`\n",
    "\n",
    "`.detach()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "y = x*x\n",
    "print(f'y.requires_grad = {y.requires_grad}')\n",
    "z = x*y\n",
    "z.backward()\n",
    "print(f'dz/dx = {x.grad}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "y = x*x\n",
    "y = y.detach() # can't say y.requires_grad = False\n",
    "print(f'y.requires_grad = {y.requires_grad}')\n",
    "z = x*y\n",
    "z.backward()\n",
    "print(f'dz/dx = {x.grad}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "with torch.no_grad():\n",
    "    y = x*x\n",
    "print(f'y.requires_grad = {y.requires_grad}')\n",
    "z = x*y\n",
    "z.backward()\n",
    "print(f'dz/dx = {x.grad}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computation graphs are not trees\n",
    "\n",
    "Re-using a parameter in multiple places makes the graph not be a tree. It's a DAG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "y = 3*x\n",
    "z = x**2\n",
    "w = y + z + x\n",
    "w.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\frac{\\partial w}{\\partial x} = \\frac{\\partial}{\\partial x}(3x + x^2 + x) = 3 + 2x + 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3 + 2*x + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The computation graph is destroyed by `backward()`\n",
    "\n",
    "To retain it for more differentiation, use `backward(retain_graph=True)`\n",
    "\n",
    "A common use case is multiple outputs with a shared subgraph\n",
    "\n",
    "Don't forget to free the graph on the last call to prevent memory leaks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(3.0, requires_grad=True)\n",
    "y = x**2\n",
    "z1 = 3*y\n",
    "z2 = 4*y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z1.backward() # (retain_graph=True)\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z2.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Derivatives of scalars with respect to tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)\n",
    "y = (x**2).sum()\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Don't do in-place modifications to tensors\n",
    "\n",
    "But it's fine to do `x = 4 * x`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[1] = x[2] + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = (x**2).sum()\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using a GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create variables on the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.ones(4, device=\"cuda\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transfer variables to/from the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1, 2, 3])\n",
    "y = torch.tensor([4, 5, 6])\n",
    "x = x.to('cuda')\n",
    "y = y.to('cuda')\n",
    "z = x * y\n",
    "z = z.to('cpu')\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `torch.optim`: All the common gradient-based optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return x**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xvec = np.linspace(-2, 2, 100)\n",
    "fvec = f(xvec)\n",
    "plt.plot(xvec, fvec);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([2.0], requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = torch.optim.SGD([x], lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_history = [x.detach().numpy().copy()]\n",
    "for i in range(10):\n",
    "    print(f'##########')\n",
    "    print(f'i = {i}')\n",
    "    print(f'initial x = {x}')\n",
    "    opt.zero_grad()\n",
    "    z = f(x)\n",
    "    print(f'z = {z}')\n",
    "    z.backward()\n",
    "    print(f'x.grad = {x.grad}')\n",
    "    opt.step()\n",
    "    print(f'updated x = {x}')\n",
    "    x_history.append(x.detach().numpy().copy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xvec = np.linspace(-2, 2, 100)\n",
    "fvec = f(xvec)\n",
    "plt.plot(xvec, fvec)\n",
    "plt.plot(x_history, np.zeros(len(x_history)), 'o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `torch.nn`: Easy neural-network construction\n",
    "\n",
    "Convention: the first index is the data-item index, so N images each of shape 128 x 128 will be in a tensor of shape N x 128 x 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.linspace(0, 2*np.pi, 100)\n",
    "y = torch.sin(x)\n",
    "plt.plot(x.numpy(), y.numpy(), 'r.');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyModel, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(1, 8)\n",
    "        self.fc2 = torch.nn.Linear(8, 8)\n",
    "        self.fc3 = torch.nn.Linear(8, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.nn.functional.relu(self.fc1(x))\n",
    "        x = torch.nn.functional.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MyModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yp = model(x.reshape(100,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x.numpy(), y.numpy(), 'r.');\n",
    "plt.plot(x.numpy(), yp.detach().numpy());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fc3.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fc3.bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fc3.bias.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in model.parameters():\n",
    "    print(p.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = torch.optim.Adam(model.parameters(), lr=0.01)\n",
    "loss_history = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    opt.zero_grad()\n",
    "    yp = model(x.reshape(100,1))\n",
    "    loss = torch.nn.MSELoss()(yp, y.reshape(100,1))\n",
    "    loss_history.append(loss.item())\n",
    "    loss.backward()\n",
    "    opt.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(loss_history);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x.numpy(), y.numpy(), 'r.');\n",
    "plt.plot(x.numpy(), yp.detach().numpy());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and restoring models\n",
    "\n",
    "Save and load the parameters, not the full models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), 'model_file.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MyModel()\n",
    "model.load_state_dict(torch.load('model_file.pkl'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing run-time output with TensorBoard\n",
    "\n",
    "Run TensorBoard with `tensorboard --logdir training_logs`\n",
    "\n",
    "Navigate to http://localhost:6006/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorboardX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = datetime.datetime.now().isoformat(timespec='seconds')\n",
    "writer = tensorboardX.SummaryWriter(f'training_logs/train-{timestamp}')\n",
    "for i in range(100):\n",
    "    loss = np.exp(-i/50)\n",
    "    writer.add_scalar('loss', loss, i)\n",
    "    # writer.add_text('parameter value = 65', i)\n",
    "    \n",
    "    # plt.ioff()\n",
    "    # fig = matplotlib.gcf()\n",
    "    # writer.add_figure('output_visualization', fig, i)\n",
    "    \n",
    "    # writer.add_distribution('input_samples', xvec, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `torch.utils.data`: Managing train/test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.utils.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([0,1,2,3,4,5,6,7,8,9])\n",
    "y = torch.tensor([10,11,12,13,14,15,16,17,18,19])\n",
    "d = torch.utils.data.TensorDataset(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_sample, y_sample) = d[random.sample(range(len(d)), 4)]\n",
    "print(f'x_sample = {x_sample}')\n",
    "print(f'y_sample = {y_sample}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "Choose a tutorial: https://pytorch.org/tutorials/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}