1 Vectors, matrices and tensors.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thinking in tensors, writing in PyTorch\n",
    "\n",
    "A hands-on course by [Piotr Migdał](https://p.migdal.pl) (2019-2022).\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stared/thinking-in-tensors-writing-in-pytorch/blob/master/1%20Vectors%2C%20matrices%20and%20tensors.ipynb)\n",
    "\n",
    "\n",
    "## Notebook 1: Vectors, matrices and tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Linear algebra is the language of deep learning... and quantum mechanics.\n",
    "\n",
    "Note: in physics and engineering, tensor is not any array. There is a one-two-many rule: \n",
    "\n",
    "* 0: scalar\n",
    "* 1: vector\n",
    "* 2: matrix\n",
    "* 3 and above: n-dimensional tensor\n",
    "\n",
    "In theory, tensors can be of an arbitrarily high dimension. In deep learning, they rare exceed 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scalar\n",
    "\n",
    "Scalar is \"just a number\". Real-world examples of a scalar are: temperature, pressure, price of an apple in a given shop, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tensor(42.)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.dim()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2 * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Food for thought\n",
    "\n",
    "> The scalar fallacy is the false but pervasive assumption that real-world things (hotels, sandwiches, people, mutual funds, chemo drugs, whatever) have some single-dimension ordering of \"goodness\".\n",
    "\n",
    "> When you project a multi-dimensional space down to one dimension, you are involving a lot of context and preferences in the act of projecting. - [rlucas on HN](https://news.ycombinator.com/item?id=8132525)\n",
    "\n",
    "See also: [Scalar fallacy](http://observationalepidemiology.blogspot.com/2011/01/scalar-fallacy.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector\n",
    "\n",
    "Vector is an ordered list of numbers, such as `[-5., 2., 0.]`.\n",
    "\n",
    "In physics and mechanical engineering, not everything is a vector:\n",
    "\n",
    "> it is not generally true that any three numbers form a vector. It is true only if, when we rotate the coordinate system, the components of the vector transform among themselves in the correct way. - [II 02: Differential Calculus of Vector Fields](http://www.feynmanlectures.caltech.edu/II_02.html) from [The Feynman Lectures on Physics](http://www.feynmanlectures.caltech.edu/)\n",
    "\n",
    "* position\n",
    "* velocity\n",
    "* electric field\n",
    "* spatial gradient of a scalar field ($\\nabla T$)\n",
    "\n",
    "\n",
    "In deep learning we are more... relaxed. Usually vectors are abstract, \n",
    "\n",
    "\n",
    "* feature vector after a ImageNet-trained vector\n",
    "* a word representation in (see: [king - man + woman is queen; but why?](https://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html))\n",
    "* user and product vectors in [Factorization Machines](https://www.reddit.com/r/MachineLearning/comments/65d3lt/r_factorization_machines_2010_a_classic_paper_in/) and related recommendation systems\n",
    "\n",
    "\n",
    "$$\\vec{v} = \\left[ v_1, v_2, \\ldots, v_n \\right]$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = tensor([1.5, -0.5, 3.0])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.dim()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector arithmetics\n",
    "\n",
    "We can multiply vectors by a scalar: \n",
    "\n",
    "$$a \\vec{v} = \\left[a v_1, a v_2, \\ldots, a v_n \\right]$$\n",
    "\n",
    "Or, provided that two vectors have the same length, add and subtract vectors to each other:\n",
    "\n",
    "$$\\vec{v} + \\vec{u}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2 * v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = tensor([1., 0., 1.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v + u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector length\n",
    "\n",
    "\n",
    "$$|\\vec{v}| = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2} = \\sqrt{\\sum_{i=1}^n v_i^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.pow(2).sum().sqrt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.pow(v, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.pow(2).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or to normalize a vector\n",
    "v / v.norm()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix\n",
    "\n",
    "[![Matrix transform - xkcd](https://imgs.xkcd.com/comics/matrix_transform.png)](https://xkcd.com/184/)\n",
    "\n",
    "Typical operations:\n",
    "\n",
    "* rotations\n",
    "* next step in a stochastic process\n",
    "* unitary operations and projections in quantum mechanics (these use complex numbers)\n",
    "* scalar products\n",
    "* [Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix) of a scalar (i.e. second order derivatives of a scalar with respect to a vector)\n",
    "* channel mixing (e.g. `RGB` to gray-scale and R-G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = tensor([[1., 2.], [3., 4.]])\n",
    "M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M.matmul(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor([1., 0.]).matmul(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for Python 3.5+\n",
    "M @ M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M * M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor([1., 2.]).matmul(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M.det()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or we can use Singular Value Decomposition, the key step of Principal Component Analysis\n",
    "M.svd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Matrix factorization visualized](https://p.migdal.pl/matrix-decomposition-viz/) by Piotr Migdał (work in progress):\n",
    "\n",
    "![](imgs/matrix_factorization_city_temperature.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor\n",
    "\n",
    "\n",
    "Tensor is a generalization of vectors and matrices for more dimensions.\n",
    "\n",
    "In physics and engineering they have more properties, as in:\n",
    "\n",
    "\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/thumb/f/fe/StressEnergyTensor_contravariant.svg/250px-StressEnergyTensor_contravariant.svg.png)\n",
    "\n",
    "[Electromagnetic tensor](https://en.wikipedia.org/wiki/Electromagnetic_tensor) from [Introduction to the mathematics of general relativity - Wikipedia](https://en.wikipedia.org/wiki/Introduction_to_the_mathematics_of_general_relativity), see also: [Tensor](https://en.wikipedia.org/wiki/Tensor).\n",
    "\n",
    "In deep learning, there are any arrays.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further reading\n",
    "\n",
    "Deep learning is just a lot of array operations, which [can be done even in Excel](http://www.deepexcel.net/). While the network was an April Fool's Day joke, it was a very educational one.\n",
    "\n",
    "For an introduction to linear algebra, I recommend [immersive linear algebra](http://immersivemath.com/ila/index.html) by J. Ström, K. Åström, and T. Akenine-Möller (from my [Interactive Machine Learning, Deep Learning and Statistics websites\n",
    "](http://p.migdal.pl/interactive-machine-learning-list/) collection).\n",
    "\n",
    "I made some points about  In particular, I recommend:\n",
    "\n",
    "* [Tensor diagrams](https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-write-a-review-simple-diagrams-of-convoluted-neural-networks-6418a63f9281).\n",
    "* [Einsum is All you Need - Einstein Summation in Deep Learning](https://rockt.github.io/2018/04/30/einsum) by Tim Rocktäschel.\n",
    "* [Matrices as Tensor Network Diagrams](https://www.math3ma.com/blog/matrices-as-tensor-network-diagrams) by [Tai-Danae Bradley](https://twitter.com/math3ma):\n",
    "\n",
    "![Scalar, vector, matrix, tensor - a drawing by Tai-Danae Bradley](https://uploads-ssl.webflow.com/5b1d427ae0c922e912eda447/5cd99a73f8ce4494ad86852e_arraychart.jpg)\n",
    "\n",
    "Beware, that PyTorch can be tricky with the tensor dimension order:\n",
    "\n",
    "* [Inconsistent dimension ordering for 1D networks - NCL vs NLC vs LNC]()\n",
    "* [Named tensors](http://nlp.seas.harvard.edu/NamedTensor) and [Named tensors (part 2)](http://nlp.seas.harvard.edu/NamedTensor2) by Alexander Rush - a proposal of type-checking tensor dimensions\n",
    "\n",
    "> Is it only me, or does \"Theano tensor dimension order\" sound like some secret convent? - [Piotr Migdał's tweet](https://twitter.com/pmigdal/status/961344490500952070)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch-macos-m1-gpu",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "032715a8d60757555a2623145b534a689e8b5634c5398e8bbec1014c4a8ede12"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}