4_video_analytics/02_cv_for_videos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computer Vision Applications for Videos\n",
    "\n",
    "Through out the course we have discussed various CV applications both in the traditional CV space as well as the deep learning space. This notebook discusses some relatively new applications in these spaces for videos. \n",
    "\n",
    "This notebook presents an overview of some intereseting applications to analyze or process videos using computer vision techniques. Along with videos, this notebook also presents some worthy real time applications of CV. These are not neccessarily for video data but on a stream of images. \n",
    "\n",
    "Let's look at these applications and try to understand the concepts behind them on a high level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CV for Videos\n",
    "\n",
    "We have seen how we can analyse videos using image processing techniques, by processing one frame at a time. But in those techniques we do not leverage the relationship which consecutive frames of the video have.\n",
    "\n",
    "As we know that a video is a sequence of a large number of frames, thus these video frames are temporally related to each other. This relation manifests in both in the form of temporal redundancies as well as logical temporal relationship between the frames.\n",
    "\n",
    "**Example**: a video of a car on a highway will have frames showing the car progressing in a particular direction as we progress forward in the frame sequence.\n",
    "\n",
    "**This temporal nature of the video makes them a good candidate input for sequential models such as RNN and LSTMs for sequential analysis.**\n",
    "\n",
    "Let us see some examples of cv applied to video data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Video Classification\n",
    "\n",
    "**Problem Statement**\n",
    "\n",
    "\n",
    "Video classification is simply the categorization of a video into defined categories. This problem is similar to image classification but only with increased complexity due to the fact a video is a collection of several hundreds or thousands of frames and also that these images might be different from each other although belonging to a single category.\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"http://blog.qure.ai/assets/images/actionrec/fronststroke.gif\" width = 200px/>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"http://blog.qure.ai/assets/images/actionrec/breaststroke.gif\" width = 200px/>\n",
    "    <figcaption style = \"text-align:center\">a) Forward Stroke b) Breast Stroke. Ref: \n",
    "        <a href=\"http://blog.qure.ai/notes/deep-learning-for-videos-action-recognition-review\">Deep Learning for Videos: A 2018 Guide to Action Recognition</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "**Challenges**:\n",
    "\n",
    "- Prediction flickering\n",
    "- Loss context over large sequence of frames\n",
    "- Lack of a proper benchmark\n",
    "\n",
    "**Datasets**:\n",
    "\n",
    "- [Sports 1M Dataset](https://github.com/gtoderici/sports-1m-dataset/): provides links to 1,133,158 YouTube videos annotated with 487 sports labels. The annotations were generated automatically using the the YouTube Topics, which has a public API.\n",
    "- [UCF 101 - Action Recognition Dataset](https://www.crcv.ucf.edu/data/UCF101.php): UCF101 is an action recognition data set of realistic action videos, collected from YouTube, having 101 action categories. \n",
    "\n",
    "**Approaches**:\n",
    "\n",
    "- Classifying each frame of a video using a 2D CNN and then averaging the predictions.\n",
    "- Using a 3D CNN to perform convolution on a set of frames.\n",
    "- Extracting features for each frame using 2D CNN in a time distributed manner and then feeding the features to an RNN\n",
    "\n",
    "**Interesting Reads**:\n",
    "\n",
    "- https://www.pyimagesearch.com/2019/07/15/video-classification-with-keras-and-deep-learning/\n",
    "- https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42455.pdf\n",
    "- http://blog.qure.ai/notes/deep-learning-for-videos-action-recognition-review\n",
    "- https://www.youtube.com/watch?v=PrPv9GV1jPI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video Captioning/Description\n",
    "\n",
    "**Problem Statement**\n",
    "\n",
    "Video captioning/description is an interesting intersection area of CV and NLP. It refers to generation of a sequence of text which describes a sequence of frames. This problem brings with it a very challenging aspect of learning the spatio-temporal dependencies in video frames and learn their mapping with the sequential representation of text.\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://github.com/AdrianHsu/S2VT-seq2seq-video-captioning-attention/raw/master/util/s2vt-1.png\" width = 900px/>\n",
    "    <figcaption style = \"text-align:center\">Image showing a Seq2Seq model for video captioning. Ref: \n",
    "        <a href=\"https://github.com/AdrianHsu/S2VT-seq2seq-video-captioning-attention\">S2VT</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "**Challenges**:\n",
    "\n",
    "- There is a huge latent space to be covered to map visual features and linguistic features and to learn a mapping from one to another.\n",
    "- Huge amount of dataset is need to train CNNs and RNNs together for this task\n",
    "\n",
    "**Datasets**:\n",
    "\n",
    "- [YouCook II](http://youcook2.eecs.umich.edu/)\n",
    "\n",
    "**Aproaches**\n",
    "\n",
    "- Seq2Seq Models\n",
    "- Attention Models\n",
    "\n",
    "**Interesting Reads**\n",
    "\n",
    "- https://github.com/DataScienceNigeria/AI-powered-by-Google-s-VideoBERT-\n",
    "- https://github.com/AdrianHsu/S2VT-seq2seq-video-captioning-attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visual Q&A\n",
    "\n",
    "**Problem Statement**\n",
    "\n",
    "Visual Question and Answer is a very interesting and complex domain and can be seen as an extension of video captioning where the models learns an even more generalized representation of visual data and is able to answer relevant questions based on the visual data.\n",
    "\n",
    "This application holds a lot of potential for developing inclusive solutions for people with visual disability. \n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://visualqa.org/static/img/vqa_examples.jpg\" width = 700px/>\n",
    "    <figcaption style = \"text-align:center\">Visual Q&A Examples. Ref: \n",
    "        <a href=\"https://visualqa.org/\">VisualQA.Org</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "**Challenges**:\n",
    "\n",
    "- There is a huge latent space to be covered to map visual features and linguistic features and to learn a mapping from one to another.\n",
    "- Huge amount of dataset is need to train CNNs and RNNs together for this task\n",
    "\n",
    "**Datasets**:\n",
    "\n",
    "- [VQA](https://visualqa.org/download.html)\n",
    "- [VizWiz](http://up.csail.mit.edu/other-pubs/vizwiz.pdf)\n",
    "\n",
    "**Aproaches**\n",
    "\n",
    "- CNN + LSTM\n",
    "- Heirarchical Co-Attention Model\n",
    "\n",
    "**Interesting Reads**\n",
    "\n",
    "- https://vqa.cloudcv.org/\n",
    "- https://visualqa.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RealTime CV\n",
    "\n",
    "This section of this notebook is dedicated to models/applications of computer vision which have been deployed for real time use. These applications are examples of meticulous use of deep learning or CV models with added customization for **fast inference** and **low memory footprint**. \n",
    "\n",
    "Libraries such as TensorflowJS, Mediapipe, OpenCV have been very helpful for developer to deploy their solutions. \n",
    "\n",
    "Let's have a look at few such applications and discuss the optimizations made to make them usable real time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenPose\n",
    "\n",
    "OpenPose is a tensorflow implementation of the pose estimation model. It comes with different backbones (cmu, dsconv, mobilenet) to choose from for varied deployment conditions.\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://github.com/infocom-tpo/tf-openpose/raw/master/etcs/openpose_tx2_mobilenet3.gif\" width = 300px/>\n",
    "    <figcaption style = \"text-align:center\">OpenPose Mobilenet Variant on Jetson TK2. Ref: \n",
    "        <a href=\"https://github.com/infocom-tpo/tf-openpose\">tf-openpose</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "This and other real time implementations of pose estimation models are being used for varied applications.\n",
    "\n",
    "**Applications**:\n",
    "\n",
    "- workout posture monitoring and correction\n",
    "- posture monitoring in factories for worker safety\n",
    "- using pose estimation to map human pose to an animated character\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://github.com/yemount/pose-animator/raw/master/resources/gifs/avatar-new-bezier-1.gif\" width = 400px/>\n",
    "    <figcaption style = \"text-align:center\">Pose Animator. Ref: \n",
    "        <a href=\"https://github.com/yemount/pose-animator\">Pose Animator</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AR Cut Paste\n",
    "\n",
    "This one is an awesome Augmented Reality application developed by [Cyril Diagne](https://github.com/cyrildiagne) to paste object pictures to your laptop in real time.\n",
    "\n",
    "It is a very cool application which demonstrates the power of CV + AR for real life use cases. The application uses an [HTTP wrapper](https://github.com/cyrildiagne/basnet-http) on [BASNET Model](https://github.com/NathanUA/BASNet) for salient object segementation.\n",
    "\n",
    "Do check out his repository and tweet for more details on how he build this application over a weekend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">5/10 - From books to presentations in 10s with AR+ML<br><br>Code: <a href=\"https://t.co/17tkhLw35T\">https://t.co/17tkhLw35T</a><br><br>Art: Molecule Shoes by <a href=\"https://twitter.com/francisbitonti?ref_src=twsrc%5Etfw\">@francisbitonti</a>, 2014<br>Book: Coder Le Monde (Editions Hyx 2018 <a href=\"https://twitter.com/CentrePompidou?ref_src=twsrc%5Etfw\">@centrepompidou</a>)<br>Technical Insights: ↓<a href=\"https://twitter.com/hashtag/ML?src=hash&amp;ref_src=twsrc%5Etfw\">#ML</a> <a href=\"https://twitter.com/hashtag/AR?src=hash&amp;ref_src=twsrc%5Etfw\">#AR</a> <a href=\"https://twitter.com/hashtag/AI?src=hash&amp;ref_src=twsrc%5Etfw\">#AI</a> <a href=\"https://twitter.com/hashtag/AIUX?src=hash&amp;ref_src=twsrc%5Etfw\">#AIUX</a> <a href=\"https://twitter.com/hashtag/Keynote?src=hash&amp;ref_src=twsrc%5Etfw\">#Keynote</a> <a href=\"https://twitter.com/hashtag/Powerpoint?src=hash&amp;ref_src=twsrc%5Etfw\">#Powerpoint</a> <a href=\"https://twitter.com/hashtag/GoogleDocs?src=hash&amp;ref_src=twsrc%5Etfw\">#GoogleDocs</a> <a href=\"https://t.co/O2sZmXCuJf\">pic.twitter.com/O2sZmXCuJf</a></p>&mdash; Cyril Diagne (@cyrildiagne) <a href=\"https://twitter.com/cyrildiagne/status/1259441154606669824?ref_src=twsrc%5Etfw\">May 10, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
      ],
      "text/plain": [
       "<__main__.Tweet at 0x26c74fc5da0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Tweet(object):\n",
    "    def __init__(self, embed_str=None):\n",
    "        self.embed_str = embed_str\n",
    "\n",
    "    def _repr_html_(self):\n",
    "        return self.embed_str\n",
    "    \n",
    "Tweet(\"\"\"<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">5/10 - From books to presentations in 10s with AR+ML<br><br>Code: <a href=\"https://t.co/17tkhLw35T\">https://t.co/17tkhLw35T</a><br><br>Art: Molecule Shoes by <a href=\"https://twitter.com/francisbitonti?ref_src=twsrc%5Etfw\">@francisbitonti</a>, 2014<br>Book: Coder Le Monde (Editions Hyx 2018 <a href=\"https://twitter.com/CentrePompidou?ref_src=twsrc%5Etfw\">@centrepompidou</a>)<br>Technical Insights: ↓<a href=\"https://twitter.com/hashtag/ML?src=hash&amp;ref_src=twsrc%5Etfw\">#ML</a> <a href=\"https://twitter.com/hashtag/AR?src=hash&amp;ref_src=twsrc%5Etfw\">#AR</a> <a href=\"https://twitter.com/hashtag/AI?src=hash&amp;ref_src=twsrc%5Etfw\">#AI</a> <a href=\"https://twitter.com/hashtag/AIUX?src=hash&amp;ref_src=twsrc%5Etfw\">#AIUX</a> <a href=\"https://twitter.com/hashtag/Keynote?src=hash&amp;ref_src=twsrc%5Etfw\">#Keynote</a> <a href=\"https://twitter.com/hashtag/Powerpoint?src=hash&amp;ref_src=twsrc%5Etfw\">#Powerpoint</a> <a href=\"https://twitter.com/hashtag/GoogleDocs?src=hash&amp;ref_src=twsrc%5Etfw\">#GoogleDocs</a> <a href=\"https://t.co/O2sZmXCuJf\">pic.twitter.com/O2sZmXCuJf</a></p>&mdash; Cyril Diagne (@cyrildiagne) <a href=\"https://twitter.com/cyrildiagne/status/1259441154606669824?ref_src=twsrc%5Etfw\">May 10, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}