deepfake-detection/Deepfake_Detection_Substra_Example_Notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deepfake-Detection_Substra_Example_Notebook",
      "provenance": [],
      "collapsed_sections": [
        "Aw4mJuJB9w-t"
      ],
      "toc_visible": true,
      "machine_shape": "hm",
      "authorship_tag": "ABX9TyNHyOq0t4mnOZ6d1PzJ7Jwf",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SubstraFoundation/substra-examples/blob/master/deepfake-detection/Deepfake_Detection_Substra_Example_Notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-4DF18EyM3rb",
        "colab_type": "text"
      },
      "source": [
        "In this Notebook, you can test the example's assets and potentially run ML tasks on a public distant VM.\n",
        "On Google Colab, you can see and modify the assets with the \"Files\" button on the left."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXe6Grsv5j03",
        "colab_type": "text"
      },
      "source": [
        "# A Substra example for Deepfakes Detection\n",
        "\n",
        "*This example is a Substra implementation of a deepfake detector.\n",
        "The Algo is based on the [inference demo Kaggle notebook](https://www.kaggle.com/humananalog/inference-demo) and use the [DFDC dataset from Kaggle](https://www.kaggle.com/c/deepfake-detection-challenge).\n",
        "The structure of the example is inspired from [Substra's Titanic Example](https://github.com/SubstraFoundation/substra/blob/master/examples/titanic/)*\n",
        "\n",
        "## Prerequisites\n",
        "\n",
        "In order to run this example, you'll need to:\n",
        "\n",
        "* use Python 3\n",
        "* have [Docker](https://www.docker.com/) installed\n",
        "* [install the `substra` cli](https://github.com/SubstraFoundation/substra#install) (supported version: 0.6.0)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FABNvfiR5s1F",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! pip3 install substra==0.6.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mONGmkG-5sVp",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "* [install the `substratools` library](https://github.com/substrafoundation/substra-tools) (supported version: 0.6.0)\n",
        "* [pull the `substra-tools` docker images](https://github.com/substrafoundation/substra-tools#pull-from-private-docker-registry)\n",
        "* have access to a Substra installation ([configure your host to a public node ip](https://doc.substra.ai/getting_started/installation/local_install_skaffold.html#network) or [install Substra on your machine](https://doc.substra.ai/getting_started/installation/local_install_skaffold.html))\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OXNWTXfLdNEr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#replace this ip by the ip of a distant VM running a substra node\n",
        "public_node_ip = \"127.0.0.1\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Sj64EGcd2bG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! echo \"{public_node_ip} substra-backend.node-1.com substra-frontend.node-1.com substra-backend.node-2.com substra-frontend.node-2.com\" | sudo tee -a /etc/hosts\n",
        "# Check if it's ok\n",
        "! curl substra-backend.node-1.com/readiness"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WiNgyy4odKzR",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "* create a substra profile to define the substra network to target, for instance:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "njy5vYAvWoEE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! substra config --profile node-1 http://substra-backend.node-1.com\n",
        "! substra login --profile node-1 --username node-1 --password 'p@$swr0d44'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFoGatph6My1",
        "colab_type": "text"
      },
      "source": [
        "* checkout this repository"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NCaLsa5LOLTu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "outputId": "c59b4699-b3c7-4486-9722-691417710c41"
      },
      "source": [
        "%cd /content\n",
        "! git clone https://github.com/SubstraFoundation/substra-examples/\n",
        "%cd /content/substra-examples/deepfake-detection/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "Cloning into 'substra-examples'...\n",
            "remote: Enumerating objects: 26, done.\u001b[K\n",
            "remote: Counting objects: 100% (26/26), done.\u001b[K\n",
            "remote: Compressing objects: 100% (25/25), done.\u001b[K\n",
            "remote: Total 285 (delta 6), reused 13 (delta 1), pack-reused 259\u001b[K\n",
            "Receiving objects: 100% (285/285), 170.32 MiB | 4.30 MiB/s, done.\n",
            "Resolving deltas: 100% (136/136), done.\n",
            "/content/substra-examples\n",
            "Checking out files: 100% (21/21), done.\n",
            "Branch 'deepfake' set up to track remote branch 'deepfake' from 'origin'.\n",
            "Switched to a new branch 'deepfake'\n",
            "/content/substra-examples/deepfake-detection\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_MwLxwQt6Vwb",
        "colab_type": "text"
      },
      "source": [
        "All commands in this example are run from the `deepfake-detection` folder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJcc8m2o6Z1K",
        "colab_type": "text"
      },
      "source": [
        "## Data preparation\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYYBlMH39eRP",
        "colab_type": "text"
      },
      "source": [
        "### Download the data\n",
        "\n",
        "The first step will be to download the data from the [Kaggle challenge source](https://www.kaggle.com/c/deepfake-detection-challenge/data)\n",
        "\n",
        "* Sign-up or login to [Kaggle](https://www.kaggle.com/) and accept the [competitions rules](https://www.kaggle.com/c/deepfake-detection-challenge/rules).\n",
        "* Download the data samples (4Go) manually (`Download All` at the bottom of the [data section](https://www.kaggle.com/c/deepfake-detection-challenge/data)), or install & configure the [Kaggle API](https://github.com/Kaggle/kaggle-api) \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aBddHdrJ7RPi",
        "colab_type": "text"
      },
      "source": [
        "#### Using Kaggle API"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5eTpK-fiM-WB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! pip install --upgrade --force-reinstall kaggle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vEyxQEIdN8YJ",
        "colab_type": "text"
      },
      "source": [
        "To use the Kaggle API, go to the 'Account' tab of your user profile (https://www.kaggle.com/<username\\>/account) and select 'Create API Token'. This will trigger the download of kaggle.json, a file containing your API credentials.  \n",
        "Upload your kaggle.json file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rSgtzV0oNZX8",
        "colab_type": "code",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "5a6da133-af44-4844-9bd0-dc03243e03b6"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "files.upload()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-784ed8c6-0ad8-48e0-bf17-899b9f6edeb6\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-784ed8c6-0ad8-48e0-bf17-899b9f6edeb6\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving kaggle.json to kaggle.json\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'kaggle.json': b'{\"username\":\"wallrider\",\"key\":\"df97e4c7538d8e5191f6d1f961580d1c\"}'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FZhslhCfOHzn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%mkdir ~/.kaggle\n",
        "%cp kaggle.json ~/.kaggle/\n",
        "! chmod 600 ~/.kaggle/kaggle.json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qs-PafWg6tu6",
        "colab_type": "text"
      },
      "source": [
        "and execute the following command:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XNzMcgKl6ZQY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "ac32878a-249b-48c7-9f83-0aea0d318433"
      },
      "source": [
        "! kaggle competitions download -c deepfake-detection-challenge"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading deepfake-detection-challenge.zip to /content/substra-examples/deepfake-detection\n",
            "100% 4.13G/4.13G [01:00<00:00, 111MB/s]\n",
            "100% 4.13G/4.13G [01:00<00:00, 73.7MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aw4mJuJB9w-t",
        "colab_type": "text"
      },
      "source": [
        "#### If you downloaded the data samples manually"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VH-raBQw97Hd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "# upload your deepfake-detection-challenge.zip file\n",
        "files.upload()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vkTW802h6fdS",
        "colab_type": "text"
      },
      "source": [
        "#### Extract the zip file and copy-paste the 'train_sample_videos' folder in the data/DFDC folder of the example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gBWZNKZ86kQA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%mkdir -p data/DFDC\n",
        "! unzip -q deepfake-detection-challenge.zip 'train_sample_videos/*' -d data/DFDC\n",
        "%rm deepfake-detection-challenge.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UGlu5tg773Z",
        "colab_type": "text"
      },
      "source": [
        "### Generate data samples\n",
        "\n",
        "The second step will be to generate train and test data samples from the [Kaggle challenge source](https://www.kaggle.com/c/deepfake-detection-challenge/data).\n",
        "To generate the data samples, run:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WG94dHhT7_ic",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "4cf893ab-ab5f-47b9-fc78-40e0679cdfa5"
      },
      "source": [
        "%cd /content/substra-examples/deepfake-detection/\n",
        "#! pip install -r scripts/requirements.txt --user\n",
        "# requirements are already satisfied in Colab, except for substratools\n",
        "! pip install substratools==0.6.0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/substra-examples/deepfake-detection\n",
            "Collecting substratools==0.6.0\n",
            "  Downloading https://files.pythonhosted.org/packages/46/90/74983a05a05259321b51516fc5404abfa169c8f39a090b53e1788b3e5cd7/substratools-0.6.0-py3-none-any.whl\n",
            "Installing collected packages: substratools\n",
            "Successfully installed substratools-0.6.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SIU4As3a8Kuk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "outputId": "e35ba5a3-bcce-4847-bfd0-d51993458c24"
      },
      "source": [
        "%cd /content/substra-examples/deepfake-detection/\n",
        "! python scripts/generate_data_samples.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/substra-examples/deepfake-detection\n",
            "Loading DFDC data from data/DFDC\n",
            "# of files in data folder: 401\n",
            "Files with extension `mp4`: 400\n",
            "Files with extension `json`: 1\n",
            "JSON file: metadata.json\n",
            "# of files in metadata: 400, # of videos: 400\n",
            "Spliting data in train/test sets...\n",
            "# of train data points:  320\n",
            "# of test data points:  80\n",
            "Data will be generated in :  /content/substra-examples/deepfake-detection/assets\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99RRo50a-vub",
        "colab_type": "text"
      },
      "source": [
        "This will create two sub-folders in the `assets` folder:\n",
        "\n",
        "* `train_data_samples` contains train data features (paths of the videos) and labels as numpy array files\n",
        "* `test_data_samples` contains test data features (paths of the videos) and labels as numpy array files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IHKRGN9I-wfS",
        "colab_type": "text"
      },
      "source": [
        "## Writing the objective and data manager\n",
        "\n",
        "Both objective and data manager will need a proper markdown description, you can check them out in their respective\n",
        "folders. Notice that the data manager's description includes a formal description of the data structure.\n",
        "\n",
        "Notice also that the `metrics.py` and `opener.py` module both rely on classes imported from the `substratools` module.\n",
        "These classes provide a simple yet rigid structure that will make algorithms pretty easy to write.\n",
        "\n",
        "## Writing a simple algorithm\n",
        "\n",
        "You'll find under `assets/algo_inference` an implementation of the `inference` model from the [inference demo Kaggle notebook](https://www.kaggle.com/humananalog/inference-demo). Like the metrics and opener scripts, it relies on a\n",
        "class imported from `substratools` that greatly simplifies the writing process. You'll notice that it handles not only\n",
        "the train and predict tasks but also a lot of data preprocessing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znr8Pg9_-zqT",
        "colab_type": "text"
      },
      "source": [
        "## Testing our assets\n",
        "\n",
        "### Using asset command line interfaces\n",
        "\n",
        "You can first test each assets with the `substratools` CLI, by running specific ML tasks in your local Python environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y4HM-NbK-3NY",
        "colab_type": "text"
      },
      "source": [
        "#### Training task"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cExE-mFS_ACM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 406
        },
        "outputId": "d4c88281-a3d2-4463-da09-b351529e362c"
      },
      "source": [
        "#for a quicker test, you can change --data-samples-path to a specific data sample, (e.g. assets/train_data_samples/data_sample_0)\n",
        "\n",
        "#train your model with the train_data\n",
        "! python assets/algo_inference/algo.py train \\\n",
        "  --debug \\\n",
        "  --opener-path assets/dataset/opener.py \\\n",
        "  --data-samples-path assets/train_data_samples \\\n",
        "  --output-model-path assets/model/model \\\n",
        "  --log-path assets/logs/train.log"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "substratools.utils - Module 'opener' loaded from path 'assets/dataset/opener.py'\n",
            "substratools.opener - loading X from '['assets/train_data_samples/data_sample_2', 'assets/train_data_samples/data_sample_49', 'assets/train_data_samples/data_sample_48', 'assets/train_data_samples/data_sample_13', 'assets/train_data_samples/data_sample_79', 'assets/train_data_samples/data_sample_74', 'assets/train_data_samples/data_sample_71', 'assets/train_data_samples/data_sample_52', 'assets/train_data_samples/data_sample_26', 'assets/train_data_samples/data_sample_3', 'assets/train_data_samples/data_sample_31', 'assets/train_data_samples/data_sample_46', 'assets/train_data_samples/data_sample_62', 'assets/train_data_samples/data_sample_6', 'assets/train_data_samples/data_sample_14', 'assets/train_data_samples/data_sample_67', 'assets/train_data_samples/data_sample_45', 'assets/train_data_samples/data_sample_28', 'assets/train_data_samples/data_sample_37', 'assets/train_data_samples/data_sample_59', 'assets/train_data_samples/data_sample_64', 'assets/train_data_samples/data_sample_51', 'assets/train_data_samples/data_sample_78', 'assets/train_data_samples/data_sample_56', 'assets/train_data_samples/data_sample_34', 'assets/train_data_samples/data_sample_0', 'assets/train_data_samples/data_sample_17', 'assets/train_data_samples/data_sample_58', 'assets/train_data_samples/data_sample_22', 'assets/train_data_samples/data_sample_5', 'assets/train_data_samples/data_sample_41', 'assets/train_data_samples/data_sample_63', 'assets/train_data_samples/data_sample_39', 'assets/train_data_samples/data_sample_15', 'assets/train_data_samples/data_sample_24', 'assets/train_data_samples/data_sample_23', 'assets/train_data_samples/data_sample_76', 'assets/train_data_samples/data_sample_53', 'assets/train_data_samples/data_sample_57', 'assets/train_data_samples/data_sample_60', 'assets/train_data_samples/data_sample_18', 'assets/train_data_samples/data_sample_44', 'assets/train_data_samples/data_sample_69', 'assets/train_data_samples/data_sample_21', 'assets/train_data_samples/data_sample_27', 'assets/train_data_samples/data_sample_11', 'assets/train_data_samples/data_sample_47', 'assets/train_data_samples/data_sample_66', 'assets/train_data_samples/data_sample_54', 'assets/train_data_samples/data_sample_72', 'assets/train_data_samples/data_sample_70', 'assets/train_data_samples/data_sample_10', 'assets/train_data_samples/data_sample_29', 'assets/train_data_samples/data_sample_65', 'assets/train_data_samples/data_sample_38', 'assets/train_data_samples/data_sample_30', 'assets/train_data_samples/data_sample_4', 'assets/train_data_samples/data_sample_19', 'assets/train_data_samples/data_sample_8', 'assets/train_data_samples/data_sample_68', 'assets/train_data_samples/data_sample_32', 'assets/train_data_samples/data_sample_16', 'assets/train_data_samples/data_sample_33', 'assets/train_data_samples/data_sample_1', 'assets/train_data_samples/data_sample_40', 'assets/train_data_samples/data_sample_61', 'assets/train_data_samples/data_sample_20', 'assets/train_data_samples/data_sample_43', 'assets/train_data_samples/data_sample_75', 'assets/train_data_samples/data_sample_35', 'assets/train_data_samples/data_sample_12', 'assets/train_data_samples/data_sample_9', 'assets/train_data_samples/data_sample_42', 'assets/train_data_samples/data_sample_50', 'assets/train_data_samples/data_sample_25', 'assets/train_data_samples/data_sample_77', 'assets/train_data_samples/data_sample_7', 'assets/train_data_samples/data_sample_36', 'assets/train_data_samples/data_sample_73', 'assets/train_data_samples/data_sample_55']'\n",
            "Finding features file...\n",
            "substratools.opener - loading y from '['assets/train_data_samples/data_sample_2', 'assets/train_data_samples/data_sample_49', 'assets/train_data_samples/data_sample_48', 'assets/train_data_samples/data_sample_13', 'assets/train_data_samples/data_sample_79', 'assets/train_data_samples/data_sample_74', 'assets/train_data_samples/data_sample_71', 'assets/train_data_samples/data_sample_52', 'assets/train_data_samples/data_sample_26', 'assets/train_data_samples/data_sample_3', 'assets/train_data_samples/data_sample_31', 'assets/train_data_samples/data_sample_46', 'assets/train_data_samples/data_sample_62', 'assets/train_data_samples/data_sample_6', 'assets/train_data_samples/data_sample_14', 'assets/train_data_samples/data_sample_67', 'assets/train_data_samples/data_sample_45', 'assets/train_data_samples/data_sample_28', 'assets/train_data_samples/data_sample_37', 'assets/train_data_samples/data_sample_59', 'assets/train_data_samples/data_sample_64', 'assets/train_data_samples/data_sample_51', 'assets/train_data_samples/data_sample_78', 'assets/train_data_samples/data_sample_56', 'assets/train_data_samples/data_sample_34', 'assets/train_data_samples/data_sample_0', 'assets/train_data_samples/data_sample_17', 'assets/train_data_samples/data_sample_58', 'assets/train_data_samples/data_sample_22', 'assets/train_data_samples/data_sample_5', 'assets/train_data_samples/data_sample_41', 'assets/train_data_samples/data_sample_63', 'assets/train_data_samples/data_sample_39', 'assets/train_data_samples/data_sample_15', 'assets/train_data_samples/data_sample_24', 'assets/train_data_samples/data_sample_23', 'assets/train_data_samples/data_sample_76', 'assets/train_data_samples/data_sample_53', 'assets/train_data_samples/data_sample_57', 'assets/train_data_samples/data_sample_60', 'assets/train_data_samples/data_sample_18', 'assets/train_data_samples/data_sample_44', 'assets/train_data_samples/data_sample_69', 'assets/train_data_samples/data_sample_21', 'assets/train_data_samples/data_sample_27', 'assets/train_data_samples/data_sample_11', 'assets/train_data_samples/data_sample_47', 'assets/train_data_samples/data_sample_66', 'assets/train_data_samples/data_sample_54', 'assets/train_data_samples/data_sample_72', 'assets/train_data_samples/data_sample_70', 'assets/train_data_samples/data_sample_10', 'assets/train_data_samples/data_sample_29', 'assets/train_data_samples/data_sample_65', 'assets/train_data_samples/data_sample_38', 'assets/train_data_samples/data_sample_30', 'assets/train_data_samples/data_sample_4', 'assets/train_data_samples/data_sample_19', 'assets/train_data_samples/data_sample_8', 'assets/train_data_samples/data_sample_68', 'assets/train_data_samples/data_sample_32', 'assets/train_data_samples/data_sample_16', 'assets/train_data_samples/data_sample_33', 'assets/train_data_samples/data_sample_1', 'assets/train_data_samples/data_sample_40', 'assets/train_data_samples/data_sample_61', 'assets/train_data_samples/data_sample_20', 'assets/train_data_samples/data_sample_43', 'assets/train_data_samples/data_sample_75', 'assets/train_data_samples/data_sample_35', 'assets/train_data_samples/data_sample_12', 'assets/train_data_samples/data_sample_9', 'assets/train_data_samples/data_sample_42', 'assets/train_data_samples/data_sample_50', 'assets/train_data_samples/data_sample_25', 'assets/train_data_samples/data_sample_77', 'assets/train_data_samples/data_sample_7', 'assets/train_data_samples/data_sample_36', 'assets/train_data_samples/data_sample_73', 'assets/train_data_samples/data_sample_55']'\n",
            "Finding label file...\n",
            "Loading labels...\n",
            "substratools.algo - launching training task\n",
            "Loading features...\n",
            "Nb of train videos: 320\n",
            "PyTorch version: 1.6.0+cu101\n",
            "CUDA version: 10.1\n",
            "cuDNN version: 7603\n",
            "torch.device:  cuda:0\n",
            "Current working directory: /content/substra-examples/deepfake-detection\n",
            "No input model, creating a new one pretrained on ImageNet\n",
            "Epoch 1:  13% 43/320 [02:18<15:38,  3.39s/it]no face found\n",
            "Epoch 1:  46% 147/320 [07:48<08:50,  3.07s/it]no face found\n",
            "Epoch 1:  82% 261/320 [13:49<03:18,  3.36s/it]no face found\n",
            "Epoch 1:  97% 310/320 [16:22<00:31,  3.15s/it]no face found\n",
            "Epoch 1: 100% 320/320 [16:52<00:00,  3.13s/it]Epoch:   1, train BCE: 0.6985\n",
            "Elapsed 1012.038173 sec. Average per video: 3.162619 sec.\n",
            "substratools.algo - saving output model to 'assets/model/model'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9riAW4yS_AwB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        },
        "outputId": "da945e13-eb3b-4740-da90-f5261ef8f4e0"
      },
      "source": [
        "#predict the labels of train_data with your previously trained model\n",
        "! python assets/algo_inference/algo.py predict \\\n",
        "  --debug \\\n",
        "  --opener-path assets/dataset/opener.py \\\n",
        "  --data-samples-path assets/train_data_samples \\\n",
        "  --output-predictions-path assets/pred-train.csv \\\n",
        "  --models-path assets/model/ \\\n",
        "  --log-path assets/logs/train_predict.log \\\n",
        "  model"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "substratools.utils - Module 'opener' loaded from path 'assets/dataset/opener.py'\n",
            "substratools.opener - loading X from '['assets/train_data_samples/data_sample_2', 'assets/train_data_samples/data_sample_49', 'assets/train_data_samples/data_sample_48', 'assets/train_data_samples/data_sample_13', 'assets/train_data_samples/data_sample_79', 'assets/train_data_samples/data_sample_74', 'assets/train_data_samples/data_sample_71', 'assets/train_data_samples/data_sample_52', 'assets/train_data_samples/data_sample_26', 'assets/train_data_samples/data_sample_3', 'assets/train_data_samples/data_sample_31', 'assets/train_data_samples/data_sample_46', 'assets/train_data_samples/data_sample_62', 'assets/train_data_samples/data_sample_6', 'assets/train_data_samples/data_sample_14', 'assets/train_data_samples/data_sample_67', 'assets/train_data_samples/data_sample_45', 'assets/train_data_samples/data_sample_28', 'assets/train_data_samples/data_sample_37', 'assets/train_data_samples/data_sample_59', 'assets/train_data_samples/data_sample_64', 'assets/train_data_samples/data_sample_51', 'assets/train_data_samples/data_sample_78', 'assets/train_data_samples/data_sample_56', 'assets/train_data_samples/data_sample_34', 'assets/train_data_samples/data_sample_0', 'assets/train_data_samples/data_sample_17', 'assets/train_data_samples/data_sample_58', 'assets/train_data_samples/data_sample_22', 'assets/train_data_samples/data_sample_5', 'assets/train_data_samples/data_sample_41', 'assets/train_data_samples/data_sample_63', 'assets/train_data_samples/data_sample_39', 'assets/train_data_samples/data_sample_15', 'assets/train_data_samples/data_sample_24', 'assets/train_data_samples/data_sample_23', 'assets/train_data_samples/data_sample_76', 'assets/train_data_samples/data_sample_53', 'assets/train_data_samples/data_sample_57', 'assets/train_data_samples/data_sample_60', 'assets/train_data_samples/data_sample_18', 'assets/train_data_samples/data_sample_44', 'assets/train_data_samples/data_sample_69', 'assets/train_data_samples/data_sample_21', 'assets/train_data_samples/data_sample_27', 'assets/train_data_samples/data_sample_11', 'assets/train_data_samples/data_sample_47', 'assets/train_data_samples/data_sample_66', 'assets/train_data_samples/data_sample_54', 'assets/train_data_samples/data_sample_72', 'assets/train_data_samples/data_sample_70', 'assets/train_data_samples/data_sample_10', 'assets/train_data_samples/data_sample_29', 'assets/train_data_samples/data_sample_65', 'assets/train_data_samples/data_sample_38', 'assets/train_data_samples/data_sample_30', 'assets/train_data_samples/data_sample_4', 'assets/train_data_samples/data_sample_19', 'assets/train_data_samples/data_sample_8', 'assets/train_data_samples/data_sample_68', 'assets/train_data_samples/data_sample_32', 'assets/train_data_samples/data_sample_16', 'assets/train_data_samples/data_sample_33', 'assets/train_data_samples/data_sample_1', 'assets/train_data_samples/data_sample_40', 'assets/train_data_samples/data_sample_61', 'assets/train_data_samples/data_sample_20', 'assets/train_data_samples/data_sample_43', 'assets/train_data_samples/data_sample_75', 'assets/train_data_samples/data_sample_35', 'assets/train_data_samples/data_sample_12', 'assets/train_data_samples/data_sample_9', 'assets/train_data_samples/data_sample_42', 'assets/train_data_samples/data_sample_50', 'assets/train_data_samples/data_sample_25', 'assets/train_data_samples/data_sample_77', 'assets/train_data_samples/data_sample_7', 'assets/train_data_samples/data_sample_36', 'assets/train_data_samples/data_sample_73', 'assets/train_data_samples/data_sample_55']'\n",
            "Finding features file...\n",
            "substratools.algo - loading model from 'assets/model/model'\n",
            "substratools.algo - launching predict task\n",
            "Nb of Test videos: 320\n",
            "PyTorch version: 1.6.0+cu101\n",
            "CUDA version: 10.1\n",
            "cuDNN version: 7603\n",
            "cuda:0\n",
            "Current working directory: /content/substra-examples/deepfake-detection\n",
            "no face found\n",
            "no face found\n",
            "no face found\n",
            "no face found\n",
            "Elapsed 399.059581 sec. Average per video: 1.247061 sec.\n",
            "substratools.opener - saving predictions to 'assets/pred-train.csv'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o74rP5X3_Fm-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154
        },
        "outputId": "0d347ccc-8bed-4f33-aadf-07e27910e98b"
      },
      "source": [
        "#calculate the score of your model on train_data predictions\n",
        "! python assets/objective/metrics.py \\\n",
        "  --debug \\\n",
        "  --opener-path assets/dataset/opener.py \\\n",
        "  --data-samples-path assets/train_data_samples \\\n",
        "  --input-predictions-path assets/pred-train.csv \\\n",
        "  --output-perf-path assets/perf-train.json \\\n",
        "  --log-path assets/logs/train_metrics.log\n",
        "  "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "substratools.opener - loading y from '['assets/train_data_samples/data_sample_2', 'assets/train_data_samples/data_sample_49', 'assets/train_data_samples/data_sample_48', 'assets/train_data_samples/data_sample_13', 'assets/train_data_samples/data_sample_79', 'assets/train_data_samples/data_sample_74', 'assets/train_data_samples/data_sample_71', 'assets/train_data_samples/data_sample_52', 'assets/train_data_samples/data_sample_26', 'assets/train_data_samples/data_sample_3', 'assets/train_data_samples/data_sample_31', 'assets/train_data_samples/data_sample_46', 'assets/train_data_samples/data_sample_62', 'assets/train_data_samples/data_sample_6', 'assets/train_data_samples/data_sample_14', 'assets/train_data_samples/data_sample_67', 'assets/train_data_samples/data_sample_45', 'assets/train_data_samples/data_sample_28', 'assets/train_data_samples/data_sample_37', 'assets/train_data_samples/data_sample_59', 'assets/train_data_samples/data_sample_64', 'assets/train_data_samples/data_sample_51', 'assets/train_data_samples/data_sample_78', 'assets/train_data_samples/data_sample_56', 'assets/train_data_samples/data_sample_34', 'assets/train_data_samples/data_sample_0', 'assets/train_data_samples/data_sample_17', 'assets/train_data_samples/data_sample_58', 'assets/train_data_samples/data_sample_22', 'assets/train_data_samples/data_sample_5', 'assets/train_data_samples/data_sample_41', 'assets/train_data_samples/data_sample_63', 'assets/train_data_samples/data_sample_39', 'assets/train_data_samples/data_sample_15', 'assets/train_data_samples/data_sample_24', 'assets/train_data_samples/data_sample_23', 'assets/train_data_samples/data_sample_76', 'assets/train_data_samples/data_sample_53', 'assets/train_data_samples/data_sample_57', 'assets/train_data_samples/data_sample_60', 'assets/train_data_samples/data_sample_18', 'assets/train_data_samples/data_sample_44', 'assets/train_data_samples/data_sample_69', 'assets/train_data_samples/data_sample_21', 'assets/train_data_samples/data_sample_27', 'assets/train_data_samples/data_sample_11', 'assets/train_data_samples/data_sample_47', 'assets/train_data_samples/data_sample_66', 'assets/train_data_samples/data_sample_54', 'assets/train_data_samples/data_sample_72', 'assets/train_data_samples/data_sample_70', 'assets/train_data_samples/data_sample_10', 'assets/train_data_samples/data_sample_29', 'assets/train_data_samples/data_sample_65', 'assets/train_data_samples/data_sample_38', 'assets/train_data_samples/data_sample_30', 'assets/train_data_samples/data_sample_4', 'assets/train_data_samples/data_sample_19', 'assets/train_data_samples/data_sample_8', 'assets/train_data_samples/data_sample_68', 'assets/train_data_samples/data_sample_32', 'assets/train_data_samples/data_sample_16', 'assets/train_data_samples/data_sample_33', 'assets/train_data_samples/data_sample_1', 'assets/train_data_samples/data_sample_40', 'assets/train_data_samples/data_sample_61', 'assets/train_data_samples/data_sample_20', 'assets/train_data_samples/data_sample_43', 'assets/train_data_samples/data_sample_75', 'assets/train_data_samples/data_sample_35', 'assets/train_data_samples/data_sample_12', 'assets/train_data_samples/data_sample_9', 'assets/train_data_samples/data_sample_42', 'assets/train_data_samples/data_sample_50', 'assets/train_data_samples/data_sample_25', 'assets/train_data_samples/data_sample_77', 'assets/train_data_samples/data_sample_7', 'assets/train_data_samples/data_sample_36', 'assets/train_data_samples/data_sample_73', 'assets/train_data_samples/data_sample_55']'\n",
            "Finding label file...\n",
            "Loading labels...\n",
            "substratools.opener - loading predictions from 'assets/pred-train.csv'\n",
            "substratools.metrics - launching scoring task\n",
            "substratools.metrics - score: 0.33066707605104295\n",
            "substratools.metrics - saving score to 'assets/perf-train.json'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-p2uD0JT-5MF",
        "colab_type": "text"
      },
      "source": [
        "#### Testing task\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ok5WZ_uz_I1s",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "b33f47b5-861a-4705-ed64-61034913acd3"
      },
      "source": [
        "#predict the labels of test_data with your previously trained model\n",
        "! python assets/algo_inference/algo.py predict \\\n",
        "  --debug \\\n",
        "  --opener-path assets/dataset/opener.py \\\n",
        "  --data-samples-path assets/test_data_samples \\\n",
        "  --output-predictions-path assets/pred-test.csv \\\n",
        "  --models-path assets/model/ \\\n",
        "  --log-path assets/logs/test_predict.log \\\n",
        "  model\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "substratools.utils - Module 'opener' loaded from path 'assets/dataset/opener.py'\n",
            "substratools.opener - loading X from '['assets/test_data_samples/data_sample_2', 'assets/test_data_samples/data_sample_13', 'assets/test_data_samples/data_sample_3', 'assets/test_data_samples/data_sample_6', 'assets/test_data_samples/data_sample_14', 'assets/test_data_samples/data_sample_0', 'assets/test_data_samples/data_sample_17', 'assets/test_data_samples/data_sample_5', 'assets/test_data_samples/data_sample_15', 'assets/test_data_samples/data_sample_18', 'assets/test_data_samples/data_sample_11', 'assets/test_data_samples/data_sample_10', 'assets/test_data_samples/data_sample_4', 'assets/test_data_samples/data_sample_19', 'assets/test_data_samples/data_sample_8', 'assets/test_data_samples/data_sample_16', 'assets/test_data_samples/data_sample_1', 'assets/test_data_samples/data_sample_12', 'assets/test_data_samples/data_sample_9', 'assets/test_data_samples/data_sample_7']'\n",
            "Finding features file...\n",
            "substratools.algo - loading model from 'assets/model/model'\n",
            "substratools.algo - launching predict task\n",
            "Nb of Test videos: 80\n",
            "PyTorch version: 1.6.0+cu101\n",
            "CUDA version: 10.1\n",
            "cuDNN version: 7603\n",
            "cuda:0\n",
            "Current working directory: /content/substra-examples/deepfake-detection\n",
            "no face found\n",
            "no face found\n",
            "no face found\n",
            "Elapsed 98.338972 sec. Average per video: 1.229237 sec.\n",
            "substratools.opener - saving predictions to 'assets/pred-test.csv'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "InNU23Bl_KgI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154
        },
        "outputId": "97fa9435-5952-47f5-e570-ffdbf5dcdbd1"
      },
      "source": [
        "#calculate the score of your model on test_data predictions\n",
        "! python assets/objective/metrics.py \\\n",
        "  --debug \\\n",
        "  --opener-path assets/dataset/opener.py \\\n",
        "  --data-samples-path assets/test_data_samples \\\n",
        "  --input-predictions-path assets/pred-test.csv \\\n",
        "  --output-perf-path assets/perf-test.json \\\n",
        "  --log-path assets/logs/test_metrics.log"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "substratools.opener - loading y from '['assets/test_data_samples/data_sample_2', 'assets/test_data_samples/data_sample_13', 'assets/test_data_samples/data_sample_3', 'assets/test_data_samples/data_sample_6', 'assets/test_data_samples/data_sample_14', 'assets/test_data_samples/data_sample_0', 'assets/test_data_samples/data_sample_17', 'assets/test_data_samples/data_sample_5', 'assets/test_data_samples/data_sample_15', 'assets/test_data_samples/data_sample_18', 'assets/test_data_samples/data_sample_11', 'assets/test_data_samples/data_sample_10', 'assets/test_data_samples/data_sample_4', 'assets/test_data_samples/data_sample_19', 'assets/test_data_samples/data_sample_8', 'assets/test_data_samples/data_sample_16', 'assets/test_data_samples/data_sample_1', 'assets/test_data_samples/data_sample_12', 'assets/test_data_samples/data_sample_9', 'assets/test_data_samples/data_sample_7']'\n",
            "Finding label file...\n",
            "Loading labels...\n",
            "substratools.opener - loading predictions from 'assets/pred-test.csv'\n",
            "substratools.metrics - launching scoring task\n",
            "substratools.metrics - score: 0.33870798705350563\n",
            "substratools.metrics - saving score to 'assets/perf-test.json'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4lDtUFN6_Oo0",
        "colab_type": "text"
      },
      "source": [
        "### Using substra cli\n",
        "\n",
        "Before pushing our assets to the platform, we need to make sure they work well. To do so, we can run them locally in a\n",
        "Docker container. This way, if the training fails, we can access the logs and debug our code.\n",
        "\n",
        "To test the assets, we'll use `substra run-local`, passing it paths to our algorithm of course, but also the opener,\n",
        "the metrics and to the data samples we want to use. It will launch a training task on the train data, a prediction task on the test data and return the accuracy score."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WRHo1_vE_dLa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#you will need Docker to run this (not available in Colab)\n",
        "! substra run-local assets/algo_inference \\\n",
        "  --train-opener=assets/dataset/opener.py \\\n",
        "  --test-opener=assets/dataset/opener.py \\\n",
        "  --metrics=assets/objective/ \\\n",
        "  --train-data-samples=assets/train_data_samples \\\n",
        "  --test-data-samples=assets/test_data_samples"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TU7FKMZd_SBG",
        "colab_type": "text"
      },
      "source": [
        "At the end of this step, you'll find in the newly created `sandbox/model` folder a `model` file that contains your\n",
        "trained model. There is also a `sandbox/pred_train` folder that contains both the predictions made by the model on\n",
        "train data and the associated performance.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L0jC-FGC_iBQ",
        "colab_type": "text"
      },
      "source": [
        "#### Debugging\n",
        "\n",
        "It's more than probable that your code won't run perfectly the first time. Since runs happen in dockers, you can't\n",
        "debug using prints. Instead, you should use the `logging` module from python. All logs can then be consulted at the end\n",
        "of the run in  `sandbox/model/log_model.log`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjaDkvLG_3eW",
        "colab_type": "text"
      },
      "source": [
        "## Adding the assets to substra\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PzRwKnVYAER_",
        "colab_type": "text"
      },
      "source": [
        "### Adding the objective, dataset and data samples to substra\n",
        "\n",
        "A script has been written that adds objective, data manager and data samples to substra. It uses the `substra` python\n",
        "sdk to perform actions. It's main goal is to create assets, get their keys and use these keys in the creation of other\n",
        "assets.\n",
        "\n",
        "To run it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6m0VBnrP_NF1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! python scripts/add_dataset_objective.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x-t_byRs__qo",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "This script just generated an `assets_keys.json` file in the `deepfake-detection` folder. This file contains the keys of all assets\n",
        "we've just created and organizes the keys of the train data samples in folds. This file will be used as input when\n",
        "adding an algorithm so that we can automatically launch all training and testing tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tyNVyep5ABxi",
        "colab_type": "text"
      },
      "source": [
        "### Adding the algorithm and training it\n",
        "\n",
        "The script `add_train_algo_inference.py` pushes our simple algo to substra and then uses the `assets_keys.json` file\n",
        "we just generated to train it against the dataset and objective we previously set up. It will then update the\n",
        "`assets_keys.json` file with the newly created assets keys (algo, traintuple and testtuple)\n",
        "\n",
        "To run it:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CiSQmPL0AGEH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! python scripts/add_train_algo_inference.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fvbcn1TYAJbZ",
        "colab_type": "text"
      },
      "source": [
        "It will end by providing a couple of commands you can use to track the progress of the train and test tuples as well\n",
        "as the associated scores. Alternatively, you can browse the frontend to look up progress and scores.\n"
      ]
    }
  ]
}