StyleGAN2_ADA_training_256.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "StyleGAN2_ADA_training_256.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "sbWQ38fBoo4h",
        "g0oZRRs2KO5A"
      ],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UDVO2FGKIQw"
      },
      "source": [
        "# StyleGAN2-ADA: training a model from scratch\n",
        "\n",
        "References:\n",
        "- this notebook: https://github.com/woctezuma/steam-stylegan2-ada\n",
        "- the original StyleGAN2-ADA repository: https://github.com/NVlabs/stylegan2-ada\n",
        "- my fork of StyleGAN2-ADA: https://github.com/woctezuma/stylegan2-ada"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VgwgYzlkyCrU"
      },
      "source": [
        "## Mount Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GmokzZyCKA5o",
        "outputId": "a4604c12-76d4-414e-8df8-e171c361019b"
      },
      "source": [
        "%pip install Google-Colab-Transfer"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: Google-Colab-Transfer in /usr/local/lib/python3.6/dist-packages (0.1.6)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PXvJm3hCKWGf",
        "outputId": "5dc4dd3a-cb53-48c7-d1e4-f71c460bb451"
      },
      "source": [
        "import colab_transfer\n",
        "\n",
        "gd = colab_transfer.get_path_to_home_of_google_drive()\n",
        "lm = colab_transfer.get_path_to_home_of_local_machine()\n",
        "\n",
        "colab_transfer.mount_google_drive()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sbWQ38fBoo4h"
      },
      "source": [
        "## Switch to Tensorflow 1.x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJiCQbhy51Nt"
      },
      "source": [
        "> TensorFlow 2.x is not supported.\n",
        "\n",
        "Reference: https://github.com/NVlabs/stylegan2-ada#requirements\n",
        "\n",
        "> Colab uses TensorFlow 2.x by default, though you can switch to 1.x by the method shown below.\n",
        "\n",
        "Reference: https://colab.research.google.com/notebooks/tensorflow_version.ipynb"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHL4_p1xnoDY"
      },
      "source": [
        "%tensorflow_version 1.x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0oZRRs2KO5A"
      },
      "source": [
        "## Install my fork of StyleGAN2-ADA\n",
        "\n",
        "Switch to the branch `google-colab`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sKYZ49zf41Sv",
        "outputId": "0c17a267-61f6-44a6-8a67-8b85d2c7853d"
      },
      "source": [
        "%cd /content/\n",
        "\n",
        "%rm -rf stylegan2-ada/\n",
        "!git clone https://github.com/woctezuma/stylegan2-ada.git\n",
        "\n",
        "%cd stylegan2-ada/\n",
        "!git checkout google-colab\n",
        "\n",
        "%cd /content/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "Cloning into 'stylegan2-ada'...\n",
            "remote: Enumerating objects: 99, done.\u001b[K\n",
            "remote: Total 99 (delta 0), reused 0 (delta 0), pack-reused 99\u001b[K\n",
            "Unpacking objects: 100% (99/99), done.\n",
            "/content/stylegan2-ada\n",
            "Branch 'google-colab' set up to track remote branch 'google-colab' from 'origin'.\n",
            "Switched to a new branch 'google-colab'\n",
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruo5KFL4yw0j"
      },
      "source": [
        "## Prepare datasets (once)\n",
        "\n",
        "Reference: https://github.com/NVlabs/stylegan2-ada#preparing-datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aj-3t7dM98dj"
      },
      "source": [
        "### Import image data from Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WxZdPFd-zyiq",
        "outputId": "42017d1d-315e-4397-86a0-fd0a08deb482"
      },
      "source": [
        "#!rm covidGanFood/ -r\r\n",
        "!git clone --single-branch --branch \"256RGB\" https://lucalin17081994:AgahamS0rr0w!@github.com/jMysliwiec/covidGanFood.git"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'covidGanFood'...\n",
            "remote: Enumerating objects: 3, done.\u001b[K\n",
            "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2/2), done.\u001b[K\n",
            "remote: Total 1364 (delta 0), reused 2 (delta 0), pack-reused 1361\u001b[K\n",
            "Receiving objects: 100% (1364/1364), 180.30 MiB | 21.63 MiB/s, done.\n",
            "Resolving deltas: 100% (1/1), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iL6_7wnZKo3w"
      },
      "source": [
        "### Prepare datasets for StyleGAN2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K1Qd8VPH9djO"
      },
      "source": [
        "custom_dataset = '/content/datasets/COVID256RGB'\n",
        "custom_images = '/content/covidGanFood/COVID256RGB'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "whUYxCDgK66P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b8e9b604-bd9f-4925-bab7-df98c080d735"
      },
      "source": [
        "!python stylegan2-ada/dataset_tool.py create_from_images {custom_dataset} {custom_images}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading images from \"/content/covidGanFood/COVID256RGB\"\n",
            "Creating dataset \"/content/datasets/COVID256RGB\"\n",
            "Added 517 images.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RWpIjP7hKzJZ"
      },
      "source": [
        "## Train new networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "McdnLndylAxa"
      },
      "source": [
        "### Computation time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzU6-RNSQ8kT"
      },
      "source": [
        "A tick requires 4 minutes of computation time, so we save a snapshot every 6 ticks (`--snap=6`) in order to save every 24 minutes.\n",
        "\n",
        "Processing 1 kimg requires 1 min of computation time, so:\n",
        "-   the total training time for **5 Mimg** should be 3.5 days of computation,\n",
        "-   the total training time for **25 Mimg** should be 2.5 weeks of computation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BWxFpc-QkJOH"
      },
      "source": [
        "### Documentation for transfer learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qjlohd3uNeEY"
      },
      "source": [
        "```\n",
        "transfer learning:\n",
        "  --resume RESUME       Resume from network pickle (default: noresume)\n",
        "  --freezed INT         Freeze-D (default: 0 discriminator layers)\n",
        "```\n",
        "```\n",
        "transfer learning source networks (--resume):\n",
        "  ffhq256        FFHQ trained at 256x256 resolution.\n",
        "  ffhq512        FFHQ trained at 512x512 resolution.\n",
        "  ffhq1024       FFHQ trained at 1024x1024 resolution.\n",
        "  celebahq256    CelebA-HQ trained at 256x256 resolution.\n",
        "  lsundog256     LSUN Dog trained at 256x256 resolution.\n",
        "  <path or URL>  Custom network pickle.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEzOo7gtBNvE"
      },
      "source": [
        "### Training recommendations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQfYzLRvBSD3"
      },
      "source": [
        "-   about `--gamma`:\n",
        "> The training configuration can be further customized with additional arguments. Common examples:\n",
        "> * `--gamma=10` overrides R1 gamma. We strongly recommend trying out at least a few different values for each new dataset.\n",
        "\n",
        "-   about `--cfg`:\n",
        "> Please note that `--cfg=auto` only serves as a reasonable first guess for the hyperparameters &mdash; it does not necessarily lead to optimal results for a given dataset. For example, `--cfg=stylegan2` yields considerably better FID  for FFHQ-140k at 1024x1024 than illustrated above. We recommend trying out at least a few different values of `--gamma` for each new dataset.\n",
        "\n",
        "-   about `--metrics`:\n",
        "> By default, `train.py` will automatically compute FID for each network pickle. We strongly recommend inspecting `metric-fid50k_full.txt` at regular intervals to monitor the training progress. When desired, the automatic computation can be disabled with `--metrics none` to speed up the training.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imBVQY2bQyFN"
      },
      "source": [
        "### Training settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbIhukS9ZK4x"
      },
      "source": [
        "Resolution is automatically determined from the dataset.\n",
        "Other settings are chosen as follows:\n",
        "- horizontal mirroring, via `--mirror=true` (instead of default to False),\n",
        "- no EMA rampup, via `--cfg=auto_no_ramp`,\n",
        "- cfg map = 8 (instead of 2), via `--cfg_map=8`,\n",
        "- freezed D (the first 10 layers, as in the article for resolution 256), via `--freezed=10`.\n",
        "\n",
        "References:\n",
        "-   https://github.com/NVlabs/stylegan2-ada#training-new-networks\n",
        "-   https://raw.githubusercontent.com/NVlabs/stylegan2-ada/main/docs/train-help.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apnOxk7UZFIL"
      },
      "source": [
        "### Train\n",
        "\n",
        "NB: to ensure your Colab session stays connected, follow: https://stackoverflow.com/a/58275370/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUKiyWY22dzf"
      },
      "source": [
        "#### Initial run (transfer learning)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_yKuIVhicPyS"
      },
      "source": [
        "Start with transfer learning from a model trained on a **diverse** dataset (`lsundog256`).\n",
        "\n",
        "Caveat: the upstream model was trained with cfg map = 8."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f8TWmeqExkda",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c7326c2-db6b-4504-ec43-568412666937"
      },
      "source": [
        "!python stylegan2-ada/train.py \\\n",
        " --gpus=1 \\\n",
        " --outdir='/content/drive/My Drive/training-runs' \\\n",
        " --snap=6 \\\n",
        " --data='datasets/COVID256RGB' \\\n",
        " --mirror=true \\\n",
        " --metrics=None \\\n",
        " --cfg=auto_no_ramp \\\n",
        " --cfg_map=8 \\\n",
        " --kimg=5000 \\\n",
        " --resume=lsundog256 \\\n",
        " --freezed=10 \\\n",
        " "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tcmalloc: large alloc 4294967296 bytes == 0x890a000 @  0x7ff28f263001 0x7ff28c4a64ff 0x7ff28c4f6b08 0x7ff28c4faac7 0x7ff28c5991a3 0x50a4a5 0x50cc96 0x507be4 0x508ec2 0x594a01 0x549e8f 0x5515c1 0x5a9dac 0x50a433 0x50cc96 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x5095c8 0x50a2fd 0x50beb4 0x507be4\n",
            "tcmalloc: large alloc 4294967296 bytes == 0x7ff0a12c4000 @  0x7ff28f2611e7 0x7ff28c4a641e 0x7ff28c4f6c2b 0x7ff28c4f730f 0x7ff28c5990a3 0x50a4a5 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4\n",
            "tcmalloc: large alloc 4294967296 bytes == 0x7fefa02c2000 @  0x7ff28f2611e7 0x7ff28c4a641e 0x7ff28c4f6c2b 0x7ff28c4f730f 0x7ff237de60c5 0x7ff237769902 0x7ff237769eb2 0x7ff237722c3e 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588c8b 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x508ec2 0x594a01\n",
            "\n",
            "Training options:\n",
            "{\n",
            "  \"G_args\": {\n",
            "    \"func_name\": \"training.networks.G_main\",\n",
            "    \"fmap_base\": 8192,\n",
            "    \"fmap_max\": 512,\n",
            "    \"mapping_layers\": 8,\n",
            "    \"num_fp16_res\": 4,\n",
            "    \"conv_clamp\": 256\n",
            "  },\n",
            "  \"D_args\": {\n",
            "    \"func_name\": \"training.networks.D_main\",\n",
            "    \"mbstd_group_size\": 4,\n",
            "    \"fmap_base\": 8192,\n",
            "    \"fmap_max\": 512,\n",
            "    \"num_fp16_res\": 4,\n",
            "    \"conv_clamp\": 256,\n",
            "    \"freeze_layers\": 10\n",
            "  },\n",
            "  \"G_opt_args\": {\n",
            "    \"beta1\": 0.0,\n",
            "    \"beta2\": 0.99,\n",
            "    \"learning_rate\": 0.0025\n",
            "  },\n",
            "  \"D_opt_args\": {\n",
            "    \"beta1\": 0.0,\n",
            "    \"beta2\": 0.99,\n",
            "    \"learning_rate\": 0.0025\n",
            "  },\n",
            "  \"loss_args\": {\n",
            "    \"func_name\": \"training.loss.stylegan2\",\n",
            "    \"r1_gamma\": 0.8192\n",
            "  },\n",
            "  \"augment_args\": {\n",
            "    \"class_name\": \"training.augment.AdaptiveAugment\",\n",
            "    \"tune_heuristic\": \"rt\",\n",
            "    \"tune_target\": 0.6,\n",
            "    \"apply_func\": \"training.augment.augment_pipeline\",\n",
            "    \"apply_args\": {\n",
            "      \"xflip\": 1,\n",
            "      \"rotate90\": 1,\n",
            "      \"xint\": 1,\n",
            "      \"scale\": 1,\n",
            "      \"rotate\": 1,\n",
            "      \"aniso\": 1,\n",
            "      \"xfrac\": 1,\n",
            "      \"brightness\": 1,\n",
            "      \"contrast\": 1,\n",
            "      \"lumaflip\": 1,\n",
            "      \"hue\": 1,\n",
            "      \"saturation\": 1\n",
            "    },\n",
            "    \"tune_kimg\": 100\n",
            "  },\n",
            "  \"num_gpus\": 1,\n",
            "  \"image_snapshot_ticks\": 6,\n",
            "  \"network_snapshot_ticks\": 6,\n",
            "  \"train_dataset_args\": {\n",
            "    \"path\": \"datasets/COVID256RGB\",\n",
            "    \"max_label_size\": 0,\n",
            "    \"resolution\": 256,\n",
            "    \"mirror_augment\": true\n",
            "  },\n",
            "  \"metric_arg_list\": [],\n",
            "  \"metric_dataset_args\": {\n",
            "    \"path\": \"datasets/COVID256RGB\",\n",
            "    \"max_label_size\": 0,\n",
            "    \"resolution\": 256,\n",
            "    \"mirror_augment\": true\n",
            "  },\n",
            "  \"total_kimg\": 5000,\n",
            "  \"minibatch_size\": 16,\n",
            "  \"minibatch_gpu\": 16,\n",
            "  \"G_smoothing_kimg\": 5.0,\n",
            "  \"G_smoothing_rampup\": null,\n",
            "  \"resume_pkl\": \"https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada/pretrained/transfer-learning-source-nets/lsundog-res256-paper256-kimg100000-noaug.pkl\",\n",
            "  \"run_dir\": \"/content/drive/My Drive/training-runs/00027-COVID256RGB-mirror-auto_no_ramp1-kimg5000-resumelsundog256-freezed10\"\n",
            "}\n",
            "\n",
            "Output directory:  /content/drive/My Drive/training-runs/00027-COVID256RGB-mirror-auto_no_ramp1-kimg5000-resumelsundog256-freezed10\n",
            "Training data:     datasets/COVID256RGB\n",
            "Training length:   5000 kimg\n",
            "Resolution:        256\n",
            "Number of GPUs:    1\n",
            "\n",
            "Creating output directory...\n",
            "Loading training set...\n",
            "tcmalloc: large alloc 4294967296 bytes == 0x7fee907d0000 @  0x7ff28f263001 0x7ff28c4a64ff 0x7ff28c4f6b08 0x7ff28c4faac7 0x7ff28c5991a3 0x50a4a5 0x50cc96 0x507be4 0x508ec2 0x594a01 0x549e8f 0x5515c1 0x5a9dac 0x50a433 0x50cc96 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x507be4 0x588e5c 0x59fd0e 0x50d256 0x5095c8 0x50a2fd 0x50beb4 0x507be4\n",
            "tcmalloc: large alloc 4294967296 bytes == 0x7fed8ffd0000 @  0x7ff28f2611e7 0x7ff28c4a641e 0x7ff28c4f6c2b 0x7ff28c4f730f 0x7ff28c5990a3 0x50a4a5 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4 0x509900 0x50a2fd 0x50cc96 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50cc96 0x507be4\n",
            "tcmalloc: large alloc 4294967296 bytes == 0x7fed8ffd0000 @  0x7ff28f2611e7 0x7ff28c4a641e 0x7ff28c4f6c2b 0x7ff28c4f730f 0x7ff237de60c5 0x7ff237769902 0x7ff237769eb2 0x7ff237722c3e 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588c8b 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x508ec2 0x594a01\n",
            "Image shape: [3, 256, 256]\n",
            "Label shape: [0]\n",
            "\n",
            "Constructing networks...\n",
            "Setting up TensorFlow plugin \"fused_bias_act.cu\": Compiling... Loading... Done.\n",
            "Setting up TensorFlow plugin \"upfirdn_2d.cu\": Compiling... Loading... Done.\n",
            "Traceback (most recent call last):\n",
            "  File \"stylegan2-ada/train.py\", line 579, in <module>\n",
            "    main()\n",
            "  File \"stylegan2-ada/train.py\", line 571, in main\n",
            "    run_training(**vars(args))\n",
            "  File \"stylegan2-ada/train.py\", line 467, in run_training\n",
            "    training_loop.training_loop(**training_options)\n",
            "  File \"/content/stylegan2-ada/training/training_loop.py\", line 124, in training_loop\n",
            "    Gs = G.clone('Gs')\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 457, in clone\n",
            "    net.copy_vars_from(self)\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 490, in copy_vars_from\n",
            "    src_net._get_vars()\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 297, in _get_vars\n",
            "    self._vars = OrderedDict(self._get_own_vars())\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 286, in _get_own_vars\n",
            "    self._init_graph()\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 151, in _init_graph\n",
            "    out_expr = self._build_func(*self._input_templates, **build_kwargs)\n",
            "  File \"/content/stylegan2-ada/training/networks.py\", line 231, in G_main\n",
            "    num_layers = components.synthesis.input_shape[1]\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 232, in input_shape\n",
            "    return self.input_shapes[0]\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 219, in input_shapes\n",
            "    self._input_shapes = [t.shape.as_list() for t in self.input_templates]\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 267, in input_templates\n",
            "    self._init_graph()\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/network.py\", line 181, in _init_graph\n",
            "    tfutil.run(remaining_inits)\n",
            "  File \"/content/stylegan2-ada/dnnlib/tflib/tfutil.py\", line 33, in run\n",
            "    return tf.get_default_session().run(*args, **kwargs)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 956, in run\n",
            "    run_metadata_ptr)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1180, in _run\n",
            "    feed_dict_tensor, options, run_metadata)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1359, in _do_run\n",
            "    run_metadata)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1365, in _do_call\n",
            "    return fn(*args)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1350, in _run_fn\n",
            "    target_list, run_metadata)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1443, in _call_tf_sessionrun\n",
            "    run_metadata)\n",
            "KeyboardInterrupt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BdjRSBEZ2Q4B",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "cc07dcc8-3af1-422e-9edf-4ee638ab5c9e"
      },
      "source": [
        "while True:pass"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-19-534b7a74019f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2FGozskE2gGM"
      },
      "source": [
        "#### Following runs (resume from latest snapshot)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvLmB4OrcSHS"
      },
      "source": [
        "Then,  **automatically** resume from the latest pickle snapshot.\n",
        "\n",
        "Caveat: you will have to **manually** specify the augmentation strength, as found in `log.txt`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7s7cdAcU2l3x"
      },
      "source": [
        "!python stylegan2-ada/train.py \\\n",
        " --gpus=1 \\\n",
        " --outdir='/content/drive/My Drive/training-runs' \\\n",
        " --snap=8 \\\n",
        " --data='/content/datasets/COVID256RGB' \\\n",
        " --mirror=true \\\n",
        " --metrics=fid50k_full \\\n",
        " --cfg=auto \\\n",
        " --cfg_map=8 \\\n",
        " --kimg=5000 \\\n",
        "\n",
        "# --gamma=10 \\\n",
        "#--p=0.713 \\\n",
        "# --freezed=0 \\"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbSO9r10vW9V"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}