Dev-Notebook-Code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adversarial perturbations can pose a serious threat for deploying machine learning systems. Recent works have shown existence of image-agnostic perturbations that can fool classifiers over most natural images. Existing methods present optimization approaches that solve for a fooling objective with an imperceptibility constraint to craft the perturbations making it very hard to defend.\n",
    "\n",
    "\n",
    "#### Motivation\n",
    "Current Approaches for crafting adversaries for a given classifier generate only one perturbation at a time, which is a single instance from the manifold of adversarial perturbations. In order to build robust models, it is essential to explore diverse manifold of adversarial perturbations. This work can be of very useful, when we are using adversarial trainning, where the cost of generation of adversaries is high(Depends on the attack). With this approach, we will be able to generate adversarial noises from the learned distribution of adversarial perturbations. \n",
    "\n",
    "\n",
    "#### Key Results: \n",
    "The author's demonstrate that perturbations crafted by our model\n",
    "1. achieve state-of-the-art fooling rates\n",
    "2. exhibit wide variety \n",
    "3. deliver excellent cross model generalizability.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aproach\n",
    "\n",
    "The architecture of the proposed model is inspired from that of GANs and is trained using fooling and diversity objectives. Our trained generator network attempts to capture the distribution of adversarial perturbations for a given classifier and readily\n",
    "generates a wide variety of such perturbations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "![Proposed approach](resources/nag.png)\n",
    "\n",
    "- **Core idea is to model the distribution of universal adversarial perturbations for a given classifier.**\n",
    "- The image shows a batch of B random vectors {z}<sub>B</sub> transforming into perturbations {delta}<sub>B</sub> by G which get added to the batch of data samples {x}<sub>B</sub>.\n",
    "- The top portion shows adversarial batch (X<sub>A</sub>), bottom portion shows shuffled adversarial batch (X<sub>S</sub>) and middle portion shows the benign batch (X<sub>B</sub>). The Fooling objective **Lf** and Diversity objective **Ld** constitute the loss. \n",
    "### Note\n",
    "- Note that the target CNN (f) is a trained classifier and its parameters are not updated during the proposed training. On the other hand, the parameters of generator (G) are randomly initialized and learned through backpropagating the loss. (Best viewed in color)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation\n",
    "- Download of Dataset \n",
    "**P.S**: Randomly Sampled 10 instances from each target class as described in the paper.\n",
    "- Option 1: Download from Archive.org\n",
    "    - [Archive Link](https://archive.org/details/Imagenet_NAG)\n",
    "    - [train.zip](https://archive.org/download/Imagenet_NAG/train.zip)\n",
    "    - [valid.zip](https://archive.org/download/Imagenet_NAG/valid.zip)\n",
    "- Option 2 : Mega Download Link for Train abd Validation data of Imagenet 2012 (Obtained from Kaggle)\n",
    "    - Validation Data: [Mega Link](https://mega.nz/#!yDoTDIyD!RjN6OBA92-KLpNqDeLS3OzwmAYesEbTsiQat9hT6p6s)\n",
    "    - Trainning Data: [Mega Link](https://mega.nz/#!vKY0WSDa!4aibnBkiXUrO9MkhQlLGXac7wLF5HY7O4LzfdFEaeQU) \n",
    "<!-- - If link fails to work use the following Colab notebook to generate your own subset of trainning examples. [Link](https://colab.research.google.com/drive/1LbZBfgqntWb3HuC3UFyF_FvwnHtd1xTA) -->\n",
    "- Setting up of Folder Structure\n",
    "For Easier handling and reproducibility of results download from mega link \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Below Assumes Dataset is downlaoded and setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Verify Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Data Verification\n",
      "Total Number of Classes: 1000 in train directory\n",
      "Total 10000 number of files in 1000 classes. i.e 10 Images/Class\n",
      "Validation Data Verification\n",
      "Validation Data has correct number of files i.e 50000\n",
      "Dataset is Setup Correctly\n"
     ]
    }
   ],
   "source": [
    "from glob import glob\n",
    "\n",
    "train_ok = True\n",
    "val_ok = True\n",
    "print(\"Training Data Verification\")\n",
    "cls_count = len(glob(\"ILSVRC/train/*\"))\n",
    "print(\"Total Number of Classes: {} in train directory\".format(cls_count))\n",
    "count = 0\n",
    "for cls_ in glob(\"ILSVRC/train/*\"):\n",
    "    imgs = glob(cls_ + \"/*\")\n",
    "    img_count = len(imgs)\n",
    "    count += img_count\n",
    "    if img_count != 10:\n",
    "        print(cls_.split(\"/\")[-1], img_count)\n",
    "        train_ok=False\n",
    "print(\"Total {} number of files in {} classes. i.e 10 Images/Class\".format(count, cls_count))\n",
    "\n",
    "print(\"Validation Data Verification\")\n",
    "val_files = glob(\"ILSVRC/valid/*\")\n",
    "val_count = len(val_files)\n",
    "if val_count == 50000:\n",
    "    print(\"Validation Data has correct number of files i.e {}\".format(val_count))\n",
    "else:\n",
    "    print(\"Validation Data has some issue. Has following number of file : {}. Kindly Check!!\".format(val_count))\n",
    "    val_ok=False\n",
    "if train_ok and val_ok:\n",
    "    print(\"Dataset is Setup Correctly\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader,Dataset\n",
    "\n",
    "import torchvision\n",
    "import torchvision.models as tvm\n",
    "\n",
    "from torchvision import transforms\n",
    "from torchvision.datasets.folder import DatasetFolder,ImageFolder\n",
    "\n",
    "import numpy as np\n",
    "from glob import glob\n",
    "from PIL import Image\n",
    "import pandas as pd\n",
    "import os,time,gc\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "import datetime,random,string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Pytorch Version : 1.4.0 and Torchvision Version : 0.5.0. Using Device cuda\n"
     ]
    }
   ],
   "source": [
    "ngpu=torch.cuda.device_count()\n",
    "device = torch.device(\"cuda\" if (torch.cuda.is_available() and ngpu > 0) else \"cpu\")\n",
    "\n",
    "print(\"Using Pytorch Version : {} and Torchvision Version : {}. Using Device {}\".format(torch.__version__,torchvision.__version__,device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset and Dataloaders Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset root Folder:ILSVRC/. Train Data Path: ILSVRC/train. Validation Data Path ILSVRC/valid\n"
     ]
    }
   ],
   "source": [
    "dataset_path=r'ILSVRC/'\n",
    "train_dataset_path=dataset_path+'train'\n",
    "test_dataset_path=dataset_path+'valid'\n",
    "print(\"Dataset root Folder:{}. Train Data Path: {}. Validation Data Path {}\".format(dataset_path,train_dataset_path,test_dataset_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparation of Labels \n",
    "label_dict={}\n",
    "label_idx={}\n",
    "\n",
    "with open('ILSVRC/LOC_synset_mapping.txt') as file:\n",
    "    lines=file.readlines()\n",
    "    for idx,line in enumerate(lines):\n",
    "        label,actual =line.strip('\\n').split(' ',maxsplit=1)\n",
    "        label_dict[label]=actual\n",
    "        label_idx[label]=idx\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transforms\n",
    "size=224\n",
    "# Imagenet Stats\n",
    "vgg_mean = [103.939, 116.779, 123.68]\n",
    "\n",
    "preprocess=transforms.Compose([transforms.Resize((size,size)),\n",
    "                               transforms.ToTensor(),\n",
    "                               transforms.Normalize(vgg_mean,(0.5, 0.5, 0.5))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset and Dataloaders\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Number of instances in valid subset of Dataset: 50000\n"
     ]
    }
   ],
   "source": [
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, subset, root_dir, transform=None):\n",
    "        self.root_dir=root_dir\n",
    "        self.transform=transform\n",
    "       \n",
    "        self.subset=subset\n",
    "        if self.subset=='train':\n",
    "            data_dir=os.path.join(self.root_dir,self.subset)\n",
    "            self.images_fn=glob(f'{data_dir}/*/*')\n",
    "            self.labels=[Path(fn).parent.name for fn in self.images_fn]\n",
    "        elif subset =='valid':\n",
    "            df=pd.read_csv('ILSVRC/LOC_val_solution.csv')\n",
    "            df['label']=df['PredictionString'].str.split(' ',n=1,expand=True)[0]\n",
    "            df=df.drop(columns=['PredictionString'])\n",
    "            self.images_fn='ILSVRC/valid/'+df['ImageId'].values+'.JPEG'\n",
    "            self.labels=df['label']\n",
    "        else:\n",
    "            raise ValueError\n",
    "        print(f\" Number of instances in {self.subset} subset of Dataset: {len(self.images_fn)}\")       \n",
    "\n",
    "    def __getitem__(self,idx):\n",
    "        fn=self.images_fn[idx]\n",
    "        label=self.labels[idx]\n",
    "        image=Image.open(fn)\n",
    "        if image.getbands()[0] == 'L':\n",
    "            image = image.convert('RGB')\n",
    "        if self.transform:\n",
    "            image = self.transform(image)    \n",
    "        return image,label_idx[label]\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.images_fn)\n",
    "        \n",
    "data_train=ImageFolder(root='ILSVRC/train',transform=preprocess)\n",
    "class2idx=data_train.class_to_idx\n",
    "data_valid=CustomDataset(subset='valid',root_dir=dataset_path,transform=preprocess)\n",
    "\n",
    "train_num = len(data_train)\n",
    "val_num = len(data_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proposed Approach\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "![Proposed approach](resources/nag.png)\n",
    "\n",
    "- **Core idea is to model the distribution of universal adversarial perturbations for a given classifier.**\n",
    "- The image shows a batch of B random vectors {z}<sub>B</sub> transforming into perturbations {delta}<sub>B</sub> by G which get added to the batch of data samples {x}<sub>B</sub>.\n",
    "- The top portion shows adversarial batch (X<sub>A</sub>), bottom portion shows shuffled adversarial batch (X<sub>S</sub>) and middle portion shows the benign batch (X<sub>B</sub>). The Fooling objective Lf (eq. 2) and Diversity objective Ld (eq. 3) constitute the loss. \n",
    "### Note\n",
    "- Note that the target CNN (f) is a trained classifier and its parameters are not updated during the proposed training. On the other hand, the parameters of generator (G) are randomly initialized and learned through backpropagating the loss. (Best viewed in color)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Functions/Objectives \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fooling_objective(qc_):\n",
    "    '''Helper function to computer compute -log(1-qc'), \n",
    "    where qc' is the adversarial probability of the class having \n",
    "    maximum probability in the corresponding clean probability\n",
    "    qc' ---> qc_\n",
    "    Parameters: \n",
    "    prob_vec : Probability vector for the clean batch\n",
    "    adv_prob_vec : Probability vecotr of the adversarial batch\n",
    "    Returns: \n",
    "    -log(1-qc') , qc'\n",
    "    \n",
    "    '''  \n",
    "    # Get the largest probablities from predictions : Shape (bs,1)\n",
    "    qc_=qc_.mean()\n",
    "    return -1*torch.log(1-qc_) , qc_\n",
    "\n",
    "def diversity_objective(prob_vec_no_shuffle, prob_vec_shuffled):\n",
    "    '''Helper function to calculate the cosine distance between two probability vectors\n",
    "    Parameters: \n",
    "    prob_vec : Probability vector for the clean batch\n",
    "    adv_prob_vec : Probability vector for the adversarial batch\n",
    "    Returns : \n",
    "    Cosine distance between the corresponding clean and adversarial batches\n",
    "    '''    \n",
    "    return torch.cosine_similarity(prob_vec_no_shuffle,prob_vec_shuffled).mean()\n",
    "\n",
    "## TODO : Not Required. As we always take the last layer.\n",
    "\n",
    "def intermediate_activation_objective(layer_name=None):\n",
    "    ''' Extract the activations of any intermediate layer for:\n",
    "    1. batch of images (of batch size=32) corrupted by the perturbations (of batch size=32) \n",
    "    2. same batch of images corrupted by same batch of perturbations but in different (random) order\n",
    "    (in this case the intermdeiate layer is set to 'res4f' of ResNet 50 architecture)\n",
    "    '''\n",
    "    if arch =='resnet50':\n",
    "        layer_name='res4f'\n",
    "    \n",
    "    pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Self Note: \n",
    "- Effect of ConvTranspose2d : It is a combination of upsampling and convolution layers used to increase the spatial resolution of the tensor\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator \n",
    "- Architecture of our generator (G) unchanged for different target CNN architectures\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![DCGAN](resources/DCGAN.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "ngf=128\n",
    "nz= latent_dim=10\n",
    "e_lim = 10\n",
    "nc=3 # Number of Channels\n",
    "\n",
    "# Fixed Architecture: Weights will be updated by Backprop.\n",
    "class AdveraryGenerator(nn.Module):\n",
    "    def __init__(self,e_lim):\n",
    "        super(AdveraryGenerator, self).__init__()\n",
    "        self.e_lim = e_lim\n",
    "        self.main = nn.Sequential(\n",
    "        nn.ConvTranspose2d( in_channels=nz,out_channels= 1024, kernel_size=4, stride=1, padding=0, bias=False),\n",
    "        nn.BatchNorm2d(1024),\n",
    "        nn.ReLU(True),\n",
    "        # state size. (ngf*8) x 4 x 4\n",
    "        nn.ConvTranspose2d(1024, 512, 4, 2, 1, bias=False),\n",
    "        nn.BatchNorm2d(512),\n",
    "        nn.ReLU(True),\n",
    "        # state size. (ngf*4) x 8 x 8\n",
    "        nn.ConvTranspose2d( 512, 256, 4, 2, 1, bias=False),\n",
    "        nn.BatchNorm2d(256),\n",
    "        nn.ReLU(True),\n",
    "        # state size. (ngf*2) x 16 x 16\n",
    "        nn.ConvTranspose2d(256, 128, 4, 2, 2, bias=False),\n",
    "        nn.BatchNorm2d(128),\n",
    "        nn.ReLU(True),\n",
    "        # state size. (ngf) x 32 x 32\n",
    "        nn.ConvTranspose2d( 128, 64, 4, 2, 2, bias=False),\n",
    "        nn.BatchNorm2d(64),\n",
    "        nn.ReLU(True),\n",
    "        # state size. (nc) x 64 x 64\n",
    "        nn.ConvTranspose2d( 64, 3, 4, 4,4, bias=False),\n",
    "        nn.BatchNorm2d(3),\n",
    "        nn.ReLU(True),\n",
    "        nn.Tanh()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.e_lim * self.main(x) # Scaling of ε\n",
    "    \n",
    "# move Generator to GPU if available\n",
    "adversarygen=AdveraryGenerator(e_lim).to(device)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Debugging\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "if debug:\n",
    "    try:\n",
    "        from torchsummary import summary\n",
    "        summary(adversarygen,(nz,1,1))\n",
    "    except:\n",
    "        raise('Check torchsummary is installed. If not install using the command pip install torchsummary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up Discriminator : Model : Architecture\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.models import googlenet, vgg16 , vgg19, resnet152, resnet50\n",
    "\n",
    "\n",
    "model_dict ={\n",
    "    'googlenet': googlenet,\n",
    "    'vgg16': vgg16 ,\n",
    "    'vgg19':vgg19, \n",
    "    'resnet152':resnet152, # TODO Generate Perturbations\n",
    "    'resnet50':resnet50    # TODO Generate Perturbations \n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run only once :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get all Pretrained Weights:\n",
    "for arch in model_dict.keys():\n",
    "    if arch !='vgg-f':\n",
    "        model=model_dict[arch](pretrained=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choice of Hyperparameters\n",
    "- The architecture of the generator consists of 5 deconv layers. The final deconv layer is followed by a tanh non-linearity and scaling by epsillon (10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# epsillon=10\n",
    "# batch_size=32\n",
    "# latent_dim = 10\n",
    "img_h,img_w,img_c=(224,224,3)\n",
    "latent_dim=10\n",
    "arch='resnet50'\n",
    "archs=model_dict.keys() # ['vgg-f','vgg16','vgg19','googlenet','resnet50','resnet152'] \n",
    "\n",
    "def get_bs(arch):\n",
    "    if torch.cuda.is_available():\n",
    "#         GPU_BENCHMARK= 8192.0\n",
    "#         GPU_MAX_MEM = torch.cuda.get_device_properties(device).total_memory / (1024*1024)\n",
    "#         BS_DIV= GPU_BENCHMARK/GPU_MAX_MEM\n",
    "#         print(f\"Current GPU MAX Size : {GPU_MAX_MEM}. {BS_DIV}\")\n",
    "\n",
    "        if arch  not in ['resnet50','resnet152']:#  ['vgg16','vgg19','vgg-f','googlenet']:\n",
    "            bs=int(64)\n",
    "        elif arch in ['resnet50','resnet152']:\n",
    "            bs=int(32)\n",
    "        else:\n",
    "            raise ValueError(f'Architecture type not supported. Please choose one from the following {archs}')\n",
    "    else:\n",
    "        bs=8 # OOM Error\n",
    "    return bs\n",
    "\n",
    "get_bs(arch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ResNet(\n",
       "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
       "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu): ReLU(inplace=True)\n",
       "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
       "  (layer1): Sequential(\n",
       "    (0): Bottleneck(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): Bottleneck(\n",
       "      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (2): Bottleneck(\n",
       "      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "  )\n",
       "  (layer2): Sequential(\n",
       "    (0): Bottleneck(\n",
       "      (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): Bottleneck(\n",
       "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (2): Bottleneck(\n",
       "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (3): Bottleneck(\n",
       "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "  )\n",
       "  (layer3): Sequential(\n",
       "    (0): Bottleneck(\n",
       "      (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (2): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (3): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (4): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (5): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "  )\n",
       "  (layer4): Sequential(\n",
       "    (0): Bottleneck(\n",
       "      (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): Bottleneck(\n",
       "      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "    (2): Bottleneck(\n",
       "      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "    )\n",
       "  )\n",
       "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
       "  (fc): Linear(in_features=2048, out_features=1000, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model=model_dict[arch](pretrained=True)\n",
    "model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_checkpoint(model, to_save, filename='checkpoint.pth'):\n",
    "    \"\"\"Save checkpoint if a new best is achieved\"\"\"\n",
    "    if to_save:\n",
    "        print (\"=> Saving a new best\")\n",
    "        torch.save(model.state_dict(), filename)  # save checkpoint\n",
    "    else:\n",
    "        print (\"=> Validation Accuracy did not improve\")\n",
    "        \n",
    "def save_perturbations(noise,arch,epoch,wabdb_flag=False):\n",
    "    rand_str= ''.join( random.choice(string.ascii_letters) for i in range(6))\n",
    "    os.makedirs(f\"{arch}-{rand_str}\",exist_ok=True)\n",
    "    perturbations=noise.permute(0,2,3,1).cpu().detach().numpy()*255\n",
    "    np.save(f'{arch}-{rand_str}/Perturbations_{arch}_{epoch}.npy', perturbations)\n",
    "    for perturb_idx,perturbation in enumerate(perturbations[:,]):\n",
    "        \n",
    "        im = Image.fromarray(perturbation.astype(np.uint8))\n",
    "        if wabdb_flag:\n",
    "            wandb.log({\"noise\": [wandb.Image(im, caption=f\"Noise_{arch}_{epoch}_{perturb_idx}\")]})\n",
    "        im.save(f'{arch}-{rand_str}/Perturbations_{arch}_{epoch}_{perturb_idx}.png')        \n",
    "\n",
    "# TODO \n",
    "def visualize_perturbations():\n",
    "    # MAtplotlib Subplot ?\n",
    "    # Subplots(4*4) or (3*3)\n",
    "    # From Memory or Disk - Epoch number ?\n",
    "    pass\n",
    "\n",
    "\n",
    "\n",
    "def get_preds(predictions,return_idx=False, k=1):\n",
    "    idxs= torch.argsort(predictions,descending=True)[:,:k]\n",
    "    if return_idx:\n",
    "        return predictions[:,idxs], idxs\n",
    "    return  predictions[:,idxs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validating Model Utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# val_iterations = val_num/bs\n",
    "\n",
    "def compute_fooling_rate(prob_adv,prob_real):\n",
    "    '''Helper function to calculate mismatches in the top index vector\n",
    "     for clean and adversarial batch\n",
    "     Parameters:\n",
    "     prob_adv : Index vector for adversarial batch\n",
    "     prob_real : Index vector for clean batch\n",
    "     Returns:\n",
    "     Number of mismatch and its percentage\n",
    "    '''\n",
    "    nfool=0\n",
    "    size = prob_real.shape[0]\n",
    "    for i in range(size):\n",
    "        if prob_real[i]!=prob_adv[i]:\n",
    "            nfool = nfool+1\n",
    "    return nfool, 100*float(nfool)/size      \n",
    "\n",
    "\n",
    "def validate_generator_old(noise,val_dl,val_iterations=10):\n",
    "    \n",
    "    total_fool=0\n",
    "    print(\"############### VALIDATION PHASE STARTED ################\")\n",
    "    train_log.writelines(\"############### VALIDATION PHASE STARTED ################\")\n",
    "    \n",
    "    for val_idx in range(val_iterations):\n",
    "        for batch_idx, data in enumerate(val_dl):\n",
    "            images = data[0].to(device)\n",
    "#             labels = data[1].to(device)\n",
    "            \n",
    "            prob_vec_clean = F.softmax(D_model(images),dim=0) # Variable q\n",
    "            prob_vec_no_shuffle = D_model(images + noise)  \n",
    "            nfool, _ = compute_fooling_rate(prob_vec_no_shuffle,prob_vec_clean)\n",
    "            total_fool += nfool\n",
    "    \n",
    "    \n",
    "    fool_rate = 100*float(total_fool)/(val_iterations*batch_size)       \n",
    "    print(f\"Fooling rate: {foolr}. Total Items Fooled :{total_fool}\")\n",
    "    train_log.writelines(f\"Fooling rate: {foolr}. Total Items Fooled :{total_fool}\")\n",
    "\n",
    "    \n",
    "    \n",
    "def validate_generator(noise,D_model,val_dl):\n",
    "    total_fool=0\n",
    "    for batch_idx, data in tqdm(enumerate(val_dl),total = val_num//val_dl.batch_size):\n",
    "        val_images = data[0].to(device)\n",
    "        val_labels = data[1].to(device)\n",
    "\n",
    "        prob_vec_clean,clean_idx = get_preds(F.softmax(D_model(val_images),dim=0),return_idx=True) # Variable q\n",
    "        prob_vec_no_shuffle,adv_idx = get_preds(F.softmax(D_model(val_images + noise),dim=0),return_idx=True)  \n",
    "        nfool, _ = compute_fooling_rate(adv_idx,clean_idx)\n",
    "        total_fool += nfool\n",
    "\n",
    "    fool_rate = 100*float(total_fool)/(val_num)\n",
    "    return fool_rate,total_fool\n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Test  Fooling Objective\n",
    "adv = torch.randint(0,1000,(32,1))\n",
    "real = torch.randint(0,1000,(32,1))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setup Wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                Logging results to <a href=\"https://wandb.com\" target=\"_blank\">Weights & Biases</a> <a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">(Documentation)</a>.<br/>\n",
       "                Project page: <a href=\"https://app.wandb.ai/gokkulnath/NAG_Pytorch\" target=\"_blank\">https://app.wandb.ai/gokkulnath/NAG_Pytorch</a><br/>\n",
       "                Run page: <a href=\"https://app.wandb.ai/gokkulnath/NAG_Pytorch/runs/hm0t7y9w\" target=\"_blank\">https://app.wandb.ai/gokkulnath/NAG_Pytorch/runs/hm0t7y9w</a><br/>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "W&B Run: https://app.wandb.ai/gokkulnath/NAG_Pytorch/runs/hm0t7y9w"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup Wandb \n",
    "\n",
    "import wandb\n",
    "wandb.login()\n",
    "wandb.init(project=\"NAG_Pytorch\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit and Train the Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(nb_epochs,D_model,dls,optimizer,adversarygen=adversarygen):\n",
    "    # Set the Discriminator in Eval mode; Weights are fixed.\n",
    "    train_dl,val_dl = dls\n",
    "    D_model=D_model.to(device)\n",
    "    D_model.eval()\n",
    "    timestamp=datetime.datetime.now().strftime(\"%d%b%Y_%H_%M\")\n",
    "    train_log = open(f'train_log_{arch}_{timestamp}.txt','w')\n",
    "    for epoch in tqdm(range(nb_epochs),total=nb_epochs):\n",
    "        running_loss=0\n",
    "        rand_str= ''.join( random.choice(string.ascii_letters) for i in range(6))\n",
    "        \n",
    "        train_log.writelines(f\"############### TRAIN PHASE STARTED : {epoch}################\")\n",
    "        for batch_idx, data in tqdm(enumerate(train_dl),total = train_num//train_dl.batch_size):\n",
    "            # Move Data and Labels to device(GPU)\n",
    "            images = data[0].to(device)\n",
    "            labels = data[1].to(device)\n",
    "\n",
    "            \n",
    "            # Generate the Adversarial Noise from Uniform Distribution U[-1,1]\n",
    "            latent_seed = 2 * torch.rand(bs, nz, 1, 1, device=device,requires_grad=True) -1 # (r1 - r2) * torch.rand(a, b) + r2\n",
    "            noise = adversarygen(latent_seed)\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # XB = images\n",
    "            #preds_XB = f(images)\n",
    "            prob_vec_clean = F.softmax(D_model(images),dim=0) # Variable q\n",
    "            clean_preds ,clean_idx = get_preds(prob_vec_clean,return_idx=True,k=1)\n",
    "            \n",
    "            #XA = images+noise\n",
    "            #preds_XA = f(images + noise)\n",
    "            prob_vec_no_shuffle = D_model(images + noise)  \n",
    "            qc_ =  F.softmax(prob_vec_no_shuffle,dim=0).gather(1,clean_idx) # Variable q'c\n",
    "\n",
    "            # 1. fooling_objective: encourages G to generate perturbations that decrease confidence of benign predictions\n",
    "            fool_obj, mean_qc_ = fooling_objective(qc_)\n",
    "            # Perturbations  are shuffled across the batch dimesion to improve diversity\n",
    "            #XS = images+ noise[torch.randperm(bs)]\n",
    "            prob_vec_shuffled =   D_model(images + noise[torch.randperm(bs)])\n",
    "            \n",
    "            # 2.  encourages Generator to explore the space of perturbations and generate a diverse set of perturbations\n",
    "            divesity_obj=diversity_objective(prob_vec_no_shuffle, prob_vec_shuffled)\n",
    "\n",
    "            # Compute Total Loss\n",
    "            total_loss = divesity_obj + fool_obj\n",
    "            \n",
    "            # Lets perform Backpropagation to compute Gradients and update the weights\n",
    "            total_loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            # wandb Logging : Expensive : Logs Perturbation Images each iteration\n",
    "#             perturbations=noise.permute(0,2,3,1).cpu().detach().numpy()*255\n",
    "#             for perturb_idx,perturbation in enumerate(perturbations[:,]):\n",
    "#                 im = Image.fromarray(perturbation.astype(np.uint8))\n",
    "#                 wandb.log({\"noise\": [wandb.Image(im, caption=f\"Noise_{arch}_{epoch}_{perturb_idx}\")]})\n",
    "            wandb.log({\"fool_obj\": fool_obj.item(),\n",
    "                       \"divesity_obj\": divesity_obj.item(),\n",
    "                       \"total_loss\":total_loss.item(),\n",
    "                      })        \n",
    "            \n",
    "            running_loss += total_loss.item()\n",
    "            \n",
    "            if batch_idx!=0  and batch_idx % 100 ==0 :\n",
    "                train_log.writelines(f\"############### VALIDATION PHASE STARTED : {epoch}, Step : {int(batch_idx / 100)} ################\")\n",
    "                fool_rate,total_fool= validate_generator(noise,D_model,val_dl)\n",
    "                print(f\"Fooling rate: {fool_rate}. Total Items Fooled :{total_fool}\")\n",
    "                train_log.writelines(f\"Fooling rate: {fool_rate}. Total Items Fooled :{total_fool}\")\n",
    "        print(f\"Diversity Loss :{divesity_obj.item()} \\n Fooling Loss: {fool_obj.item()} \\n\")\n",
    "        print(f\"Total Loss after Epoch No: {epoch +1} - {running_loss/(train_num//train_dl.batch_size)}\")\n",
    "        train_log.writelines(f\"Loss after Epoch No: {epoch +1} is {running_loss/(train_num//train_dl.batch_size)}\")\n",
    "        # to_save can be any expression/condition that returns a bool\n",
    "        \n",
    "        save_checkpoint(adversarygen, to_save= True, filename=f'GeneratorW_{arch}_{epoch}_{rand_str}.pth') \n",
    "        if epoch % 1 == 0:\n",
    "#             save_perturbations(noise,arch,epoch)\n",
    "            save_perturbations(noise,arch,epoch,wabdb_flag=True)\n",
    "    train_log.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start Actual Trainning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Generator for Arch resnet50\n",
      "32\n",
      "Elsasped Time 0.6291134357452393 Seconds\n"
     ]
    }
   ],
   "source": [
    "total_epochs = 20\n",
    "lr = 1e-3\n",
    "# Setting up Dataloaders\n",
    "import time,gc\n",
    "\n",
    "arch='resnet50'\n",
    "start= time.time()\n",
    "print(f\"Training Generator for Arch {arch}\")\n",
    "model= model_dict[arch](pretrained=True)\n",
    "bs = get_bs(arch)\n",
    "print(bs)\n",
    "train_dl=DataLoader(data_train,batch_size=bs,shuffle=True,num_workers=4,pin_memory=True,drop_last=True)\n",
    "val_dl=DataLoader(data_valid,batch_size=bs,shuffle=True,num_workers=4,pin_memory=True,drop_last=True)\n",
    "dls = [train_dl,val_dl]\n",
    "optimizer = optim.Adam(adversarygen.parameters(), lr=lr)\n",
    "\n",
    "print(f\"Elsasped Time {time.time()-start} Seconds\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.7/site-packages/ipykernel_launcher.py:8: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
      "Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n",
      "  \n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8aca44cd3f9841c5b9ab58fca7e6fa86",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=20.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.7/site-packages/ipykernel_launcher.py:13: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
      "Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n",
      "  del sys.path[0]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5aa38d9588cd4894982b1d5603d8d760",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.7/site-packages/ipykernel_launcher.py:45: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
      "Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "902af7c1f69747fc98d655acb52e9e5d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.126. Total Items Fooled :49563\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7b9ef3e9f1d4895998504bc4b2c85ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.506. Total Items Fooled :49753\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74950f3c8bac40c989c8afde8c6b7b95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.364. Total Items Fooled :49682\n",
      "\n",
      "Diversity Loss :0.9992280602455139 \n",
      " Fooling Loss: 0.03950555995106697 \n",
      "\n",
      "Total Loss after Epoch No: 1 - 1.0430664758269603\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dd4f5c9669f4478ae678ccce77b85ec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4a42f2146d944c6a9f79ca05ec5b303",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.332. Total Items Fooled :49666\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c1cf1aa2bf1458c83673d0d4a24131c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.486. Total Items Fooled :49743\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a671fc2fe9c946ed859208781a4b9c4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 98.67. Total Items Fooled :49335\n",
      "\n",
      "Diversity Loss :0.9981168508529663 \n",
      " Fooling Loss: 0.02838512323796749 \n",
      "\n",
      "Total Loss after Epoch No: 2 - 1.039582596757473\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "387bef93b781471084698b56726faeb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8945f598fdf49ddb1051e7e5a75e33e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.446. Total Items Fooled :49723\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3b6c0b2b8a894a82a46fba0bc494118b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.52. Total Items Fooled :49760\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec361ff919e040bb882f64e94581fdee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.38. Total Items Fooled :49690\n",
      "\n",
      "Diversity Loss :0.9990019798278809 \n",
      " Fooling Loss: 0.004721864592283964 \n",
      "\n",
      "Total Loss after Epoch No: 3 - 1.039885509854708\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9420071213484152b42c83ca065338e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69734cf0550b4fa287e803b68c10bd90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.356. Total Items Fooled :49678\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec35fb037e59452fb4b79aa270e045a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.658. Total Items Fooled :49829\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f9bc2828ae044229db017bde96fdac7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.662. Total Items Fooled :49831\n",
      "\n",
      "Diversity Loss :0.999189555644989 \n",
      " Fooling Loss: 0.003563906066119671 \n",
      "\n",
      "Total Loss after Epoch No: 4 - 1.0383393155076566\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e1adfdea23e4da2aacdd3dd255faa37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "280fd486e3524e98b65734fa231ad5d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.62. Total Items Fooled :49810\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c6e1eef28b34f9296d9e19dfd4b7211",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.486. Total Items Fooled :49743\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bceaa4d595e54bf78dc55550fcd9febc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.686. Total Items Fooled :49843\n",
      "\n",
      "Diversity Loss :0.9990421533584595 \n",
      " Fooling Loss: 0.008808250539004803 \n",
      "\n",
      "Total Loss after Epoch No: 5 - 1.0367735035908527\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11d1e1446ece4c639e99af4be38f39f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a308fc4b40ac4b37970ebe1cb1b6e1ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.744. Total Items Fooled :49872\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "caf7ff581474496c9a8362abb227b743",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.338. Total Items Fooled :49669\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "244b9bf4c1aa4d95bec1a5d7312ed331",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.692. Total Items Fooled :49846\n",
      "\n",
      "Diversity Loss :0.9993197917938232 \n",
      " Fooling Loss: 0.005549242720007896 \n",
      "\n",
      "Total Loss after Epoch No: 6 - 1.037304274737835\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80b3e2255e7048368701e5881f028c5c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "185f44a7574d48b5988ed065ac844a06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.494. Total Items Fooled :49747\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35b863abccf842669e34f4fd5090bf50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.422. Total Items Fooled :49711\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0462a06a8104354bb8a819971619b97",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.454. Total Items Fooled :49727\n",
      "\n",
      "Diversity Loss :0.9981052875518799 \n",
      " Fooling Loss: 0.04303847253322601 \n",
      "\n",
      "Total Loss after Epoch No: 7 - 1.038387955763401\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17d9f1f8cf9a4ac4b76379a3b68ac6b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ce535abce7f4fcf967e6320876bca68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.29. Total Items Fooled :49645\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d28c2430c8a14256ae02885de1476d3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.416. Total Items Fooled :49708\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69913bbc3236409c926cb1e770230dce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.558. Total Items Fooled :49779\n",
      "\n",
      "Diversity Loss :0.998153567314148 \n",
      " Fooling Loss: 6.139297056506621e-06 \n",
      "\n",
      "Total Loss after Epoch No: 8 - 1.0364860896116648\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3de145daecad4ef58a1a3b70adddc791",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "854a4053618b48439b652803402b5ea6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.59. Total Items Fooled :49795\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6c58163cc86482daf00b659f8c4d3cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.78. Total Items Fooled :49890\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b776196db804e2683b0a815aad1c435",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.444. Total Items Fooled :49722\n",
      "\n",
      "Diversity Loss :0.998258113861084 \n",
      " Fooling Loss: 0.002013623248785734 \n",
      "\n",
      "Total Loss after Epoch No: 9 - 1.0362467425755966\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e728089fab1549698b646a4e4b2dcedd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8684a02ef50c4e14b2526834c74f0b50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.598. Total Items Fooled :49799\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9dac234075cd4f24968653926ddd99a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.486. Total Items Fooled :49743\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4efa9828b1e4dc19e6862ba09b959de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.456. Total Items Fooled :49728\n",
      "\n",
      "Diversity Loss :0.9985426664352417 \n",
      " Fooling Loss: 0.008954382501542568 \n",
      "\n",
      "Total Loss after Epoch No: 10 - 1.038268227989857\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9123a14ee2f040cba30907c5ba0aa7c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a48017f088e4b599057814aa78d3abe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.72. Total Items Fooled :49860\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46702a3bfe0647f0885c3f25c8e0c40a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.426. Total Items Fooled :49713\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01be9ff69001403e83d21558e5303b26",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.692. Total Items Fooled :49846\n",
      "\n",
      "Diversity Loss :0.9976068735122681 \n",
      " Fooling Loss: 0.020556485280394554 \n",
      "\n",
      "Total Loss after Epoch No: 11 - 1.0406007697949042\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e5f0fba39b14cd68db68b10c5c505e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c864b11d9194c0dad8287eca09e06eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.436. Total Items Fooled :49718\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d8d366c6aaf4e2badbeba2370ffafe1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.63. Total Items Fooled :49815\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ded95ab2631c42248e6ee78859b9d92d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.554. Total Items Fooled :49777\n",
      "\n",
      "Diversity Loss :0.9977318048477173 \n",
      " Fooling Loss: 0.04298632964491844 \n",
      "\n",
      "Total Loss after Epoch No: 12 - 1.0370476129345405\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0c38694f42c490b9d48516e87226116",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb56c70ae518448d936c43c59c87edb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.498. Total Items Fooled :49749\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0f2dd82c6564c6bbf9479aad30f5b3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.562. Total Items Fooled :49781\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35862993839c44edbba68ab93288953e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.458. Total Items Fooled :49729\n",
      "\n",
      "Diversity Loss :0.9988154172897339 \n",
      " Fooling Loss: 0.04159717634320259 \n",
      "\n",
      "Total Loss after Epoch No: 13 - 1.037070428713774\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47ca1a5a1ed54a5b9e13f0f344709b56",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "478410c57efe47d9a992ec2351edb0b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.368. Total Items Fooled :49684\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14c9451d16dd44f9baa1669ffa6ae7d5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.644. Total Items Fooled :49822\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5310a077a1d64e848860cf0fbb117c82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.446. Total Items Fooled :49723\n",
      "\n",
      "Diversity Loss :0.9994094371795654 \n",
      " Fooling Loss: 0.0624578632414341 \n",
      "\n",
      "Total Loss after Epoch No: 14 - 1.0377162058766072\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6eb37dd07f7d40f8ae37ebfa7299ba38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "beae471c513d413e9a1c5099e30f98d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.696. Total Items Fooled :49848\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f29b887d5c384ada932753e3a721baca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.788. Total Items Fooled :49894\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09c51a13c3154915b8bf65adfe038864",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.494. Total Items Fooled :49747\n",
      "\n",
      "Diversity Loss :0.9997531175613403 \n",
      " Fooling Loss: 0.035558152943849564 \n",
      "\n",
      "Total Loss after Epoch No: 15 - 1.0360386572205103\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d450c2ed8c2489c8e7a264eddf9fc8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6d0a73182844d45b0e4535d96d368cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.678. Total Items Fooled :49839\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aec20e2e7ef849ceb2d90ff12aabd1cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.574. Total Items Fooled :49787\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d28fdf360f274d398d66e96963914d87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.548. Total Items Fooled :49774\n",
      "\n",
      "Diversity Loss :0.9999127388000488 \n",
      " Fooling Loss: 0.07229295372962952 \n",
      "\n",
      "Total Loss after Epoch No: 16 - 1.041024495011721\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c450f86424ef44bbbc009eb493f1512f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff15ee543bf24522a6026876ac12bd3e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.7. Total Items Fooled :49850\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "380fe9965d4a46f0a814a5af763acd82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.542. Total Items Fooled :49771\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf6be17f0d2845acabb81aacf2a776c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.654. Total Items Fooled :49827\n",
      "\n",
      "Diversity Loss :0.9986363649368286 \n",
      " Fooling Loss: 0.0533723421394825 \n",
      "\n",
      "Total Loss after Epoch No: 17 - 1.0388616713193746\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8cbbc50e6004118b661006387d0497a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "071bfdf262344f4f91efb2a485bed12f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.634. Total Items Fooled :49817\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37e2a105ebee489f8c7a8a586731fa6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.78. Total Items Fooled :49890\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6072cb9b2a424cb2931bf9d4179da304",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.24. Total Items Fooled :49620\n",
      "\n",
      "Diversity Loss :0.9978522658348083 \n",
      " Fooling Loss: 0.03446746990084648 \n",
      "\n",
      "Total Loss after Epoch No: 18 - 1.0345487464696934\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60810a90ac9345cdab26a6e8285f9fe1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "006cc20e01fb4a888dcae6ca119de6ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.37. Total Items Fooled :49685\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da76fa4833c14fe6912e19c8e35c2fa2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.69. Total Items Fooled :49845\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5b10da12b854a5584b313d3404da5cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.492. Total Items Fooled :49746\n",
      "\n",
      "Diversity Loss :0.9980953335762024 \n",
      " Fooling Loss: 0.04048139601945877 \n",
      "\n",
      "Total Loss after Epoch No: 19 - 1.0351752294943883\n",
      "=> Saving a new best\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82cbf6f80bcb426f93c86404fc423230",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=312.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68c261aff127455aa325fb3a3d813722",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.674. Total Items Fooled :49837\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8852304d585344bdb78108b322244aed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.714. Total Items Fooled :49857\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee4e5f1232954587882e96e80aad3ea0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1562.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fooling rate: 99.434. Total Items Fooled :49717\n",
      "\n",
      "Diversity Loss :0.9989607334136963 \n",
      " Fooling Loss: 0.021351832896471024 \n",
      "\n",
      "Total Loss after Epoch No: 20 - 1.0392777105936637\n",
      "=> Saving a new best\n",
      "\n"
     ]
    }
   ],
   "source": [
    "fit(nb_epochs=total_epochs,D_model=model,dls=dls,optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Misc:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Setup Caffenet and VGG-F  (TODO)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Paper  : http://www.robots.ox.ac.uk/~vgg/publications/2015/Parkhi15/parkhi15.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the below code first before loading VGG-F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} PrepareCaffenetModel.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading VGG-F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from vgg import VGG_F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = vgg_f()\n",
    "model.load_state_dict(torch.load('VGG_FACE.caffemodel.pt'))\n",
    "model_dict['vgg-f'] =  model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model(torch.rand((3,224,224)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading Trained Weights\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pretrained Generator Weigths for Googlenet, Resnet50, VGG16 and VGG19 Avalaible as a Kaggle Dataset\n",
    "- Link : https://www.kaggle.com/gokkulnath/nag-pytorch-pretrained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below line after setting up kaggle api key\n",
    "# !kaggle datasets download -d gokkulnath/nag-pytorch-pretrained"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating NAG performance across Models: (TODO)\n",
    "- For Tabular Column Generation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Steps to evaluate the perturbations generated by Generator Network (TODO)\n",
    "```arch='Fixed'\n",
    "for modelarch, model in model_dict.items():\n",
    "    num_iteration = 10 # Blackbox Settings\n",
    "    if modelarch == arch:\n",
    "        num_iteration =100 # Whitebox Settings \n",
    "    for i range(num_iteration)\n",
    "        1. Load the Weights of the Generator\n",
    "        2. Generate a Perturbation using a random vector of dimension latent_dim,1\n",
    "        3. Add the noise to a sample image \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpolating Latent Dimension for NAG \n",
    "\n",
    "[![Interpolating Latent Dimension for NAG](https://img.youtube.com/vi/2lojORAu8vA/0.jpg)](https://www.youtube.com/watch?v=2lojORAu8vA&feature=youtu.be)\n",
    "###### Youtube Link : [Link](https://www.youtube.com/watch?v=2lojORAu8vA&feature=youtu.be) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Obtained Perturbations\n",
    "![](resources/Collage_perturbation.v1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References:\n",
    "- Official Code Repo : https://github.com/val-iisc/nag\n",
    "- GAN Architecture : Pytorch Tutorial\n",
    "- [Transpose Convolution Docs](https://pytorch.org/docs/stable/nn.html?highlight=convtranspose2d#torch.nn.ConvTranspose2d)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (fastai)",
   "language": "python",
   "name": "fastai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}