Generative Adversarial Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Adversarial Networks\n",
    "\n",
    "The objective today is to use a neural network library written from scratch to create a generative adversarial network, from which we will create data that resembles the ever so popular MNIST data set.  \n",
    "\n",
    "First let's get a few questions out of the way\n",
    "\n",
    "## Why?\n",
    "\n",
    "1.  Scala - because it is a great language\n",
    "2.  ND4J - it is the main linear algebra library for DL4J (Deep learning 4 Java).  The author's objective was to shorten the gap between JVM languages and Numpy or Matlab\n",
    "3.  From Scratch - it is challenging a.f. and a lot of fun.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "505c4fb4-3aa9-4265-9087-4d3bb3e48ae4",
       "version_major": 2,
       "version_minor": 0
      },
      "method": "display_data"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12a1aee3-fb1b-4b9e-a03d-41b2e7b8bc3d",
       "version_major": 2,
       "version_minor": 0
      },
      "method": "display_data"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%classpath add mvn \n",
    "org.nd4j nd4j-native-platform 0.7.2\n",
    "org.nd4j nd4s_2.11 0.7.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%classpath add jar ../scala-miniflow/target/scala-2.11/scala-miniflow_2.11-0.1.0-SNAPSHOT.jar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Feed Forward Neural networks\n",
    "\n",
    "Here we will begin to get into how we will model our neural network framework. Per Deep Learning by Goodfellow, the feedforward neural network is called such because\n",
    "\n",
    "* (feedforward) information flows through the function being evaluated from the input $x$, through intermediate computations used to define $f$, and finally to the output $y$. We are not considering any feedback connections\n",
    "* (network) They are typically represented by composing together many different functions. The model is associated with a directy acyclic graph describing how functions are componsed together.\n",
    "* (neural) The models are loosly inspired by neuroscience.  \n",
    "\n",
    "While this is from scratch, we will spend little time on the framework with the exception of an example of how to set up simple neural network.  \n",
    "\n",
    "Before that, we'll introduce our main building block - the Node class.  \n",
    "\n",
    "Our Node will\n",
    "\n",
    "* take, as arguments, incoming nodes\n",
    "* have a method which captures outbound nodes\n",
    "* have a forward method (feed forward)\n",
    "* have a backward method (for back propagation)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GAN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generative Adversarial Networks, or GANs for short, are amazing.  You put two networks in a game, where one network's objective is to trick the other network. \n",
    "\n",
    "In what follows, the tricktser (aka the generator), will generate images, those images will be mixed in with real images and the other network (aka the discriminator) will attempt to label the images correctly as fake vs real.  \n",
    "\n",
    "Based on how we set up the objective functions for both networks\n",
    "\n",
    "* The generator's objective will help it get better a generating fake images\n",
    "* The discriminator's objective will help it get better at picking out the fake images.  \n",
    "\n",
    "By the end the generator will be able to take a vector of noise and turn it into something that resembles an image from the mnist dataset.\n",
    "\n",
    "## Generative vs Discimrinative Models\n",
    "\n",
    "Survised machine learning models can be characterized as either Generative or Discriminative. \n",
    "\n",
    "Generative models learn joint probability distributions, whereas Discriminative models learn conditional distributions. \n",
    "\n",
    "Examples of generative models include\n",
    "* Gaussian Mixture Models\n",
    "* Hidden Markov Model\n",
    "\n",
    "Examples of discriminative models include\n",
    "* Logistic regression\n",
    "* Support Vector Machines\n",
    "\n",
    "In the generative adversarial network, we are pitting a generative model (the generator) against a discriminitive model (the discriminator).  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why does this work?\n",
    "\n",
    "Let's refer to the Generator as $G$ and the Discriminator as $D$.  \n",
    "\n",
    "$G$ takes as an input an $1 \\times n$ vector of random noise, and outputs a $1 \\times p$ vectors that now represents an images.  \n",
    "\n",
    "$D$ takes as an input an $1 \\times p$ vector representing an image a produces a scaler between 0 and 1 which will represent the probability that the input is real.  \n",
    "\n",
    "Finally, let $x$ represent an image and $z$ a noise vector.  \n",
    "\n",
    "### Discriminator objective \n",
    "\n",
    "The cost function for $D$ should be obvious - binary cross entropy. \n",
    "\n",
    "$$cost_D = - \\sum 1_{y = real}\\ln(p) + 1_{y = fake}\\ln(1 - p) $$\n",
    "\n",
    "The smaller $cost_D$ is the better the discriminator is at discriminating\n",
    "\n",
    "### Generator objective\n",
    "\n",
    "For $G$, we want $G$ to generate real looking pictures, so suppose that we told the discriminator that the images $G$ generated are real?  That would be equivalent to the following cost function \n",
    "\n",
    "$$cost_G = -\\sum \\ln( (D \\circ G)(z) )$$\n",
    "\n",
    "So, the cost for $G$ is natural log of the discriminator evaluated at the images created by the generator.   Suppose that $D \\circ G$ is slightly larger than 0, then the discriminator thinks the image is fake, then $\\ln ( D \\circ G)$ will be very negative.  Then multiplying this by a negative gives a very large number.  So when cost of $G$ is very large, this means the discriminator knows it is seeing a fake.  On the other hand, if $D \\circ G$ is closer to 1, then $\\ln(D \\circ G)$ will be very small (in the absolute sense), and we'll see $cost_G$ be very small.  This implies that the discriminator is believing the data that we provided is real!  So now if we work to optimize this function, we want to minimize this, that means the optimization work towards making the generator generate real looking images!!  Pretty damn cool if you ask me "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "\n",
    "In the original GAN paper, the optimization goes as follows\n",
    "```\n",
    "for number of training iterations do\n",
    "  for k steps do\n",
    "    * sample minibatch of m noise samples from noise prior \n",
    "    * sample minibatch of m examples from d.g.p.\n",
    "    * update the discriminator \n",
    "  end for\n",
    "  * sample minibatch of m noise samples from noise prior\n",
    "  * update the generator\n",
    "end for\n",
    "```\n",
    "In what follows we set $k = 1$.  There are slight order of operation differences between what follows and the optimization routine above.  Morver, the original GAN architecture relies on Maxout activations and drop out in the discriminator and ReLUs in the generator.  See the configuration [here](https://github.com/goodfeli/adversarial/blob/master/mnist.yaml).  \n",
    "\n",
    "This process can be described as a zero sum game, where increases and descreases in the objective function of one network is balanceed by decreasess and increases in the objective function of the other network.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the graph\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we train the gan, we'll do a forward and backward on the generator's graph.  \n",
    "\n",
    "![generator graph](img/generator_graph.png)\n",
    "\n",
    "and then we do a forward and a backward on the discriminator's graph\n",
    "\n",
    "![discriminator graph](img/discriminator_graph.png)\n",
    "\n",
    "It is important to note that we will not use any derivatives calculated for the discriminator's parameter during the back prop of the generator's graph!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import com.github.timsetsfire.nn.node._\n",
    "import com.github.timsetsfire.nn.activation._\n",
    "import com.github.timsetsfire.nn.costfunctions._\n",
    "import com.github.timsetsfire.nn.batchnormalization._\n",
    "import com.github.timsetsfire.nn.regularization.Dropout\n",
    "import com.github.timsetsfire.nn.optimize._\n",
    "import com.github.timsetsfire.nn.graph._\n",
    "\n",
    "import scala.util.Try\n",
    "import org.nd4j.linalg.factory.Nd4j\n",
    "import org.nd4j.linalg.api.ndarray.INDArray\n",
    "import org.nd4j.linalg.ops.transforms.Transforms.{sigmoid,exp,log,pow,sqrt}\n",
    "import org.nd4s.Implicits._\n",
    "\n",
    "import breeze.linalg.DenseMatrix\n",
    "import breeze.plot._\n",
    "\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Data \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using the test data set from MNIST.  \n",
    "\n",
    "Run `curl -s https://pjreddie.com/media/files/mnist_test.csv > ./data/mnist_test.csv` in terminal (or git bash on windows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val x_ = Nd4j.readNumpy(\"data/mnist_test.csv\", \",\").getColumns( (1 until 785):_*).div(255d)\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inputs\n",
    "\n",
    "For inputs, we'll have the following \n",
    "* Images - this will be used for real and fake images alike.  It will serve as the input to our _discriminator_.  \n",
    "* Labels - this will be used for real and fake lables.  It will serve as an input to our _discriminator_ in particular the objective function \n",
    "* Noise - this is the noise that will be used to generate images.  It will be an input to the _generator_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val images = new Input()\n",
    "images.setName(\"images\")\n",
    "val labels = new Input()\n",
    "labels.setName(\"labels\")\n",
    "val noise = new Input()  // noise is used to generate fake images\n",
    "noise.setName(\"noise\")\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator Network\n",
    "\n",
    "This is the network that will generate fake data. Notice that the network does not \"end\" with an objective.  It ends with a fakeImage output.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val h1Generator= LeakyReLU(noise, (100,128), 0.2)\n",
    "h1Generator.setName(\"generator_hidden1\")\n",
    "\n",
    "val h2Generator= LeakyReLU(h1Generator, (128, 256), 0.2)\n",
    "h2Generator.setName(\"generator_hidden2\")\n",
    "\n",
    "val h3Generator= LeakyReLU(h2Generator, (256, 512), 0.2)\n",
    "h3Generator.setName(\"generator_hidden3\")\n",
    "\n",
    "val fakeImages = Sigmoid(h3Generator, (512,784))  // the output, 784, is the dimensions of the image (28x28)\n",
    "fakeImages.setName(\"fake_images\")\n",
    "\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Variable@58043370, Variable@1d1ecbd0, Variable@50d68622, Variable@21dd9b1a, Variable@5011a007, Variable@548aa46, Variable@33af23eb, Variable@6ae7e583]]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val generator = topologicalSort{ buildGraph(fakeImages) }\n",
    "val generatorTrainables = generator.filter{ _.getClass.getSimpleName == \"Variable\" }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discriminator Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val h1Discrim = LeakyReLU(images, (784,256), 0.1)  // the input is the dimensions of the image\n",
    "h1Discrim.setName(\"discriminator_hidden_layer1\")\n",
    "\n",
    "val d1 = new Dropout(h1Discrim, 0.20)\n",
    "d1.setName(\"dropout_h1_layer\")\n",
    "\n",
    "val h2Discrim = LeakyReLU(d1, (256,64), 0.1)\n",
    "h2Discrim.setName(\"discriminator_hidden_layer2\")\n",
    "\n",
    "val d2 = new Dropout(h2Discrim, 0.20)\n",
    "d2.setName(\"dropout_h2_layer\")\n",
    "\n",
    "val h3Discrim = LeakyReLU(d2, (64,16), 0.1)\n",
    "h3Discrim.setName(\"discriminator_hidden_layer3\")\n",
    "\n",
    "val logits = Linear(h3Discrim, (16, 1))\n",
    "logits.setName(\"discriminator_logits\")\n",
    "\n",
    "val cost = new BceWithLogits(labels, logits)\n",
    "cost.setName(\"cost_function\")\n",
    "\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Variable@34805f2f, Variable@11edaa5f, Variable@3f8ef57, Variable@3537a120, Variable@11c068f5, Variable@1d07b279, Variable@69c0b628, Variable@15034985]]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val discriminator = topologicalSort{ buildGraph(cost) }\n",
    "val discriminatorTrainables = discriminator.filter{ _.getClass.getSimpleName == \"Variable\" }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize\n",
    "\n",
    "Next, we'll initialize the trainable parameters of the networks.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "// initialize generator and discriminator parameters\n",
    "discriminatorTrainables.foreach{ node =>\n",
    "    val size = node.size\n",
    "    val (m,n) = (size._1.asInstanceOf[Int], size._2.asInstanceOf[Int])\n",
    "    node.value = Nd4j.randn(m, n) * math.sqrt(3/(m.toDouble + n.toDouble))\n",
    "  }\n",
    "\n",
    "// initialize generator and discriminator\n",
    "generatorTrainables.foreach{ node =>\n",
    "    val size = node.size\n",
    "    val (m,n) = (size._1.asInstanceOf[Int], size._2.asInstanceOf[Int])\n",
    "    node.value = Nd4j.randn(m, n) * math.sqrt(3/(m.toDouble + n.toDouble))\n",
    "  }\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam\n",
    "\n",
    "We'll use Adam to train this graph and we're not going to cover the details of the algorithm\n",
    "\n",
    "## Setting up first and second moment maps for Adam Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val Array(xrows, xcols) = x_.shape\n",
    "val batchSize = 128\n",
    "val stepsPerEpoch = xrows / batchSize\n",
    "\n",
    "val firstMomentGenerator = generatorTrainables.map{ i => (i, Nd4j.zerosLike(i.value))}.toMap\n",
    "val secondMomentGenerator = generatorTrainables.map{ i => (i, Nd4j.zerosLike(i.value))}.toMap\n",
    "val firstMomentDiscriminator = discriminatorTrainables.map{ i => (i, Nd4j.zerosLike(i.value))}.toMap\n",
    "val secondMomentDiscriminator = discriminatorTrainables.map{ i => (i, Nd4j.zerosLike(i.value))}.toMap\n",
    "val t = new java.util.concurrent.atomic.AtomicInteger\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "\n",
    "We're using dropout through the discriminator.  We we do not want to do though is have dropout turned on when we are training the generator, so below is a helper function to set the training field of the dropout nodes to true or false as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "setDropoutTraining: (n: com.github.timsetsfire.nn.node.Node, training: Boolean)Unit\n"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def setDropoutTraining(n: Node, training: Boolean = false): Unit = {\n",
    "  n.asInstanceOf[Dropout[Node]].train = training\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "var stepSize: Double = 0.002 // 0.001 tf default\n",
    "val beta1: Double = 0.2  // 0.9 tf default\n",
    "val beta2: Double = 0.999  // 0.999 tf default\n",
    "val delta: Double = 1e-8\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we are going to track 16 noise vectors and how they look after $n$ number of epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we tune the generator, we \"connect\" the generator output to the discriminator input.  We forward propagate the fake data through, but we label them as real.  When we backprop we backprop from the output of the discriminator all the way back to the input to the generator.  This has the affect of updating parameters as to make the images generated look better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val noiseDataForPicture = Nd4j.rand(16,100).mul(2).sub(1)\n",
    "OutputCell.HIDDEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0\tdiscriminator -> loss: 0.546\tgenerator -> loss: 4.900\n",
      "epoch: 10\tdiscriminator -> loss: 0.447\tgenerator -> loss: 2.083\n",
      "epoch: 20\tdiscriminator -> loss: 0.590\tgenerator -> loss: 1.165\n",
      "epoch: 30\tdiscriminator -> loss: 0.626\tgenerator -> loss: 1.033\n",
      "epoch: 40\tdiscriminator -> loss: 0.645\tgenerator -> loss: 0.932\n",
      "epoch: 50\tdiscriminator -> loss: 0.656\tgenerator -> loss: 0.889\n",
      "epoch: 60\tdiscriminator -> loss: 0.656\tgenerator -> loss: 0.869\n",
      "epoch: 70\tdiscriminator -> loss: 0.655\tgenerator -> loss: 0.872\n",
      "epoch: 80\tdiscriminator -> loss: 0.656\tgenerator -> loss: 0.864\n",
      "epoch: 90\tdiscriminator -> loss: 0.653\tgenerator -> loss: 0.874\n",
      "epoch: 100\tdiscriminator -> loss: 0.648\tgenerator -> loss: 0.899\n"
     ]
    }
   ],
   "source": [
    "for(epoch <- 0 to 100) {\n",
    "\n",
    "      var loss = 0d\n",
    "      var genCost = 0d\n",
    "      var n = 0d\n",
    "    \n",
    "      for(steps <- 0 to stepsPerEpoch) {\n",
    "\n",
    "        t.addAndGet(1)\n",
    "\n",
    "        val noiseData = Nd4j.rand(batchSize,100).mul(2).sub(1)\n",
    "        val fakeLabelData = Nd4j.ones(batchSize, 1)\n",
    "\n",
    "        val generatorFeedDict: Map[Node, INDArray] = Map(\n",
    "          noise -> noiseData,\n",
    "          labels -> fakeLabelData\n",
    "        )\n",
    "\n",
    "        // generator\n",
    "          \n",
    "        // set the discriminator dropout nodes training method to false\n",
    "        discriminator.filter{ _.getClass.getSimpleName == \"Dropout\"}.foreach(d => setDropoutTraining(d, false))\n",
    "          \n",
    "        // initialize the input nodes for the generator\n",
    "        generatorFeedDict.foreach{ case (n, v) => n.forward(v)}\n",
    "        // feed forward the generator network\n",
    "        generator.foreach(_.forward())\n",
    "        \n",
    "        // pass the fake images created by the generator to the discriminator\n",
    "        images.forward(fakeImages.value)\n",
    "        // pass the fake labels to the discriminator as well.  We'll telling the discriminator\n",
    "        // that the labels are real, and forward prop\n",
    "        discriminator.foreach(_.forward())\n",
    "          \n",
    "        // now we backprop the discriminator\n",
    "        discriminator.reverse.foreach(_.backward())\n",
    "        \n",
    "        // the generator and discriminator are connected by the images node at this point\n",
    "        // so we grab the derivative out of the discriminator input images, and put it into\n",
    "        // the generator output fake images\n",
    "        fakeImages.backward(images.gradients(images).dup)\n",
    "        // and we continue to backprop\n",
    "        generator.reverse.tail.foreach(_.backward())\n",
    "       \n",
    "        // update the generator cost function\n",
    "        genCost += (cost.value.sumT*batchSize)\n",
    "\n",
    "        // update the parameters of the generator via Adam optimizer\n",
    "        for( n <- generatorTrainables) {\n",
    "          firstMomentGenerator(n).muli(beta1).addi(n.gradients(n).mul(1 - beta1))\n",
    "          secondMomentGenerator(n).muli(beta2).addi( pow(n.gradients(n),2).mul(1 - beta2))\n",
    "          val fhat = firstMomentGenerator(n).div(1 - math.pow(beta1, t.get))\n",
    "          val shat = secondMomentGenerator(n).div(1 - math.pow(beta2, t.get))\n",
    "          n.value.addi( fhat.mul(-stepSize).div(sqrt(shat).add(delta)))\n",
    "        }\n",
    "\n",
    "        // forward through the generator\n",
    "        generator.foreach(_.forward())\n",
    "        \n",
    "        // get the fake data\n",
    "        val fakeImageData = fakeImages.value\n",
    "          \n",
    "        // shuffle x\n",
    "        Nd4j.shuffle(x_,1)\n",
    "          \n",
    "        // grab batchsize rows of x\n",
    "        val realImageData = x_.getRows((0 until batchSize):_*)\n",
    "        \n",
    "        // label the real images with a 1\n",
    "        val realLabelData = Nd4j.ones(batchSize, 1)\n",
    "          \n",
    "        // label the fake images with a 0\n",
    "        val fakeLabelData0 = Nd4j.zeros(batchSize, 1)\n",
    "\n",
    "        // concatenate the data and update the feed dicationary for the generator\n",
    "        val labelData = Nd4j.concat(0, fakeLabelData0, realLabelData)\n",
    "        val imageData = Nd4j.concat(0, fakeImageData, realImageData)\n",
    "        val discriminatorFeedDict: Map[Node, INDArray] = Map(\n",
    "          images -> imageData,\n",
    "          labels -> labelData\n",
    "        )\n",
    "        \n",
    "        // turn dropout back on\n",
    "        discriminator.filter{ _.getClass.getSimpleName == \"Dropout\"}.foreach(d => setDropoutTraining(d, true))\n",
    "          \n",
    "        // feed forward\n",
    "        discriminatorFeedDict.foreach{ case (n, v) => n.forward(v)}\n",
    "        discriminator.foreach(_.forward())\n",
    "          \n",
    "        // back prop\n",
    "        discriminator.reverse.foreach(_.backward())\n",
    "          \n",
    "        // update the discriminator weights.  \n",
    "        for( n <- discriminatorTrainables) {\n",
    "          firstMomentDiscriminator(n).muli(beta1).addi(n.gradients(n).mul(1d - beta1))\n",
    "          secondMomentDiscriminator(n).muli(beta2).addi( pow(n.gradients(n),2).mul(1d - beta2))\n",
    "          val fhat = firstMomentDiscriminator(n).div(1 - math.pow(beta1, t.get))\n",
    "          val shat = secondMomentDiscriminator(n).div(1 - math.pow(beta2, t.get))\n",
    "          n.value.addi( fhat.mul(-stepSize).div(sqrt(shat).add(delta)))\n",
    "        }\n",
    "\n",
    "        loss += ((cost.value(0,0)) * images.value.shape.apply(0))\n",
    "        n += images.value.shape.apply(0)\n",
    "      }\n",
    "    \n",
    "      if(epoch % 10 == 0) {\n",
    "        print(f\"epoch: ${epoch}\")\n",
    "        print(f\"\\tdiscriminator -> loss: ${loss / n.toDouble}%2.3f\")\n",
    "        println(f\"\\tgenerator -> loss: ${genCost / (n.toDouble/2d)}%2.3f\")\n",
    "          \n",
    "      }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "306627a8-699f-47d4-9f39-4ae10c23059e",
       "version_major": 2,
       "version_minor": 0
      },
      "method": "display_data"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "noise.forward(noiseDataForPicture)\n",
    "generator.foreach(_.forward())\n",
    "\n",
    "val d1 = fakeImages.value.getRows(1,2,3,4).data.asDouble\n",
    "val d2 = fakeImages.value.getRows(5,6,7,8).data.asDouble\n",
    "val d3 = fakeImages.value.getRows(9,10,11,12).data.asDouble\n",
    "val d4 = fakeImages.value.getRows(13,14,15,0).data.asDouble\n",
    "\n",
    "val items = 4 * 28\n",
    "\n",
    "val da = (for{ i <- 0 to items} yield d1.drop(i*28).take(28) ++ (d2.drop(i*28).take(28)) ++ (d3.drop(i*28).take(28)) ++ (d4.drop(i*28).take(28))).init \n",
    "\n",
    "\n",
    "val hm = new HeatMap\n",
    "hm.data_=( da.reverse )\n",
    "hm.color_=(GradientColor.GREEN_YELLOW_WHITE )\n",
    "\n",
    "hm\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I had a hard time generating the associated figures here, so I had to run the Main method in the jar, with 100 epochs.  \n",
    "\n",
    "First Epoch\n",
    "\n",
    "<img src=\"img/genfig0.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 10\n",
    "<img src=\"img/genfig10.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 20\n",
    "<img src=\"img/genfig20.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 30\n",
    "<img src=\"img/genfig30.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 40\n",
    "<img src=\"img/genfig40.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 50\n",
    "<img src=\"img/genfig50.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 60\n",
    "<img src=\"img/genfig60.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 70\n",
    "<img src=\"img/genfig70.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 80\n",
    "<img src=\"img/genfig80.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 90\n",
    "<img src=\"img/genfig90.png\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Epoch 100\n",
    "<img src=\"img/genfig100.png\" alt=\"drawing\" width=\"350\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Scala",
   "language": "scala",
   "name": "scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "",
   "name": "Scala",
   "nbconverter_exporter": "",
   "version": "2.11.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}