Fashion_MNIST_Competition.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/todnewman/coe_training/blob/master/Fashion_MNIST_Competition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHwxGemYV_FT"
      },
      "source": [
        "# Fashion MNIST Competition!\n",
        "**Author**: jlwhelan28\n",
        "\n",
        "**Source**: Forked from https://github.com/todnewman/coe_training\n",
        "\n",
        "## Problem\n",
        "\n",
        "Classify images from the \"Fashion MNIST\" data set.   Optimize the test accuracy.\n",
        "\n",
        "## Metrics\n",
        "\n",
        "This competition is evaluated on the mean Dice coefficient. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:![alt text](https://user-images.githubusercontent.com/26015273/41822460-2ca0a90a-77f0-11e8-9c71-7e88fa6b5c61.gif)\n",
        "\n",
        "\n",
        "The double sum is over the observations `i`, whose number is `N`, and the categories `c`, whose number is `C`. The term `1_{y_i \\in C_c}` is the indicator function of the `i`th observation belonging to the `c`th category. The `p_{model}[y_i \\in C_c]` is the probability predicted by the model for the `i`th observation to belong to the `c`th category. When there are more than two categories, the neural network outputs a vector of `C` probabilities, each giving the probability that the network input should be classified as belonging to the respective category. When the number of categories is just two, the neural network outputs a single probability `\\hat{y}_i`, with the other one being `1` minus the output. This is why the binary cross entropy looks a bit different from categorical cross entropy, despite being a special case of it.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "This dataset is the Fashion MNIST dataset\n",
        "\n",
        "Recently, the researchers at Zalando, an e-commerce company, introduced Fashion MNIST as a drop-in replacement for the original MNIST dataset. Like MNIST, Fashion MNIST consists of a training set consisting of 60,000 examples belonging to 10 different classes and a test set of 10,000 examples. Each training example is a gray-scale image, 28x28 in size. The authors of the work further claim that the Fashion MNIST should actually replace MNIST dataset for benchmarking of new Machine Learning or Computer Vision models.\n",
        "\n",
        "Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255. The training and test data sets have 785 columns. The first column consists of the class labels (see above), and represents the article of clothing. The rest of the columns contain the pixel-values of the associated image.\n",
        "\n",
        "The Labels are:  \n",
        "0 T-shirt/top\n",
        "1 Trouser\n",
        "2 Pullover\n",
        "3 Dress\n",
        "4 Coat\n",
        "5 Sandal\n",
        "6 Shirt\n",
        "7 Sneaker\n",
        "8 Bag\n",
        "9 Ankle boot \n",
        "\n",
        "## Objective\n",
        "\n",
        "In this competition, you can try different variations of the CNN model given as a reference, you may evaluate techniques to squeeze more performance out of a CNN, or you might even try a completely different model, neural network or otherwise.  You will note that there are tips/tricks/techniques documented in many locations on the internet that could be useful.\n",
        "\n",
        "## Rules and Timeline\n",
        "\n",
        "The primary measure for the competition will be the accuracy of prediction on the test data.  Ties will be broken by Precision accuracy first, then Recall Accuracy if needed.\n",
        "\n",
        "The results will be revealed at the end of the last day of class.  Please submit your Metrics blocks (Starts with SUBMIT... and ends with END SUBMISSION) to instructors (wtnewman@raytheon.com) before lunch.\n",
        "\n",
        "A prize will be given to the top finisher(s) based on the judgement of the instructor and the availability of prizes.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "use_local_gpu = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "V0A2_Mo0KxWc"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "if use_local_gpu:\n",
        "    os.environ[\"LD_LIBRARY_PATH\"] = os.environ.get(\"LD_LIBRARY_PATH\", \"\") + (f\":{os.environ['CONDA_PREFIX']}/lib\")\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from keras.datasets import fashion_mnist\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Flatten,  Conv2D, MaxPooling2D, Activation, BatchNormalization\n",
        "from keras import backend as K\n",
        "from keras.callbacks import EarlyStopping\n",
        "from sklearn import metrics\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import train_test_split\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "from keras.callbacks import TensorBoard,  ModelCheckpoint\n",
        "from keras.layers import LeakyReLU\n",
        "%matplotlib inline\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = \"2\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/physical_device:CPU:0\n"
          ]
        }
      ],
      "source": [
        "if use_local_gpu:\n",
        "    found_device = tf.config.list_physical_devices(\"GPU\")[0].name\n",
        "    device_name = \"/gpu:0\"\n",
        "    print(found_device)\n",
        "else:\n",
        "    found_device = tf.config.list_physical_devices(\"CPU\")[0].name\n",
        "    device_name = \"/cpu:0\"\n",
        "    print(found_device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3y6VjlzZLKPB"
      },
      "source": [
        "## Set Up Variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "LtL1H333LHGz"
      },
      "outputs": [],
      "source": [
        "epochs = 30                 # Number of Training Epochs\n",
        "num_classes = 10            # This is the number of classes in the Fashion MNIST dataset\n",
        "batch_size = 128            # This parameter can be adjusted\n",
        "img_rows, img_cols = 28, 28 # Pixel sizes of the Images in the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SW52B7dT9H5"
      },
      "source": [
        "## Gather and Process Fashion MNIST data\n",
        "\n",
        "1. First, collect the data from Keras (our goal is someday that our organizational data is this easy to get!)\n",
        "2. Then split into train and test sets.\n",
        "3. Next we need to process the data into the proper shape for the CNN\n",
        "4. Then scale the floats to land between 0 and 1.  Often times we use sklearn's MinMaxScaler for this, but in this case we're going for simplicity.\n",
        "5. Next take the y_train and y_test labels and encode them one-hot.  This will enable the CNN to function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "GZENio2YLPUy"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(12000, 10)\n"
          ]
        }
      ],
      "source": [
        "# Grab the data from the keras repository\n",
        "\n",
        "mnist_data = fashion_mnist.load_data()\n",
        "x = mnist_data[0][0]\n",
        "y = mnist_data[0][1]\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=41)\n",
        "\n",
        "# Process the date into the right tensor shape.  This is a good practice, but\n",
        "# usually tensorflow uses channels last (the 'else' here)\n",
        "\n",
        "if K.image_data_format() == \"channels first\":\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "    \n",
        "#\n",
        "#  Cast to a 32 bit float and then scale so the value is a float between 0 and 1\n",
        "    \n",
        "x_train = x_train.astype(\"float32\")\n",
        "x_test = x_test.astype(\"float32\")\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "#\n",
        "# Convert Class Vector to Binary Class Matrices (one-hot encoding).\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
        "print(y_test.shape)\n",
        "\n",
        "#\n",
        "# Function to decode one-hot encoding later on when we want to evaluate performance.\n",
        "def decode_one_hot(y):\n",
        "    y_classes = [np.argmax(yi, axis=None, out=None) for yi in y]\n",
        "    return y_classes\n",
        "\n",
        "'''\n",
        "\n",
        "Below we're experimenting with the Keras ImageDataGenerator.  From my experience, if these parameters\n",
        "are set too aggressively, the loss/accuracy will either never improve or it will take too long to improve.\n",
        "Below is an example of a complex data augmentation regime.  This is just for reference.  See my more simple\n",
        "one at the bottom.\n",
        "\n",
        "    \n",
        "datagen = ImageDataGenerator(rotation_range=0.5, \n",
        "                                 zoom_range=0.1,\n",
        "                                 featurewise_center=True,\n",
        "                                 #featurewise_std_normalization=True,\n",
        "                                 width_shift_range=0.1, \n",
        "                                 height_shift_range=0.1, \n",
        "                                 shear_range=0.1,\n",
        "                                 horizontal_flip=True, \n",
        "                                 fill_mode=\"nearest\")\n",
        "'''\n",
        "#\n",
        "#  Set up our Image Augmentation Data Generator\n",
        "#\n",
        "datagen = ImageDataGenerator(\n",
        "    rotation_range=5,\n",
        "    # shear_range=0.1,\n",
        "    # horizontal_flip=True,\n",
        "    # fill_mode=\"nearest\",\n",
        ")\n",
        "datagen.fit(x_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FcFnUPDLvnO"
      },
      "source": [
        "## Build the Model\n",
        "\n",
        "* In this example, we define the below block as a Sequential Model. \n",
        "* See the excellent [Keras Documentation](https://keras.io/guides/sequential_model/) on Sequential Models for info.\n",
        "* Many of these parameters can be experimented with.  The documentation will help you understand how much to experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "8EWdo8mALXB9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.\n"
          ]
        }
      ],
      "source": [
        "#\n",
        "# This is what is known as a Tensorflow (Keras) Sequential model\n",
        "# We will talk at some level about each of these layer types in class.\n",
        "#\n",
        "model = Sequential()\n",
        "\n",
        "model.add(Conv2D(64, kernel_size=(3,3), input_shape=input_shape, activation=\"relu\"))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "model.add(Dropout(0.1))\n",
        "model.add(Conv2D(32, kernel_size=(3,3), activation=\"relu\"))\n",
        "model.add(Dropout(0.1))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "model.add(Dropout(0.1))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(256, activation=\"relu\"))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(num_classes, activation=\"softmax\"))\n",
        "\n",
        "my_callbacks = [ModelCheckpoint('model_out.hdf5', monitor='acc', save_best_only=True, period=1)]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvIfjap-L8TO"
      },
      "source": [
        "## Fit and Evaluate the Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "n5U20dV1L-eP"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/Kepler/anaconda3/envs/tensorflow/lib/python3.7/site-packages/ipykernel_launcher.py:34: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "188/187 [==============================] - ETA: 0s - loss: 0.7958 - accuracy: 0.7072WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.7958 - accuracy: 0.7072 - val_loss: 0.4957 - val_accuracy: 0.8151\n",
            "Epoch 2/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.5019 - accuracy: 0.8147WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.5019 - accuracy: 0.8147 - val_loss: 0.4116 - val_accuracy: 0.8438\n",
            "Epoch 3/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.4367 - accuracy: 0.8412WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.4366 - accuracy: 0.8412 - val_loss: 0.3620 - val_accuracy: 0.8676\n",
            "Epoch 4/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3964 - accuracy: 0.8542WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 75ms/step - loss: 0.3960 - accuracy: 0.8544 - val_loss: 0.3482 - val_accuracy: 0.8710\n",
            "Epoch 5/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3765 - accuracy: 0.8626WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.3763 - accuracy: 0.8627 - val_loss: 0.3259 - val_accuracy: 0.8809\n",
            "Epoch 6/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.3585 - accuracy: 0.8698WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.3585 - accuracy: 0.8698 - val_loss: 0.3251 - val_accuracy: 0.8798\n",
            "Epoch 7/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3411 - accuracy: 0.8747WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.3410 - accuracy: 0.8746 - val_loss: 0.3027 - val_accuracy: 0.8885\n",
            "Epoch 8/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3302 - accuracy: 0.8782WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.3304 - accuracy: 0.8780 - val_loss: 0.2962 - val_accuracy: 0.8905\n",
            "Epoch 9/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3170 - accuracy: 0.8820WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.3172 - accuracy: 0.8819 - val_loss: 0.2974 - val_accuracy: 0.8902\n",
            "Epoch 10/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3098 - accuracy: 0.8862WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.3096 - accuracy: 0.8861 - val_loss: 0.2781 - val_accuracy: 0.8978\n",
            "Epoch 11/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.3017 - accuracy: 0.8885WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.3015 - accuracy: 0.8886 - val_loss: 0.2728 - val_accuracy: 0.9001\n",
            "Epoch 12/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2916 - accuracy: 0.8920WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2918 - accuracy: 0.8919 - val_loss: 0.2709 - val_accuracy: 0.8998\n",
            "Epoch 13/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2895 - accuracy: 0.8943WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2895 - accuracy: 0.8942 - val_loss: 0.2656 - val_accuracy: 0.9020\n",
            "Epoch 14/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2819 - accuracy: 0.8972WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2819 - accuracy: 0.8972 - val_loss: 0.2633 - val_accuracy: 0.9026\n",
            "Epoch 15/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2707 - accuracy: 0.8998WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2707 - accuracy: 0.8998 - val_loss: 0.2595 - val_accuracy: 0.9026\n",
            "Epoch 16/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2672 - accuracy: 0.9011WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2673 - accuracy: 0.9010 - val_loss: 0.2610 - val_accuracy: 0.9047\n",
            "Epoch 17/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2665 - accuracy: 0.9005WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2665 - accuracy: 0.9005 - val_loss: 0.2498 - val_accuracy: 0.9084\n",
            "Epoch 18/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2557 - accuracy: 0.9063WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.2560 - accuracy: 0.9062 - val_loss: 0.2603 - val_accuracy: 0.9043\n",
            "Epoch 19/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2574 - accuracy: 0.9052WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.2574 - accuracy: 0.9052 - val_loss: 0.2493 - val_accuracy: 0.9097\n",
            "Epoch 20/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2494 - accuracy: 0.9063WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.2496 - accuracy: 0.9063 - val_loss: 0.2548 - val_accuracy: 0.9070\n",
            "Epoch 21/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2455 - accuracy: 0.9081WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.2455 - accuracy: 0.9080 - val_loss: 0.2440 - val_accuracy: 0.9106\n",
            "Epoch 22/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2431 - accuracy: 0.9102WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2429 - accuracy: 0.9102 - val_loss: 0.2469 - val_accuracy: 0.9118\n",
            "Epoch 23/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2366 - accuracy: 0.9115WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2364 - accuracy: 0.9116 - val_loss: 0.2436 - val_accuracy: 0.9108\n",
            "Epoch 24/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2357 - accuracy: 0.9118WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2355 - accuracy: 0.9119 - val_loss: 0.2407 - val_accuracy: 0.9109\n",
            "Epoch 25/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2311 - accuracy: 0.9153WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2312 - accuracy: 0.9152 - val_loss: 0.2634 - val_accuracy: 0.9031\n",
            "Epoch 26/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2280 - accuracy: 0.9132WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2282 - accuracy: 0.9131 - val_loss: 0.2442 - val_accuracy: 0.9107\n",
            "Epoch 27/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2275 - accuracy: 0.9150WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2277 - accuracy: 0.9149 - val_loss: 0.2367 - val_accuracy: 0.9124\n",
            "Epoch 28/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2234 - accuracy: 0.9165WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2234 - accuracy: 0.9165 - val_loss: 0.2426 - val_accuracy: 0.9119\n",
            "Epoch 29/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2210 - accuracy: 0.9169WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.2210 - accuracy: 0.9169 - val_loss: 0.2387 - val_accuracy: 0.9127\n",
            "Epoch 30/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2189 - accuracy: 0.9179WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2189 - accuracy: 0.9179 - val_loss: 0.2385 - val_accuracy: 0.9132\n",
            "Epoch 31/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2138 - accuracy: 0.9203WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2138 - accuracy: 0.9203 - val_loss: 0.2359 - val_accuracy: 0.9128\n",
            "Epoch 32/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2117 - accuracy: 0.9210WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.2117 - accuracy: 0.9210 - val_loss: 0.2385 - val_accuracy: 0.9137\n",
            "Epoch 33/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2083 - accuracy: 0.9208WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.2085 - accuracy: 0.9206 - val_loss: 0.2350 - val_accuracy: 0.9143\n",
            "Epoch 34/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2085 - accuracy: 0.9208WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2085 - accuracy: 0.9208 - val_loss: 0.2327 - val_accuracy: 0.9155\n",
            "Epoch 35/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.2081 - accuracy: 0.9212WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.2081 - accuracy: 0.9212 - val_loss: 0.2328 - val_accuracy: 0.9155\n",
            "Epoch 36/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2030 - accuracy: 0.9233WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.2032 - accuracy: 0.9232 - val_loss: 0.2421 - val_accuracy: 0.9133\n",
            "Epoch 37/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.2028 - accuracy: 0.9234WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.2027 - accuracy: 0.9234 - val_loss: 0.2337 - val_accuracy: 0.9162\n",
            "Epoch 38/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.1981 - accuracy: 0.9251WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 79ms/step - loss: 0.1981 - accuracy: 0.9251 - val_loss: 0.2309 - val_accuracy: 0.9185\n",
            "Epoch 39/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1966 - accuracy: 0.9248WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.1969 - accuracy: 0.9247 - val_loss: 0.2331 - val_accuracy: 0.9144\n",
            "Epoch 40/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1979 - accuracy: 0.9256WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.1978 - accuracy: 0.9256 - val_loss: 0.2332 - val_accuracy: 0.9183\n",
            "Epoch 41/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1947 - accuracy: 0.9264WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.1947 - accuracy: 0.9265 - val_loss: 0.2312 - val_accuracy: 0.9184\n",
            "Epoch 42/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1918 - accuracy: 0.9274WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.1918 - accuracy: 0.9274 - val_loss: 0.2350 - val_accuracy: 0.9167\n",
            "Epoch 43/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.1896 - accuracy: 0.9265WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.1896 - accuracy: 0.9265 - val_loss: 0.2320 - val_accuracy: 0.9175\n",
            "Epoch 44/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1881 - accuracy: 0.9293WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 77ms/step - loss: 0.1881 - accuracy: 0.9292 - val_loss: 0.2362 - val_accuracy: 0.9172\n",
            "Epoch 45/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1887 - accuracy: 0.9283WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 79ms/step - loss: 0.1889 - accuracy: 0.9281 - val_loss: 0.2341 - val_accuracy: 0.9174\n",
            "Epoch 46/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.1871 - accuracy: 0.9290WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 78ms/step - loss: 0.1871 - accuracy: 0.9290 - val_loss: 0.2428 - val_accuracy: 0.9158\n",
            "Epoch 47/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1876 - accuracy: 0.9286WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.1877 - accuracy: 0.9285 - val_loss: 0.2313 - val_accuracy: 0.9207\n",
            "Epoch 48/50\n",
            "188/187 [==============================] - ETA: 0s - loss: 0.1842 - accuracy: 0.9303WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 15s 77ms/step - loss: 0.1842 - accuracy: 0.9303 - val_loss: 0.2380 - val_accuracy: 0.9175\n",
            "Epoch 49/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1830 - accuracy: 0.9304WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.1827 - accuracy: 0.9304 - val_loss: 0.2321 - val_accuracy: 0.9203\n",
            "Epoch 50/50\n",
            "187/187 [============================>.] - ETA: 0s - loss: 0.1802 - accuracy: 0.9301WARNING:tensorflow:Can save best model only with acc available, skipping.\n",
            "187/187 [==============================] - 14s 76ms/step - loss: 0.1803 - accuracy: 0.9300 - val_loss: 0.2309 - val_accuracy: 0.9208\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2309 - accuracy: 0.9208\n",
            "375/375 [==============================] - 1s 2ms/step\n",
            "Model: \"sequential_4\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d_8 (Conv2D)           (None, 26, 26, 64)        640       \n",
            "                                                                 \n",
            " max_pooling2d_8 (MaxPooling  (None, 13, 13, 64)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " dropout_16 (Dropout)        (None, 13, 13, 64)        0         \n",
            "                                                                 \n",
            " conv2d_9 (Conv2D)           (None, 11, 11, 32)        18464     \n",
            "                                                                 \n",
            " dropout_17 (Dropout)        (None, 11, 11, 32)        0         \n",
            "                                                                 \n",
            " max_pooling2d_9 (MaxPooling  (None, 5, 5, 32)         0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " dropout_18 (Dropout)        (None, 5, 5, 32)          0         \n",
            "                                                                 \n",
            " flatten_4 (Flatten)         (None, 800)               0         \n",
            "                                                                 \n",
            " dense_8 (Dense)             (None, 256)               205056    \n",
            "                                                                 \n",
            " dropout_19 (Dropout)        (None, 256)               0         \n",
            "                                                                 \n",
            " dense_9 (Dense)             (None, 10)                2570      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 226,730\n",
            "Trainable params: 226,730\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# Flag to determine whether we use Keras' Image augmentation data generator\n",
        "augmentation = True\n",
        "\n",
        "#\n",
        "# Compile the model so we can fit it. Researching loss functions and optimizers\n",
        "# might be a good thing to do.\n",
        "#\n",
        "model.compile(loss=keras.losses.categorical_crossentropy, \n",
        "              optimizer=\"adam\", \n",
        "              metrics=['accuracy'])\n",
        "\n",
        "with tf.device(device_name):\n",
        "    if not augmentation:\n",
        "        #\n",
        "        # Fit the model.  Once the model is trained we'll evaluate the performance.\n",
        "        print('not using image augmentation')\n",
        "        hist = model.fit(x_train, y_train,\n",
        "            batch_size=batch_size,\n",
        "            epochs=epochs,\n",
        "            verbose=1,\n",
        "            validation_data=(x_test, y_test),\n",
        "            callbacks=my_callbacks)\n",
        "    else:\n",
        "    # fits the model on batches with real-time data augmentation:\n",
        "        hist = model.fit_generator(\n",
        "            datagen.flow(\n",
        "                x_train, y_train, batch_size=batch_size\n",
        "            ),\n",
        "            steps_per_epoch=len(x_train) / batch_size,\n",
        "            validation_data=(x_test, y_test),\n",
        "            epochs=epochs,\n",
        "            verbose=1,\n",
        "            callbacks=my_callbacks,\n",
        "            workers = 2\n",
        "        )\n",
        "\n",
        "\n",
        "    score = model.evaluate(x_test, y_test)\n",
        "\n",
        "#\n",
        "# Predict on the test data and pass to metrics function\n",
        "yhat = np.argmax(model.predict(x_test), axis=-1)\n",
        "y_dec = decode_one_hot(y_test)\n",
        "\n",
        "\n",
        "\n",
        "model.summary()\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "SUBMIT THIS BLOCK for the Competition\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.86      0.91      0.88      1211\n",
            "           1       0.99      0.99      0.99      1204\n",
            "           2       0.87      0.87      0.87      1216\n",
            "           3       0.91      0.94      0.92      1188\n",
            "           4       0.87      0.85      0.86      1252\n",
            "           5       0.99      0.98      0.99      1172\n",
            "           6       0.79      0.75      0.77      1189\n",
            "           7       0.96      0.98      0.97      1180\n",
            "           8       0.99      0.98      0.99      1162\n",
            "           9       0.98      0.97      0.97      1226\n",
            "\n",
            "    accuracy                           0.92     12000\n",
            "   macro avg       0.92      0.92      0.92     12000\n",
            "weighted avg       0.92      0.92      0.92     12000\n",
            "\n",
            "Testing Loss: 0.2308681458234787\n",
            "Testing Accuracy: 0.9208333492279053\n",
            "END SUBMISSION BLOCK\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(\"\\nSUBMIT THIS BLOCK for the Competition\\n\")\n",
        "print(metrics.classification_report(y_dec, yhat))\n",
        "print(\"Testing Loss:\", score[0])\n",
        "print(\"Testing Accuracy:\", score[1])\n",
        "print(\"END SUBMISSION BLOCK\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTXLg2MJMFhy"
      },
      "source": [
        "## Plot the accuracy vs. validation accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P2Z97VRkMIRh"
      },
      "outputs": [],
      "source": [
        "epoch_list = list(range(1, len(hist.history['accuracy']) + 1))\n",
        "plt.plot(epoch_list, hist.history['accuracy'], epoch_list, hist.history['val_accuracy'])\n",
        "plt.legend((\"Training Accuracy\", \"Validation Accuracy\"))\n",
        "plt.show()\n",
        "\n",
        "plt.plot(epoch_list, hist.history['loss'], epoch_list, hist.history['val_loss'])\n",
        "plt.legend((\"Training Loss\", \"Validation Loss\"))\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-imOV3gbDji"
      },
      "source": [
        "## Visualization of Performance on the Test Set\n",
        "\n",
        "Here is a visualization of how well our classifier can do inference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hfsF8TlLwarT"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "from imutils import build_montages\n",
        "from google.colab.patches import cv2_imshow\n",
        "\n",
        "# initialize our list of output images\n",
        "images = []\n",
        "\n",
        "labelNames = [\"top\", \"trouser\", \"pullover\", \"dress\", \"coat\",\n",
        "\t\"sandal\", \"shirt\", \"sneaker\", \"bag\", \"ankle boot\"]\n",
        " \n",
        "# randomly select a few testing fashion items\n",
        "for i in np.random.choice(np.arange(0, len(y_test)), size=(16,)):\n",
        "\t# classify the clothing\n",
        "\tprobs = model.predict(x_test[np.newaxis, i])\n",
        "\tprediction = probs.argmax(axis=1)\n",
        "\tlabel = labelNames[prediction[0]]\n",
        " \n",
        "\t# extract the image from the testData if using \"channels_first\"\n",
        "\t# ordering\n",
        "\tif K.image_data_format() == \"channels_first\":\n",
        "\t\timage = (x_test[i][0] * 255).astype(\"uint8\")\n",
        " \n",
        "\t# otherwise we are using \"channels_last\" ordering\n",
        "\telse:\n",
        "\t\timage = (x_test[i] * 255).astype(\"uint8\")\n",
        "    # initialize the text label color as green (correct)\n",
        "\tcolor = (0, 255, 0)\n",
        " \n",
        "\t# otherwise, the class label prediction is incorrect\n",
        "\tif prediction[0] != np.argmax(y_test[i]):\n",
        "\t\tcolor = (0, 0, 255)\n",
        " \n",
        "\t# merge the channels into one image and resize the image from\n",
        "\t# 28x28 to 96x96 so we can better see it and then draw the\n",
        "\t# predicted label on the image\n",
        "\timage = cv2.merge([image] * 3)\n",
        "\timage = cv2.resize(image, (96, 96), interpolation=cv2.INTER_LINEAR)\n",
        "\tcv2.putText(image, label, (5, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.75,\n",
        "\t\tcolor, 2)\n",
        " \n",
        "\t# add the image to our list of output images\n",
        "\timages.append(image)\n",
        "# construct the montage for the images\n",
        "montage = build_montages(images, (96, 96), (4, 4))[0]\n",
        " \n",
        "# show the output montage\n",
        "cv2_imshow( montage)\n",
        "cv2.waitKey(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dMDjuN_anxKM"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Fashion MNIST Competition",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.7.13 ('tensorflow')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "9ffe0057f7f573817286207e663cabc03af1ad7947e6e0e3f5918ea0a7ce3ef1"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}