3-5-13-PretrainedModel.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3-5-13-PretrainedModel.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/z4ziad/colabs/blob/master/3-5-13-PretrainedModel.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pO4-CY_TCZZS"
      },
      "source": [
        "# Generate the Pretrained Model\n",
        "This notebook uses the pre-trained [micro_speech](https://github.com/tensorflow/tensorflow/tree/v2.4.1/tensorflow/lite/micro/examples/micro_speech) example for [TensorFlow Lite for MicroControllers](https://www.tensorflow.org/lite/microcontrollers/overview) 20 kB [Simple Audio Recognition](https://www.tensorflow.org/tutorials/sequences/audio_recognition) model to recognize keywords! **We strongly suggest you take your time working through this file to start to understand the code as we will be using a very similar file to train the model with your choice of keywords during the assignment.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtAfqQ9MGSiX"
      },
      "source": [
        "### Import packages\n",
        "Clone the TensorFlow Github Repository, which contains the relevant code required to run this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "olCcGuF7GRVO"
      },
      "source": [
        "!wget https://github.com/tensorflow/tensorflow/archive/v2.4.1.zip\n",
        "!unzip v2.4.1.zip &> 0\n",
        "!mv tensorflow-2.4.1/ tensorflow/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Important - Reverting to TF1 - Start\n",
        "We need to install Tensorflow V1 in order to use the required features for Keyword Spotting. \n",
        "\n",
        "**Run the following cell and click \"restart runtime\" that appears once it completes.**\n",
        "\n",
        "**Do not rerun the previous cells after restarting the runtime.**"
      ],
      "metadata": {
        "id": "bTILyH4VwC7K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip uninstall tensorflow -y\n",
        "!pip install tensorflow-gpu==1.15"
      ],
      "metadata": {
        "id": "5rI0qFT1wIpd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Important - Reverting to TF1 - End\n",
        "\n",
        "**Make sure to click the \"RESTART RUNTIME\" button in the output of the previous cell. Then move onto the next cell!**"
      ],
      "metadata": {
        "id": "J8l2VwdrwKTB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "assert(tf.VERSION == \"1.15.0\") #Make sure you install Tensorflow 1.15.0 and restart the runtime or this will fail!\n",
        "# Now that the runtime is set import things!\n",
        "import sys\n",
        "# We add this path so we can import the speech processing modules.\n",
        "sys.path.append(\"/content/tensorflow/tensorflow/examples/speech_commands/\")\n",
        "import input_data\n",
        "import models\n",
        "import numpy as np\n",
        "import pickle"
      ],
      "metadata": {
        "id": "Kb1aqhddwMx1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XaVtYN4nlCft"
      },
      "source": [
        "### Configure Defaults\n",
        "In this Colab we will just run with the default configurations to use the pre-trained model. However, in your assignment you will try the model to recognize a new word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ludfxbNIaegy"
      },
      "source": [
        "# A comma-delimited list of the words you want to train for.\n",
        "# All the other words you do not select will be used to train \n",
        "# an \"unknown\" label so that the model does not just recognize\n",
        "# speech but your specific words. Audio data with no spoken \n",
        "# words will be used to train a \"silence\" label.\n",
        "WANTED_WORDS = \"yes,no\"\n",
        "\n",
        "# Print the configuration to confirm it\n",
        "print(\"Spotting these words: %s\" % WANTED_WORDS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gCgeOpvY9pAi"
      },
      "source": [
        "**DO NOT MODIFY** the following constants as they include filepaths used in this notebook and data that is shared during training and inference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nd1iM1o2ymvA"
      },
      "source": [
        "# Calculate the percentage of 'silence' and 'unknown' training samples required\n",
        "# to ensure that we have equal number of samples for each label.\n",
        "number_of_labels = WANTED_WORDS.count(',') + 1\n",
        "number_of_total_labels = number_of_labels + 2 # for 'silence' and 'unknown' label\n",
        "equal_percentage_of_training_samples = int(100.0/(number_of_total_labels))\n",
        "SILENT_PERCENTAGE = equal_percentage_of_training_samples\n",
        "UNKNOWN_PERCENTAGE = equal_percentage_of_training_samples\n",
        "\n",
        "# Constants which are shared during training and inference\n",
        "PREPROCESS = 'micro'\n",
        "WINDOW_STRIDE = 20\n",
        "MODEL_ARCHITECTURE = 'tiny_conv'\n",
        "\n",
        "# Constants for training directories and filepaths\n",
        "DATASET_DIR =  'dataset/'\n",
        "LOGS_DIR = 'logs/'\n",
        "TRAIN_DIR = 'train/' # for training checkpoints and other files.\n",
        "\n",
        "# Constants for inference directories and filepaths\n",
        "import os\n",
        "MODELS_DIR = 'models'\n",
        "if not os.path.exists(MODELS_DIR):\n",
        "  os.mkdir(MODELS_DIR)\n",
        "MODEL_TF = os.path.join(MODELS_DIR, 'model.pb')\n",
        "MODEL_TFLITE = os.path.join(MODELS_DIR, 'model.tflite')\n",
        "FLOAT_MODEL_TFLITE = os.path.join(MODELS_DIR, 'float_model.tflite')\n",
        "MODEL_TFLITE_MICRO = os.path.join(MODELS_DIR, 'model.cc')\n",
        "SAVED_MODEL = os.path.join(MODELS_DIR, 'saved_model')\n",
        "\n",
        "# Constants for Quantization\n",
        "QUANT_INPUT_MIN = 0.0\n",
        "QUANT_INPUT_MAX = 26.0\n",
        "QUANT_INPUT_RANGE = QUANT_INPUT_MAX - QUANT_INPUT_MIN\n",
        "\n",
        "# Constants for audio process during Quantization and Evaluation\n",
        "SAMPLE_RATE = 16000\n",
        "CLIP_DURATION_MS = 1000\n",
        "WINDOW_SIZE_MS = 30.0\n",
        "FEATURE_BIN_COUNT = 40\n",
        "BACKGROUND_FREQUENCY = 0.8\n",
        "BACKGROUND_VOLUME_RANGE = 0.1\n",
        "TIME_SHIFT_MS = 100.0\n",
        "\n",
        "# URL for the dataset and train/val/test split\n",
        "DATA_URL = 'https://storage.googleapis.com/download.tensorflow.org/data/speech_commands_v0.02.tar.gz'\n",
        "VALIDATION_PERCENTAGE = 10\n",
        "TESTING_PERCENTAGE = 10"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UczQKtqLi7OJ"
      },
      "source": [
        "### Loading the pre-trained model\n",
        "\n",
        "These commands will download a pre-trained model checkpoint file (the output from training) that we can use to build a model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RZw3VNlnla-J"
      },
      "source": [
        "!curl -O \"https://storage.googleapis.com/download.tensorflow.org/models/tflite/speech_micro_train_2020_05_10.tgz\"\n",
        "!tar xzf speech_micro_train_2020_05_10.tgz\n",
        "TOTAL_STEPS = 15000 # used to identify which checkpoint file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XQUJLrdS-ftl"
      },
      "source": [
        "### Generate a TensorFlow Model for Inference\n",
        "\n",
        "Combine relevant training results (graph, weights, etc) into a single file for inference. This process is known as freezing a model and the resulting model is known as a frozen model/graph, as it cannot be further re-trained after this process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyc3_eLh9sAg"
      },
      "source": [
        "!rm -rf {SAVED_MODEL}\n",
        "!python tensorflow/tensorflow/examples/speech_commands/freeze.py \\\n",
        "--wanted_words=$WANTED_WORDS \\\n",
        "--window_stride_ms=$WINDOW_STRIDE \\\n",
        "--preprocess=$PREPROCESS \\\n",
        "--model_architecture=$MODEL_ARCHITECTURE \\\n",
        "--start_checkpoint=$TRAIN_DIR$MODEL_ARCHITECTURE'.ckpt-'{TOTAL_STEPS} \\\n",
        "--save_format=saved_model \\\n",
        "--output_file={SAVED_MODEL}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_DBGDxVI-nKG"
      },
      "source": [
        "### Generate a TensorFlow Lite Model\n",
        "\n",
        "Convert the frozen graph into a TensorFlow Lite model, which is fully quantized for use with embedded devices.\n",
        "\n",
        "The following cell will also print the model size, which will be under 20 kilobytes.\n",
        "\n",
        "We download the dataset to use as a representative dataset for more thoughtful post training quantization. \n",
        "\n",
        "**Note: this may take a little time as it is a relatively large file**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rNQdAplJV1fz"
      },
      "source": [
        "model_settings = models.prepare_model_settings(\n",
        "    len(input_data.prepare_words_list(WANTED_WORDS.split(','))),\n",
        "    SAMPLE_RATE, CLIP_DURATION_MS, WINDOW_SIZE_MS,\n",
        "    WINDOW_STRIDE, FEATURE_BIN_COUNT, PREPROCESS)\n",
        "audio_processor = input_data.AudioProcessor(\n",
        "    DATA_URL, DATASET_DIR,\n",
        "    SILENT_PERCENTAGE, UNKNOWN_PERCENTAGE,\n",
        "    WANTED_WORDS.split(','), VALIDATION_PERCENTAGE,\n",
        "    TESTING_PERCENTAGE, model_settings, LOGS_DIR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lBj_AyCh1cC0"
      },
      "source": [
        "with tf.Session() as sess:\n",
        "# with tf.compat.v1.Session() as sess: #replaces the above line for use with TF2.x\n",
        "  float_converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL)\n",
        "  float_tflite_model = float_converter.convert()\n",
        "  float_tflite_model_size = open(FLOAT_MODEL_TFLITE, \"wb\").write(float_tflite_model)\n",
        "  print(\"Float model is %d bytes\" % float_tflite_model_size)\n",
        "\n",
        "  converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL)\n",
        "  converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "  converter.inference_input_type = tf.lite.constants.INT8\n",
        "  # converter.inference_input_type = tf.compat.v1.lite.constants.INT8 #replaces the above line for use with TF2.x   \n",
        "  converter.inference_output_type = tf.lite.constants.INT8\n",
        "  # converter.inference_output_type = tf.compat.v1.lite.constants.INT8 #replaces the above line for use with TF2.x\n",
        "  def representative_dataset_gen():\n",
        "    for i in range(100):\n",
        "      data, _ = audio_processor.get_data(1, i*1, model_settings,\n",
        "                                         BACKGROUND_FREQUENCY, \n",
        "                                         BACKGROUND_VOLUME_RANGE,\n",
        "                                         TIME_SHIFT_MS,\n",
        "                                         'testing',\n",
        "                                         sess)\n",
        "      flattened_data = np.array(data.flatten(), dtype=np.float32).reshape(1, 1960)\n",
        "      yield [flattened_data]\n",
        "  converter.representative_dataset = representative_dataset_gen\n",
        "  tflite_model = converter.convert()\n",
        "  tflite_model_size = open(MODEL_TFLITE, \"wb\").write(tflite_model)\n",
        "  print(\"Quantized model is %d bytes\" % tflite_model_size)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeLiDZTbLkzv"
      },
      "source": [
        "### Testing the accuracy after Quantization\n",
        "\n",
        "Verify that the model we've exported is still accurate, using the TF Lite Python API and our test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQsEteKRLryJ"
      },
      "source": [
        "# Helper function to run inference\n",
        "def run_tflite_inference_testSet(tflite_model_path, model_type=\"Float\"):\n",
        "  #\n",
        "  # Load test data\n",
        "  #\n",
        "  np.random.seed(0) # set random seed for reproducible test results.\n",
        "  with tf.Session() as sess:\n",
        "  # with tf.compat.v1.Session() as sess: #replaces the above line for use with TF2.x\n",
        "    test_data, test_labels = audio_processor.get_data(\n",
        "        -1, 0, model_settings, BACKGROUND_FREQUENCY, BACKGROUND_VOLUME_RANGE,\n",
        "        TIME_SHIFT_MS, 'testing', sess)\n",
        "  test_data = np.expand_dims(test_data, axis=1).astype(np.float32)\n",
        "\n",
        "  #\n",
        "  # Initialize the interpreter\n",
        "  #\n",
        "  interpreter = tf.lite.Interpreter(tflite_model_path)\n",
        "  interpreter.allocate_tensors()\n",
        "  input_details = interpreter.get_input_details()[0]\n",
        "  output_details = interpreter.get_output_details()[0]\n",
        "  \n",
        "  #\n",
        "  # For quantized models, manually quantize the input data from float to integer\n",
        "  #\n",
        "  if model_type == \"Quantized\":\n",
        "    input_scale, input_zero_point = input_details[\"quantization\"]\n",
        "    test_data = test_data / input_scale + input_zero_point\n",
        "    test_data = test_data.astype(input_details[\"dtype\"])\n",
        "\n",
        "  #\n",
        "  # Evaluate the predictions\n",
        "  #\n",
        "  correct_predictions = 0\n",
        "  for i in range(len(test_data)):\n",
        "    interpreter.set_tensor(input_details[\"index\"], test_data[i])\n",
        "    interpreter.invoke()\n",
        "    output = interpreter.get_tensor(output_details[\"index\"])[0]\n",
        "    top_prediction = output.argmax()\n",
        "    correct_predictions += (top_prediction == test_labels[i])\n",
        "\n",
        "  print('%s model accuracy is %f%% (Number of test samples=%d)' % (\n",
        "      model_type, (correct_predictions * 100) / len(test_data), len(test_data)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l-pD52Na6jRa"
      },
      "source": [
        "# Compute float model accuracy\n",
        "run_tflite_inference_testSet(FLOAT_MODEL_TFLITE)\n",
        "\n",
        "# Compute quantized model accuracy\n",
        "run_tflite_inference_testSet(MODEL_TFLITE, model_type='Quantized')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9GT6xQCDNMll"
      },
      "source": [
        "# Testing the model on example Audio\n",
        "Now that we know the model is accurate on the test set lets explore with some hand crafted examples just how accurate the model is in the real world!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0oc-U-G9N9Du"
      },
      "source": [
        "### Load and listen to the example files\n",
        "What is interesting about them? Can you tell them all apart?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h1Qkm4riNP93"
      },
      "source": [
        "from IPython.display import HTML, Audio\n",
        "!wget --no-check-certificate --content-disposition https://github.com/tinyMLx/colabs/blob/master/yes_no.pkl?raw=true\n",
        "print(\"Wait a minute for the file to sync in the Colab and then run the next cell!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ow4x1IsVNm9H"
      },
      "source": [
        "fid = open('yes_no.pkl', 'rb')\n",
        "audio_files = pickle.load(fid)\n",
        "yes1 = audio_files['yes1']\n",
        "yes2 = audio_files['yes2']\n",
        "yes3 = audio_files['yes3']\n",
        "yes4 = audio_files['yes4']\n",
        "no1 = audio_files['no1']\n",
        "no2 = audio_files['no2']\n",
        "no3 = audio_files['no3']\n",
        "no4 = audio_files['no4']\n",
        "sr_yes1 = audio_files['sr_yes1']\n",
        "sr_yes2 = audio_files['sr_yes2']\n",
        "sr_yes3 = audio_files['sr_yes3']\n",
        "sr_yes4 = audio_files['sr_yes4']\n",
        "sr_no1 = audio_files['sr_no1']\n",
        "sr_no2 = audio_files['sr_no2']\n",
        "sr_no3 = audio_files['sr_no3']\n",
        "sr_no4 = audio_files['sr_no4']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E7_Z2zwhOBjm"
      },
      "source": [
        "Audio(yes1, rate=sr_yes1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8wslnF-lOuQJ"
      },
      "source": [
        "Audio(yes2, rate=sr_yes2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-lGGD_AkOuui"
      },
      "source": [
        "Audio(yes3, rate=sr_yes3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wuqri-nsyI1Y"
      },
      "source": [
        "Audio(yes4, rate=sr_yes4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-lVNGSZFOu3_"
      },
      "source": [
        "Audio(no1, rate=sr_no1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xSWSOuUsOvCK"
      },
      "source": [
        "Audio(no2, rate=sr_no2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SD-AnwSbOvKy"
      },
      "source": [
        "Audio(no3, rate=sr_no3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "btKBlSZwyKrS"
      },
      "source": [
        "Audio(no4, rate=sr_no4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2VrEGTt5Pt1e"
      },
      "source": [
        "### Test the model on the example files\n",
        "We first need to import a series of packages and build the loudest section tool so that we can process audio files manually to send them to our model. These packages will also be used later for you to record your own audio to test the model!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IoTsiK2Xtf3s"
      },
      "source": [
        "!pip install ffmpeg-python &> 0\n",
        "from google.colab.output import eval_js\n",
        "from base64 import b64decode\n",
        "import numpy as np\n",
        "from scipy.io.wavfile import read as wav_read\n",
        "import io\n",
        "import ffmpeg\n",
        "!pip install librosa\n",
        "import librosa\n",
        "import scipy.io.wavfile\n",
        "!git clone https://github.com/petewarden/extract_loudest_section.git\n",
        "!make -C extract_loudest_section/\n",
        "print(\"Packages Imported, Extract_Loudest_Section Built\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kuz390pSPxcG"
      },
      "source": [
        "# Helper function to run inference (on a single input this time)\n",
        "# Note: this also includes additional manual pre-processing\n",
        "TF_SESS = tf.compat.v1.InteractiveSession()\n",
        "def run_tflite_inference_singleFile(tflite_model_path, custom_audio, sr_custom_audio, model_type=\"Float\"):\n",
        "  #\n",
        "  # Preprocess the sample to get the features we pass to the model\n",
        "  #\n",
        "  # First re-sample to the needed rate (and convert to mono if needed)\n",
        "  custom_audio_resampled = librosa.resample(librosa.to_mono(np.float64(custom_audio)), sr_custom_audio, SAMPLE_RATE)\n",
        "  # Then extract the loudest one second\n",
        "  scipy.io.wavfile.write('custom_audio.wav', SAMPLE_RATE, np.int16(custom_audio_resampled))\n",
        "  !/tmp/extract_loudest_section/gen/bin/extract_loudest_section custom_audio.wav ./trimmed\n",
        "  # Finally pass it through the TFLiteMicro preprocessor to produce the \n",
        "  # spectrogram/MFCC input that the model expects\n",
        "  custom_model_settings = models.prepare_model_settings(\n",
        "      0, SAMPLE_RATE, CLIP_DURATION_MS, WINDOW_SIZE_MS,\n",
        "      WINDOW_STRIDE, FEATURE_BIN_COUNT, PREPROCESS)\n",
        "  custom_audio_processor = input_data.AudioProcessor(None, None, 0, 0, '', 0, 0,\n",
        "                                                    model_settings, None)\n",
        "  custom_audio_preprocessed = custom_audio_processor.get_features_for_wav(\n",
        "                                        'trimmed/custom_audio.wav', model_settings, TF_SESS)\n",
        "  # Reshape the output into a 1,1960 matrix as that is what the model expects\n",
        "  custom_audio_input = custom_audio_preprocessed[0].flatten()\n",
        "  test_data = np.reshape(custom_audio_input,(1,len(custom_audio_input)))\n",
        "\n",
        "  #\n",
        "  # Initialize the interpreter\n",
        "  #\n",
        "  interpreter = tf.lite.Interpreter(tflite_model_path)\n",
        "  interpreter.allocate_tensors()\n",
        "  input_details = interpreter.get_input_details()[0]\n",
        "  output_details = interpreter.get_output_details()[0]\n",
        "\n",
        "  #\n",
        "  # For quantized models, manually quantize the input data from float to integer\n",
        "  #\n",
        "  if model_type == \"Quantized\":\n",
        "    input_scale, input_zero_point = input_details[\"quantization\"]\n",
        "    test_data = test_data / input_scale + input_zero_point\n",
        "    test_data = test_data.astype(input_details[\"dtype\"])\n",
        "\n",
        "  #\n",
        "  # Run the interpreter\n",
        "  #\n",
        "  interpreter.set_tensor(input_details[\"index\"], test_data)\n",
        "  interpreter.invoke()\n",
        "  output = interpreter.get_tensor(output_details[\"index\"])[0]\n",
        "  top_prediction = output.argmax()\n",
        "\n",
        "  #\n",
        "  # Translate the output\n",
        "  #\n",
        "  top_prediction_str = ''\n",
        "  if top_prediction == 2 or top_prediction == 3:\n",
        "    top_prediction_str = WANTED_WORDS.split(',')[top_prediction-2]\n",
        "  elif top_prediction == 0:\n",
        "    top_prediction_str = 'silence'\n",
        "  else:\n",
        "    top_prediction_str = 'unknown'\n",
        "\n",
        "  print('%s model guessed the value to be %s' % (model_type, top_prediction_str))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g7eZJQUxn-Ri"
      },
      "source": [
        "# Then test the model -- do they all work as you'd expect?\n",
        "print(\"Testing yes1\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, yes1, sr_yes1, model_type=\"Quantized\")\n",
        "print(\"Testing yes2\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, yes2, sr_yes2, model_type=\"Quantized\")\n",
        "print(\"Testing yes3\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, yes3, sr_yes3, model_type=\"Quantized\")\n",
        "print(\"Testing yes4\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, yes4, sr_yes4, model_type=\"Quantized\")\n",
        "print(\"Testing no1\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, no1, sr_no1, model_type=\"Quantized\")\n",
        "print(\"Testing no2\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, no2, sr_no2, model_type=\"Quantized\")\n",
        "print(\"Testing no3\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, no3, sr_no3, model_type=\"Quantized\")\n",
        "print(\"Testing no4\")\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, no4, sr_no4, model_type=\"Quantized\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uDdZHRuFLPdH"
      },
      "source": [
        "# Testing the model with your own data!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41DVwrZBM8Jf"
      },
      "source": [
        "### Define the audio importing function\n",
        "Adapted from: https://ricardodeazambuja.com/deep_learning/2019/03/09/audio_and_video_google_colab/ and https://colab.research.google.com/drive/1Z6VIRZ_sX314hyev3Gm5gBqvm1wQVo-a#scrollTo=RtMcXr3o6gxN"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iy8gUzGtM5FK"
      },
      "source": [
        "AUDIO_HTML = \"\"\"\n",
        "<script>\n",
        "var my_div = document.createElement(\"DIV\");\n",
        "var my_p = document.createElement(\"P\");\n",
        "var my_btn = document.createElement(\"BUTTON\");\n",
        "var t = document.createTextNode(\"Press to start recording\");\n",
        "\n",
        "my_btn.appendChild(t);\n",
        "my_div.appendChild(my_btn);\n",
        "document.body.appendChild(my_div);\n",
        "\n",
        "var base64data = 0;\n",
        "var reader;\n",
        "var recorder, gumStream;\n",
        "var recordButton = my_btn;\n",
        "\n",
        "var handleSuccess = function(stream) {\n",
        "  gumStream = stream;\n",
        "  var options = {\n",
        "    bitsPerSecond: 128000, //chrome seems to ignore, always 48k\n",
        "    audioBitsPerSecond: 128000, //chrome seems to ignore, always 48k\n",
        "    mimeType : 'audio/mp4'\n",
        "    // mimeType : 'audio/webm;codecs=opus' // try me if the above fails\n",
        "  };            \n",
        "  recorder = new MediaRecorder(stream);\n",
        "  recorder.ondataavailable = function(e) {            \n",
        "    var url = URL.createObjectURL(e.data);\n",
        "    var preview = document.createElement('audio');\n",
        "    preview.controls = true;\n",
        "    preview.src = url;\n",
        "    document.body.appendChild(preview);\n",
        "\n",
        "    reader = new FileReader();\n",
        "    reader.readAsDataURL(e.data); \n",
        "    reader.onloadend = function() {\n",
        "      base64data = reader.result;\n",
        "    }\n",
        "  };\n",
        "  recorder.start();\n",
        "  };\n",
        "\n",
        "recordButton.innerText = \"Recording... press to stop\";\n",
        "\n",
        "navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
        "\n",
        "\n",
        "function toggleRecording() {\n",
        "  if (recorder && recorder.state == \"recording\") {\n",
        "      recorder.stop();\n",
        "      gumStream.getAudioTracks()[0].stop();\n",
        "      recordButton.innerText = \"Saving the recording... pls wait!\"\n",
        "  }\n",
        "}\n",
        "\n",
        "// https://stackoverflow.com/a/951057\n",
        "function sleep(ms) {\n",
        "  return new Promise(resolve => setTimeout(resolve, ms));\n",
        "}\n",
        "\n",
        "var data = new Promise(resolve=>{\n",
        "//recordButton.addEventListener(\"click\", toggleRecording);\n",
        "recordButton.onclick = ()=>{\n",
        "toggleRecording()\n",
        "\n",
        "sleep(2000).then(() => {\n",
        "  // wait 2000ms for the data to be available...\n",
        "  // ideally this should use something like await...\n",
        "  resolve(base64data.toString())\n",
        "\n",
        "});\n",
        "\n",
        "}\n",
        "});\n",
        "      \n",
        "</script>\n",
        "\"\"\"\n",
        "\n",
        "def get_audio():\n",
        "  display(HTML(AUDIO_HTML))\n",
        "  data = eval_js(\"data\")\n",
        "  binary = b64decode(data.split(',')[1])\n",
        "  \n",
        "  process = (ffmpeg\n",
        "    .input('pipe:0')\n",
        "    .output('pipe:1', format='wav', ac='1')\n",
        "    .run_async(pipe_stdin=True, pipe_stdout=True, pipe_stderr=True, quiet=True, overwrite_output=True)\n",
        "  )\n",
        "  output, err = process.communicate(input=binary)\n",
        "  \n",
        "  riff_chunk_size = len(output) - 8\n",
        "  # Break up the chunk size into four bytes, held in b.\n",
        "  q = riff_chunk_size\n",
        "  b = []\n",
        "  for i in range(4):\n",
        "      q, r = divmod(q, 256)\n",
        "      b.append(r)\n",
        "\n",
        "  # Replace bytes 4:8 in proc.stdout with the actual size of the RIFF chunk.\n",
        "  riff = output[:4] + bytes(b) + output[8:]\n",
        "\n",
        "  sr, audio = wav_read(io.BytesIO(riff))\n",
        "\n",
        "  return audio, sr\n",
        "print(\"Chrome Audio Recorder Defined\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8aLBVlkeNC8B"
      },
      "source": [
        "### Record your own audio and test the model!\n",
        "After you run the record cell wait for the stop button to appear then start recording and then press the button to stop the recording once you have said the word!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qYaZxXE0M_C9"
      },
      "source": [
        "custom_audio, sr_custom_audio = get_audio()\n",
        "print(\"DONE\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PloZwcVhpZuY"
      },
      "source": [
        "# Then test the model\n",
        "run_tflite_inference_singleFile(MODEL_TFLITE, custom_audio, sr_custom_audio, model_type=\"Quantized\")"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}