Tensorflow2 and Keras Part-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Data pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Data may also be passed into the fit method as a tf.data.Dataset() iterator\n",
    "> The from_tensor_slices() method converts the NumPy arrays into a dataset\n",
    "> The batch() and shuffle() methods chained together. \n",
    "\n",
    ">Next, the map() method invokes a method on the input images, x, that randomly flips one in two of them across\n",
    "the y-axis, effectively increasing the size of the image set\n",
    "\n",
    ">Finally, the repeat() method means that the dataset will be re-fed from the beginning when its end is\n",
    "reached (continuously)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(train_x,train_y), (test_x, test_y) = mnist.load_data()\n",
    "train_x, test_x = train_x/255.0, test_x/255.0\n",
    "epochs=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "buffer_size = 10000\n",
    "training_dataset = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(32).shuffle(10000)\n",
    "training_dataset = training_dataset.map(lambda x, y: (tf.image.random_flip_left_right(x), y))\n",
    "training_dataset = training_dataset.repeat()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "testing_dataset = tf.data.Dataset.from_tensor_slices((test_x, test_y)).batch(batch_size).shuffle(10000)\n",
    "testing_dataset = training_dataset.repeat()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the model architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now in the fit() function, we can pass the dataset directly in, as follows:\n",
    "model5 = tf.keras.models.Sequential([\n",
    " tf.keras.layers.Flatten(),\n",
    " tf.keras.layers.Dense(512,activation=tf.nn.relu),\n",
    " tf.keras.layers.Dropout(0.2),\n",
    " tf.keras.layers.Dense(10,activation=tf.nn.softmax)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compiling the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps_per_epoch = len(train_x)//batch_size #required becuase of the repeat() on the dataset\n",
    "optimiser = tf.keras.optimizers.Adam()\n",
    "model5.compile (optimizer= optimiser, loss='sparse_categorical_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1875/1875 [==============================] - 13s 6ms/step - loss: 0.3541 - accuracy: 0.8936\n",
      "Epoch 2/10\n",
      "1875/1875 [==============================] - 10s 5ms/step - loss: 0.1750 - accuracy: 0.9452\n",
      "Epoch 3/10\n",
      "1875/1875 [==============================] - 10s 5ms/step - loss: 0.1368 - accuracy: 0.9574\n",
      "Epoch 4/10\n",
      "1875/1875 [==============================] - 10s 5ms/step - loss: 0.1128 - accuracy: 0.9649\n",
      "Epoch 5/10\n",
      "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0989 - accuracy: 0.9692\n",
      "Epoch 6/10\n",
      "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0868 - accuracy: 0.9729\n",
      "Epoch 7/10\n",
      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0844 - accuracy: 0.9736\n",
      "Epoch 8/10\n",
      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0729 - accuracy: 0.9764\n",
      "Epoch 9/10\n",
      "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0683 - accuracy: 0.9784\n",
      "Epoch 10/10\n",
      "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0637 - accuracy: 0.9791\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2b02621add0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(training_dataset, epochs=epochs, steps_per_epoch = steps_per_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10/10 [==============================] - 1s 3ms/step - loss: 0.0403 - accuracy: 0.9844\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.04026519134640694, 0.984375]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.evaluate(testing_dataset,steps=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "callbacks = [\n",
    "  # Write TensorBoard logs to `./logs` directory\n",
    "  tf.keras.callbacks.TensorBoard(log_dir='log/{}/'.format(dt.datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")))\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0617 - accuracy: 0.9797 - val_loss: 0.0726 - val_accuracy: 0.9792\n",
      "Epoch 2/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0571 - accuracy: 0.9812 - val_loss: 0.0562 - val_accuracy: 0.9792\n",
      "Epoch 3/10\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0560 - accuracy: 0.9820 - val_loss: 0.0265 - val_accuracy: 1.0000\n",
      "Epoch 4/10\n",
      "1875/1875 [==============================] - 15s 8ms/step - loss: 0.0506 - accuracy: 0.9840 - val_loss: 0.0203 - val_accuracy: 0.9896\n",
      "Epoch 5/10\n",
      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0476 - accuracy: 0.9845 - val_loss: 0.0522 - val_accuracy: 0.9688\n",
      "Epoch 6/10\n",
      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0454 - accuracy: 0.9855 - val_loss: 0.0184 - val_accuracy: 0.9896\n",
      "Epoch 7/10\n",
      "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0442 - accuracy: 0.9856 - val_loss: 0.0080 - val_accuracy: 1.0000\n",
      "Epoch 8/10\n",
      "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0416 - accuracy: 0.9863 - val_loss: 0.0214 - val_accuracy: 1.0000\n",
      "Epoch 9/10\n",
      "1875/1875 [==============================] - 12s 7ms/step - loss: 0.0399 - accuracy: 0.9869 - val_loss: 0.0072 - val_accuracy: 1.0000\n",
      "Epoch 10/10\n",
      "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0424 - accuracy: 0.9861 - val_loss: 0.0263 - val_accuracy: 0.9896\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2b03e5c2d10>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(training_dataset, epochs=epochs, steps_per_epoch=steps_per_epoch,\n",
    "          validation_data=testing_dataset,\n",
    "          validation_steps=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10/10 [==============================] - 0s 3ms/step - loss: 0.0038 - accuracy: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.003776536788791418, 1.0]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.evaluate(testing_dataset,steps=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and loading Keras models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The Keras API in TensorFlow has the ability to save and restore models easily. This is done as follows, and saves the model in the current directory. Of course, a longer path may be passed here:\n",
    "\n",
    "#### Saving a model\n",
    "    \n",
    "`model.save('./model_name.h5')`\n",
    "\n",
    ">This will save the model architecture, its weights, its training state (loss, optimizer), and the state of the optimizer, so that you can carry on training the model from where you left off.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Loading a saved model is done as follows. Note that if you have compiled your model, the load will compile your model using the saved training configuration:\n",
    "\n",
    "#### Loding a model\n",
    "\n",
    "`from tensorflow.keras.models import load_model\n",
    "new_model = load_model('./model_name.h5')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">It is also possible to save just the model weights and load them with this (in which case, you must build your architecture to load the weights into):\n",
    "\n",
    "#### Saving the model weights only\n",
    "    \n",
    "    `model.save_weights('./model_weights.h5')`\n",
    "    \n",
    ">Then use the following to load it:\n",
    "\n",
    "#### Loding the weights\n",
    "    \n",
    "    `model.load_weights('./model_weights.h5')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras datasets\n",
    "\n",
    ">The following datasets are available from within Keras: boston_housing, cifar10, cifar100, fashion_mnist, imdb, mnist,and reuters.\n",
    "\n",
    ">They are all accessed with the function.\n",
    "\n",
    "`load_data()`  \n",
    "\n",
    ">For example, to load the fashion_mnist dataset, use the following:\n",
    "\n",
    "`(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using NumPy arrays with datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "number_items = 11\n",
    "number_list1 = np.arange(number_items)\n",
    "number_list2 = np.arange(number_items,number_items*2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create datasets, using the from_tensor_slices() method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_list1_dataset = tf.data.Dataset.from_tensor_slices(number_list1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create an iterator on it using the make_one_shot_iterator() method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = tf.compat.v1.data.make_one_shot_iterator(number_list1_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using them together, with the get_next method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "for item in number_list1_dataset:\n",
    "    number = iterator.get_next().numpy()\n",
    "    print(number)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note that executing this code twice in the same program run will raise an error because we are using a one-shot iterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It's also possible to access the data in batches() with the batch method. Note that the first argument is the number of elements to put in each batch and the second is the self-explanatory drop_remainder argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2]\n",
      "[3 4 5]\n",
      "[6 7 8]\n",
      "[ 9 10]\n"
     ]
    }
   ],
   "source": [
    "number_list1_dataset = tf.data.Dataset.from_tensor_slices(number_list1).batch(3, drop_remainder = False)\n",
    "iterator = tf.compat.v1.data.make_one_shot_iterator(number_list1_dataset)\n",
    "for item in number_list1_dataset:\n",
    "    number = iterator.get_next().numpy()\n",
    "    print(number)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### There is also a zip method, which is useful for presenting features and labels together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(<tf.Tensor: shape=(), dtype=int32, numpy=1>, <tf.Tensor: shape=(), dtype=string, numpy=b'a'>)\n",
      "(<tf.Tensor: shape=(), dtype=int32, numpy=2>, <tf.Tensor: shape=(), dtype=string, numpy=b'e'>)\n",
      "(<tf.Tensor: shape=(), dtype=int32, numpy=3>, <tf.Tensor: shape=(), dtype=string, numpy=b'i'>)\n",
      "(<tf.Tensor: shape=(), dtype=int32, numpy=4>, <tf.Tensor: shape=(), dtype=string, numpy=b'o'>)\n",
      "(<tf.Tensor: shape=(), dtype=int32, numpy=5>, <tf.Tensor: shape=(), dtype=string, numpy=b'u'>)\n"
     ]
    }
   ],
   "source": [
    "data_set1 = [1,2,3,4,5]\n",
    "data_set2 = ['a','e','i','o','u']\n",
    "data_set1 = tf.data.Dataset.from_tensor_slices(data_set1)\n",
    "data_set2 = tf.data.Dataset.from_tensor_slices(data_set2)\n",
    "zipped_datasets = tf.data.Dataset.zip((data_set1, data_set2))\n",
    "iterator = tf.compat.v1.data.make_one_shot_iterator(zipped_datasets)\n",
    "for item in zipped_datasets:\n",
    "    number = iterator.get_next()\n",
    "    print(number)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can concatenate two datasets as follows, using the concatenate method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<ConcatenateDataset element_spec=TensorSpec(shape=(), dtype=tf.int32, name=None)>\n",
      "tf.Tensor(1, shape=(), dtype=int32)\n",
      "tf.Tensor(2, shape=(), dtype=int32)\n",
      "tf.Tensor(3, shape=(), dtype=int32)\n",
      "tf.Tensor(5, shape=(), dtype=int32)\n",
      "tf.Tensor(7, shape=(), dtype=int32)\n",
      "tf.Tensor(11, shape=(), dtype=int32)\n",
      "tf.Tensor(13, shape=(), dtype=int32)\n",
      "tf.Tensor(17, shape=(), dtype=int32)\n",
      "tf.Tensor(19, shape=(), dtype=int32)\n",
      "tf.Tensor(23, shape=(), dtype=int32)\n",
      "tf.Tensor(29, shape=(), dtype=int32)\n",
      "tf.Tensor(31, shape=(), dtype=int32)\n",
      "tf.Tensor(37, shape=(), dtype=int32)\n",
      "tf.Tensor(41, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "datas1 = tf.data.Dataset.from_tensor_slices([1,2,3,5,7,11,13,17])\n",
    "datas2 = tf.data.Dataset.from_tensor_slices([19,23,29,31,37,41])\n",
    "datas3 = datas1.concatenate(datas2)\n",
    "print(datas3)\n",
    "iterator = tf.compat.v1.data.make_one_shot_iterator(datas3)\n",
    "for i in range(14):\n",
    "    number = iterator.get_next()\n",
    "    print(number)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can also do away with iterators altogether, as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(1, shape=(), dtype=int32)\n",
      "tf.Tensor(2, shape=(), dtype=int32)\n",
      "tf.Tensor(3, shape=(), dtype=int32)\n",
      "tf.Tensor(5, shape=(), dtype=int32)\n",
      "tf.Tensor(7, shape=(), dtype=int32)\n",
      "tf.Tensor(11, shape=(), dtype=int32)\n",
      "tf.Tensor(13, shape=(), dtype=int32)\n",
      "tf.Tensor(17, shape=(), dtype=int32)\n",
      "tf.Tensor(19, shape=(), dtype=int32)\n",
      "tf.Tensor(23, shape=(), dtype=int32)\n",
      "tf.Tensor(29, shape=(), dtype=int32)\n",
      "tf.Tensor(31, shape=(), dtype=int32)\n",
      "tf.Tensor(37, shape=(), dtype=int32)\n",
      "tf.Tensor(41, shape=(), dtype=int32)\n",
      "tf.Tensor(1, shape=(), dtype=int32)\n",
      "tf.Tensor(2, shape=(), dtype=int32)\n",
      "tf.Tensor(3, shape=(), dtype=int32)\n",
      "tf.Tensor(5, shape=(), dtype=int32)\n",
      "tf.Tensor(7, shape=(), dtype=int32)\n",
      "tf.Tensor(11, shape=(), dtype=int32)\n",
      "tf.Tensor(13, shape=(), dtype=int32)\n",
      "tf.Tensor(17, shape=(), dtype=int32)\n",
      "tf.Tensor(19, shape=(), dtype=int32)\n",
      "tf.Tensor(23, shape=(), dtype=int32)\n",
      "tf.Tensor(29, shape=(), dtype=int32)\n",
      "tf.Tensor(31, shape=(), dtype=int32)\n",
      "tf.Tensor(37, shape=(), dtype=int32)\n",
      "tf.Tensor(41, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "epochs=2\n",
    "for e in range(epochs):\n",
    "    for item in datas3:\n",
    "        print(item)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using comma-separated value (CSV)files with datasets.\n",
    "\n",
    ">CSV files are a very popular method of storing data. TensorFlow 2 contains flexible methods for dealing with them. \n",
    "\n",
    ">The main method here is tf.data.experimental.CsvDataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CSV Example 1\n",
    "With the following arguments, our dataset will consist of two items taken from each row of the filename file, both of the float type, with the first line of the file ignored and columns 1 and 2 used (column numbering is, of course, 0-based):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = [\"./size_1000.csv\"]\n",
    "record_defaults = [tf.float32] * 2 # two required float columns\n",
    "data_set = tf.data.experimental.CsvDataset(filename, record_defaults, header=True, select_cols=[1,2])\n",
    "for item in data_set:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CSV example 2\n",
    "In this example, and with the following arguments, our dataset will consist of one required float,\n",
    "one optional float with a default value of 0.0, and an int, where there is no header in the CSV file and\n",
    "only columns 1, 2, and 3 are imported:\n",
    "#file Chapter_2.ipynb\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"mycsvfile.txt\"\n",
    "record_defaults = [tf.float32, tf.constant([0.0], dtype=tf.float32), tf.int32,]\n",
    "data_set = tf.data.experimental.CsvDataset(filename, record_defaults, header=False, select_cols=[1,2,3])\n",
    "for item in data_set:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CSV example 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For our final example, our dataset will consist of two required floats and a required string, where the\n",
    "#CSV file has a header variable:\n",
    "filename = \"file1.txt\"\n",
    "record_defaults = [tf.float32, tf.float32, tf.string ,]\n",
    "dataset = tf.data.experimental.CsvDataset(filename, record_defaults, header=False)\n",
    "for item in dataset:\n",
    "    print(item[0].numpy(), item[1].numpy(),item[2].numpy().decode() )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFRecords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TFRecord format is a binary file format. For large files, it is a good choice because binary files take up less disc space, take less time to copy, and can be read very efficiently from the disc. All this can have a significant effect on the efficiency of your data pipeline and thus, the training time of your model. The format is also optimized in a\n",
    "variety of ways for use with TensorFlow. It is a little complex because data has to be converted into\n",
    "the binary format prior to storage and decoded when read back."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TFRecord example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    ">A TFRecord file is a sequence of binary strings, its structure must be specified prior to\n",
    "saving so that it can be properly written and subsequently read back.\n",
    "\n",
    ">TensorFlow has two structures for this, \n",
    "\n",
    "`tf.train.Example and tf.train.SequenceExample. `\n",
    "\n",
    ">We have to store each sample of your data in one of these structures, then serialize it, and use `tf.python_io.TFRecordWriter` to save it to disk.\n",
    "\n",
    ">In the next example, \n",
    "the  data, is first converted to the binary format and then saved to disc.\n",
    "\n",
    ">A feature is a dictionary containing the data that is passed to tf.train.Example prior to serialization and saving the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "data = np.array([10.,11.,12.,13.,14.,15.])\n",
    "def npy_to_tfrecords(fname,data):\n",
    "    writer = tf.io.TFRecordWriter(fname)\n",
    "    feature={}\n",
    "    feature['data'] = tf.train.Feature(float_list=tf.train.FloatList(value=data))\n",
    "    example = tf.train.Example(features=tf.train.Features(feature=feature))\n",
    "    serialized = example.SerializeToString()\n",
    "    writer.write(serialized)\n",
    "    writer.close()\n",
    "npy_to_tfrecords(\"./myfile.tfrecords\",data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The code to read the record back is as follows. \n",
    "\n",
    ">A parse_function function is constructed that decodes the dataset read back from the file. This requires a dictionary (keys_to_features) with the same name and structure as the saved data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([10. 11. 12. 13. 14. 15.], shape=(6,), dtype=float32)\n",
      "[10. 11. 12. 13. 14. 15.]\n",
      "12.0\n"
     ]
    }
   ],
   "source": [
    "data_set = tf.data.TFRecordDataset(\"./myfile.tfrecords\")\n",
    "def parse_function(example_proto):\n",
    "    keys_to_features = {'data':tf.io.FixedLenSequenceFeature([], dtype = tf.float32, allow_missing = True) }\n",
    "    parsed_features = tf.io.parse_single_example(serialized=example_proto, features=keys_to_features)\n",
    "    return parsed_features['data']\n",
    "data_set = data_set.map(parse_function)\n",
    "iterator = tf.compat.v1.data.make_one_shot_iterator(data_set)\n",
    "# array is retrieved as one item\n",
    "item = iterator.get_next()\n",
    "print(item)\n",
    "print(item.numpy())\n",
    "print(item[2].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TFRecord example 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = './students.tfrecords'\n",
    "dataset = {\n",
    "'ID': 61553,\n",
    "'Name': ['Jones', 'Felicity'],\n",
    "'Scores': [45.6, 97.2]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Using this, we can construct a tf.train.Example class, again using the `Feature()` method. Note how we have to encode our string:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "ID = tf.train.Feature(int64_list=tf.train.Int64List(value=[dataset['ID']]))\n",
    "Name = tf.train.Feature(bytes_list=tf.train.BytesList(value=[n.encode('utf-8') for n in dataset['Name']]))\n",
    "Scores = tf.train.Feature(float_list=tf.train.FloatList(value=dataset['Scores']))\n",
    "example = tf.train.Example(features=tf.train.Features(feature={'ID': ID, 'Name': Name, 'Scores': Scores }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Serializing and writing this record to disc is the same as TFRecord example 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "writer_rec = tf.io.TFRecordWriter(filename)\n",
    "writer_rec.write(example.SerializeToString())\n",
    "writer_rec.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To read this back, we just need to construct our parse_function function to reflect the structure of the record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_set = tf.data.TFRecordDataset(\"./students.tfrecords\")\n",
    "def parse_function(example_proto):\n",
    "    keys_to_features = {'ID':tf.io.FixedLenFeature([], dtype = tf.int64),\n",
    "    'Name':tf.io.VarLenFeature(dtype = tf.string),\n",
    "    'Scores':tf.io.VarLenFeature(dtype = tf.float32)}\n",
    "    parsed_features = tf.io.parse_single_example(serialized=example_proto, features=keys_to_features)\n",
    "    return parsed_features[\"ID\"], parsed_features[\"Name\"],parsed_features[\"Scores\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parsing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = data_set.map(parse_function)\n",
    "iterator = tf.compat.v1.data.make_one_shot_iterator(dataset)\n",
    "items = iterator.get_next()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Record is retrieved as one item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(<tf.Tensor: shape=(), dtype=int64, numpy=61553>, <tensorflow.python.framework.sparse_tensor.SparseTensor object at 0x000002B0262747F0>, <tensorflow.python.framework.sparse_tensor.SparseTensor object at 0x000002B03E5C38B0>)\n"
     ]
    }
   ],
   "source": [
    "print(items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Now we can extract our data from item (note that the string must be decoded (from bytes) where the default for our Python 3 is utf8). Note also that the string and\n",
    "the array of floats are returned as sparse arrays, and to extract them from the record, we use the sparse array value method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID:  10.0\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'values'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[1;32mIn [32], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mID: \u001b[39m\u001b[38;5;124m\"\u001b[39m,item[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mnumpy())\n\u001b[1;32m----> 2\u001b[0m name \u001b[38;5;241m=\u001b[39m \u001b[43mitem\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalues\u001b[49m\u001b[38;5;241m.\u001b[39mnumpy()\n\u001b[0;32m      3\u001b[0m name1\u001b[38;5;241m=\u001b[39m name[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mdecode()\n\u001b[0;32m      4\u001b[0m name2 \u001b[38;5;241m=\u001b[39m name[\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mdecode(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mutf8\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\tensorflow\\python\\framework\\ops.py:446\u001b[0m, in \u001b[0;36mTensor.__getattr__\u001b[1;34m(self, name)\u001b[0m\n\u001b[0;32m    437\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mT\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mastype\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mravel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtranspose\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mreshape\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mclip\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msize\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m    438\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtolist\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdata\u001b[39m\u001b[38;5;124m\"\u001b[39m}:\n\u001b[0;32m    439\u001b[0m   \u001b[38;5;66;03m# TODO(wangpeng): Export the enable_numpy_behavior knob\u001b[39;00m\n\u001b[0;32m    440\u001b[0m   \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mAttributeError\u001b[39;00m(\n\u001b[0;32m    441\u001b[0m       \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m)\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m object has no attribute \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m. \u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[0;32m    442\u001b[0m \u001b[38;5;124m    If you are looking for numpy-related methods, please run the following:\u001b[39m\n\u001b[0;32m    443\u001b[0m \u001b[38;5;124m    from tensorflow.python.ops.numpy_ops import np_config\u001b[39m\n\u001b[0;32m    444\u001b[0m \u001b[38;5;124m    np_config.enable_numpy_behavior()\u001b[39m\n\u001b[0;32m    445\u001b[0m \u001b[38;5;124m  \u001b[39m\u001b[38;5;124m\"\"\"\u001b[39m)\n\u001b[1;32m--> 446\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__getattribute__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'values'"
     ]
    }
   ],
   "source": [
    "print(\"ID: \",item[0].numpy())\n",
    "name = item[1].values.numpy()\n",
    "name1= name[0].decode()\n",
    "name2 = name[1].decode('utf8')\n",
    "print(\"Name:\",name1,\",\",name2)\n",
    "print(\"Scores: \",item[2].values.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">One-hot encoding (OHE) is where a tensor is constructed from the data labels with a 1 in each of\n",
    "the elements corresponding to a label's value, and 0 everywhere else; that is, one of the bits in the\n",
    "tensor is hot (1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One-hot Encoding Example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">In this example, we are converting a decimal value of 7 to a one-hot encoded value of 0000000100 using\n",
    "\n",
    "`the tf.one_hot() method:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 is  [0. 0. 0. 0. 0. 0. 0. 1. 0. 0.] when one-hot encoded with a depth of 10\n"
     ]
    }
   ],
   "source": [
    "z = 7\n",
    "z_train_ohe = tf.one_hot(z, depth=10).numpy()\n",
    "print(z, \"is \",z_train_ohe,\"when one-hot encoded with a depth of 10\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One-hot Encoding Example 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Using the fashion MNIST dataset.\n",
    "\n",
    ">The original labels are integers from 0 to 9, so, for example, a label of 5 becomes 0000010000 when onehot encoded, but note the difference between the index and the label stored at that index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.python.keras.datasets import fashion_mnist\n",
    "\n",
    "width, height, = 28,28\n",
    "# total classes\n",
    "n_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split feature training set into training and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "split = 50000\n",
    "(y_train, y_valid) = y_train[:split], y_train[split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### one-hot encode the labels using TensorFlow then convert back to numpy for display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_ohe = tf.one_hot(y_train, depth=n_classes).numpy()\n",
    "y_valid_ohe = tf.one_hot(y_valid, depth=n_classes).numpy()\n",
    "y_test_ohe = tf.one_hot(y_test, depth=n_classes).numpy()\n",
    "\n",
    "# show difference between the original label and a one-hot-encoded label\n",
    "i=8\n",
    "print(y_train[i]) # 'ordinary' number value of label at index i=8 is 5\n",
    "# note the difference between the index of 8 and the label at that index which is 5\n",
    "print(y_train_ohe[i]) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}