C3_W2_Lab_1_Manual_Dimensionality.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "colab": {
      "name": "C3_W2_Lab_1_Manual_Dimensionality.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ns-tolu/ml_ops/blob/master/C3_W2_Lab_1_Manual_Dimensionality.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-oaTnIQF9V9m"
      },
      "source": [
        "# Ungraded lab: Manual Feature Engineering\n",
        "------------------------\n",
        " \n",
        "Welcome, during this ungraded lab you are going to perform feature engineering using TensorFlow and Keras. By having a deeper understanding of the problem you are dealing with and proposing transformations to the raw features you will see how the predictive power of your model increases. In particular you will:\n",
        "\n",
        "\n",
        "1. Define the model using feature columns.\n",
        "2. Use Lambda layers to perform feature engineering on some of these features.\n",
        "3. Compare the training history and predictions of the model before and after feature engineering.\n",
        "\n",
        "**Note**: This lab has some tweaks compared to the code you just saw on the lectures. The major one being that time-related variables are not used in the feature engineered model.\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0P7VGW6w9V9n"
      },
      "source": [
        "First, install and import the necessary packages, set up paths to work on and download the dataset.\n",
        "\n",
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GJlL4SaV9V9v"
      },
      "source": [
        "# Import the packages\n",
        "\n",
        "# Utilities\n",
        "import os\n",
        "import logging\n",
        "\n",
        "# For visualization\n",
        "import matplotlib as mpl\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "\n",
        "# For modelling\n",
        "import tensorflow as tf\n",
        "from tensorflow import feature_column as fc\n",
        "from tensorflow.keras import layers, models\n",
        "\n",
        "# Set TF logger to only print errors (dismiss warnings)\n",
        "logging.getLogger(\"tensorflow\").setLevel(logging.ERROR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RvhL6GAI9V9z"
      },
      "source": [
        "## Load taxifare dataset\n",
        "\n",
        "For this lab you are going to use a tweaked version of the [Taxi Fare dataset](https://www.kaggle.com/c/new-york-city-taxi-fare-prediction/data), which has been pre-processed and split beforehand. \n",
        "\n",
        "First, create the directory where the data is going to be saved.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SCvWVB4H9V90"
      },
      "source": [
        "if not os.path.isdir(\"/tmp/data\"):\n",
        "    os.makedirs(\"/tmp/data\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yC5BEPLxm2uP"
      },
      "source": [
        "Now download the data in `csv` format from a cloud storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rKH_WXZH9V94"
      },
      "source": [
        "!gsutil cp gs://cloud-training-demos/feat_eng/data/taxi*.csv /tmp/data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HIEJqz359V96"
      },
      "source": [
        "Let's check that the files were copied correctly and look like we expect them to."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RW6Sm4799V97"
      },
      "source": [
        "!ls -l /tmp/data/*.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EStEEdDsnXHW"
      },
      "source": [
        "Everything looks fine. Notice that there are three files, one for each split of `training`, `testing` and `validation`.\n",
        "\n",
        "## Inspect tha data\n",
        "\n",
        "Now take a look at the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1bUQlBaCCfSD"
      },
      "source": [
        "pd.read_csv('/tmp/data/taxi-train.csv').head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1ybIPxDnGTS"
      },
      "source": [
        "The data contains a total of 8 variables.\n",
        "\n",
        "The `fare_amount` is the target, the continuous value we’ll train a model to predict. This leaves you with 7 features. \n",
        "\n",
        "However this lab is going to focus on transforming the geospatial ones so the time features `hourofday` and `dayofweek` will be ignored."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pcIlWn3H9V-E"
      },
      "source": [
        "## Create an input pipeline \n",
        "\n",
        "To load the data for the model you are going to use an experimental feature of Tensorflow that lets loading directly from a `csv` file.\n",
        "\n",
        "For this you need to define some lists containing relevant information of the dataset such as the type of the columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cxC3Qjx19V-E"
      },
      "source": [
        "# Specify which column is the target\n",
        "LABEL_COLUMN = 'fare_amount'\n",
        "\n",
        "# Specify numerical columns\n",
        "# Note you should create another list with STRING_COLS if you \n",
        "# had text data but in this case all features are numerical\n",
        "NUMERIC_COLS = ['pickup_longitude', 'pickup_latitude',\n",
        "                'dropoff_longitude', 'dropoff_latitude',\n",
        "                'passenger_count', 'hourofday', 'dayofweek']\n",
        "\n",
        "\n",
        "# A function to separate features and labels\n",
        "def features_and_labels(row_data):\n",
        "    label = row_data.pop(LABEL_COLUMN)\n",
        "    return row_data, label\n",
        "\n",
        "\n",
        "# A utility method to create a tf.data dataset from a CSV file\n",
        "def load_dataset(pattern, batch_size=1, mode='eval'):\n",
        "    dataset = tf.data.experimental.make_csv_dataset(pattern, batch_size)\n",
        "    \n",
        "    dataset = dataset.map(features_and_labels)  # features, label\n",
        "    if mode == 'train':\n",
        "        # Notice the repeat method is used so this dataset will loop infinitely\n",
        "        dataset = dataset.shuffle(1000).repeat()\n",
        "        # take advantage of multi-threading; 1=AUTOTUNE\n",
        "        dataset = dataset.prefetch(1)\n",
        "    return dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uAhDWtck9V-K"
      },
      "source": [
        "## Create a DNN Model in Keras\n",
        "\n",
        "Now you will build a simple Neural Network with the numerical features as input represented by a [`DenseFeatures`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/DenseFeatures) layer (which produces a dense Tensor based on the given features), two dense layers with ReLU activation functions and an output layer with a linear activation function (since this is a regression problem).\n",
        "\n",
        "Since the model is defined using `feature columns` the first layer might look different to what you are used to. This is done by declaring two dictionaries, one for the inputs (defined as Input layers) and one for the features (defined as feature columns).\n",
        "\n",
        "Then computing the `DenseFeatures` tensor by passing in the feature columns to the constructor of the `DenseFeatures` layer and passing in the inputs to the resulting tensor (this is easier to understand with code):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5jheXumL9V-K"
      },
      "source": [
        "def build_dnn_model():\n",
        "    # input layer\n",
        "    inputs = {\n",
        "        colname: layers.Input(name=colname, shape=(), dtype='float32')\n",
        "        for colname in NUMERIC_COLS\n",
        "    }\n",
        "\n",
        "    # feature_columns\n",
        "    feature_columns = {\n",
        "        colname: fc.numeric_column(colname)\n",
        "        for colname in NUMERIC_COLS\n",
        "    }\n",
        "\n",
        "    # Constructor for DenseFeatures takes a list of numeric columns\n",
        "    # and the resulting tensor takes a dictionary of Input layers\n",
        "    dnn_inputs = layers.DenseFeatures(feature_columns.values())(inputs)\n",
        "\n",
        "    # two hidden layers of 32 and 8 units, respectively\n",
        "    h1 = layers.Dense(32, activation='relu', name='h1')(dnn_inputs)\n",
        "    h2 = layers.Dense(8, activation='relu', name='h2')(h1)\n",
        "\n",
        "    # final output is a linear activation because this is a regression problem\n",
        "    output = layers.Dense(1, activation='linear', name='fare')(h2)\n",
        "\n",
        "    # Create model with inputs and output\n",
        "    model = models.Model(inputs, output)\n",
        "\n",
        "    # compile model (Mean Squared Error is suitable for regression)\n",
        "    model.compile(optimizer='adam', \n",
        "                  loss='mse', \n",
        "                  metrics=[\n",
        "                      tf.keras.metrics.RootMeanSquaredError(name='rmse'), \n",
        "                      'mse'\n",
        "                  ])\n",
        "\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FuKLnld69V-N"
      },
      "source": [
        "We'll build our DNN model and inspect the model architecture."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EnvtEUnd9V-N"
      },
      "source": [
        "# Save compiled model into a variable\n",
        "model = build_dnn_model()\n",
        "\n",
        "# Plot the layer architecture and relationship between input features\n",
        "tf.keras.utils.plot_model(model, 'dnn_model.png', show_shapes=False, rankdir='LR')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ol9FVnFnayeU"
      },
      "source": [
        "With the model architecture defined it is time to train it!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G5R1zEJj9V-Q"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "You are going to train the model for 20 epochs using a batch size of 32."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AwW7dL8t9V-Q"
      },
      "source": [
        "NUM_EPOCHS = 20\n",
        "TRAIN_BATCH_SIZE = 32 \n",
        "NUM_TRAIN_EXAMPLES = len(pd.read_csv('/tmp/data/taxi-train.csv'))\n",
        "NUM_EVAL_EXAMPLES = len(pd.read_csv('/tmp/data/taxi-valid.csv'))\n",
        "\n",
        "print(f\"training split has {NUM_TRAIN_EXAMPLES} examples\\n\")\n",
        "print(f\"evaluation split has {NUM_EVAL_EXAMPLES} examples\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_D_sr3UQfpGj"
      },
      "source": [
        "Use the previously defined function to load the datasets from the original csv files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NWwcyV6I9V-S"
      },
      "source": [
        "# Training dataset\n",
        "trainds = load_dataset('/tmp/data/taxi-train*', TRAIN_BATCH_SIZE, 'train')\n",
        "\n",
        "# Evaluation dataset\n",
        "evalds = load_dataset('/tmp/data/taxi-valid*', 1000, 'eval').take(NUM_EVAL_EXAMPLES//1000)\n",
        "\n",
        "# Needs to be specified since the dataset is infinite \n",
        "# This happens because the repeat method was used when creating the dataset\n",
        "steps_per_epoch = NUM_TRAIN_EXAMPLES // TRAIN_BATCH_SIZE\n",
        "\n",
        "# Train the model and save the history\n",
        "history = model.fit(trainds,\n",
        "                    validation_data=evalds,\n",
        "                    epochs=NUM_EPOCHS,\n",
        "                    steps_per_epoch=steps_per_epoch)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbcJqYFu9V-V"
      },
      "source": [
        "### Visualize training curves\n",
        "\n",
        "Now lets visualize the training history of the model with the raw features:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVlOutPY9V-V"
      },
      "source": [
        "# Function for plotting metrics for a given history\n",
        "def plot_curves(history, metrics):\n",
        "    nrows = 1\n",
        "    ncols = 2\n",
        "    fig = plt.figure(figsize=(10, 5))\n",
        "\n",
        "    for idx, key in enumerate(metrics):  \n",
        "        ax = fig.add_subplot(nrows, ncols, idx+1)\n",
        "        plt.plot(history.history[key])\n",
        "        plt.plot(history.history[f'val_{key}'])\n",
        "        plt.title(f'model {key}')\n",
        "        plt.ylabel(key)\n",
        "        plt.xlabel('epoch')\n",
        "        plt.legend(['train', 'validation'], loc='upper left')\n",
        "\n",
        "\n",
        "# Plot history metrics\n",
        "plot_curves(history, ['loss', 'mse'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NUpJkrpNhm8-"
      },
      "source": [
        "The training history doesn't look very promising showing an erratic behaviour. Looks like the training process struggled to transverse the high dimensional space that the current features create. \n",
        "\n",
        "Nevertheless let's use it for prediction.\n",
        "\n",
        "Notice that the latitude and longitude values should revolve around (`37`, `45`) and (`-70`, `-78`) respectively since these are the range of coordinates for New York city."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mtNMWIg69V-a"
      },
      "source": [
        "# Define a taxi ride (a data point)\n",
        "taxi_ride = {\n",
        "    'pickup_longitude': tf.convert_to_tensor([-73.982683]),\n",
        "    'pickup_latitude': tf.convert_to_tensor([40.742104]),\n",
        "    'dropoff_longitude': tf.convert_to_tensor([-73.983766]),\n",
        "    'dropoff_latitude': tf.convert_to_tensor([40.755174]),\n",
        "    'passenger_count': tf.convert_to_tensor([3.0]),\n",
        "    'hourofday': tf.convert_to_tensor([3.0]),\n",
        "    'dayofweek': tf.convert_to_tensor([3.0]),\n",
        "}\n",
        "\n",
        "# Use the model to predict\n",
        "prediction = model.predict(taxi_ride, steps=1)\n",
        "\n",
        "# Print prediction\n",
        "print(f\"the model predicted a fare total of {float(prediction):.2f} USD for the ride.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZhFMdh7ixN6"
      },
      "source": [
        "The model predicted this particular ride to be around 12 USD. However you know the model performance is not the best as it was showcased by the training history. Let's improve it by using **Feature Engineering**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e1TNR8nI3Bdv"
      },
      "source": [
        "## Improve Model Performance Using Feature Engineering \n",
        "\n",
        "Going forward you will only use geo-spatial features as these are the most relevant when calculating the fare since this value is mostly dependant on the distance transversed:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gRSTOWC7285u"
      },
      "source": [
        "# Drop dayofweek and hourofday features\n",
        "NUMERIC_COLS = ['pickup_longitude', 'pickup_latitude',\n",
        "                'dropoff_longitude', 'dropoff_latitude']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cm5rY2B_9V-d"
      },
      "source": [
        "Since you are dealing exclusively with geospatial data you will create some transformations that are aware of this geospatial nature. This help the model make a better representation of the problem at hand.\n",
        "\n",
        "For instance the model cannot magically understand what a coordinate is supposed to represent and since the data is taken from New York only, the latitude and longitude revolve around (`37`, `45`) and (`-70`, `-78`) respectively, which is arbitrary for the model. A good first step is to scale these values. \n",
        "\n",
        "**Notice all transformations are created by defining functions**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-dOFcJJn9V-j"
      },
      "source": [
        "def scale_longitude(lon_column):\n",
        "    return (lon_column + 78)/8."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rGLNozez9V-m"
      },
      "source": [
        "def scale_latitude(lat_column):\n",
        "    return (lat_column - 37)/8."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhMqPABOm4h1"
      },
      "source": [
        "Another important fact is that the fare of a taxi ride is proportional to the distance of the ride. But as the features currently are, there is no way for the model to infer that the pair of (`pickup_latitude`, `pickup_longitude`) represent the point where the passenger started the ride and the pair (`dropoff_latitude`, `dropoff_longitude`) represent the point where the ride ended. More importantly, the model is not aware that the distance between these two points is crucial for predicting the fare.\n",
        "\n",
        "To solve this, a new feature (which is a transformation of the other ones) that provides this information is required."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x-_yQj1S9V-g"
      },
      "source": [
        "def euclidean(params):\n",
        "    lon1, lat1, lon2, lat2 = params\n",
        "    londiff = lon2 - lon1\n",
        "    latdiff = lat2 - lat1\n",
        "    return tf.sqrt(londiff*londiff + latdiff*latdiff)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ss4nvMbP9V-o"
      },
      "source": [
        "### Applying transformations\n",
        "\n",
        "Now you will define the `transform` function which will apply the previously defined transformation functions. To apply the actual transformations you will be using `Lambda` layers apply a function to values (in this case the inputs).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0kXOft7O9V-o"
      },
      "source": [
        "def transform(inputs, numeric_cols):\n",
        "\n",
        "    # Make a copy of the inputs to apply the transformations to\n",
        "    transformed = inputs.copy()\n",
        "\n",
        "    # Define feature columns\n",
        "    feature_columns = {\n",
        "        colname: tf.feature_column.numeric_column(colname)\n",
        "        for colname in numeric_cols\n",
        "    }\n",
        "\n",
        "    # Scaling longitude from range [-70, -78] to [0, 1]\n",
        "    for lon_col in ['pickup_longitude', 'dropoff_longitude']:\n",
        "        transformed[lon_col] = layers.Lambda(\n",
        "            scale_longitude,\n",
        "            name=f\"scale_{lon_col}\")(inputs[lon_col])\n",
        "\n",
        "    # Scaling latitude from range [37, 45] to [0, 1]\n",
        "    for lat_col in ['pickup_latitude', 'dropoff_latitude']:\n",
        "        transformed[lat_col] = layers.Lambda(\n",
        "            scale_latitude,\n",
        "            name=f'scale_{lat_col}')(inputs[lat_col])\n",
        "\n",
        "    # add Euclidean distance\n",
        "    transformed['euclidean'] = layers.Lambda(\n",
        "        euclidean,\n",
        "        name='euclidean')([inputs['pickup_longitude'],\n",
        "                           inputs['pickup_latitude'],\n",
        "                           inputs['dropoff_longitude'],\n",
        "                           inputs['dropoff_latitude']])\n",
        "        \n",
        "    \n",
        "    # Add euclidean distance to feature columns\n",
        "    feature_columns['euclidean'] = fc.numeric_column('euclidean')\n",
        "\n",
        "    return transformed, feature_columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5HQT79B9V-s"
      },
      "source": [
        "## Update the model\n",
        "\n",
        "Next, you'll create the DNN model now with the engineered (transformed) features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ULaSC25j9V-s"
      },
      "source": [
        "def build_dnn_model():\n",
        "    \n",
        "    # input layer (notice type of float32 since features are numeric)\n",
        "    inputs = {\n",
        "        colname: layers.Input(name=colname, shape=(), dtype='float32')\n",
        "        for colname in NUMERIC_COLS\n",
        "    }\n",
        "\n",
        "    # transformed features\n",
        "    transformed, feature_columns = transform(inputs, numeric_cols=NUMERIC_COLS)\n",
        "\n",
        "    # Constructor for DenseFeatures takes a list of numeric columns\n",
        "    # and the resulting tensor takes a dictionary of Lambda layers\n",
        "    dnn_inputs = layers.DenseFeatures(feature_columns.values())(transformed)\n",
        "\n",
        "    # two hidden layers of 32 and 8 units, respectively\n",
        "    h1 = layers.Dense(32, activation='relu', name='h1')(dnn_inputs)\n",
        "    h2 = layers.Dense(8, activation='relu', name='h2')(h1)\n",
        "\n",
        "    # final output is a linear activation because this is a regression problem\n",
        "    output = layers.Dense(1, activation='linear', name='fare')(h2)\n",
        "\n",
        "    # Create model with inputs and output\n",
        "    model = models.Model(inputs, output)\n",
        "\n",
        "    # Compile model (Mean Squared Error is suitable for regression)\n",
        "    model.compile(optimizer='adam', \n",
        "                  loss='mse', \n",
        "                  metrics=[tf.keras.metrics.RootMeanSquaredError(name='rmse'), 'mse'])\n",
        "    \n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cyNi1qgB9V-v"
      },
      "source": [
        "# Save compiled model into a variable\n",
        "model = build_dnn_model()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LI9UTFc29V-x"
      },
      "source": [
        "Let's see how the model architecture has changed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-vtdSjyJ9V-y"
      },
      "source": [
        "# Plot the layer architecture and relationship between input features\n",
        "tf.keras.utils.plot_model(model, 'dnn_model_engineered.png', show_shapes=False, rankdir='LR')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hQqxaAL-3X-V"
      },
      "source": [
        "This plot is very useful for understanding the relationships and dependencies between the original and the transformed features!\n",
        "\n",
        "**Notice that the input of the model now consists of 5 features instead of the original 7, thus reducing the dimensionality of the problem.**\n",
        "\n",
        "Let's now train the model that includes feature engineering."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M_rE0U9W9V-0"
      },
      "source": [
        "# Train the model and save the history\n",
        "history = model.fit(trainds,\n",
        "                    validation_data=evalds,\n",
        "                    epochs=NUM_EPOCHS,\n",
        "                    steps_per_epoch=steps_per_epoch)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vjGcFRx9V-2"
      },
      "source": [
        "Notice that the features `passenger_count`, `hourofday` and `dayofweek` were excluded since they were omitted when defining the input pipeline.\n",
        "\n",
        "Now lets visualize the training history of the model with the engineered features. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vfacl2gP9V-2"
      },
      "source": [
        "# Plot history metrics\n",
        "plot_curves(history, ['loss', 'mse'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WV8ZGZev9V-5"
      },
      "source": [
        "This looks a lot better than the previous training history! Now the loss and error metrics are decreasing with each epoch and both curves (train and validation) are very close to each other. Nice job!\n",
        "\n",
        "Let's do a prediction with this new model on the example we previously used. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Biqounrh9V-5"
      },
      "source": [
        "# Use the model to predict\n",
        "prediction = model.predict(taxi_ride, steps=1)\n",
        "\n",
        "# Print prediction\n",
        "print(f\"the model predicted a fare total of {float(prediction):.2f} USD for the ride.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JVZCFMi74xrj"
      },
      "source": [
        "Wow, now the model predicts a fare that is roughly half of what the previous model predicted! Looks like the model with the raw features was overestimating the fare by a great margin.\n",
        "\n",
        "Notice that you get a warning since the `taxi_ride` dictionary contains information about the unused features. You can supress it by redefining `taxi_ride` without these values but it is useful to know that Keras is smart enough to handle it on its own."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qeMvthqfugsm"
      },
      "source": [
        "**Congratulations on finishing this ungraded lab!** Now you should have a clearer understanding of the importance and impact of performing feature engineering on your data. \n",
        "\n",
        "This process is very domain-specific and requires a great understanding of the situation that is being modelled. Because of this, new techniques that switch from a manual to an automatic feature engineering have been developed and you will check some of them in an upcoming lab.\n",
        "\n",
        "\n",
        "**Keep it up!**"
      ]
    }
  ]
}