2_Augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data augmentation\n",
    "\n",
    "Data augmentation is a straighforward way to artificially increase the size\n",
    "of the dataset while embedding invariances into the model.\n",
    "\n",
    "## AWS info\n",
    "Run this notebook with the **Data Science** image and **ml.t3.2xlarge** instance type.\n",
    "\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from imblearn.ensemble import BalancedRandomForestClassifier\n",
    "from sklearn import metrics\n",
    "from tqdm.auto import tqdm\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "%matplotlib inline\n",
    "\n",
    "# For reproducibility\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to your extracted windows\n",
    "DATASET_PATH = 'processed_data/'\n",
    "X_FEATS_PATH = 'X_feats.pkl'  # path to your extracted features, if have one\n",
    "print(f'Content of {DATASET_PATH}')\n",
    "print(os.listdir(DATASET_PATH))\n",
    "\n",
    "X = np.load(DATASET_PATH+'X.npy', mmap_mode='r')\n",
    "Y = np.load(DATASET_PATH+'Y.npy')\n",
    "T = np.load(DATASET_PATH+'T.npy')\n",
    "pid = np.load(DATASET_PATH+'pid.npy')\n",
    "X_feats = pd.read_pickle(DATASET_PATH+'X_feats.pkl')\n",
    "\n",
    "# As before, let's map the text annotations to simplified labels\n",
    "ANNO_LABEL_DICT_PATH = 'capture24/annotation-label-dictionary.csv'\n",
    "anno_label_dict = pd.read_csv(ANNO_LABEL_DICT_PATH, index_col='annotation', dtype='string')\n",
    "Y = anno_label_dict.loc[Y, 'label:Willetts2018'].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hold out participants P101-P151 for testing (51 participants)\n",
    "test_ids = [f'P{i}' for i in range(101,152)]\n",
    "mask_test = np.isin(pid, test_ids)\n",
    "mask_train = ~mask_test\n",
    "X_train, Y_train, T_train, pid_train = \\\n",
    "    X_feats[mask_train], Y[mask_train], T[mask_train], pid[mask_train]\n",
    "X_test, Y_test, T_test, pid_test = \\\n",
    "    X_feats[mask_test], Y[mask_test], T[mask_test], pid[mask_test]\n",
    "print(\"Shape of X_train:\", X_train.shape)\n",
    "print(\"Shape of X_test:\", X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a random forest classifier\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = BalancedRandomForestClassifier(\n",
    "    n_estimators=1000,\n",
    "    replacement=True,\n",
    "    sampling_strategy='not minority',\n",
    "    n_jobs=-1,\n",
    "    random_state=42,\n",
    "    verbose=1\n",
    ")\n",
    "clf.fit(X_train, Y_train)\n",
    "\n",
    "Y_test_pred = clf.predict(X_test)\n",
    "print('\\nClassifier performance')\n",
    "print('Out of sample:\\n', metrics.classification_report(Y_test, Y_test_pred, zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Robustness to unforseen scenarios\n",
    "\n",
    "What if the subjects in the test set wore the device differently from\n",
    "those in the training set? For example, suppose that all the subjects in the\n",
    "training set were right-handed, then the model could underperform on a test\n",
    "subject who is left-handed. This would more or less result in the device having\n",
    "been rotated. Another typical scenario happens when we want our model to be\n",
    "deployable on other accelerometer devices with different axis orientations.\n",
    "\n",
    "<img src=\"wrist_accelerometer.jpg\" width=\"400\"/>\n",
    "\n",
    "Let's generate an artificial test set simulating this scenario by flipping two\n",
    "of the axes signs (this may simulate a different device specs, but it does not\n",
    "exactly simulate handedness since the movement dynamics are also mirrored, but\n",
    "it is enough to demonstrate our point). For this, we will need to grab the raw\n",
    "test data, rotate it, and re-compute the same features.\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_features(xyz):\n",
    "    ''' Extract features. xyz is an array of shape (N,3) '''\n",
    "\n",
    "    feats = {}\n",
    "    feats['xMean'], feats['yMean'], feats['zMean'] = np.mean(xyz, axis=0)\n",
    "    feats['xStd'], feats['yStd'], feats['zStd'] = np.std(xyz, axis=0)\n",
    "    v = np.linalg.norm(xyz, axis=1)  # magnitude stream\n",
    "    feats['mean'], feats['std'] = np.mean(v), np.std(v)\n",
    "\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1**: Replace this extract_features function with the function used to engineer your own features in Notebook 1_Baseline.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating test set with 'rotated device'...\")\n",
    "X_raw_test = X[mask_test]\n",
    "X_test_new = []\n",
    "for i in tqdm(range(X_raw_test.shape[0])):\n",
    "    # Rotate device\n",
    "    x = X_raw_test[i].copy()\n",
    "    x[:,1] *= -1\n",
    "    x[:,2] *= -1\n",
    "    X_test_new.append(extract_features(x))\n",
    "X_test_new = pd.DataFrame(X_test_new)\n",
    "del X_raw_test  # free memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2**: What other valid augmentation could be made to this dataset, that would simulate real world behaviour? Think about how you could implement this using code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Performance on simulated test set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_test_new_pred = clf.predict(X_test_new)\n",
    "print('\\nClassifier performance -- simulated test set')\n",
    "print('Out of sample:\\n', metrics.classification_report(Y_test, Y_test_new_pred, zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model performance is notably worse on the simulated test set. The solution\n",
    "is to simply augment the training dataset with the same rotation &mdash; we want\n",
    "our model to perform well no matter how/what device was worn.\n",
    "\n",
    "## Data augmentation\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating training set with 'rotated device'...\")\n",
    "X_raw_train = X[mask_train]\n",
    "X_train_new = []\n",
    "for i in tqdm(range(X_raw_train.shape[0])):\n",
    "    # Rotate device\n",
    "    x = X_raw_train[i].copy()\n",
    "    x[:,1] *= -1\n",
    "    x[:,2] *= -1\n",
    "    X_train_new.append(extract_features(x))\n",
    "X_train_new = pd.DataFrame(X_train_new)\n",
    "\n",
    "# Add the \"new data\" to training set\n",
    "X_aug_train = pd.concat((X_train, X_train_new))\n",
    "Y_aug_train = np.concatenate((Y_train, Y_train))\n",
    "print(\"X_aug_train shape:\", X_aug_train.shape)\n",
    "del X_raw_train  # free memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Re-train with augmented dataset\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = BalancedRandomForestClassifier(\n",
    "    n_estimators=1000,\n",
    "    replacement=True,\n",
    "    sampling_strategy='not minority',\n",
    "    n_jobs=-1,\n",
    "    random_state=42,\n",
    "    verbose=1\n",
    ")\n",
    "clf.fit(X_aug_train, Y_aug_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Re-check performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_test_new_pred = clf.predict(X_test_new)\n",
    "print('\\nClassifier performance -- augmented model on simulated test set')\n",
    "print('Out of sample:\\n', metrics.classification_report(Y_test, Y_test_new_pred, zero_division=0))\n",
    "\n",
    "Y_test_pred = clf.predict(X_test)\n",
    "print('\\nClassifier performance -- augmented model on original test set')\n",
    "print('Out of sample:\\n', metrics.classification_report(Y_test, Y_test_pred, zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the performance loss is recovered with the augmented model.\n",
    "Also, note how that the performance on the original test set remained almost\n",
    "unchanged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3**: How does the performance of the augmented model on the original test set comapre to the original model on the same test set? Why is it useful/necessary to make this check?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4**: In this notebook, we observe the performance of the augmented model on both the original test data, and the augmented test data. Which of these datasets should be used to report the metrics of this model and why?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('wearables_workshop')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "ba62a1b4a2d59cbe868c83bdd535aa95e1f6d51e6a8dbfe9705911f17ded0148"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}