demos/netops/04-infer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a Model Inference Nuclio Function\n",
    "\n",
    "- [Initialize Nuclio Emulation, Environment Variables, and Configuration](#netops-infer-init-func)\n",
    "  - [Install Required Packages](#netops-infer-install-pkgs)\n",
    "- [Set a cron Function Trigger](#netops-infer-set-func-trigger)\n",
    "- [Initialize the Function](#netops-infer-func-init)\n",
    "- [Deploy the Function](#netops-infer-func-deploy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"netops-infer-init-func\"></a>\n",
    "## Initialize Nuclio Emulation, Environment Variables, and Configuration\n",
    "\n",
    "> **Note:** Use `# nuclio: ignore` for sections that dont need to be copied to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: ignore\n",
    "import nuclio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: ignore\n",
    "from os import environ\n",
    "version = '1.0'\n",
    "environ['MODEL_FILE'] = 'netops.v{}.model'.format(version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%nuclio: setting 'MODEL_FILEPATH' environment variable\n"
     ]
    }
   ],
   "source": [
    "%nuclio env -c V3IO_USERNAME=${V3IO_USERNAME}\n",
    "%nuclio env -c V3IO_ACCESS_KEY=${V3IO_ACCESS_KEY}\n",
    "%nuclio env -c MODEL_FILE=${MODEL_FILE}\n",
    "%nuclio env -c MODEL_FILEPATH=/tmp/mlmodel/${MODEL_FILE}\n",
    "%nuclio env -l MODEL_FILEPATH=models/${MODEL_FILE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"netops-infer-install-pkgs\"></a>\n",
    "### Install Required Packages\n",
    "\n",
    "`%nuclio cmd` allows you to run image build instructions and install packages.\n",
    "\n",
    "> **Note:** When using the `-c` option, commands will only run in Nuclio and not locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%nuclio cmd -c\n",
    "pip install pandas\n",
    "pip install sklearn\n",
    "pip install v3io_frames\n",
    "pip install dask_ml\n",
    "apt-get update && apt-get install -y wget\n",
    "mkdir -p /tmp/mlmodel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy the model file into the function\n",
    "%nuclio cmd -c wget -O /tmp/mlmodel/${MODEL_FILE} --header \"x-v3io-session-key: ${V3IO_ACCESS_KEY}\" http://${V3IO_WEBAPI_SERVICE_HOST}:8081/users/${V3IO_USERNAME}/demos/netops/models/${MODEL_FILE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"netops-infer-set-func-trigger\"></a>\n",
    "## Set a cron Function Trigger\n",
    "\n",
    "Set a cron function trigger with 1-minute interval:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%nuclio: setting spec.triggers.secs.kind to 'cron'\n",
      "%nuclio: setting spec.triggers.secs.attributes.interval to '600s'\n",
      "%nuclio: setting spec.build.baseImage to 'python:3.6-jessie'\n"
     ]
    }
   ],
   "source": [
    "%%nuclio config \n",
    "spec.triggers.secs.kind = \"cron\"\n",
    "spec.triggers.secs.attributes.interval = \"600s\"\n",
    "spec.build.baseImage = \"python:3.6-jessie\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"netops-infer-func-init\"></a>\n",
    "## Initialize the Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pickle\n",
    "import sklearn\n",
    "from dask_ml.wrappers import ParallelPostFit\n",
    "import v3io_frames as v3f\n",
    "from os import environ\n",
    "\n",
    "# Define a V3IO TSDB client\n",
    "client = v3f.Client('framesd:8081')\n",
    "\n",
    "# Load the model\n",
    "with open(environ['MODEL_FILEPATH'], 'rb+') as f:\n",
    "    model = pickle.load(f) \n",
    "    \n",
    "clf = ParallelPostFit(estimator=model,\n",
    "                      scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'time                 company                     data_center        device       \\n2019-06-17 13:42:04  Ortega__Nelson_and_Parrish  Chen_Rest          9999032632997    0.0\\n2019-06-17 14:42:04  Ortega__Nelson_and_Parrish  Chen_Rest          9999032632997    0.0\\n2019-06-17 13:42:04  Ortega__Nelson_and_Parrish  Chen_Rest          2588823083436    0.0\\n2019-06-17 14:42:04  Ortega__Nelson_and_Parrish  Chen_Rest          2588823083436    0.0\\n2019-06-17 13:42:04  Ortega__Nelson_and_Parrish  Gallagher_Springs  9909218583192    0.0\\n2019-06-17 14:42:04  Ortega__Nelson_and_Parrish  Gallagher_Springs  9909218583192    0.0\\n2019-06-17 13:42:04  Washington-Lambert          Tamara_Greens      9025508674926    0.0\\n2019-06-17 14:42:04  Washington-Lambert          Tamara_Greens      9025508674926    0.0\\n2019-06-17 13:42:04  Washington-Lambert          Nicole_Trail       5079628775825    0.0\\n2019-06-17 14:42:04  Washington-Lambert          Nicole_Trail       5079628775825    0.0\\n2019-06-17 13:42:04  Washington-Lambert          Nicole_Trail       0243526550889    0.0\\n2019-06-17 14:42:04  Washington-Lambert          Nicole_Trail       0243526550889    0.0\\n2019-06-17 13:42:04  Washington-Lambert          Tamara_Greens      1416256267840    0.0\\n2019-06-17 14:42:04  Washington-Lambert          Tamara_Greens      1416256267840    0.0\\n2019-06-17 13:42:04  Ortega__Nelson_and_Parrish  Gallagher_Springs  6842603445464    0.0\\n2019-06-17 14:42:04  Ortega__Nelson_and_Parrish  Gallagher_Springs  6842603445464    0.0\\nName: prediction, dtype: float64'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%nuclio handler\n",
    "# Load last hour data\n",
    "df = client.read(backend='tsdb', query='select avg(cpu_utilization) , avg(latency) , avg(packet_loss) , avg(throughput) from netops_metrics_jupyter group by company, data_center, device',\n",
    "            start=\"now-1d\", end='now+1d', multi_index=True, step='1h')\n",
    "df['prediction'] = clf.predict(df)\n",
    "#df.reset_index(inplace=True)\n",
    "#client.write(backend='tsdb', table='netops_predictions', dfs=[df], index_cols=['company', 'data_center', 'device', 'time'])\n",
    "str(df['prediction']) # nuclio: return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"netops-infer-func-deploy\"></a>\n",
    "## Deploy the Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%nuclio deploy -n netops-infer -p netops -c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>avg(cpu_utilization)</th>\n",
       "      <th>avg(latency)</th>\n",
       "      <th>avg(packet_loss)</th>\n",
       "      <th>avg(throughput)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>company</th>\n",
       "      <th>data_center</th>\n",
       "      <th>device</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-06-17 14:38:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>77.711198</td>\n",
       "      <td>30.534627</td>\n",
       "      <td>9.209405</td>\n",
       "      <td>233.679544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 14:43:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>79.533761</td>\n",
       "      <td>29.101493</td>\n",
       "      <td>14.341944</td>\n",
       "      <td>179.910993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 14:48:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>70.554952</td>\n",
       "      <td>1.738742</td>\n",
       "      <td>0.911834</td>\n",
       "      <td>250.449321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 14:53:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>69.348307</td>\n",
       "      <td>1.350994</td>\n",
       "      <td>0.524656</td>\n",
       "      <td>249.755806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 14:58:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>68.872050</td>\n",
       "      <td>2.495677</td>\n",
       "      <td>0.712196</td>\n",
       "      <td>252.960565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 15:03:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>80.429847</td>\n",
       "      <td>31.620278</td>\n",
       "      <td>14.888336</td>\n",
       "      <td>182.140072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 15:08:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>76.459967</td>\n",
       "      <td>25.483856</td>\n",
       "      <td>12.512834</td>\n",
       "      <td>201.100587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 15:13:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>85.523864</td>\n",
       "      <td>45.786581</td>\n",
       "      <td>21.547878</td>\n",
       "      <td>149.405401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 15:18:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>84.407529</td>\n",
       "      <td>48.858549</td>\n",
       "      <td>20.431261</td>\n",
       "      <td>158.104648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-17 15:23:29</th>\n",
       "      <th>Washington-Lambert</th>\n",
       "      <th>Tamara_Greens</th>\n",
       "      <th>1416256267840</th>\n",
       "      <td>69.697620</td>\n",
       "      <td>4.529601</td>\n",
       "      <td>1.604976</td>\n",
       "      <td>246.599148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    avg(cpu_utilization)  \\\n",
       "time                company            data_center   device                                \n",
       "2019-06-17 14:38:29 Washington-Lambert Tamara_Greens 1416256267840             77.711198   \n",
       "2019-06-17 14:43:29 Washington-Lambert Tamara_Greens 1416256267840             79.533761   \n",
       "2019-06-17 14:48:29 Washington-Lambert Tamara_Greens 1416256267840             70.554952   \n",
       "2019-06-17 14:53:29 Washington-Lambert Tamara_Greens 1416256267840             69.348307   \n",
       "2019-06-17 14:58:29 Washington-Lambert Tamara_Greens 1416256267840             68.872050   \n",
       "2019-06-17 15:03:29 Washington-Lambert Tamara_Greens 1416256267840             80.429847   \n",
       "2019-06-17 15:08:29 Washington-Lambert Tamara_Greens 1416256267840             76.459967   \n",
       "2019-06-17 15:13:29 Washington-Lambert Tamara_Greens 1416256267840             85.523864   \n",
       "2019-06-17 15:18:29 Washington-Lambert Tamara_Greens 1416256267840             84.407529   \n",
       "2019-06-17 15:23:29 Washington-Lambert Tamara_Greens 1416256267840             69.697620   \n",
       "\n",
       "                                                                    avg(latency)  \\\n",
       "time                company            data_center   device                        \n",
       "2019-06-17 14:38:29 Washington-Lambert Tamara_Greens 1416256267840     30.534627   \n",
       "2019-06-17 14:43:29 Washington-Lambert Tamara_Greens 1416256267840     29.101493   \n",
       "2019-06-17 14:48:29 Washington-Lambert Tamara_Greens 1416256267840      1.738742   \n",
       "2019-06-17 14:53:29 Washington-Lambert Tamara_Greens 1416256267840      1.350994   \n",
       "2019-06-17 14:58:29 Washington-Lambert Tamara_Greens 1416256267840      2.495677   \n",
       "2019-06-17 15:03:29 Washington-Lambert Tamara_Greens 1416256267840     31.620278   \n",
       "2019-06-17 15:08:29 Washington-Lambert Tamara_Greens 1416256267840     25.483856   \n",
       "2019-06-17 15:13:29 Washington-Lambert Tamara_Greens 1416256267840     45.786581   \n",
       "2019-06-17 15:18:29 Washington-Lambert Tamara_Greens 1416256267840     48.858549   \n",
       "2019-06-17 15:23:29 Washington-Lambert Tamara_Greens 1416256267840      4.529601   \n",
       "\n",
       "                                                                    avg(packet_loss)  \\\n",
       "time                company            data_center   device                            \n",
       "2019-06-17 14:38:29 Washington-Lambert Tamara_Greens 1416256267840          9.209405   \n",
       "2019-06-17 14:43:29 Washington-Lambert Tamara_Greens 1416256267840         14.341944   \n",
       "2019-06-17 14:48:29 Washington-Lambert Tamara_Greens 1416256267840          0.911834   \n",
       "2019-06-17 14:53:29 Washington-Lambert Tamara_Greens 1416256267840          0.524656   \n",
       "2019-06-17 14:58:29 Washington-Lambert Tamara_Greens 1416256267840          0.712196   \n",
       "2019-06-17 15:03:29 Washington-Lambert Tamara_Greens 1416256267840         14.888336   \n",
       "2019-06-17 15:08:29 Washington-Lambert Tamara_Greens 1416256267840         12.512834   \n",
       "2019-06-17 15:13:29 Washington-Lambert Tamara_Greens 1416256267840         21.547878   \n",
       "2019-06-17 15:18:29 Washington-Lambert Tamara_Greens 1416256267840         20.431261   \n",
       "2019-06-17 15:23:29 Washington-Lambert Tamara_Greens 1416256267840          1.604976   \n",
       "\n",
       "                                                                    avg(throughput)  \n",
       "time                company            data_center   device                          \n",
       "2019-06-17 14:38:29 Washington-Lambert Tamara_Greens 1416256267840       233.679544  \n",
       "2019-06-17 14:43:29 Washington-Lambert Tamara_Greens 1416256267840       179.910993  \n",
       "2019-06-17 14:48:29 Washington-Lambert Tamara_Greens 1416256267840       250.449321  \n",
       "2019-06-17 14:53:29 Washington-Lambert Tamara_Greens 1416256267840       249.755806  \n",
       "2019-06-17 14:58:29 Washington-Lambert Tamara_Greens 1416256267840       252.960565  \n",
       "2019-06-17 15:03:29 Washington-Lambert Tamara_Greens 1416256267840       182.140072  \n",
       "2019-06-17 15:08:29 Washington-Lambert Tamara_Greens 1416256267840       201.100587  \n",
       "2019-06-17 15:13:29 Washington-Lambert Tamara_Greens 1416256267840       149.405401  \n",
       "2019-06-17 15:18:29 Washington-Lambert Tamara_Greens 1416256267840       158.104648  \n",
       "2019-06-17 15:23:29 Washington-Lambert Tamara_Greens 1416256267840       246.599148  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.read(backend='tsdb', query='select avg(cpu_utilization) , avg(latency) , avg(packet_loss) , avg(throughput)  from netops_metrics_jupyter group by company, data_center, device',\n",
    "            start=\"now-1d\", end='now+1d', multi_index=True, step='5m').head(10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}