Train%2Band%2Bpredict%2Bwith%2BPython%2Bmachine%2Blearning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Using DSX Local Machine Learning Service for Model Training and Making Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This notebook shows how to use machine learning libraries and services from DSX Local to train, save, deploy and evaluate a model and make a prediction for new data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Table of contents\n",
    "- [Prepare the environment](#prepare_environment)\n",
    "- [Load data](#load_data)\n",
    "- [Access and manipulate data](#access_manipulate_data)\n",
    "- [Evaluate the model](#evaluate_model)\n",
    "- [Save the model](#save_model)\n",
    "- [Make an online scoring prediction](#make_prediction)\n",
    "- [Summary](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"prepare_environment\"></a>\n",
    "## Prepare the environment\n",
    "\n",
    "Import machine learning libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import StringIndexer, IndexToString, VectorIndexer, VectorAssembler\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.classification import LogisticRegression, DecisionTreeClassifier\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"load_data\"></a>\n",
    "## Load data \n",
    "The 1983 Data Exposition dataset was collected by Ernesto Ramos and David Donoho and dealt with automobiles. Data on mpg, cylinders, displacement, was provided for 406 different cars, each identified by name. The dataset is freely available on the Data Science Experience home page.\n",
    "\n",
    "\n",
    "Perform the following steps to upload this dataset:\n",
    "1. Go to the <a href=\"https://apsportal.ibm.com/exchange/public/entry/view/c81e9be8daf6941023b9dc86f303053b\" target=\"_blank\">Car performance data</a> card on the Data Science Experience home page.\n",
    "1. Click the download button.\n",
    "1. Click the **Create new** icon on the notebook action bar, and use **Add data set** button to add the downloaded cars.csv file as a `Local File`. \n",
    "\n",
    "The data file is listed on the **Local Data** pane in the notebook.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"access_manipulate_data\"></a>\n",
    "## Access and manipulate data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To add the code to access the data file, click the next code cell and select **Insert Spark DataFrame in Python** in the **Insert To Code** drop-down list below the data file in the `Local Data` pane in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---------+------+----------+------+------------+----+--------+--------------------+\n",
      "|mpg|cylinders|engine|horsepower|weight|acceleration|year|  origin|                name|\n",
      "+---+---------+------+----------+------+------------+----+--------+--------------------+\n",
      "| 18|        8| 307.0|       130|  3504|        12.0|  70|American|chevrolet chevell...|\n",
      "| 15|        8| 350.0|       165|  3693|        11.5|  70|American|   buick skylark 320|\n",
      "| 18|        8| 318.0|       150|  3436|        11.0|  70|American|  plymouth satellite|\n",
      "| 16|        8| 304.0|       150|  3433|        12.0|  70|American|       amc rebel sst|\n",
      "| 17|        8| 302.0|       140|  3449|        10.5|  70|American|         ford torino|\n",
      "+---+---------+------+----------+------+------------+----+--------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-block alert-info\"> Note: Make sure the df variable in the following cell is the same as the generated code from insertToCode.</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Due to missing data in `mpg` and `horsepower` columns, they will be excluded from the dataset for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+------+------+------------+----+--------+--------------------+\n",
      "|cylinders|engine|weight|acceleration|year|  origin|                name|\n",
      "+---------+------+------+------------+----+--------+--------------------+\n",
      "|        8| 307.0|  3504|        12.0|  70|American|chevrolet chevell...|\n",
      "|        8| 350.0|  3693|        11.5|  70|American|   buick skylark 320|\n",
      "|        8| 318.0|  3436|        11.0|  70|American|  plymouth satellite|\n",
      "|        8| 304.0|  3433|        12.0|  70|American|       amc rebel sst|\n",
      "|        8| 302.0|  3449|        10.5|  70|American|         ford torino|\n",
      "+---------+------+------+------------+----+--------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "carsDataRaw = df_data_2\n",
    "carsModData = carsDataRaw.drop(\"mpg\").drop(\"horsepower\")\n",
    "carsModData.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the model training process, the original dataset will be split into training dataset and testing dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training dataset: 348\n",
      "Number of testing dataset: 58\n"
     ]
    }
   ],
   "source": [
    "splitted_data = carsModData.randomSplit([0.85, 0.15], 24)\n",
    "train_data = splitted_data[0]\n",
    "test_data = splitted_data[1]\n",
    "\n",
    "print(\"Number of training dataset: {}\".format(train_data.count()))\n",
    "print(\"Number of testing dataset: {}\".format(test_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The following task is to set the input columns for model training, and use the corresponding algorithms to train the model. In this example, Linear Regression method is used to evaluate `weight` in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "originIndexer = StringIndexer().setInputCol(\"origin\").setOutputCol(\"origin_code\")\n",
    "\n",
    "vectorAssembler_features = VectorAssembler().setInputCols([\"cylinders\",\n",
    "                                                                 \"engine\",\n",
    "                                                                 \"acceleration\",\n",
    "                                                                 \"year\",\n",
    "                                                                 \"origin_code\"]).setOutputCol(\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rf = LinearRegression().setLabelCol(\"weight\").setFeaturesCol(\"features\")\n",
    "pipeline = Pipeline().setStages([originIndexer,vectorAssembler_features,rf])\n",
    "model = pipeline.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"evaluate_model\"></a>\n",
    "## Evaluate the model\n",
    "The model performance can be evaluated using the R Square for test data and the evaluation result can be saved on Cloudant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R Square of Test Data: 0.863976844308\n"
     ]
    }
   ],
   "source": [
    "testData = model.transform(test_data).drop(\"prediction\")\n",
    "metric = model.stages[2].evaluate(testData)\n",
    "print(\"R Square of Test Data: {}\".format(metric.r2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"save_model\"></a>\n",
    "## Save the model\n",
    "After the model is successfully trained, repository service is used to save the model. The model name and author information can be customized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/opt/conda/lib/python2.7/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from dsx_ml.ml import save\n",
    "saved_model_output = save(name='CarsModelPython', model=model, test_data=test_data,algorithm_type='Regression')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-block alert-info\"> Note: The warnings in the cell above are expected.</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"make_prediction\"></a>\n",
    "## Make an online scoring prediction\n",
    "\n",
    "Upon saving a model, an internal online scoring endpoint is automatically created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://dsxl-api.ibm-private-cloud.svc.cluster.local/v3/project/score/Python27/spark-2.0/dsx-samples/CarsModelPython/1\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import requests\n",
    "\n",
    "header_online = {'Content-Type': 'application/json', 'Authorization': os.environ['DSX_TOKEN']}\n",
    "\n",
    "print(saved_model_output['scoring_endpoint'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "New data is provided in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'engine': 289, 'acceleration': 11.1, 'cylinders': 6, 'origin': 'American', 'year': 79}\n"
     ]
    }
   ],
   "source": [
    "new_data = {\"cylinders\" : 6, \"engine\" : 289, \"acceleration\" : 11.1, \"year\" : 79, \"origin\" : \"American\" }\n",
    "print(new_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The model evaluates new data and give an estimate scoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"success\":true,\"description\":\"Success\",\"object\":{\"error\":\"\",\"output\":{\"classes\":[],\"predictions\":[3553],\"probabilities\":[]},\"returnCode\":\"0\"}}\n"
     ]
    }
   ],
   "source": [
    "payload = [new_data]\n",
    "scoring_response = requests.post(saved_model_output['scoring_endpoint'], json=payload, headers=header_online, verify=False)\n",
    "\n",
    "print(scoring_response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"summary\"></a>\n",
    "## Summary\n",
    "In this sample, you learned how to use DSX Local machine learning services and libraries. You also learned how to split data for model training, how to customize, save and deploy the model, and how to use model endpoint for new data evaluation and scoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-block alert-info\"> Note: To save resources and get the best performance please use the code below to stop the kernel before exiting your notebook.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%javascript\n",
    "Jupyter.notebook.session.delete();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<hr>\n",
    "Copyright &copy; IBM Corp. 2017. Released as licensed Sample Materials."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python2 with Spark Local Mode",
   "language": "python",
   "name": "py2localspark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}