Predicting Telco Customer Churn using SparkML.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "# Predicting Telco Customer Churn using SparkML on IBM Cloud Pak for Data (ICP4D)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We'll use this notebook to create a machine learning model to predict customer churn. In this notebook we will build the prediction model using the SparkML library.\n\nThis notebook walks you through these steps:\n\n- Load and Visualize data set. (https://raw.githubusercontent.com/IBM/telco-customer-churn-on-icp4d/master/data/Telco-Customer-Churn.csv)\n- Build a predictive model with SparkML API\n- Save the model in the ML repository\n\n* This notebook has been updated, in order to compatibilize to the services new versions, and it is part of code pattern at: https://developer.ibm.com/patterns/data-analysis-model-building-and-deploying-with-wml/"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "\n# The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.\n# Generate the TOKEN on settings session, and insert here the code using menu above (3 points) \"Insert Project Token\"\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# The code was removed by Watson Studio for sharing."
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## 1.0 Install required packages\n\nThere are a couple of Python packages we will use in this notebook. First we make sure the Watson Machine Learning client v3 is removed (its not installed by default) and then install/upgrade the v4 version of the client (this package is installed by default on CP4D).\n\nWML Client: https://wml-api-pyclient-dev-v4.mybluemix.net/#repository"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!pip uninstall --yes watson-machine-learning-client-V4\n!pip install --user watson-machine-learning-client-V4\n!pip install --user pyspark==2.4 --upgrade|tail -n 1\n!pip install --user scikit-learn==0.20.3 --upgrade|tail -n 1"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "import pandas as pd\nimport numpy as np\nimport json\nimport os\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## 2.0 Load and Clean data\n\nWe'll load our data as a pandas data frame.\n\n**<font color='red'><< FOLLOW THE INSTRUCTIONS BELOW TO LOAD THE DATASET >></font>**\n\n* Highlight the cell below by clicking it.\n* Click the `10/01` \"Find data\" icon in the upper right of the notebook.\n* If you are using Virtualized data, begin by choosing the `Files` tab. Then choose your virtualized data (i.e. MYSCHEMA.BILLINGPRODUCTCUSTOMERS), click `Insert to code` and choose `Insert Pandas DataFrame`.\n* If you are using this notebook without virtualized data, add the locally uploaded file `Telco-Customer-Churn.csv` by choosing the `Files` tab. Then choose the `Telco-Customer-Churn.csv`. Click `Insert to code` and choose `Insert Pandas DataFrame`.\n* The code to bring the data into the notebook environment and create a Pandas DataFrame will be added to the cell below.\n* Run the cell\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Place cursor below and insert the Pandas DataFrame for the Telco churn data\nimport os, types\nimport pandas as pd\nfrom botocore.client import Config\nimport ibm_boto3\n\ndef __iter__(self): return 0\n\n# Insert pandas code below\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# The code was removed by Watson Studio for sharing."
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We'll use the Pandas naming convention df for our DataFrame. Make sure that the cell below uses the name for the dataframe used above. For the locally uploaded file it should look like df_data_1 or df_data_2 or df_data_x. For the virtualized data case it should look like data_df_1 or data_df_2 or data_df_x.\n\n**<font color='red'><< UPDATE THE VARIABLE ASSIGNMENT TO THE VARIABLE GENERATED ABOVE. >></font>**"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# for virtualized data\n# df = data_df_1\n\n# for local upload\ndf = df_data_2"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 2.1 Drop CustomerID feature (column)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "df = df.drop('customerID', axis=1)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "df.head()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 2.2 Examine the data types of the features"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "df.info()"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Statistics for the columns (features). Set it to all, since default is to describe just the numeric features.\ndf.describe(include = 'all')"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We see that Tenure ranges from 0 (new customer) to 6 years, Monthly charges range from $18 to $118, etc"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 2.3 Check for need to Convert TotalCharges column to numeric if it is detected as object\n\nIf the above `df.info` shows the \"TotalCharges\" columnn as an object, we'll need to convert it to numeric. If you have already done this during a previous exercise for \"Data Visualization with Data Refinery\", you can skip to step `2.4`."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "totalCharges = df.columns.get_loc(\"TotalCharges\")\nnew_col = pd.to_numeric(df.iloc[:, totalCharges], errors='coerce')\ndf.iloc[:, totalCharges] = pd.Series(new_col)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Statistics for the columns (features). Set it to all, since default is to describe just the numeric features.\ndf.describe(include = 'all')"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We now see statistics for the `TotalCharges` feature."
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n\n### 2.4 Any NaN values should be removed to create a more accurate model."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Check if we have any NaN values and see which features have missing values that should be addressed\nprint(df.isnull().values.any())\ndf.isnull().sum()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We should see that the `TotalCharges` column has missing values. There are various ways we can address this issue:\n\n- Drop records with missing values \n- Fill in the missing value with one of the following strategies: Zero, Mean of the values for the column, Random value, etc)."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Handle missing values for nan_column (TotalCharges)\nfrom sklearn.impute import SimpleImputer\n\n# Find the column number for TotalCharges (starting at 0).\ntotal_charges_idx = df.columns.get_loc(\"TotalCharges\")\nimputer = SimpleImputer(missing_values=np.nan, strategy='mean')\n\ndf.iloc[:, total_charges_idx] = imputer.fit_transform(df.iloc[:, total_charges_idx].values.reshape(-1, 1))\ndf.iloc[:, total_charges_idx] = pd.Series(df.iloc[:, total_charges_idx])"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Validate that we have addressed any NaN values\nprint(df.isnull().values.any())\ndf.isnull().sum()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n### 2.5 Categorize Features\n\nWe will categorize some of the columns / features based on wether they are categorical values or continuous (i.e numerical) values. We will use this in later sections to build visualizations."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "columns_idx = np.s_[0:] # Slice of first row(header) with all columns.\nfirst_record_idx = np.s_[0] # Index of first record\n\nstring_fields = [type(fld) is str for fld in df.iloc[first_record_idx, columns_idx]] # All string fields\nall_features = [x for x in df.columns if x != 'Churn']\ncategorical_columns = list(np.array(df.columns)[columns_idx][string_fields])\ncategorical_features = [x for x in categorical_columns if x != 'Churn']\ncontinuous_features = [x for x in all_features if x not in categorical_features]\n\n#print('All Features: ', all_features)\n#print('\\nCategorical Features: ', categorical_features)\n#print('\\nContinuous Features: ', continuous_features)\n#print('\\nAll Categorical Columns: ', categorical_columns)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 2.6 Visualize data\n\nData visualization can be used to find patterns, detect outliers, understand distribution and more. We can use graphs such as:\n\n- Histograms, boxplots, etc: To find distribution / spread of our continuous variables.\n- Bar charts: To show frequency in categorical values.\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "import seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom sklearn.preprocessing import LabelEncoder\n\n%matplotlib inline\nsns.set(style=\"darkgrid\")\nsns.set_palette(\"hls\", 3)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "First, we get a high level view of the distribution of `Churn`. What percentage of customer in our dataset are churning vs not churning. "
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "print(df.groupby(['Churn']).size())\nchurn_plot = sns.countplot(data=df, x='Churn', order=df.Churn.value_counts().index)\nplt.ylabel('Count')\nfor p in churn_plot.patches:\n    height = p.get_height()\n    churn_plot.text(p.get_x()+p.get_width()/2., height + 1,'{0:.0%}'.format(height/float(len(df))),ha=\"center\") \nplt.show()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We can get use frequency counts charts to get an understanding of the categorical features relative to `Churn`  \n\n- We can see that for the `gender` feature. We have relatively equal rates of churn by `gender`\n- We can see that for the `InternetService` feature. We have higher churn for those that have \"Fiber optic\" service versus those with \"DSL\"\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Categorical feature count plots\nf, ((ax1, ax2, ax3), (ax4, ax5, ax6), (ax7, ax8, ax9), (ax10, ax11, ax12), (ax13, ax14, ax15)) = plt.subplots(5, 3, figsize=(20, 20))\nax = [ax1, ax2, ax3, ax4, ax5, ax6, ax7, ax8, ax9, ax10, ax11, ax12, ax13, ax14, ax15 ]\n\nfor i in range(len(categorical_features)):\n    sns.countplot(x = categorical_features[i], hue=\"Churn\", data=df, ax=ax[i])"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We can get use histrogram charts to get an understanding of the distribution of our continuous / numerical features relative to Churn.\n\n- We can see that for the `MonthlyCharges` feature, customers that churn tend to pay higher monthly fees than those that stay.\n- We can see that for the `tenure` feature, customers that churn tend to be relatively new customers."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Continuous feature histograms.\nfig, ax = plt.subplots(2, 2, figsize=(28, 8))\ndf[df.Churn == 'No'][continuous_features].hist(bins=20, color=\"blue\", alpha=0.5, ax=ax)\ndf[df.Churn == 'Yes'][continuous_features].hist(bins=20, color=\"orange\", alpha=0.5, ax=ax)\n\n# Or use displots\n#sns.set_palette(\"hls\", 3)\n#f, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(25, 25))\n#ax = [ax1, ax2, ax3, ax4]\n#for i in range(len(continuous_features)):\n#    sns.distplot(df[continuous_features[i]], bins=20, hist=True, ax=ax[i])"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "scrolled": true
            },
            "outputs": [],
            "source": "# Create Grid for pairwise relationships\ngr = sns.PairGrid(df, height=5, hue=\"Churn\")\ngr = gr.map_diag(plt.hist)\ngr = gr.map_offdiag(plt.scatter)\ngr = gr.add_legend()"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Plot boxplots of numerical columns. More variation in the boxplot implies higher significance. \nf, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(25, 25))\nax = [ax1, ax2, ax3, ax4]\n\nfor i in range(len(continuous_features)):\n    sns.boxplot(x = 'Churn', y = continuous_features[i], data=df, ax=ax[i])"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## 3.0 Create a model\n\nNow we can create our machine learning model. You could use the insights / intuition gained from the data visualization steps above to what kind of model to create or which features to use. We will create a simple classification model."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "from pyspark.sql import SparkSession\nimport pandas as pd\nimport json\n\nspark = SparkSession.builder.getOrCreate()\ndf_data = spark.createDataFrame(df)\ndf_data.head()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 3.1 Split the data into training and test sets"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "spark_df = df_data\n(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n\nprint(\"Number of records for training: \" + str(train_data.count()))\nprint(\"Number of records for evaluation: \" + str(test_data.count()))"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 3.2 Examine the Spark DataFrame Schema\nLook at the data types to determine requirements for feature engineering"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "spark_df.printSchema()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 3.3 Use StringIndexer to encode a string column of labels to a column of label indices\n\nWe are using the Pipeline package to build the development steps as pipeline. \nWe are using StringIndexer to handle categorical / string features from the dataset. StringIndexer encodes a string column of labels to a column of label indices\n\nWe then use VectorAssembler to asemble these features into a vector. Pipelines API requires that input variables are passed in  a vector"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "from pyspark.ml.classification import RandomForestClassifier\nfrom pyspark.ml.feature import StringIndexer, IndexToString, VectorAssembler\nfrom pyspark.ml.evaluation import BinaryClassificationEvaluator\nfrom pyspark.ml import Pipeline, Model\n\n\nsi_gender = StringIndexer(inputCol = 'gender', outputCol = 'gender_IX')\nsi_Partner = StringIndexer(inputCol = 'Partner', outputCol = 'Partner_IX')\nsi_Dependents = StringIndexer(inputCol = 'Dependents', outputCol = 'Dependents_IX')\nsi_PhoneService = StringIndexer(inputCol = 'PhoneService', outputCol = 'PhoneService_IX')\nsi_MultipleLines = StringIndexer(inputCol = 'MultipleLines', outputCol = 'MultipleLines_IX')\nsi_InternetService = StringIndexer(inputCol = 'InternetService', outputCol = 'InternetService_IX')\nsi_OnlineSecurity = StringIndexer(inputCol = 'OnlineSecurity', outputCol = 'OnlineSecurity_IX')\nsi_OnlineBackup = StringIndexer(inputCol = 'OnlineBackup', outputCol = 'OnlineBackup_IX')\nsi_DeviceProtection = StringIndexer(inputCol = 'DeviceProtection', outputCol = 'DeviceProtection_IX')\nsi_TechSupport = StringIndexer(inputCol = 'TechSupport', outputCol = 'TechSupport_IX')\nsi_StreamingTV = StringIndexer(inputCol = 'StreamingTV', outputCol = 'StreamingTV_IX')\nsi_StreamingMovies = StringIndexer(inputCol = 'StreamingMovies', outputCol = 'StreamingMovies_IX')\nsi_Contract = StringIndexer(inputCol = 'Contract', outputCol = 'Contract_IX')\nsi_PaperlessBilling = StringIndexer(inputCol = 'PaperlessBilling', outputCol = 'PaperlessBilling_IX')\nsi_PaymentMethod = StringIndexer(inputCol = 'PaymentMethod', outputCol = 'PaymentMethod_IX')\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "si_Label = StringIndexer(inputCol=\"Churn\", outputCol=\"label\").fit(spark_df)\nlabel_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 3.4 Create a single vector"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "va_features = VectorAssembler(inputCols=['gender_IX',  'SeniorCitizen', 'Partner_IX', 'Dependents_IX', 'PhoneService_IX', 'MultipleLines_IX', 'InternetService_IX', \\\n                                         'OnlineSecurity_IX', 'OnlineBackup_IX', 'DeviceProtection_IX', 'TechSupport_IX', 'StreamingTV_IX', 'StreamingMovies_IX', \\\n                                         'Contract_IX', 'PaperlessBilling_IX', 'PaymentMethod_IX', 'TotalCharges', 'MonthlyCharges'], outputCol=\"features\")"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 3.5 Create a pipeline, and fit a model using RandomForestClassifier \nAssemble all the stages into a pipeline. We don't expect a clean linear regression, so we'll use RandomForestClassifier to find the best decision tree for the data."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "classifier = RandomForestClassifier(featuresCol=\"features\")\n\npipeline = Pipeline(stages=[si_gender, si_Partner, si_Dependents, si_PhoneService, si_MultipleLines, si_InternetService, si_OnlineSecurity, si_OnlineBackup, si_DeviceProtection, \\\n                            si_TechSupport, si_StreamingTV, si_StreamingMovies, si_Contract, si_PaperlessBilling, si_PaymentMethod, si_Label, va_features, \\\n                            classifier, label_converter])\n\nmodel = pipeline.fit(train_data)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "predictions = model.transform(test_data)\nevaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\narea_under_curve = evaluatorDT.evaluate(predictions)\n\nevaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\",  metricName='areaUnderROC')\narea_under_curve = evaluatorDT.evaluate(predictions)\nevaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\",  metricName='areaUnderPR')\narea_under_PR = evaluatorDT.evaluate(predictions)\nprint(\"areaUnderROC = %g\" % area_under_curve)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## 4.0 Save the model and test data\n\nNow the model can be saved for future deployment. The model will be saved using the Watson Machine Learning client, to a deployment space.\n\n**<font color='red'><< UPDATE THE VARIABLE 'MODEL_NAME' TO A UNIQUE NAME>></font>**\n\n**<font color='red'><< UPDATE THE VARIABLE 'DEPLOYMENT_SPACE_NAME' TO THE NAME OF THE DEPLOYMENT SPACE CREATED PREVIOUSLY>></font>**"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "\nMODEL_NAME = \"GAMA-PREDICT-CHURN\"\nDEPLOYMENT_SPACE_NAME = 'Big-Data'\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### 4.1 Save the model to ICP4D local Watson Machine Learning\n\n1. Generate an API Key: https://cloud.ibm.com/iam/apikeys\n2. Generate a TOKEN for your Watson Machine Learning:\n   curl --insecure -X POST --header \"Content-Type: application/x-www-form-urlencoded\" --header \"Accept: application/json\" --data-urlencode \"grant_type=urn:ibm:params:oauth:grant-type:apikey\" --data-urlencode \"apikey=$API_key\" \"https://iam.ng.bluemix.net/identity/token\"\n3.   <font color='red'>Replace the `token` value of `*****` with `token` generate. The value for `url` should match the `url` for your Watson Machine Learning.</font>"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "from ibm_watson_machine_learning import APIClient\n\nwml_credentials = {\n                   \"url\": \"https://us-south.ml.cloud.ibm.com\",\n                   \"token\":\"*******\"\n}\n\nclient = APIClient(wml_credentials)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# The code was removed by Watson Studio for sharing."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "client.spaces.list()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### Use the desired space as the `default_space`\n\nThe deployment space ID will be looked up based on the name specified above. If you do not receive a space GUID as an output to the next cell, do not proceed until you have created a deployment space."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Be sure to update the name of the space with the one you want to use.\n#client.spaces.list()\nall_spaces = client.spaces.get_details()['resources']\nspace_id = None\n#print(all_spaces)\nfor space in all_spaces:\n    if space['entity']['name'] == DEPLOYMENT_SPACE_NAME:\n        space_id = space[\"metadata\"][\"id\"]\n        print(\"\\nDeployment Space GUID: \", space_id)\n\nif space_id is None:\n    print(\"WARNING: Your space does not exist. Create a deployment space before proceeding to the next cell.\")\n    #space_id = client.spaces.store(meta_props={client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"guid\"]"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "**<font color='red'><< REPLACE space_id BELOW with the id for your space. For e.g.<br/>client.set.default_space(\"6b39c537-f707-4078-9dc7-ce70b70ab22f\") >></font>**"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Now set the default space to the GUID for your deployment space. If this is successful, you will see a 'SUCCESS' message.\nclient.set.default_space(space_id)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "#### Save the Model"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# case you need check the services, uncomment th eline below and run it.\n#client.software_specifications.list()"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "software_spec_id =  client.software_specifications.get_id_by_name('spark-mllib_2.4')\nprint(software_spec_id)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Store our model\nmodel_props = {client.repository.ModelMetaNames.NAME: MODEL_NAME,\n               client.repository.ModelMetaNames.SOFTWARE_SPEC_UID : software_spec_id,\n               client.repository.ModelMetaNames.TYPE : \"mllib_2.4\"}\npublished_model = client.repository.store_model(model=model, pipeline=pipeline, meta_props=model_props, training_data=train_data)\n\nprint(json.dumps(published_model, indent=3))"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# Use this cell to do any cleanup of previously created models and deployments\nclient.repository.list_models()\nclient.deployments.list()\n\n# client.repository.delete('GUID of stored model')\n# client.deployments.delete('GUID of deployed model')\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## 5.0 Save Test Data\n\nWe will save the test data we used to evaluate the model to our project."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "write_score_CSV=test_data.toPandas().drop(['Churn'], axis=1)\n#write_score_CSV.to_csv('/project_data/data_asset/TelcoCustomerSparkMLBatchScore.csv', sep=',', index=False)\nproject.save_data('TelcoCustomerSparkMLBatchScore.csv', write_score_CSV.to_csv(), overwrite=True)\n\nwrite_eval_CSV=test_data.toPandas()\n#write_eval_CSV.to_csv('/project_data/data_asset/TelcoCustomerSparkMLEval.csv', sep=',', index=False)\nproject.save_data('TelcoCustomerSparkMLEval.csv', write_eval_CSV.to_csv(), overwrite=True)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Congratulations, you have created a model based on customer churn data, and deployed it to Watson Machine Learning!"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.7",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.10"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}