diff --git a/notebooks/10 Automated Machine Learning.ipynb b/notebooks/10 Automated Machine Learning.ipynb index b6301cc..bc49289 100644 --- a/notebooks/10 Automated Machine Learning.ipynb +++ b/notebooks/10 Automated Machine Learning.ipynb @@ -11,7 +11,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this notebook, we demonstrated how to use the Kx kdb+/q Automated Machine Learning library. The example below use samples from the Telco Customer Churn dataset.\n", + "In this notebook, we demonstrate how to use the Kx kdb+/q Automated Machine Learning library. The example below uses samples from the Telco Customer Churn dataset and IMBD movie review dataset.\n", "\n", "
\n", "To run the below notebook, ensure that dependencies specified in requirements.txt have been correctly installed.\n", @@ -29,7 +29,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The Machine Learning Toolkit ([ML-Toolkit](https://github.com/KxSystems/ml)) contains general use utilities, an implementation of the FRESH (Feature Extraction based on Scalable Hypothesis tests) algorithm and cross validation functions. The primary purpose of these libraries are to provide kdb+/q users with access to commonly-used ML functions for preprocessing data, extracting features and scoring results." + "The Machine Learning Toolkit ([ML-Toolkit](https://github.com/KxSystems/ml)) contains general use utilities, an implementation of the FRESH (Feature Extraction based on Scalable Hypothesis tests) algorithm, cross validation functions, clustering libraries and time series functionality. The primary purpose of these libraries are to provide kdb+/q users with access to commonly-used ML functions for preprocessing data, extracting features and scoring results." ] }, { @@ -48,15 +48,16 @@ "- Data preprocessing\n", "- Feature engineering and feature selection\n", "- Model selection\n", - "- Hyperparameter Tuning\n", + "- Hyperparameter tuning\n", "- Report generation and model persistence\n", "\n", "Each of these steps is outlined in depth within the documentation for this platform [here](https://code.kx.com/q/ml/automl). This allows users to understand the processes by which decisions are being made and the transformations which their data undergo during the production of the output models.\n", "\n", - "At present the supported machine learning problem types are classification and regression and based on:\n", + "At present the supported machine learning problem types for classification and regression tasks and based on:\n", "\n", "- One-to-one feature to target non time-series\n", "- FRESH based feature extraction and model production\n", + "- NLP-based feature creation and word2vec transformation\n", "\n", "The problems which can be solved by this framework will be expanded over time as will the available functionality." ] @@ -67,7 +68,7 @@ "source": [ "### Multi-processing\n", "\n", - "This library supports multi-processed grid-search/cross-validation procedures and FRESH feature creation provided a user set `-s -8` in the JUPYTERQ_SERVERARGS, access to which can be found [here](https://code.kx.com/q/ml/jupyterq/notebooks/#server-command-line-arguments). In this demo, we use 8 worker processes and open a centralised port as below." + "This library supports multi-processed grid-search/cross-validation procedures and FRESH feature creation provided a user set `-s -8` in the JUPYTERQ_SERVERARGS entry to the appropriate JSON file, instructions to facilitate this can be found [here](https://code.kx.com/q/ml/jupyterq/notebooks/#server-command-line-arguments). In this demo, we use 8 worker processes and open a centralised port as below." ] }, { @@ -92,11 +93,24 @@ "metadata": { "scrolled": true }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Documentation can be found at https://code.kx.com/q/ml/automl/\n" + ] + } + ], "source": [ "// load in automl\n", "\\l automl/automl.q\n", - ".automl.loadfile`:init.q" + ".automl.loadfile`:init.q\n", + "\n", + "// load utils\n", + "\\l ../utils/util.q\n", + "\\l ../utils/graphics.q" ] }, { @@ -125,13 +139,6 @@ "---" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Default Configurations" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -143,9 +150,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The [Telco Customer Churn dataset](https://www.kaggle.com/blastchar/telco-customer-churn/data) contains entries for 7043 customers. In each case below, we aim to create a model which can accurately predict customer churn based on 20 features relating to each customer.\n", + "The [Telco Customer Churn dataset](https://www.kaggle.com/blastchar/telco-customer-churn/data) contains the following information.\n", + "* Data on 7,043 customers of a telecom provider provided by IBM\n", + "* Customer feature information including\n", + " * What form of internet the user has (DSL/Fiber Optic)?\n", + " * What are the users monthly payments?\n", + " * How long has the customer been in their contract?\n", + " * What services does the customer use? i.e. phone, internet, online backup, streaming etc.\n", + "* A target variable 'Churn' indicating if a user has cancelled their contract in the last month\n", "\n", - "Below we load in the data and select a subset of 5000 random data points to train and test the pipeline on. We also load in additional graphics and utility functions required throughout this notebook." + "\n", + "In each case of the examples below, we aim to create a model which can accurately predict customer churn based on 20 features relating to each customer." ] }, { @@ -159,17 +174,6 @@ "cell_type": "code", "execution_count": 3, "metadata": {}, - "outputs": [], - "source": [ - "// load utils\n", - "\\l ../utils/util.q\n", - "\\l ../utils/graphics.q" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, "outputs": [ { "name": "stdout", @@ -220,12 +224,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As we want to run both `.automl.run` and `.automl.new` we start by splitting our data into a training and testing set, where 10% has been chosen for the testing set. Note that we have set a random seed so that results can be replicated." + "In order to test both the model generation and prediction steps of the workflow we split the dataset into a training and testing set where\n", + "\n", + "| Dataset form | Purpose | Percentage (%)|\n", + "|--------------|:-------------------------------------------------------------------|---------------|\n", + "| Training | Generate model for deployment using `.automl.fit` | 90 |\n", + "| Testing | Independent dataset to test application of `predict` functionality | 10 |\n", + "\n", + "__*Note:*__ \n", + "\n", + " We have set a random seed so that results can be replicated." ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 4, "metadata": {}, "outputs": [ { @@ -248,19 +261,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### User Interface" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "At the highest level the automated machine learning library contains two primary callable functions:\n", - "\n", - "- `.automl.run` = Run the automated machine learning pipeline on user defined data and target\n", - "- `.automl.new` = Using a previously fit model and set of instructions to produce an appropriate pipeline derived from a defined run, predict the target value for new tabular data\n", - "\n", - "Both of these functions are modifiable by a user to suit specific use cases and have been designed to cover a wide range of functional options and to be extensible to a users needs." + "## Default Configurations" ] }, { @@ -274,61 +275,49 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Below we demonstrate how to apply `.automl.run` to our features and targets in the default setting, where the function has the syntax:\n", - "\n", - "```.automl.run[tab;tgt;ftype;ptype;dict]```\n", + "The automated machine learning pipeline will use the training features (`xtrain`) and targets (`ytrain`) from `telcoInputs` above as input to `automl.fit`. \n", "\n", - "Where:\n", - "- `tab` is unkeyed tabular data from which the models will be created\n", - "- `tgt` is the target vector\n", - "- `ftype` type of feature extraction being completed on the dataset as a symbol (``` `fresh```/``` `normal```)\n", - "- `ptype` type of problem, regression/class, as a symbol (``` `reg```/``` `class```)\n", - "- `dict` is one of `(::)` for default behaviour, a kdb+ dictionary or path to a user defined flat file for modifying default parameters.\n", + "Appropriate preprocessing steps including feature creation and selection will be applied to the data before being passed to a variety of machine learning models, choosing the best performing model. \n", "\n", "In this case, we select ``` `normal``` feature extraction as we have a 1-to-1 mapping between features and targets. We also use ``` `class``` for the problem type as we are dealing with a binary classification problem.\n", "\n", - "**NB:** For the purposes of this demonstration we will pass in a dictionary in place of the default parameter `(::)`. In order to ensure replication for users of this notebook the random seed parameter ``` `seed``` is set in this example with the remaining parameters defaulted." + "**Inportant:** \n", + "\n", + " For the purposes of this demonstration we will pass in a dictionary in place of the default parameter (::). In order to ensure replication for users of this notebook the random seed parameter `seed is set in this example with the remaining parameters defaulted." ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ - "tab:telcoInputs`xtrain / features\n", - "tgt:telcoInputs`ytrain / targets\n", - "ftype:`normal / normal feature extraction\n", - "ptype:`class / classification problem\n", - "dict:enlist[`seed]!enlist 42 / default configuration" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Outputs" + "telcoFeats :telcoInputs`xtrain / features\n", + "telcoTarget :telcoInputs`ytrain / targets\n", + "featureType1:`normal / normal feature extraction\n", + "problemType1:`class / classification problem\n", + "paramDict1 :enlist[`seed]!enlist 350 / default configuration" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "In the default configuration, the following items will be returned/saved during an individual run:\n", + "In the default configuration, information generated during the fitting of the model will we saved to the outputs folder. This includes metadata information, graphs, reports and the fitted model.\n", "\n", - "- The best model, saved as a hdf5 file for keras models, or \"pickled\" byte objects for sklearn models.\n", - "- A saved report indicating the procedure taken and scores achieved.\n", - "- A saved byte encoded dictionary denoting the procedure to be taken for reproducing results or running on new data.\n", - "- Results from each step of the pipeline published to console.\n", + "In addition to saving outputs, the function returns a dictionary with two keys:\n", "\n", - "In addition to the saved outputs, the function will also return the date and time of the current run. This allows users to run the best model from a defined run on new data by passing the date and time to `.automl.new` (see example [below](#Test-on-new-data)).\n", + " Return key | Description\n", + "-------------|:-------------\n", + " `modelInfo` | Metadata information generated from the pipeline such as preprocessing steps taken, significant features chosen and any other information needed to replicate the results.\n", + " `predict` | A function containing all relevant information and procedures required to generate new predictions using the fit model\n", "\n", - "We can now run `.aml.run` using the default setting with out training set from the Telco Customer Churn dataset." + "We can now run `.automl.fit` using the default setting with out training set from the Telco Customer Churn dataset." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "metadata": { "scrolled": false }, @@ -337,62 +326,131 @@ "name": "stdout", "output_type": "stream", "text": [ + "Executing node: automlConfig\n", + "Executing node: configuration\n", + "Executing node: targetDataConfig\n", + "Executing node: targetData\n", + "Executing node: featureDataConfig\n", + "Executing node: featureData\n", + "Executing node: dataCheck\n", + "Executing node: featureDescription\n", "\n", "The following is a breakdown of information for each of the relevant columns in the dataset\n", "\n", - " | count unique mean std min max type\n", - "-------| ----------------------------------\n", - "comment| 1350 1350 :: :: :: :: text\n", + "\n", + " | count unique mean std min max type \n", + "------ | --------------------------------------------------------\n", + "tenure | 4500 73 32.326 24.55931 0i 72i numeric \n", + "MonthlyCharges | 4500 1251 64.88498 30.49795 18.55 118.75 numeric \n", + "TotalCharges | 4500 3178 2284.252 2275.078 18.85 8672.45 numeric \n", + "customerID | 4500 3310 :: :: :: :: categorical\n", + "gender | 4500 2 :: :: :: :: categorical\n", + "Partner | 4500 2 :: :: :: :: categorical\n", + "Dependents | 4500 2 :: :: :: :: categorical\n", + "PhoneService | 4500 2 :: :: :: :: categorical\n", + "MultipleLines | 4500 3 :: :: :: :: categorical\n", + "InternetService | 4500 3 :: :: :: :: categorical\n", + "OnlineSecurity | 4500 3 :: :: :: :: categorical\n", + "OnlineBackup | 4500 3 :: :: :: :: categorical\n", + "DeviceProtection| 4500 3 :: :: :: :: categorical\n", + "TechSupport | 4500 3 :: :: :: :: categorical\n", + "StreamingTV | 4500 3 :: :: :: :: categorical\n", + "StreamingMovies | 4500 3 :: :: :: :: categorical\n", + "Contract | 4500 3 :: :: :: :: categorical\n", + "PaperlessBilling| 4500 2 :: :: :: :: categorical\n", + "PaymentMethod | 4500 4 :: :: :: :: categorical\n", + "SeniorCitizen | 4500 2 :: :: :: :: boolean \n", + "\n", + "\n", + "Executing node: dataPreprocessing\n", "\n", "Data preprocessing complete, starting feature creation\n", "\n", - "Feature creation and significance testing complete\n", + "Executing node: featureCreation\n", + "Executing node: labelEncode\n", + "Executing node: featureSignificance\n", + "\n", + "Total number of significant features being passed to the models = 40\n", + "\n", + "Executing node: trainTestSplit\n", + "Executing node: modelGeneration\n", + "Executing node: selectModels\n", + "\n", "Starting initial model selection - allow ample time for large datasets\n", "\n", - "Total features being passed to the models = 88\n", + "Executing node: runModels\n", "\n", - "Scores for all models, using .ml.accuracy\n", - "RandomForestClassifier | 0.7546512\n", - "GradientBoostingClassifier| 0.7512166\n", - "AdaBoostClassifier | 0.748891\n", - "SVC | 0.7476879\n", - "LinearSVC | 0.746552\n", - "MLPClassifier | 0.7454093\n", - "LogisticRegression | 0.7430972\n", - "binarykeras | 0.7338083\n", - "KNeighborsClassifier | 0.7291504\n", - "GaussianNB | 0.620453\n", + "Scores for all models using .ml.accuracy\n", + "\n", + "\n", + "RandomForestClassifier | 0.8482639\n", + "LogisticRegression | 0.7993056\n", + "GradientBoostingClassifier| 0.7989583\n", + "AdaBoostClassifier | 0.7920139\n", + "KNeighborsClassifier | 0.7666667\n", + "MLPClassifier | 0.7493056\n", + "GaussianNB | 0.7451389\n", + "SVC | 0.7225694\n", + "BinaryKeras | 0.7038194\n", + "LinearSVC | 0.59375\n", "\n", - "Best scoring model = RandomForestClassifier\n", - "Score for validation predictions using best model = 0.6944444\n", "\n", "\n", - "Feature impact calculated for features associated with RandomForestClassifier model\n", - "Plots saved in /outputs/2020.09.22/run_13.34.41.579/images/\n", + "Best scoring model = RandomForestClassifier\n", + "\n", + "Executing node: optimizeModels\n", "\n", "Continuing to hyperparameter search and final model fitting on testing set\n", "\n", - "Best model fitting now complete - final score on testing set = 0.7407407\n", + "\n", + "Best model fitting now complete - final score on testing set = 0.8655556\n", + "\n", "\n", "Confusion matrix for testing set:\n", "\n", - " | pred_0 pred_1\n", + "\n", + " | true_0 true_1\n", "------| -------------\n", - "true_0| 86 35 \n", - "true_1| 35 114 \n", + "pred_0| 619 49 \n", + "pred_1| 72 160 \n", + "\n", + "\n", + "Executing node: predictParams\n", + "Executing node: preprocParams\n", + "Executing node: pathConstruct\n", + "Executing node: saveGraph\n", + "\n", + "Saving down graphs to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.23.40.301/images/\n", + "\n", + "Executing node: saveReport\n", "\n", - "Saving down procedure report to /outputs/2020.09.22/run_13.34.41.579/report/\n", - "Saving down RandomForestClassifier model to /outputs/2020.09.22/run_13.34.41.579/models/\n", - "Saving down model parameters to /outputs/2020.09.22/run_13.34.41.579/config/\n", + "Saving down procedure report to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.23.40.301/report/\n", "\n", - ".automl.run took 00:02:30.535\n" + "Executing node: saveMeta\n", + "\n", + "Saving down model parameters to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.23.40.301/config/\n", + "\n", + "Executing node: saveModels\n", + "\n", + "Saving down model to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.23.40.301/models/\n", + "\n", + "\n", + ".automl.fit took 00:00:37.773\n", + "\n", + "Return of .automl.fit:\n", + "modelInfo| `startDate`startTime`featureExtractionType`problemType`saveOption`..\n", + "predict | {[config;features]\n", + " original_print:utils.printing;\n", + " utils.printi..\n", + "\n" ] } ], "source": [ "start:.z.t\n", - "r1:.automl.run[tab;tgt;ftype;ptype;dict]\n", - "-1\"\\n.automl.run took \",string .z.t-start;" + "model1:.automl.fit[telcoFeats;telcoTarget;featureType1;problemType1;paramDict1]\n", + "-1\"\\n.automl.fit took \",string .z.t-start;\n", + "-1\"\\nReturn of .automl.fit:\\n\",.Q.s[model1];" ] }, { @@ -403,19 +461,17 @@ "\n", "\n", "\n", - "We see that in the above example, 8 features were passed to the model following the application of feature extraction and significance testing. \n", + "We see that in the above example, 40 features were passed to the model following the application of feature extraction and significance testing. \n", "\n", - "**NB:** In the default case, normal feature extraction only uses the original features passed into the system, while FRESH feature extraction would apply the functions available for FRESH within the ML-Toolkit as defined by `.ml.fresh.params`.\n", - "\n", - "Looking at the feature impact above, we can see that `tenure` had the highest feature impact in the dataset when applied to the best model.\n", + "Looking at the feature impact plot above, we can see that `Contract_One year` had the highest feature impact in the dataset when applied to the best model, indicating this was the most important feature when generating predictions.\n", "\n", "#### Confusion matrix\n", "\n", "\n", "\n", - "A confusion matrix is also produced within the pipeline for classification problems. We see that the final `RandomForestClassifier` model correctly classified 724 data points. \n", + "A confusion matrix is also produced within the pipeline for classification problems. We see that the final `RandomForestClassifier` model correctly classified 779 data points. \n", "\n", - "All other outputs from this run have been stored in a directory of format `/outputs/date/run_time/`" + "All other outputs from this run have been stored in a directory of format `/outputs/dateTimeModels/date/run_time/`" ] }, { @@ -429,38 +485,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can apply the workflow and fitted model associated with our specified run to new data using:\n", - "\n", - "```.automl.new[tab;dt;tm]```\n", - "\n", - "Where:\n", - "\n", - "- `tab` is an unkeyed tabular dataset which has the same schema as the input data from the run specified in fpath\n", - "- `dt` is the date for a specified run as a date `yyyy.mm.dd` or a string of format `\"yyyy.mm.dd\"`\n", - "- `tm` is the timestamp for a specified run as a timestamp `hh:mm:ss.xxx` or a string of format `\"hh:mm:ss.xxx\"` or `\"hh.mm.ss.xxx\"` \n", - "\n", - "**NB:** Outputs from previous runs, such as `models` or `config`, are stored in the `outputs` directory and are organised such that we have the following file structure: `outputs/dt/run_tm/`, e.g. `outputs/2001.01.01/run_12.00.00.000\"`.\n", + "We can apply the workflow associated with our specified run to new data using the `predict` attribute returned\n", "\n", - "The function will return the target predictions for new data based on the previously fitted model and workflow.\n", - "\n", - "Below we apply `.automl.new` to the test set we created earlier and pass in the date and time of the previous run. This will apply the best model from the run above to our new data:" + "The function will return the target predictions for new data based on the previously fitted model and workflow." ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Run applied to dataset:\n", + "Model applied to dataset:\n", "\n", - "Run date: 2020.09.22. Run time: 13:10:17.224.\n", + "Model date: 2020.12.22. Model time: 13:23:40.301.\n", "\n", "Predictions: \n", - "0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 1 0 0 0 0 1 0..\n", + "0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0..\n", "\n", "Targets:\n", "0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0..\n" @@ -468,10 +512,10 @@ } ], "source": [ - "-1\"Run applied to dataset:\\n\";\n", - ".util.print_runid . r1;\n", + "-1\"Model applied to dataset:\\n\";\n", + ".util.printDateTimeId model1.modelInfo;\n", "-1\"\\nPredictions: \";\n", - "show pred:.automl.new[telcoInputs`xtest]. r1\n", + "show pred1:model1.predict[telcoInputs`xtest]\n", "-1\"\\nTargets:\";\n", "show telcoInputs`ytest" ] @@ -485,30 +529,30 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Accuracy on model run using hold-out data: 0.816\n" + "Accuracy on model run using hold-out data: 0.866\n" ] } ], "source": [ - "-1\"Accuracy on model run using hold-out data: \",string acc1:.ml.accuracy[telcoInputs`ytest;pred];" + "-1\"Accuracy on model run using hold-out data: \",string accuracy1:.ml.accuracy[telcoInputs`ytest;pred1];" ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "
" + "
" ] }, "metadata": {}, @@ -516,7 +560,7 @@ }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAcB0lEQVR4nO3deZhcZZn38e+ddAKELUASCJCw7+ACMVHx1bAIOBJgFGQZCCOjzLiM+ioir8OouBERhtHBGQYFhIRFFEXCvoOorMOroggTkJhAEhJCIGBYktzzxzlhyjbp1BO6u6rI93Nd5+qq85x6zn2qu3/9nKVOR2YiSWrOgFYXIEmdxNCUpAKGpiQVMDQlqYChKUkFDE1JKmBo6nUpIr4ZEU9HxOOvoY/tI2JBL5bVEhFxSkSc1eo6Xi8MzTYXEc83TEsjYlHD8795Df3eFRFHr2SZj0TEI/W6ZkfE1IhYq4m+D4iIaU0s9/aIuCEinq0D7q7Xsk0N/W4HfATYLjO3XNV+MvORzBz6WuvpLiLWjIiMiBkRMaBh/hoR8UxEvNhkP029z5n5xcz8+GupWf/L0GxzmbnOsgn4IzChYd5FfbXeiNgfOBl4f73uXYCf9GL/44EbgOuArYBhwCeA9/ZC91sAszNzfi/01Zf+BOzT8Pxg4KneXEFEdPVmfwIy06lDJuBxYN9u8wYC/ww8BswDLgKG1m1rA5cC84EFwN3ABsAZwBLgReB54IzlrOtk4NIealkL+FdgBjAb+DdgDWAjYBGwtO77eWCj5bz+vuWtt9syHwMeBZ4GfgxsXM9fE0jgw3X7M8CZdduB3dZ/NnAAMK1b37OBd9SP9wQeAJ6r559az98RWNzwmtHANfX7+QhwbEPbpPq9vwRYCPwaeNMKtmtZ/ScDkxvmXwX8E/Biw7y/B35f9zkNOK6ev9z3ua7jYuAH9WuOrud9r37dsXXta9fP/xqYCWzQ6p/vTplaXoBTwTdr+aF5EvAzYNP6l/H7wPl12yeBH9UB1wW8peGX5S7g6B7WtS/VSOgLwNuAwd3az677HgqsD1wPfLFu+4uQ6vbaoXVovK2HZf6qDrA31Nt1DnBj3bYsdH4MrEc1Ul0AjF/e+psIzQeAw+rH6wLj6sfdQ/Nu4EyqPw5j6vDcs26bVL9f76b6Q3YmcNsKtm1Z/TtSjSzXAUYAs4A3dwvNg+rti/p7sgjYpYftmgS8VL9/A+rv/auhWS9zef392xiYA7y71T/bnTS5e975/h44KTOfzMwXgVOAwyMigFeA4cA2mbk4M+/NzBea6TQzbwKOAMZRBeK8iPhGRAyod/mOAz6ZmQsy81mqX8wjmqx5o/rrrB6W+RvgnMz8db1dJwL7RMQmDct8PTOfy8w/AHcAb2py/d29AmwfERtl5sLMvLv7AvVx0jcCn8/MlzLzPuAC4JiGxW7JzBszcwkwuYl6nqc6RPF+4CiqP0KvNC6QmVdm5h+ychNwO/COlfR7e2Zek5lLM3PRctqPpwrjm6n2Jm5cSX9qYGh2sDoYRwHXRMSC+kzvA1Tf142Ac6l+yX4UETMj4usRMbDZ/utf2PdSjQwPozq5cgzVqHYQ8NuG9V5BNVpqxtP115E9LLMpML2hlgVUu8+bNSwzu+Hxn6hGbKviWKoR7SMRcXd9PHd59cztFkLTe6GeC4GJ9XRh98aIOCgi7omI+fX7vDfV8d+ezOipMTOfpjo+vTPwL03UqAaGZgfLal/rCWDvzBzaMK2ZmfPqEdEXMnNH4J1UwbdsNNj07a3qEcv1VKO5XalGiIupRrDL1rl+Zi4bQfbYdx2A91ONsFbkSaoTOgBExPpUu+JPNFt3gxeAIQ19DQI2bKjnocw8nCr0vw38OCIGL6ee4d2uHhi9ivU0ugnYHlgrM+9tbIiItYEfAl8BRmR1Jv8Wql11WPH73OP7HxFjgSPrvr+96qWvngzNznc2MCkiRgFExIiImFA/3jcidq4va3mOKuiW1K+bA2y9ok4j4tCIOCwihkbl7VQnTO7KzFeA84BvRcSwun1URLy7oe8REdHTSOsE4B8i4lMRsWHdxx4RMaVuvwT4cETsGhFrAt+g2v2dvcIeV+whYMOI2KcOzFNo+NmPiIn1rvkS4Fmq0FnarY9pVCd3vlpfGrQ71Qj1NV3BkJlLqY4/vm85zWtRjeifApZGxEHA+Ib2Zt7nPxMRQ6gOHXwG+Ftgh4g4bpWKX00Zmp3vNKrRyi0RsRD4BbB73bYZ8FOqs6gPUp35vaxuOxOYWF8XeNpy+n0G+CjV2ennqELylMy8vG7/FNXo6z6qoLkO2LZu+xVwJTC93n3fkG4y8zZgP6rAeJzqzP9ZwNV1+1XAqXU/TwKb8OfHD5uWmfOoTopdRHWmeHa9vmUOBB6u379TgQ9k5uJufSTwAapd2tlUZ6c/m5k/W5WauvX9m8x8aAV1nwBMpTqkcQjV93CZlb7Py3EG8FBmnl8fajgGOD0itnxtW7H6iOpnQZLUDEeaklTA0JSkAoamJBUwNCWpQNt+mD+61soYvG6ry1CHevNOo1tdgjrY9OmPM2/evFheW/uG5uB1WWOHD7S6DHWon9/t7SO16vYcN2aFbe6eS1IBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBbpaXYD+0hqDu7jp3E8xeHAXXQMH8pObHuCrZ1/DBusNYfI3jmOLTTdk+pPzOfrEc1mwcBGDugZy1slHsvvOo1maSznhtMv52f3/3erNUBtZsmQJe44bw6abbcaPf3oV8+fP55ijDmf69MfZYostmXLJZWywwQatLrMjONJsQy+9vJgDjv824w6fxLgjTmW/t+/M2N225IQPvpvb7nmY3Q7+Mrfd8zAnfHA/AI57354AvOUDX+fAfziLSZ/+ayKilZugNnPWt7/FDjvt9Orz00+bxPi99+HBh/6b8Xvvw+mnTWphdZ3F0GxTLyx6GYBBXQPp6hpIZnLg+DcwZerdAEyZejcT9noDADtuvQm33vMwAHOfeZ5nFy5ij51Ht6ZwtZ2ZM2dy3bVX88HjPvTqvKum/pSjjzkWgKOPOZapV17RqvI6jqHZpgYMCO669CT+ePMkbrnr99z74HRGbLQus+c9B8Dsec8xfMN1AfjNI08wYfxuDBw4gC023Yg37zyKzTdxV0uVz37mU3zt1NMYMOB/f92fmjOHkSNHAjBy5EjmPvVUq8rrOP0WmhFxQEQ8HBHTIuKk/lpvp1q6NHnrEZPYdv+TGbPrFuy8zcgVLnvBT3/JE3MW8POLTuSbn30/d/3qDyxesqQfq1W7uubqqxgxfAS777FHq0t53eiXE0ERMRD4DvBuYCZwb0RcmZm/64/1d7Jnn1/EHff9N/u9fWeeenohmwxbj9nznmOTYesxd/5CAJYsWcqJZ/z41dfc+v1PM+2Pc1tVstrIL3/xc6666kquu+4aXnrxRZ577jk+OPFoRmy8MbNmzWLkyJHMmjWL4SNGtLrUjtFfI82xwLTMfCwzXwYuBQ7up3V3nGEbrMP666wFwJprDGLvcTvw8ONzuPr233D0hHEAHD1hHFfd9msA1lpzEEPWHAzA3uN2ZPGSpfz+sdmtKV5t5StfO5VHH5/Jw9Me58KLLmX8Xntz/oVTeO+BBzFl8gUATJl8AQdO8NexWf11ydFmwIyG5zOBcf207o6zybD1+O6Xj2HggAEMGBBcfuN/ce3PHuTuX/+BKd84jmMPeRszZj3D35x4LgDDN1iXqf/+MZYuTZ6cu4C/O/mCFm+B2t0JJ57E0Ud+gAvOP5dRo0Zz0aU/bHVJHSMys+9XEnEYsH9mfqh+fgwwNjP/sdtyxwPHAzBonT3W3OXYPq9Nr0/P3HtWq0tQB9tz3Bjuv/++5V6311+75zOBUQ3PNwee7L5QZp6TmWMyc0x0rdVPpUlS8/orNO8FtouIrSJiMHAEcGU/rVuSek2/HNPMzMUR8XHgemAgcF5m/rY/1i1JvanfPnuemdcA1/TX+iSpL/iJIEkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSrQtaKGiGgqUDNzae+VI0ntbYWhCSwGsof2qNsH9mpFktTGegrNrfqtCknqECsMzcyc3n1evcu+cWbO6tOqJKlNNXXcMiKGRsTFwIvAtHreQRHx1b4sTpLaTbNnz88GngW2AF6u5/0SOLwvipKkdtXTMc1G+wCbZuYrEZEAmTk3Ikb0XWmS1H6aHWk+CwxrnBERowGPbUparTQbmt8DLo+IvYABEfE24AKq3XZJWm00u3v+DaqTQN8BBgHnAf8JfKuP6pKkttRUaGZmAv9aT5K02mp2pElE7A0cCWwKPAlcmpk391VhktSOmr1O89PApcB84GrgaeDiiPhMH9YmSW2n2ZHmZ4C9M/PBZTMiYjJwI3BGXxQmSe2o5NZw07o9f4yeb+ghSa87KwzNiBiwbAK+BJwbEdtFxFoRsT1wDvDFfqpTktpCs7eGi/rrkd3mHUV1DackrRa8NZwkFSi6NZwkre5KrtM8CHgX1WfQl+2uk5kT+6AuSWpLzV6n+UWqj00OAA6juk5zf2BB35UmSe2n2UuOjgPenZn/F3i5/joB2LKvCpOkdtRsaA5tuLD95YgYlJn3UO2uS9Jqo9ljmo9GxC6Z+VvgQeAjEfEM8EzflSZJ7afZ0DwZ2Kh+/P+Ai4B1gI/2RVGS1K6avTXcNQ2P7wa27bOKJKmNrTA0I2LrZjrIzMd6rxxJam89jTSnUX1kMnpYJoGBvVqRJLWxnj4RVHIHJElaLRiMklTA0JSkAoamJBUwNCWpQNN3Oepvb9xxNLfc6b9V16p5cMazrS5BHWzRy0tW2NbTdZozaOJ/AGXm6FUrS5I6T08jzaP7rQpJ6hA9Xad5e38WIkmdoOTO7W8C/g9/eef2L/RBXZLUlpq9c/vxwM+BvYHPAbsBn8Ebd0hazTR7ydGJwAGZ+dfAovrrocArfVaZJLWhZkNzRGb+rH68NCIGZOa1VP/yQpJWG80e05wZEVtm5uPAI8DBETEPeLnPKpOkNtRsaJ4G7AQ8DnwZ+BEwGPhE35QlSe2p2Tu3f7/h8bURsQEwODOf76vCJKkdNRWaEdH92OdiYHF9bHNp75clSe2p2d3zxaz4I5XeuV3SaqPZ0Nyq2/ORwEnA1N4tR5LaW7PHNKd3mzU9Io4F7gXO7fWqJKlNvZb7aa4HDO+tQiSpEzR7Imgyf35McwjwTmBKXxQlSe2q2WOa07o9fwE4OzNv6uV6JKmtNRua12Xm3d1nRsTYzLynl2uSpLbV7DHNG1cw/7reKkSSOkGPI836ovaoHkbQcB9NYBuq6zclabWxst3zxovauwfkUuBrvV6RJLWxlYXmVlSjy9upzpYvk8DczFzUV4VJUjvqMTSXXdQeETsASzLz1ZsOR8SgiFgjM1/q4xolqW00eyLoBmCPbvP2AK7v3XIkqb01G5pvALpfcnQP8MbeLUeS2luzobkA2LjbvI2pLnKXpNVGs6F5OXBxROwaEUMiYjfgQuCyvitNktpPs6H5T8BDVLvkC4G7gIeBz/dRXZLUlpq9NdyLwMci4uPAMGBeZuZy7uguSa9rRaGXlbnArhHxTWBm35QlSe2p6dCMiOER8cmI+C/g/wNjgU/2WWWS1IZW9tnzQcBBwN8C+1PdIu4SYAvgsMx8qq8LlKR2srKR5hzgP6lO+rw1M3fOzK8AL/d5ZZLUhlYWmr8GhgLjgLfU/+9cklZbPYZmZo6nugXcDcAJwOyImAqsDQzq8+okqc2s9ERQZk7PzK9k5nbAPsAsqtvC/SoiTuvrAiWpnZRecnRnZh4PbAL8I7Bbn1QlSW1qlS5Oz8wXM/OSzHxPbxckSe3MT/RIUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKdLW6AK3cG3fahnXWWZeBAwfS1dXFLXfezRc+fyLXX3s1gwYNZqutt+ass89l/aFDW12q2tAl5/8HV/zgQjKTQw6fyFHHfZRHHvoNk07+NH964QVGbj6Kr5z5XdZZd71Wl9oRHGl2iCuvvYk77rqfW+68G4Dxe+/Lz+/9FXfe8wDbbLsdZ54+qcUVqh1Ne/h3XPGDC7ngJzdz8dV3cuct1/PHPzzKV0/6BB878Ytcet0v2Gu/A5n83W+3utSOYWh2qL333Y+urmpHYczYt/LkE0+0uCK1o8cffYTd3jSGNdcaQldXF7uP25PbbriKP/5hGruP3ROAse/Yi1uvm9riSjuHodkBIoL3H/Qe9tpzLN8/77t/0X7Rheez734HtKAytbtttt+JB+75BQuemc+Li/7EL267kTmzZrL19jtxx03XAHDzNVcwZ5Z/dJvVL8c0I+I84EDgqczctT/W+Xpy7c13MHLkpsx96ineN+EAtt9+B97+jncCcMZpX6erq4vDjjiqxVWqHW217Q5M/PtP8vGJhzBkyNpst+OuDBzYxRe+cRann/I5vvdvp/HOfd7DoEGDWl1qx+ivkeb3AYdCq2jkyE0BGD5iBO896GDuv+9eAC6ZciHXX3s1/3neZCKilSWqjR18+ESmTL2Dc35wLesN3YBRW27Dlttsz1kX/oTJV97OfhMOZbPRW7W6zI7RL6GZmXcA8/tjXa83L7zwAgsXLnz18a0338hOO+/CTTdcx7fO/CYXX3YFQ4YMaXGVamfz580FYPYTM7j1+qnsf9Chr85bunQp533nm7z/qA+2ssSO4iVHbW7uU3M45ohDAVi8ZDGHfuAI9t3vAPbYbQdeeukl3jehGsCPGTuOf/n2v7eyVLWpz310Is8umE9XVxcnnnI6660/lEvO/w9+NPl7AIzffwITDju6xVV2jsjM/llRxJbAVT0d04yI44HjATYfNXqPX//+sX6pTa8/j855vtUlqINNPGg8v/vNA8s95tVWZ88z85zMHJOZY4YNG97qciTpL7RVaEpSu+uX0IyIS4BfAjtExMyI+Lv+WK8k9bZ+ORGUmUf2x3okqa+5ey5JBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUoHIzFbXsFwRMReY3uo62tgwYF6ri1DH8uenZ1tk5vDlNbRtaKpnEXFfZo5pdR3qTP78rDp3zyWpgKEpSQUMzc51TqsLUEfz52cVeUxTkgo40pSkAoamJBUwNDtMRBwQEQ9HxLSIOKnV9aizRMR5EfFURDzY6lo6laHZQSJiIPAd4D3AzsCREbFza6tSh/k+cECri+hkhmZnGQtMy8zHMvNl4FLg4BbXpA6SmXcA81tdRyczNDvLZsCMhucz63mS+omh2VliOfO8ZkzqR4ZmZ5kJjGp4vjnwZItqkVZLhmZnuRfYLiK2iojBwBHAlS2uSVqtGJodJDMXAx8HrgceAi7LzN+2tip1koi4BPglsENEzIyIv2t1TZ3Gj1FKUgFHmpJUwNCUpAKGpiQVMDQlqYChKUkFDE21TERsGREZEV3182sj4th+WO+XImJKL/f5Z9vSX69V/zM01aOIeDwiFkXE8xExJyLOj4h1+mJdmfmezLygyZr27YsaImJ8RMzsi771+mBoqhkTMnMdYHfgLcDJ3ReIij9Pet3zh1xNy8wngGuBXQEi4raI+FpE/Bz4E7B1RKwfEedGxKyIeCIivlrfB5SIGBgRp0fEvIh4DHhvY/91fx9qeP7hiHgoIhZGxO8iYveImAyMBqbWo98T62XfGhG/iIgFEfGriBjf0M9WEXF73c+NwLBV2f6IeG9EPBARz0XEjIj40nIWOy4inqy3/zMNrx0QESdFxKMR8XREXBYRG65KHWotQ1NNi4hRwF8BDzTMPgY4HlgXmA5cACwGtgXeDOwHLAvCDwMH1vPHAIf2sK7DgC8BE4H1gIOApzPzGOCP1KPfzDwtIjYDrga+CmwInABcHhHD6+4uBu6nCsuvAKt63PSFup6hVIH/kYg4pNsyewHb1dt9UsNhhE8AhwDvAjYFnqG6obQ6TWY6Oa1wAh4HngcWUIXivwNr1W23AV9uWHZj4KVl7fW8I4Fb68e3AP/Q0LYf1a3tuhr6+1D9+Hrgkz3UtG/D888Bk7stcz1VOI6mCvG1G9ouBqasoO/xwMwm35t/Bc6sH29Zb8uODe2nAefWjx8C9mloGwm8AnQ1vLar1d9vp5VPnq1TMw7JzJtW0NZ4U+QtgEHArIhXb/05oGGZTbstP72HdY4CHm2yvi2AwyJiQsO8QcCt9TqfycwXuq238RZ7TYmIccAkqsMTg4E1gB92W6z79u3WUONPImJpQ/sSqj806iDunuu1arzjywyqkeawzBxaT+tl5i51+yz+PKxG99DvDGCbJta5bNnJDescmplrZ+akep0bRMTaTa63JxdT3YpvVGauD5zNX94Yuvv2Lbvf6QzgPd1qXDOr48TqIIamek1mzgJuAM6IiPXqkx/bRMS76kUuAz4REZtHxAZAT/9N83vACRGxR31mftuI2KJumwNs3bDsFGBCROxfn2xas750aPPMnA7cB5wSEYMj4h3ABFai7qNxCqrjtvMz88WIGAsctZyX/nNEDImIXYAPAj+o558NfG3ZNkTE8Ijw/zt1IENTvW0i1a7r76hOdvyI6vgdwHepjjX+Cvgv4Mcr6iQzfwh8jWp0txC4guokD8CpwMn1mfITMnMG1T+Y+zwwl2pU91n+9+f7KGAc1T8U+yJw4Uq2YTNgUbdpG+CjwJcjYiHwBao/At3dDkwDbgZOz8wb6vnfohql3lC//q66JnUY76cpSQUcaUpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKvA/rJASlnPEWjQAAAAASUVORK5CYII=", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbqElEQVR4nO3deZgcZbn38e8dJmGHgAQIssoWNmUzUfRo2HeEI1tEQFFxFxVUVBQQ0YjyoohH5LATICBwVJBAWAQRBEFxQVEMmJBAwhogIBFC7vePqsRmTCb9hJnp7uT7ua66prue6qfu7un5zVNLV0dmIklqzoBWFyBJncTQlKQChqYkFTA0JamAoSlJBQxNSSpgaGqRFBHfjoinImLia+hjo4h4phfLaomIODEizmh1HYsKQ7PNRcTzDdPsiHix4f4hr6HfOyPivQtY5qMR8UC9rmkRcXVELN1E37tFxIQmltsuIsZHxLN1wN35Wp5TQ78bAh8FNszMdRe2n8x8IDMHv9Z6uouIpSIiI2JyRAxomL9kREyPiJlN9tPU65yZx2fmJ15Lzfo3Q7PNZeZycybgYWDvhnkX99V6I2JX4Djg3fW6NwP+rxf7HwmMB64D1gNWAT4F7NkL3a8DTMvMp3uhr770T2DHhvvvAh7vzRVERFdv9icgM506ZAImAjt1m7cE8BXgIeBJ4GJgcN22LDAWeBp4BrgLWAk4FXgFmAk8D5w6j3UdB4ztoZalge8Ck4FpwPeBJYHXAS8Cs+u+nwdeN4/H3zOv9XZb5uPAg8BTwFXAavX8pYAEPlS3TwdOq9v26rb+M4HdgAnd+p4GvL2+/TbgXuC5ev436/nDgFkNj1kbuLZ+PR8ADm9oG12/9pcCM4A/AlvO53nNqf844KKG+dcAXwZmNsz7MPDXus8JwBH1/Hm+znUdlwCX1Y95bz3v7Ppxh9e1L1vf3w+YAqzU6vd3p0wtL8Cp4Jc179A8FrgNWKP+YzwfOK9uOwq4og64LuDNDX8sdwLv7WFdO1GNhL4KvBUY1K39zLrvwcCKwPXA8XXbf4RUt8cOrkPjrT0ss0cdYG+sn9dZwA1125zQuQpYgWqk+gwwcl7rbyI07wUOqG8vD4yob3cPzbuA06j+OWxbh+fb6rbR9eu1M9U/stOAW+bz3ObUP4xqZLkcsCowFdiqW2juUz+/qH8nLwKb9fC8RgP/ql+/AfXvfm5o1stcWf/+VgMeA3Zu9Xu7kyY3zzvfh4FjM/PRzJwJnAgcFBEBvAwMAdbPzFmZeXdmvtBMp5l5I3AwMIIqEJ+MiG9FxIB6k+8I4KjMfCYzn6X6wzy4yZpfV/+c2sMyhwBnZeYf6+f1eWDHiFi9YZlvZOZzmfkP4JfAlk2uv7uXgY0i4nWZOSMz7+q+QL2f9E3AlzLzX5l5D3ABcGjDYjdn5g2Z+QpwURP1PE+1i+LdwHuo/gm93LhAZv4sM/+RlRuBW4G3L6DfWzPz2sycnZkvzqP9SKowvolqa+KGBfSnBoZmB6uDcS3g2oh4pj7Sey/V7/V1wDlUf2RXRMSUiPhGRCzRbP/1H+yeVCPDA6gOrhxKNaodCPy5Yb0/oRotNeOp+ufQHpZZA5jUUMszVJvPr29YZlrD7X9SjdgWxuFUI9oHIuKuen/uvOp5olsITeqFei4EDqunC7s3RsQ+EfGbiHi6fp13oNr/25PJPTVm5lNU+6c3Bf5fEzWqgaHZwbLa1noE2CEzBzdMS2Xmk/WI6KuZOQx4B1XwzRkNNn15q3rEcj3VaG5zqhHiLKoR7Jx1rpiZc0aQPfZdB+BvqUZY8/Mo1QEdACJiRapN8UearbvBC8AyDX0NBFZuqOf+zDyIKvRPB66KiEHzqGdIt7MH1l7IehrdCGwELJ2Zdzc2RMSywI+Bk4BVszqSfzPVpjrM/3Xu8fWPiOHAqLrv0xe+9MWTodn5zgRGR8RaABGxakTsXd/eKSI2rU9reY4q6F6pH/cY8Ib5dRoR+0fEARExOCrbUR0wuTMzXwbOBb4XEavU7WtFxM4Nfa8aET2NtI4BPhIRn46Iles+tomIMXX7pcCHImLziFgK+BbV5u+0+fY4f/cDK0fEjnVgnkjDez8iDqs3zV8BnqUKndnd+phAdXDn6/WpQVtTjVBf0xkMmTmbav/jf8+jeWmqEf3jwOyI2AcY2dDezOv8KhGxDNWug6OB9wEbR8QRC1X8YsrQ7HynUI1Wbo6IGcAdwNZ12+uBn1IdRb2P6sjv5XXbacBh9XmBp8yj3+nAx6iOTj9HFZInZuaVdfunqUZf91AFzXXABnXbH4CfAZPqzfeV6SYzbwF2oQqMiVRH/s8Afl63XwN8s+7nUWB1Xr3/sGmZ+STVQbGLqY4UT6vXN8dewN/q1++bwIGZOatbHwkcSLVJO43q6PTnMvO2hampW99/ysz751P3McDVVLs09qX6Hc6xwNd5Hk4F7s/M8+pdDYcC34mIdV/bs1h8RPVekCQ1w5GmJBUwNCWpgKEpSQUMTUkq0LYf5o+upTMGLd/qMtSh3jRs7VaXoA728MMTeerJJ2Nebe0bmoOWZ8mND2x1GepQv/jV91pdgjrY9m8fMd82N88lqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChmYbWnJQF7dddAx3XXYsv73iyxz3kT0A+OrH9uQ3l32RO8cey9X/83GGDlnxVY9ba/WVeOL2U/n0oTu2omy1sVdeeYV3vHVbDnr3PgBMf/pp9ttrV7Z54zD222tXnpk+vcUVdg5Dsw3966VZ7Hbk6Yw4aDQjDv4mu2y3KcO3WJfTLriJ4Qd9k7ccPJpxt93HF4/c/VWPO+WYdzP+9j+3qGq1szN/cDobbTxs7v3TTv0W7xi5A7/94195x8gdOO3Ub7Wwus5iaLapF158CYCBXUvQ1bUEmcmMF2bObV9m6SXJzLn39x75Rv4x5Un+8uC0fq9V7e2RR6Yw/rprOex9R8ydN+7nVzPqkMMAGHXIYVx7zc9aVV7HMTTb1IABwZ1jj+Xhm0Zz851/5e77JgFwwsf35u/jTuLg3bflpB/+HIBllhrE0e/fmZN/dG0rS1ab+tLnP8uJJ49mwIB//7k//vhjrD50KACrDx3KE0883qryOk6/hWZE7BYRf4uICRFxbH+tt1PNnp285eDRbLDrcWy7+Tpsun71Bj/hB1ez4e5fYey4e/jIQe8A4Csf3ZPvj7l57uhUmuO6cdewypBV2XKrbVpdyiKjqz9WEhFLAD8AdgamAHdHxM8y8y/9sf5O9uzzL/LLe/7OLtttyl8enDp3/uXj7uaq0z/K18+8ljdvvg777bQlJ396X1Zcfmlmz05mvvQyZ172yxZWrnZw16/v4LqfX80N14/jXzNnMmPGcxx5xGGsuupqTJs6ldWHDmXa1KkMGbJqq0vtGP010hwOTMjMhzLzJWAs8K5+WnfHWWWl5VhxuaUBWGrJgewwYmP+NvEx1l97yNxl9nznG3lg4mMA7PSB7zJsz+MZtufxnHHxLXz7nPEGpgA4/mvf4M9/n8Qf73+Qcy64mP965/acde6F7LbHXlx68YUAXHrxhey+594trrRz9MtIE3g9MLnh/hRgRD+tu+OsvsoK/O/XDmWJAQMYMCC48obfMe62+7j0Ox9kw3VWZfbs5OGpT/Opk8e2ulR1qM8c/QXef+jBjLnwPNZccy3OH3NZq0vqGNF4BLbPVhJxALBrZn6wvn8oMDwzP9ltuSOBIwEYuNw2S212eJ/XpkXT1Nu/1+oS1MG2f/sI7v3dPTGvtv7aPJ8CrNVwf03g0e4LZeZZmbltZm4bXUv3U2mS1Lz+Cs27gQ0jYr2IGAQcDHhimKSO0y/7NDNzVkR8ArgeWAI4NzP96IqkjtNfB4LIzGsBz76W1NH8RJAkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQV6JpfQ0Q0FaiZObv3ypGk9jbf0ARmAdlDe9TtS/RqRZLUxnoKzfX6rQpJ6hDzDc3MnNR9Xr3JvlpmTu3TqiSpTTW13zIiBkfEJcBMYEI9b5+I+HpfFidJ7abZo+dnAs8C6wAv1fN+DRzUF0VJUrvqaZ9mox2BNTLz5YhIgMx8IiJW7bvSJKn9NDvSfBZYpXFGRKwNuG9T0mKl2dA8G7gyIrYHBkTEW4ELqDbbJWmx0ezm+beoDgL9ABgInAv8CPheH9UlSW2pqdDMzAS+W0+StNhqdqRJROwAjALWAB4FxmbmTX1VmCS1o2bP0/wsMBZ4Gvg58BRwSUQc3Ye1SVLbaXakeTSwQ2beN2dGRFwE3ACc2heFSVI7Krk03IRu9x+i5wt6SNIiZ76hGRED5kzACcA5EbFhRCwdERsBZwHH91OdktQWmr00XNQ/R3Wb9x6qczglabHgpeEkqUDRpeEkaXFXcp7mPsA7qT6DPmdzncw8rA/qkqS21Ox5msdTfWxyAHAA1XmauwLP9F1pktR+mj3l6Ahg58z8DPBS/XNvYN2+KkyS2lGzoTm44cT2lyJiYGb+hmpzXZIWG83u03wwIjbLzD8D9wEfjYjpwPS+K02S2k+zoXkc8Lr69heBi4HlgI/1RVGS1K6avTTctQ237wI26LOKJKmNzTc0I+INzXSQmQ/1XjmS1N56GmlOoPrIZPSwTAJL9GpFktTGevpEUMkVkCRpsWAwSlIBQ1OSChiaklTA0JSkAk1f5ai/bbXJ2tx+1xmtLkMd6g+TvJaMFt7Ml1+Zb1tP52lOponvAMrMtReuLEnqPD2NNN/bb1VIUofo6TzNW/uzEEnqBCVXbt8S+C/+88rtX+2DuiSpLTV75fYjgduBHYAvAFsAR+OFOyQtZpo95ejzwG6ZuR/wYv1zf+DlPqtMktpQs6G5ambeVt+eHREDMnMc1VdeSNJio9l9mlMiYt3MnAg8ALwrIp4EXuqzyiSpDTUbmqcAmwATga8BVwCDgE/1TVmS1J6avXL7+Q23x0XESsCgzHy+rwqTpHbUVGhGRPd9n7OAWfW+zdm9X5YktadmN89nMf+PVHrldkmLjWZDc71u94cCxwJX9245ktTemt2nOanbrEkRcThwN3BOr1clSW3qtVxPcwVgSG8VIkmdoNkDQRfx6n2aywDvAMb0RVGS1K6a3ac5odv9F4AzM/PGXq5Hktpas6F5XWbe1X1mRAzPzN/0ck2S1Laa3ad5w3zmX9dbhUhSJ+hxpFmf1B7VzQgarqMJrE91/qYkLTYWtHneeFJ794CcDZzc6xVJUhtbUGiuRzW6vJXqaPkcCTyRmS/2VWGS1I56DM05J7VHxMbAK5k596LDETEwIpbMzH/1cY2S1DaaPRA0Htim27xtgOt7txxJam/NhuYbge6nHP0GeFPvliNJ7a3Z0HwGWK3bvNWoTnKXpMVGs6F5JXBJRGweEctExBbAhcDlfVeaJLWfZkPzy8D9VJvkM4A7gb8BX+qjuiSpLTV7abiZwMcj4hPAKsCTmZnzuKK7JC3SikIvK08Am0fEt4EpfVOWJLWnpkMzIoZExFER8Tvg98Bw4Kg+q0yS2tCCPns+ENgHeB+wK9Ul4i4F1gEOyMzH+7pASWonCxppPgb8iOqgz1syc9PMPAl4qc8rk6Q2tKDQ/CMwGBgBvLn+vnNJWmz1GJqZOZLqEnDjgWOAaRFxNbAsMLDPq5OkNrPAA0GZOSkzT8rMDYEdgalUl4X7Q0Sc0tcFSlI7KT3l6FeZeSSwOvBJYIs+qUqS2tRCnZyemTMz89LM3L23C5KkduYneiSpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMzTY3efJkdt1pe7bcYhO2ftNmnHH69wB473sOYsQ2WzJimy3ZeIN1GbHNli2uVO3qsvPP5D27v5VRu72Vsef9EIC/3/8nPrj/Lhyyx3Yc/aGDeWHGcy2usnN0tboA9ayrq4vRp5zKVltvzYwZM9huxDbsuNPOjLnksrnLfOFzR7Piiiu2sEq1qwcf+As/vewCzr3qJroGDuLTR+zPdtvvwje+dBSfPPYkth7xNq7+8RjGnP19PvyZL7e63I7gSLPNDR06lK223hqA5ZdfnmHDNuHRRx+Z256ZXHnF5Rx40KhWlag2NnHCA2y25ZtZaull6OrqYuvhb+PW8dcw6aEJbDV8OwCGv20kv7ju6hZX2jkMzQ4yaeJEfv/7e3nz8BFz593+q9tYbdXV2GDDDVtYmdrVGzbahN/ffQfPTn+amS/+kztuuYHHpj7C+hsN47YbxwFw07if8vi0RxbQk+bol9CMiHMj4vGIuK8/1rcoev755xl14Lv59qnfZYUVVpg7//Kxl3LAwY4yNW/rbbAxhx55FJ88fD8+fcT+bLjJZnQt0cWXR5/BFWPO5vB3jeSfLzxP18CBrS61Y/TXPs3zgTOAC/tpfYuUl19+mVEHvpuDRh3Cvvv999z5s2bN4qc/uYrb7/ptC6tTu9vnwEPZ58BDAfjhd77GkNXXYN31N+L0C64C4OF/TOCOW8a3ssSO0i8jzcz8JfB0f6xrUZOZfORDH2DjYZtw1Gc++6q2m2+6kY02Hsaaa67ZourUCZ5+6gkApj06mVvGX8Mue+8/d97s2bM57wffYb9R729liR3Fo+dt7o7bb+eSiy9i8823mHta0Ylf/wa77b4HP75srAeAtEBf/PhhPDt9Ol0DuzjmhG+zwoqDuez8M7lizNkAjNxlL/ba/5AWV9k5IjP7Z0UR6wLXZObmPSxzJHAkwFprr73NAw9O6pfatOj5w6RnWl2COtj79t2e+/90b8yrra2OnmfmWZm5bWZuO2SVIa0uR5L+Q1uFpiS1u/465ehS4NfAxhExJSI+0B/rlaTe1i8HgjLToxWSFglunktSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVCAys9U1zFNEPAFManUdbWwV4MlWF6GO5funZ+tk5pB5NbRtaKpnEXFPZm7b6jrUmXz/LDw3zyWpgKEpSQUMzc51VqsLUEfz/bOQ3KcpSQUcaUpSAUNTkgoYmh0mInaLiL9FxISIOLbV9aizRMS5EfF4RNzX6lo6laHZQSJiCeAHwO7ApsCoiNi0tVWpw5wP7NbqIjqZodlZhgMTMvOhzHwJGAu8q8U1qYNk5i+Bp1tdRyczNDvL64HJDfen1PMk9RNDs7PEPOZ5zpjUjwzNzjIFWKvh/prAoy2qRVosGZqd5W5gw4hYLyIGAQcDP2txTdJixdDsIJk5C/gEcD1wP3B5Zv65tVWpk0TEpcCvgY0jYkpEfKDVNXUaP0YpSQUcaUpSAUNTkgoYmpJUwNCUpAKGpiQVMDTVMhGxbkRkRHTV98dFxOH9sN4TImJML/f5qufSX49V/zM01aOImBgRL0bE8xHxWEScFxHL9cW6MnP3zLygyZp26osaImJkREzpi761aDA01Yy9M3M5YGvgzcBx3ReIiu8nLfJ8k6tpmfkIMA7YHCAibomIkyPiduCfwBsiYsWIOCcipkbEIxHx9fo6oETEEhHxnYh4MiIeAvZs7L/u74MN9z8UEfdHxIyI+EtEbB0RFwFrA1fXo9/P18u+JSLuiIhnIuIPETGyoZ/1IuLWup8bgFUW5vlHxJ4RcW9EPBcRkyPihHksdkREPFo//6MbHjsgIo6NiAcj4qmIuDwiVl6YOtRahqaaFhFrAXsA9zbMPhQ4ElgemARcAMwCNgC2AnYB5gThh4C96vnbAvv3sK4DgBOAw4AVgH2ApzLzUOBh6tFvZp4SEa8Hfg58HVgZOAa4MiKG1N1dAvyWKixPAhZ2v+kLdT2DqQL/oxGxb7dltgc2rJ/3sQ27ET4F7Au8E1gDmE51QWl1msx0cprvBEwEngeeoQrF/wGWrttuAb7WsOxqwL/mtNfzRgG/qG/fDHykoW0XqkvbdTX098H69vXAUT3UtFPD/S8AF3Vb5nqqcFybKsSXbWi7BBgzn75HAlOafG2+C5xW3163fi7DGtpPAc6pb98P7NjQNhR4GehqeGxXq3/fTguePFqnZuybmTfOp63xosjrAAOBqRFzL/05oGGZNbotP6mHda4FPNhkfesAB0TE3g3zBgK/qNc5PTNf6LbexkvsNSUiRgCjqXZPDAKWBH7cbbHuz2+Lhhr/LyJmN7S/QvWPRh3EzXO9Vo1XfJlMNdJcJTMH19MKmblZ3T6VV4fV2j30OxlYv4l1zln2ooZ1Ds7MZTNzdL3OlSJi2SbX25NLqC7Ft1ZmrgicyX9eGLr785tzvdPJwO7dalwqq/3E6iCGpnpNZk4FxgOnRsQK9cGP9SPinfUilwOfiog1I2IloKdv0zwbOCYitqmPzG8QEevUbY8Bb2hYdgywd0TsWh9sWqo+dWjNzJwE3AOcGBGDIuLtwN4sQN1H4xRU+22fzsyZETEceM88HvqViFgmIjYD3g9cVs8/Ezh5znOIiCER4fc7dSBDU73tMKpN179QHey4gmr/HcD/Uu1r/APwO+Cq+XWSmT8GTqYa3c0AfkJ1kAfgm8Bx9ZHyYzJzMtUXzH0JeIJqVPc5/v3+fg8wguoLxY4HLlzAc3g98GK3aX3gY8DXImIG8FWqfwLd3QpMAG4CvpOZ4+v536MapY6vH39nXZM6jNfTlKQCjjQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBX4/1wy7VGkDXsiAAAAAElFTkSuQmCC", "text/plain": [ "
" ] @@ -526,7 +570,7 @@ } ], "source": [ - "displayCM[value .ml.confmat[telcoInputs`ytest;pred];`0`1;\"Test Set Confusion Matrix\";()];" + ".util.displayCM[value .ml.confmat[telcoInputs`ytest;pred1];`0`1;\"Test Set Confusion Matrix\";()];" ] }, { @@ -547,7 +591,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The function `.automl.run` can also be applied to textual data using its default configuration.\n", + "The function `.automl.fit` can also be applied to text data using its default configuration.\n", "\n", "As with the example above, data must be presented with a 1-to-1 mapping between features and targets.\n", "\n", @@ -560,7 +604,7 @@ "source": [ "### IMBD Dataset\n", "\n", - "The [IMBD](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews) dataset contains reviews of over 50,000 movies for NLP or text analytics. The dataset consists of 2 columns, containing textual reviews and their associated positive or negative sentiment classification." + "The [IMBD](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews) dataset contains reviews of over 50,000 movies for NLP or text analysis. The dataset consists of 2 columns, containing text reviews and the target indicating if they were positively or negatively reviewed." ] }, { @@ -579,14 +623,14 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Shape of feature data is: 1500 x 1 x 585\n", + "Shape of feature data is: 1000 x 1 x 585\n", "\n", "comment \n", "-------------------------------------------------------------------------------\n", @@ -598,16 +642,16 @@ "\n", "Distribution of target values:\n", "\n", - "target| num pcnt \n", - "------| ---------\n", - "0 | 740 49.33\n", - "1 | 760 50.67\n" + "target| num pcnt\n", + "------| --------\n", + "0 | 477 47.7\n", + "1 | 523 52.3\n" ] } ], "source": [ "// load data\n", - "imdbData:1500#(\"SI\";enlist \",\")0:`:../data/IMBD.csv\n", + "imdbData:1000#(\"SI\";enlist \",\")0:`:../data/IMBD.csv\n", "\n", "// convert text data to string\n", "imdbData:update string each comment from imdbData\n", @@ -627,22 +671,22 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We now split the data into training and testing sets to be used with `.automl.run` and `.automl.new`." + "We now split the data into training and testing sets to be used with `.automl.fit` and as an independent testing set for application of the `predict` attribute." ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "xtrain| +(,`comment)!,(\"Three years ago, Rachel(Therese Fretwell) was partyin..\n", - "ytrain| 0 1 0 1 0 1 0 1 1 0 0 0 0 0 1 0 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 1 1..\n", - "xtest | +(,`comment)!,(\"CAROL'S JOURNEY is a pleasure to watch for so many re..\n", - "ytest | 1 0 1 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 1 0 1 1 1 1..\n" + "xtrain| +(,`comment)!,(\"The creativeness of this movie was lost from the begi..\n", + "ytrain| 0 0 1 0 1 1 1 1 1 1 0 0 1 1 1 0 1 0 1 0 0 1 1 1 0 0 0 0 1 0 1 1 1 1 0..\n", + "xtest | +(,`comment)!,(\"I'm watching this on the Star World network overseas ..\n", + "ytest | 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 1 0 0 1 1..\n" ] } ], @@ -663,34 +707,27 @@ "source": [ "The below example demonstrates a binary classification problem. Notice that this time `nlp` is being passed as the feature extraction type.\n", "\n", - "The default configuration for AutoML will again be used, with a random seed included so that results can be replicated." + "A slight modification will be made to the default parameters as this model will be saved under the name `nlpModelNotebook` and the overWriteFiles parameter will also be set to `1b` to allow users to run this notebook multiple times, overwriting the saved model each iteration." ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ - "tab:imdbInputs`xtrain / features\n", - "tgt:imdbInputs`ytrain / targets\n", - "ftype:`nlp / NLP feature extraction\n", - "ptype:`class / classification problem\n", - "dict:enlist[`seed]!enlist 168 / default configuration" + "IMBDfeats :imdbInputs`xtrain / features\n", + "IMBDtarget :imdbInputs`ytrain / targets\n", + "featureType2:`nlp / NLP feature extraction\n", + "problemType2:`class / classification problem\n", + "paramDict2 :`savedModelName`overWriteFiles`seed!(`nlpModelNotebook;1b;100) / define name of model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Outputs" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now run `automl.run` utilizing the NLP functionality." + "We can now run `automl.fit` utilizing the NLP functionality." ] }, { @@ -704,7 +741,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 13, "metadata": { "scrolled": false }, @@ -713,62 +750,107 @@ "name": "stdout", "output_type": "stream", "text": [ + "Executing node: automlConfig\n", + "Executing node: configuration\n", + "Executing node: targetDataConfig\n", + "Executing node: targetData\n", + "Executing node: featureDataConfig\n", + "Executing node: featureData\n", + "Executing node: dataCheck\n", + "\n", + "For full reproducibility between q processes of the NLP word2vec implementation, the PYTHONHASHSEED environment variable must be set upon initialization of q. See https://code.kx.com/q/ml/automl/ug/options/#seed for details.\n", + "\n", + "Executing node: featureDescription\n", "\n", "The following is a breakdown of information for each of the relevant columns in the dataset\n", "\n", + "\n", " | count unique mean std min max type\n", "-------| ----------------------------------\n", - "comment| 1350 1350 :: :: :: :: text\n", + "comment| 900 900 :: :: :: :: text\n", + "\n", + "\n", + "Executing node: dataPreprocessing\n", "\n", "Data preprocessing complete, starting feature creation\n", "\n", - "Feature creation and significance testing complete\n", + "Executing node: featureCreation\n", + "Executing node: labelEncode\n", + "Executing node: featureSignificance\n", + "\n", + "Total number of significant features being passed to the models = 254\n", + "\n", + "Executing node: trainTestSplit\n", + "Executing node: modelGeneration\n", + "Executing node: selectModels\n", + "\n", "Starting initial model selection - allow ample time for large datasets\n", "\n", - "Total features being passed to the models = 88\n", + "Executing node: runModels\n", + "\n", + "Scores for all models using .ml.accuracy\n", + "\n", + "\n", + "RandomForestClassifier | 0.7586657\n", + "GradientBoostingClassifier| 0.7447826\n", + "SVC | 0.736027\n", + "MLPClassifier | 0.7342579\n", + "AdaBoostClassifier | 0.7327286\n", + "LinearSVC | 0.7291904\n", + "KNeighborsClassifier | 0.7186657\n", + "LogisticRegression | 0.709985\n", + "BinaryKeras | 0.6977961\n", + "GaussianNB | 0.6910795\n", "\n", - "Scores for all models, using .ml.accuracy\n", - "RandomForestClassifier | 0.7546512\n", - "GradientBoostingClassifier| 0.7512166\n", - "AdaBoostClassifier | 0.748891\n", - "SVC | 0.7476879\n", - "LinearSVC | 0.746552\n", - "MLPClassifier | 0.7454093\n", - "LogisticRegression | 0.7430972\n", - "binarykeras | 0.7338083\n", - "KNeighborsClassifier | 0.7291504\n", - "GaussianNB | 0.620453\n", "\n", - "Best scoring model = RandomForestClassifier\n", - "Score for validation predictions using best model = 0.6944444\n", "\n", + "Best scoring model = RandomForestClassifier\n", "\n", - "Feature impact calculated for features associated with RandomForestClassifier model\n", - "Plots saved in /outputs/2020.09.22/run_13.10.48.282/images/\n", + "Executing node: optimizeModels\n", "\n", "Continuing to hyperparameter search and final model fitting on testing set\n", "\n", - "Best model fitting now complete - final score on testing set = 0.7407407\n", + "\n", + "Best model fitting now complete - final score on testing set = 0.7611111\n", + "\n", "\n", "Confusion matrix for testing set:\n", "\n", - " | pred_0 pred_1\n", + "\n", + " | true_0 true_1\n", "------| -------------\n", - "true_0| 86 35 \n", - "true_1| 35 114 \n", + "pred_0| 56 27 \n", + "pred_1| 16 81 \n", + "\n", + "\n", + "Executing node: predictParams\n", + "Executing node: preprocParams\n", + "Executing node: pathConstruct\n", + "Executing node: saveGraph\n", + "\n", + "Saving down graphs to /Users/dianeodonoghue/q/automl/outputs/namedModels/nlpModelNotebook/images/\n", + "\n", + "Executing node: saveReport\n", + "\n", + "Saving down procedure report to /Users/dianeodonoghue/q/automl/outputs/namedModels/nlpModelNotebook/report/\n", + "\n", + "Executing node: saveMeta\n", + "\n", + "Saving down model parameters to /Users/dianeodonoghue/q/automl/outputs/namedModels/nlpModelNotebook/config/\n", "\n", - "Saving down procedure report to /outputs/2020.09.22/run_13.10.48.282/report/\n", - "Saving down RandomForestClassifier model to /outputs/2020.09.22/run_13.10.48.282/models/\n", - "Saving down model parameters to /outputs/2020.09.22/run_13.10.48.282/config/\n", + "Executing node: saveModels\n", "\n", - ".automl.run took 00:02:15.215\n" + "Saving down model to /Users/dianeodonoghue/q/automl/outputs/namedModels/nlpModelNotebook/models/\n", + "\n", + "\n", + ".automl.fit took 00:01:17.022\n" ] } ], "source": [ "start:.z.t\n", - "r2:.automl.run[tab;tgt;ftype;ptype;dict]\n", - "-1\"\\n.automl.run took \",string .z.t-start;" + ".automl.fit[IMBDfeats;IMBDtarget;featureType2;problemType2;paramDict2];\n", + "-1\"\\n.automl.fit took \",string .z.t-start;" ] }, { @@ -781,15 +863,15 @@ "\n", "From the above example, we can see that even though one feature was passed to the model, multiple features were created using the `nlp` feature creation methods. If there was any additional non textual data present, the `normal` feature creation procedures would of been applied to them. \n", "\n", - "Looking at the feature impact above, we can see that the features created by the `word2vec` module (`colx`) were deemed to be the most important \n", + "Looking at the feature impact above, we can see that the majority of features created by the `word2vec` module (`colx`) were deemed to be important along with various features created from the NLP spacy library\n", "\n", "#### Confusion matrix\n", "\n", "\n", "\n", - "A confusion matrix is also produced within the pipeline for classification problems. We see that the final `MLPClassifier` model correctly classified 284 data points. \n", + "A confusion matrix is also produced within the pipeline for classification problems. We see that the final `RandomForestClassifier` model correctly classified 137 out of 180 data points. \n", "\n", - "All other outputs from this run have been stored in a directory of format `/outputs/date/run_time/`" + "All other outputs from this run have been stored in a directory of format `/outputs/namedModels/modelName/`" ] }, { @@ -803,7 +885,34 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The best model created within `automl.run` , is applied to the unseen test data to evaluate the models performance" + "To retrieve a model, `.automl.getModel` can be used to retrieve the metadata and associated prediction function to be used on new data from disk, either, by passing the name or the date/time of the desired model." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "modelInfo| `modelLib`modelFunc`startDate`startTime`featureExtractionType`prob..\n", + "predict | {[config;features]\n", + " original_print:utils.printing;\n", + " utils.printi..\n" + ] + } + ], + "source": [ + "show model2:.automl.getModel[enlist[`savedModelName]!enlist \"nlpModelNotebook\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The best model created within `automl.fit` is applied to the unseen test data to evaluate the models performance" ] }, { @@ -815,23 +924,23 @@ "name": "stdout", "output_type": "stream", "text": [ - "Run applied to dataset:\n", + "Model applied to dataset:\n", "\n", - "Run date: 2020.09.22. Run time: 13:10:48.282.\n", + "Model Name: nlpModelNotebook.\n", "\n", "Predictions: \n", - "1 0 0 0 0 0 0 1 0 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 1 0..\n", + "1 1 1 1 0 1 1 0 1 0 0 0 1 0 0 1 1 1 1 1 0 1 1 1 1 1 0 1 0 1 1 0 0 1 1 0 1 0 1..\n", "\n", "Targets:\n", - "1 0 1 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 1 0 1 1 1 1 0 1 1 1..\n" + "1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 1 0 0 1 1 1 1 1 1..\n" ] } ], "source": [ - "-1\"Run applied to dataset:\\n\";\n", - ".util.print_runid . r2;\n", + "-1\"Model applied to dataset:\\n\";\n", + ".util.printSavedModelId model2.modelInfo;\n", "-1\"\\nPredictions: \";\n", - "show imdbPred:.automl.new[imdbInputs`xtest]. r2\n", + "show pred2:model2.predict[imdbInputs`xtest]\n", "-1\"\\nTargets:\";\n", "show imdbInputs`ytest" ] @@ -845,12 +954,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Accuracy on model run using hold-out data: 0.6933333\n" + "Accuracy on model run using hold-out data: 0.78\n" ] } ], "source": [ - "-1\"Accuracy on model run using hold-out data: \",string acc2:.ml.accuracy[imdbInputs`ytest;imdbPred];" + "-1\"Accuracy on model run using hold-out data: \",string accuracy2:.ml.accuracy[imdbInputs`ytest;pred2];" ] }, { @@ -861,7 +970,7 @@ { "data": { "text/plain": [ - "
" + "
" ] }, "metadata": {}, @@ -869,7 +978,7 @@ }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAau0lEQVR4nO3dd7hcVb3/8fc3jRoIoUoowdAEVAQkF0FEOiJFFAWEIFy6sfwUFZUqIIgidgEFlA4C+gMEKSKoKPUiCiIYuMTEJCaBAAlSEvK9f+wdGI7JyaxwzpmZ5P16nnkys9aetb8zZ84na5fZJzITSVJz+rW6AEnqJIamJBUwNCWpgKEpSQUMTUkqYGhKUgFDUwuliPh6RDwVEU++gTHWjYhnerCsloiIkyLie62uY2FhaLa5iJjRcJsdES80PP7oGxj3rojYfz7LHBkRj9XrmhQR10XEEk2MvXNEjGliuXdFxM0R8WwdcHe9kdfUMO46wJHAOpk5fEHHyczHMnPIG62nq4hYPCIyIsZFRL+G9sUiYlpEvNjkOE29z5l5QmaOfiM16zWGZpvLzKXn3IB/ALs1tF3SW+uNiJ2AY4EP1uveEPh5D46/DXAz8CtgLWAF4JPArj0w/JrApMx8ugfG6k3/BrZreLwHMLknVxARA3pyPAGZ6a1DbsCTwPZd2voDxwFPAFOBS4Ahdd9SwOXA08AzwN3AcsCZwCvAi8AM4My5rOtY4PJualkC+BYwDpgEfBdYDFgeeAGYXY89A1h+Ls+/b27r7bLMx4HHgaeAa4CV6/bFgQQOrfunAWfVfe/vsv6zgZ2BMV3GngRsVd/fEngAeK5uP61uXx+Y1fCcNYAb6vfzMeDAhr7T6/f+MmA68Gdg43m8rjn1Hwtc1NB+PfBl4MWGtsOBv9VjjgEOrtvn+j7XdVwKXFE/Z/+67cf18w6sa1+qfvwBYDywXKs/351ya3kB3gp+WHMPzWOA3wGr1r+MPwEuqPs+BVxVB9wA4J0Nvyx3Aft3s67tqWZCxwNbAIO69J9djz0EWBa4CTih7vuPkOry3CF1aGzRzTLvqwPsbfXrOhe4pe6bEzrXAMtQzVSfAbaZ2/qbCM0HgL3r+4OBkfX9rqF5N3AW1X8Om9XhuWXdd3r9fu1A9R/ZWcDt83htc+pfn2pmuTSwEjAReEeX0Ny9fn1R/0xeADbs5nWdDrxUv3/96p/9q6FZL3N1/fNbGfgXsEOrP9uddHPzvPMdDhyTmRMy80XgJOAjERHATGBFYERmzsrMezPz+WYGzcxbgX2AkVSBODUivhYR/epNvoOBT2XmM5n5LNUv5j5N1rx8/e/Ebpb5KHBuZv65fl2fB7aLiFUalvlqZj6Xmf8L/BbYuMn1dzUTWDcils/M6Zl5d9cF6v2kbwe+lJkvZeZ9wE+BAxoWuy0zb8nMV4CLmqhnBtUuig8C+1H9JzSzcYHMvDYz/zcrtwJ3AFvNZ9w7MvOGzJydmS/Mpf8wqjD+NdXWxC3zGU8NDM0OVgfj6sANEfFMfaT3Aaqf6/LAeVS/ZFdFxPiI+GpE9G92/PoXdleqmeHeVAdXDqCa1Q4EHm5Y7y+oZkvNeKr+903dLLMqMLahlmeoNp+HNSwzqeH+v6lmbAviQKoZ7WMRcXe9P3du9UzpEkJje6CeC4FR9e3Crp0RsXtE3BMRT9fv87ZU+3+7M667zsx8imr/9AbAN5uoUQ0MzQ6W1bbWP4FtM3NIw23xzJxaz4iOz8z1ga2pgm/ObLDpy1vVM5abqGZzG1HNEGdRzWDnrHPZzJwzg+x27DoA76eaYc3LBKoDOgBExLJUm+L/bLbuBs8DSzaMNRAY2lDPI5n5EarQ/w5wTUQMmks9K3Y5e2CNBayn0a3AusASmXlvY0dELAX8DDgZWCmrI/m3UW2qw7zf527f/4jYHNi3Hvs7C176osnQ7HxnA6dHxOoAEbFSROxW398+IjaoT2t5jiroXqmf9y/gzfMaNCI+FBF7R8SQqLyL6oDJXZk5Ezgf+HZErFD3rx4ROzSMvVJEdDfTOho4IiI+HRFD6zE2jYiL6/7LgEMjYqOIWBz4GtXm76R5jjhvjwBDI2K7OjBPouGzHxGj6k3zV4BnqUJndpcxxlAd3DmlPjVoE6oZ6hs6gyEzZ1Ptf9xrLt1LUM3oJwOzI2J3YJuG/mbe59eJiCWpdh18FvgYsF5EHLxAxS+iDM3OdwbVbOW2iJgO/AHYpO4bBvx/qqOoD1Ed+b2y7jsLGFWfF3jGXMadBhxFdXT6OaqQPCkzr677P001+7qPKmh+Baxd9z0IXAuMrTffh9JFZt4O7EgVGE9SHfn/HvDLuv964LR6nAnAKrx+/2HTMnMq1UGxS6iOFE+q1zfH+4FH6/fvNODDmTmryxgJfJhqk3YS1dHpz2Xm7xakpi5j/yUzH5lH3UcD11Ht0tiT6mc4x3zf57k4E3gkMy+odzUcAHwjIoa/sVex6IjqsyBJaoYzTUkqYGhKUgFDU5IKGJqSVKBtv8wfA5bIGDS41WWoQ73jLWu0ugR1sLFjn2Tq1Kkxt772Dc1Bg1lsvQ+3ugx1qDvv9vKRWnBbjtxsnn1unktSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUGtLoAzd/ffnkS059/iVdmz2bWK7PZ6qNn8OXD38fBe72LKdNmAHDC967lpt//tcWVqh0dfsjB3HjD9ay40krc/6eHAPjzgw/yiY8fwfMzZrDm8OFccOElLLPMMi2utDMYmh1i58O+zVPPPP+6tu9e/Bu+ddGvW1SROsUBB36MI44azSEHj3q17cjDD+H0M77Bu7d+Dz+94HzOOvPrnHDSyS2ssnO4eS4t5LZ699YMHTr0dW1/f+xRtnr31gBsu/0O/OLnV7eitI5kaHaAzOS6H4zmzks+z8F7bflq+xH7bM09V3yRs0/4KEMGL9HCCtVpNthwI66/7loArrnqZ4wfN67FFXWOPgvNiNg5Ih6NiDERcUxfrXdhsO1BZ/Gu/b7GnqN/wOEfeTdbbjKCH/3sd2yw24mM3Od0Jk19jtM/s1ery1QHOedH53POD7/PuzbflBkzpjNo0KBWl9Qx+iQ0I6I/8H1gF2ADYN+I2KAv1r0wmDjlWQCmTJvBtbf9mXduOJzJT09n9uwkMzn/mjvZbKM1W1ylOsl666/P9TfezB/uuZ8Pf2Rf1nrziFaX1DH6aqa5OTAmM5/IzJeBy4E9+mjdHW3JxQex9JKLvXp/+y3W5+HHJ7DKCq8d6dxj27fz18cntqpEdaDJkycDMHv2bE7/6ikcetgRLa6oc/TV0fNhQONOk/HAyD5ad0dbafnBXPHNQwEY0L8/V9x4H7f84RHOO3kUb1tvNTKTsROf5hOnXNbiStWuRu2/L7+743amTp3KiOGrcdzxJzFjxgzOOfv7AOyx516M+thBLa6yc0Rm9v5KIvYGdsrMQ+rHBwCbZ+Ynuix3GHAYAAOX3nTxDQ/s9dq0cJp27/daXYI62JYjN+P++++LufX11eb5eGD1hserARO6LpSZ52bmZpm5WQzwaLCk9tNXoXkvsE5ErBURg4B9gGv7aN2S1GP6ZJ9mZs6KiNHATUB/4PzMfLgv1i1JPanPvkaZmTcAN/TV+iSpN/iNIEkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSowYF4dEdFUoGbm7J4rR5La2zxDE5gFZDf9Uff379GKJKmNdReaa/VZFZLUIeYZmpk5tmtbvcm+cmZO7NWqJKlNNbXfMiKGRMSlwIvAmLpt94g4pTeLk6R20+zR87OBZ4E1gZfrtj8CH+mNoiSpXXW3T7PRdsCqmTkzIhIgM6dExEq9V5oktZ9mZ5rPAis0NkTEGoD7NiUtUpoNzR8DV0fEe4F+EbEF8FOqzXZJWmQ0u3n+NaqDQN8HBgLnA+cA3+6luiSpLTUVmpmZwLfqmyQtspqdaRIR2wL7AqsCE4DLM/PXvVWYJLWjZs/T/AxwOfA08EvgKeDSiPhsL9YmSW2n2ZnmZ4FtM/OhOQ0RcRFwC3BmbxQmSe2o5NJwY7o8foLuL+ghSQudeYZmRPSbcwNOBM6LiHUiYomIWBc4Fzihj+qUpLbQ7KXhov533y5t+1GdwylJiwQvDSdJBYouDSdJi7qS8zR3B95D9R30OZvrZOaoXqhLktpSs+dpnkD1tcl+wN5U52nuBDzTe6VJUvtp9pSjg4EdMvP/AS/X/+4GDO+twiSpHTUbmkMaTmx/OSIGZuY9VJvrkrTIaHaf5uMRsWFmPgw8BBwZEdOAab1XmiS1n2ZD81hg+fr+F4FLgKWBo3qjKElqV81eGu6Ghvt3A2v3WkWS1MbmGZoR8eZmBsjMJ3quHElqb93NNMdQfWUyulkmgf49WpEktbHuvhFUcgUkSVokGIySVMDQlKQChqYkFTA0JalA01c56mvrjxjGhdec2uoy1KFW+djFrS5BHWzGk0/Ps6+78zTH0cTfAMrMNRasLEnqPN3NNPfvsyokqUN0d57mHX1ZiCR1gpIrt28MvJv/vHL78b1QlyS1pWav3H4YcCewLfAF4K3AZ/HCHZIWMc2ecvR5YOfM/ADwQv3vh4CZvVaZJLWhZkNzpcz8XX1/dkT0y8wbqf7khSQtMprdpzk+IoZn5pPAY8AeETEVeLnXKpOkNtRsaJ4BvAV4EvgKcBUwCPhk75QlSe2p2Su3/6Th/o0RsRwwKDNn9FZhktSOmgrNiOi673MWMKvetzm758uSpPbU7Ob5LOb9lUqv3C5pkdFsaK7V5fGbgGOA63q2HElqb83u0xzbpWlsRBwI3Auc1+NVSVKbeiPX01wGWLGnCpGkTtDsgaCLeP0+zSWBrQEvWihpkdLsPs0xXR4/D5ydmbf2cD2S1NaaDc1fZebdXRsjYvPMvKeHa5KkttXsPs1b5tH+q54qRJI6Qbczzfqk9qjuRtBwHU1gBNX5m5K0yJjf5nnjSe1dA3I24F8+k7RImV9orkU1u7yD6mj5HAlMycwXeqswSWpH3YbmnJPaI2I94JXMfPWiwxExMCIWy8yXerlGSWobzR4IuhnYtEvbpsBNPVuOJLW3ZkPzbUDXU47uAd7es+VIUntrNjSfAVbu0rYy1UnukrTIaDY0rwYujYiNImLJiHgrcCFwZe+VJkntp9nQ/DLwCNUm+XTgLuBR4Eu9VJcktaVmLw33IvDxiBgNrABMzcycyxXdJWmhVhR6WZkCbBQRXwfG905ZktSemg7NiFgxIj4VEf8D/AnYHPhUr1UmSW1oft89HwjsDnwM2InqEnGXAWsCe2fm5N4uUJLayfxmmv8CzqE66PNfmblBZp4MvNzrlUlSG5pfaP4ZGAKMBN5Z/71zSVpkdRuambkN1SXgbgaOBiZFxHXAUsDAXq9OktrMfA8EZebYzDw5M9cBtgMmUl0W7sGIOKO3C5SkdlJ6ytHvM/MwYBXgE8Bbe6UqSWpTC3Ryema+mJmXZeYuPV2QJLUzv9EjSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqMKDVBah7kyaM58Sjj+CpKZOJfv34wD4Hsu9BR/LFTxzE2Cf+DsCM555l6WWW5dJf/r7F1apd9Yvg9pN3YcK0f7PPmbez0RpD+OZBI1lq8QGMm/I8h/7wTqa/MLPVZXYEQ7PNDRgwgE9/6RTW32hjnp8xnVG7b8PIrd7Lad+94NVlzjr1yyw9eJkWVql2d+TO6/PohGcZvMRAAL5zyBYcd+n93Pm3yey/9Qg+uesGnHrVgy2usjO4ed7mVlhpFdbfaGMAllp6MMPXXpcpkya+2p+Z3HrDL9hptw+1qkS1uVWHLsmOG6/KRbePebVt7TcN5s6/TQbgNw9NZLd3rt6q8jqOodlBJowfy6MP/4UNN9701bYH7v0Dyy+/ImusNaKFlamdnbb/phx/2QPMztfaHhn3LO/bZDUA9hy5JsOGLtWi6jpPn4RmRJwfEZMj4qG+WN/C6N/Pz+ALR43iM8d99XWb4jdfezU77v7BFlamdrbTxsOY8tyLPPjk069rH/2jP3LIDuty+8m7sPTiA5g5a3aLKuw8fbVP8yfA94AL+2h9C5VZM2fyhaNGsfPue7Ptzru/1j5rFr+56TouvPb21hWntjZy3RXZZZPV2PHtw1hsYH8GLzGQc47cksN/eCd7fe02AEasMpgdNx7W4ko7R5+EZmb+NiKG98W6FjaZycnHjGb4iHX56CGjX9d3z523s+aIdVj5TX7gNXdfufJPfOXKPwGw1VtWZvT73sLhP7yTFZZZjKnPvUQEfG6Pt3LBr//e4ko7h0fP29yD993FDT+/grXX24D9dt0KgI8ffTxbvndHbr7+ag8AaYF8aIvhHLL9egBcd98/uPi3j7e4os4RmTn/pXpiRdVM8/rM3KibZQ4DDgNYZdXVN73u93/pk9q08Nn+uOtaXYI62Iwbj2fWU0/E3Pra6uh5Zp6bmZtl5mbLDV2+1eVI0n9oq9CUpHbXV6ccXQb8EVgvIsZHxH/3xXolqaf11dHzfftiPZLU29w8l6QChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpQGRmq2uYq4iYAoxtdR1tbAVgaquLUMfy89O9NTNzxbl1tG1oqnsRcV9mbtbqOtSZ/PwsODfPJamAoSlJBQzNznVuqwtQR/Pzs4DcpylJBZxpSlIBQ1OSChiaHSYido6IRyNiTEQc0+p61Fki4vyImBwRD7W6lk5laHaQiOgPfB/YBdgA2DciNmhtVeowPwF2bnURnczQ7CybA2My84nMfBm4HNijxTWpg2Tmb4GnW11HJzM0O8swYFzD4/F1m6Q+Ymh2lphLm+eMSX3I0Ows44HVGx6vBkxoUS3SIsnQ7Cz3AutExFoRMQjYB7i2xTVJixRDs4Nk5ixgNHAT8AhwZWY+3Nqq1Eki4jLgj8B6ETE+Iv671TV1Gr9GKUkFnGlKUgFDU5IKGJqSVMDQlKQChqYkFTA01TIRMTwiMiIG1I9vjIgD+2C9J0bExT085uteS189V33P0FS3IuLJiHghImZExL8i4oKIWLo31pWZu2TmT5usafveqCEitomI8b0xthYOhqaasVtmLg1sArwTOLbrAlHx86SFnh9yNS0z/wncCGwEEBG3R8SpEXEn8G/gzRGxbEScFxETI+KfEXFKfR1QIqJ/RHwjIqZGxBPAro3j1+Md0vD40Ih4JCKmR8RfI2KTiLgIWAO4rp79fr5e9r8i4g8R8UxEPBgR2zSMs1ZE3FGPcwuwwoK8/ojYNSIeiIjnImJcRJw4l8UOjogJ9ev/bMNz+0XEMRHxeEQ8FRFXRsTQBalDrWVoqmkRsTrwPuCBhuYDgMOAwcBY4KfALGBt4B3AjsCcIDwUeH/dvhnwoW7WtTdwIjAKWAbYHXgqMw8A/kE9+83MMyJiGPBL4BRgKHA0cHVErFgPdylwP1VYngws6H7T5+t6hlAF/pERsWeXZd4LrFO/7mMadiN8EtgTeA+wKjCN6oLS6jSZ6c3bPG/Ak8AM4BmqUPwBsETddzvwlYZlVwZemtNft+0L/Ka+fxtwREPfjlSXthvQMN4h9f2bgE91U9P2DY+/AFzUZZmbqMJxDaoQX6qh71Lg4nmMvQ0wvsn35lvAWfX94fVrWb+h/wzgvPr+I8B2DX1vAmYCAxqeO6DVP29v8795tE7N2DMzb51HX+NFkdcEBgITI1699Ge/hmVW7bL82G7WuTrweJP1rQnsHRG7NbQNBH5Tr3NaZj7fZb2Nl9hrSkSMBE6n2j0xCFgM+FmXxbq+vrc21PjziJjd0P8K1X806iBunuuNarziyziqmeYKmTmkvi2TmRvW/RN5fVit0c2444ARTaxzzrIXNaxzSGYulZmn1+tcLiKWanK93bmU6lJ8q2fmssDZ/OeFobu+vjnXOx0H7NKlxsWz2k+sDmJoqsdk5kTgZuDMiFimPvgxIiLeUy9yJfDJiFgtIpYDuvtrmj8Gjo6ITesj82tHxJp137+ANzcsezGwW0TsVB9sWrw+dWi1zBwL3AecFBGDImIrYDfmox6j8RZU+22fzswXI2JzYL+5PPW4iFgyIjYEDgKuqNvPBk6d8xoiYsWI8O87dSBDUz1tFNWm61+pDnZcRbX/DuBHVPsaHwT+B7hmXoNk5s+AU6lmd9OBX1Ad5AE4DTi2PlJ+dGaOo/oDc18CplDN6j7Ha5/v/YCRVH9Q7ATgwvm8hmHAC11uI4CjgK9ExHTgeKr/BLq6AxgD/Br4RmbeXLd/m2qWenP9/LvqmtRhvJ6mJBVwpilJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqcD/AegXkmXdBd24AAAAAElFTkSuQmCC", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAaHElEQVR4nO3deZhcZZn38e/dSRpCEJKQAELYRRiWEdkCihpZBERwQVRQcECJorzgjKiMIgjiGFFcxnFeBhWQXcANFFncQGUXRFGENywxkTUbBEhIQu73j3PCFE13p57Q3VWVfD/Xda6uOs+p59xVXf3r5yx1KjITSVJzulpdgCR1EkNTkgoYmpJUwNCUpAKGpiQVMDQlqYChqRVSRHw5ImZFxIMvoY9XRsTcASyrJSLi5Ij4r1bXsaIwNNtcRDzVMC2JiPkN99/7Evq9KSLet4xljoqIe+t1PRIRV0TEyCb63icipjax3Gsi4pqIeKIOuJteynNq6Hdz4Chg88zceHn7ycx7M3P0S62np4hYNSIyIqZHRFfD/FUiYk5ELGiyn6Ze58w8KTOPfik1638Zmm0uM1dfOgF/B/ZvmHfBYK03IvYGTgAOrNe9NfCjAex/EnANcBWwCTAOOAbYbwC63wh4JDNnD0Bfg+kZYI+G+28FHhvIFUTE8IHsT0BmOnXIBDwI7Nlj3jDgs8D9wEzgAmB03TYKuBiYDcwFbgbGAKcDzwELgKeA03tZ1wnAxf3UMhL4OjAdeAT4JrAKsBYwH1hS9/0UsFYvj7+tt/X2WOajwH3ALOCHwDr1/FWBBI6s2+cAX6vb3tJj/WcA+wBTe/T9CLBbffu1wB3Ak/X8L9bztwQWNzxmQ+DK+vW8F3h/Q9uU+rW/CJgH/AnYro/ntbT+E4DzGub/FPgMsKBh3oeAv9V9TgWOqOf3+jrXdVwIfL9+zPvqed+pH/f+uvZR9f23AzOAMa1+f3fK1PICnAp+Wb2H5vHAb4H16j/Gc4Cz67ZjgcvqgBsO7NTwx3IT8L5+1rUn1UjoRGBXoLtH+xl136OBNYGrgZPqtheFVI/Hjq5DY9d+lnlzHWD/XD+vM4Fr67alofNDYA2qkepcYFJv628iNO8ADqpvvwyYWN/uGZo3A1+j+uewYx2er63bptSv115U/8i+Bvymj+e2tP4tqUaWqwNrAw8Dr+4RmgfUzy/q38l8YOt+ntcU4Nn69euqf/fPh2a9zA/q3986wKPAXq1+b3fS5OZ55/sQcHxmPpSZC4CTgXdHRACLgPHAZpm5ODNvzcynm+k0M38BvAeYSBWIMyPiSxHRVW/yHQEcm5lzM/MJqj/M9zRZ81r1z4f7Wea9wJmZ+af6eX0S2CMi1m1Y5j8y88nMfAC4HtiuyfX3tAh4ZUSslZnzMvPmngvU+0lfBXw6M5/NzNuA7wGHNiz2q8y8NjOfA85rop6nqHZRHAgcQvVPaFHjApl5eWY+kJVfANcBuy2j3+sy88rMXJKZ83tpn0wVxr+k2pq4dhn9qYGh2cHqYNwAuDIi5tZHeu+g+r2uBXyX6o/ssoiYERH/ERHDmu2//oPdj2pkeBDVwZVDqUa1I4C/NKz3x1SjpWbMqn++vJ9l1gOmNdQyl2rzef2GZR5puP0M1YhtebyfakR7b0TcXO/P7a2ex3uE0LQBqOdc4LB6OrdnY0QcEBG3RMTs+nXenWr/b3+m99eYmbOo9k9vBXy1iRrVwNDsYFlta/0D2D0zRzdMq2bmzHpEdGJmbgm8nir4lo4Gm768VT1iuZpqNLcN1QhxMdUIduk618zMpSPIfvuuA/APVCOsvjxEdUAHgIhYk2pT/B/N1t3gaWC1hr5GAGMb6rk7M99NFfr/CfwwIrp7qWd8j7MHNlzOehr9AnglMDIzb21siIhRwKXA54G1szqS/yuqTXXo+3Xu9/WPiJ2Bg+u+/3P5S185GZqd7wxgSkRsABARa0fE/vXtPSNiq/q0liepgu65+nGPApv21WlEvDMiDoqI0VF5DdUBk5sycxFwFvCNiBhXt28QEXs19L12RPQ30joO+HBEfCwixtZ97BAR59ftFwFHRsQ2EbEq8CWqzd9H+uyxb3cDYyNijzowT6bhvR8Rh9Wb5s8BT1CFzpIefUylOrhzan1q0PZUI9SXdAZDZi6h2v/4jl6aR1KN6B8DlkTEAcCkhvZmXucXiIjVqHYdfBz4F2CLiDhiuYpfSRmane80qtHKryJiHnADsH3dtj7wE6qjqHdRHfm9pG77GnBYfV7gab30Owf4CNXR6SepQvLkzPxB3f4xqtHXbVRBcxXwirrtTuByYFq9+T6WHjLzN8CbqALjQaoj//8F/Kxu/ynwxbqfh4B1eeH+w6Zl5kyqg2IXUB0pfqRe31JvAe6pX78vAu/KzMU9+kjgXVSbtI9QHZ3+RGb+dnlq6tH3nzPz7j7qPg64gmqXxtuofodLLfN17sXpwN2ZeXa9q+FQ4CsRsfFLexYrj6jeC5KkZjjSlKQChqYkFTA0JamAoSlJBdr2w/zdq4/OkWP7O/dZ6tsrxo9qdQnqYNOmPcjMmTOjt7a2Dc2RY1/OLp86u9VlqEP95EO7tLoEdbDXTtyxzzY3zyWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpALDW12A+jdiWHD627dmxLBgWFfw2/tmc94tM3jdZmM5dOcJbDBmJMdcehf/7/GnW12qOsDcuXM56kMf5K9/uYuI4Iwzz2KXXXdtdVkdxdBsc4ueSz75k7+yYNEShnUFX33H1tw6bS4Pzn6GU35+L8dM2rTVJaqDHPevx/KmN+3DRd+/jIULF/LMM8+0uqSOY2h2gAWLlgAwvKsabSYwfc6C1haljvPkk0/yu99dz7fPOgeA7u5uuru7W1tUB3KfZgfoCvjvd2/L94/YgTumP8E9jz7V6pLUgR64/37GjRvP5A8czi47vpqjJn+Qp592t06pIQvNiNgnIu6JiKkRcfxQrXdFsCThI9//M+8953a2WHsUG40d2eqS1IEWL17MH++4nSM/dBQ33XYHq40axVdOm9LqsjrOkIRmRAwDvgXsC2wFHBwRWw3FulckTy98jjsfepKdNhzd6lLUgdafMIH1J0xg54kTAXj7ge/kj3fc3uKqOs9QjTR3BqZm5v2ZuRC4GHjrEK27o6256nBGdQ8DoHtYsP2ENZk+Z36Lq1InWnfddZkwYQPuveceAH7zq1+y5T85dik1VAeC1gemN9yfAUwconV3tLGjujluj83oCuiK4Pqps7h52lxes8kYPvL6jVlz5Ag+/5YtuG/mM3zmir+1uly1ua9+/Zscfth7WbhwIRtvuilnfufsVpfUcYYqNKOXefmihSImA5MBVh2z7mDX1BEemPUMH73kzy+af8MDc7jhgTktqEid7FXbbcfvb76t1WV0tKHaPJ8BbNBwfwLwUM+FMvPMzNwxM3fsXt39dpLaz1CF5q3A5hGxSUR0A+8BLh+idUvSgBmSzfPMXBwRRwNXA8OAszLzL0OxbkkaSEP2iaDMvBK4cqjWJ0mDwU8ESVIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUoHhfTVERFOBmplLBq4cSWpvfYYmsBjIftqjbh82oBVJUhvrLzQ3GbIqJKlD9BmamTmt57x6k32dzHx4UKuSpDbV1H7LiBgdERcCC4Cp9bwDIuLUwSxOktpNs0fPzwCeADYCFtbzbgTePRhFSVK76m+fZqM9gPUyc1FEJEBmPh4Raw9eaZLUfpodaT4BjGucEREbAu7blLRSaTY0vwP8ICLeCHRFxK7A96g22yVppdHs5vmXqA4CfQsYAZwF/A/wjUGqS5LaUlOhmZkJfL2eJGml1exIk4jYHTgYWA94CLg4M385WIVJUjtq9jzNfwMuBmYDPwNmARdGxMcHsTZJajvNjjQ/DuyemXctnRER5wHXAqcPRmGS1I5KLg03tcf9++n/gh6StMLpMzQjomvpBHwO+G5EbB4RIyPilcCZwElDVKcktYVmLw0X9c+De8w7hOocTklaKXhpOEkqUHRpOEla2ZWcp3kA8Aaqz6Av3VwnMw8bhLokqS01e57mSVQfm+wCDqI6T3NvYO7glSZJ7afZU46OAPbKzH8FFtY/9wc2HqzCJKkdNRuaoxtObF8YESMy8xaqzXVJWmk0u0/zvojYOjP/AtwFHBURc4A5g1eaJLWfZkPzBGCt+va/AxcAqwMfGYyiJKldNXtpuCsbbt8MvGLQKpKkNtZnaEbEps10kJn3D1w5ktTe+htpTqX6yGT0s0wCwwa0IklqY/19IqjkCkiStFIwGCWpgKEpSQUMTUkqYGhKUoGmr3I01NZbc1VO3nvLVpehDjVmp6NbXYI62LP3/L3Ptv7O05xOE98BlJkbLl9ZktR5+htpvm/IqpCkDtHfeZrXDWUhktQJSq7cvh3wOl585fYTB6EuSWpLzV65fTLwe2B34FPAtsDH8cIdklYyzZ5y9Elgn8x8OzC//vlOYNGgVSZJbajZ0Fw7M39b314SEV2Z+XOqr7yQpJVGs/s0Z0TExpn5IHAv8NaImAksHLTKJKkNNRuapwH/BDwInAJcBnQDxwxOWZLUnpq9cvs5Dbd/HhFjgO7MfGqwCpOkdtRUaEZEz32fi4HF9b7NJQNfliS1p2Y3zxfT90cqvXK7pJVGs6G5SY/7LweOB64Y2HIkqb01u09zWo9Z0yLi/cCtwHcHvCpJalMv5XqaawDjB6oQSeoEzR4IOo8X7tNcDXg9cP5gFCVJ7arZfZpTe9x/GjgjM38xwPVIUltrNjSvysybe86MiJ0z85YBrkmS2laz+zSv7WP+VQNViCR1gn5HmvVJ7VHdjKDhOprAZlTnb0rSSmNZm+eNJ7X3DMglwBcGvCJJamPLCs1NqEaX11EdLV8qgcczc/5gFSZJ7ajf0Fx6UntEbAE8l5nPX3Q4IkZExCqZ+ewg1yhJbaPZA0HXADv0mLcDcPXAliNJ7a3Z0PxnoOcpR7cArxrYciSpvTUbmnOBdXrMW4fqJHdJWmk0G5o/AC6MiG0iYrWI2BY4F7hk8EqTpPbTbGh+BribapN8HnATcA/w6UGqS5LaUrOXhlsAfDQijgbGATMzM3u5orskrdCKQi8rjwPbRMSXgRmDU5YktaemQzMixkfEsRFxO/BHYGfg2EGrTJLa0LI+ez4COAD4F2BvqkvEXQRsBByUmY8NdoGS1E6WNdJ8FPgfqoM+u2TmVpn5eWDhoFcmSW1oWaH5J2A0MBHYqf6+c0laafUbmpk5ieoScNcAxwGPRMQVwChgxKBXJ0ltZpkHgjJzWmZ+PjM3B/YAHqa6LNydEXHaYBcoSe2k9JSj32XmZGBd4P8A2w5KVZLUppbr5PTMXJCZF2XmvgNdkCS1Mz/RI0kFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKjC81QWof6cefzQ3/Ppqxqw1jguuvPH5+ZeeeyaXnf9thg0bzmsm7cXRnzqlhVWq3XV1Bb+/4JM89NgTHHjsGZw35XA233gdAEa/bCRz581nl/dMaXGVncHQbHP7veNgDjr0SE75xIefn/eHm37L9b+8kvOu+B3dq6zC7FmPt7BCdYKjD3kj9zzwKC8btSoAhx5/9vNtU/7t7Tzx1PxWldZx3Dxvc6/e+bWsseaYF8z74YVncejkj9G9yioAjF1rfCtKU4dYf+3R7LPb1pz9oxt6bT9wr+255Ko/DHFVncvQ7EDTH5jKnbfdyAcO3JOjDtmPv/7p9laXpDb25U8cyGe+8WOWLMkXtb12+814dPY87vu7WyvNGpLQjIizIuKxiLhrKNa3onvuucXMe2Iu37nsWo7+1CmccOzhZL74D0La93Xb8Njsedxx9/Re29+1z45cetVtQ1xVZxuqkeY5wD5DtK4V3vh112fS3vsTEWz9qh3oii7mzp7V6rLUhnbdblPe8oZt+dvPTubcKYczaadXctaphwEwbFgXb939VVx2tVsqJYbkQFBmXh8RGw/FulYGr9/zzdx24/VsP3E3/v7AVBYtWsjosWu1uiy1oRO/eTknfvNyAF63w+Z87LA9OOKEcwHYfeIW3Pvgo/zjsbmtLLHjePS8zZ34sQ9w+y2/Z+6cWRyw29Z88Njj2f+d7+ML/340733zrgwf0c1nT/u/RESrS1WHOWjvHTwAtBxiqPaF1SPNn2bmNv0sMxmYDLDuehN2+NF1fx6S2rTieeNBJ7S6BHWwZ++5hCXPPNbrSKStjp5n5pmZuWNm7jh67LhWlyNJL9JWoSlJ7W6oTjm6CLgR2CIiZkTEB4ZivZI00Ibq6PnBQ7EeSRpsbp5LUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklQgMrPVNfQqIh4HprW6jjY2DpjZ6iLUsXz/9G+jzBzfW0Pbhqb6FxG3ZeaOra5Dncn3z/Jz81ySChiaklTA0OxcZ7a6AHU03z/LyX2aklTAkaYkFTA0JamAodlhImKfiLgnIqZGxPGtrkedJSLOiojHIuKuVtfSqQzNDhIRw4BvAfsCWwEHR8RWra1KHeYcYJ9WF9HJDM3OsjMwNTPvz8yFwMXAW1tckzpIZl4PzG51HZ3M0Ows6wPTG+7PqOdJGiKGZmeJXuZ5zpg0hAzNzjID2KDh/gTgoRbVIq2UDM3OciuweURsEhHdwHuAy1tck7RSMTQ7SGYuBo4GrgbuBi7JzL+0tip1koi4CLgR2CIiZkTEB1pdU6fxY5SSVMCRpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1MtExEbR0RGxPD6/s8j4v1DsN7PRcT5A9znC57LUD1WQ8/QVL8i4sGImB8RT0XEoxFxdkSsPhjrysx9M/N7Tda052DUEBGTImLGYPStFYOhqWbsn5mrA9sDOwEn9FwgKr6ftMLzTa6mZeY/gJ8D2wBExG8i4gsR8XvgGWDTiFgzIr4bEQ9HxD8i4tT6OqBExLCI+EpEzIyI+4H9Gvuv+/tgw/0jI+LuiJgXEX+NiO0j4jxgQ+CKevT7yXrZXSLihoiYGxF3RsSkhn42iYjr6n6uBcYtz/OPiP0i4o6IeDIipkfE53pZ7IiIeKh+/h9veGxXRBwfEfdFxKyIuCQixi5PHWotQ1NNi4gNgDcDdzTMPhSYDLwMmAZ8D1gMvAJ4NfAmYGkQHgm8pZ6/I/DOftZ1EPA54DBgDeAAYFZmHgr8nXr0m5mnRcT6wM+AU4GxwHHADyJifN3dhcAfqMLy88Dy7jd9uq5nNFXgHxURb+uxzBuBzevnfXzDboRjgLcBbwDWA+ZQXVBanSYznZz6nIAHgaeAuVSh+N/AyLrtN8ApDcuuAzy7tL2edzDw6/r2r4APN7S9ierSdsMb+vtgfftq4Nh+atqz4f6ngPN6LHM1VThuSBXioxraLgTO76PvScCMJl+brwNfq29vXD+XLRvaTwO+W9++G9ijoe3lwCJgeMNjh7f69+207MmjdWrG2zLzF320NV4UeSNgBPBwxPOX/uxqWGa9HstP62edGwD3NVnfRsBBEbF/w7wRwK/rdc7JzKd7rLfxEntNiYiJwBSq3RPdwCrApT0W6/n8tm2o8UcRsaSh/TmqfzTqIG6e66VqvOLLdKqR5rjMHF1Pa2Tm1nX7w7wwrDbsp9/pwGZNrHPpsuc1rHN0Zo7KzCn1OsdExKgm19ufC6kuxbdBZq4JnMGLLwzd8/ktvd7pdGDfHjWumtV+YnUQQ1MDJjMfBq4BTo+INeqDH5tFxBvqRS4BjomICRExBujv2zS/AxwXETvUR+ZfEREb1W2PAps2LHs+sH9E7F0fbFq1PnVoQmZOA24DTo6I7ojYDdifZaj7aJyCar/t7MxcEBE7A4f08tDPRsRqEbE1cDjw/Xr+GcAXlj6HiBgfEX6/UwcyNDXQDqPadP0r1cGOy6j23wF8m2pf453A7cAP++okMy8FvkA1upsH/JjqIA/AF4ET6iPlx2XmdKovmPs08DjVqO4T/O/7+xBgItUXip0EnLuM57A+ML/HtBnwEeCUiJgHnEj1T6Cn64CpwC+Br2TmNfX8b1CNUq+pH39TXZM6jNfTlKQCjjQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBX4/0RzUdouPvL6AAAAAElFTkSuQmCC", "text/plain": [ "
" ] @@ -879,7 +988,7 @@ } ], "source": [ - "displayCM[value .ml.confmat[imdbInputs`ytest;imdbPred];`0`1;\"Test Set Confusion Matrix\";()];" + ".util.displayCM[value .ml.confmat[imdbInputs`ytest;pred2];`0`1;\"Test Set Confusion Matrix\";()];" ] }, { @@ -900,44 +1009,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In the previous section of the notebook, we showcased how to apply default parameters within the pipeline (excluding the random seed). In this section we will focus on how the final parameter of `.automl.run` can be modified to apply changes to the default behaviour.\n", - "\n", - "There are two options for how this final parameter can be input:\n", - "- **kdb+ dictionary** outlining the changes to default behaviour that are to be made\n", - "- The path to a **flat file** containing more human readable configuration updates." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Advanced parameters" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The following lists the parameters which can be altered by users to modify the functionality of the automl platform. In each case, the parameter name corresponds to the kdb+ dictionary key which would be passed, alongside its user defined value, to the `.automl.run` function in order to update functionality.\n", - "\n", - "Parameters:\n", - "\n", - "```txt\n", - "aggcols Aggregation columns for FRESH\n", - "funcs Functions to be applied for feature extraction\n", - "gs Grid search function and no. of folds/percentage of data in validation set\n", - "hld Size of the testing set on which the final model is tested\n", - "hp Type of hyperparameter search to perform - `grid`random`sobol\n", - "rs Random search function and no. of folds/percentage of data in validation set\n", - "saveopt Saving options outlining what is to be saved to disk from a run\n", - "scf Scoring functions for classification/regression tasks\n", - "seed Random seed to be used\n", - "sigfeats Feature significance procedure to be applied to the data\n", - "sz Size of validation set used.\n", - "trials Number of random/Sobol-random hyperparameters to generate\n", - "tts Train-test split function to be applied\n", - "xv Cross-validation function and # of folds/percentage of data in validation set\n", - "```" + "In the previous section of the notebook, we showcased how to apply default parameters within the pipeline. In this section we will focus on how the final parameter of `.automl.fit` can be modified to apply changes to the default behaviour." ] }, { @@ -951,11 +1023,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this case we use the Telco dataset and alter the parameter dictionary `p` in the following ways:\n", - "1. Added a **random seed**: Here we have altered the ``` `seed``` parameter to be `75`.\n", - "2. Added **feature extraction**: As mentioned above, in the default setting no functions are applied to the table during feature extraction. Below we apply `.automl.prep.i.truncsvd` to the data, this is a truncated singular value decomposition outlined [here](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html) and applied to all combinations of columns of type float.\n", - "3. Changed the size of the **testing** and **holdout** sets to be 10% of the data at each stage.\n", - "4. Changed the **hyperparameter search** type from the default of grid search to random search. Note that Sobol-random search is also available." + "Below we apply `.automl.featureCreation.normal.truncSingleDecomp` to the data, this is a truncated singular value decomposition outlined [here](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html) and applied to all combinations of columns of type float.\n", + "\n", + "A random seed of `100` will also be set." ] }, { @@ -967,16 +1037,17 @@ "name": "stdout", "output_type": "stream", "text": [ - "seed | 75\n", - "funcs| `.automl.prep.i.truncsvd\n", - "sz | 0.1\n", - "hld | 0.1\n", - "hp | `random\n" + "seed | 100\n", + "functions| {[features]\n", + " truncCols:.ml.i.fndcols[features;\"f\"];\n", + " truncCols@:..\n" ] } ], "source": [ - "show p:`seed`funcs`sz`hld`hp!(75;`.automl.prep.i.truncsvd;.1;.1;`random)" + "paramKeys:`seed`functions // parameter names to amend\n", + "paramVals:(100;.automl.featureCreation.normal.truncSingleDecomp) // amended values\n", + "show paramDict3:paramKeys!paramVals" ] }, { @@ -990,9 +1061,18 @@ "name": "stdout", "output_type": "stream", "text": [ + "Executing node: automlConfig\n", + "Executing node: configuration\n", + "Executing node: targetDataConfig\n", + "Executing node: targetData\n", + "Executing node: featureDataConfig\n", + "Executing node: featureData\n", + "Executing node: dataCheck\n", + "Executing node: featureDescription\n", "\n", "The following is a breakdown of information for each of the relevant columns in the dataset\n", "\n", + "\n", " | count unique mean std min max type \n", "------ | --------------------------------------------------------\n", "tenure | 4500 73 32.326 24.55931 0i 72i numeric \n", @@ -1016,62 +1096,95 @@ "PaymentMethod | 4500 4 :: :: :: :: categorical\n", "SeniorCitizen | 4500 2 :: :: :: :: boolean \n", "\n", + "\n", + "Executing node: dataPreprocessing\n", + "\n", "Data preprocessing complete, starting feature creation\n", "\n", - "Feature creation and significance testing complete\n", + "Executing node: featureCreation\n", + "Executing node: labelEncode\n", + "Executing node: featureSignificance\n", + "\n", + "Total number of significant features being passed to the models = 1021\n", + "\n", + "Executing node: trainTestSplit\n", + "Executing node: modelGeneration\n", + "Executing node: selectModels\n", + "\n", "Starting initial model selection - allow ample time for large datasets\n", "\n", - "Total features being passed to the models = 270\n", + "Executing node: runModels\n", "\n", - "Scores for all models, using .ml.accuracy\n", - "RandomForestClassifier | 0.8403292\n", - "GradientBoostingClassifier| 0.8101509\n", - "AdaBoostClassifier | 0.7975309\n", - "LogisticRegression | 0.7964335\n", - "MLPClassifier | 0.7876543\n", - "KNeighborsClassifier | 0.7805213\n", - "binarykeras | 0.7780521\n", - "SVC | 0.7572016\n", - "LinearSVC | 0.7322359\n", - "GaussianNB | 0.7308642\n", + "Scores for all models using .ml.accuracy\n", + "\n", + "\n", + "RandomForestClassifier | 0.8204861\n", + "GradientBoostingClassifier| 0.8065972\n", + "AdaBoostClassifier | 0.7982639\n", + "LogisticRegression | 0.7975694\n", + "KNeighborsClassifier | 0.771875\n", + "MLPClassifier | 0.7670139\n", + "GaussianNB | 0.7392361\n", + "SVC | 0.7309028\n", + "BinaryKeras | 0.6506944\n", + "LinearSVC | 0.6340278\n", "\n", - "Best scoring model = RandomForestClassifier\n", - "Score for validation predictions using best model = 0.8691358\n", "\n", "\n", - "Feature impact calculated for features associated with RandomForestClassifier model\n", - "Plots saved in /outputs/2020.09.22/run_13.13.15.766/images/\n", + "Best scoring model = RandomForestClassifier\n", + "\n", + "Executing node: optimizeModels\n", "\n", "Continuing to hyperparameter search and final model fitting on testing set\n", - "Number of distinct hp sets less than n, returning 222 sets.\n", "\n", - "Best model fitting now complete - final score on testing set = 0.8177778\n", + "\n", + "Best model fitting now complete - final score on testing set = 0.8644444\n", + "\n", "\n", "Confusion matrix for testing set:\n", "\n", - " | pred_0 pred_1\n", + "\n", + " | true_0 true_1\n", "------| -------------\n", - "true_0| 302 32 \n", - "true_1| 50 66 \n", + "pred_0| 596 41 \n", + "pred_1| 81 182 \n", + "\n", + "\n", + "Executing node: predictParams\n", + "Executing node: preprocParams\n", + "Executing node: pathConstruct\n", + "Executing node: saveGraph\n", + "\n", + "Saving down graphs to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.35.10.726/images/\n", "\n", - "Saving down procedure report to /outputs/2020.09.22/run_13.13.15.766/report/\n", - "Saving down RandomForestClassifier model to /outputs/2020.09.22/run_13.13.15.766/models/\n", - "Saving down model parameters to /outputs/2020.09.22/run_13.13.15.766/config/\n", + "Executing node: saveReport\n", "\n", - ".automl.run took 00:12:50.186\n" + "Saving down procedure report to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.35.10.726/report/\n", + "\n", + "Executing node: saveMeta\n", + "\n", + "Saving down model parameters to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.35.10.726/config/\n", + "\n", + "Executing node: saveModels\n", + "\n", + "Saving down model to /Users/dianeodonoghue/q/automl/outputs/dateTimeModels/2020.12.22/run_13.35.10.726/models/\n", + "\n", + "\n", + ".automl.fit took 00:02:01.191\n" ] } ], "source": [ "start:.z.t\n", - "r3:.automl.run[telcoInputs`xtrain;telcoInputs`ytrain;`normal;`class;p]\n", - "-1\"\\n.automl.run took \",string .z.t-start;" + "model3:.automl.fit[telcoInputs`xtrain;telcoInputs`ytrain;`normal;`class;paramDict3]\n", + "-1\"\\n.automl.fit took \",string .z.t-start;" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "#### Feature impact\n", "" ] }, @@ -1079,9 +1192,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can see by looking at the feature impact that many of the most impactful features are now derived from those generated when `.automl.prep.i.truncsvd` was applied during feature extraction, this gives some insight into the potential benefit of this form of feature extraction. \n", + "We can see by looking at the feature impact that a number of the most impactful features are now derived from those generated when `.automl.featureCreation.normal.truncSingleDecomp` was applied during feature extraction, this gives some insight into the potential benefit of this form of feature extraction. \n", "\n", - "While benefiting from increases in accuracy the addition of larger numbers of features can have the effect of slowing training time and scoring time which have have an impact in time critical use-cases.\n", + "While the model may benefit from an increases in accuracy, the addition of larger numbers of features can have the effect of slowing training time and scoring time which have have an impact in time critical use-cases.\n", "\n", "We can now predict on the hold-out dataset in order to compare accuracy results to the default case." ] @@ -1095,12 +1208,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Run applied to dataset:\n", + "Model applied to dataset:\n", "\n", - "Run date: 2020.09.22. Run time: 13:13:15.766.\n", + "Model date: 2020.12.22. Model time: 13:35:10.726.\n", "\n", "Predictions: \n", - "0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0..\n", + "0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0..\n", "\n", "Targets:\n", "0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0..\n" @@ -1108,14 +1221,21 @@ } ], "source": [ - "-1\"Run applied to dataset:\\n\";\n", - ".util.print_runid . r3;\n", + "-1\"Model applied to dataset:\\n\";\n", + ".util.printDateTimeId model3.modelInfo;\n", "-1\"\\nPredictions: \";\n", - "show pred:.automl.new[telcoInputs`xtest]. r3\n", + "show pred3:model3.predict[telcoInputs`xtest]\n", "-1\"\\nTargets:\";\n", "show telcoInputs`ytest" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can see that by adding feature extraction in the normal case, we have improved the accuracy slightly. This is highlighted in the confusion matrix below." + ] + }, { "cell_type": "code", "execution_count": 21, @@ -1125,26 +1245,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Run date: 2020.09.22. Run time: 13:10:17.224.\n", - "Accuracy on default model run using held-out data: 0.816\n", + "Model date: 2020.12.22. Model time: 13:23:40.301.\n", + "Accuracy on default model run using held-out data: 0.866\n", "\n", - "Run date: 2020.09.22. Run time: 13:13:15.766.\n", - "Accuracy on custom model run using held-out data : 0.82\n" + "Model date: 2020.12.22. Model time: 13:35:10.726.\n", + "Accuracy on custom model run using held-out data : 0.878\n" ] } ], "source": [ - ".util.print_runid . r1;\n", - "-1\"Accuracy on default model run using held-out data: \",string[acc1],\"\\n\";\n", - ".util.print_runid . r3;\n", - "-1\"Accuracy on custom model run using held-out data : \",string .ml.accuracy[telcoInputs`ytest;pred];" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that by adding feature extraction in the normal case we have improved our accuracy by ~ 8%. This is highlighted in the confusion matrix below." + ".util.printDateTimeId model1.modelInfo;\n", + "-1\"Accuracy on default model run using held-out data: \",string[accuracy1],\"\\n\";\n", + ".util.printDateTimeId model3.modelInfo;\n", + "-1\"Accuracy on custom model run using held-out data : \",string .ml.accuracy[telcoInputs`ytest;pred3];" ] }, { @@ -1154,16 +1267,7 @@ "outputs": [ { "data": { - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAcB0lEQVR4nO3deZxdZX3H8c8vmYQAAcISVglrIBIUWQNqlX01gBUoqIDFSqu1aAUpKgWRRUSpS62lqJRNQAVUUJawSylhkUWjKA1LJBIgLGEJCSQzv/5xTvAyJpP7hJm595LP+/U6r7n3POc+53fuzHznOcs9E5mJJKk5Q1pdgCR1EkNTkgoYmpJUwNCUpAKGpiQVMDQlqYChqTeliPhqRDwTEY++gT42iYhZ/VhWS0TESRHx7VbX8WZhaLa5iHipYeqJiDkNzz/0BvqdHBEfXswyH4+IB+t1PRERV0bEsk30vWdETG1iuXdGxKSIeL4OuMlvZJsa+h0LfBwYm5nrL2k/mflgZo56o/X0FhEjIiIj4rGIGNIwf5mIeC4i5jbZT1Pvc2aemJmffCM1688MzTaXmSMXTMAfgYkN834wUOuNiD2A44EP1OseD/ykH/vfEZgEXANsAKwGHAXs0w/drwc8kZnP9kNfA+llYJeG5/sBT/XnCiKiqz/7E5CZTh0yAY8Cu/aaNxT4V+Bh4GngB8Coum154BLgWWAWcAewMnAm0A3MBV4CzlzIuo4HLumjlmWBbwCPAU8A/w4sA6wKzAF66r5fAlZdyOvvXth6ey3zj8BDwDPA5cAa9fwRQAIfq9ufA75et72v1/rPAvYEpvbq+wng3fXjdwH3Ai/U879czx8HzG94zRjgqvr9fBA4vKHt9Pq9vxh4Efg18I5FbNeC+o8HLmiY/3PgC8Dchnl/D/y+7nMqcEQ9f6Hvc13HRcAP69d8uJ73vfp1h9e1L18/fz8wHVi51T/fnTK1vACngm/WwkPzOOBWYO36l/Fc4L/rtk8Bl9YB1wVs2/DLMhn4cB/r2pVqJHQCsAMwvFf7WXXfo4CVgGuBE+u2vwipXq8dVYfGDn0ss3cdYG+vt+ts4Lq6bUHoXA6sSDVSnQXsuLD1NxGa9wIH1o9XACbUj3uH5h3A16n+OGxTh+e76rbT6/drN6o/ZF8Hbl7Eti2ofxzVyHIksDowA9iyV2juW29f1N+TOcD4PrbrdOCV+v0bUn/vXwvNepnL6u/fGsCTwG6t/tnupMnd887398Bxmfl4Zs4FTgL+JiICmAeMBjbKzPmZeVdmzm6m08y8HjgYmEAViE9HxFciYki9y3cE8KnMnJWZz1P9Yh7cZM2r1l9n9LHMh4CzM/PX9XYdC+wSEWs2LHNaZr6QmY8AvwTe0eT6e5sHbBIRq2bmi5l5R+8F6uOkWwCfz8xXMvNu4Dzg0IbFbszM6zKzG7igiXpeojpE8QHgg1R/hOY1LpCZV2TmI1m5HrgFePdi+r0lM6/KzJ7MnLOQ9iOpwvgGqr2J6xbTnxoYmh2sDsZ1gasiYlZ9pvdequ/rqsD3qX7JLo2I6RFxWkQMbbb/+hd2H6qR4YFUJ1cOpRrVDgN+27Den1KNlprxTP11rT6WWRuY1lDLLKrd53Ualnmi4fHLVCO2JXE41Yj2wYi4oz6eu7B6ZvYKoWn9UM/5wGH1dH7vxojYNyLujIhn6/d5Z6rjv315rK/GzHyG6vj0ZsC/NVGjGhiaHSyrfa0/ATtn5qiGaURmPl2PiE7IzHHAe6iCb8FosOnbW9UjlmupRnObU40Q51ONYBesc6XMXDCC7LPvOgB/RTXCWpTHqU7oABARK1Htiv+p2bobzAaWa+hrGLBKQz0PZObfUIX+t4DLI2L4QuoZ3evqgTFLWE+j64FNgGUz867GhohYHvgxcDKwelZn8m+k2lWHRb/Pfb7/EbEdcEjd97eWvPSlk6HZ+c4CTo+IdQEiYvWImFg/3jUiNqsva3mBKui669c9CWy4qE4j4oCIODAiRkXlnVQnTCZn5jzgHOCbEbFa3b5uROzW0PfqEdHXSOsY4B8i4tMRsUrdx9YRcWHdfjHwsYjYPCJGAF+h2v19YpE9LtoDwCoRsUsdmCfR8LMfEYfVu+bdwPNUodPTq4+pVCd3TqkvDdqKaoT6hq5gyMwequOPf72Q5mWpRvRPAT0RsS+wY0N7M+/z60TEclSHDo4GPgJsGhFHLFHxSylDs/OdQTVauTEiXgT+F9iqblsH+BnVWdQpVGd+f1S3fR04rL4u8IyF9Psc8Amqs9MvUIXkSZl5Wd3+aarR191UQXMNsHHddj9wBTCt3n1fhV4y82Zgd6rAeJTqzP+3gV/U7T8Hvlz38ziwJq8/fti0zHya6qTYD6jOFD9Rr2+B9wF/qN+/LwMHZeb8Xn0kcBDVLu0TVGenP5uZty5JTb36/k1mPrCIuo8BrqQ6pLE/1fdwgcW+zwtxJvBAZv53fajhUOBrEbH+G9uKpUdUPwuSpGY40pSkAoamJBUwNCWpgKEpSQXa9sP80bVsxvAVWl2GOtTbNl231SWogz32x2k8+8zTsbC29g3N4SuwzKYHtboMdairb/KDLlpye+20wyLb3D2XpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKdLW6AP2lZYZ3cf33P83w4V10DR3KT66/l1POuorTPr0/e79nc16d180j05/myBMv5PmX5jCsayjfPv4QttpsDD3ZwzFnXMatv/q/Vm+G2sSEt2/CyJEjGTJ0KF1dXVx90+1M+c39HPeZT/LK3Ll0dXVx2te+xZZbb9vqUjuCodmGXnl1Pnse+S1mz3mVrq4h3HjOZ5h02++4YfLv+dd/v4Lu7h5OOWo/PnvE7hz/rZ9xxF+/C4BtDzqN0SuP5Kff/gTv/vBXycwWb4naxY+vnMQqq6722vNTT/wcnzn2C+y8257cMOlqTj3x81z68+taWGHncPe8Tc2e8yoAw7qG0tU1lMzkhsm/p7u7B4A7f/MI66wxCoBxG67JTXf+AYCZz73E8y/OYevNxrSmcHWEiODFF18E4MUXXmCNNddqcUWdw9BsU0OGBJMvOY4/3nA6N07+PXdNmfa69sP224Frb/sdAL958E9M3PFtDB06hPXWXpUtN1uXt6y5civKVhuKgEP+eh/23HF7Ljz3ewCcdNrXOOWEz7HN+I04+YTj+NwJJ7e4ys4xaLvnEbEn8E1gKPC9zDx9sNbdiXp6ku0PPp2VRi7LD//tY2y20Vr87qEZABz70T3o7u7hkqvuAuC8n93OuA3W4LYfHMsfZzzL5PsfYX53dyvLVxv56TU3s+Zaa/P0zKc4+P17s/HYTfnFFZfzxdO+yj77vp8rfnIpRx/19/zwp9e0utSOMCgjzYgYCvwHsBewGXBIRGw2GOvudM+/NIdf3v1/7P7O6u360MQJ7P2ezfnIF859bZnu7h6OPfNytj/4dA7657MZtcKyTP3jzBZVrHaz5lprA7Da6NXZ6337cd89d/Hjiy9k74n7AzBx/w9w3z13t7LEjjJYu+fbAVMz8+HMfBW4BNhvkNbdcVZbeSQrjVwWgBHLDGPnCZvyh0efZLd3vpWjP7IrB3z6v5gzd95ryy87YhjLjRgOwM4TxjG/u4ffP/xES2pXe3l59mxeqo9dvjx7NrfceD2bvnU8a6y1Frff9ksA/ueXN7HBhhu3ssyOMli75+sAjzU8nw5MGKR1d5w1V1uR737pUIYOGcKQIcFl193D1bdOYcrPTmSZ4V38/D8/CcCdv3mUo069hNErr8CV3/lHenqSx2fO4qPHn9fiLVC7mDnzST764YMA6O6ez/4fOJiddt2D5ZcfyQmfO5r58+czYsQIzvjGd1pcaeeIwbgsJSIOBPbIzL+rnx8KbJeZ/9RruSOBIwEYNnLrEeMPH/Da9Ob00E3/1uoS1MH22mkH7r/3V7GwtsHaPZ8OrNvw/C3A470XysyzM3ObzNwmupYdpNIkqXmDFZp3AWMjYoOIGA4cDFwxSOuWpH4zKMc0M3N+RHwSuJbqkqNzMvO3g7FuSepPg3adZmZeBVw1WOuTpIHgJ4IkqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpQNeiGiKiqUDNzJ7+K0eS2tsiQxOYD2Qf7VG3D+3XiiSpjfUVmhsMWhWS1CEWGZqZOa33vHqXfY3MnDGgVUlSm2rquGVEjIqIi4C5wNR63r4RccpAFidJ7abZs+dnAc8D6wGv1vNuB/5mIIqSpHbV1zHNRrsAa2fmvIhIgMycGRGrD1xpktR+mh1pPg+s1jgjIsYAHtuUtFRpNjS/B1wWETsBQyJiB+A8qt12SVpqNLt7/hWqk0D/AQwDzgH+C/jmANUlSW2pqdDMzAS+UU+StNRqdqRJROwMHAKsDTwOXJKZNwxUYZLUjpq9TvMzwCXAs8AvgGeAiyLi6AGsTZLaTrMjzaOBnTNzyoIZEXEBcB1w5kAUJkntqOTWcFN7PX+Yvm/oIUlvOosMzYgYsmACvgh8PyLGRsSyEbEJcDZw4iDVKUltodlbw0X99ZBe8z5IdQ2nJC0VvDWcJBUoujWcJC3tSq7T3Bd4L9Vn0BfsrpOZhw1AXZLUlpq9TvNEqo9NDgEOpLpOcw9g1sCVJkntp9lLjo4AdsvMfwZerb9OBNYfqMIkqR01G5qjGi5sfzUihmXmnVS765K01Gj2mOZDETE+M38LTAE+HhHPAc8NXGmS1H6aDc3jgVXrx58DfgCMBD4xEEVJUrtq9tZwVzU8vgPYeMAqkqQ2tsjQjIgNm+kgMx/uv3Ikqb31NdKcSvWRyehjmQSG9mtFktTG+vpEUMkdkCRpqWAwSlIBQ1OSChiaklTA0JSkAk3f5WiwbfnWMdx2x7dbXYY61CNPzW51Cepg3T2L/k8+fV2n+RhN/A+gzByzZGVJUufpa6T54UGrQpI6RF/Xad4ymIVIUicouXP7O4C/4i/v3H7CANQlSW2p2Tu3HwncBuwM/AvwNuBovHGHpKVMs5ccHQvsmZnvB+bUXw8A5g1YZZLUhpoNzdUz89b6cU9EDMnMq6n+5YUkLTWaPaY5PSLWz8xHgQeB/SLiaeDVAatMktpQs6F5BvBW4FHgS8ClwHDgqIEpS5LaU7N3bj+34fHVEbEyMDwzXxqowiSpHTUVmhHR+9jnfGB+fWyzp//LkqT21Ozu+XwW/ZFK79wuaanRbGhu0Ov5WsBxwJX9W44ktbdmj2lO6zVrWkQcDtwFfL/fq5KkNvVG7qe5IjC6vwqRpE7Q7ImgC3j9Mc3lgPcAFw5EUZLUrpo9pjm11/PZwFmZeX0/1yNJba3Z0LwmM+/oPTMitsvMO/u5JklqW80e07xuEfOv6a9CJKkT9DnSrC9qj+phBA330QQ2orp+U5KWGovbPW+8qL13QPYAp/Z7RZLUxhYXmhtQjS5voTpbvkACMzNzzkAVJkntqM/QXHBRe0RsCnRn5ms3HY6IYRGxTGa+MsA1SlLbaPZE0CRg617ztgau7d9yJKm9NRuabwd6X3J0J7BF/5YjSe2t2dCcBazRa94aVBe5S9JSo9nQvAy4KCI2j4jlIuJtwPnAjwauNElqP82G5heAB6h2yV8EJgN/AD4/QHVJUltq9tZwc4F/jIhPAqsBT2dmLuSO7pL0plYUelmZCWweEV8Fpg9MWZLUnpoOzYgYHRGfioh7gPuA7YBPDVhlktSGFvfZ82HAvsBHgD2obhF3MbAecGBmPjXQBUpSO1ncSPNJ4L+oTvpsn5mbZebJwKsDXpkktaHFheavgVHABGDb+v+dS9JSq8/QzMwdqW4BNwk4BngiIq4ElgeGDXh1ktRmFnsiKDOnZebJmTkW2AWYQXVbuPsj4oyBLlCS2knpJUf/k5lHAmsC/wS8bUCqkqQ2tUQXp2fm3My8ODP36u+CJKmd+YkeSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUOzzc2dO5d377Ad2221BVttMZ6TTzoRgM/9y2fZYvNxbLvl2znogPcza9asFleqdnXe2d9m4k7bsO/O23LMJz7CK3Pncs2VlzNxp20Y/5YVmHL/Pa0usaMYmm1umWWW4ZrrbuTOe+7njrvvY9K113DH5Mnssutu/Oq+Kdx1768ZO3YTvvqVL7e6VLWhJ2c8zoXn/Cc/vupWrrjxLrq7u7nqZ5cydtxmfOu7F7HN9u9qdYkdx9BscxHByJEjAZg3bx7z580jIth1t93p6uoCYLsJ2/On6dNbWabaWPf8+cydO4f58+czd84cVl9zLTYaO44NNt6k1aV1JEOzA3R3dzNh63cwZu3V2XnX3dhuwoTXtZ9/7jnssedeLapO7WyNtdbmb//hKHbZ7q28d8uNGLniirzrvbu0uqyONiihGRHnRMRTETFlMNb3ZjN06FDu+NV9TH10OnffdSe/nfLnt/ErXz6VoV1dHPzBD7WwQrWr52c9x43X/oLrJk/h5numMufll7nisktaXVZHG6yR5rnAnoO0rjetUaNG8Z737sikSdcAcOH553HVL37Ouef/gIhocXVqR7ffehPrjFmfVVYdzbBhw9htr3257+7JrS6row1KaGbmL4FnB2NdbzYzZ8587cz4nDlzuPGG69l003FMuvYazvzaV7j0J1ew3HLLtbhKtau11lmX+++5kzlzXiYzmfw/N7Ph2E1bXVZH62p1AerbEzNm8LEjDqe7u5ue7OEDBxzE3vu8j/HjNuaVV17hfXvuBlQng/79O2e1uFq1my222pbd99mfA/Z4F0O7unjr+C046ENHcP3VV3Dq8cfw7LNP8/HDPsC48W/nuxf9rNXldoTIzMFZUcT6wM8zc/M+ljkSOBJg3TFjtn7woWmDUpvefB55anarS1AHO3Cvv2LK/fcs9JhXW509z8yzM3ObzNxm9GqjW12OJP2FtgpNSWp3g3XJ0cXA7cCmETE9Ij46GOuVpP42KCeCMvOQwViPJA00d88lqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSoQmdnqGhYqImYC01pdRxtbDXi61UWoY/nz07f1MnP0whraNjTVt4i4OzO3aXUd6kz+/Cw5d88lqYChKUkFDM3OdXarC1BH8+dnCXlMU5IKONKUpAKGpiQVMDQ7TETsGRF/iIipEXFcq+tRZ4mIcyLiqYiY0upaOpWh2UEiYijwH8BewGbAIRGxWWurUoc5F9iz1UV0MkOzs2wHTM3MhzPzVeASYL8W16QOkpm/BJ5tdR2dzNDsLOsAjzU8n17PkzRIDM3OEguZ5zVj0iAyNDvLdGDdhudvAR5vUS3SUsnQ7Cx3AWMjYoOIGA4cDFzR4pqkpYqh2UEycz7wSeBa4AHgR5n529ZWpU4SERcDtwObRsT0iPhoq2vqNH6MUpIKONKUpAKGpiQVMDQlqYChKUkFDE1JKmBoqmUiYv2IyIjoqp9fHRGHD8J6vxgRF/Zzn6/blsF6rQafoak+RcSjETEnIl6KiCcj4r8jYuRArCsz98rM85qsadeBqCEidoyI6QPRt94cDE01Y2JmjgS2ArYFju+9QFT8edKbnj/kalpm/gm4GtgcICJujohTI+I24GVgw4hYKSK+HxEzIuJPEXFKfR9QImJoRHwtIp6OiIeBfRr7r/v7u4bnH4uIByLixYj4XURsFREXAGOAK+vR77H1sttHxP9GxKyIuD8idmzoZ4OIuKXu5zpgtSXZ/ojYJyLujYgXIuKxiPjiQhY7IiIer7f/6IbXDomI4yLioYh4JiJ+FBGrLEkdai1DU02LiHWBvYF7G2YfChwJrABMA84D5gMbA1sCuwMLgvBjwPvq+dsAB/SxrgOBLwKHASsC+wLPZOahwB+pR7+ZeUZErAP8AjgFWAU4BrgsIkbX3V0E/IoqLE8GlvS46ey6nlFUgf/xiNi/1zI7AWPr7T6u4TDCUcD+wHuBtYHnqG4orU6TmU5Oi5yAR4GXgFlUofgdYNm67WbgSw3LrgG8sqC9nncIcFP9+EbgHxradqe6tV1XQ39/Vz++FvhUHzXt2vD8X4ALei1zLVU4jqEK8eUb2i4CLlxE3zsC05t8b74BfL1+vH69LeMa2s8Avl8/fgDYpaFtLWAe0NXw2q5Wf7+dFj95tk7N2D8zr19EW+NNkdcDhgEzIl679eeQhmXW7rX8tD7WuS7wUJP1rQccGBETG+YNA26q1/lcZs7utd7GW+w1JSImAKdTHZ4YDiwD/LjXYr23720NNf4kInoa2rup/tCog7h7rjeq8Y4vj1GNNFfLzFH1tGJmjq/bZ/D6sBrTR7+PARs1sc4Fy17QsM5Rmbl8Zp5er3PliFi+yfX25SKqW/Gtm5krAWfxlzeG7r19C+53+hiwV68aR2R1nFgdxNBUv8nMGcAk4MyIWLE++bFRRLy3XuRHwFER8ZaIWBno679pfg84JiK2rs/MbxwR69VtTwIbNix7ITAxIvaoTzaNqC8dektmTgPuBk6KiOER8W5gIotR99E4BdVx22czc25EbAd8cCEv/deIWC4ixgN/C/ywnn8WcOqCbYiI0RHh/3fqQIam+tthVLuuv6M62XEp1fE7gO9SHWu8H7gHuHxRnWTmj4FTqUZ3LwI/pTrJA/Bl4Pj6TPkxmfkY1T+Y+zwwk2pU91n+/PP9QWAC1T8UOxE4fzHbsA4wp9e0EfAJ4EsR8SJwAtUfgd5uAaYCNwBfy8xJ9fxvUo1SJ9Wvn1zXpA7j/TQlqYAjTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBf4fkZkaSEs2jW0AAAAASUVORK5CYII=", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFgCAYAAAA/wissAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbj0lEQVR4nO3deZgcVbnH8e+bTMIOQQiEHWQLArITcWPfZFfCakBRuK7gVa6ioAiKIIoCFxURFNlBQWUnoBfkgmwKKhrlBkxICAECCRBkC3nvH1XBZkwmfcLMdHfy/TxPPdNdp/rUWz0zvzm1dE1kJpKk5gxodQGS1EkMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqbmSxHxrYh4OiLGvYk+1omIab1YVktExAkRcVar65hfGJptLiKmN0wzI+LFhucHv4l+74qID85lmY9HxEP1uiZHxDURsUgTfe8SEWObWO6dETE6Ip6tA+6uN7NNDf2uDXwcWDszV5/XfjLzocwc8mbr6S4iFo6IjIgJETGgYf5CETE1Il5qsp+m3ufMPD4zP/Vmata/GJptLjMXnzUBjwJ7NMy7uK/WGxE7A8cBH6jXvT7wi17sfxtgNHAjsAawLHAksFsvdL8aMDkzn+mFvvrSP4HtG57vBTzZmyuIiK7e7E9AZjp1yASMA3boNm8g8GXgEWAKcDEwpG5bDLgMeAaYBtwNLA2cBrwGvARMB06bzbqOAy7roZZFgNOBCcBk4L+BhYBlgBeBmXXf04FlZvP6+2a33m7LfBJ4GHgauApYvp6/MJDA4XX7VOC7ddvu3dZ/NrALMLZb35OBd9eP3wXcDzxXzz+5nj8cmNHwmlWB6+v38yHg0Ia2U+r3/lLgeeBPwMZz2K5Z9R8HXNgw/1rgWOClhnn/Afyt7nMscFg9f7bvc13HJcDl9Ws+WM87t37doXXti9XP9wEmAku3+ue7U6aWF+BU8M2afWgeA9wOrFj/Mp4P/KRuOwr4eR1wXcAWDb8sdwEf7GFdO1CNhL4CbAUM7tZ+dt33EGAp4Cbg+Lrt30Kq22uH1KGxVQ/LvK8OsLfX23UOcHPdNit0rgKWpBqpTgO2md36mwjN+4GR9eMlgBH14+6heTfwXao/DpvX4fmuuu2U+v3akeoP2XeBW+ewbbPqH041slwcWA54HNikW2juWW9f1N+TF4H1e9iuU4CX6/dvQP29fz0062WurL9/ywNPADu2+me7kyZ3zzvffwDHZOakzHwJOAHYPyICeBUYCqyZmTMy897MfKGZTjPzFuAAYARVIE6JiG9GxIB6l+8w4KjMnJaZz1L9Yh7QZM3L1F8f72GZg4FzMvNP9XZ9Htg+IoY1LPONzHwuM/8B/BbYuMn1d/cqsE5ELJOZz2fm3d0XqI+TbgR8KTNfzsz7gJ8CoxoW+01m3pyZrwEXNlHPdKpDFB8ADqL6I/Rq4wKZeXVm/iMrtwC3Ae+eS7+3Zeb1mTkzM1+cTfsRVGH8a6q9iZvn0p8aGJodrA7GVYDrI2Jafab3fqrv6zLAeVS/ZD+PiIkR8Y2IGNhs//Uv7G5UI8ORVCdXRlGNagcBf2lY7y+pRkvNeLr+ukIPy6wIjG+oZRrV7vNKDctMbnj8T6oR27w4lGpE+1BE3F0fz51dPU91C6HxvVDPBcAh9XRB98aI2DMi7omIZ+r3eTuq4789mdBTY2Y+TXV8+m3Ad5qoUQ0MzQ6W1b7WY8B2mTmkYVo4M6fUI6KvZOZw4L1UwTdrNNj07a3qEctNVKO5DahGiDOoRrCz1rlUZs4aQfbYdx2Av6caYc3JJKoTOgBExFJUu+KPNVt3gxeARRv6GgS8paGeMZm5P1XonwlcFRGDZ1PP0G5XD6w6j/U0ugVYB1gkM+9tbIiIxYCfAV8DlsvqTP5vqHbVYc7vc4/vf0RsCRxY933mvJe+YDI0O9/ZwCkRsQpARCwXEXvUj3eIiLfVl7U8RxV0r9WvewJ465w6jYh9I2JkRAyJyjupTpjclZmvAj8GzoiIZev2VSJix4a+l4uInkZaRwMfi4jPRMRb6j42i4iL6vZLgcMjYoOIWBj4JtXu7+Q59jhnY4C3RMT2dWCeQMPPfkQcUu+avwY8SxU6M7v1MZbq5M7X60uDNqUaob6pKxgycybV8cf3z6Z5EaoR/ZPAzIjYE9imob2Z9/kNImJRqkMHnwM+BKwbEYfNU/ELKEOz851KNVr5TUQ8D9wJbFq3rQT8iuos6oNUZ36vqNu+CxxSXxd46mz6nQp8gurs9HNUIXlCZl5Zt3+GavR1H1XQ3AisVbf9EbgaGF/vvr+FbjLzVmAnqsAYR3Xm/yzgurr9WuDkup9JwDDeePywaZk5heqk2MVUZ4on1+ubZXfg7/X7dzKwX2bO6NZHAvtR7dJOpjo7/V+Zefu81NSt7z9n5pg51H00cA3VIY29qb6Hs8z1fZ6N04AxmfmT+lDDKODbEbH6m9uKBUdUPwuSpGY40pSkAoamJBUwNCWpgKEpSQXa9sP80bVIxuAlWl2GOtTbh6/S6hLUwSY8Op6np0yJ2bW1b2gOXoKF1t2v1WWoQ93y29NbXYI62A7vHTHHNnfPJamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoZmG1pocBe3X3g0d19+DL//+bEc97H3vaH9M6O258X7z2KZIYu9Pu/ow3biwV8dzx9/8WV22Gq9/i5Zbe61115j23dtzkH77gXA1GeeYd89d2HLjddj3z13YdrUqS2usHMYmm3o5VdmsMsRZzJi/1MYccDJ7PTOt7HlhqsDsPLyQ9juHcN59PFnXl9++FuHMXLnTdl035PY85Pf54wv7seAAdGi6tWOzvn+mayz7r/+mJ75nVN5z9bbcc8DY3jP1ttx5ndObWF1ncXQbFMvvPgKAIO6BtLVNZDMBODUoz/AsWf88vXnALtv83Z+dtMfeOXVGYyf9DQPT5jCFhus3oqy1YYmPTaRm2+6gQ8eetjr82647hr2P3gUAPsfPIrrr726VeV1HEOzTQ0YENx12TE8+utT+M1df+PeB8ez29YbMunJafz5ocfesOxKQ5di4uR/7V499uRUVlxuqf4uWW3q2C98juO/djIDBvzr1/2pp55g2LAVABg2bAWmTHmyVeV1nH4LzYjYJSL+HhFjI+KY/lpvp5o5M3nHAaew1s7HsfkGq7HB2ivyhY/szIk/uO7fF45/3xVvGIhqATb6husYOnQoG22yWatLmW909cdKImIg8D1gR2AicG9EXJ2Zf+2P9XeyZ6e/yG/v+z923+btrLbSMtxz+RcBWGm5Ifzuki/wnlHf4rEnp7HysKVff81Kyy3N408926qS1UbuvutObrz+Wm4ZfSMvvfQS059/jo9/9BCGDl2eyZMfZ9iwFZg8+XGWXXa5VpfaMfprpLklMDYzH8nMV4DLgL36ad0dZ9mlF2epxRcBYOGFBrHdiHX5498mstr2X2T4bsczfLfjeezJaWx10Dd54unnue7WPzFy500ZPKiL1VZchrVWHcq9D45r7UaoLXz5hJP409/H8Ye/jOVH51/Mu9+7LT849wJ2ed/uXH7xhQBcfvGF7LrbHi2utHP0y0gTWAmY0PB8IjCin9bdcYYtuyQ/OnEUAwcMYMCA4Mqb/8ANtz84x+XHPDKZK0ffz/1XHsuM12bymVOuYOZM9881Z0d+9vN89NADufjCn7Dyyqtw3gWXtbqkjhHZDwe/ImIksHNmfrR+PgrYMjM/3W25I4AjABi0+GYLr39on9em+dOE209vdQnqYDu8dwQP/OH3s71ur792zycCqzQ8XxmY1H2hzDwnMzfPzM2ja5F+Kk2SmtdfoXkvsHZErBERg4EDAC8Mk9Rx+uWYZmbOiIhPATcBA4EfZ+Zf+mPdktSb+utEEJl5PXB9f61PkvqCnwiSpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAl1zaoiIpgI1M2f2XjmS1N7mGJrADCB7aI+6fWCvViRJbayn0Fyj36qQpA4xx9DMzPHd59W77Mtn5uN9WpUktammjltGxJCIuAR4CRhbz9szIr7el8VJUrtp9uz52cCzwGrAK/W83wH790VRktSuejqm2Wh7YMXMfDUiEiAzn4qI5fquNElqP82ONJ8Flm2cERGrAh7blLRAaTY0zwWujIhtgQERsRXwU6rddklaYDS7e/5NqpNA3wMGAT8Gfgic0Ud1SVJbaio0MzOB0+tJkhZYzY40iYjtgAOBFYFJwGWZ+eu+KkyS2lGz12l+FrgMeAa4DngauCQiPteHtUlS22l2pPk5YLvMfHDWjIi4ELgZOK0vCpOkdlRya7ix3Z4/Qs839JCk+c4cQzMiBsyagK8C50XE2hGxSESsA5wDHN9PdUpSW2j21nBRfz2w27yDqK7hlKQFgreGk6QCRbeGk6QFXcl1mnsCW1N9Bn3W7jqZeUgf1CVJbanZ6zSPp/rY5ABgJNV1mjsD0/quNElqP81ecnQYsGNm/ifwSv11D2D1vipMktpRs6E5pOHC9lciYlBm3kO1uy5JC4xmj2k+HBHrZ+ZfgAeBj0fEVGBq35UmSe2n2dA8DlimfvxF4GJgceATfVGUJLWrZm8Nd33D47uBtfqsIklqY3MMzYh4azMdZOYjvVeOJLW3nkaaY6k+Mhk9LJPAwF6tSJLaWE+fCCq5A5IkLRAMRkkqYGhKUgFDU5IKGJqSVKDpuxz1t03WW5U77j6r1WWoQz0wznvJaN69/OrMObb1dJ3mBJr4H0CZueq8lSVJnaenkeYH+60KSeoQPV2neVt/FiJJnaDkzu0bA+/h3+/c/pU+qEuS2lKzd24/ArgD2A74ArAh8Dm8cYekBUyzlxx9HtglM/cBXqy/7gu82meVSVIbajY0l8vM2+vHMyNiQGbeQPUvLyRpgdHsMc2JEbF6Zo4DHgL2iogpwCt9VpkktaFmQ/NUYD1gHHAi8HNgMHBk35QlSe2p2Tu3n9/w+IaIWBoYnJnT+6owSWpHTYVmRHQ/9jkDmFEf25zz540kaT7T7O75DOb8kUrv3C5pgdFsaK7R7fkKwDHANb1bjiS1t2aPaY7vNmt8RBwK3Auc1+tVSVKbejP301wSGNpbhUhSJ2j2RNCFvPGY5qLAe4GL+qIoSWpXzR7THNvt+QvA2Zl5Sy/XI0ltrdnQvDEz7+4+MyK2zMx7erkmSWpbzR7TvHkO82/srUIkqRP0ONKsL2qP6mEEDffRBNakun5TkhYYc9s9b7yovXtAzgRO6vWKJKmNzS0016AaXd5GdbZ8lgSeyswX+6owSWpHPYbmrIvaI2Jd4LXMfP2mwxExKCIWysyX+7hGSWobzZ4IGg1s1m3eZsBNvVuOJLW3ZkPz7UD3S47uATbq3XIkqb01G5rTgOW7zVue6iJ3SVpgNBuaVwKXRMQGEbFoRGwIXABc0XelSVL7aTY0jwXGUO2SPw/cBfwd+FIf1SVJbanZW8O9BHwyIj4FLAtMycyczR3dJWm+VhR6WXkK2CAivgVM7JuyJKk9NR2aETE0Io6KiD8ADwBbAkf1WWWS1Ibm9tnzQcCewIeAnaluEXcpsBowMjOf7OsCJamdzG2k+QTwQ6qTPu/IzLdl5teAV/q8MklqQ3MLzT8BQ4ARwBb1/zuXpAVWj6GZmdtQ3QJuNHA0MDkirgEWAwb1eXWS1GbmeiIoM8dn5tcyc21ge+BxqtvC/TEiTu3rAiWpnZRecvS/mXkEMAz4NLBhn1QlSW1qni5Oz8yXMvPSzNy1twuSpHbmJ3okqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDM02N2HCBHbeYVs23nA9Nt1ofc468wwArvz5z9h0o/VZdPAAfn/ffS2uUu3s8vPP5uD3bcVBu27FZT/5AQD/N+bPHD5yJw7e7Z0cfcQBvPD8cy2usnMYmm2uq6uLU049jQf+PIbb/vcufnj29xjz17+y/vobcNkVV/Hu97y31SWqjT380F+5+oqfct6Vv+aCa27njltvYsK4hzn52KP4+NHHc/F1d7L1jrtz0bn/3epSO4ah2eZWWGEFNtl0UwCWWGIJhg9fj0mTHmP4euuxzrrrtrg6tbtxDz/E+htvwcKLLEpXVxebbPEubht9LeMfGcsmW74TgC3fvQ233nRNiyvtHIZmBxk/bhwPPHA/W2w5otWlqEOsufZ6PHDvnTw79RleevGf/O62m3li8mO8dZ3h3P7rGwD4zQ2/4snJj7W40s7RL6EZET+OiCcj4sH+WN/8aPr06Ry43wf41mmns+SSS7a6HHWI1ddalw8ecRRHfmgf/vOwfVlr+PoMHNjFsSefxZUXncuH9t6Gf74wna5Bg1pdasfo6qf1nA+cBVzQT+ubr7z66qscuN8H2P/Ag9l7n/e3uhx1mD1HjmLPkaMA+MFpJ7LcsBVZfc11OOP8qwB49B9juePW0a0ssaP0y0gzM38LPNMf65rfZCYfO/wjrDt8PY76z8+2uhx1oGeefgqAyZMmcOvoa9lx931fnzdz5kx+8v1vs88BH25liR2lv0aamkd33nEHl1x8IRtssCEjNtsYgBO+/g1efvllPvuZTzPlqad4/1678faNNuaa629qcbVqR1/61CE8O3UqXYO6OPr4b7HkUkO4/PyzufLicwHYZqfd2X3fg1tcZeeIzOyfFUWsDlybmRv0sMwRwBEAq6y66mYPPTy+X2rT/OeBcdNaXYI62If32ZYxf74/ZtfWVmfPM/OczNw8MzcfuuzQVpcjSf+mrUJTktpdf11ydCnwO2DdiJgYER/pj/VKUm/rlxNBmXlgf6xHkvqau+eSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBUwNCWpgKEpSQUMTUkqYGhKUgFDU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSAUNTkgoYmpJUwNCUpAKGpiQVMDQlqYChKUkFDE1JKmBoSlIBQ1OSChiaklTA0JSkAoamJBWIzGx1DbMVEU8B41tdRxtbFpjS6iLUsfz56dlqmTl0dg1tG5rqWUTcl5mbt7oOdSZ/fuadu+eSVMDQlKQChmbnOqfVBaij+fMzjzymKUkFHGlKUgFDU5IKGJodJiJ2iYi/R8TYiDim1fWos0TEjyPiyYh4sNW1dCpDs4NExEDge8CuwNuAAyPiba2tSh3mfGCXVhfRyQzNzrIlMDYzH8nMV4DLgL1aXJM6SGb+Fnim1XV0MkOzs6wETGh4PrGeJ6mfGJqdJWYzz2vGpH5kaHaWicAqDc9XBia1qBZpgWRodpZ7gbUjYo2IGAwcAFzd4pqkBYqh2UEycwbwKeAmYAxwRWb+pbVVqZNExKXA74B1I2JiRHyk1TV1Gj9GKUkFHGlKUgFDU5IKGJqSVMDQlKQChqYkFTA01TIRsXpEZER01c9viIhD+2G9X42Ii3q5zzdsS3+9Vv3P0FSPImJcRLwYEdMj4omI+ElELN4X68rMXTPzp03WtENf1BAR20TExL7oW/MHQ1PN2CMzFwc2BbYAjuu+QFT8edJ8zx9yNS0zHwNuADYAiIhbI+KkiLgD+Cfw1ohYKiLOi4jHI+KxiPh6fR9QImJgRHw7IqZExCPAbo391/19tOH54RExJiKej4i/RsSmEXEhsCpwTT36/Xy97Dsi4s6ImBYRf4yIbRr6WSMibqv7uRlYdl62PyJ2i4j7I+K5iJgQEV+dzWKHRcSkevs/1/DaARFxTEQ8HBFPR8QVEfGWealDrWVoqmkRsQrwPuD+htmjgCOAJYDxwE+BGcBawCbATsCsIDwc2L2evzmwbw/rGgl8FTgEWBLYE3g6M0cBj1KPfjPz1IhYCbgO+DrwFuBo4MqIGFp3dwnwe6qw/Bowr8dNX6jrGUIV+B+PiL27LbMtsHa93cc0HEY4Etgb2BpYEZhKdUNpdZrMdHKa4wSMA6YD06hC8fvAInXbrcCJDcsuD7w8q72edyDwP/Xj3wAfa2jbierWdl0N/X20fnwTcFQPNe3Q8PwLwIXdlrmJKhxXpQrxxRraLgEumkPf2wATm3xvTge+Wz9evd6W4Q3tpwLn1Y/HANs3tK0AvAp0Nby2q9Xfb6e5T56tUzP2zsxb5tDWeFPk1YBBwOMRr9/6c0DDMit2W358D+tcBXi4yfpWA0ZGxB4N8wYB/1Ovc2pmvtBtvY232GtKRIwATqE6PDEYWAj4WbfFum/fhg01/iIiZja0v0b1h0YdxN1zvVmNd3yZQDXSXDYzh9TTkpm5ft3+OG8Mq1V76HcCsGYT65y17IUN6xySmYtl5in1OpeOiMWaXG9PLqG6Fd8qmbkUcDb/fmPo7ts3636nE4Bdu9W4cFbHidVBDE31msx8HBgNnBYRS9YnP9aMiK3rRa4AjoyIlSNiaaCn/6Z5LnB0RGxWn5lfKyJWq9ueAN7asOxFwB4RsXN9smnh+tKhlTNzPHAfcEJEDI6IdwN7MBd1H41TUB23fSYzX4qILYGDZvPSL0fEohGxPvBh4PJ6/tnASbO2ISKGRoT/36kDGZrqbYdQ7br+lepkx8+pjt8B/IjqWOMfgT8AV82pk8z8GXAS1ejueeCXVCd5AE4GjqvPlB+dmROo/sHcl4CnqEZ1/8W/fr4PAkZQ/UOx44EL5rINKwEvdpvWBD4BnBgRzwNfofoj0N1twFjg18C3M3N0Pf8MqlHq6Pr1d9U1qcN4P01JKuBIU5IKGJqSVMDQlKQChqYkFTA0JamAoSlJBQxNSSpgaEpSgf8HtynnHoT7aZoAAAAASUVORK5CYII=", "text/plain": [ "
" ] @@ -1173,7 +1277,7 @@ } ], "source": [ - "displayCM[value .ml.confmat[telcoInputs`ytest;pred];`0`1;\"Test Set Confusion Matrix\";()];" + ".util.displayCM[value .ml.confmat[telcoInputs`ytest;pred3];`0`1;\"Test Set Confusion Matrix\";()];" ] }, { @@ -1187,40 +1291,68 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this example we again use the Telco dataset and highlight how to change the save options, contained under `saveopt` within the parameter dictionary.\n", + "In this example we again use the Telco dataset and highlight how to change the save options contained under `saveOption` within the parameter dictionary.\n", "\n", - "In the default case, not modified in the examples above, the system will save all outputs to disk (reports, images, config file and models). This can be altered by the user to reduce the number of outputs saved to disk, where:\n", + "In the default case, not modified in the examples above, the system will save all outputs to disk (reports, images, config file and models). Below we will set the `saveOption` to be `0`, which means that the results will be displayed to console but nothing is persisted to disk.\n", "\n", - "- `0` = Nothing is saved the models will run and display results to console but nothing persisted\n", - "- `1` = Save the model and configuration file only, will not generate a report for the user or any images\n", - "- `2` = Save all possible outputs to disk for the user including reports, images, config and models\n", + "Other alterations made to the `paramDict` in the below model were\n", + "1. Added a **random seed**: Here we have altered the ``` `seed``` parameter to be `175`.\n", + "2. Changed the size of the **holdout** sets to be 30% of the data at each stage.\n", + "3. Changed the **hyperparameter search** type from the default of grid search to random search and set the number of repetitions to `2`. Note that Sobol-random search is also available.\n", "\n", - "We demonstrate the case for `0` below for a subset of 1000 data points." + "A smaller subset of 1000 datapoints will be used" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "saveOption | 0\n", + "seed | 175\n", + "holdoutSize | 0.3\n", + "hyperparameterSearchType| `random\n", + "randomSearchArgument | 2\n" + ] + } + ], "source": [ "\\S 42\n", - "feat:1000?telcoFeat\n", - "targ:1000?telcoTarg" + "features:1000?telcoFeat\n", + "target :1000?telcoTarg\n", + "\n", + "paramKeys:`saveOption`seed`holdoutSize`hyperparameterSearchType`randomSearchArgument // parameter names to amend\n", + "paramVals:(0;175;.3;`random;2) // amended values\n", + "show paramDict:paramKeys!paramVals" ] }, { "cell_type": "code", "execution_count": 24, - "metadata": {}, + "metadata": { + "scrolled": false + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ + "Executing node: automlConfig\n", + "Executing node: configuration\n", + "Executing node: targetDataConfig\n", + "Executing node: targetData\n", + "Executing node: featureDataConfig\n", + "Executing node: featureData\n", + "Executing node: dataCheck\n", + "Executing node: featureDescription\n", "\n", "The following is a breakdown of information for each of the relevant columns in the dataset\n", "\n", + "\n", " | count unique mean std min max type \n", "------ | --------------------------------------------------------\n", "tenure | 1000 73 33.551 25.0546 0i 72i numeric \n", @@ -1244,43 +1376,72 @@ "PaymentMethod | 1000 4 :: :: :: :: categorical\n", "SeniorCitizen | 1000 2 :: :: :: :: boolean \n", "\n", + "\n", + "Executing node: dataPreprocessing\n", + "\n", "Data preprocessing complete, starting feature creation\n", "\n", - "Feature creation and significance testing complete\n", + "Executing node: featureCreation\n", + "Executing node: labelEncode\n", + "Executing node: featureSignificance\n", + "\n", + "Total number of significant features being passed to the models = 12\n", + "\n", + "Executing node: trainTestSplit\n", + "Executing node: modelGeneration\n", + "Executing node: selectModels\n", + "\n", "Starting initial model selection - allow ample time for large datasets\n", "\n", - "Total features being passed to the models = 12\n", + "Executing node: runModels\n", + "\n", + "Scores for all models using .ml.accuracy\n", + "\n", + "\n", + "MLPClassifier | 0.7482143\n", + "LogisticRegression | 0.7446429\n", + "AdaBoostClassifier | 0.7428571\n", + "SVC | 0.7375\n", + "LinearSVC | 0.7357143\n", + "GaussianNB | 0.725\n", + "GradientBoostingClassifier| 0.7125\n", + "BinaryKeras | 0.7017857\n", + "KNeighborsClassifier | 0.6857143\n", + "RandomForestClassifier | 0.6839286\n", "\n", - "Scores for all models, using .ml.accuracy\n", - "AdaBoostClassifier | 0.775\n", - "LogisticRegression | 0.775\n", - "LinearSVC | 0.775\n", - "SVC | 0.771875\n", - "MLPClassifier | 0.765625\n", - "GradientBoostingClassifier| 0.759375\n", - "KNeighborsClassifier | 0.7546875\n", - "RandomForestClassifier | 0.74375\n", - "GaussianNB | 0.7328125\n", - "binarykeras | 0.684375\n", "\n", - "Best scoring model = AdaBoostClassifier\n", - "Score for validation predictions using best model = 0.71875\n", + "\n", + "Best scoring model = MLPClassifier\n", + "\n", + "Executing node: optimizeModels\n", "\n", "Continuing to hyperparameter search and final model fitting on testing set\n", "\n", - "Best model fitting now complete - final score on testing set = 0.71\n", + "\n", + "Best model fitting now complete - final score on testing set = 0.805\n", + "\n", "\n", "Confusion matrix for testing set:\n", "\n", - " | pred_0 pred_1\n", + "\n", + " | true_0 true_1\n", "------| -------------\n", - "true_0| 142 0 \n", - "true_1| 58 0 \n" + "pred_0| 161 0 \n", + "pred_1| 39 0 \n", + "\n", + "\n", + "Executing node: predictParams\n", + "Executing node: preprocParams\n", + "Executing node: pathConstruct\n", + "Executing node: saveGraph\n", + "Executing node: saveReport\n", + "Executing node: saveMeta\n", + "Executing node: saveModels\n" ] } ], "source": [ - ".automl.run[feat;targ;`normal;`class;enlist[`saveopt]!enlist 0];" + ".automl.fit[features;target;`normal;`class;paramDict];" ] }, { @@ -1301,11 +1462,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this example, the IMDB dataset is used with the following changes made to the input dictionary `p`:\n", + "In this example, the IMDB dataset is used with the following changes made to the input dictionary `paramDict`:\n", "\n", "1. **Word2vec transformation** changed from default `continuous bag of words` method to `skip-gram`.\n", - "2. **Significant feature function** changed to use the Benjamini-Hochberg-Yekutieli procedure.\n", - "3. **Random seed** set as `100`." + "2. **Significant feature function** changed to use the percentile based procedure.\n", + "3. **Random seed** set as `275`.\n", + "\n", + "In this example, printing to screen will also be suppressed and redirected to a log file called `LogFile`" ] }, { @@ -1317,20 +1480,27 @@ "name": "stdout", "output_type": "stream", "text": [ - "sigfeats| `.automl.newsigfeat\n", - "w2v | 1\n", - "seed | 150\n" + "significantFeatures| `.automl.newSigFeat\n", + "w2v | 1\n", + "seed | 275\n", + "loggingFile | \"logFile\"\n" ] } ], "source": [ + ".automl.updatePrinting[] // Disable printing to screen \n", + ".automl.updateLogging[] // Redirect printing to log file\n", + "\n", + "\n", "// new significant feature function \n", - ".automl.newsigfeat:{[x;y]\n", - " .ml.fresh.significantfeatures[x;y;.ml.fresh.benjhoch 0.05]\n", + ".automl.newSigFeat:{[x;y]\n", + " .ml.fresh.significantfeatures[x;y;.ml.fresh.percentile 0.10]\n", " }\n", "\n", "// new parameter dictionary\n", - "show p:`sigfeats`w2v`seed!(`.automl.newsigfeat;1;150)" + "paramKeys:`significantFeatures`w2v`seed`loggingFile // parameter names to amend\n", + "paramVals:(`.automl.newSigFeat;1;275;\"logFile\") // amended values\n", + "show paramDict4:paramKeys!paramVals" ] }, { @@ -1344,66 +1514,43 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "The following is a breakdown of information for each of the relevant columns in the dataset\n", - "\n", - " | count unique mean std min max type\n", - "-------| ----------------------------------\n", - "comment| 1350 1350 :: :: :: :: text\n", - "\n", - "Data preprocessing complete, starting feature creation\n", - "\n", - "Feature creation and significance testing complete\n", - "Starting initial model selection - allow ample time for large datasets\n", - "\n", - "Total features being passed to the models = 230\n", - "\n", - "Scores for all models, using .ml.accuracy\n", - "MLPClassifier | 0.8090335\n", - "LinearSVC | 0.7940046\n", - "RandomForestClassifier | 0.7881839\n", - "GradientBoostingClassifier| 0.7824304\n", - "AdaBoostClassifier | 0.773202\n", - "LogisticRegression | 0.743077\n", - "SVC | 0.7350181\n", - "KNeighborsClassifier | 0.7245665\n", - "binarykeras | 0.7130125\n", - "GaussianNB | 0.6747883\n", - "\n", - "Best scoring model = MLPClassifier\n", - "Score for validation predictions using best model = 0.8518519\n", - "\n", - "\n", - "Feature impact calculated for features associated with MLPClassifier model\n", - "Plots saved in /outputs/2020.09.22/run_13.26.49.103/images/\n", - "\n", - "Continuing to hyperparameter search and final model fitting on testing set\n", - "\n", - "Best model fitting now complete - final score on testing set = 0.7814815\n", - "\n", - "Confusion matrix for testing set:\n", - "\n", - " | pred_0 pred_1\n", - "------| -------------\n", - "true_0| 95 32 \n", - "true_1| 27 116 \n", - "\n", - "Saving down procedure report to /outputs/2020.09.22/run_13.26.49.103/report/\n", - "Saving down MLPClassifier model to /outputs/2020.09.22/run_13.26.49.103/models/\n", - "Saving down model parameters to /outputs/2020.09.22/run_13.26.49.103/config/\n" + "Executing node: automlConfig\n", + "Executing node: configuration\n", + "Executing node: targetDataConfig\n", + "Executing node: targetData\n", + "Executing node: featureDataConfig\n", + "Executing node: featureData\n", + "Executing node: dataCheck\n", + "Executing node: featureDescription\n", + "Executing node: dataPreprocessing\n", + "Executing node: featureCreation\n", + "Executing node: labelEncode\n", + "Executing node: featureSignificance\n", + "Executing node: trainTestSplit\n", + "Executing node: modelGeneration\n", + "Executing node: selectModels\n", + "Executing node: runModels\n", + "Executing node: optimizeModels\n", + "Executing node: predictParams\n", + "Executing node: preprocParams\n", + "Executing node: pathConstruct\n", + "Executing node: saveGraph\n", + "Executing node: saveReport\n", + "Executing node: saveMeta\n", + "Executing node: saveModels\n" ] } ], "source": [ "// run automl with new parameters\n", - "r4:.automl.run[imdbInputs`xtrain;imdbInputs`ytrain;`nlp;`class;p]" + "model4:.automl.fit[imdbInputs`xtrain;imdbInputs`ytrain;`nlp;`class;paramDict4]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "We can see that the features deemed important compared with the initial run have varied. In this iteration, some of the features that were created from the NLP parser such as `CARDINAL` and compound were identified as significant in predicting the target value\n", + "We can see that the features deemed important compared with the initial run have varied. In this iteration, very few of the features created from the NLP spacy library were deemed to be significant when predicting the target value\n", "\n", "" ] @@ -1417,162 +1564,53 @@ "name": "stdout", "output_type": "stream", "text": [ - "Run date: 2020.09.22. Run time: 13:26:49.103.\n", + "Model date: 2020.12.22. Model time: 13:38:49.990.\n", "\n", "Predictions: \n", - "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0..\n", + "1 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 1 0 1 0 1 1 1 0 1 0 1 1 1 0 1 1 0 1 1 1..\n", "\n", "Targets:\n", - "1 0 1 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 1 0 1 1 1 1 0 1 1 1..\n" + "1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 1 0 0 1 1 1 1 1 1..\n" ] } ], "source": [ - ".util.print_runid . r4;\n", + ".util.printDateTimeId model4.modelInfo;\n", "-1\"\\nPredictions: \";\n", - "show pred:.automl.new[imdbInputs`xtest]. r4\n", + "show pred4:model4.predict[imdbInputs`xtest]\n", "-1\"\\nTargets:\";\n", "show imdbInputs`ytest" ] }, - { - "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Run date: 2020.09.22. Run time: 13:10:48.282.\n", - "Accuracy on default model run using held-out data: 0.6933333\n", - "\n", - "Run date: 2020.09.22. Run time: 13:10:48.282.\n", - "Accuracy on custom model run using held-out data : 0.6933333\n" - ] - } - ], - "source": [ - ".util.print_runid . r2;\n", - "-1\"Accuracy on default model run using held-out data: \",string[acc2],\"\\n\";\n", - ".util.print_runid . r2;\n", - "-1\"Accuracy on custom model run using held-out data : \",string .ml.accuracy[imdbInputs`ytest;imdbPred];" - ] - }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Example 4" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We have shown in the previous examples that the pipeline can be altered by passing in a dictionary of parameters as the last argument in `.automl.run`. As mentioned previously, we can also pass the path to a flat file.\n", - "\n", - "Default flat files are saved to `automl/code/models/` where users can change parameters within each file. These are generated by a user using the function `.automl.savedefault` as follows:" + "Below we can see how changing the `w2v` implementation decreased the accuracy of the model compared with the initial run" ] }, { "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [], - "source": [ - ".automl.savedefault[\"new_normal_defaults.txt\";`normal]" - ] - }, - { - "cell_type": "markdown", + "execution_count": 28, "metadata": {}, - "source": [ - "We can then run the pipeline using this new file as our final argument:" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "metadata": { - "scrolled": false - }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ + "Model date: 2020.12.22. Model time: 13:24:18.472.\n", + "Accuracy on default model run using held-out data: 0.78\n", "\n", - "The following is a breakdown of information for each of the relevant columns in the dataset\n", - "\n", - " | count unique mean std min max type \n", - "------ | --------------------------------------------------------\n", - "tenure | 5000 73 32.2504 24.55779 0i 72i numeric \n", - "MonthlyCharges | 5000 1285 64.82904 30.46505 18.55 118.75 numeric \n", - "TotalCharges | 5000 3413 2281.232 2279.37 18.85 8672.45 numeric \n", - "customerID | 5000 3566 :: :: :: :: categorical\n", - "gender | 5000 2 :: :: :: :: categorical\n", - "Partner | 5000 2 :: :: :: :: categorical\n", - "Dependents | 5000 2 :: :: :: :: categorical\n", - "PhoneService | 5000 2 :: :: :: :: categorical\n", - "MultipleLines | 5000 3 :: :: :: :: categorical\n", - "InternetService | 5000 3 :: :: :: :: categorical\n", - "OnlineSecurity | 5000 3 :: :: :: :: categorical\n", - "OnlineBackup | 5000 3 :: :: :: :: categorical\n", - "DeviceProtection| 5000 3 :: :: :: :: categorical\n", - "TechSupport | 5000 3 :: :: :: :: categorical\n", - "StreamingTV | 5000 3 :: :: :: :: categorical\n", - "StreamingMovies | 5000 3 :: :: :: :: categorical\n", - "Contract | 5000 3 :: :: :: :: categorical\n", - "PaperlessBilling| 5000 2 :: :: :: :: categorical\n", - "PaymentMethod | 5000 4 :: :: :: :: categorical\n", - "SeniorCitizen | 5000 2 :: :: :: :: boolean \n", - "\n", - "Data preprocessing complete, starting feature creation\n", - "\n", - "Feature creation and significance testing complete\n", - "Starting initial model selection - allow ample time for large datasets\n", - "\n", - "Total features being passed to the models = 7\n", - "\n", - "Scores for all models, using .ml.accuracy\n", - "AdaBoostClassifier | 0.8009375\n", - "MLPClassifier | 0.8003125\n", - "LogisticRegression | 0.796875\n", - "GradientBoostingClassifier| 0.795625\n", - "RandomForestClassifier | 0.790625\n", - "KNeighborsClassifier | 0.7903125\n", - "binarykeras | 0.78125\n", - "SVC | 0.75875\n", - "GaussianNB | 0.7475\n", - "LinearSVC | 0.7353125\n", - "\n", - "Best scoring model = AdaBoostClassifier\n", - "Score for validation predictions using best model = 0.8025\n", - "\n", - "\n", - "Feature impact calculated for features associated with AdaBoostClassifier model\n", - "Plots saved in /outputs/2020.09.22/run_13.33.53.803/images/\n", - "\n", - "Continuing to hyperparameter search and final model fitting on testing set\n", - "\n", - "Best model fitting now complete - final score on testing set = 0.733\n", - "\n", - "Confusion matrix for testing set:\n", - "\n", - " | pred_0 pred_1\n", - "------| -------------\n", - "true_0| 733 0 \n", - "true_1| 267 0 \n", - "\n", - "Saving down procedure report to /outputs/2020.09.22/run_13.33.53.803/report/\n", - "Saving down AdaBoostClassifier model to /outputs/2020.09.22/run_13.33.53.803/models/\n", - "Saving down model parameters to /outputs/2020.09.22/run_13.33.53.803/config/\n" + "Model date: 2020.12.22. Model time: 13:38:49.990.\n", + "Accuracy on custom model run using held-out data : 0.78\n" ] } ], "source": [ - ".automl.run[telcoFeat;telcoTarg;`normal;`class;\"new_normal_defaults.txt\"];" + ".util.printDateTimeId model2.modelInfo;\n", + "-1\"Accuracy on default model run using held-out data: \",string[accuracy2],\"\\n\";\n", + ".util.printDateTimeId model4.modelInfo;\n", + "-1\"Accuracy on custom model run using held-out data : \",string .ml.accuracy[imdbInputs`ytest;pred4];" ] }, { diff --git a/notebooks/images/run1conf.png b/notebooks/images/run1conf.png index 6029608..dd3ce1e 100644 Binary files a/notebooks/images/run1conf.png and b/notebooks/images/run1conf.png differ diff --git a/notebooks/images/run1impact.png b/notebooks/images/run1impact.png index 28476ae..eb111df 100644 Binary files a/notebooks/images/run1impact.png and b/notebooks/images/run1impact.png differ diff --git a/notebooks/images/run2conf.png b/notebooks/images/run2conf.png index 5a90067..bc48251 100644 Binary files a/notebooks/images/run2conf.png and b/notebooks/images/run2conf.png differ diff --git a/notebooks/images/run2impact.png b/notebooks/images/run2impact.png index 4afaadf..cf1c132 100644 Binary files a/notebooks/images/run2impact.png and b/notebooks/images/run2impact.png differ diff --git a/notebooks/images/run3impact.png b/notebooks/images/run3impact.png index 861b37c..7d770e5 100644 Binary files a/notebooks/images/run3impact.png and b/notebooks/images/run3impact.png differ diff --git a/notebooks/images/run4impact.png b/notebooks/images/run4impact.png index 8ff941f..25fe9a2 100644 Binary files a/notebooks/images/run4impact.png and b/notebooks/images/run4impact.png differ diff --git a/utils/util.q b/utils/util.q index 74cf50e..4e849b6 100644 --- a/utils/util.q +++ b/utils/util.q @@ -8,7 +8,8 @@ npa:.p.import[`numpy]`:array round:{y*"j"$x%y} imax:{x?max x} mattab:{flip value flip x} -print_runid:{-1"Run date: ",string[x],". Run time: ",string[y],"."} +printDateTimeId:{-1"Model date: ",string[x`startDate],". Model time: ",string[x`startTime],"."} +printSavedModelId:{-1"Model Name: ",string[x`savedModelName],"."} // @kind function // @category misc