notebooks/03_machine_learning/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Tutorial 3"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "An introduction to machine learning using MLJ and the Titanic\n",
    "dataset. Explains how to train a simple decision tree model and\n",
    "evaluate it's performance on a holdout set."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "MLJ is a *multi-paradigm* machine learning toolbox (i.e., not just\n",
    "deep-learning)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "For other MLJ learning resources see the [Learning\n",
    "MLJ](https://JuliaAI.github.io/MLJ.jl/dev/learning_mlj/)\n",
    "section of the\n",
    "[manual](https://JuliaAI.github.io/MLJ.jl/dev/)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Activate package environment"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Activating project at `~/GoogleDrive/Julia/HelloJulia`\n"
     ]
    }
   ],
   "cell_type": "code",
   "source": [
    "using Pkg\n",
    "Pkg.activate(joinpath(@__DIR__, \"..\", \"..\"))\n",
    "Pkg.instantiate()"
   ],
   "metadata": {},
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Establishing correct data representation"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "using MLJ\n",
    "import DataFrames"
   ],
   "metadata": {},
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "A [\"scientific\n",
    "type\"](https://juliaai.github.io/ScientificTypes.jl/dev/) or\n",
    "*scitype* indicates how MLJ will *interpret* data (as opposed to how\n",
    "it is represented on your machine). For example, while we have"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "Float64"
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "cell_type": "code",
   "source": [
    "typeof(3.14)"
   ],
   "metadata": {},
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "source": [
    "we have"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "ScientificTypesBase.Continuous"
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "cell_type": "code",
   "source": [
    "scitype(3.14)"
   ],
   "metadata": {},
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "source": [
    "and also"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "ScientificTypesBase.Continuous"
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "cell_type": "code",
   "source": [
    "scitype(3.143f0)"
   ],
   "metadata": {},
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "source": [
    "In MLJ, model data requirements are articulated using scitypes."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here are common \"scalar\" scitypes:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "HTML{String}(\"<div style=\\\"text-align: left\\\";>\\n        <img src=\\\"https://github.com/ablaom/MLJTutorial.jl/blob/dev/notebooks/01_data_representation/scitypes.png?raw=true\\\">\\n</div>\\n\")",
      "text/html": [
       "<div style=\"text-align: left\";>\n",
       "        <img src=\"https://github.com/ablaom/MLJTutorial.jl/blob/dev/notebooks/01_data_representation/scitypes.png?raw=true\">\n",
       "</div>\n"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "cell_type": "code",
   "source": [
    "html\"\"\"\n",
    "<div style=\"text-align: left\";>\n",
    "        <img src=\"https://github.com/ablaom/MLJTutorial.jl/blob/dev/notebooks/01_data_representation/scitypes.png?raw=true\">\n",
    "</div>\n",
    "\"\"\""
   ],
   "metadata": {},
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "source": [
    "There are also container scitypes. For example, the scitype of any\n",
    "vector is `AbstractVector{S}`, where `S` is the scitype of its\n",
    "elements:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "AbstractVector{Textual}\u001b[90m (alias for \u001b[39m\u001b[90mAbstractArray{ScientificTypesBase.Textual, 1}\u001b[39m\u001b[90m)\u001b[39m"
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "cell_type": "code",
   "source": [
    "scitype([\"cat\", \"mouse\", \"dog\"])"
   ],
   "metadata": {},
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "source": [
    "We'll be using [OpenML](https://www.openml.org/home) to grab the\n",
    "Titanic dataset:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m8×7 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean     \u001b[0m\u001b[1m min    \u001b[0m\u001b[1m median  \u001b[0m\u001b[1m max     \u001b[0m\u001b[1m nmissing \u001b[0m\u001b[1m eltype         \u001b[0m ⋯\n     │\u001b[90m Symbol   \u001b[0m\u001b[90m Union…   \u001b[0m\u001b[90m Any    \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Any     \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m Type           \u001b[0m ⋯\n─────┼──────────────────────────────────────────────────────────────────────────\n   1 │ pclass   \u001b[90m          \u001b[0m 1      \u001b[90m         \u001b[0m 3               0  CategoricalValu ⋯\n   2 │ sex      \u001b[90m          \u001b[0m female \u001b[90m         \u001b[0m male            0  CategoricalValu\n   3 │ age       29.7589   0.42    30.0     80.0            0  Float64\n   4 │ sibsp     0.523008  0.0     0.0      8.0             0  Float64\n   5 │ fare      32.2042   0.0     14.4542  512.329         0  Float64         ⋯\n   6 │ cabin    \u001b[90m          \u001b[0m E31    \u001b[90m         \u001b[0m C148          687  Union{Missing,\n   7 │ embarked \u001b[90m          \u001b[0m C      \u001b[90m         \u001b[0m S               2  Union{Missing,\n   8 │ survived \u001b[90m          \u001b[0m 0      \u001b[90m         \u001b[0m 1               0  CategoricalValu\n\u001b[36m                                                                1 column omitted\u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>8×7 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">mean</th><th style = \"text-align: left;\">min</th><th style = \"text-align: left;\">median</th><th style = \"text-align: left;\">max</th><th style = \"text-align: left;\">nmissing</th><th style = \"text-align: left;\">eltype</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Type\" style = \"text-align: left;\">Type</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">pclass</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">1</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">3</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sex</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">female</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">age</td><td style = \"text-align: left;\">29.7589</td><td style = \"text-align: left;\">0.42</td><td style = \"text-align: left;\">30.0</td><td style = \"text-align: left;\">80.0</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">sibsp</td><td style = \"text-align: left;\">0.523008</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">8.0</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">fare</td><td style = \"text-align: left;\">32.2042</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">14.4542</td><td style = \"text-align: left;\">512.329</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: left;\">cabin</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">E31</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">C148</td><td style = \"text-align: right;\">687</td><td style = \"text-align: left;\">Union{Missing, CategoricalValue{String, UInt32}}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: left;\">embarked</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">C</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">S</td><td style = \"text-align: right;\">2</td><td style = \"text-align: left;\">Union{Missing, CategoricalValue{String, UInt32}}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">8</td><td style = \"text-align: left;\">survived</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">0</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">1</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "cell_type": "code",
   "source": [
    "table = OpenML.load(42638)\n",
    "df0 = DataFrames.DataFrame(table)\n",
    "DataFrames.describe(df0)"
   ],
   "metadata": {},
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `schema` operator summarizes the column scitypes of a table:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "┌──────────┬─────────────────────────────────┬──────────────────────────────────\n│\u001b[22m names    \u001b[0m│\u001b[22m scitypes                        \u001b[0m│\u001b[22m types                          \u001b[0m ⋯\n├──────────┼─────────────────────────────────┼──────────────────────────────────\n│ pclass   │ Multiclass{3}                   │ CategoricalValue{String, UInt32 ⋯\n│ sex      │ Multiclass{2}                   │ CategoricalValue{String, UInt32 ⋯\n│ age      │ Continuous                      │ Float64                         ⋯\n│ sibsp    │ Continuous                      │ Float64                         ⋯\n│ fare     │ Continuous                      │ Float64                         ⋯\n│ cabin    │ Union{Missing, Multiclass{186}} │ Union{Missing, CategoricalValue ⋯\n│ embarked │ Union{Missing, Multiclass{3}}   │ Union{Missing, CategoricalValue ⋯\n│ survived │ Multiclass{2}                   │ CategoricalValue{String, UInt32 ⋯\n└──────────┴─────────────────────────────────┴──────────────────────────────────\n\u001b[36m                                                                1 column omitted\u001b[0m\n"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "cell_type": "code",
   "source": [
    "schema(df0)"
   ],
   "metadata": {},
   "execution_count": 9
  },
  {
   "cell_type": "markdown",
   "source": [
    "Looks like we need to fix `:sibsp`, the number of siblings/spouses:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "┌──────────┬─────────────────────────────────┬──────────────────────────────────\n│\u001b[22m names    \u001b[0m│\u001b[22m scitypes                        \u001b[0m│\u001b[22m types                          \u001b[0m ⋯\n├──────────┼─────────────────────────────────┼──────────────────────────────────\n│ pclass   │ Multiclass{3}                   │ CategoricalValue{String, UInt32 ⋯\n│ sex      │ Multiclass{2}                   │ CategoricalValue{String, UInt32 ⋯\n│ age      │ Continuous                      │ Float64                         ⋯\n│ sibsp    │ Count                           │ Int64                           ⋯\n│ fare     │ Continuous                      │ Float64                         ⋯\n│ cabin    │ Union{Missing, Multiclass{186}} │ Union{Missing, CategoricalValue ⋯\n│ embarked │ Union{Missing, Multiclass{3}}   │ Union{Missing, CategoricalValue ⋯\n│ survived │ Multiclass{2}                   │ CategoricalValue{String, UInt32 ⋯\n└──────────┴─────────────────────────────────┴──────────────────────────────────\n\u001b[36m                                                                1 column omitted\u001b[0m\n"
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "cell_type": "code",
   "source": [
    "df1 = coerce(df0, :sibsp => Count)\n",
    "schema(df1)"
   ],
   "metadata": {},
   "execution_count": 10
  },
  {
   "cell_type": "markdown",
   "source": [
    "Lets take a closer look at our target column :survived. Here a value\n",
    "`0` means that the individual didn't survive while a value of `1` indicates\n",
    "an individual survived."
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "2-element Vector{String}:\n \"0\"\n \"1\""
     },
     "metadata": {},
     "execution_count": 11
    }
   ],
   "cell_type": "code",
   "source": [
    "levels(df1.survived)"
   ],
   "metadata": {},
   "execution_count": 11
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `:cabin` feature has a lot of missing values, and low frequency\n",
    "for other classes:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "Dict{Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}, Int64} with 148 entries:\n  \"C104\"    => 1\n  \"E50\"     => 1\n  \"D20\"     => 2\n  \"E58\"     => 1\n  \"C46\"     => 1\n  \"D37\"     => 1\n  \"B96 B98\" => 4\n  \"C86\"     => 1\n  \"C106\"    => 1\n  \"A5\"      => 1\n  \"C52\"     => 2\n  \"B19\"     => 1\n  \"C65\"     => 2\n  \"C30\"     => 1\n  \"D48\"     => 1\n  \"B42\"     => 1\n  \"C128\"    => 1\n  \"E38\"     => 1\n  \"E10\"     => 1\n  ⋮         => ⋮"
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "cell_type": "code",
   "source": [
    "import StatsBase\n",
    "StatsBase.countmap(df0.cabin)"
   ],
   "metadata": {},
   "execution_count": 12
  },
  {
   "cell_type": "markdown",
   "source": [
    "We'll make `missing` into a bona fide class and group all the other\n",
    "classes into one:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "class (generic function with 1 method)"
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "cell_type": "code",
   "source": [
    "function class(c)\n",
    "    if ismissing(c)\n",
    "        return \"without cabin\"\n",
    "    else\n",
    "        return \"has cabin\"\n",
    "    end\n",
    "end"
   ],
   "metadata": {},
   "execution_count": 13
  },
  {
   "cell_type": "markdown",
   "source": [
    "Shorthand syntax would be `class(c) = ismissing(c) ? \"without cabin\" :\n",
    "\"has cabin\"`. Now to transform the whole column:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "┌──────────┬───────────────────────────────┬────────────────────────────────────\n│\u001b[22m names    \u001b[0m│\u001b[22m scitypes                      \u001b[0m│\u001b[22m types                            \u001b[0m ⋯\n├──────────┼───────────────────────────────┼────────────────────────────────────\n│ pclass   │ Multiclass{3}                 │ CategoricalValue{String, UInt32}  ⋯\n│ sex      │ Multiclass{2}                 │ CategoricalValue{String, UInt32}  ⋯\n│ age      │ Continuous                    │ Float64                           ⋯\n│ sibsp    │ Count                         │ Int64                             ⋯\n│ fare     │ Continuous                    │ Float64                           ⋯\n│ cabin    │ Textual                       │ String                            ⋯\n│ embarked │ Union{Missing, Multiclass{3}} │ Union{Missing, CategoricalValue{S ⋯\n│ survived │ Multiclass{2}                 │ CategoricalValue{String, UInt32}  ⋯\n└──────────┴───────────────────────────────┴────────────────────────────────────\n\u001b[36m                                                                1 column omitted\u001b[0m\n"
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "cell_type": "code",
   "source": [
    "df2 = DataFrames.transform(\n",
    "    df1,\n",
    "    :cabin => DataFrames.ByRow(class) => :cabin\n",
    ") # :cabin now has `Textual` scitype\n",
    "coerce!(df2, :class => Multiclass)\n",
    "schema(df2)"
   ],
   "metadata": {},
   "execution_count": 14
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Splitting into train and test sets\n",
    "Here we split off 30% of our observations into a\n",
    "lock-and-throw-away-the-key holdout set, called `df_test`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "624"
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "cell_type": "code",
   "source": [
    "df, df_test = partition(df2, 0.7, rng=123)\n",
    "DataFrames.nrow(df)"
   ],
   "metadata": {},
   "execution_count": 15
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "267"
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "cell_type": "code",
   "source": [
    "DataFrames.nrow(df_test)"
   ],
   "metadata": {},
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Cleaning the data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's construct an MLJ model to impute missing data:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "FillImputer(\n  features = Symbol[], \n  continuous_fill = MLJModels._median, \n  count_fill = MLJModels._round_median, \n  finite_fill = MLJModels._mode)"
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "cell_type": "code",
   "source": [
    "cleaner = FillImputer()"
   ],
   "metadata": {},
   "execution_count": 17
  },
  {
   "cell_type": "markdown",
   "source": [
    "In MLJ a *model* is just a container for hyper-parameters associated with some ML\n",
    "algorithm. It does not store learned parameters (unlike scikit-learn \"estimators\"). In\n",
    "this case the hyper-parameters `features`, `continuous_fill`, `count_fill`, and\n",
    "`finite_fill` specify which features should be imputed and how imputation should be\n",
    "carried out, depending on the scitype. Since we didn't specify any features in our\n",
    "constructor, we are using default values."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now bind the model with training data in a *machine*:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "untrained Machine; caches model-specific representations of data\n  model: FillImputer(features = Symbol[], …)\n  args: \n    1:\tSource @090 ⏎ ScientificTypesBase.Table{Union{AbstractVector{ScientificTypesBase.Continuous}, AbstractVector{ScientificTypesBase.Count}, AbstractVector{ScientificTypesBase.Multiclass{3}}, AbstractVector{ScientificTypesBase.Textual}, AbstractVector{Union{Missing, ScientificTypesBase.Multiclass{3}}}, AbstractVector{ScientificTypesBase.Multiclass{2}}}}\n"
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "cell_type": "code",
   "source": [
    "machc = machine(cleaner, df)"
   ],
   "metadata": {},
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "source": [
    "And train the machine to store learned parameters there (the column\n",
    "modes and medians to be used to impute missings):"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ Info: Training machine(FillImputer(features = Symbol[], …), …).\n"
     ]
    }
   ],
   "cell_type": "code",
   "source": [
    "fit!(machc);"
   ],
   "metadata": {},
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can inspect the learned parameters if we want:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "Dict{Symbol, Any} with 7 entries:\n  :sibsp    => 0\n  :pclass   => \"3\"\n  :survived => \"0\"\n  :sex      => \"male\"\n  :age      => 30.0\n  :fare     => 14.5\n  :embarked => \"S\""
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "cell_type": "code",
   "source": [
    "fitted_params(machc).filler_given_feature"
   ],
   "metadata": {},
   "execution_count": 20
  },
  {
   "cell_type": "markdown",
   "source": [
    "Next, we apply the learned transformation on our data:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "┌──────────┬───────────────┬──────────────────────────────────┐\n│\u001b[22m names    \u001b[0m│\u001b[22m scitypes      \u001b[0m│\u001b[22m types                            \u001b[0m│\n├──────────┼───────────────┼──────────────────────────────────┤\n│ pclass   │ Multiclass{3} │ CategoricalValue{String, UInt32} │\n│ sex      │ Multiclass{2} │ CategoricalValue{String, UInt32} │\n│ age      │ Continuous    │ Float64                          │\n│ sibsp    │ Count         │ Int64                            │\n│ fare     │ Continuous    │ Float64                          │\n│ cabin    │ Textual       │ String                           │\n│ embarked │ Multiclass{3} │ CategoricalValue{String, UInt32} │\n│ survived │ Multiclass{2} │ CategoricalValue{String, UInt32} │\n└──────────┴───────────────┴──────────────────────────────────┘\n"
     },
     "metadata": {},
     "execution_count": 21
    }
   ],
   "cell_type": "code",
   "source": [
    "dfc     =  transform(machc, df)\n",
    "dfc_test = transform(machc, df_test)\n",
    "schema(dfc)"
   ],
   "metadata": {},
   "execution_count": 21
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Split the data into input features and target"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The following method puts the column with name equal to `:survived`\n",
    "into the vector `y`, and everything else into a table (`DataFrame`)\n",
    "called `X`."
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "AbstractVector{Multiclass{2}}\u001b[90m (alias for \u001b[39m\u001b[90mAbstractArray{ScientificTypesBase.Multiclass{2}, 1}\u001b[39m\u001b[90m)\u001b[39m"
     },
     "metadata": {},
     "execution_count": 22
    }
   ],
   "cell_type": "code",
   "source": [
    "y, X = unpack(dfc, ==(:survived));\n",
    "scitype(y)"
   ],
   "metadata": {},
   "execution_count": 22
  },
  {
   "cell_type": "markdown",
   "source": [
    "While we're here, we'll do the same for the holdout test set:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "y_test, X_test = unpack(dfc_test, ==(:survived));"
   ],
   "metadata": {},
   "execution_count": 23
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Choosing a supervised model:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "There are not many models that can directly handle a mixture of\n",
    "scitypes, as we have here:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "4-element Vector{NamedTuple{(:name, :package_name, :is_supervised, :abstract_type, :deep_properties, :docstring, :fit_data_scitype, :human_name, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :inverse_transform_scitype, :is_pure_julia, :is_wrapper, :iteration_parameter, :load_path, :package_license, :package_url, :package_uuid, :predict_scitype, :prediction_type, :reporting_operations, :reports_feature_importances, :supports_class_weights, :supports_online, :supports_training_losses, :supports_weights, :transform_scitype, :input_scitype, :target_scitype, :output_scitype)}}:\n (name = ConstantClassifier, package_name = MLJModels, ... )\n (name = DecisionTreeClassifier, package_name = BetaML, ... )\n (name = DeterministicConstantClassifier, package_name = MLJModels, ... )\n (name = RandomForestClassifier, package_name = BetaML, ... )"
     },
     "metadata": {},
     "execution_count": 24
    }
   ],
   "cell_type": "code",
   "source": [
    "models(matching(X, y))"
   ],
   "metadata": {},
   "execution_count": 24
  },
  {
   "cell_type": "markdown",
   "source": [
    "This can be mitigated with further pre-processing (such as one-hot\n",
    "encoding) but we'll settle for one the above models here:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[36m  mutable struct DecisionTreeClassifier <: MLJModelInterface.Probabilistic\u001b[39m\n\n  A simple Decision Tree model for classification with support for Missing\n  data, from the Beta Machine Learning Toolkit (BetaML).\n\n\u001b[1m  Hyperparameters:\u001b[22m\n\u001b[1m  ≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡\u001b[22m\n\n    •  \u001b[36mmax_depth::Int64\u001b[39m: The maximum depth the tree is allowed to reach.\n       When this is reached the node is forced to become a leaf [def: \u001b[36m0\u001b[39m,\n       i.e. no limits]\n\n    •  \u001b[36mmin_gain::Float64\u001b[39m: The minimum information gain to allow for a\n       node's partition [def: \u001b[36m0\u001b[39m]\n\n    •  \u001b[36mmin_records::Int64\u001b[39m: The minimum number of records a node must\n       holds to consider for a partition of it [def: \u001b[36m2\u001b[39m]\n\n    •  \u001b[36mmax_features::Int64\u001b[39m: The maximum number of (random) features to\n       consider at each partitioning [def: \u001b[36m0\u001b[39m, i.e. look at all features]\n\n    •  \u001b[36msplitting_criterion::Function\u001b[39m: This is the name of the function to\n       be used to compute the information gain of a specific partition.\n       This is done by measuring the difference betwwen the \"impurity\" of\n       the labels of the parent node with those of the two child nodes,\n       weighted by the respective number of items. [def: \u001b[36mgini\u001b[39m]. Either\n       \u001b[36mgini\u001b[39m, \u001b[36mentropy\u001b[39m or a custom function. It can also be an anonymous\n       function.\n\n    •  \u001b[36mrng::Random.AbstractRNG\u001b[39m: A Random Number Generator to be used in\n       stochastic parts of the code [deafult: \u001b[36mRandom.GLOBAL_RNG\u001b[39m]\n\n\u001b[1m  Example:\u001b[22m\n\u001b[1m  ≡≡≡≡≡≡≡≡\u001b[22m\n\n\u001b[36m  julia> using MLJ\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> X, y        = @load_iris;\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> modelType   = @load DecisionTreeClassifier pkg = \"BetaML\" verbosity=0\u001b[39m\n\u001b[36m  BetaML.Trees.DecisionTreeClassifier\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> model       = modelType()\u001b[39m\n\u001b[36m  DecisionTreeClassifier(\u001b[39m\n\u001b[36m    max_depth = 0, \u001b[39m\n\u001b[36m    min_gain = 0.0, \u001b[39m\n\u001b[36m    min_records = 2, \u001b[39m\n\u001b[36m    max_features = 0, \u001b[39m\n\u001b[36m    splitting_criterion = BetaML.Utils.gini, \u001b[39m\n\u001b[36m    rng = Random._GLOBAL_RNG())\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> mach        = machine(model, X, y);\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> fit!(mach);\u001b[39m\n\u001b[36m  [ Info: Training machine(DecisionTreeClassifier(max_depth = 0, …), …).\u001b[39m\n\u001b[36m  \u001b[39m\n\u001b[36m  julia> cat_est    = predict(mach, X)\u001b[39m\n\u001b[36m  150-element CategoricalDistributions.UnivariateFiniteVector{Multiclass{3}, String, UInt32, Float64}:\u001b[39m\n\u001b[36m   UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)\u001b[39m\n\u001b[36m   UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)\u001b[39m\n\u001b[36m   ⋮\u001b[39m\n\u001b[36m   UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\u001b[39m\n\u001b[36m   UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\u001b[39m\n\u001b[36m   UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\u001b[39m",
      "text/markdown": "```julia\nmutable struct DecisionTreeClassifier <: MLJModelInterface.Probabilistic\n```\n\nA simple Decision Tree model for classification with support for Missing data, from the Beta Machine Learning Toolkit (BetaML).\n\n# Hyperparameters:\n\n  * `max_depth::Int64`: The maximum depth the tree is allowed to reach. When this is reached the node is forced to become a leaf [def: `0`, i.e. no limits]\n  * `min_gain::Float64`: The minimum information gain to allow for a node's partition [def: `0`]\n  * `min_records::Int64`: The minimum number of records a node must holds to consider for a partition of it [def: `2`]\n  * `max_features::Int64`: The maximum number of (random) features to consider at each partitioning [def: `0`, i.e. look at all features]\n  * `splitting_criterion::Function`: This is the name of the function to be used to compute the information gain of a specific partition. This is done by measuring the difference betwwen the \"impurity\" of the labels of the parent node with those of the two child nodes, weighted by the respective number of items. [def: `gini`]. Either `gini`, `entropy` or a custom function. It can also be an anonymous function.\n  * `rng::Random.AbstractRNG`: A Random Number Generator to be used in stochastic parts of the code [deafult: `Random.GLOBAL_RNG`]\n\n# Example:\n\n```julia\njulia> using MLJ\n\njulia> X, y        = @load_iris;\n\njulia> modelType   = @load DecisionTreeClassifier pkg = \"BetaML\" verbosity=0\nBetaML.Trees.DecisionTreeClassifier\n\njulia> model       = modelType()\nDecisionTreeClassifier(\n  max_depth = 0, \n  min_gain = 0.0, \n  min_records = 2, \n  max_features = 0, \n  splitting_criterion = BetaML.Utils.gini, \n  rng = Random._GLOBAL_RNG())\n\njulia> mach        = machine(model, X, y);\n\njulia> fit!(mach);\n[ Info: Training machine(DecisionTreeClassifier(max_depth = 0, …), …).\n\njulia> cat_est    = predict(mach, X)\n150-element CategoricalDistributions.UnivariateFiniteVector{Multiclass{3}, String, UInt32, Float64}:\n UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)\n UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)\n ⋮\n UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\n UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\n UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0)\n```\n"
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "cell_type": "code",
   "source": [
    "doc(\"DecisionTreeClassifier\", pkg=\"BetaML\")"
   ],
   "metadata": {},
   "execution_count": 25
  },
  {
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ Info: For silent loading, specify `verbosity=0`. \n",
      "import BetaML ✔\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "DecisionTreeClassifier(\n  max_depth = 0, \n  min_gain = 0.0, \n  min_records = 2, \n  max_features = 0, \n  splitting_criterion = BetaML.Utils.gini, \n  rng = Random._GLOBAL_RNG())"
     },
     "metadata": {},
     "execution_count": 26
    }
   ],
   "cell_type": "code",
   "source": [
    "Tree = @load DecisionTreeClassifier pkg=BetaML  # model type\n",
    "tree = Tree()                                   # default instance"
   ],
   "metadata": {},
   "execution_count": 26
  },
  {
   "cell_type": "markdown",
   "source": [
    "Notice that by calling `Tree` with no arguments we get default\n",
    "values for the various hyperparameters that control how the tree is\n",
    "trained. We specify keyword arguments to overide these defaults. For example:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "DecisionTreeClassifier(\n  max_depth = 3, \n  min_gain = 0.0, \n  min_records = 2, \n  max_features = 0, \n  splitting_criterion = BetaML.Utils.gini, \n  rng = Random._GLOBAL_RNG())"
     },
     "metadata": {},
     "execution_count": 27
    }
   ],
   "cell_type": "code",
   "source": [
    "small_tree = Tree(max_depth=3)"
   ],
   "metadata": {},
   "execution_count": 27
  },
  {
   "cell_type": "markdown",
   "source": [
    "A decision tree is frequently not the best performing model, but it\n",
    "is easy to interpret (and the algorithm is relatively easy to\n",
    "explain). For example, here's an diagramatic representation of a\n",
    "tree trained on (some part of) the Titanic data set, which suggests\n",
    "how prediction works:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "HTML{String}(\"<div style=\\\"text-align: left\\\";>\\n        <img src=\\\"https://upload.wikimedia.org/wikipedia/commons/5/58/Decision_Tree_-_survival_of_passengers_on_the_Titanic.jpg\\\">\\n</div>\\n\")",
      "text/html": [
       "<div style=\"text-align: left\";>\n",
       "        <img src=\"https://upload.wikimedia.org/wikipedia/commons/5/58/Decision_Tree_-_survival_of_passengers_on_the_Titanic.jpg\">\n",
       "</div>\n"
      ]
     },
     "metadata": {},
     "execution_count": 28
    }
   ],
   "cell_type": "code",
   "source": [
    "html\"\"\"\n",
    "<div style=\"text-align: left\";>\n",
    "        <img src=\"https://upload.wikimedia.org/wikipedia/commons/5/58/Decision_Tree_-_survival_of_passengers_on_the_Titanic.jpg\">\n",
    "</div>\n",
    "\"\"\""
   ],
   "metadata": {},
   "execution_count": 28
  },
  {
   "cell_type": "markdown",
   "source": [
    "## The fit/predict worflow"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now the bind data to be used for training and evaluation to the model (ie, choice of\n",
    "hyperparameters) in a machine, just like we did for missing value imputation. In this\n",
    "case, however, we also need to specify the training target `y`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "untrained Machine; caches model-specific representations of data\n  model: DecisionTreeClassifier(max_depth = 0, …)\n  args: \n    1:\tSource @126 ⏎ ScientificTypesBase.Table{Union{AbstractVector{ScientificTypesBase.Continuous}, AbstractVector{ScientificTypesBase.Count}, AbstractVector{ScientificTypesBase.Multiclass{2}}, AbstractVector{ScientificTypesBase.Multiclass{3}}, AbstractVector{ScientificTypesBase.Textual}}}\n    2:\tSource @872 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n"
     },
     "metadata": {},
     "execution_count": 29
    }
   ],
   "cell_type": "code",
   "source": [
    "macht = machine(tree, X, y)"
   ],
   "metadata": {},
   "execution_count": 29
  },
  {
   "cell_type": "markdown",
   "source": [
    "To train using *all* the bound data:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ Info: Training machine(DecisionTreeClassifier(max_depth = 0, …), …).\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "trained Machine; caches model-specific representations of data\n  model: DecisionTreeClassifier(max_depth = 0, …)\n  args: \n    1:\tSource @126 ⏎ ScientificTypesBase.Table{Union{AbstractVector{ScientificTypesBase.Continuous}, AbstractVector{ScientificTypesBase.Count}, AbstractVector{ScientificTypesBase.Multiclass{2}}, AbstractVector{ScientificTypesBase.Multiclass{3}}, AbstractVector{ScientificTypesBase.Textual}}}\n    2:\tSource @872 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n"
     },
     "metadata": {},
     "execution_count": 30
    }
   ],
   "cell_type": "code",
   "source": [
    "fit!(macht)"
   ],
   "metadata": {},
   "execution_count": 30
  },
  {
   "cell_type": "markdown",
   "source": [
    "And get predictions on the holdout set:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "p = predict(macht, X_test);"
   ],
   "metadata": {},
   "execution_count": 31
  },
  {
   "cell_type": "markdown",
   "source": [
    "These are *probabilistic* predictions:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "UnivariateFinite{ScientificTypesBase.Multiclass{2}}(0=>0.2, 1=>0.8)"
     },
     "metadata": {},
     "execution_count": 32
    }
   ],
   "cell_type": "code",
   "source": [
    "p[3]"
   ],
   "metadata": {},
   "execution_count": 32
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "0.2"
     },
     "metadata": {},
     "execution_count": 33
    }
   ],
   "cell_type": "code",
   "source": [
    "pdf(p[3], \"0\")"
   ],
   "metadata": {},
   "execution_count": 33
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also get \"point\" predictions:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "267-element CategoricalArrays.CategoricalArray{String,1,UInt32}:\n \"1\"\n \"1\"\n \"1\"\n \"0\"\n \"0\"\n \"0\"\n \"1\"\n \"0\"\n \"0\"\n \"0\"\n ⋮\n \"0\"\n \"0\"\n \"0\"\n \"0\"\n \"1\"\n \"0\"\n \"1\"\n \"0\"\n \"0\""
     },
     "metadata": {},
     "execution_count": 34
    }
   ],
   "cell_type": "code",
   "source": [
    "yhat = mode.(p)"
   ],
   "metadata": {},
   "execution_count": 34
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can evaluate performance using a probabilistic measure, as in"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "8.307615579043436"
     },
     "metadata": {},
     "execution_count": 35
    }
   ],
   "cell_type": "code",
   "source": [
    "log_loss(p, y_test) |> mean"
   ],
   "metadata": {},
   "execution_count": 35
  },
  {
   "cell_type": "markdown",
   "source": [
    "Or using a deterministic measure:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "0.7265917602996255"
     },
     "metadata": {},
     "execution_count": 36
    }
   ],
   "cell_type": "code",
   "source": [
    "accuracy(yhat, y_test)"
   ],
   "metadata": {},
   "execution_count": 36
  },
  {
   "cell_type": "markdown",
   "source": [
    "List all performance measures with `measures()`. Naturally, MLJ\n",
    "includes functions to automate this kind of performance evaluation,\n",
    "but this is beyond the scope of this tutorial. See, eg,\n",
    "[here](https://JuliaAI.github.io/MLJ.jl/dev/getting_started/#Getting-Started)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Learning more"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Some suggestions for next steps are\n",
    "[here](https://JuliaAI.github.io/MLJ.jl/dev/getting_started/#Getting-Started)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "---\n",
    "\n",
    "*This notebook was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).*"
   ],
   "metadata": {}
  }
 ],
 "nbformat_minor": 3,
 "metadata": {
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.10.3"
  },
  "kernelspec": {
   "name": "julia-1.10",
   "display_name": "Julia 1.10.3",
   "language": "julia"
  }
 },
 "nbformat": 4
}