lectures/09_ensembles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CPSC 330 Lecture 9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Lecture plan\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements (5 min)\n",
    "- Evaluation metrics True/False (10 min)\n",
    "- Introducing some other classifiers (20 min)\n",
    "- Break (5 mins)\n",
    "- Ensembles: averaging (20 min)\n",
    "- Ensembles: stacking (20 min)\n",
    "- Summary\n",
    "\n",
    "Piazza:\n",
    "\n",
    "- Ensembles True/False questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Name a few popular tree-based classifier.\n",
    "- Name a few key hyperparameters of random forest classifiers.\n",
    "- Weigh the pros and cons of using a classifier from outside scikit-learn.\n",
    "- Employ ensemble classifier approaches, in particular model averaging and stacking\n",
    "- Compare averaging vs. stacking\n",
    "- Interpret the coefficients of the meta-classifier in the case of stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements\n",
    "\n",
    "- hw schedule for the rest of the course posted at https://github.com/UBC-CS/cpsc330#homework-schedule\n",
    "- hw4 posted, due Monday 11:59pm\n",
    "- Evaluation metrics true/false: https://piazza.com/class/kb2e6nwu3uj23?cid=283\n",
    "- Midterm format: https://piazza.com/class/kb2e6nwu3uj23?cid=281 and https://piazza.com/class/kb2e6nwu3uj23?cid=284"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_validate\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n",
    "from sklearn.ensemble import VotingClassifier, StackingClassifier\n",
    "\n",
    "from xgboost import XGBClassifier\n",
    "from lightgbm import LGBMClassifier\n",
    "from catboost import CatBoostClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copied from lecture 2\n",
    "import re\n",
    "import graphviz\n",
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "def display_tree(feature_names, tree):\n",
    "    \"\"\" For binary classification only \"\"\"\n",
    "    dot = export_graphviz(tree, out_file=None, feature_names=feature_names, class_names=tree.classes_.astype(str), impurity=False, precision=4)\n",
    "    # adapted from https://stackoverflow.com/questions/44821349/python-graphviz-remove-legend-on-nodes-of-decisiontreeclassifier\n",
    "    dot = re.sub('(\\\\\\\\nsamples = [0-9]+)(\\\\\\\\nvalue = \\[[0-9]+, [0-9]+\\])(\\\\\\\\nclass = [A-Za-z0-9]+)', '', dot)\n",
    "    dot = re.sub(     '(samples = [0-9]+)(\\\\\\\\nvalue = \\[[0-9]+, [0-9]+\\])\\\\\\\\n', '', dot)\n",
    "    return graphviz.Source(dot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducing some other classifiers (20 min)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "cc_df = pd.read_csv('data/creditcard.csv', encoding='latin-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_test = train_test_split(cc_df, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today I will use a smaller data set for speed of the demos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.sample(frac=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21360, 31)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_train.drop(columns=['Class'])\n",
    "y_train = df_train['Class']\n",
    "\n",
    "X_test = df_test.drop(columns=['Class'])\n",
    "y_test = df_test['Class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr = Pipeline([\n",
    "    ('scale', StandardScaler()),\n",
    "    ('lr', LogisticRegression(max_iter=1000, class_weight='balanced'))\n",
    "]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_method = 'average_precision'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.215157\n",
       "score_time     0.006149\n",
       "test_score     0.746118\n",
       "train_score    0.820779\n",
       "dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_results = pd.DataFrame(cross_validate(pipe_lr, X_train, y_train, scoring=score_method, return_train_score=True)).mean()\n",
    "lr_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = DecisionTreeClassifier(class_weight='balanced', random_state=123) # scaling not needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.189141\n",
       "score_time     0.003725\n",
       "test_score     0.518204\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt_results = pd.DataFrame(cross_validate(dt, X_train, y_train, scoring=score_method, return_train_score=True)).mean()\n",
    "dt_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about some other classifiers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This is basically the average of a bunch of random decision trees. \n",
    "- We'll talk about averaging later today!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(class_weight='balanced', random_state=999) # scaling not needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       2.274872\n",
       "score_time     0.027058\n",
       "test_score     0.788906\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_results = pd.DataFrame(cross_validate(rf, X_train, y_train, scoring=score_method, return_train_score=True)).mean()\n",
    "rf_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gets a better score!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>2.274872</td>\n",
       "      <td>0.027058</td>\n",
       "      <td>0.788906</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.215157</td>\n",
       "      <td>0.006149</td>\n",
       "      <td>0.746118</td>\n",
       "      <td>0.820779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.189141</td>\n",
       "      <td>0.003725</td>\n",
       "      <td>0.518204</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     fit_time  score_time  test_score  train_score\n",
       "random forest        2.274872    0.027058    0.788906     1.000000\n",
       "logistic regression  0.215157    0.006149    0.746118     0.820779\n",
       "decision tree        0.189141    0.003725    0.518204     1.000000"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = pd.DataFrame([lr_results, dt_results, rf_results], index=[\"logistic regression\", \"decision tree\", \"random forest\"])\n",
    "summary.sort_values(by=[\"test_score\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are these random forests you speak of?\n",
    "\n",
    "- They are a collection of decision trees.\n",
    "- Each tree \"votes\" on the prediction, majority rules (more on this later).\n",
    "- Each tree (and split) is limited in the number of features it can look at.\n",
    "- Each tree is training on a slightly different version of the dataset. \n",
    "\n",
    "We can actually look at the sub-trees in a trained random forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Tree 1\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"380pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 380.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 376,-184 376,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"241,-180 131,-180 131,-144 241,-144 241,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"186\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V17 &lt;= &#45;2.924.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"177.5,-108 72.5,-108 72.5,-72 177.5,-72 177.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"125\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V13 &lt;= 1.704.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M170.6071,-143.8314C163.5096,-135.454 154.952,-125.3531 147.176,-116.1749\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"149.7348,-113.7807 140.6001,-108.4133 144.3939,-118.3056 149.7348,-113.7807\"/>\n",
       "<text text-anchor=\"middle\" x=\"138.5395\" y=\"-129.6282\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"298.5,-108 195.5,-108 195.5,-72 298.5,-72 298.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"247\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V9 &lt;= &#45;3.641.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M201.3929,-143.8314C208.4904,-135.454 217.048,-125.3531 224.824,-116.1749\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"227.6061,-118.3056 231.3999,-108.4133 222.2652,-113.7807 227.6061,-118.3056\"/>\n",
       "<text text-anchor=\"middle\" x=\"233.4605\" y=\"-129.6282\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"76,-36 0,-36 0,0 76,0 76,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"38\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M103.0462,-71.8314C92.3305,-62.9632 79.2811,-52.1637 67.691,-42.5718\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"69.8822,-39.8422 59.9467,-36.1628 65.4192,-45.2349 69.8822,-39.8422\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"170,-36 94,-36 94,0 170,0 170,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"132\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M126.7664,-71.8314C127.515,-64.131 128.4053,-54.9743 129.2373,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"132.7256,-46.7051 130.2098,-36.4133 125.7585,-46.0276 132.7256,-46.7051\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"278,-36 202,-36 202,0 278,0 278,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"240\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M245.2336,-71.8314C244.485,-64.131 243.5947,-54.9743 242.7627,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"246.2415,-46.0276 241.7902,-36.4133 239.2744,-46.7051 246.2415,-46.0276\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"372,-36 296,-36 296,0 372,0 372,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"334\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M268.9538,-71.8314C279.6695,-62.9632 292.7189,-52.1637 304.309,-42.5718\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"306.5808,-45.2349 312.0533,-36.1628 302.1178,-39.8422 306.5808,-45.2349\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbceb0a6d60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Tree 2\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"383pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 383.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 379,-184 379,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"236,-180 138,-180 138,-144 236,-144 236,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"187\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V8 &lt;= 8.923.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"178,-108 68,-108 68,-72 178,-72 178,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"123\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V16 &lt;= &#45;4.317.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M170.8501,-143.8314C163.3283,-135.3694 154.2435,-125.1489 146.0196,-115.8971\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"148.627,-113.5621 139.3673,-108.4133 143.3951,-118.2127 148.627,-113.5621\"/>\n",
       "<text text-anchor=\"middle\" x=\"137.8993\" y=\"-129.6701\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"306,-108 196,-108 196,-72 306,-72 306,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"251\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V17 &lt;= &#45;4.737.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M203.1499,-143.8314C210.6717,-135.3694 219.7565,-125.1489 227.9804,-115.8971\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"230.6049,-118.2127 234.6327,-108.4133 225.373,-113.5621 230.6049,-118.2127\"/>\n",
       "<text text-anchor=\"middle\" x=\"236.1007\" y=\"-129.6701\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"76,-36 0,-36 0,0 76,0 76,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"38\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M101.5509,-71.8314C91.1822,-63.0485 78.5771,-52.3712 67.3354,-42.8489\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"69.3349,-39.9556 59.4422,-36.1628 64.8105,-45.297 69.3349,-39.9556\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"170,-36 94,-36 94,0 170,0 170,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"132\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M125.2711,-71.8314C126.2336,-64.131 127.3782,-54.9743 128.4479,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"131.9309,-46.7702 129.6983,-36.4133 124.9849,-45.9019 131.9309,-46.7702\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"281,-36 205,-36 205,0 281,0 281,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"243\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M248.9813,-71.8314C248.1257,-64.131 247.1083,-54.9743 246.1574,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"249.6289,-45.9656 245.0459,-36.4133 242.6717,-46.7386 249.6289,-45.9656\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"375,-36 299,-36 299,0 375,0 375,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"337\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M272.7014,-71.8314C283.1921,-63.0485 295.9455,-52.3712 307.3194,-42.8489\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"309.8847,-45.2659 315.3055,-36.1628 305.3911,-39.8986 309.8847,-45.2659\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbceb0a6130>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Tree 3\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"380pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 380.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 376,-184 376,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"237.5,-180 134.5,-180 134.5,-144 237.5,-144 237.5,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"186\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V3 &lt;= &#45;7.406.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"177.5,-108 72.5,-108 72.5,-72 177.5,-72 177.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"125\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V19 &lt;= 0.477.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M170.6071,-143.8314C163.5096,-135.454 154.952,-125.3531 147.176,-116.1749\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"149.7348,-113.7807 140.6001,-108.4133 144.3939,-118.3056 149.7348,-113.7807\"/>\n",
       "<text text-anchor=\"middle\" x=\"138.5395\" y=\"-129.6282\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"298.5,-108 195.5,-108 195.5,-72 298.5,-72 298.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"247\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">V17 &lt;= &#45;4.06.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M201.3929,-143.8314C208.4904,-135.454 217.048,-125.3531 224.824,-116.1749\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"227.6061,-118.3056 231.3999,-108.4133 222.2652,-113.7807 227.6061,-118.3056\"/>\n",
       "<text text-anchor=\"middle\" x=\"233.4605\" y=\"-129.6282\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"76,-36 0,-36 0,0 76,0 76,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"38\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M103.0462,-71.8314C92.3305,-62.9632 79.2811,-52.1637 67.691,-42.5718\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"69.8822,-39.8422 59.9467,-36.1628 65.4192,-45.2349 69.8822,-39.8422\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"170,-36 94,-36 94,0 170,0 170,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"132\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M126.7664,-71.8314C127.515,-64.131 128.4053,-54.9743 129.2373,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"132.7256,-46.7051 130.2098,-36.4133 125.7585,-46.0276 132.7256,-46.7051\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"278,-36 202,-36 202,0 278,0 278,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"240\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M245.2336,-71.8314C244.485,-64.131 243.5947,-54.9743 242.7627,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"246.2415,-46.0276 241.7902,-36.4133 239.2744,-46.7051 246.2415,-46.0276\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"372,-36 296,-36 296,0 372,0 372,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"334\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M268.9538,-71.8314C279.6695,-62.9632 292.7189,-52.1637 304.309,-42.5718\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"306.5808,-45.2349 312.0533,-36.1628 302.1178,-39.8422 306.5808,-45.2349\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbceb0a6d60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf_demo = RandomForestClassifier(max_depth=2, n_estimators=3, random_state=321)\n",
    "rf_demo.fit(X_train, y_train)\n",
    "for i, tree in enumerate(rf_demo.estimators_):\n",
    "    print(\"\\n\\nTree\", i+1)\n",
    "    display(display_tree(X_train.columns, tree))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that they look at different features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some notable hyperparameters:\n",
    "\n",
    "- `n_estimators`: number of decision trees (higher = more complexity)\n",
    "- `max_depth`: max depth of each decision tree (higher = more complexity)\n",
    "- `max_features`: the number of features you get to look at each split (higher = more complexity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ??RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We can also try some other tree-based classifiers using \"gradient boosting\".\n",
    "  - We won't go into the details of how they work, but we can think of them as more complicated random forests.\n",
    "- Important to note that all of these **do** implement `predict_proba` so we can use something like ROC AUC.\n",
    "  - The probability scores come from the variation in the votes across trees, and other fancier sources.\n",
    "- But not all of them implement `class_weight`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb = GradientBoostingClassifier(random_state=999) # no class_weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       14.985047\n",
       "score_time      0.006453\n",
       "test_score      0.638259\n",
       "train_score     0.866346\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb_results = pd.DataFrame(cross_validate(gb, X_train, y_train, scoring=score_method, return_train_score=True)).mean()\n",
    "gb_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>2.274872</td>\n",
       "      <td>0.027058</td>\n",
       "      <td>0.788906</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.215157</td>\n",
       "      <td>0.006149</td>\n",
       "      <td>0.746118</td>\n",
       "      <td>0.820779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gradient boosting</th>\n",
       "      <td>14.985047</td>\n",
       "      <td>0.006453</td>\n",
       "      <td>0.638259</td>\n",
       "      <td>0.866346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.189141</td>\n",
       "      <td>0.003725</td>\n",
       "      <td>0.518204</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      fit_time  score_time  test_score  train_score\n",
       "random forest         2.274872    0.027058    0.788906     1.000000\n",
       "logistic regression   0.215157    0.006149    0.746118     0.820779\n",
       "gradient boosting    14.985047    0.006453    0.638259     0.866346\n",
       "decision tree         0.189141    0.003725    0.518204     1.000000"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = pd.DataFrame([lr_results, dt_results, rf_results, gb_results], index=[\"logistic regression\", \"decision tree\", \"random forest\", \"gradient boosting\"])\n",
    "summary.sort_values(by=[\"test_score\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Not very good, possible because of `class_weight`. (Though `class_weight` should not matter so much for this scoring metric - to be investigated further...)\n",
    "- It's also _very_ slow, so we'll drop it going forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classifiers outside sklearn\n",
    "\n",
    "- We can also try some other gradient boosting tree-based classifiers from outside sklearn.\n",
    "- There are more \"fancy\" are are **extremely popular**.  \n",
    "- Over time, people created \"sklearn-friendly\" wrappers so that these classifiers are compatible with all that we know & love.\n",
    "  - As long as they implement `fit` and `predict` and `predict_proba` and `score` we can plug them right in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "from lightgbm import LGBMClassifier\n",
    "from catboost import CatBoostClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`XGBClassifier` doesn't have an easy implementation of `class_weight='balanced'` AFAIK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    453.468085\n",
       "1      1.000000\n",
       "Name: Class, dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vc = y_train.value_counts()\n",
    "vc = vc/vc.iloc[1]\n",
    "vc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifiers = {\n",
    "    'logistic regression' : pipe_lr,\n",
    "    'decision tree' : dt,\n",
    "    'random forest' : rf,\n",
    "    'XGBoost' : XGBClassifier(scale_pos_weight=500, random_state=999), \n",
    "    'LightGBM' : LGBMClassifier(class_weight='balanced', random_state=999),\n",
    "    'CatBoost' : CatBoostClassifier(auto_class_weights='Balanced', verbose=0, random_state=999)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logistic regression\n",
      "decision tree\n",
      "random forest\n",
      "XGBoost\n",
      "LightGBM\n",
      "CatBoost\n"
     ]
    }
   ],
   "source": [
    "mean_results_dict = dict()\n",
    "std_results_dict = dict()\n",
    "for name, classifier in classifiers.items():\n",
    "    print(name)\n",
    "    result = pd.DataFrame(cross_validate(classifier, X_train, y_train, scoring=score_method, return_train_score=True))\n",
    "    mean_results_dict[name] = result.mean()\n",
    "    std_results_dict[name] = result.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = pd.DataFrame(mean_results_dict).T\n",
    "std_results  = pd.DataFrame(std_results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = results.assign(test_score_std=std_results[\"test_score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "      <th>test_score_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>XGBoost</th>\n",
       "      <td>0.926108</td>\n",
       "      <td>0.008277</td>\n",
       "      <td>0.835194</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.092386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>9.252291</td>\n",
       "      <td>0.006278</td>\n",
       "      <td>0.789193</td>\n",
       "      <td>0.999145</td>\n",
       "      <td>0.071582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>2.289599</td>\n",
       "      <td>0.027257</td>\n",
       "      <td>0.788906</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.087181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.491935</td>\n",
       "      <td>0.014840</td>\n",
       "      <td>0.788775</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.074766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.237156</td>\n",
       "      <td>0.005316</td>\n",
       "      <td>0.746118</td>\n",
       "      <td>0.820779</td>\n",
       "      <td>0.153621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.183530</td>\n",
       "      <td>0.003058</td>\n",
       "      <td>0.518204</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.085817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     fit_time  score_time  test_score  train_score  \\\n",
       "XGBoost              0.926108    0.008277    0.835194     1.000000   \n",
       "CatBoost             9.252291    0.006278    0.789193     0.999145   \n",
       "random forest        2.289599    0.027257    0.788906     1.000000   \n",
       "LightGBM             0.491935    0.014840    0.788775     1.000000   \n",
       "logistic regression  0.237156    0.005316    0.746118     0.820779   \n",
       "decision tree        0.183530    0.003058    0.518204     1.000000   \n",
       "\n",
       "                     test_score_std  \n",
       "XGBoost                    0.092386  \n",
       "CatBoost                   0.071582  \n",
       "random forest              0.087181  \n",
       "LightGBM                   0.074766  \n",
       "logistic regression        0.153621  \n",
       "decision tree              0.085817  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.sort_values(by=[\"test_score\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My thoughts:\n",
    "\n",
    "- Keep in mind all of the above is with:\n",
    "  - Default hyperparameters (except `class_weight`)\n",
    "  - A particular scoring metric\n",
    "- Look at the std for test score (across 5 folds) - it's very high!\n",
    "  - So we can't exactly trust the ordering above, or even the scores that much.\n",
    "  - Stratified splits might help here? To investigate.\n",
    "- All the trees totally overfit\n",
    "  - We may do better by tweaking the hyperparameters\n",
    "- Look at the fit times! And the score times are also interesting.\n",
    "  - Is is typical that `predict`/`score` are much faster than `fit` (with one notable exception, coming later in the course)\n",
    "- Note that CatBoost took about 10x longer than XGBoost and the scores only differ by <1%\n",
    "  - We took the mean of the sub-scores, but we might want to look more closely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What classifier should I use?!\n",
    "\n",
    "- Cop out answer: whichever gets the highest CV score.\n",
    "  - But we shouldn't overuse the validation set.\n",
    "  - This seems more reasonable when you have more data.\n",
    "  - It's time-consuming.\n",
    "- You may also have a reason to choose one over another.\n",
    "- One reason is _interpretability_, which we'll talk a bit more about next week.\n",
    "  - This is an area of growing interest and concern in ML.\n",
    "- Other considerations could be speed (fit and/or predict), maintainability of the code.\n",
    "- Finally, you could use all of them! (That's what the rest of today is about!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)\n",
    "\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensembles: averaging (20 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier we looked at a bunch of classifiers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logistic regression': Pipeline(steps=[('scale', StandardScaler()),\n",
       "                 ('lr',\n",
       "                  LogisticRegression(class_weight='balanced', max_iter=1000))]),\n",
       " 'decision tree': DecisionTreeClassifier(class_weight='balanced', random_state=123),\n",
       " 'random forest': RandomForestClassifier(class_weight='balanced', random_state=999),\n",
       " 'XGBoost': XGBClassifier(base_score=None, booster=None, colsample_bylevel=None,\n",
       "               colsample_bynode=None, colsample_bytree=None, gamma=None,\n",
       "               gpu_id=None, importance_type='gain', interaction_constraints=None,\n",
       "               learning_rate=None, max_delta_step=None, max_depth=None,\n",
       "               min_child_weight=None, missing=nan, monotone_constraints=None,\n",
       "               n_estimators=100, n_jobs=None, num_parallel_tree=None,\n",
       "               random_state=999, reg_alpha=None, reg_lambda=None,\n",
       "               scale_pos_weight=500, subsample=None, tree_method=None,\n",
       "               validate_parameters=None, verbosity=None),\n",
       " 'LightGBM': LGBMClassifier(class_weight='balanced', random_state=999),\n",
       " 'CatBoost': <catboost.core.CatBoostClassifier at 0x7fbceb0a00a0>}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "averaging_model = VotingClassifier(list(classifiers.items()), voting='soft') # need the list() here for cross_val to work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This `VotingClassifier` will take a _vote_ using the predictions of the constituent classifiers.\n",
    "  - Or, more generally, classifier pipelines!\n",
    "- Note the `voting='soft'`\n",
    "  - By default (`voting='hard'`) it uses the output of `predict` and actually votes.\n",
    "  - with `voting='soft'` it averages the output of `predict_proba` and then thresholds / takes the larger.\n",
    "  - The choice depends on whether you trust `predict_proba` from your base classifiers - if so, it's nice to access that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "averaging_model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: it seems sklearn requires us to actually call `fit` on the `VotingClassifier`, instead of passing in pre-fit models. This is an implementation choice rather than a conceptual limitation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a particular test case that is actually a fraud case (y=1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = np.where(y_test)[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaging_model.predict(X_test.iloc[t:t+1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For hard voting, these are the votes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logistic regression': 1,\n",
       " 'decision tree': 0,\n",
       " 'random forest': 1,\n",
       " 'XGBoost': 1,\n",
       " 'LightGBM': 1,\n",
       " 'CatBoost': 1}"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1 = {name : classifier.predict(X_test[t:t+1])[0] for name, classifier in averaging_model.named_estimators_.items()}\n",
    "r1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For soft voting, these are the scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBoost</th>\n",
       "      <td>0.000133</td>\n",
       "      <td>0.999867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.000046</td>\n",
       "      <td>0.999954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>0.001123</td>\n",
       "      <td>0.998877</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            0         1\n",
       "logistic regression  0.000000  1.000000\n",
       "decision tree        1.000000  0.000000\n",
       "random forest        0.400000  0.600000\n",
       "XGBoost              0.000133  0.999867\n",
       "LightGBM             0.000046  0.999954\n",
       "CatBoost             0.001123  0.998877"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2 = {name : classifier.predict_proba(X_test[t:t+1])[0] for name, classifier in averaging_model.named_estimators_.items()}\n",
    "pd.DataFrame(r2).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.23355\n",
       "1    0.76645\n",
       "dtype: float64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(r2).T.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.23355042, 0.76644958]])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaging_model.predict_proba(X_test.iloc[t:t+1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Aside: the probability scores from `DecisionTreeClassifier` are pretty bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how well this model performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "vot_res = pd.DataFrame(cross_validate(averaging_model, X_train, y_train, scoring=score_method, return_train_score=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "vot_res_mean = vot_res.mean()\n",
    "vot_res_std = vot_res.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "vot_res_df = pd.DataFrame(vot_res_mean, columns=[\"Voting\"]).T\n",
    "vot_res_df = vot_res_df.assign(test_score_std=[vot_res_std[\"test_score\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "      <th>test_score_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>XGBoost</th>\n",
       "      <td>0.926108</td>\n",
       "      <td>0.008277</td>\n",
       "      <td>0.835194</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.092386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>9.252291</td>\n",
       "      <td>0.006278</td>\n",
       "      <td>0.789193</td>\n",
       "      <td>0.999145</td>\n",
       "      <td>0.071582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>2.289599</td>\n",
       "      <td>0.027257</td>\n",
       "      <td>0.788906</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.087181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.491935</td>\n",
       "      <td>0.014840</td>\n",
       "      <td>0.788775</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.074766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting</th>\n",
       "      <td>10.028424</td>\n",
       "      <td>0.065686</td>\n",
       "      <td>0.786425</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.093385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.237156</td>\n",
       "      <td>0.005316</td>\n",
       "      <td>0.746118</td>\n",
       "      <td>0.820779</td>\n",
       "      <td>0.153621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.183530</td>\n",
       "      <td>0.003058</td>\n",
       "      <td>0.518204</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.085817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      fit_time  score_time  test_score  train_score  \\\n",
       "XGBoost               0.926108    0.008277    0.835194     1.000000   \n",
       "CatBoost              9.252291    0.006278    0.789193     0.999145   \n",
       "random forest         2.289599    0.027257    0.788906     1.000000   \n",
       "LightGBM              0.491935    0.014840    0.788775     1.000000   \n",
       "Voting               10.028424    0.065686    0.786425     1.000000   \n",
       "logistic regression   0.237156    0.005316    0.746118     0.820779   \n",
       "decision tree         0.183530    0.003058    0.518204     1.000000   \n",
       "\n",
       "                     test_score_std  \n",
       "XGBoost                    0.092386  \n",
       "CatBoost                   0.071582  \n",
       "random forest              0.087181  \n",
       "LightGBM                   0.074766  \n",
       "Voting                     0.093385  \n",
       "logistic regression        0.153621  \n",
       "decision tree              0.085817  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_with_avg = pd.concat((results, vot_res_df))\n",
    "results_with_avg.sort_values(by=[\"test_score\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that we did a bit better than even our best classifier!\n",
    "\n",
    "TODO: jk, this was true in a previous random seed but not now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Question: how could the average do better than the best model???\n",
    "  - From the perspective of the best estimator (in this case CatBoost), why are you adding on worse estimators??\n",
    "  \n",
    "<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how this can work:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Example | log reg    | rand for     | cat boost    | Averaged model |\n",
    "|--------|--------|--------|---------|---------------|\n",
    "|  1     | ✅    |   ✅    | ❌     | ✅✅❌=>✅  |\n",
    "|  2     | ✅    |   ❌    | ✅     | ✅❌✅=>✅  |\n",
    "|  3     | ❌    |   ✅    | ✅     | ❌✅✅=>✅  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In short, as long as the different models make different mistakes, this can work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is amazing, why not always do this?\n",
    "\n",
    "1. `fit`/`predict` time.\n",
    "2. Reduction in interpretability.\n",
    "3. Reduction in code maintainability (e.g. Netflix prize)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back to ensembling: when can I do this?\n",
    "\n",
    "- You can comebine completely different estimators, or similar estimators.\n",
    "  - (In fact, a random forest itself is an ensemble of decision trees.)\n",
    "- But you can also try different hyperparameter values.\n",
    "  - So... the hyperparameters of the ensemble now go even deeper.\n",
    "  - The inception never ends... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensembles: stacking (20 min)\n",
    "\n",
    "- Another type of ensemble is stacking.\n",
    "- Instead of averaging the outputs of each estimator, instead use their outputs as _inputs to another model_.\n",
    "- By default for classification, it uses logistic regression.\n",
    "  - We don't need a complex model here necessarily, more of a weighted average.\n",
    "  - The features going into the logistic regression are the classifier outputs, _not_ the original features!\n",
    "  - So the number of coefficients = the number of base estimators!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import StackingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code starts to get too slow here; I'm going to remove CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifiers_nocat = classifiers.copy()\n",
    "del classifiers_nocat[\"CatBoost\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_model = StackingClassifier(list(classifiers_nocat.items()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's going on in here? \n",
    "\n",
    "- It is doing cross-validation by itself by default (see [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html))\n",
    "  - It is fitting the base estimators on the training fold\n",
    "  - And the predicting on the validation fold\n",
    "  - And then fitting the meta-estimator on that output (on the validation fold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.predict(X_test[t:t+1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.28045983, 0.71954017]])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.predict_proba(X_test[t:t+1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(This is the `predict_proba` from logistic regression)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how well this model performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       16.328395\n",
       "score_time      0.053243\n",
       "test_score      0.783362\n",
       "train_score     1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(stacking_model, X_train, y_train, scoring=score_method, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The situation here is a bit mind-boggling.\n",
    "- AFAIK on each fold of cross-validation it is doing cross-validation.\n",
    "- And then in there you might have pipelines and...\n",
    "- This is really loops within loops within loops within loops..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the input features (X) to the meta-model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBoost</th>\n",
       "      <td>0.000133</td>\n",
       "      <td>0.999867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.000046</td>\n",
       "      <td>0.999954</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            0         1\n",
       "logistic regression  0.000000  1.000000\n",
       "decision tree        1.000000  0.000000\n",
       "random forest        0.400000  0.600000\n",
       "XGBoost              0.000133  0.999867\n",
       "LightGBM             0.000046  0.999954"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r3 = {name : classifier.predict_proba(X_test[t:t+1])[0] for name, classifier in stacking_model.named_estimators_.items()}\n",
    "r3 = pd.DataFrame(r3).T\n",
    "r3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Because this needs to work for multi-class now I'm not quite sure if both probabilities are going in or just one - I think just one - this is a bit simpler for regression!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- If the meta-model is logistic regression (which it is by default), you are taking a weighted average of these outputs and learning the weights from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.59483414, 1.300195  , 1.74174005, 2.06847608, 2.87194886]])"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.final_estimator_.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>2.594834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>1.300195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>1.741740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBoost</th>\n",
       "      <td>2.068476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>2.871949</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Coefficient\n",
       "logistic regression     2.594834\n",
       "decision tree           1.300195\n",
       "random forest           1.741740\n",
       "XGBoost                 2.068476\n",
       "LightGBM                2.871949"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data=stacking_model.final_estimator_.coef_[0], index=classifiers_nocat.keys(), columns=[\"Coefficient\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "#??StackingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It seems that the XGBoost is being trusted the most / counting for most.\n",
    "- We can also try a different final estimator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_model_tree = StackingClassifier(list(classifiers_nocat.items()), \n",
    "                 final_estimator=DecisionTreeClassifier(max_depth=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pd.DataFrame(cross_validate(stacking_model_tree, X_train, y_train, scoring=score_method, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not very good. But we can look at the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_model_tree.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"716pt\" height=\"260pt\"\n",
       " viewBox=\"0.00 0.00 716.00 260.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 256)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-256 712,-256 712,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"438.5,-252 289.5,-252 289.5,-216 438.5,-216 438.5,-252\"/>\n",
       "<text text-anchor=\"middle\" x=\"364\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">random forest &lt;= 0.185</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"352.5,-180 221.5,-180 221.5,-144 352.5,-144 352.5,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"287\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">XGBoost &lt;= 0.0001</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M344.5696,-215.8314C335.268,-207.1337 323.9796,-196.5783 313.8714,-187.1265\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"316.1189,-184.4363 306.4241,-180.1628 311.3379,-189.5493 316.1189,-184.4363\"/>\n",
       "<text text-anchor=\"middle\" x=\"307.2626\" y=\"-201.4488\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">True</text>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"511,-180 371,-180 371,-144 511,-144 511,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"441\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">LightGBM &lt;= 0.1604</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>0&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M383.4304,-215.8314C392.732,-207.1337 404.0204,-196.5783 414.1286,-187.1265\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"416.6621,-189.5493 421.5759,-180.1628 411.8811,-184.4363 416.6621,-189.5493\"/>\n",
       "<text text-anchor=\"middle\" x=\"420.7374\" y=\"-201.4488\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"203,-108 23,-108 23,-72 203,-72 203,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"113\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">logistic regression &lt;= 0.0002</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M243.0925,-143.8314C219.7176,-134.159 190.7903,-122.1891 166.1532,-111.9944\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"167.1702,-108.6275 156.5918,-108.038 164.4937,-115.0956 167.1702,-108.6275\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"352.5,-108 221.5,-108 221.5,-72 352.5,-72 352.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"287\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">XGBoost &lt;= 0.0001</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M287,-143.8314C287,-136.131 287,-126.9743 287,-118.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"290.5001,-118.4132 287,-108.4133 283.5001,-118.4133 290.5001,-118.4132\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"66,-36 0,-36 0,0 66,0 66,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"33\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M92.8126,-71.8314C83.1486,-63.1337 71.4204,-52.5783 60.9184,-43.1265\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"62.9553,-40.251 53.1809,-36.1628 58.2725,-45.454 62.9553,-40.251\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"150,-36 84,-36 84,0 150,0 150,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M114.0094,-71.8314C114.4372,-64.131 114.9459,-54.9743 115.4213,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"118.9169,-46.592 115.977,-36.4133 111.9276,-46.2037 118.9169,-46.592\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"265,-36 199,-36 199,0 265,0 265,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"232\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M273.1212,-71.8314C266.7864,-63.5386 259.1616,-53.557 252.2074,-44.4533\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"254.9175,-42.2353 246.0657,-36.4133 249.3548,-46.4847 254.9175,-42.2353\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"349,-36 283,-36 283,0 349,0 349,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"316\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M294.3179,-71.8314C297.4876,-63.9617 301.27,-54.5712 304.7813,-45.8533\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"308.0939,-46.9968 308.5835,-36.4133 301.6008,-44.3815 308.0939,-46.9968\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"511,-108 371,-108 371,-72 511,-72 511,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"441\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">LightGBM &lt;= 0.0007</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M441,-143.8314C441,-136.131 441,-126.9743 441,-118.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"444.5001,-118.4132 441,-108.4133 437.5001,-118.4133 444.5001,-118.4132\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"660.5,-108 529.5,-108 529.5,-72 660.5,-72 660.5,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"595\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">XGBoost &lt;= 0.9997</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>8&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M479.8607,-143.8314C500.2742,-134.2874 525.4723,-122.5065 547.086,-112.4013\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"548.8423,-115.4439 556.4187,-108.038 545.8775,-109.1027 548.8423,-115.4439\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"448,-36 382,-36 382,0 448,0 448,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"415\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M434.4391,-71.8314C431.5973,-63.9617 428.2063,-54.5712 425.0581,-45.8533\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"428.3377,-44.63 421.6492,-36.4133 421.7538,-47.0076 428.3377,-44.63\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"532,-36 466,-36 466,0 532,0 532,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"499\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 0</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>9&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M455.6358,-71.8314C462.3161,-63.5386 470.3568,-53.557 477.6904,-44.4533\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"480.6194,-46.3965 484.1671,-36.4133 475.1681,-42.0052 480.6194,-46.3965\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"624,-36 558,-36 558,0 624,0 624,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"591\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M593.9906,-71.8314C593.5628,-64.131 593.0541,-54.9743 592.5787,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"596.0724,-46.2037 592.023,-36.4133 589.0831,-46.592 596.0724,-46.2037\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"708,-36 642,-36 642,0 708,0 708,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"675\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">class = 1</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>12&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M615.1874,-71.8314C624.8514,-63.1337 636.5796,-52.5783 647.0816,-43.1265\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"649.7275,-45.454 654.8191,-36.1628 645.0447,-40.251 649.7275,-45.454\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbcee43d460>"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(list(classifiers_nocat.keys()), stacking_model_tree.final_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is so cool!! (Well,  think so)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### An effective strategy\n",
    "\n",
    "- Randomly generate a bunch of models with different hyperparameter configurations, and then stack all the models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary \n",
    "\n",
    "- There's a big world of classifiers out there.\n",
    "  - Tree-based classifiers are particularly popular and effective on a wide range of problems. \n",
    "  - These can be much more complex than LogisticRegression, which only learns one coefficient per feature.\n",
    "- We sometimes need to leave the world of scikit-learn (and more of that later in the course!).\n",
    "- Ensembles are usually pretty effective.\n",
    "  - But they trade off code complexity and code speed for prediction accuracy.\n",
    "  - Don't forget that hyperparameter optimization multiplies the slowness of the code!\n",
    "- Stacking is a bit slower than voting, but generally more accuracy.\n",
    "  - As a bonus, you get to see the coefficients for each base classifier.\n",
    "- Remember, everything we've done is subject to earlier warnings:\n",
    "  - Check for small datasets\n",
    "  - Check the CV folds\n",
    "  - Check the test set\n",
    "  - Think carefully about which scoring metric is suitable for your problem"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}