lectures/07_missingness-scaling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CPSC 330 Lecture 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Lecture plan\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements\n",
    "- Missing data: abridged version (10 min)\n",
    "- Feature scaling (25 min)\n",
    "- Break (5 min)\n",
    "- Putting it all together with `ColumnTransformer` (30 min)\n",
    "- Hyperparameter search, revisited (5 min)\n",
    "- Summary (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements\n",
    "\n",
    "- New office hours starting this week, see [calendar](https://htmlpreview.github.io/?https://github.com/UBC-CS/cpsc330/blob/master/docs/calendar.html).\n",
    "- I'll try to release hw4 earlier than Tuesday (I have some time later today to work on it)\n",
    "- Will try Live Q&A one more time (today) and then will send out a poll to get your feedback.\n",
    "  - Reminder - turn on Live Q&A\n",
    "- New plot in [Tuesday's lecture notebook](https://github.com/UBC-CS/cpsc330/blob/master/lectures/06_optimization-bias-categoricals.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Use `SimpleImputer` to impute values where data is missing.\n",
    "- Explain the motivation for feature scaling.\n",
    "- Compare standardization vs. normalization.\n",
    "  - (but we won't really discuss much about the pros and cons of each).\n",
    "- Identify which preprocessing steps apply to which types of data (e.g. `StandardScaler` is for numeric features)\n",
    "- Use `ColumnTransformer` for more complex pipelines.\n",
    "- Use `GridSearchCV` and `RandomizedSearchCV` on nested pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['font.size'] = 16\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_validate, GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn import set_config, config_context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with missing data: abridged version (10 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we'll continue with the census data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "census = pd.read_csv('data/adult.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed last time, we'll drop the `education` column because it's already been ordinally encoded in `education.num`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "census = census.drop(columns=[\"education\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "census_train, census_test = train_test_split(census, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>native.country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>90</td>\n",
       "      <td>?</td>\n",
       "      <td>77053</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>?</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>82</td>\n",
       "      <td>Private</td>\n",
       "      <td>132870</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>18</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>66</td>\n",
       "      <td>?</td>\n",
       "      <td>186061</td>\n",
       "      <td>10</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>?</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54</td>\n",
       "      <td>Private</td>\n",
       "      <td>140359</td>\n",
       "      <td>4</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>Private</td>\n",
       "      <td>264663</td>\n",
       "      <td>10</td>\n",
       "      <td>Separated</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age workclass  fnlwgt  education.num marital.status         occupation  \\\n",
       "0   90         ?   77053              9        Widowed                  ?   \n",
       "1   82   Private  132870              9        Widowed    Exec-managerial   \n",
       "2   66         ?  186061             10        Widowed                  ?   \n",
       "3   54   Private  140359              4       Divorced  Machine-op-inspct   \n",
       "4   41   Private  264663             10      Separated     Prof-specialty   \n",
       "\n",
       "    relationship   race     sex  capital.gain  capital.loss  hours.per.week  \\\n",
       "0  Not-in-family  White  Female             0          4356              40   \n",
       "1  Not-in-family  White  Female             0          4356              18   \n",
       "2      Unmarried  Black  Female             0          4356              40   \n",
       "3      Unmarried  White  Female             0          3900              40   \n",
       "4      Own-child  White  Female             0          3900              40   \n",
       "\n",
       "  native.country income  \n",
       "0  United-States  <=50K  \n",
       "1  United-States  <=50K  \n",
       "2  United-States  <=50K  \n",
       "3  United-States  <=50K  \n",
       "4  United-States  <=50K  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "census_train.sort_index().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We can see we have a bunch of missing values, where presumably the person did not answer that question on the census.\n",
    "- Interestingly, these were not picked up: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 26048 entries, 17064 to 19966\n",
      "Data columns (total 14 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   age             26048 non-null  int64 \n",
      " 1   workclass       26048 non-null  object\n",
      " 2   fnlwgt          26048 non-null  int64 \n",
      " 3   education.num   26048 non-null  int64 \n",
      " 4   marital.status  26048 non-null  object\n",
      " 5   occupation      26048 non-null  object\n",
      " 6   relationship    26048 non-null  object\n",
      " 7   race            26048 non-null  object\n",
      " 8   sex             26048 non-null  object\n",
      " 9   capital.gain    26048 non-null  int64 \n",
      " 10  capital.loss    26048 non-null  int64 \n",
      " 11  hours.per.week  26048 non-null  int64 \n",
      " 12  native.country  26048 non-null  object\n",
      " 13  income          26048 non-null  object\n",
      "dtypes: int64(6), object(8)\n",
      "memory usage: 3.0+ MB\n"
     ]
    }
   ],
   "source": [
    "census_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Everything is non-null because the missing values were encoded as the string \"?\" instead of an actual NaN in Python.\n",
    "- We saw those last class, where \"?\" was a category generated by OHE.\n",
    "- Let's change them to actual nulls:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>native.country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>77053</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>82</td>\n",
       "      <td>Private</td>\n",
       "      <td>132870</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>18</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>66</td>\n",
       "      <td>NaN</td>\n",
       "      <td>186061</td>\n",
       "      <td>10</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54</td>\n",
       "      <td>Private</td>\n",
       "      <td>140359</td>\n",
       "      <td>4</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>Private</td>\n",
       "      <td>264663</td>\n",
       "      <td>10</td>\n",
       "      <td>Separated</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age workclass  fnlwgt  education.num marital.status         occupation  \\\n",
       "0   90       NaN   77053              9        Widowed                NaN   \n",
       "1   82   Private  132870              9        Widowed    Exec-managerial   \n",
       "2   66       NaN  186061             10        Widowed                NaN   \n",
       "3   54   Private  140359              4       Divorced  Machine-op-inspct   \n",
       "4   41   Private  264663             10      Separated     Prof-specialty   \n",
       "\n",
       "    relationship   race     sex  capital.gain  capital.loss  hours.per.week  \\\n",
       "0  Not-in-family  White  Female             0          4356              40   \n",
       "1  Not-in-family  White  Female             0          4356              18   \n",
       "2      Unmarried  Black  Female             0          4356              40   \n",
       "3      Unmarried  White  Female             0          3900              40   \n",
       "4      Own-child  White  Female             0          3900              40   \n",
       "\n",
       "  native.country income  \n",
       "0  United-States  <=50K  \n",
       "1  United-States  <=50K  \n",
       "2  United-States  <=50K  \n",
       "3  United-States  <=50K  \n",
       "4  United-States  <=50K  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_nan = census_train.replace('?', np.NaN)\n",
    "df_test_nan  = census_test.replace( '?', np.NaN)\n",
    "\n",
    "df_train_nan.sort_index().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 26048 entries, 17064 to 19966\n",
      "Data columns (total 14 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   age             26048 non-null  int64 \n",
      " 1   workclass       24600 non-null  object\n",
      " 2   fnlwgt          26048 non-null  int64 \n",
      " 3   education.num   26048 non-null  int64 \n",
      " 4   marital.status  26048 non-null  object\n",
      " 5   occupation      24595 non-null  object\n",
      " 6   relationship    26048 non-null  object\n",
      " 7   race            26048 non-null  object\n",
      " 8   sex             26048 non-null  object\n",
      " 9   capital.gain    26048 non-null  int64 \n",
      " 10  capital.loss    26048 non-null  int64 \n",
      " 11  hours.per.week  26048 non-null  int64 \n",
      " 12  native.country  25573 non-null  object\n",
      " 13  income          26048 non-null  object\n",
      "dtypes: int64(6), object(8)\n",
      "memory usage: 3.0+ MB\n"
     ]
    }
   ],
   "source": [
    "df_train_nan.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now we can see the null values, and likely these would be picked out by pandas profiler.\n",
    "  - Note: we'll address null values in the features, not in the targets.\n",
    "- So, how should we address these?\n",
    "- Disclaimer: we will only cover this in a super simplistic way.\n",
    "- See STAT courses for a proper treatment of this topic!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gotta drop 'em all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_nan = df_train_nan.drop(columns=['income'])\n",
    "X_test_nan  = df_test_nan.drop(columns=['income'])\n",
    "y_train = df_train_nan['income']\n",
    "y_test = df_test_nan['income']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26048, 13)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(24144, 13)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.dropna(axis=0).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- So, we dropped about 2000 rows.\n",
    "- We'd need to do the same in our test set.\n",
    "- But what if we get a missing value in deployment?\n",
    "- And furthermore, what if the missing values don't occur at random and we're systematically dropping certain data?\n",
    "- This is not a great solution, especially if there's a lot of missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26048, 10)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.dropna(axis=1).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- One can also drop all _columns_ with missing values using `axis=1`. \n",
    "- This generally throws away a lot of information, because you lose a whole column just for 1 missing value.\n",
    "- But I might drop a column if it's 99.9% missing values, for example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imputation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Imputation means inventing values for the missing data.\n",
    "- The strategies are different for numeric vs. categorical.\n",
    "- In this dataset it turns out we only have missing values in the categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "imp = SimpleImputer(strategy='most_frequent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This imputer is another transformer, like the other ones we've seen (`CountVectorizer`, `OrdinalEncoder`, `OneHotEncoder`).\n",
    "- The \"most_frequent\" strategy puts in the most frequent value seen in that column.\n",
    "- There are also strategies for numeric variables, like taking the mean or median value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_features = ['age', 'fnlwgt', 'education.num', 'capital.gain', \n",
    "                    'capital.loss', 'hours.per.week']\n",
    "categorical_features = ['workclass', 'marital.status', 'occupation', \n",
    "                        'relationship', 'race', 'sex', 'native.country']\n",
    "target_column = 'income'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "imp.fit(X_train_nan[categorical_features]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_imp_cat = pd.DataFrame(imp.transform(X_train_nan[categorical_features]),\n",
    "                           columns=categorical_features, index=X_train_nan.index)\n",
    "X_test_imp_cat = pd.DataFrame(imp.transform(X_test_nan[categorical_features]),\n",
    "                           columns=categorical_features, index=X_test_nan.index)\n",
    "\n",
    "X_train_imp = X_train_nan.copy()\n",
    "X_train_imp.update(X_train_imp_cat)\n",
    "\n",
    "X_test_imp = X_test_nan.copy()\n",
    "X_test_imp.update(X_test_imp_cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the missing values filled in. Before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>native.country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>77053</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>82</td>\n",
       "      <td>Private</td>\n",
       "      <td>132870</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>18</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>66</td>\n",
       "      <td>NaN</td>\n",
       "      <td>186061</td>\n",
       "      <td>10</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54</td>\n",
       "      <td>Private</td>\n",
       "      <td>140359</td>\n",
       "      <td>4</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>Private</td>\n",
       "      <td>264663</td>\n",
       "      <td>10</td>\n",
       "      <td>Separated</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age workclass  fnlwgt  education.num marital.status         occupation  \\\n",
       "0   90       NaN   77053              9        Widowed                NaN   \n",
       "1   82   Private  132870              9        Widowed    Exec-managerial   \n",
       "2   66       NaN  186061             10        Widowed                NaN   \n",
       "3   54   Private  140359              4       Divorced  Machine-op-inspct   \n",
       "4   41   Private  264663             10      Separated     Prof-specialty   \n",
       "\n",
       "    relationship   race     sex  capital.gain  capital.loss  hours.per.week  \\\n",
       "0  Not-in-family  White  Female             0          4356              40   \n",
       "1  Not-in-family  White  Female             0          4356              18   \n",
       "2      Unmarried  Black  Female             0          4356              40   \n",
       "3      Unmarried  White  Female             0          3900              40   \n",
       "4      Own-child  White  Female             0          3900              40   \n",
       "\n",
       "  native.country  \n",
       "0  United-States  \n",
       "1  United-States  \n",
       "2  United-States  \n",
       "3  United-States  \n",
       "4  United-States  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.sort_index().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>native.country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>90</td>\n",
       "      <td>Private</td>\n",
       "      <td>77053</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>82</td>\n",
       "      <td>Private</td>\n",
       "      <td>132870</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>18</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>66</td>\n",
       "      <td>Private</td>\n",
       "      <td>186061</td>\n",
       "      <td>10</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>4356</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54</td>\n",
       "      <td>Private</td>\n",
       "      <td>140359</td>\n",
       "      <td>4</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>Private</td>\n",
       "      <td>264663</td>\n",
       "      <td>10</td>\n",
       "      <td>Separated</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3900</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age workclass  fnlwgt  education.num marital.status         occupation  \\\n",
       "0   90   Private   77053              9        Widowed     Prof-specialty   \n",
       "1   82   Private  132870              9        Widowed    Exec-managerial   \n",
       "2   66   Private  186061             10        Widowed     Prof-specialty   \n",
       "3   54   Private  140359              4       Divorced  Machine-op-inspct   \n",
       "4   41   Private  264663             10      Separated     Prof-specialty   \n",
       "\n",
       "    relationship   race     sex  capital.gain  capital.loss  hours.per.week  \\\n",
       "0  Not-in-family  White  Female             0          4356              40   \n",
       "1  Not-in-family  White  Female             0          4356              18   \n",
       "2      Unmarried  Black  Female             0          4356              40   \n",
       "3      Unmarried  White  Female             0          3900              40   \n",
       "4      Own-child  White  Female             0          3900              40   \n",
       "\n",
       "  native.country  \n",
       "0  United-States  \n",
       "1  United-States  \n",
       "2  United-States  \n",
       "3  United-States  \n",
       "4  United-States  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_imp.sort_index().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't go into any detail about methods of imputation, but you can consider the different approaches as hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fill value\n",
    "\n",
    "- By the way, another option would be to just leave in the \"?\" and have this be its own category for categorical variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "SimpleImputer(strategy='constant', fill_value=\"?\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- I won't get too deep into this in this course. \n",
    "- We can just say what we always say - treat it as a hyperparameter unless you have a better idea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pipeline\n",
    "\n",
    "Let's build a Pipeline with what we have so far for categorical features only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([('imputation', SimpleImputer(strategy='most_frequent')),\n",
    "                 ('ohe', OneHotEncoder(handle_unknown='ignore')),\n",
    "                 ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now we have a Pipeline with 3 stages: 2 transformers followed by a classifier.\n",
    "- Now we can go back to that image from Lecture 5 and it's more appropriate:\n",
    "\n",
    "<img src=\"img/pipeline.png\" width=\"700\">\n",
    "\n",
    "[Source](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#18)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can run the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.994029</td>\n",
       "      <td>0.016973</td>\n",
       "      <td>0.813244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.954693</td>\n",
       "      <td>0.015995</td>\n",
       "      <td>0.808829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.972394</td>\n",
       "      <td>0.017054</td>\n",
       "      <td>0.805950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.962992</td>\n",
       "      <td>0.016949</td>\n",
       "      <td>0.819159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.940569</td>\n",
       "      <td>0.016533</td>\n",
       "      <td>0.815512</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score\n",
       "0  0.994029    0.016973    0.813244\n",
       "1  0.954693    0.015995    0.808829\n",
       "2  0.972394    0.017054    0.805950\n",
       "3  0.962992    0.016949    0.819159\n",
       "4  0.940569    0.016533    0.815512"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(pipe, X_train_nan[categorical_features], y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Great, so this all works, but we only used the categorical features.\n",
    "- Later today we'll see how to combine everything nicely with `ColumnTransformer`.\n",
    "- But first, one more thing: preprocessing of the numeric variables!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature scaling (25 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the numeric features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>20</td>\n",
       "      <td>110998</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>22</td>\n",
       "      <td>263670</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>51</td>\n",
       "      <td>335997</td>\n",
       "      <td>9</td>\n",
       "      <td>4386</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>53</td>\n",
       "      <td>111939</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>52</td>\n",
       "      <td>51048</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28636</th>\n",
       "      <td>48</td>\n",
       "      <td>70668</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17730</th>\n",
       "      <td>35</td>\n",
       "      <td>340018</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28030</th>\n",
       "      <td>26</td>\n",
       "      <td>373553</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15725</th>\n",
       "      <td>28</td>\n",
       "      <td>155621</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19966</th>\n",
       "      <td>40</td>\n",
       "      <td>151294</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26048 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  fnlwgt  education.num  capital.gain  capital.loss  hours.per.week\n",
       "17064   20  110998             10             0             0              30\n",
       "18434   22  263670              9             0             0              80\n",
       "3294    51  335997              9          4386             0              55\n",
       "31317   53  111939             13             0             0              35\n",
       "4770    52   51048             13             0             0              55\n",
       "...    ...     ...            ...           ...           ...             ...\n",
       "28636   48   70668              9             0             0              50\n",
       "17730   35  340018              6             0             0              38\n",
       "28030   26  373553             10             0             0              42\n",
       "15725   28  155621              3             0             0              40\n",
       "19966   40  151294              9             0             0              48\n",
       "\n",
       "[26048 rows x 6 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_imp[numeric_features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a model using only these features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.149345</td>\n",
       "      <td>0.012755</td>\n",
       "      <td>0.793858</td>\n",
       "      <td>0.800557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.141597</td>\n",
       "      <td>0.012674</td>\n",
       "      <td>0.798848</td>\n",
       "      <td>0.800125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.114308</td>\n",
       "      <td>0.011775</td>\n",
       "      <td>0.800960</td>\n",
       "      <td>0.799261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.113999</td>\n",
       "      <td>0.012915</td>\n",
       "      <td>0.803993</td>\n",
       "      <td>0.798503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.128847</td>\n",
       "      <td>0.013455</td>\n",
       "      <td>0.798426</td>\n",
       "      <td>0.799079</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.149345    0.012755    0.793858     0.800557\n",
       "1  0.141597    0.012674    0.798848     0.800125\n",
       "2  0.114308    0.011775    0.800960     0.799261\n",
       "3  0.113999    0.012915    0.803993     0.798503\n",
       "4  0.128847    0.013455    0.798426     0.799079"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "pd.DataFrame(cross_validate(lr, X_train_imp[numeric_features], y_train, return_train_score=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so `DummyClassifier` gets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7605190417690417"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DummyClassifier(strategy='prior').fit(None, y_train).score(None, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- And here we do a few percent better.\n",
    "- But let's look at the coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.fit(X_train_imp[numeric_features], y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>-0.007233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fnlwgt</th>\n",
       "      <td>-0.000004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>education.num</th>\n",
       "      <td>-0.001697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital.gain</th>\n",
       "      <td>0.000337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital.loss</th>\n",
       "      <td>0.000785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hours.per.week</th>\n",
       "      <td>-0.007883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Coefficient\n",
       "age               -0.007233\n",
       "fnlwgt            -0.000004\n",
       "education.num     -0.001697\n",
       "capital.gain       0.000337\n",
       "capital.loss       0.000785\n",
       "hours.per.week    -0.007883"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data=lr.coef_[0], index=numeric_features, columns=['Coefficient'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What we see here is a very small coefficient for `fnlwgt` (description of this feature [here](https://www.kaggle.com/uciml/adult-census-income), I couldn't quite decipher it).\n",
    "- Why is this coefficient so small?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>26048.000000</td>\n",
       "      <td>2.604800e+04</td>\n",
       "      <td>26048.000000</td>\n",
       "      <td>26048.000000</td>\n",
       "      <td>26048.000000</td>\n",
       "      <td>26048.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>38.586686</td>\n",
       "      <td>1.892295e+05</td>\n",
       "      <td>10.070485</td>\n",
       "      <td>1075.695754</td>\n",
       "      <td>87.629991</td>\n",
       "      <td>40.433239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>13.619181</td>\n",
       "      <td>1.050005e+05</td>\n",
       "      <td>2.572231</td>\n",
       "      <td>7334.297499</td>\n",
       "      <td>404.192112</td>\n",
       "      <td>12.346313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>1.376900e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>1.175830e+05</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>40.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>37.000000</td>\n",
       "      <td>1.777850e+05</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>40.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>48.000000</td>\n",
       "      <td>2.368852e+05</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>90.000000</td>\n",
       "      <td>1.366120e+06</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>99999.000000</td>\n",
       "      <td>4356.000000</td>\n",
       "      <td>99.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age        fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "count  26048.000000  2.604800e+04   26048.000000  26048.000000  26048.000000   \n",
       "mean      38.586686  1.892295e+05      10.070485   1075.695754     87.629991   \n",
       "std       13.619181  1.050005e+05       2.572231   7334.297499    404.192112   \n",
       "min       17.000000  1.376900e+04       1.000000      0.000000      0.000000   \n",
       "25%       28.000000  1.175830e+05       9.000000      0.000000      0.000000   \n",
       "50%       37.000000  1.777850e+05      10.000000      0.000000      0.000000   \n",
       "75%       48.000000  2.368852e+05      12.000000      0.000000      0.000000   \n",
       "max       90.000000  1.366120e+06      16.000000  99999.000000   4356.000000   \n",
       "\n",
       "       hours.per.week  \n",
       "count    26048.000000  \n",
       "mean        40.433239  \n",
       "std         12.346313  \n",
       "min          1.000000  \n",
       "25%         40.000000  \n",
       "50%         40.000000  \n",
       "75%         45.000000  \n",
       "max         99.000000  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Answer: because the values are so big (avg = 200,000)\n",
    "- And what if these values happened to be even larger? Or what if capital gain/loss was measured in thousands of dollars?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_mod = X_train_imp[numeric_features].copy()\n",
    "X_train_mod[\"capital.gain\"] /= 1000\n",
    "X_train_mod[\"capital.loss\"] /= 1000\n",
    "X_train_mod[\"fnlwgt\"] *= 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>20</td>\n",
       "      <td>110998000</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>22</td>\n",
       "      <td>263670000</td>\n",
       "      <td>9</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>51</td>\n",
       "      <td>335997000</td>\n",
       "      <td>9</td>\n",
       "      <td>4.386</td>\n",
       "      <td>0.0</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>53</td>\n",
       "      <td>111939000</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>52</td>\n",
       "      <td>51048000</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age     fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064   20  110998000             10         0.000           0.0   \n",
       "18434   22  263670000              9         0.000           0.0   \n",
       "3294    51  335997000              9         4.386           0.0   \n",
       "31317   53  111939000             13         0.000           0.0   \n",
       "4770    52   51048000             13         0.000           0.0   \n",
       "\n",
       "       hours.per.week  \n",
       "17064              30  \n",
       "18434              80  \n",
       "3294               55  \n",
       "31317              35  \n",
       "4770               55  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_mod.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.061292\n",
       "score_time     0.012938\n",
       "test_score     0.760519\n",
       "train_score    0.760519\n",
       "dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "pd.DataFrame(cross_validate(lr, X_train_mod, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now our train & test scores went down to basically `DummyClassifier` level!\n",
    "- But what is up with that, these units are arbitrary to begin with!!\n",
    "- BTW, decision trees don't have this problem because they're only about thresholds, rather than crunching the actual number.\n",
    "  - [Great post on Piazza](https://piazza.com/class/kb2e6nwu3uj23?cid=256) pointing out something similar with the spacing in ordinal encodings!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7707694308794502"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier(random_state=1)\n",
    "cross_val_score(dt, X_train_imp[numeric_features], y_train).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7707694308794502"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier(random_state=1)\n",
    "cross_val_score(dt, X_train_mod, y_train).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But this problem affects plenty of ML methods.\n",
    "- So it would be nice to just take care of this issue.\n",
    "- The general approach is to rescale the features.\n",
    "- Two specific approaches we'll cover are standardization and normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Approach | What it does | How to update $X$ (but see below!) | sklearn implementation | \n",
    "|---------|------------|-----------------------|----------------|\n",
    "| normalization | sets range to $[0,1]$   | `X -= np.min(X,axis=0)`<br>`X /= np.max(X,axis=0)`  | [`MinMaxScaler()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)\n",
    "| standardization | sets sample mean to $0$, s.d. to $1$   | `X -= np.mean(X,axis=0)`<br>`X /=  np.std(X,axis=0)` | [`StandardScaler()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler) |\n",
    "\n",
    "There are all sorts of articles on this; see, e.g. [here](http://www.dataminingblog.com/standardization-vs-normalization/) and [here](https://medium.com/@rrfd/standardize-or-normalize-examples-in-python-e3f174b65dfc)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use these scaling methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train_imp[numeric_features]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.36476947, -0.74507317, -0.02740291, -0.14666932, -0.21680698,\n",
       "        -0.84506515],\n",
       "       [-1.21791495,  0.70896709, -0.41617802, -0.14666932, -0.21680698,\n",
       "         3.20480459],\n",
       "       [ 0.91147565,  1.39780572, -0.41617802,  0.45135445, -0.21680698,\n",
       "         1.17986972],\n",
       "       ...,\n",
       "       [-0.9242059 ,  1.75548713, -0.02740291, -0.14666932, -0.21680698,\n",
       "         0.12690359],\n",
       "       [-0.77735138, -0.32008602, -2.74882863, -0.14666932, -0.21680698,\n",
       "        -0.0350912 ],\n",
       "       [ 0.10377577, -0.36129614, -0.41617802, -0.14666932, -0.21680698,\n",
       "         0.61288796]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.transform(X_train_imp[numeric_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.364769</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.217915</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.204805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.397806</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.058330</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.316034</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064 -1.364769 -0.745073      -0.027403     -0.146669     -0.216807   \n",
       "18434 -1.217915  0.708967      -0.416178     -0.146669     -0.216807   \n",
       "3294   0.911476  1.397806      -0.416178      0.451354     -0.216807   \n",
       "31317  1.058330 -0.736111       1.138922     -0.146669     -0.216807   \n",
       "4770   0.984903 -1.316034       1.138922     -0.146669     -0.216807   \n",
       "\n",
       "       hours.per.week  \n",
       "17064       -0.845065  \n",
       "18434        3.204805  \n",
       "3294         1.179870  \n",
       "31317       -0.440078  \n",
       "4770         1.179870  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_train_df = pd.DataFrame(scaler.transform(X_train_imp[numeric_features]),\n",
    "                           columns=numeric_features, index=X_train_imp.index)\n",
    "scaled_train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the same Golden Rule issue we talked about before.\n",
    "  - We fit the transformer on the training data, and then transform both data sets.\n",
    "  - We need to use a Pipeline for cross-validation because the transformation of each row depends on the other rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_test_df = pd.DataFrame(scaler.transform(X_test_imp[numeric_features]),\n",
    "                           columns=numeric_features, index=X_test_imp.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that it did what we expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               3.634832e-16\n",
       "fnlwgt           -4.961863e-17\n",
       "education.num     1.371028e-15\n",
       "capital.gain     -3.863724e-16\n",
       "capital.loss      7.671122e-16\n",
       "hours.per.week    9.381657e-16\n",
       "dtype: float64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_train_df.mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are basically all zero ($10^{-16}$ is zero to numerical precision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               1.000019\n",
       "fnlwgt            1.000019\n",
       "education.num     1.000019\n",
       "capital.gain      1.000019\n",
       "capital.loss      1.000019\n",
       "hours.per.week    1.000019\n",
       "dtype: float64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_train_df.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for test we get something different - that is OK!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age              -0.001850\n",
       "fnlwgt            0.026132\n",
       "education.num     0.019814\n",
       "capital.gain      0.001331\n",
       "capital.loss     -0.004034\n",
       "hours.per.week    0.001708\n",
       "dtype: float64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_test_df.mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               1.007872\n",
       "fnlwgt            1.025728\n",
       "education.num     1.000891\n",
       "capital.gain      1.034391\n",
       "capital.loss      0.984757\n",
       "hours.per.week    1.000546\n",
       "dtype: float64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_test_df.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's re-run our experiments now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Without scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.125385\n",
       "score_time     0.012492\n",
       "test_score     0.799217\n",
       "train_score    0.799505\n",
       "dtype: float64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "pd.DataFrame(cross_validate(lr, X_train_imp[numeric_features], y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. With scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([('scaling', StandardScaler()),\n",
    "                 ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.061689\n",
       "score_time     0.012663\n",
       "test_score     0.814727\n",
       "train_score    0.815024\n",
       "dtype: float64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(pipe, X_train_imp[numeric_features], y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we actually do a little better! Cool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. After messing with the data by rescaling some features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.061530\n",
       "score_time     0.012299\n",
       "test_score     0.760519\n",
       "train_score    0.760519\n",
       "dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "pd.DataFrame(cross_validate(lr, X_train_mod[numeric_features], y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the same bad results we saw earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. After messing with the data, but using feature scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([('scaling', StandardScaler()),\n",
    "                 ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.067296\n",
       "score_time     0.013045\n",
       "test_score     0.814727\n",
       "train_score    0.815024\n",
       "dtype: float64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(pipe, X_train_mod[numeric_features], y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BAM! The scaling always sets the variance to 1, so the fact that we scaled up/down by 1000 is irrelevant!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can redo the same experiments but with min/max scaling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([('scaling', MinMaxScaler()),\n",
    "                 ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.086178\n",
       "score_time     0.012658\n",
       "test_score     0.810427\n",
       "train_score    0.810888\n",
       "dtype: float64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(pipe, X_train_imp[numeric_features], y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here, we get similar results. \n",
    "- We can also check that it does what it's supposed to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler()\n",
    "minmax.fit(X_train_imp[numeric_features])\n",
    "normalized_train = minmax.transform(X_train_imp[numeric_features])\n",
    "normalized_test = minmax.transform(X_test_imp[numeric_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's again check the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalized_train.min(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalized_train.max(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again for test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.        , -0.00109735,  0.        ,  0.        ,  0.        ,\n",
       "        0.        ])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalized_test.min(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 1.08768803, 1.        , 1.        , 0.84550046,\n",
       "       1.        ])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalized_test.max(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.7000e+01, 1.3769e+04, 1.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "       1.0000e+00])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minmax.data_min_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9.00000e+01, 1.36612e+06, 1.60000e+01, 9.99990e+04, 4.35600e+03,\n",
       "       9.90000e+01])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minmax.data_max_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing the targets?\n",
    "\n",
    "- We'll discuss this when we get to numeric targets (regression) in a couple weeks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together with `ColumnTransformer` (30 min)\n",
    "\n",
    "- Ok, so this is all great, but now we have ourselves a BIG MESS.\n",
    "- We have a Pipeline for the categorical features\n",
    "- Attribution: some code in this section adapted from the [sklearn documentation](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_cat = Pipeline([('imputation', SimpleImputer(strategy='most_frequent')),\n",
    "                     ('ohe', OneHotEncoder(handle_unknown='ignore')),\n",
    "                     ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a pipeline for the numeric features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_num = Pipeline([('scaler', StandardScaler()), # there were no missing values here\n",
    "                     ('lr', LogisticRegression(max_iter=1000))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But we need to join together the scaled numeric features with the imputed/OHE'd categorical features BEFORE passing the whole dataframe into the classifier!\n",
    "- We can do this awkwardly without a Pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "ohe     = OneHotEncoder(handle_unknown='ignore', sparse=False)\n",
    "scaler  = StandardScaler()\n",
    "lr      = LogisticRegression(max_iter=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we process the categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>workclass_Federal-gov</th>\n",
       "      <th>workclass_Local-gov</th>\n",
       "      <th>workclass_Never-worked</th>\n",
       "      <th>workclass_Private</th>\n",
       "      <th>workclass_Self-emp-inc</th>\n",
       "      <th>workclass_Self-emp-not-inc</th>\n",
       "      <th>workclass_State-gov</th>\n",
       "      <th>workclass_Without-pay</th>\n",
       "      <th>marital.status_Divorced</th>\n",
       "      <th>marital.status_Married-AF-spouse</th>\n",
       "      <th>...</th>\n",
       "      <th>native.country_Portugal</th>\n",
       "      <th>native.country_Puerto-Rico</th>\n",
       "      <th>native.country_Scotland</th>\n",
       "      <th>native.country_South</th>\n",
       "      <th>native.country_Taiwan</th>\n",
       "      <th>native.country_Thailand</th>\n",
       "      <th>native.country_Trinadad&amp;Tobago</th>\n",
       "      <th>native.country_United-States</th>\n",
       "      <th>native.country_Vietnam</th>\n",
       "      <th>native.country_Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 83 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       workclass_Federal-gov  workclass_Local-gov  workclass_Never-worked  \\\n",
       "17064                    0.0                  0.0                     0.0   \n",
       "18434                    0.0                  0.0                     0.0   \n",
       "3294                     0.0                  0.0                     0.0   \n",
       "31317                    0.0                  0.0                     0.0   \n",
       "4770                     0.0                  0.0                     0.0   \n",
       "\n",
       "       workclass_Private  workclass_Self-emp-inc  workclass_Self-emp-not-inc  \\\n",
       "17064                1.0                     0.0                         0.0   \n",
       "18434                1.0                     0.0                         0.0   \n",
       "3294                 1.0                     0.0                         0.0   \n",
       "31317                1.0                     0.0                         0.0   \n",
       "4770                 0.0                     1.0                         0.0   \n",
       "\n",
       "       workclass_State-gov  workclass_Without-pay  marital.status_Divorced  \\\n",
       "17064                  0.0                    0.0                      0.0   \n",
       "18434                  0.0                    0.0                      0.0   \n",
       "3294                   0.0                    0.0                      0.0   \n",
       "31317                  0.0                    0.0                      0.0   \n",
       "4770                   0.0                    0.0                      0.0   \n",
       "\n",
       "       marital.status_Married-AF-spouse  ...  native.country_Portugal  \\\n",
       "17064                               0.0  ...                      0.0   \n",
       "18434                               0.0  ...                      0.0   \n",
       "3294                                0.0  ...                      0.0   \n",
       "31317                               0.0  ...                      0.0   \n",
       "4770                                0.0  ...                      0.0   \n",
       "\n",
       "       native.country_Puerto-Rico  native.country_Scotland  \\\n",
       "17064                         0.0                      0.0   \n",
       "18434                         0.0                      0.0   \n",
       "3294                          0.0                      0.0   \n",
       "31317                         0.0                      0.0   \n",
       "4770                          0.0                      0.0   \n",
       "\n",
       "       native.country_South  native.country_Taiwan  native.country_Thailand  \\\n",
       "17064                   0.0                    0.0                      0.0   \n",
       "18434                   0.0                    0.0                      0.0   \n",
       "3294                    0.0                    0.0                      0.0   \n",
       "31317                   0.0                    0.0                      0.0   \n",
       "4770                    0.0                    0.0                      0.0   \n",
       "\n",
       "       native.country_Trinadad&Tobago  native.country_United-States  \\\n",
       "17064                             0.0                           1.0   \n",
       "18434                             0.0                           1.0   \n",
       "3294                              0.0                           1.0   \n",
       "31317                             0.0                           1.0   \n",
       "4770                              0.0                           1.0   \n",
       "\n",
       "       native.country_Vietnam  native.country_Yugoslavia  \n",
       "17064                     0.0                        0.0  \n",
       "18434                     0.0                        0.0  \n",
       "3294                      0.0                        0.0  \n",
       "31317                     0.0                        0.0  \n",
       "4770                      0.0                        0.0  \n",
       "\n",
       "[5 rows x 83 columns]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_train = X_train_nan[categorical_features]\n",
    "cat_train_imp = imputer.fit_transform(cat_train)\n",
    "cat_train_imp_ohe = ohe.fit_transform(cat_train_imp)\n",
    "cat_train_imp_ohe_df = pd.DataFrame(data=cat_train_imp_ohe, columns=ohe.get_feature_names(categorical_features), index=X_train_nan.index)\n",
    "cat_train_imp_ohe_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we process the numeric variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.364769</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.217915</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.204805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.397806</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.058330</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.316034</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28636</th>\n",
       "      <td>0.691194</td>\n",
       "      <td>-1.129174</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.774883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17730</th>\n",
       "      <td>-0.263361</td>\n",
       "      <td>1.436102</td>\n",
       "      <td>-1.582503</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.197086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28030</th>\n",
       "      <td>-0.924206</td>\n",
       "      <td>1.755487</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.126904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15725</th>\n",
       "      <td>-0.777351</td>\n",
       "      <td>-0.320086</td>\n",
       "      <td>-2.748829</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.035091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19966</th>\n",
       "      <td>0.103776</td>\n",
       "      <td>-0.361296</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.612888</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26048 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064 -1.364769 -0.745073      -0.027403     -0.146669     -0.216807   \n",
       "18434 -1.217915  0.708967      -0.416178     -0.146669     -0.216807   \n",
       "3294   0.911476  1.397806      -0.416178      0.451354     -0.216807   \n",
       "31317  1.058330 -0.736111       1.138922     -0.146669     -0.216807   \n",
       "4770   0.984903 -1.316034       1.138922     -0.146669     -0.216807   \n",
       "...         ...       ...            ...           ...           ...   \n",
       "28636  0.691194 -1.129174      -0.416178     -0.146669     -0.216807   \n",
       "17730 -0.263361  1.436102      -1.582503     -0.146669     -0.216807   \n",
       "28030 -0.924206  1.755487      -0.027403     -0.146669     -0.216807   \n",
       "15725 -0.777351 -0.320086      -2.748829     -0.146669     -0.216807   \n",
       "19966  0.103776 -0.361296      -0.416178     -0.146669     -0.216807   \n",
       "\n",
       "       hours.per.week  \n",
       "17064       -0.845065  \n",
       "18434        3.204805  \n",
       "3294         1.179870  \n",
       "31317       -0.440078  \n",
       "4770         1.179870  \n",
       "...               ...  \n",
       "28636        0.774883  \n",
       "17730       -0.197086  \n",
       "28030        0.126904  \n",
       "15725       -0.035091  \n",
       "19966        0.612888  \n",
       "\n",
       "[26048 rows x 6 columns]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_train = X_train_nan[numeric_features]\n",
    "num_train_scaler = scaler.fit_transform(num_train)\n",
    "num_train_scaler_df = pd.DataFrame(data=num_train_scaler, columns=numeric_features, index=X_train_nan.index)\n",
    "num_train_scaler_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we smush them together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>workclass_Federal-gov</th>\n",
       "      <th>workclass_Local-gov</th>\n",
       "      <th>workclass_Never-worked</th>\n",
       "      <th>workclass_Private</th>\n",
       "      <th>...</th>\n",
       "      <th>native.country_Portugal</th>\n",
       "      <th>native.country_Puerto-Rico</th>\n",
       "      <th>native.country_Scotland</th>\n",
       "      <th>native.country_South</th>\n",
       "      <th>native.country_Taiwan</th>\n",
       "      <th>native.country_Thailand</th>\n",
       "      <th>native.country_Trinadad&amp;Tobago</th>\n",
       "      <th>native.country_United-States</th>\n",
       "      <th>native.country_Vietnam</th>\n",
       "      <th>native.country_Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.364769</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.217915</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.204805</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.397806</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.058330</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.316034</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28636</th>\n",
       "      <td>0.691194</td>\n",
       "      <td>-1.129174</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.774883</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17730</th>\n",
       "      <td>-0.263361</td>\n",
       "      <td>1.436102</td>\n",
       "      <td>-1.582503</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.197086</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28030</th>\n",
       "      <td>-0.924206</td>\n",
       "      <td>1.755487</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.126904</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15725</th>\n",
       "      <td>-0.777351</td>\n",
       "      <td>-0.320086</td>\n",
       "      <td>-2.748829</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.035091</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19966</th>\n",
       "      <td>0.103776</td>\n",
       "      <td>-0.361296</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.612888</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26048 rows × 89 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064 -1.364769 -0.745073      -0.027403     -0.146669     -0.216807   \n",
       "18434 -1.217915  0.708967      -0.416178     -0.146669     -0.216807   \n",
       "3294   0.911476  1.397806      -0.416178      0.451354     -0.216807   \n",
       "31317  1.058330 -0.736111       1.138922     -0.146669     -0.216807   \n",
       "4770   0.984903 -1.316034       1.138922     -0.146669     -0.216807   \n",
       "...         ...       ...            ...           ...           ...   \n",
       "28636  0.691194 -1.129174      -0.416178     -0.146669     -0.216807   \n",
       "17730 -0.263361  1.436102      -1.582503     -0.146669     -0.216807   \n",
       "28030 -0.924206  1.755487      -0.027403     -0.146669     -0.216807   \n",
       "15725 -0.777351 -0.320086      -2.748829     -0.146669     -0.216807   \n",
       "19966  0.103776 -0.361296      -0.416178     -0.146669     -0.216807   \n",
       "\n",
       "       hours.per.week  workclass_Federal-gov  workclass_Local-gov  \\\n",
       "17064       -0.845065                    0.0                  0.0   \n",
       "18434        3.204805                    0.0                  0.0   \n",
       "3294         1.179870                    0.0                  0.0   \n",
       "31317       -0.440078                    0.0                  0.0   \n",
       "4770         1.179870                    0.0                  0.0   \n",
       "...               ...                    ...                  ...   \n",
       "28636        0.774883                    0.0                  0.0   \n",
       "17730       -0.197086                    0.0                  0.0   \n",
       "28030        0.126904                    0.0                  0.0   \n",
       "15725       -0.035091                    0.0                  0.0   \n",
       "19966        0.612888                    0.0                  0.0   \n",
       "\n",
       "       workclass_Never-worked  workclass_Private  ...  \\\n",
       "17064                     0.0                1.0  ...   \n",
       "18434                     0.0                1.0  ...   \n",
       "3294                      0.0                1.0  ...   \n",
       "31317                     0.0                1.0  ...   \n",
       "4770                      0.0                0.0  ...   \n",
       "...                       ...                ...  ...   \n",
       "28636                     0.0                1.0  ...   \n",
       "17730                     0.0                1.0  ...   \n",
       "28030                     0.0                1.0  ...   \n",
       "15725                     0.0                1.0  ...   \n",
       "19966                     0.0                1.0  ...   \n",
       "\n",
       "       native.country_Portugal  native.country_Puerto-Rico  \\\n",
       "17064                      0.0                         0.0   \n",
       "18434                      0.0                         0.0   \n",
       "3294                       0.0                         0.0   \n",
       "31317                      0.0                         0.0   \n",
       "4770                       0.0                         0.0   \n",
       "...                        ...                         ...   \n",
       "28636                      0.0                         0.0   \n",
       "17730                      0.0                         0.0   \n",
       "28030                      0.0                         0.0   \n",
       "15725                      0.0                         0.0   \n",
       "19966                      0.0                         0.0   \n",
       "\n",
       "       native.country_Scotland  native.country_South  native.country_Taiwan  \\\n",
       "17064                      0.0                   0.0                    0.0   \n",
       "18434                      0.0                   0.0                    0.0   \n",
       "3294                       0.0                   0.0                    0.0   \n",
       "31317                      0.0                   0.0                    0.0   \n",
       "4770                       0.0                   0.0                    0.0   \n",
       "...                        ...                   ...                    ...   \n",
       "28636                      0.0                   0.0                    0.0   \n",
       "17730                      0.0                   0.0                    0.0   \n",
       "28030                      0.0                   0.0                    0.0   \n",
       "15725                      0.0                   0.0                    0.0   \n",
       "19966                      0.0                   0.0                    0.0   \n",
       "\n",
       "       native.country_Thailand  native.country_Trinadad&Tobago  \\\n",
       "17064                      0.0                             0.0   \n",
       "18434                      0.0                             0.0   \n",
       "3294                       0.0                             0.0   \n",
       "31317                      0.0                             0.0   \n",
       "4770                       0.0                             0.0   \n",
       "...                        ...                             ...   \n",
       "28636                      0.0                             0.0   \n",
       "17730                      0.0                             0.0   \n",
       "28030                      0.0                             0.0   \n",
       "15725                      0.0                             0.0   \n",
       "19966                      0.0                             0.0   \n",
       "\n",
       "       native.country_United-States  native.country_Vietnam  \\\n",
       "17064                           1.0                     0.0   \n",
       "18434                           1.0                     0.0   \n",
       "3294                            1.0                     0.0   \n",
       "31317                           1.0                     0.0   \n",
       "4770                            1.0                     0.0   \n",
       "...                             ...                     ...   \n",
       "28636                           1.0                     0.0   \n",
       "17730                           1.0                     0.0   \n",
       "28030                           1.0                     0.0   \n",
       "15725                           0.0                     0.0   \n",
       "19966                           1.0                     0.0   \n",
       "\n",
       "       native.country_Yugoslavia  \n",
       "17064                        0.0  \n",
       "18434                        0.0  \n",
       "3294                         0.0  \n",
       "31317                        0.0  \n",
       "4770                         0.0  \n",
       "...                          ...  \n",
       "28636                        0.0  \n",
       "17730                        0.0  \n",
       "28030                        0.0  \n",
       "15725                        0.0  \n",
       "19966                        0.0  \n",
       "\n",
       "[26048 rows x 89 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_this_is_so_annoying = pd.concat((num_train_scaler_df, cat_train_imp_ohe_df), axis=1)\n",
    "X_train_this_is_so_annoying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8523495085995086"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.fit(X_train_this_is_so_annoying, y_train)\n",
    "lr.score(X_train_this_is_so_annoying, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, time to do all that again for the test data!\n",
    "\n",
    "![](img/mike_not_gonna_happen.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Right, so the above is horribly messy and also results in a Golden Rule violation if we do cross-validation.\n",
    "- Enter `ColumnTransformer` to save the day!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/column-transformer.png\" width=800>\n",
    "\n",
    "Image adapted from [here](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#37)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A big advantage here is that we build all our transformations together into one object, and that way we're sure we do the same operations to all splits of the data. \n",
    "- Otherwise we might, for example, do the OHE on both train and test but forget to scale the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get to work! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['age',\n",
       " 'fnlwgt',\n",
       " 'education.num',\n",
       " 'capital.gain',\n",
       " 'capital.loss',\n",
       " 'hours.per.week']"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "    ('scale', StandardScaler(), numeric_features),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore', sparse=False), categorical_features)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sklearn.compose._column_transformer.ColumnTransformer"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above:\n",
    "\n",
    "- The `ColumnTransformer` syntax is somewhat similar to `Pipeline` in that you pass in a list of tuples.\n",
    "- But here each tuple has 3 values instead of 2: (name, object, list of columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor.fit(X_train_imp);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.36476947, -0.74507317, -0.02740291, ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-1.21791495,  0.70896709, -0.41617802, ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.91147565,  1.39780572, -0.41617802, ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       ...,\n",
       "       [-0.9242059 ,  1.75548713, -0.02740291, ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.77735138, -0.32008602, -2.74882863, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.10377577, -0.36129614, -0.41617802, ...,  1.        ,\n",
       "         0.        ,  0.        ]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_preproc = preprocessor.transform(X_train_imp)\n",
    "X_train_preproc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train_preproc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>workclass_Federal-gov</th>\n",
       "      <th>workclass_Local-gov</th>\n",
       "      <th>workclass_Never-worked</th>\n",
       "      <th>workclass_Private</th>\n",
       "      <th>...</th>\n",
       "      <th>native.country_Portugal</th>\n",
       "      <th>native.country_Puerto-Rico</th>\n",
       "      <th>native.country_Scotland</th>\n",
       "      <th>native.country_South</th>\n",
       "      <th>native.country_Taiwan</th>\n",
       "      <th>native.country_Thailand</th>\n",
       "      <th>native.country_Trinadad&amp;Tobago</th>\n",
       "      <th>native.country_United-States</th>\n",
       "      <th>native.country_Vietnam</th>\n",
       "      <th>native.country_Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.364769</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.217915</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.204805</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.397806</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.058330</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.316034</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28636</th>\n",
       "      <td>0.691194</td>\n",
       "      <td>-1.129174</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.774883</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17730</th>\n",
       "      <td>-0.263361</td>\n",
       "      <td>1.436102</td>\n",
       "      <td>-1.582503</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.197086</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28030</th>\n",
       "      <td>-0.924206</td>\n",
       "      <td>1.755487</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.126904</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15725</th>\n",
       "      <td>-0.777351</td>\n",
       "      <td>-0.320086</td>\n",
       "      <td>-2.748829</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.035091</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19966</th>\n",
       "      <td>0.103776</td>\n",
       "      <td>-0.361296</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>0.612888</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26048 rows × 89 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064 -1.364769 -0.745073      -0.027403     -0.146669     -0.216807   \n",
       "18434 -1.217915  0.708967      -0.416178     -0.146669     -0.216807   \n",
       "3294   0.911476  1.397806      -0.416178      0.451354     -0.216807   \n",
       "31317  1.058330 -0.736111       1.138922     -0.146669     -0.216807   \n",
       "4770   0.984903 -1.316034       1.138922     -0.146669     -0.216807   \n",
       "...         ...       ...            ...           ...           ...   \n",
       "28636  0.691194 -1.129174      -0.416178     -0.146669     -0.216807   \n",
       "17730 -0.263361  1.436102      -1.582503     -0.146669     -0.216807   \n",
       "28030 -0.924206  1.755487      -0.027403     -0.146669     -0.216807   \n",
       "15725 -0.777351 -0.320086      -2.748829     -0.146669     -0.216807   \n",
       "19966  0.103776 -0.361296      -0.416178     -0.146669     -0.216807   \n",
       "\n",
       "       hours.per.week  workclass_Federal-gov  workclass_Local-gov  \\\n",
       "17064       -0.845065                    0.0                  0.0   \n",
       "18434        3.204805                    0.0                  0.0   \n",
       "3294         1.179870                    0.0                  0.0   \n",
       "31317       -0.440078                    0.0                  0.0   \n",
       "4770         1.179870                    0.0                  0.0   \n",
       "...               ...                    ...                  ...   \n",
       "28636        0.774883                    0.0                  0.0   \n",
       "17730       -0.197086                    0.0                  0.0   \n",
       "28030        0.126904                    0.0                  0.0   \n",
       "15725       -0.035091                    0.0                  0.0   \n",
       "19966        0.612888                    0.0                  0.0   \n",
       "\n",
       "       workclass_Never-worked  workclass_Private  ...  \\\n",
       "17064                     0.0                1.0  ...   \n",
       "18434                     0.0                1.0  ...   \n",
       "3294                      0.0                1.0  ...   \n",
       "31317                     0.0                1.0  ...   \n",
       "4770                      0.0                0.0  ...   \n",
       "...                       ...                ...  ...   \n",
       "28636                     0.0                1.0  ...   \n",
       "17730                     0.0                1.0  ...   \n",
       "28030                     0.0                1.0  ...   \n",
       "15725                     0.0                1.0  ...   \n",
       "19966                     0.0                1.0  ...   \n",
       "\n",
       "       native.country_Portugal  native.country_Puerto-Rico  \\\n",
       "17064                      0.0                         0.0   \n",
       "18434                      0.0                         0.0   \n",
       "3294                       0.0                         0.0   \n",
       "31317                      0.0                         0.0   \n",
       "4770                       0.0                         0.0   \n",
       "...                        ...                         ...   \n",
       "28636                      0.0                         0.0   \n",
       "17730                      0.0                         0.0   \n",
       "28030                      0.0                         0.0   \n",
       "15725                      0.0                         0.0   \n",
       "19966                      0.0                         0.0   \n",
       "\n",
       "       native.country_Scotland  native.country_South  native.country_Taiwan  \\\n",
       "17064                      0.0                   0.0                    0.0   \n",
       "18434                      0.0                   0.0                    0.0   \n",
       "3294                       0.0                   0.0                    0.0   \n",
       "31317                      0.0                   0.0                    0.0   \n",
       "4770                       0.0                   0.0                    0.0   \n",
       "...                        ...                   ...                    ...   \n",
       "28636                      0.0                   0.0                    0.0   \n",
       "17730                      0.0                   0.0                    0.0   \n",
       "28030                      0.0                   0.0                    0.0   \n",
       "15725                      0.0                   0.0                    0.0   \n",
       "19966                      0.0                   0.0                    0.0   \n",
       "\n",
       "       native.country_Thailand  native.country_Trinadad&Tobago  \\\n",
       "17064                      0.0                             0.0   \n",
       "18434                      0.0                             0.0   \n",
       "3294                       0.0                             0.0   \n",
       "31317                      0.0                             0.0   \n",
       "4770                       0.0                             0.0   \n",
       "...                        ...                             ...   \n",
       "28636                      0.0                             0.0   \n",
       "17730                      0.0                             0.0   \n",
       "28030                      0.0                             0.0   \n",
       "15725                      0.0                             0.0   \n",
       "19966                      0.0                             0.0   \n",
       "\n",
       "       native.country_United-States  native.country_Vietnam  \\\n",
       "17064                           1.0                     0.0   \n",
       "18434                           1.0                     0.0   \n",
       "3294                            1.0                     0.0   \n",
       "31317                           1.0                     0.0   \n",
       "4770                            1.0                     0.0   \n",
       "...                             ...                     ...   \n",
       "28636                           1.0                     0.0   \n",
       "17730                           1.0                     0.0   \n",
       "28030                           1.0                     0.0   \n",
       "15725                           0.0                     0.0   \n",
       "19966                           1.0                     0.0   \n",
       "\n",
       "       native.country_Yugoslavia  \n",
       "17064                        0.0  \n",
       "18434                        0.0  \n",
       "3294                         0.0  \n",
       "31317                        0.0  \n",
       "4770                         0.0  \n",
       "...                          ...  \n",
       "28636                        0.0  \n",
       "17730                        0.0  \n",
       "28030                        0.0  \n",
       "15725                        0.0  \n",
       "19966                        0.0  \n",
       "\n",
       "[26048 rows x 89 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_columns = numeric_features + list(preprocessor.named_transformers_['ohe'].get_feature_names(categorical_features))\n",
    "pd.DataFrame(data=X_train_preproc, columns=new_columns, index=X_train_imp.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- BAM! Scaling and OHE applied at the same time!\n",
    "- When we `fit` the `ColumnTransformer`, it fits _all_ the transformers. And likewise for `transform`.\n",
    "- Warning: by default `ColumnTransformer` throws away any columns not accounted for in its steps.\n",
    "- Setting `remainder='passthrough'` keeps the rest of the columns in tact as in the image above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's put everything together in a pipeline: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attempt 1: one pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We use the preprocessor as a step in the pipeline!\n",
    "- This step treats numeric features and categorical features differently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('impute', SimpleImputer(strategy='most_frequent')),\n",
    "    ('preproc', preprocessor), # <-- this is the ColumnTransformer!\n",
    "    ('lr', LogisticRegression(max_iter=1000))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Specifying the columns using strings is only supported for pandas DataFrames",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/utils/__init__.py\u001b[0m in \u001b[0;36m_get_column_indices\u001b[0;34m(X, key)\u001b[0m\n\u001b[1;32m    424\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 425\u001b[0;31m             \u001b[0mall_columns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    426\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'columns'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-84-a2e0e5c89735>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mpipe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train_nan\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/pipeline.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, **fit_params)\u001b[0m\n\u001b[1;32m    328\u001b[0m         \"\"\"\n\u001b[1;32m    329\u001b[0m         \u001b[0mfit_params_steps\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_fit_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 330\u001b[0;31m         \u001b[0mXt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m         with _print_elapsed_time('Pipeline',\n\u001b[1;32m    332\u001b[0m                                  self._log_message(len(self.steps) - 1)):\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/pipeline.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, X, y, **fit_params_steps)\u001b[0m\n\u001b[1;32m    290\u001b[0m                 \u001b[0mcloned_transformer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtransformer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    291\u001b[0m             \u001b[0;31m# Fit or load from cache the current transformer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 292\u001b[0;31m             X, fitted_transformer = fit_transform_one_cached(\n\u001b[0m\u001b[1;32m    293\u001b[0m                 \u001b[0mcloned_transformer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    294\u001b[0m                 \u001b[0mmessage_clsname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Pipeline'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/joblib/memory.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    350\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    351\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 352\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    353\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    354\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcall_and_shelve\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/pipeline.py\u001b[0m in \u001b[0;36m_fit_transform_one\u001b[0;34m(transformer, X, y, weight, message_clsname, message, **fit_params)\u001b[0m\n\u001b[1;32m    738\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0m_print_elapsed_time\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage_clsname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    739\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtransformer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'fit_transform'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 740\u001b[0;31m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtransformer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    741\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    742\u001b[0m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtransformer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py\u001b[0m in \u001b[0;36mfit_transform\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m    527\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_transformers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    528\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_column_callables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 529\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_remainder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    530\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    531\u001b[0m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_fit_transform_one\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py\u001b[0m in \u001b[0;36m_validate_remainder\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    325\u001b[0m         \u001b[0mcols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    326\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mcolumns\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_columns\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 327\u001b[0;31m             \u001b[0mcols\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_get_column_indices\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    329\u001b[0m         \u001b[0mremaining_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_features\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcols\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/utils/__init__.py\u001b[0m in \u001b[0;36m_get_column_indices\u001b[0;34m(X, key)\u001b[0m\n\u001b[1;32m    425\u001b[0m             \u001b[0mall_columns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    426\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 427\u001b[0;31m             raise ValueError(\"Specifying the columns using strings is only \"\n\u001b[0m\u001b[1;32m    428\u001b[0m                              \"supported for pandas DataFrames\")\n\u001b[1;32m    429\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Specifying the columns using strings is only supported for pandas DataFrames"
     ]
    }
   ],
   "source": [
    "pipe.fit(X_train_nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ugh. What happened?\n",
    "- The problem is that `SimpleImputer.transform` outputs a numpy array, not a pandas dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(imputer.transform(X_train_nan[categorical_features]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've been putting it into a dataframe manually so that it looks nice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>workclass</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>native.country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>Private</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Asian-Pac-Islander</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>Private</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>Private</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>Private</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28636</th>\n",
       "      <td>Private</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17730</th>\n",
       "      <td>Private</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28030</th>\n",
       "      <td>Private</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15725</th>\n",
       "      <td>Private</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>Columbia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19966</th>\n",
       "      <td>Private</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26048 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          workclass      marital.status         occupation   relationship  \\\n",
       "17064       Private       Never-married       Adm-clerical      Own-child   \n",
       "18434       Private       Never-married      Other-service      Own-child   \n",
       "3294        Private  Married-civ-spouse    Exec-managerial        Husband   \n",
       "31317       Private  Married-civ-spouse      Other-service        Husband   \n",
       "4770   Self-emp-inc  Married-civ-spouse              Sales        Husband   \n",
       "...             ...                 ...                ...            ...   \n",
       "28636       Private  Married-civ-spouse  Machine-op-inspct           Wife   \n",
       "17730       Private       Never-married      Other-service      Unmarried   \n",
       "28030       Private  Married-civ-spouse       Adm-clerical           Wife   \n",
       "15725       Private       Never-married       Craft-repair  Not-in-family   \n",
       "19966       Private            Divorced  Handlers-cleaners  Not-in-family   \n",
       "\n",
       "                     race     sex native.country  \n",
       "17064  Asian-Pac-Islander  Female  United-States  \n",
       "18434               Black    Male  United-States  \n",
       "3294                White    Male  United-States  \n",
       "31317               White    Male  United-States  \n",
       "4770                White    Male  United-States  \n",
       "...                   ...     ...            ...  \n",
       "28636               White  Female  United-States  \n",
       "17730               Black  Female  United-States  \n",
       "28030               White  Female  United-States  \n",
       "15725               White    Male       Columbia  \n",
       "19966               White  Female  United-States  \n",
       "\n",
       "[26048 rows x 7 columns]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data=imputer.transform(X_train_nan[categorical_features]), columns=categorical_features, index=X_train_nan.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, when we made the `ColumnTransformer` we referred to columns _by name_ rather than by index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "    ('scale', StandardScaler(), numeric_features),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore', sparse=False), categorical_features)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['age',\n",
       " 'fnlwgt',\n",
       " 'education.num',\n",
       " 'capital.gain',\n",
       " 'capital.loss',\n",
       " 'hours.per.week']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['workclass',\n",
       " 'marital.status',\n",
       " 'occupation',\n",
       " 'relationship',\n",
       " 'race',\n",
       " 'sex',\n",
       " 'native.country']"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can't use these names if it's getting a numpy array where the columns aren't named."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('impute', SimpleImputer(strategy='most_frequent')),\n",
    "    ('preproc', preprocessor), # <-- this is the ColumnTransformer!\n",
    "    ('lr', LogisticRegression(max_iter=1000))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attempt 2: separate pipeline for each feature type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a pipeline for the categorical features (preprocessing only):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_cat = Pipeline([\n",
    "    ('impute', SimpleImputer(strategy='most_frequent')),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore', sparse=False))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't need a pipeline for the numerical features because there's only one step, scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's put these together to make a `ColumnTransformer` for all the preprocessing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "    ('cat', pipe_cat, categorical_features),\n",
    "    ('num', StandardScaler(), numeric_features)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Take a minute to digest this... what does it do.\n",
    "- When ready, let's combine this with the classifier in _another pipeline_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('preprocessing', preprocessor),\n",
    "    ('classifier', LogisticRegression(max_iter=1000))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.fit(X_train_nan, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, what just happened here?\n",
    "\n",
    "- We fit the imputer, one-hot encoder, standard scaler, and logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['<=50K', '>50K', '<=50K', ..., '<=50K', '<=50K', '<=50K'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.predict(X_test_nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fit_time': array([1.68061996, 1.45436502, 1.71592689, 1.49593711, 1.52128315]),\n",
       " 'score_time': array([0.03714681, 0.03344703, 0.03336501, 0.03313684, 0.03448796]),\n",
       " 'test_score': array([0.84932821, 0.84913628, 0.84414587, 0.85717028, 0.85486658])}"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_validate(pipe, X_train_nan, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A LOT of steps just happened here!\n",
    "- This is so cool (if you ask me)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two images for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"db95553e-3db2-493a-a8f5-0c4a52739dcf\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"db95553e-3db2-493a-a8f5-0c4a52739dcf\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('cat',\n",
       "                                                  Pipeline(steps=[('impute',\n",
       "                                                                   SimpleImputer(strategy='most_frequent')),\n",
       "                                                                  ('ohe',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                                 sparse=False))]),\n",
       "                                                  ['workclass',\n",
       "                                                   'marital.status',\n",
       "                                                   'occupation', 'relationship',\n",
       "                                                   'race', 'sex',\n",
       "                                                   'native.country']),\n",
       "                                                 ('num', StandardScaler(),\n",
       "                                                  ['age', 'fnlwgt',\n",
       "                                                   'education.num',\n",
       "                                                   'capital.gain',\n",
       "                                                   'capital.loss',\n",
       "                                                   'hours.per.week'])])),\n",
       "                ('classifier', LogisticRegression(max_iter=1000))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"6e2cfbd4-561c-4243-b830-926ee6a42095\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"6e2cfbd4-561c-4243-b830-926ee6a42095\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('cat',\n",
       "                                 Pipeline(steps=[('impute',\n",
       "                                                  SimpleImputer(strategy='most_frequent')),\n",
       "                                                 ('ohe',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse=False))]),\n",
       "                                 ['workclass', 'marital.status', 'occupation',\n",
       "                                  'relationship', 'race', 'sex',\n",
       "                                  'native.country']),\n",
       "                                ('num', StandardScaler(),\n",
       "                                 ['age', 'fnlwgt', 'education.num',\n",
       "                                  'capital.gain', 'capital.loss',\n",
       "                                  'hours.per.week'])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a549beae-d25f-49d4-bbfe-172c3a6957cd\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"a549beae-d25f-49d4-bbfe-172c3a6957cd\">cat</label><div class=\"sk-toggleable__content\"><pre>['workclass', 'marital.status', 'occupation', 'relationship', 'race', 'sex', 'native.country']</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7416df72-6114-4984-af8d-cc236bc18120\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"7416df72-6114-4984-af8d-cc236bc18120\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8b1c8399-80ec-44b9-865e-d64ef9cd6739\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"8b1c8399-80ec-44b9-865e-d64ef9cd6739\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore', sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"bb471755-78be-4ee2-a3e8-7a570ff789ad\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"bb471755-78be-4ee2-a3e8-7a570ff789ad\">num</label><div class=\"sk-toggleable__content\"><pre>['age', 'fnlwgt', 'education.num', 'capital.gain', 'capital.loss', 'hours.per.week']</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b5e830a5-b945-47a2-b8ff-5757da2ba080\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b5e830a5-b945-47a2-b8ff-5757da2ba080\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"932b7aec-e33f-48c6-8d02-7adfeeac2515\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"932b7aec-e33f-48c6-8d02-7adfeeac2515\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('cat',\n",
       "                                                  Pipeline(steps=[('impute',\n",
       "                                                                   SimpleImputer(strategy='most_frequent')),\n",
       "                                                                  ('ohe',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                                 sparse=False))]),\n",
       "                                                  ['workclass',\n",
       "                                                   'marital.status',\n",
       "                                                   'occupation', 'relationship',\n",
       "                                                   'race', 'sex',\n",
       "                                                   'native.country']),\n",
       "                                                 ('num', StandardScaler(),\n",
       "                                                  ['age', 'fnlwgt',\n",
       "                                                   'education.num',\n",
       "                                                   'capital.gain',\n",
       "                                                   'capital.loss',\n",
       "                                                   'hours.per.week'])])),\n",
       "                ('classifier', LogisticRegression(max_iter=1000))])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set_config(display='diagram')\n",
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/pipeline_columntransformer.png\" width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter search, revisited (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's do hyperparameter optimization on this pipeline.\n",
    "- Let's optimize `C` from logistic regression and `strategy` from `SimpleImputer` (either `most_frequent` or `constant`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypers = {\n",
    "    'classifier__C' : [0.01, 0.1, 1, 10, 100]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we access the imputer strategy?\n",
    "- Well, it's the strategy of the imputer of the categorical part of the preprocessing..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypers = {\n",
    "    'classifier__C' : [0.01, 0.1, 1, 10, 100],\n",
    "    'preprocessing__cat__impute__strategy' : ['most_frequent', 'constant']\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We are indexing into the mess of pipelines and `ColumnsTransformer`s here.\n",
    "- Note that we're using the names given when made those objects, _not_ the Python variable names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "searcher = GridSearchCV(pipe, hypers, n_jobs=-1, verbose=2, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:    9.4s\n",
      "[Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:   26.6s finished\n"
     ]
    }
   ],
   "source": [
    "searcher.fit(X_train_nan, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>param_classifier__C</th>\n",
       "      <th>param_preprocessing__cat__impute__strategy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.851659</td>\n",
       "      <td>0.853492</td>\n",
       "      <td>2.342403</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>constant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.851467</td>\n",
       "      <td>0.852733</td>\n",
       "      <td>0.900799</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>constant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.851313</td>\n",
       "      <td>0.853703</td>\n",
       "      <td>5.383548</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>constant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.851237</td>\n",
       "      <td>0.853626</td>\n",
       "      <td>3.530070</td>\n",
       "      <td>4</td>\n",
       "      <td>100</td>\n",
       "      <td>constant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.850929</td>\n",
       "      <td>0.852695</td>\n",
       "      <td>4.088326</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>most_frequent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.850737</td>\n",
       "      <td>0.851956</td>\n",
       "      <td>2.270041</td>\n",
       "      <td>6</td>\n",
       "      <td>0.1</td>\n",
       "      <td>most_frequent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.850661</td>\n",
       "      <td>0.852897</td>\n",
       "      <td>6.673370</td>\n",
       "      <td>7</td>\n",
       "      <td>100</td>\n",
       "      <td>most_frequent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.850584</td>\n",
       "      <td>0.852849</td>\n",
       "      <td>7.300103</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>most_frequent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.848511</td>\n",
       "      <td>0.849125</td>\n",
       "      <td>0.439311</td>\n",
       "      <td>9</td>\n",
       "      <td>0.01</td>\n",
       "      <td>constant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.848434</td>\n",
       "      <td>0.848904</td>\n",
       "      <td>1.715281</td>\n",
       "      <td>10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>most_frequent</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_test_score  mean_train_score  mean_fit_time  rank_test_score  \\\n",
       "5         0.851659          0.853492       2.342403                1   \n",
       "3         0.851467          0.852733       0.900799                2   \n",
       "7         0.851313          0.853703       5.383548                3   \n",
       "9         0.851237          0.853626       3.530070                4   \n",
       "4         0.850929          0.852695       4.088326                5   \n",
       "2         0.850737          0.851956       2.270041                6   \n",
       "8         0.850661          0.852897       6.673370                7   \n",
       "6         0.850584          0.852849       7.300103                8   \n",
       "1         0.848511          0.849125       0.439311                9   \n",
       "0         0.848434          0.848904       1.715281               10   \n",
       "\n",
       "  param_classifier__C param_preprocessing__cat__impute__strategy  \n",
       "5                   1                                   constant  \n",
       "3                 0.1                                   constant  \n",
       "7                  10                                   constant  \n",
       "9                 100                                   constant  \n",
       "4                   1                              most_frequent  \n",
       "2                 0.1                              most_frequent  \n",
       "8                 100                              most_frequent  \n",
       "6                  10                              most_frequent  \n",
       "1                0.01                                   constant  \n",
       "0                0.01                              most_frequent  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [\n",
    "    'mean_test_score', 'mean_train_score', 'mean_fit_time', 'rank_test_score', 'param_classifier__C', 'param_preprocessing__cat__impute__strategy'\n",
    "]\n",
    "pd.DataFrame(searcher.cv_results_)[columns].sort_values(by=['rank_test_score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, the range of scores is super small here; it seems like these hyperparameters don't matter too much in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8519883310302472"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "searcher.score(X_test_nan, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here it looks like the overfitting on the validation set was not a serious issue, likely mainly due to the larger dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26048, 13)"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_nan.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary (5 min)\n",
    "\n",
    "- We can use `SimpleImputer` to impute values where data is missing.\n",
    "- Feature scaling...\n",
    "  - improves performance for some models (so far: logistic regression but not decision trees)\n",
    "  - is generally a good idea for numeric features\n",
    "  - I'll say more about `StandardScaler` vs. `MinMaxScaler` later in the course\n",
    "- `ColumnTransformer` is great for more complex pipelines, though it's not simple!\n",
    "  - It allows us to perform different operations (possible pipelines!) on different sets of columns.\n",
    "  - We have to be carefully reference the hyperparameters in `GridSearchCV` and `RandomizedSearchCV`."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}