Hyperparameter Tuning with sklearn/Hyperparameter Tuning with sklearn.ipynb

{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "nbformat_minor": 2,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell1",
        "collapsed": false
      },
      "source": "# Hyperparameter Tuning with sklearn\n\nIn this tutorial, we will walk you through an example workflow for how you can use hyperparameter tuning with [sci-kit learn](https://scikit-learn.org/stable/) to pick the best model for your machine learning problem.\n\n**Requirements:** Please add the `scikit-learn` package from the package picker on the top right. We will be using this package in the notebook.\n\n## Load Iris Dataset\n\n We will be using sklearn's built-in [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) to predict the flower type based on features, such as the length and width of the sepal and petal.",
      "id": "baf3385b-ebd1-47dd-b68c-d91832ed3077"
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "name": "cell2",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "# Load iris flower dataset\nfrom sklearn import svm, datasets\niris = datasets.load_iris()",
      "id": "88b3d55e-3122-44dc-8527-ab87ac3537e6"
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "scrolled": false,
        "name": "cell3",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepal length (cm)</th>\n",
              "      <th>sepal width (cm)</th>\n",
              "      <th>petal length (cm)</th>\n",
              "      <th>petal width (cm)</th>\n",
              "      <th>flower</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>setosa</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>5.3</td>\n",
              "      <td>3.7</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>setosa</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.3</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>setosa</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>7.0</td>\n",
              "      <td>3.2</td>\n",
              "      <td>4.7</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>6.4</td>\n",
              "      <td>3.2</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>6.9</td>\n",
              "      <td>3.1</td>\n",
              "      <td>4.9</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>5.5</td>\n",
              "      <td>2.3</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>6.5</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.6</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>5.7</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>6.3</td>\n",
              "      <td>3.3</td>\n",
              "      <td>4.7</td>\n",
              "      <td>1.6</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>4.9</td>\n",
              "      <td>2.4</td>\n",
              "      <td>3.3</td>\n",
              "      <td>1.0</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>6.6</td>\n",
              "      <td>2.9</td>\n",
              "      <td>4.6</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>5.2</td>\n",
              "      <td>2.7</td>\n",
              "      <td>3.9</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.0</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.2</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>6.0</td>\n",
              "      <td>2.2</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>6.1</td>\n",
              "      <td>2.9</td>\n",
              "      <td>4.7</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>5.6</td>\n",
              "      <td>2.9</td>\n",
              "      <td>3.6</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.1</td>\n",
              "      <td>4.4</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>5.6</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>5.8</td>\n",
              "      <td>2.7</td>\n",
              "      <td>4.1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>6.2</td>\n",
              "      <td>2.2</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>5.6</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.9</td>\n",
              "      <td>1.1</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.2</td>\n",
              "      <td>4.8</td>\n",
              "      <td>1.8</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>6.1</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>4.9</td>\n",
              "      <td>1.5</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>6.1</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.7</td>\n",
              "      <td>1.2</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>6.4</td>\n",
              "      <td>2.9</td>\n",
              "      <td>4.3</td>\n",
              "      <td>1.3</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>6.6</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.4</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>6.8</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.8</td>\n",
              "      <td>1.4</td>\n",
              "      <td>versicolor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120</th>\n",
              "      <td>6.9</td>\n",
              "      <td>3.2</td>\n",
              "      <td>5.7</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121</th>\n",
              "      <td>5.6</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.9</td>\n",
              "      <td>2.0</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122</th>\n",
              "      <td>7.7</td>\n",
              "      <td>2.8</td>\n",
              "      <td>6.7</td>\n",
              "      <td>2.0</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>123</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.7</td>\n",
              "      <td>4.9</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>124</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.3</td>\n",
              "      <td>5.7</td>\n",
              "      <td>2.1</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>125</th>\n",
              "      <td>7.2</td>\n",
              "      <td>3.2</td>\n",
              "      <td>6.0</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>126</th>\n",
              "      <td>6.2</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.8</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>127</th>\n",
              "      <td>6.1</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.9</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>128</th>\n",
              "      <td>6.4</td>\n",
              "      <td>2.8</td>\n",
              "      <td>5.6</td>\n",
              "      <td>2.1</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>129</th>\n",
              "      <td>7.2</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.8</td>\n",
              "      <td>1.6</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>130</th>\n",
              "      <td>7.4</td>\n",
              "      <td>2.8</td>\n",
              "      <td>6.1</td>\n",
              "      <td>1.9</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>131</th>\n",
              "      <td>7.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>6.4</td>\n",
              "      <td>2.0</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>6.4</td>\n",
              "      <td>2.8</td>\n",
              "      <td>5.6</td>\n",
              "      <td>2.2</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>133</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.8</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>134</th>\n",
              "      <td>6.1</td>\n",
              "      <td>2.6</td>\n",
              "      <td>5.6</td>\n",
              "      <td>1.4</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>135</th>\n",
              "      <td>7.7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>6.1</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>136</th>\n",
              "      <td>6.3</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.6</td>\n",
              "      <td>2.4</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>137</th>\n",
              "      <td>6.4</td>\n",
              "      <td>3.1</td>\n",
              "      <td>5.5</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>138</th>\n",
              "      <td>6.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.8</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>139</th>\n",
              "      <td>6.9</td>\n",
              "      <td>3.1</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.1</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>140</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.1</td>\n",
              "      <td>5.6</td>\n",
              "      <td>2.4</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>141</th>\n",
              "      <td>6.9</td>\n",
              "      <td>3.1</td>\n",
              "      <td>5.1</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142</th>\n",
              "      <td>5.8</td>\n",
              "      <td>2.7</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.9</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>143</th>\n",
              "      <td>6.8</td>\n",
              "      <td>3.2</td>\n",
              "      <td>5.9</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>144</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.3</td>\n",
              "      <td>5.7</td>\n",
              "      <td>2.5</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>145</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>virginica</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>103 rows \u00d7 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
              "47                 4.6               3.2                1.4               0.2   \n",
              "48                 5.3               3.7                1.5               0.2   \n",
              "49                 5.0               3.3                1.4               0.2   \n",
              "50                 7.0               3.2                4.7               1.4   \n",
              "51                 6.4               3.2                4.5               1.5   \n",
              "52                 6.9               3.1                4.9               1.5   \n",
              "53                 5.5               2.3                4.0               1.3   \n",
              "54                 6.5               2.8                4.6               1.5   \n",
              "55                 5.7               2.8                4.5               1.3   \n",
              "56                 6.3               3.3                4.7               1.6   \n",
              "57                 4.9               2.4                3.3               1.0   \n",
              "58                 6.6               2.9                4.6               1.3   \n",
              "59                 5.2               2.7                3.9               1.4   \n",
              "60                 5.0               2.0                3.5               1.0   \n",
              "61                 5.9               3.0                4.2               1.5   \n",
              "62                 6.0               2.2                4.0               1.0   \n",
              "63                 6.1               2.9                4.7               1.4   \n",
              "64                 5.6               2.9                3.6               1.3   \n",
              "65                 6.7               3.1                4.4               1.4   \n",
              "66                 5.6               3.0                4.5               1.5   \n",
              "67                 5.8               2.7                4.1               1.0   \n",
              "68                 6.2               2.2                4.5               1.5   \n",
              "69                 5.6               2.5                3.9               1.1   \n",
              "70                 5.9               3.2                4.8               1.8   \n",
              "71                 6.1               2.8                4.0               1.3   \n",
              "72                 6.3               2.5                4.9               1.5   \n",
              "73                 6.1               2.8                4.7               1.2   \n",
              "74                 6.4               2.9                4.3               1.3   \n",
              "75                 6.6               3.0                4.4               1.4   \n",
              "76                 6.8               2.8                4.8               1.4   \n",
              "..                 ...               ...                ...               ...   \n",
              "120                6.9               3.2                5.7               2.3   \n",
              "121                5.6               2.8                4.9               2.0   \n",
              "122                7.7               2.8                6.7               2.0   \n",
              "123                6.3               2.7                4.9               1.8   \n",
              "124                6.7               3.3                5.7               2.1   \n",
              "125                7.2               3.2                6.0               1.8   \n",
              "126                6.2               2.8                4.8               1.8   \n",
              "127                6.1               3.0                4.9               1.8   \n",
              "128                6.4               2.8                5.6               2.1   \n",
              "129                7.2               3.0                5.8               1.6   \n",
              "130                7.4               2.8                6.1               1.9   \n",
              "131                7.9               3.8                6.4               2.0   \n",
              "132                6.4               2.8                5.6               2.2   \n",
              "133                6.3               2.8                5.1               1.5   \n",
              "134                6.1               2.6                5.6               1.4   \n",
              "135                7.7               3.0                6.1               2.3   \n",
              "136                6.3               3.4                5.6               2.4   \n",
              "137                6.4               3.1                5.5               1.8   \n",
              "138                6.0               3.0                4.8               1.8   \n",
              "139                6.9               3.1                5.4               2.1   \n",
              "140                6.7               3.1                5.6               2.4   \n",
              "141                6.9               3.1                5.1               2.3   \n",
              "142                5.8               2.7                5.1               1.9   \n",
              "143                6.8               3.2                5.9               2.3   \n",
              "144                6.7               3.3                5.7               2.5   \n",
              "145                6.7               3.0                5.2               2.3   \n",
              "146                6.3               2.5                5.0               1.9   \n",
              "147                6.5               3.0                5.2               2.0   \n",
              "148                6.2               3.4                5.4               2.3   \n",
              "149                5.9               3.0                5.1               1.8   \n",
              "\n",
              "         flower  \n",
              "47       setosa  \n",
              "48       setosa  \n",
              "49       setosa  \n",
              "50   versicolor  \n",
              "51   versicolor  \n",
              "52   versicolor  \n",
              "53   versicolor  \n",
              "54   versicolor  \n",
              "55   versicolor  \n",
              "56   versicolor  \n",
              "57   versicolor  \n",
              "58   versicolor  \n",
              "59   versicolor  \n",
              "60   versicolor  \n",
              "61   versicolor  \n",
              "62   versicolor  \n",
              "63   versicolor  \n",
              "64   versicolor  \n",
              "65   versicolor  \n",
              "66   versicolor  \n",
              "67   versicolor  \n",
              "68   versicolor  \n",
              "69   versicolor  \n",
              "70   versicolor  \n",
              "71   versicolor  \n",
              "72   versicolor  \n",
              "73   versicolor  \n",
              "74   versicolor  \n",
              "75   versicolor  \n",
              "76   versicolor  \n",
              "..          ...  \n",
              "120   virginica  \n",
              "121   virginica  \n",
              "122   virginica  \n",
              "123   virginica  \n",
              "124   virginica  \n",
              "125   virginica  \n",
              "126   virginica  \n",
              "127   virginica  \n",
              "128   virginica  \n",
              "129   virginica  \n",
              "130   virginica  \n",
              "131   virginica  \n",
              "132   virginica  \n",
              "133   virginica  \n",
              "134   virginica  \n",
              "135   virginica  \n",
              "136   virginica  \n",
              "137   virginica  \n",
              "138   virginica  \n",
              "139   virginica  \n",
              "140   virginica  \n",
              "141   virginica  \n",
              "142   virginica  \n",
              "143   virginica  \n",
              "144   virginica  \n",
              "145   virginica  \n",
              "146   virginica  \n",
              "147   virginica  \n",
              "148   virginica  \n",
              "149   virginica  \n",
              "\n",
              "[103 rows x 5 columns]"
            ]
          },
          "execution_count": 104,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "import pandas as pd\ndf = pd.DataFrame(iris.data,columns=iris.feature_names)\ndf['flower'] = iris.target\ndf['flower'] = df['flower'].apply(lambda x: iris.target_names[x])\ndf",
      "id": "174993f5-413b-4ce5-ba31-0e35866ad542"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell4",
        "collapsed": false
      },
      "source": "Let's start by using `train_test_split` to manually tune parameters by trial and error.",
      "id": "264ab30b-72cd-4f8b-99e0-d802cb6ae83d"
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "name": "cell5",
        "language": "python",
        "collapsed": false,
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3)",
      "id": "439ae0dd-2dd8-4153-9edc-48cc665a6868"
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {
        "name": "cell6",
        "language": "python",
        "collapsed": false,
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.9111111111111111"
            ]
          },
          "execution_count": 110,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "from sklearn.model_selection import cross_val_score\n# Build a Support Vector Classification (SVC) model\nmodel = svm.SVC(kernel='rbf',C=30,gamma='auto')\nmodel.fit(X_train,y_train)\nmodel.score(X_test, y_test)",
      "id": "bfcb51f2-c02f-4a57-9f8b-5fde651dfb38"
    },
    {
      "cell_type": "markdown",
      "id": "4c11ca12-a146-4874-94ee-4118c93d1517",
      "metadata": {
        "name": "cell7",
        "collapsed": false
      },
      "source": "It's difficult to guess the right parameters to use for the model, so let's loop over different parameter values and use K Fold cross validation and compute the average score."
    },
    {
      "cell_type": "code",
      "id": "22071ccf-77d1-4de3-89c4-001e8cee21fe",
      "metadata": {
        "language": "python",
        "name": "cell8",
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "# Using a for-loop to iterate over different kernel types and regularization parameter (C) values\nimport numpy as np\nkernels = ['rbf', 'linear']\nC = [1,10,20]\navg_scores = {}\nfor kval in kernels:\n    for cval in C:\n        cv_scores = cross_val_score(svm.SVC(kernel=kval,C=cval,gamma='auto'),iris.data, iris.target, cv=5)\n        avg_scores[kval + '_' + str(cval)] = np.average(cv_scores)\n\navg_scores",
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell9",
        "collapsed": false
      },
      "source": "## Hyperparameter Search with Grid Search\nsklearn's [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) provides a convenient way for us to perform the parameter search without having to write a for-loop.",
      "id": "4cc03092-812a-499a-bd8c-5950b67ebfef"
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "name": "cell10",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'mean_fit_time': array([0.00040021, 0.00079727, 0.00119658, 0.00039496, 0.00080886,\n",
              "        0.00018702]),\n",
              " 'std_fit_time': array([0.00049016, 0.00039864, 0.00039806, 0.00048377, 0.00076687,\n",
              "        0.00037403]),\n",
              " 'mean_score_time': array([0.00059886, 0.00019956, 0.00019951, 0.00019903, 0.00079527,\n",
              "        0.00019898]),\n",
              " 'std_score_time': array([0.00048897, 0.00039911, 0.00039902, 0.00039806, 0.00074207,\n",
              "        0.00039797]),\n",
              " 'param_C': masked_array(data=[1, 1, 10, 10, 20, 20],\n",
              "              mask=[False, False, False, False, False, False],\n",
              "        fill_value='?',\n",
              "             dtype=object),\n",
              " 'param_kernel': masked_array(data=['rbf', 'linear', 'rbf', 'linear', 'rbf', 'linear'],\n",
              "              mask=[False, False, False, False, False, False],\n",
              "        fill_value='?',\n",
              "             dtype=object),\n",
              " 'params': [{'C': 1, 'kernel': 'rbf'},\n",
              "  {'C': 1, 'kernel': 'linear'},\n",
              "  {'C': 10, 'kernel': 'rbf'},\n",
              "  {'C': 10, 'kernel': 'linear'},\n",
              "  {'C': 20, 'kernel': 'rbf'},\n",
              "  {'C': 20, 'kernel': 'linear'}],\n",
              " 'split0_test_score': array([0.96666667, 0.96666667, 0.96666667, 1.        , 0.96666667,\n",
              "        1.        ]),\n",
              " 'split1_test_score': array([1., 1., 1., 1., 1., 1.]),\n",
              " 'split2_test_score': array([0.96666667, 0.96666667, 0.96666667, 0.9       , 0.9       ,\n",
              "        0.9       ]),\n",
              " 'split3_test_score': array([0.96666667, 0.96666667, 0.96666667, 0.96666667, 0.96666667,\n",
              "        0.93333333]),\n",
              " 'split4_test_score': array([1., 1., 1., 1., 1., 1.]),\n",
              " 'mean_test_score': array([0.98      , 0.98      , 0.98      , 0.97333333, 0.96666667,\n",
              "        0.96666667]),\n",
              " 'std_test_score': array([0.01632993, 0.01632993, 0.01632993, 0.03887301, 0.03651484,\n",
              "        0.0421637 ]),\n",
              " 'rank_test_score': array([1, 1, 1, 4, 5, 5])}"
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "from sklearn.model_selection import GridSearchCV\ngs = GridSearchCV(svm.SVC(gamma='auto'), {\n    'C': [1,5,10,20,30,50],\n    'kernel': ['rbf','linear','poly']\n}, cv=5, return_train_score=False)\ngs.fit(iris.data, iris.target)",
      "id": "fd98618f-02e6-4063-908c-cdd361a1d91b"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "name": "cell11",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_C</th>\n",
              "      <th>param_kernel</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>split3_test_score</th>\n",
              "      <th>split4_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.000598</td>\n",
              "      <td>0.000489</td>\n",
              "      <td>0.000598</td>\n",
              "      <td>0.000488</td>\n",
              "      <td>1</td>\n",
              "      <td>rbf</td>\n",
              "      <td>{'C': 1, 'kernel': 'rbf'}</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.980000</td>\n",
              "      <td>0.016330</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000600</td>\n",
              "      <td>0.000490</td>\n",
              "      <td>0.000599</td>\n",
              "      <td>0.000489</td>\n",
              "      <td>1</td>\n",
              "      <td>linear</td>\n",
              "      <td>{'C': 1, 'kernel': 'linear'}</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.980000</td>\n",
              "      <td>0.016330</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.000605</td>\n",
              "      <td>0.000495</td>\n",
              "      <td>0.000394</td>\n",
              "      <td>0.000483</td>\n",
              "      <td>10</td>\n",
              "      <td>rbf</td>\n",
              "      <td>{'C': 10, 'kernel': 'rbf'}</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.980000</td>\n",
              "      <td>0.016330</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.000798</td>\n",
              "      <td>0.000399</td>\n",
              "      <td>0.000200</td>\n",
              "      <td>0.000399</td>\n",
              "      <td>10</td>\n",
              "      <td>linear</td>\n",
              "      <td>{'C': 10, 'kernel': 'linear'}</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.973333</td>\n",
              "      <td>0.038873</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.000205</td>\n",
              "      <td>0.000410</td>\n",
              "      <td>0.000193</td>\n",
              "      <td>0.000386</td>\n",
              "      <td>20</td>\n",
              "      <td>rbf</td>\n",
              "      <td>{'C': 20, 'kernel': 'rbf'}</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>0.036515</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.000200</td>\n",
              "      <td>0.000400</td>\n",
              "      <td>0.000198</td>\n",
              "      <td>0.000397</td>\n",
              "      <td>20</td>\n",
              "      <td>linear</td>\n",
              "      <td>{'C': 20, 'kernel': 'linear'}</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>0.933333</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>0.042164</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
              "0       0.000598      0.000489         0.000598        0.000488       1   \n",
              "1       0.000600      0.000490         0.000599        0.000489       1   \n",
              "2       0.000605      0.000495         0.000394        0.000483      10   \n",
              "3       0.000798      0.000399         0.000200        0.000399      10   \n",
              "4       0.000205      0.000410         0.000193        0.000386      20   \n",
              "5       0.000200      0.000400         0.000198        0.000397      20   \n",
              "\n",
              "  param_kernel                         params  split0_test_score  \\\n",
              "0          rbf      {'C': 1, 'kernel': 'rbf'}           0.966667   \n",
              "1       linear   {'C': 1, 'kernel': 'linear'}           0.966667   \n",
              "2          rbf     {'C': 10, 'kernel': 'rbf'}           0.966667   \n",
              "3       linear  {'C': 10, 'kernel': 'linear'}           1.000000   \n",
              "4          rbf     {'C': 20, 'kernel': 'rbf'}           0.966667   \n",
              "5       linear  {'C': 20, 'kernel': 'linear'}           1.000000   \n",
              "\n",
              "   split1_test_score  split2_test_score  split3_test_score  split4_test_score  \\\n",
              "0                1.0           0.966667           0.966667                1.0   \n",
              "1                1.0           0.966667           0.966667                1.0   \n",
              "2                1.0           0.966667           0.966667                1.0   \n",
              "3                1.0           0.900000           0.966667                1.0   \n",
              "4                1.0           0.900000           0.966667                1.0   \n",
              "5                1.0           0.900000           0.933333                1.0   \n",
              "\n",
              "   mean_test_score  std_test_score  rank_test_score  \n",
              "0         0.980000        0.016330                1  \n",
              "1         0.980000        0.016330                1  \n",
              "2         0.980000        0.016330                1  \n",
              "3         0.973333        0.038873                4  \n",
              "4         0.966667        0.036515                5  \n",
              "5         0.966667        0.042164                5  "
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "# Putting GridSearchCV result into a dataframe\ndf = pd.DataFrame(gs.cv_results_)\ndf",
      "id": "7b5d4228-8d7b-4b80-b52d-43deefff4c33"
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "name": "cell12",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>param_C</th>\n",
              "      <th>param_kernel</th>\n",
              "      <th>mean_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>rbf</td>\n",
              "      <td>0.980000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>linear</td>\n",
              "      <td>0.980000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10</td>\n",
              "      <td>rbf</td>\n",
              "      <td>0.980000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>10</td>\n",
              "      <td>linear</td>\n",
              "      <td>0.973333</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>20</td>\n",
              "      <td>rbf</td>\n",
              "      <td>0.966667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>20</td>\n",
              "      <td>linear</td>\n",
              "      <td>0.966667</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  param_C param_kernel  mean_test_score\n",
              "0       1          rbf         0.980000\n",
              "1       1       linear         0.980000\n",
              "2      10          rbf         0.980000\n",
              "3      10       linear         0.973333\n",
              "4      20          rbf         0.966667\n",
              "5      20       linear         0.966667"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "results = df[['param_C','param_kernel','mean_test_score']]\nresults",
      "id": "0c7c4506-461f-4f07-a137-66c447764134"
    },
    {
      "cell_type": "markdown",
      "id": "bed28995-602a-4445-ae24-459c189f4aa7",
      "metadata": {
        "name": "cell13",
        "collapsed": false
      },
      "source": "Let's plot the scores for each parameter combination in a heatmap to visualize the grid search results: "
    },
    {
      "cell_type": "code",
      "id": "40b83dc6-0357-4ead-8af3-7f4b964135f5",
      "metadata": {
        "language": "python",
        "name": "cell14",
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "import altair as alt\nalt.Chart(results).mark_rect().encode(\n    x='param_C:O',\n    y='param_kernel:O',\n    color='mean_test_score:Q'\n)",
      "execution_count": null
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "name": "cell15",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'C': 1, 'kernel': 'rbf'}"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "print(f\"From GridSearch, this is the best parameter combination {gs.best_params_} and resulting score {gs.best_score_:.2f}.\")",
      "id": "40157ffd-749a-4ab5-b052-4cac6c9401c7"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell16",
        "collapsed": false
      },
      "source": "## Hyperparameter Search with Randomized Search\nGridSearchCV performs an exhaustive search over all parameter combinations. That can become very expensive when there is large numbers of parameters to search through. \n[RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) performs a random search over the space of possible parameters. ",
      "id": "29e45732-181b-4d29-9f89-34010e9c5b90"
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "name": "cell17",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>param_C</th>\n",
              "      <th>param_kernel</th>\n",
              "      <th>mean_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>10</td>\n",
              "      <td>rbf</td>\n",
              "      <td>0.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>linear</td>\n",
              "      <td>0.98</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  param_C param_kernel  mean_test_score\n",
              "0      10          rbf             0.98\n",
              "1       1       linear             0.98"
            ]
          },
          "execution_count": 102,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": "from sklearn.model_selection import RandomizedSearchCV\nrs = RandomizedSearchCV(svm.SVC(gamma='auto'), {\n        'C': [1,5,10,20,30,50],\n        'kernel': ['rbf','linear','poly']\n    }, \n    cv=5, \n    return_train_score=False, \n    n_iter=2\n)\nimport time\nstart = time.time()\nrs.fit(iris.data, iris.target)\nend = time.time()\nprint(f\"RandomizedSearchCV took {end-start:.2f} seconds\")\ndf = pd.DataFrame(rs.cv_results_)[['param_C','param_kernel','mean_test_score']]\ndf",
      "id": "6d49590c-6a90-4a11-ba97-9507402cbde5"
    },
    {
      "cell_type": "code",
      "id": "34a4f5ce-4973-402c-815b-8151b91abaca",
      "metadata": {
        "language": "python",
        "name": "cell18",
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "start = time.time()\ngs.fit(iris.data, iris.target)\nend = time.time()\nprint(f\"GridSearchCV took {end-start:.2f} seconds\")",
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "id": "386ef4a7-5ac8-4463-bf7c-3ef080a0732a",
      "metadata": {
        "name": "cell19",
        "collapsed": false
      },
      "source": "We can see that GridSearchCV takes longer than RandomizedSearchCV. The runtime difference is even larger when you have many parameters to try or if the training time is long (e.g., when you are working with large datasets). Read more about the difference between the two hyperparameter tuning methods [here](https://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html)."
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell20",
        "collapsed": false
      },
      "source": "## Model Comparison with Grid Search\n\nNow that we learned how we can find the optimized parameter for a given model, we can loop through different model and parameter ranges and perform grid search to determine the most optimal parameters for different types of classification models.",
      "id": "b1c54800-2a71-4ff2-8532-7064cd0bc5ed"
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {
        "name": "cell21",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [],
      "source": "from sklearn import svm\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegression\n\nmodel_params = {\n    'svm': {\n        'model': svm.SVC(gamma='auto'),\n        'params' : {\n            'C': [1,5,10,20,30,50],\n            'kernel': ['rbf','linear','poly']\n        }  \n    },\n    'random_forest': {\n        'model': RandomForestClassifier(),\n        'params' : {\n            'n_estimators': [1,5,10]\n        }\n    },\n    'logistic_regression' : {\n        'model': LogisticRegression(solver='liblinear',multi_class='auto'),\n        'params': {\n            'C': [1,5,10]\n        }\n    }\n}\n",
      "id": "0e369727-865b-40e3-b2fa-c39e4cb880a9"
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {
        "scrolled": false,
        "name": "cell22",
        "language": "python",
        "codeCollapsed": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>model</th>\n",
              "      <th>best_score</th>\n",
              "      <th>best_params</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>svm</td>\n",
              "      <td>0.980000</td>\n",
              "      <td>{'C': 1, 'kernel': 'rbf'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>random_forest</td>\n",
              "      <td>0.953333</td>\n",
              "      <td>{'n_estimators': 5}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>logistic_regression</td>\n",
              "      <td>0.966667</td>\n",
              "      <td>{'C': 5}</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 model  best_score                best_params\n",
              "0                  svm    0.980000  {'C': 1, 'kernel': 'rbf'}\n",
              "1        random_forest    0.953333        {'n_estimators': 5}\n",
              "2  logistic_regression    0.966667                   {'C': 5}"
            ]
          },
          "execution_count": 93,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scores = []\n",
        "\n",
        "for model_name, mp in model_params.items():\n",
        "    clf =  GridSearchCV(mp['model'], mp['params'], cv=5, return_train_score=False)\n",
        "    clf.fit(iris.data, iris.target)\n",
        "    scores.append({\n",
        "        'model': model_name,\n",
        "        'best_score': clf.best_score_,\n",
        "        'best_params': clf.best_params_\n",
        "    })\n",
        "    \n",
        "df = pd.DataFrame(scores,columns=['model','best_score','best_params'])\n",
        "df"
      ],
      "id": "9f554154-6e89-4469-9ba9-f65f58c12e74"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "name": "cell23",
        "collapsed": false
      },
      "source": "Based on above, we can determine that SVM with C=1 and Radial Basis Function (RBF) kernel is the most accurate model for this iris flower classification task.",
      "id": "a1c399f8-68d3-469c-90bf-c9174ab4393c"
    },
    {
      "cell_type": "markdown",
      "id": "0a13f084-70a2-4b68-901a-339dda453e6d",
      "metadata": {
        "name": "cell24",
        "collapsed": false
      },
      "source": "## Conclusion\nIn this example, we looked at how you can use scikit-learn to search over a range of parameter combinations and find the best parameter settings for your model using GridSearchCV and RandomizedSearchCV. We also looked at how you can apply this technique to search over different models to look for the best model-parameter combination for your machine learning task. "
    }
  ]
}