update lecture 16

Author: kuleshov

slides/lecture16-boosting.ipynb

@@ -19,6 +19,22 @@
     "__Volodymyr Kuleshov__<br>Cornell Tech"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Announcement\n",
+    "\n",
+    "* Project proposals have been graded over the weekend. \n",
+    "* Milestone due on 11/7.\n",
+    "* HW3 will be graded next week. Prelim will be graded after that.\n",
+    "* We will hold review session for the prelim."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -70,7 +86,7 @@
     "    ensemble.append(model)\n",
     "\n",
     "# output average prediction at test time:\n",
-    "y_test = ensemble.average_prediction(y_test)\n",
+    "y_test = ensemble.average_prediction(x_test)\n",
     "```\n",
     "<!-- Data samples taken with replacement are known as bootstrap samples. -->"
    ]
@@ -162,7 +178,7 @@
    "source": [
     "# Structure of a Boosting Algorithm\n",
     "\n",
-    "Boosting reduces *underfitting* by combining models that correct each others' errors."
+    "Boosting reduces *underfitting* via models that correct each others' errors."
    ]
   },
   {
@@ -173,7 +189,7 @@
     }
    },
    "source": [
-    "1. Fit a weak learner $g_0$ on dataset $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$. Let $f=g_0$."
+    "1. Compute weights $w^{(i)}$ for each $i$ based on $t$-th model predictions $f_t(x^{(i)})$ and targets $y^{(i)}$. Give more weight to points with errors."
    ]
   },
   {
@@ -184,7 +200,7 @@
     }
    },
    "source": [
-    "2. Compute weights $w^{(i)}$ for each $i$ based on model predictions $f(x^{(i)})$ and targets $y^{(i)}$. Give more weight to points with errors."
+    "2. Fit new weak learner $g_t$ on $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$ with weights $w^{(i)}$."
    ]
   },
   {
@@ -195,18 +211,7 @@
     }
    },
    "source": [
-    "3. Fit new weak learner $g_1$ on $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$ with weights $w^{(i)}$."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "fragment"
-    }
-   },
-   "source": [
-    "4. Set $f_1 = g_0 + \\alpha_1 g$ for some weight $\\alpha_1$. Go to Step 2 and repeat."
+    "3. Set $f_{t+1} = f_t + \\alpha_t g_t$ for some weight $\\alpha_t$. Go to Step 1 and repeat."
    ]
   },
   {
@@ -735,7 +740,7 @@
     }
    },
    "source": [
-    "* $f(x)$ consists of $T$ smaller models $g$ with weights $\\alpha_t$ and params $\\phi_t$."
+    "* $f(x)$ consists of $T$ smaller models $g$ with weights $\\alpha_t$ & params $\\phi_t$."
    ]
   },
   {
@@ -781,18 +786,7 @@
     }
    },
    "source": [
-    "1. Fit a weak learner $g_0$ on dataset $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$. Let $f=g_0$."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "fragment"
-    }
-   },
-   "source": [
-    "2. Compute weights $w^{(i)}$ for each $i$ based on model predictions $f(x^{(i)})$ and targets $y^{(i)}$. Give more weight to points with errors."
+    "1. Compute weights $w^{(i)}$ for each $i$ based on $t$-th model predictions $f_t(x^{(i)})$ and targets $y^{(i)}$. Give more weight to points with errors."
    ]
   },
   {
@@ -803,7 +797,7 @@
     }
    },
    "source": [
-    "3. Fit new weak learner $g_1$ on $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$ with weights $w^{(i)}$."
+    "2. Fit new weak learner $g_t$ on $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\}$ with weights $w^{(i)}$."
    ]
   },
   {
@@ -814,7 +808,7 @@
     }
    },
    "source": [
-    "4. Set $f_1 = g_0 + \\alpha_1 g$ for some weight $\\alpha_1$. Go to Step 2 and repeat."
+    "3. Set $f_{t+1} = f_t + \\alpha_t g_t$ for some weight $\\alpha_t$. Go to Step 1 and repeat."
    ]
   },
   {
@@ -887,8 +881,8 @@
    },
    "source": [
     "The resulting algorithm is often called L2Boost. At step $t$ we minimize\n",
-    "$$\\sum_{i=1}^n (r^{(i)}_t - g(x^{(i)}; \\phi))^2, $$\n",
-    "where $r^{(i)}_t = y^{(i)} - f(x^{(i)})_{t-1}$ is the residual from the model at time $t-1$."
+    "$$\\sum_{i=1}^n (r^{(i)}_t - \\alpha g(x^{(i)}; \\phi))^2, $$\n",
+    "where $r^{(i)}_t = y^{(i)} - f(x^{(i)})_{t-1}$ is the residual from the model $f_{t-1}$."
    ]
   },
   {
@@ -1416,7 +1410,7 @@
    },
    "source": [
     "At step $t$ we minimize\n",
-    "$$\\sum_{i=1}^n (r^{(i)}_t - g(x^{(i)}; \\phi))^2, $$\n",
+    "$$\\sum_{i=1}^n (r^{(i)}_t - \\alpha g(x^{(i)}; \\phi))^2, $$\n",
     "where $r^{(i)}_t = y^{(i)} - f_{t-1}(x^{(i)})$ is the residual error of the model $f_{t-1}$."
    ]
   },
@@ -1936,7 +1930,7 @@
    "source": [
     "We will then perform approximate functional gradient descent using\n",
     "\n",
-    "$$f_t \\gets f_{t-1} - \\alpha_t \\nabla g_t$$\n",
+    "$$f_t \\gets f_{t-1} - \\alpha_t g_t$$\n",
     "\n",
     "which is approximately $f_t \\gets f_{t-1} - \\alpha_t \\nabla J(f_{t-1}).$"
    ]
@@ -2158,7 +2152,7 @@
    },
    "source": [
     "At step $t$ we minimize\n",
-    "$$\\sum_{i=1}^n (r^{(i)}_t - g(x^{(i)}; \\phi))^2, $$\n",
+    "$$\\sum_{i=1}^n (r^{(i)}_t - \\alpha g(x^{(i)}; \\phi))^2, $$\n",
     "where $r^{(i)}_t = y^{(i)} - f_{t-1}(x^{(i)})$ is the residual from $f_{t-1}$."
    ]
   },