lectures/24_deployment-conclusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 Lecture 24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outline:\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements + survey (15 min)\n",
    "- Model deployment (30 min)\n",
    "- Instructor/TA evaluations + Break (10 min)\n",
    "- Review / conclusion (25 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Describe the goals and challenges of model deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import train_test_split, cross_validate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements + survey (15 min)\n",
    "\n",
    "- Last lecture today!\n",
    "- Learning objectives should now be posted for all lectures.\n",
    "- hw7 grades posted\n",
    "- Recordings on README now link to YouTube, which has some parts cut out. See Canvas for whole videos.\n",
    "- Extremely short survey on CPSC 330 vs. 340: https://ubc.ca1.qualtrics.com/jfe/form/SV_2ayfs2EcNUJdYKV\n",
    "- We will take time later for the formal course evaluations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "REMINDER TO RESUME RECORDING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Final exam\n",
    "\n",
    "- Two parts: written part on Canvas (40%), coding part on GitHub (60%)\n",
    "- Both parts will be available from Dec 7 at 6:00pm to Dec 9 at 6:00pm\n",
    "\n",
    "Canvas part:\n",
    "\n",
    "- No time limit within the 48 hours. Poll: https://piazza.com/class/kb2e6nwu3uj23?cid=647\n",
    "- Similar length to the midterm: approximately 20 short-answer questions\n",
    "- Open-book\n",
    "- Answer must be in your own words\n",
    "- Questions will appear in random order\n",
    "- Once you submit an answer you cannot go back\n",
    "- I am guessing the written part will take you 1-2 hours (similar to midterm), but you can take as long as you want.\n",
    "- More emphasis on 2nd half of the course.\n",
    "\n",
    "Coding part:\n",
    "\n",
    "- No time limit within the 48 hours\n",
    "- Submission process exactly the same as assignments (push to github and Canvas)\n",
    "- Make sure you're using the course environment (conda or pip)\n",
    "- Open-book (course notes, internet posts other than you asking a question, anything except another human)\n",
    "- Can use code from class with attribution\n",
    "- Cannot use code from elsewhere\n",
    "- Similar format to [last year's final exam](https://github.com/UBC-CS/cpsc330/blob/master/exams/2019W2/final_exam.ipynb) (except Q7)\n",
    "- I am guessing the coding part will take you 2-3 hours (similar to last year's final), but you can take as long as you want.\n",
    "- Will cover the whole course but more emphasis on the 1st half.\n",
    "\n",
    "Joint rules:\n",
    "\n",
    "- No communication \n",
    "- No public Piazza posts\n",
    "- Questions via private Piazza posts\n",
    "  - I will check several times a day (probably between 7am and 8pm)\n",
    "- There will be an integrity pledge\n",
    "\n",
    "I will write this all up in a Piazza post soon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model deployment (30 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attribution\n",
    "\n",
    "This material adapted from the [model deployment tutorial](https://github.com/TomasBeuzen/machine-learning-tutorials/blob/master/ml-deploy-model/deploy-with-flask.ipynb) by [Tomas Beuzen](https://www.tomasbeuzen.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What is deployment?\n",
    "\n",
    "- After we train a model, we want to use it!\n",
    "- The user likely does not want to install your Python stack, train your model.\n",
    "- You don't necessarily want to share your dataset.\n",
    "- So we need to do two things:\n",
    "\n",
    "1. Save/store your model for later use.\n",
    "2. Make the saved model conveniently accessible.\n",
    "\n",
    "We will use [Joblib](https://joblib.readthedocs.io/) for (1) and [Flask](https://flask.palletsprojects.com/) & [Heroku](https://www.heroku.com/) for (2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Requirements (I already did these)\n",
    "\n",
    "- Heroku account. Register [here](https://www.heroku.com/).\n",
    "- Heroku CLI. Download [here](https://devcenter.heroku.com/categories/command-line)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More python installations (not in the course environment):\n",
    "\n",
    "```\n",
    "pip install Flask\n",
    "pip install Flask-WTF\n",
    "pip install joblib\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preparing the model we wish to deploy\n",
    "\n",
    "We'll be training a regression model to predict the age of abalone based on the classic abalone dataset hosted [here](https://archive.ics.uci.edu/ml/datasets/abalone). We aim to predict the age of abalone based on four physical measurements. I've renamed `abalone.data` to `abalone.csv` after downloading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "abalone_df = pd.read_csv('data/abalone.csv',\n",
    "                       names = ['sex', 'length', 'diameter', 'height',\n",
    "                                'whole_weight', 'shucked_weight', 'viscera_weight',\n",
    "                                'shell_weight', 'rings'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, only use 4 features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['length', 'diameter', 'height', 'whole_weight']\n",
    "\n",
    "X = abalone_df[features]\n",
    "y = abalone_df['rings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>length</th>\n",
       "      <th>diameter</th>\n",
       "      <th>height</th>\n",
       "      <th>whole_weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3906</th>\n",
       "      <td>0.245</td>\n",
       "      <td>0.180</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.0635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2562</th>\n",
       "      <td>0.440</td>\n",
       "      <td>0.325</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.4165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2197</th>\n",
       "      <td>0.405</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.105</td>\n",
       "      <td>0.3625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1405</th>\n",
       "      <td>0.655</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.205</td>\n",
       "      <td>1.6445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1903</th>\n",
       "      <td>0.575</td>\n",
       "      <td>0.445</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.8760</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      length  diameter  height  whole_weight\n",
       "3906   0.245     0.180   0.065        0.0635\n",
       "2562   0.440     0.325   0.100        0.4165\n",
       "2197   0.405     0.305   0.105        0.3625\n",
       "1405   0.655     0.535   0.205        1.6445\n",
       "1903   0.575     0.445   0.145        0.8760"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3906     4\n",
       "2562     6\n",
       "2197    10\n",
       "1405    13\n",
       "1903    10\n",
       "Name: rings, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build and score model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RandomForestRegressor(n_estimators=10, random_state=123)\n",
    "model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8716166306386149"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27082133598283975"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We'll re-fit the model on the full dataset to get it ready for deployment. \n",
    "  - This is probably a good idea, because more data is better.\n",
    "  - It's also a little scary, because we can't test this new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RandomForestRegressor(n_estimators=10, random_state=123)\n",
    "model.fit(X, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save trained model using `joblib`. This will be loaded up when we start our \"app\": "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('web_api/abalone_predictor.joblib', 'wb') as f:\n",
    "    joblib.dump(model, f)\n",
    "with open('web_application/abalone_predictor.joblib', 'wb') as f:\n",
    "    joblib.dump(model, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define a function that accepts input data as a dictionary and returns a prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def return_prediction(model, input_json):\n",
    "    \n",
    "    input_data = [[input_json[k] for k in features]]\n",
    "    prediction = model.predict(input_data)[0]\n",
    "    \n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_input_json = {\n",
    "    'length': 0.41,\n",
    "    'diameter': 0.33,\n",
    "    'height': 0.10,\n",
    "    'whole_weight': 0.36\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.6"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "return_prediction(model, example_input_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function appears in the `app.py` that we'll be using shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Setting up a directory structure and environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We need a specific directory structure to help us easily deploy our machine learning model. \n",
    "- This is already set up in this repo.\n",
    "\n",
    "```shell\n",
    "lectures\n",
    "├── web_api\n",
    "│   └── abalone_predictor.joblib  # this is the machine learning model we have built locally\n",
    "│   └── app.py  # the file that defines our flask API\n",
    "│   └── Procfile  # required by Heroku to help start flask app\n",
    "│   └── requirements.txt  # file containing required packages\n",
    "│   \n",
    "└── web_application\n",
    "    └── abalone_predictor.joblib  # this is the machine learning model we have built locally\n",
    "    └── app.py  # the file that defines our flask API\n",
    "    └── Procfile  # required by Heroku to help start flask app\n",
    "    └── requirements.txt  # file containing required packages\n",
    "    └── templates  # this subdirectory contains HTML templates to help us build the web application\n",
    "    │   └── style.css  # css template to be used in web application\n",
    "    └── static  # this subdirectory contains CSS style sheets\n",
    "        └── home.html  # html template to be used in web application\n",
    "        └── prediction.html  # html template to be used in web application\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two options for deploying our abalone prediction model. We can:\n",
    "\n",
    "1. Develop a RESTful web API that accepts HTTP requests in the form of input data and returns a prediction.\n",
    "2. Build a web application with a HTML user-interface that interacts directly with our API.\n",
    "\n",
    "We'll explore both options below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building and deploying a web API\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| API  |    you are here          |                          |\n",
    "| app  |                          |                          |\n",
    "\n",
    "- I have a separate Python file called `app.py` that handles this part.\n",
    "- We can open it up here in Jupyter Lab and take a look.\n",
    "- If we run `python app.py` we'll bring it to life.\n",
    "  - If you get an error, you may need to install those extra packages and make sure you have the environment loaded.\n",
    "- We won't go into details here. If you want to learn more about Flask, see:\n",
    "  - [Flask tutorial video series by Corey Schafer](https://www.youtube.com/playlist?list=PL-osiE80TeTs4UjLw5MM6OjgkjFeUxCYH)\n",
    "  - [Flask docs](https://flask.palletsprojects.com/en/1.1.x/)\n",
    "  - [Flask tutorial by Miguel Grinberg](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)\n",
    "- If you don't know what an API is, that's OK.\n",
    "  - For our purposes, it's something that exists at a particular address, that can accept information and return information.\n",
    "  - Sort of like a function but not Python-specific and potentially accessible by anyone on the internet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing that the Flask API is alive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's test this out in a terminal here in Jupyter Lab. \n",
    "- Ok, now let's go to the URL `http://127.0.0.1:5000/`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sending a request to the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.6\n"
     ]
    }
   ],
   "source": [
    "!curl -d '{\"length\":0.41,\"diameter\":0.33,\"height\":0.10,\"whole_weight\":0.36}' \\\n",
    "      -H \"Content-Type: application/json\" \\\n",
    "      -X POST http://localhost:5000/predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Or, we can open up another Terminal tab and do it there)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploying the API on a server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| API  |                          |      you are here   |\n",
    "| app  |                          |                          |\n",
    "\n",
    "- Okay, so we have a working API running on localhost, but we don't want to host this service on my laptop!\n",
    "- We now want to deploy it on a \"real\" server so others can send it requests. \n",
    "- We will use Heroku to deploy our app but you could also use other services such as AWS.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Heroku set-up (I already did these):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Go to [Heroku](https://dashboard.heroku.com/), log-in, and click \"Create new app\".\n",
    "2. Choose a unique name for your app.\n",
    "3. Create app.\n",
    "\n",
    "<img src=\"img/flask_images/fl_6.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We will be using the Heroku CLI to deploy our model. \n",
    "- We'll open up another terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "heroku login\n",
    "cd my-project/\n",
    "git init\n",
    "heroku git:remote -a my-abalone-predictor\n",
    "git add .\n",
    "git commit -am \"Initial commit\"\n",
    "git push heroku master\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that for more complex applications, you may choose to containerize everything in a Docker container to deploy to Heroku)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the API on Heroku"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.6\n"
     ]
    }
   ],
   "source": [
    "!curl -d '{\"length\":0.41,\"diameter\":0.35,\"height\":0.12,\"whole_weight\":0.36}' \\\n",
    "      -H \"Content-Type: application/json\" \\\n",
    "      -X POST https://my-abalone-predictor.herokuapp.com/predict "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- OK so what this means is that anyone can do this.\n",
    "- In fact, you all have your laptops - give it a try!\n",
    "- You can also do the `curl` from a terminal:\n",
    "\n",
    "```\n",
    "curl -d '{\"length\":0.41,\"diameter\":0.33,\"height\":0.10,\"whole_weight\":0.36}' \\\n",
    "     -H \"Content-Type: application/json\" \\ \n",
    "     -X POST https://my-abalone-predictor.herokuapp.com/predict \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/mike_highfive.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it for the API approach. Next:\n",
    "\n",
    "#### Building and deploying a web application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| API  |                          |                          |\n",
    "| app  |   you are here           |                          |\n",
    "\n",
    "- Flask can create entire web applications.\n",
    "- We only need to refactor our code a little bit and link it up with some html and css to create our web application.\n",
    "- We will use Flask to create a html form, accept data submitted to the form, and return a prediction using the submitted data. \n",
    "- Again, I won't go into too much detail here, but we can open up `web_application/` and take a quick look."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the web application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's terminate our API Flask app, navigate to `../web_application`, and run again.\n",
    "- Now let's go back to `http://127.0.0.1:5000/`.\n",
    "\n",
    "We can try it again on localhost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploying the web application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| API  |                          |                          |\n",
    "| app  |                        |          you are here       |\n",
    "\n",
    "- I already logged in to Heroku and created the app.\n",
    "- Now the same commands:\n",
    "\n",
    "```\n",
    "heroku login\n",
    "cd my-project/\n",
    "git init\n",
    "heroku git:remote -a my-abalone-web-app\n",
    "git add .\n",
    "git commit -am \"Initial commit\"\n",
    "git push heroku master\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's try it out: https://my-abalone-web-app.herokuapp.com/\n",
    "- You can try it too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "\n",
    "- There are many ways to deploy a model; a RESTful API is very common and convenient. \n",
    "- As you can see, a simple deployment is fairly straightward. \n",
    "- However, there may be other considerations such as:\n",
    "  - Privacy/security\n",
    "  - Scaling\n",
    "  - Error handling\n",
    "  - Real-time / speed\n",
    "  - Low-resource environments (e.g. edge computing)\n",
    "  - etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (10 min)\n",
    "\n",
    "- We'll take a longer break today.\n",
    "- Consider taking this time to fill out the instructor/TA evaluations if you haven't already.\n",
    "- You may have seen [my post about these evaluations](https://www.reddit.com/r/UBC/comments/k18qj7/teaching_evaluations_the_good_the_bad_and_the_ugly/) on r/ubc.\n",
    "\n",
    "Evaluation link: https://canvas.ubc.ca/courses/53561/external_tools/4732"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Course review / conclusion (25 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Learning objectives\n",
    "\n",
    "Here are the course learning outcomes I came up with when proposing this new course:\n",
    "\n",
    "1. Identify problems that may be addressed with machine learning.\n",
    "2. Select the appropriate machine learning tool for a problem.\n",
    "3. Transform data of various types into usable features.\n",
    "4. Apply standard tools implementing supervised and unsupervised learning techniques.\n",
    "5. Describe core differences between training, validation, and testing regimes.\n",
    "6. Effectively communicate the results of a machine learning pipeline.\n",
    "7. Be realistic about the limitations of individual approaches and machine learning as a whole. \n",
    "8. Create reproducible workflows and pipelines.\n",
    "\n",
    "- How did we do? \n",
    "- Hopefully OK, except we skipped the last point (that will likely be its own new course).\n",
    "- I would also add:\n",
    "\n",
    "9. Identify and avoid scenarios in which training and testing data are accidentally mixed (the \"Golden Rule\").\n",
    "10. Employ good habits for applying ML, such as starting an analysis with a baseline estimator.\n",
    "\n",
    "because I think they are important enough to make it to the course-level list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What did we cover?\n",
    "\n",
    "I see the course roughly like this (not in order):\n",
    "\n",
    "Part 1: Supervised learning on tabular data\n",
    "\n",
    "- Overfitting, train/validation/test/deployment, cross-validation\n",
    "- Feature preprocessing, pipelines, imputation, OHE, etc\n",
    "- The Golden Rule, various ways to accidentally violate it\n",
    "- Classification metrics: confusion matrix, precision/recall, ROC, AUC\n",
    "- Regression metrics: MSE, MAPE\n",
    "- Regression: transforming the targets\n",
    "- Feature importances, feature selection\n",
    "- Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2: Other data types (non-tabular)\n",
    "\n",
    "- Computer vision with deep learning\n",
    "- Language data, text preprocessing\n",
    "- Ratings data\n",
    "- Time series\n",
    "- Right-censored data / survival analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also: Other stuff\n",
    "\n",
    "- Ensembles\n",
    "- Outlier detection\n",
    "- Clustering\n",
    "- A bunch of models: \n",
    "  - baselines\n",
    "  - linear models (ridge, lasso, huber, logistic regression, SGD)\n",
    "  - tree-based models (random forest, gradient boosted trees)\n",
    "  - KNN classifier/regressor\n",
    "  - pre-trained deep learning models\n",
    "- Communicating your results (including visualizations)\n",
    "- ML skepticism\n",
    "- Ethics for ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Some key takeaways\n",
    "\n",
    "Some useful guidelines:\n",
    "\n",
    "- Do train-test split right away and only once\n",
    "- Don't look at the test set until the end\n",
    "- Don't call `fit` on test/validation data\n",
    "- Use pipelines\n",
    "- Use baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recipe to approach a supervised learning problem with tabular data:\n",
    "\n",
    "1. Have a long conversation with the stakeholder(s) who will be using your pipeline.\n",
    "2. Have a long conversation with the person(s) who collected the data.\n",
    "3. Think about the ethical implications - are you sure you want to do this project? If so, should ethics guide your approach?\n",
    "4. Random train-test split with fixed random seed; do not touch the test data until Step 16.\n",
    "5. Exploratory data analysis, outlier detection.\n",
    "6. Choose a scoring metric -> higher values should make you & your stakeholders happier.\n",
    "7. Fit a baseline model, e.g. `DummyClassifier` or `DummyRegressor`.\n",
    "8. Create a preprocessing pipeline. May involve feature engineering. (This is usually a time-consuming step!)\n",
    "9. Try a linear model, e.g. `LogisticRegression` or `Ridge`; tune hyperparameters with CV.\n",
    "10. Try other sensible model(s), e.g. LightGBM; tune hyperparameters with CV.\n",
    "11. For each model, look at sub-scores from the folds of cross-validation to get a sense of \"error bars\" on the scores.\n",
    "12. Pick a model that you like. Best CV score is a reasonable metric, though you may choose to favour simpler models.\n",
    "13. Look at feature importances.\n",
    "14. (optional) Perform some more diagnostics like confusion matrix for classification, or \"predicted vs. true\" scatterplots for regression.\n",
    "15. (optional) Try to calibrate the uncertainty/confidence outputted by your model.\n",
    "16. Test set evaluation.\n",
    "17. Question everything again: validity of results, bias/fairness of trained model, etc.\n",
    "18. Discuss your results with stakeholders.\n",
    "19. (optional) Retrain on all your data.\n",
    "20. Deployment & integration.\n",
    "21. Profit?\n",
    "\n",
    "PS: the order of steps is approximate, and some steps may need to be repeated during prototyping, experimentation, and as needed over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What would I do differently?\n",
    "\n",
    "- Find a dataset with multi-class classification for an early part of the course.\n",
    "- Reordering the material a bit:\n",
    "  - Move \"feature importances for computer vision\" into computer vision lecture (not ethics).\n",
    "  - Introduce random forests and feature importances a bit earlier\n",
    "  - Move outlier lecture much earlier\n",
    "- Allocate 2 lectures to time series data \n",
    "\n",
    "I'm sure you have other suggestions - feel free to drop me an email, submit my contact form anonymously at mikegelbart.com, or drop them in the course evaluations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 330 vs. 340\n",
    "\n",
    "- I am hoping lots of people will take both courses.\n",
    "- There is some overlap but not a crazy amount (I hope).\n",
    "- If you want to learn how these methods work under the hood, CPSC 340 will give you a lot of that, such as:\n",
    "  - Implementing `Ridge.fit()` from scratch\n",
    "  - Mathematically speaking, what is `C` in `LogisticRegression`?\n",
    "  - How fast do these algorithms run in terms of the number of rows and columns of your dataset? \n",
    "  - Etc.\n",
    "- There are also a bunch of other methods covered. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unsolicited advice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- I sometimes end my courses with \"unsolicited life advice\".\n",
    "- Today I will just say let's take care of ourselves and each other and try to get through this.\n",
    "- If you're interested, the advice from CPSC 340 a couple years ago [is on YouTube](https://www.youtube.com/watch?v=_7zYxpzrKmQ&list=PLWmXHcz_53Q02ZLeAxigki1JZFfCO6M-b&index=34&t=0s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion & farewell\n",
    "\n",
    "That's all, folks. You made it!\n",
    "\n",
    "<table style=\"float:left\"><tr>\n",
    "<td><img src=\"img/mike_hanginthere.png\"/></td>\n",
    "<td><img src=\"img/mike_believeinyou.png\"/></td>\n",
    "</tr></table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I believe in you! Thank you for believing in me and entrusting me with this part of your education."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}