lectures/16_natural-language-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 16: Introduction to natural language processing \n",
    "\n",
    "UBC 2021-22\n",
    "\n",
    "Instructor: Giulia Toti"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "## Imports\n",
    "\n",
    "import os\n",
    "import re\n",
    "import string\n",
    "import sys\n",
    "import time\n",
    "from collections import Counter, defaultdict\n",
    "\n",
    "import IPython\n",
    "import nltk\n",
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "from ipywidgets import interactive\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Announcements\n",
    "\n",
    "- Final exam: December 19th at 12:00pm [PHRM 1101](https://learningspaces.ubc.ca/classrooms/phrm-1101)\n",
    "- Homework 5 grades have been posted. \n",
    "- Homework 6 was due on Monday, Nov 15th at 11:59pm (submission window extended till November 16 at 10am.)\n",
    "- Homework 7 due Wednesday, Nov 17th at 11:59pm\n",
    "- It seems like many of you are quite overwhelmed. So I decided to skip one homework. We'll only have one more homework after this.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Broadly explain what is natural language processing (NLP). \n",
    "- Name some common NLP applications. \n",
    "- Explain the general idea of a vector space model.\n",
    "- Explain the difference between different word representations: term-term co-occurrence matrix representation and Word2Vec representation.\n",
    "- Describe the reasons and benefits of using pre-trained embeddings. \n",
    "- Load and use pre-trained word embeddings to find word similarities and analogies. \n",
    "- Demonstrate biases in embeddings and learn to watch out for such biases in pre-trained embeddings.\n",
    "- Use word embeddings in text classification and document clustering using `spaCy`.\n",
    "- Explain the general idea of topic modeling. \n",
    "- Describe the input and output of topic modeling. \n",
    "- Carry out basic text preprocessing using `spaCy`.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is Natural Language Processing (NLP)?\n",
    "\n",
    "- What should a search engine return when asked the following question? \n",
    "\n",
    "![](img/lexical_ambiguity.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/lexical_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is Natural Language Processing (NLP)?\n",
    "#### How often do you search everyday? \n",
    "\n",
    "![](img/Google_search.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/Google_search.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is Natural Language Processing (NLP)?\n",
    "\n",
    "![](img/WhatisNLP.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/WhatisNLP.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Everyday NLP applications\n",
    "\n",
    "![](img/annotation-image.png)\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/annotation-image.png\" height=\"1200\" width=\"1200\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### NLP in news \n",
    "\n",
    "Often you'll hear about NLP in news. Some examples: \n",
    "- [How suicide prevention is getting a boost from artificial intelligence](https://abcnews.go.com/GMA/Wellness/suicide-prevention-boost-artificial-intelligence-exclusive/story?id=76541481)\n",
    "- [Meet GPT-3. It Has Learned to Code (and Blog and Argue).](https://www.nytimes.com/2020/11/24/science/artificial-intelligence-ai-gpt3.html)\n",
    "- [How Do You Know a Human Wrote This?](https://www.nytimes.com/2020/07/29/opinion/gpt-3-ai-automation.html)\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why is NLP hard?\n",
    "\n",
    "- Language is complex and subtle. \n",
    "- Language is ambiguous at different levels. \n",
    "- Language understanding involves common-sense knowledge and real-world reasoning.\n",
    "- All the problems related to representation and reasoning in artificial intelligence arise in this domain. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Lexical ambiguity\n",
    "\n",
    "<br><br>\n",
    "\n",
    "![](img/lexical_ambiguity.png)\n",
    "\n",
    "<!-- <img src=\"img/lexical_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Referential ambiguity\n",
    "<br><br>\n",
    "\n",
    "<!-- <img src=\"img/referential_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "\n",
    "![](img/referential_ambiguity.png)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [Ambiguous news headlines](http://www.fun-with-words.com/ambiguous_headlines.html)\n",
    "\n",
    "<blockquote>\n",
    "PROSTITUTES APPEAL TO POPE\n",
    "</blockquote>    \n",
    "\n",
    "- **appeal to** means make a serious or urgent request or be attractive or interesting?\n",
    "\n",
    "<blockquote>\n",
    "KICKING BABY CONSIDERED TO BE HEALTHY    \n",
    "</blockquote> \n",
    "\n",
    "- **kicking** is used as an adjective or a verb?\n",
    "\n",
    "<blockquote>\n",
    "MILK DRINKERS ARE TURNING TO POWDER\n",
    "</blockquote>\n",
    "\n",
    "- **turning** means becoming or take up?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overall goal\n",
    "\n",
    "- Give you a quick introduction to you of this important field in artificial intelligence which extensively used machine learning.   \n",
    "\n",
    "![](img/NLP_in_industry.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/NLP_in_industry.png\" width=\"900\" height=\"800\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Today's plan\n",
    "\n",
    "- Word embeddings\n",
    "- Topic modeling \n",
    "- Basic text preprocessing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embeddings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The idea is to represent word meaning so that similar words are close together. \n",
    "\n",
    "![](img/t-SNE_word_embeddings.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/t-SNE_word_embeddings.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->    \n",
    "\n",
    "(Attribution: Jurafsky and Martin 3rd edition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why do we care about word representation?\n",
    "\n",
    "- So far we have been talking about sentence or document representation. \n",
    "- Now we are going one step back and talking about word representation. \n",
    "- Although word representation cannot be directly used in text classification tasks such as sentiment analysis using tradition ML models, it's good to know about word embeddings because they are so widely used. \n",
    "- They are quite useful in more advanced machine learning models such as recurrent neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word meaning \n",
    "\n",
    "- A favourite topic of philosophers for centuries. \n",
    "- An example from legal domain: [Are hockey gloves gloves or \"articles of plastics\"?](https://www.scc-csc.ca/case-dossier/info/sum-som-eng.aspx?cas=36258)\n",
    "\n",
    "<blockquote>\n",
    "Canada (A.G.) v. Igloo Vikski Inc. was a tariff code case that made its way to the SCC (Supreme Court of Canada). The case disputed the definition of hockey gloves as either gloves or as \"articles of plastics.\"\n",
    "</blockquote>\n",
    "\n",
    "![](img/hockey_gloves_case.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/hockey_gloves_case.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word meaning: ML and NLP view\n",
    "- Modeling word meaning that allows us to \n",
    "    * draw useful inferences to solve meaning-related problems \n",
    "    * find relationship between words, \n",
    "        * E.g., which words are similar, which ones have positive or negative connotations\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How do we represent words?  \n",
    "\n",
    "- Suppose you are building a question answering system and you are given the following question and three candidate answers. \n",
    "- What kind of relationship between words would we like our representation to capture in order to arrive at the correct answer?  \n",
    "    \n",
    "<blockquote>       \n",
    "<p style=\"font-size:30px\"><b>Question:</b> How <b>tall</b> is Machu Picchu?</p>\n",
    "    <p style=\"font-size:30px\"><b>Candidate 1:</b> Machu Picchu is 13.164 degrees south of the equator.</p>    \n",
    "<p style=\"font-size:30px\"><b>Candidate 2:</b> The official height of Machu Picchu is 2,430 m.</p>\n",
    "<p style=\"font-size:30px\"><b>Candidate 3:</b> Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.</p>    \n",
    "</blockquote> \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Need a representation that captures relationships between words.\n",
    "\n",
    "- We will be looking at two such representations.  \n",
    "    1. Sparse representation with **term-term co-occurrence matrix**\n",
    "    2. Dense representation with **Word2Vec**\n",
    "- Both are based on two ideas: **distributional hypothesis** and **vector space model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Distributional hypothesis\n",
    "\n",
    "<blockquote> \n",
    "    <p>You shall know a word by the company it keeps.</p>\n",
    "    <footer>Firth, 1957</footer>        \n",
    "</blockquote>\n",
    "\n",
    "<blockquote> \n",
    "If A and B have almost identical environments we say that they are synonyms.\n",
    "<footer>Harris, 1954</footer>    \n",
    "</blockquote>    \n",
    "\n",
    "Example: \n",
    "\n",
    "- Her **child** loves to play in the playground. \n",
    "- Her **kid** loves to play in the playground. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vector space model\n",
    "\n",
    "- Model the meaning of a word by placing it into a vector space.  \n",
    "- A standard way to represent meaning in NLP\n",
    "- The idea is to create **embeddings of words** so that distances among words in the vector space indicate the relationship between them. \n",
    "\n",
    "![](img/t-SNE_word_embeddings.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/t-SNE_word_embeddings.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->    \n",
    "\n",
    "(Attribution: Jurafsky and Martin 3rd edition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Term-term co-occurrence matrix\n",
    "\n",
    "- So far we have been talking about documents and we created document-term co-occurrence matrix (e.g., bag-of-words representation of text). \n",
    "- We can also do this with words. The idea is to go through a corpus of text, keeping a count of all of the words that appear in context of each word (within a window).\n",
    "\n",
    "- An example: \n",
    "\n",
    "![](img/term-term_comat.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/term-term_comat.png\" width=\"600\" height=\"600\"> -->\n",
    "<!-- </center> -->\n",
    "(Credit: Jurafsky and Martin 3rd edition)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Visualizing word vectors and similarity \n",
    "\n",
    "![](img/word_vectors_and_angles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/word_vectors_and_angles.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->\n",
    "    \n",
    "(Credit: Jurafsky and Martin 3rd edition)\n",
    "\n",
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualizing word vectors and similarity\n",
    "\n",
    "![](img/word_vectors_and_angles.png)\n",
    "\n",
    "<!-- <img src=\"img/word_vectors_and_angles.png\" width=\"600\" height=\"600\"> -->\n",
    "\n",
    "(Credit: Jurafsky and Martin 3rd edition)\n",
    "\n",
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words.\n",
    "\n",
    "- We can also calculate a normalized version of dot products. \n",
    "    $$similarity_{cosine}(w_1,w_2) = \\frac{w_1.w_2}{\\left\\lVert w_1\\right\\rVert_2 \\left\\lVert w_2\\right\\rVert_2}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tall</th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>13.164</th>\n",
       "      <th>degrees</th>\n",
       "      <th>south</th>\n",
       "      <th>equator</th>\n",
       "      <th>official</th>\n",
       "      <th>height</th>\n",
       "      <th>2,430</th>\n",
       "      <th>...</th>\n",
       "      <th>mean</th>\n",
       "      <th>sea</th>\n",
       "      <th>level</th>\n",
       "      <th>1,000</th>\n",
       "      <th>3,300</th>\n",
       "      <th>ft</th>\n",
       "      <th>lower</th>\n",
       "      <th>elevation</th>\n",
       "      <th>3,400</th>\n",
       "      <th>11,200</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tall</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13.164</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>degrees</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         tall  machu  picchu  13.164  degrees  south  equator  official  \\\n",
       "tall        0      1       1       0        0      0        0         0   \n",
       "machu       1      0       5       1        1      0        0         1   \n",
       "picchu      1      5       0       1        1      1        0         1   \n",
       "13.164      0      1       1       0        1      1        1         0   \n",
       "degrees     0      1       1       1        0      1        1         0   \n",
       "\n",
       "         height  2,430  ...  mean  sea  level  1,000  3,300  ft  lower  \\\n",
       "tall          0      0  ...     0    0      0      0      0   0      0   \n",
       "machu         1      2  ...     0    0      0      0      0   0      0   \n",
       "picchu        1      2  ...     0    0      0      0      0   0      0   \n",
       "13.164        0      0  ...     0    0      0      0      0   0      0   \n",
       "degrees       0      0  ...     0    0      0      0      0   0      0   \n",
       "\n",
       "         elevation  3,400  11,200  \n",
       "tall             0      0       0  \n",
       "machu            0      0       0  \n",
       "picchu           0      0       0  \n",
       "13.164           0      0       0  \n",
       "degrees          0      0       0  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Let's build term-term co-occurrence matrix for our text.\n",
    "sys.path.append(\"code/.\")\n",
    "\n",
    "from comat import CooccurrenceMatrix\n",
    "from preprocessing import MyPreprocessor\n",
    "\n",
    "corpus = [\n",
    "    \"How tall is Machu Picchu?\",\n",
    "    \"Machu Picchu is 13.164 degrees south of the equator.\",\n",
    "    \"The official height of Machu Picchu is 2,430 m.\",\n",
    "    \"Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.\",\n",
    "    \"It is 80 kilometres (50 miles) northwest of Cusco, on the crest of the mountain Machu Picchu, located about 2,430 metres (7,970 feet) above mean sea level, over 1,000 metres (3,300 ft) lower than Cusco, which has an elevation of 3,400 metres (11,200 ft).\",\n",
    "]\n",
    "pp = MyPreprocessor()\n",
    "pp_corpus = pp.preprocess_corpus(corpus)\n",
    "cm = CooccurrenceMatrix(pp_corpus)\n",
    "vocab, comat = cm.fit_transform()\n",
    "words = [\n",
    "    key for key, value in sorted(vocab.items(), key=lambda item: (item[1], item[0]))\n",
    "]\n",
    "df = pd.DataFrame(comat.todense(), columns=words, index=words, dtype=np.int8)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height is 2.00 and cosine similarity is 0.71\n",
      "The dot product between tall and official is 2.00 and cosine similarity is 0.82\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "\n",
    "def similarity(word1, word2):\n",
    "    \"\"\"\n",
    "    Returns similarity score between word1 and word2\n",
    "    Arguments\n",
    "    ---------\n",
    "    word1 -- (str)\n",
    "        The first word\n",
    "    word2 -- (str)\n",
    "        The second word\n",
    "\n",
    "    Returns\n",
    "    --------\n",
    "    None. Prints the similarity score between word1 and word2.\n",
    "    \"\"\"\n",
    "    vec1 = cm.get_word_vector(word1).todense().flatten()\n",
    "    vec2 = cm.get_word_vector(word2).todense().flatten()\n",
    "    v1 = np.squeeze(np.asarray(vec1))\n",
    "    v2 = np.squeeze(np.asarray(vec2))\n",
    "    print(\n",
    "        \"The dot product between %s and %s is %0.2f and cosine similarity is %0.2f\"\n",
    "        % (word1, word2, np.dot(v1, v2), cosine_similarity(vec1, vec2))\n",
    "    )\n",
    "\n",
    "\n",
    "similarity(\"tall\", \"height\")\n",
    "similarity(\"tall\", \"official\")\n",
    "\n",
    "### Not very reliable similarity scores because we used only 4 sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We are able to capture some similarities between words now. \n",
    "- That said similarities do not make much sense in the toy example above because we're using a tiny corpus. \n",
    "- To find meaningful patterns of similarities between words, we need a large corpus. \n",
    "- Let's try a bit larger corpus and check whether the similarities make sense. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences in the corpus:  682\n"
     ]
    }
   ],
   "source": [
    "import wikipedia\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "corpus = []\n",
    "\n",
    "queries = [\"Machu Picchu\", \"human stature\", \"Everest\"]\n",
    "\n",
    "for i in range(len(queries)):\n",
    "    sents = sent_tokenize(wikipedia.page(queries[i]).content)\n",
    "    corpus.extend(sents)\n",
    "print(\"Number of sentences in the corpus: \", len(sents))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>15th-century</th>\n",
       "      <th>inca</th>\n",
       "      <th>citadel</th>\n",
       "      <th>located</th>\n",
       "      <th>eastern</th>\n",
       "      <th>cordillera</th>\n",
       "      <th>southern</th>\n",
       "      <th>peru</th>\n",
       "      <th>...</th>\n",
       "      <th>panorama</th>\n",
       "      <th>drawing</th>\n",
       "      <th>imaging</th>\n",
       "      <th>panoramas</th>\n",
       "      <th>interactive</th>\n",
       "      <th>summitpost</th>\n",
       "      <th>format</th>\n",
       "      <th>quicktime</th>\n",
       "      <th>virtual</th>\n",
       "      <th>info-graphic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15th-century</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>inca</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>citadel</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 5581 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              machu  picchu  15th-century  inca  citadel  located  eastern  \\\n",
       "machu             0      90             1     6        2        2        1   \n",
       "picchu           90       0             1     6        4        1        1   \n",
       "15th-century      1       1             0     1        1        1        0   \n",
       "inca              6       6             1     0        1        1        1   \n",
       "citadel           2       4             1     1        0        1        1   \n",
       "\n",
       "              cordillera  southern  peru  ...  panorama  drawing  imaging  \\\n",
       "machu                  0         1     5  ...         0        0        0   \n",
       "picchu                 0         1     4  ...         0        0        0   \n",
       "15th-century           0         0     0  ...         0        0        0   \n",
       "inca                   0         0     2  ...         0        0        0   \n",
       "citadel                1         0     0  ...         0        0        0   \n",
       "\n",
       "              panoramas  interactive  summitpost  format  quicktime  virtual  \\\n",
       "machu                 0            0           0       0          0        0   \n",
       "picchu                0            0           0       0          0        0   \n",
       "15th-century          0            0           0       0          0        0   \n",
       "inca                  0            0           0       0          0        0   \n",
       "citadel               0            0           0       0          0        0   \n",
       "\n",
       "              info-graphic  \n",
       "machu                    0  \n",
       "picchu                   0  \n",
       "15th-century             0  \n",
       "inca                     0  \n",
       "citadel                  0  \n",
       "\n",
       "[5 rows x 5581 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pp = MyPreprocessor()\n",
    "pp_corpus = pp.preprocess_corpus(corpus)\n",
    "cm = CooccurrenceMatrix(pp_corpus)\n",
    "vocab, comat = cm.fit_transform()\n",
    "words = [\n",
    "    key for key, value in sorted(vocab.items(), key=lambda item: (item[1], item[0]))\n",
    "]\n",
    "df = pd.DataFrame(comat.todense(), columns=words, index=words, dtype=np.int8)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height is 38.00 and cosine similarity is 0.13\n",
      "The dot product between tall and official is 0.00 and cosine similarity is 0.00\n"
     ]
    }
   ],
   "source": [
    "similarity(\"tall\", \"height\")\n",
    "similarity(\"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sparse vs. dense word vectors\n",
    "\n",
    "- Term-term co-occurrence matrices are long and sparse. \n",
    "    - length |V| is usually large (e.g., > 50,000) \n",
    "    - most elements are zero\n",
    "- OK because there are efficient ways to deal with sparse matrices.\n",
    "\n",
    "\n",
    "### Alternative \n",
    "- Learn short (~100 to 1000 dimensions) and dense vectors. \n",
    "- Short vectors may be easier to train with ML models (less weights to train).\n",
    "- They may generalize better.\n",
    "- In practice they work much better! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word2Vec \n",
    "\n",
    "- A family of algorithms to create dense word embeddings\n",
    "\n",
    "![](img/word2vec.png)\n",
    "\n",
    "<!-- <img src=\"img/word2vec.png\" width=\"1000\" height=\"1000\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Success of Word2Vec\n",
    "\n",
    "- Able to capture complex relationships between words.\n",
    "- Example: What is the word that is similar to **WOMAN** in the same sense as **KING** is similar to **MAN**?\n",
    "- Perform a simple algebraic operations with the vector representation of words.\n",
    "    $\\vec{X} = \\vec{\\text{KING}} − \\vec{\\text{MAN}} + \\vec{\\text{WOMAN}}$\n",
    "- Search in the vector space for the word closest to $\\vec{X}$ measured by cosine distance.\n",
    "\n",
    "![](img/word_analogies1.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/word_analogies1.png\" width=\"500\" height=\"500\"> -->\n",
    "<!-- </center> -->\n",
    "    \n",
    "(Credit: Mikolov et al. 2013)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We can create a dense representation with a library called `gensim`. \n",
    "\n",
    "```\n",
    "conda install -c anaconda gensim\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/gensim/similarities/__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package <https://pypi.org/project/python-Levenshtein/> is unavailable. Install Levenhstein (e.g. `pip install python-Levenshtein`) to suppress this warning.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\n",
    "model = Word2Vec(sentences, min_count=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the word vector of the word _cat_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00713902,  0.00124103, -0.00717672, -0.00224462,  0.0037193 ,\n",
       "        0.00583312,  0.00119818,  0.00210273, -0.00411039,  0.00722533,\n",
       "       -0.00630704,  0.00464721, -0.00821997,  0.00203647, -0.00497705,\n",
       "       -0.00424769, -0.00310899,  0.00565521,  0.0057984 , -0.00497465,\n",
       "        0.00077333, -0.00849578,  0.00780981,  0.00925729, -0.00274233,\n",
       "        0.00080022,  0.00074665,  0.00547788, -0.00860608,  0.00058445,\n",
       "        0.00686942,  0.00223159,  0.00112468, -0.00932216,  0.00848237,\n",
       "       -0.00626413, -0.00299237,  0.00349379, -0.00077263,  0.00141129,\n",
       "        0.00178199, -0.0068289 , -0.00972481,  0.00904058,  0.00619805,\n",
       "       -0.00691293,  0.00340348,  0.00020606,  0.00475374, -0.00711994,\n",
       "        0.00402695,  0.00434743,  0.00995737, -0.00447374, -0.00138927,\n",
       "       -0.00731732, -0.00969783, -0.00908026, -0.00102276, -0.00650329,\n",
       "        0.00484973, -0.00616403,  0.00251919,  0.00073944, -0.00339216,\n",
       "       -0.00097922,  0.00997912,  0.00914589, -0.00446183,  0.00908303,\n",
       "       -0.00564176,  0.00593092, -0.00309722,  0.00343175,  0.00301723,\n",
       "        0.00690046, -0.00237388,  0.00877504,  0.00758943, -0.00954765,\n",
       "       -0.00800821, -0.0076379 ,  0.00292326, -0.00279472, -0.00692952,\n",
       "       -0.00812826,  0.00830918,  0.00199049, -0.00932802, -0.00479272,\n",
       "        0.00313674, -0.00471321,  0.00528084, -0.00423344,  0.00264179,\n",
       "       -0.00804569,  0.00620989,  0.00481889,  0.00078719,  0.00301345],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv[\"cat\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What's the most similar word to the word _cat_? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dog', 0.17018885910511017),\n",
       " ('woof', 0.004503009375184774),\n",
       " ('say', -0.027750369161367416),\n",
       " ('meow', -0.04461709037423134)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(\"cat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is good. But if you want good and meaningful representations of words you need to train models on a large corpus such as the whole Wikipedia, which is computationally intensive. \n",
    "\n",
    "So instead of training our own models, we use the **pre-trained embeddings**. These are the word embeddings people have trained embeddings on huge corpora and made them available for us to use. \n",
    "\n",
    "Let's try out Google news pre-trained word vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# It'll take a while to run this when you try it out for the first time.\n",
    "import gensim.downloader as api\n",
    "\n",
    "google_news_vectors = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary:  3000000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of vocabulary: \", len(google_news_vectors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `google_news_vectors` above has 300 dimensional word vectors for 3,000,000 unique words from Google news. \n",
    "- What can we do with these word vectors?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding similar words \n",
    "\n",
    "- Given word $w$, search in the vector space for the word closest to $w$ as measured by cosine distance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('UVic', 0.7886475920677185),\n",
       " ('SFU', 0.7588528394699097),\n",
       " ('Simon_Fraser', 0.7356574535369873),\n",
       " ('UFV', 0.6880435943603516),\n",
       " ('VIU', 0.6778583526611328),\n",
       " ('Kwantlen', 0.6771429181098938),\n",
       " ('UBCO', 0.6734487414360046),\n",
       " ('UPEI', 0.6731126308441162),\n",
       " ('UBC_Okanagan', 0.6709133982658386),\n",
       " ('Lakehead_University', 0.6622507572174072)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"UBC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('info', 0.7363681793212891),\n",
       " ('infomation', 0.680029571056366),\n",
       " ('infor_mation', 0.673384964466095),\n",
       " ('informaiton', 0.6639009118080139),\n",
       " ('informa_tion', 0.6601256728172302),\n",
       " ('informationon', 0.6339334845542908),\n",
       " ('informationabout', 0.6320980191230774),\n",
       " ('Information', 0.6186580657958984),\n",
       " ('informaion', 0.6093292236328125),\n",
       " ('details', 0.6063088774681091)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"information\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to extract all documents containing information or similar words, you could use this information.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding similarity scores between words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27610135"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"Canada\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.060384665"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"China\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The similarity between height and tall is 0.473\n",
      "The similarity between pineapple and mango is 0.668\n",
      "The similarity between pineapple and juice is 0.418\n",
      "The similarity between sun and robot is 0.029\n",
      "The similarity between GPU and lion is 0.002\n"
     ]
    }
   ],
   "source": [
    "word_pairs = [\n",
    "    (\"height\", \"tall\"),\n",
    "    (\"pineapple\", \"mango\"),\n",
    "    (\"pineapple\", \"juice\"),\n",
    "    (\"sun\", \"robot\"),\n",
    "    (\"GPU\", \"lion\"),\n",
    "]\n",
    "for pair in word_pairs:\n",
    "    print(\n",
    "        \"The similarity between %s and %s is %0.3f\"\n",
    "        % (pair[0], pair[1], google_news_vectors.similarity(pair[0], pair[1]))\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def analogy(word1, word2, word3, model=google_news_vectors):\n",
    "    \"\"\"\n",
    "    Returns analogy word using the given model.\n",
    "\n",
    "    Parameters\n",
    "    --------------\n",
    "    word1 : (str)\n",
    "        word1 in the analogy relation\n",
    "    word2 : (str)\n",
    "        word2 in the analogy relation\n",
    "    word3 : (str)\n",
    "        word3 in the analogy relation\n",
    "    model :\n",
    "        word embedding model\n",
    "\n",
    "    Returns\n",
    "    ---------------\n",
    "        pd.dataframe\n",
    "    \"\"\"\n",
    "    print(\"%s : %s :: %s : ?\" % (word1, word2, word3))\n",
    "    sim_words = model.most_similar(positive=[word3, word2], negative=[word1])\n",
    "    return pd.DataFrame(sim_words, columns=[\"Analogy word\", \"Score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : king :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>queen</td>\n",
       "      <td>0.711819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>monarch</td>\n",
       "      <td>0.618967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>princess</td>\n",
       "      <td>0.590243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>crown_prince</td>\n",
       "      <td>0.549946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>prince</td>\n",
       "      <td>0.537732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>kings</td>\n",
       "      <td>0.523684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Queen_Consort</td>\n",
       "      <td>0.523595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>queens</td>\n",
       "      <td>0.518113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sultan</td>\n",
       "      <td>0.509859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>monarchy</td>\n",
       "      <td>0.508741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Analogy word     Score\n",
       "0          queen  0.711819\n",
       "1        monarch  0.618967\n",
       "2       princess  0.590243\n",
       "3   crown_prince  0.549946\n",
       "4         prince  0.537732\n",
       "5          kings  0.523684\n",
       "6  Queen_Consort  0.523595\n",
       "7         queens  0.518113\n",
       "8         sultan  0.509859\n",
       "9       monarchy  0.508741"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"king\", \"woman\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Montreal : Canadiens :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Canucks</td>\n",
       "      <td>0.821327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Vancouver_Canucks</td>\n",
       "      <td>0.750401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Calgary_Flames</td>\n",
       "      <td>0.705470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Leafs</td>\n",
       "      <td>0.695783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Maple_Leafs</td>\n",
       "      <td>0.691617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thrashers</td>\n",
       "      <td>0.687504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Avs</td>\n",
       "      <td>0.681716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sabres</td>\n",
       "      <td>0.665307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Blackhawks</td>\n",
       "      <td>0.664625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Habs</td>\n",
       "      <td>0.661023</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Analogy word     Score\n",
       "0            Canucks  0.821327\n",
       "1  Vancouver_Canucks  0.750401\n",
       "2     Calgary_Flames  0.705470\n",
       "3              Leafs  0.695783\n",
       "4        Maple_Leafs  0.691617\n",
       "5          Thrashers  0.687504\n",
       "6                Avs  0.681716\n",
       "7             Sabres  0.665307\n",
       "8         Blackhawks  0.664625\n",
       "9               Habs  0.661023"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Montreal\", \"Canadiens\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Toronto : UofT :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SFU</td>\n",
       "      <td>0.579245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UVic</td>\n",
       "      <td>0.576921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UBC</td>\n",
       "      <td>0.571431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Simon_Fraser</td>\n",
       "      <td>0.543464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Langara_College</td>\n",
       "      <td>0.541347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>UVIC</td>\n",
       "      <td>0.520495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Grant_MacEwan</td>\n",
       "      <td>0.517273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>UFV</td>\n",
       "      <td>0.514150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ubyssey</td>\n",
       "      <td>0.510421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Kwantlen</td>\n",
       "      <td>0.503807</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Analogy word     Score\n",
       "0              SFU  0.579245\n",
       "1             UVic  0.576921\n",
       "2              UBC  0.571431\n",
       "3     Simon_Fraser  0.543464\n",
       "4  Langara_College  0.541347\n",
       "5             UVIC  0.520495\n",
       "6    Grant_MacEwan  0.517273\n",
       "7              UFV  0.514150\n",
       "8          Ubyssey  0.510421\n",
       "9         Kwantlen  0.503807"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Toronto\", \"UofT\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples of semantic and syntactic relationships\n",
    "\n",
    "![](img/word_analogies2.png)\n",
    "\n",
    "<!-- <img src=\"img/word_analogies2.png\" width=\"800\" height=\"800\"> -->\n",
    "\n",
    "(Credit: Mikolov 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Implicit biases and stereotypes in word embeddings\n",
    "\n",
    "- Reflect gender stereotypes present in broader society.\n",
    "- They may also amplify these stereotypes because of their widespread usage. \n",
    "- See the paper [Man is to Computer Programmer as Woman is to ...](http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : computer_programmer :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>homemaker</td>\n",
       "      <td>0.562712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>housewife</td>\n",
       "      <td>0.510505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>graphic_designer</td>\n",
       "      <td>0.505180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>schoolteacher</td>\n",
       "      <td>0.497949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>businesswoman</td>\n",
       "      <td>0.493489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>paralegal</td>\n",
       "      <td>0.492551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>registered_nurse</td>\n",
       "      <td>0.490797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>saleswoman</td>\n",
       "      <td>0.488163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>electrical_engineer</td>\n",
       "      <td>0.479773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>mechanical_engineer</td>\n",
       "      <td>0.475540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Analogy word     Score\n",
       "0            homemaker  0.562712\n",
       "1            housewife  0.510505\n",
       "2     graphic_designer  0.505180\n",
       "3        schoolteacher  0.497949\n",
       "4        businesswoman  0.493489\n",
       "5            paralegal  0.492551\n",
       "6     registered_nurse  0.490797\n",
       "7           saleswoman  0.488163\n",
       "8  electrical_engineer  0.479773\n",
       "9  mechanical_engineer  0.475540"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"computer_programmer\", \"woman\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/eva-srsly.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the modern embeddings are de-biased. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "(NLP)### Other pre-trained embeddings\n",
    "\n",
    "A number of pre-trained word embeddings are available. The most popular ones are:  \n",
    "\n",
    "- [Word2Vec](https://code.google.com/archive/p/word2vec/)\n",
    "    * trained on several corpora using the word2vec algorithm \n",
    "- [wikipedia2vec](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)\n",
    "    * pretrained embeddings for 12 languages \n",
    "- [GloVe](https://nlp.stanford.edu/projects/glove/)\n",
    "    * trained using [the GloVe algorithm](https://nlp.stanford.edu/pubs/glove.pdf) \n",
    "    * published by Stanford University \n",
    "- [fastText pre-trained embeddings for 294 languages](https://fasttext.cc/docs/en/pretrained-vectors.html) \n",
    "    * trained using [the fastText algorithm](http://aclweb.org/anthology/Q17-1010)\n",
    "    * published by Facebook    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Note that these pre-trained word vectors are of big size (in several gigabytes). \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a list of all pre-trained embeddings available with `gensim`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300', 'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', 'glove-wiki-gigaword-100', 'glove-wiki-gigaword-200', 'glove-wiki-gigaword-300', 'glove-twitter-25', 'glove-twitter-50', 'glove-twitter-100', 'glove-twitter-200', '__testing_word2vec-matrix-synopsis']\n"
     ]
    }
   ],
   "source": [
    "import gensim.downloader\n",
    "\n",
    "print(list(gensim.downloader.info()[\"models\"].keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word vectors with spaCy\n",
    "\n",
    "- spaCy also gives you access to word vectors with bigger models: `en_core_web_md` or `en_core_web_lr`\n",
    "- `spaCy`'s pre-trained embeddings are trained on [OntoNotes corpus](https://catalog.ldc.upenn.edu/LDC2013T19).\n",
    "- This corpus has a collection of different styles of texts such as telephone conversations, newswire, newsgroups, broadcast news, broadcast conversation, weblogs, religious texts. \n",
    "- Let's try it out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.8445e-01,  5.5363e-01,  4.9800e-01, -2.7769e-01,  9.3481e-02,\n",
       "        5.8978e-01, -6.1267e-01,  1.5415e-01,  3.7752e-01,  1.4348e-01,\n",
       "       -2.0126e-01,  2.6869e-01, -7.2758e-01,  2.4405e-01,  3.5321e-01,\n",
       "       -3.8314e-01,  1.8920e-01,  9.0860e-01,  2.0685e-01,  6.7174e-02,\n",
       "        4.9190e-01,  4.8224e-01,  1.2929e-01,  4.9490e-01, -9.3981e-02,\n",
       "       -6.0443e-01,  2.8314e-01, -7.4459e-02,  5.9333e-02, -9.5484e-01,\n",
       "       -1.2755e-01,  1.1871e-01,  2.9725e-01, -1.8604e-01, -4.9672e-01,\n",
       "       -1.4352e-01, -1.2770e-02,  1.0423e-01, -6.7861e-01,  7.6421e-01,\n",
       "        2.0364e-02, -3.7836e-02,  2.9399e-01, -4.1602e-01,  3.1965e-01,\n",
       "        9.8503e-01, -6.8950e-02, -4.0057e-01, -1.3972e-01,  3.2916e-01,\n",
       "       -8.3725e-02,  7.1081e-02,  4.5407e-01, -6.0930e-02,  8.5099e-01,\n",
       "       -7.3595e-01,  3.5860e-01, -1.7554e-01, -1.5838e-01,  3.1525e-02,\n",
       "       -1.4029e-01,  9.3486e-02,  4.2341e-01,  2.2277e-01, -8.6968e-02,\n",
       "       -4.4123e-01,  1.1326e-01,  1.1435e-01, -8.7185e-01,  5.2618e-01,\n",
       "       -8.7317e-02,  8.8284e-01, -2.1643e-01,  4.1260e-01, -1.9629e-01,\n",
       "        2.4883e-01,  8.2752e-02, -1.0800e-01, -3.7362e-01, -2.0995e-01,\n",
       "        7.9307e-02, -2.3805e-01, -1.8537e-01, -1.1211e-01, -8.4294e-02,\n",
       "       -2.9430e-01,  1.2991e+00,  8.2207e-01, -8.0108e-01, -3.9065e-01,\n",
       "       -8.9309e-02, -3.7571e-01,  5.4184e-01, -3.5247e-02, -2.2929e-01,\n",
       "        6.1834e-01, -1.7637e-01, -1.0375e-01, -5.3831e-01,  2.0187e-01,\n",
       "        4.1651e-01, -4.8564e-01, -9.3836e-02, -1.6272e-01, -2.3528e-01,\n",
       "       -1.0950e+00,  1.7226e-01, -5.8347e-01,  1.0375e+00,  3.8504e-01,\n",
       "       -4.3338e-01, -2.8801e-01, -9.3194e-01,  1.1582e-01,  2.1183e-01,\n",
       "       -4.7209e-02,  5.2106e-02,  1.0386e-01,  3.1998e-01,  3.3064e-01,\n",
       "       -7.5731e-02,  1.5550e-01, -1.3200e-01, -4.2069e-01, -4.0630e-01,\n",
       "       -3.9150e-02, -1.1429e-01,  7.1862e-02, -4.1747e-01, -3.3113e-02,\n",
       "       -5.8413e-01, -5.5429e-01, -4.2042e-01, -1.1280e-02, -3.3124e-01,\n",
       "       -9.0115e-02,  3.8209e-01,  1.2274e-01, -3.2571e-01,  3.5727e-02,\n",
       "       -2.1267e+00,  6.5307e-01, -8.7850e-02,  2.7738e-01, -3.8208e-01,\n",
       "       -6.4254e-02, -3.8063e-01,  7.2693e-01,  2.5506e-01, -4.4696e-01,\n",
       "        5.4132e-01,  1.2553e-01,  5.2933e-01,  3.7021e-01, -1.0748e+00,\n",
       "       -5.3942e-01,  3.4178e-01,  1.5028e-01,  3.1055e-01,  6.1673e-01,\n",
       "       -9.9166e-02,  4.2590e-01,  1.4452e-01, -1.6359e-01, -1.5430e-01,\n",
       "       -8.1012e-05, -7.2212e-02, -1.7403e-01,  1.6741e-01,  2.3766e-01,\n",
       "        2.2990e-01,  3.2401e-01,  3.2437e-01,  1.4617e-01, -8.6237e-01,\n",
       "       -1.6606e-01,  2.5771e-01, -2.3734e-01,  3.6835e-01, -6.2753e-01,\n",
       "       -2.7425e-01, -1.2848e-01,  2.9455e-01, -2.4697e-01, -3.8050e-01,\n",
       "       -3.1025e-01, -2.7280e-01, -4.8886e-01, -5.8880e-01, -1.1161e-01,\n",
       "       -8.8502e-01, -2.7199e-02, -3.8848e-02, -2.2087e-01, -1.6948e-01,\n",
       "       -2.2928e-01,  6.5381e-01,  1.0211e-01, -2.4038e-01,  3.0906e-01,\n",
       "       -6.8702e-01,  9.5880e-02,  5.4251e-01,  2.5873e-01,  3.2799e-01,\n",
       "        8.3257e-02,  3.0840e-01,  3.2223e-01,  5.4355e-01,  7.9113e-01,\n",
       "       -7.5587e-02,  4.0929e-01,  3.6541e-01, -2.4388e-02, -6.5300e-01,\n",
       "        1.9694e-01, -4.9399e-02, -3.0366e-01, -1.9463e-01, -2.2815e-01,\n",
       "       -6.4144e-01,  2.7839e-02,  5.5869e-01,  3.9416e-01,  5.1166e-01,\n",
       "        1.4871e-01, -2.2047e-01,  9.8443e-02, -2.8448e-01,  1.8304e-02,\n",
       "       -7.6165e-01, -1.8388e-01, -2.8458e-01,  1.3345e-01, -5.5309e-01,\n",
       "       -4.3729e-01,  7.7534e-02, -4.9492e-01,  4.6410e-01,  2.4458e-01,\n",
       "       -1.6128e-01, -6.6307e-01, -8.2413e-01, -3.3230e-01, -9.4846e-02,\n",
       "       -1.1402e-01,  1.6437e-01, -1.4815e-01,  9.4412e-03,  4.3647e-01,\n",
       "       -1.3735e-01,  5.2660e-02, -4.8225e-01,  3.3501e-01, -1.4767e-01,\n",
       "        3.3204e-01,  3.3985e-01,  1.5218e-01,  3.5595e-01,  1.4562e-01,\n",
       "       -1.3912e-01, -5.4815e-02, -5.7572e-01, -2.6978e-01, -1.4739e-01,\n",
       "       -2.7083e-01, -4.4642e-01,  2.3675e-02, -7.0934e-01, -3.0456e-01,\n",
       "       -3.2839e-01, -1.1621e-02,  1.7080e-01,  7.2882e-02, -4.0110e-01,\n",
       "        8.1377e-01, -3.8940e-01,  3.7111e-01, -6.4290e-01,  4.0496e-01,\n",
       "        2.7992e-02,  1.7528e-01,  3.3878e-01,  4.3610e-01,  4.3892e-01,\n",
       "        1.5003e-01,  4.4302e-01,  4.0898e-01,  3.8504e-01, -1.5575e-01,\n",
       "        2.0590e-01,  1.0659e-02, -2.5892e-01, -1.3161e-01, -5.7390e-01,\n",
       "        6.2052e-01, -2.7414e-01, -1.0214e+00, -5.5313e-02,  5.6540e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc = nlp(\"pineapple\")\n",
    "doc.vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Representing documents using word embeddings\n",
    "\n",
    "- Assuming that we have reasonable representations of words. \n",
    "- How do we represent meaning of paragraphs or documents?\n",
    "- Two simple approaches\n",
    "    - Averaging embeddings\n",
    "    - Concatenating embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Averaging embeddings\n",
    "\n",
    "<blockquote>\n",
    "All empty promises\n",
    "</blockquote>\n",
    "    \n",
    "$(embedding(all) + embedding(empty) + embedding(promise))/3$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Average embeddings with spaCy\n",
    "\n",
    "- We can do this conveniently with [spaCy](https://spacy.io/usage/linguistic-features#vectors-similarity). \n",
    "- We need `en_core_web_md` model to access word vectors. \n",
    "- You can download the model by going to command line and in your course `conda` environment and download `en_core_web_md` as follows.   \n",
    "\n",
    "```\n",
    "conda activate cpsc330\n",
    "python -m spacy download en_core_web_md\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access word vectors for individual words in `spaCy` as follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.0127  , -0.20527 , -0.24555 , -0.076636,  0.11981 ,  0.21642 ,\n",
       "       -0.55584 ,  0.020206,  0.39778 ,  1.7218  ], dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp(\"empty\").vector[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get average embeddings for a sentence or a document in `spaCy` as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300,)\n",
      "Vector for: All empty promises\n",
      "[ 0.28903252 -0.09684668 -0.11497    -0.26554868  0.01983    -0.111594\n",
      " -0.10229333  0.126915    0.17705734  2.1837332 ]\n"
     ]
    }
   ],
   "source": [
    "s = \"All empty promises\"\n",
    "doc = nlp(s)\n",
    "avg_sent_emb = doc.vector\n",
    "print(avg_sent_emb.shape)\n",
    "print(\"Vector for: {}\\n{}\".format((s), (avg_sent_emb[0:10])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Similarity between documents \n",
    "\n",
    "- We can also get similarity between documents as follows. \n",
    "- Note that this is based on average embeddings of each sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep learning is very popular these days. <-> Machine learning is dominated by neural networks. 0.7564516644025884\n",
      "Machine learning is dominated by neural networks. <-> A home-made fresh bread with butter and cheese. 0.5363564587815752\n"
     ]
    }
   ],
   "source": [
    "doc1 = nlp(\"Deep learning is very popular these days.\")\n",
    "doc2 = nlp(\"Machine learning is dominated by neural networks.\")\n",
    "doc3 = nlp(\"A home-made fresh bread with butter and cheese.\")\n",
    "\n",
    "# Similarity of two documents\n",
    "print(doc1, \"<->\", doc2, doc1.similarity(doc2))\n",
    "print(doc2, \"<->\", doc3, doc2.similarity(doc3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Do these scores make sense? \n",
    "- There are no common words, but we are still able to identify that doc1 and doc2 are more similar that doc2 and doc3.  \n",
    "- You can use such average embedding representation in text classification tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  Airline sentiment analysis using average embedding representation \n",
    "\n",
    "- Let's try average embedding representation for airline sentiment analysis.\n",
    "- You can download the dataset [here](https://www.kaggle.com/jaskarancr/airline-sentiment-dataset). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/Airline-Sentiment-2-w-AA.csv\", encoding=\"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(df, test_size=0.2, random_state=123)\n",
    "X_train, y_train = train_df[\"text\"], train_df[\"airline_sentiment\"]\n",
    "X_test, y_test = test_df[\"text\"], test_df[\"airline_sentiment\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_unit_id</th>\n",
       "      <th>_golden</th>\n",
       "      <th>_unit_state</th>\n",
       "      <th>_trusted_judgments</th>\n",
       "      <th>_last_judgment_at</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment:confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason:confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>681455792</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 4:21</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>Southwest</td>\n",
       "      <td>NaN</td>\n",
       "      <td>mrssuperdimmock</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@SouthwestAir link doesn't work</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/19/15 18:53</td>\n",
       "      <td>5.686040e+17</td>\n",
       "      <td>Lake Arrowhead, CA</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>681459957</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:45</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Delta</td>\n",
       "      <td>NaN</td>\n",
       "      <td>labeles</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@JetBlue okayyyy. But I had huge irons on way ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 10:18</td>\n",
       "      <td>5.677500e+17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>681462990</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:53</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Customer Service Issue</td>\n",
       "      <td>0.6727</td>\n",
       "      <td>US Airways</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DropMeAnywhere</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@USAirways They're all reservations numbers an...</td>\n",
       "      <td>[0.0, 0.0]</td>\n",
       "      <td>2/17/15 14:50</td>\n",
       "      <td>5.678190e+17</td>\n",
       "      <td>Here, There and Everywhere</td>\n",
       "      <td>Arizona</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>681448905</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:10</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jsamaudio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica no A's channel this year?</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/18/15 12:25</td>\n",
       "      <td>5.681440e+17</td>\n",
       "      <td>St. Francis (Calif.)</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>681454122</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:08</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.3544</td>\n",
       "      <td>United</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CajunSQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@united missed it.  Incoming on time, then Sat...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 14:20</td>\n",
       "      <td>5.678110e+17</td>\n",
       "      <td>Baton Rouge, LA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        _unit_id  _golden _unit_state  _trusted_judgments _last_judgment_at  \\\n",
       "5789   681455792    False   finalized                   3      2/25/15 4:21   \n",
       "8918   681459957    False   finalized                   3      2/25/15 9:45   \n",
       "11688  681462990    False   finalized                   3      2/25/15 9:53   \n",
       "413    681448905    False   finalized                   3     2/25/15 10:10   \n",
       "4135   681454122    False   finalized                   3     2/25/15 10:08   \n",
       "\n",
       "      airline_sentiment  airline_sentiment:confidence          negativereason  \\\n",
       "5789           negative                           1.0              Can't Tell   \n",
       "8918            neutral                           1.0                     NaN   \n",
       "11688          negative                           1.0  Customer Service Issue   \n",
       "413             neutral                           1.0                     NaN   \n",
       "4135           negative                           1.0              Bad Flight   \n",
       "\n",
       "       negativereason:confidence         airline airline_sentiment_gold  \\\n",
       "5789                      0.6667       Southwest                    NaN   \n",
       "8918                         NaN           Delta                    NaN   \n",
       "11688                     0.6727      US Airways                    NaN   \n",
       "413                          NaN  Virgin America                    NaN   \n",
       "4135                      0.3544          United                    NaN   \n",
       "\n",
       "                  name negativereason_gold  retweet_count  \\\n",
       "5789   mrssuperdimmock                 NaN              0   \n",
       "8918           labeles                 NaN              0   \n",
       "11688   DropMeAnywhere                 NaN              0   \n",
       "413          jsamaudio                 NaN              0   \n",
       "4135          CajunSQL                 NaN              0   \n",
       "\n",
       "                                                    text tweet_coord  \\\n",
       "5789                     @SouthwestAir link doesn't work         NaN   \n",
       "8918   @JetBlue okayyyy. But I had huge irons on way ...         NaN   \n",
       "11688  @USAirways They're all reservations numbers an...  [0.0, 0.0]   \n",
       "413             @VirginAmerica no A's channel this year?         NaN   \n",
       "4135   @united missed it.  Incoming on time, then Sat...         NaN   \n",
       "\n",
       "       tweet_created      tweet_id              tweet_location  \\\n",
       "5789   2/19/15 18:53  5.686040e+17          Lake Arrowhead, CA   \n",
       "8918   2/17/15 10:18  5.677500e+17                         NaN   \n",
       "11688  2/17/15 14:50  5.678190e+17  Here, There and Everywhere   \n",
       "413    2/18/15 12:25  5.681440e+17        St. Francis (Calif.)   \n",
       "4135   2/17/15 14:20  5.678110e+17             Baton Rouge, LA   \n",
       "\n",
       "                    user_timezone  \n",
       "5789   Pacific Time (US & Canada)  \n",
       "8918                          NaN  \n",
       "11688                     Arizona  \n",
       "413    Pacific Time (US & Canada)  \n",
       "4135                          NaN  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag-of-words representation for sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data matrix shape: (11712, 13064)\n"
     ]
    }
   ],
   "source": [
    "pipe = make_pipeline(\n",
    "    CountVectorizer(stop_words=\"english\"), LogisticRegression(max_iter=1000)\n",
    ")\n",
    "pipe.named_steps[\"countvectorizer\"].fit(X_train)\n",
    "X_train_transformed = pipe.named_steps[\"countvectorizer\"].transform(X_train)\n",
    "print(\"Data matrix shape:\", X_train_transformed.shape)\n",
    "pipe.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.94\n",
      "Test accuracy 0.80\n"
     ]
    }
   ],
   "source": [
    "print(\"Train accuracy {:.2f}\".format(pipe.score(X_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(pipe.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment analysis with average embedding representation\n",
    "\n",
    "- Let's see how can we get word vectors using `spaCy`. \n",
    "- Let's create average embedding representation for each example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_train)])\n",
    "X_test_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_test)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We have reduced dimensionality from 13,064 to 300! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11712, 300)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.102596</td>\n",
       "      <td>0.089241</td>\n",
       "      <td>-0.236022</td>\n",
       "      <td>0.030403</td>\n",
       "      <td>-0.183757</td>\n",
       "      <td>0.188970</td>\n",
       "      <td>-0.057322</td>\n",
       "      <td>-0.194976</td>\n",
       "      <td>-0.025315</td>\n",
       "      <td>1.887160</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.186171</td>\n",
       "      <td>-0.001768</td>\n",
       "      <td>0.088228</td>\n",
       "      <td>0.095176</td>\n",
       "      <td>0.195018</td>\n",
       "      <td>-0.085129</td>\n",
       "      <td>-0.113560</td>\n",
       "      <td>-0.108034</td>\n",
       "      <td>0.095647</td>\n",
       "      <td>0.274220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.063396</td>\n",
       "      <td>0.124589</td>\n",
       "      <td>-0.122292</td>\n",
       "      <td>-0.189419</td>\n",
       "      <td>-0.101237</td>\n",
       "      <td>0.002607</td>\n",
       "      <td>0.032192</td>\n",
       "      <td>-0.151461</td>\n",
       "      <td>-0.055412</td>\n",
       "      <td>1.796026</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.183419</td>\n",
       "      <td>-0.016063</td>\n",
       "      <td>-0.005495</td>\n",
       "      <td>-0.045449</td>\n",
       "      <td>0.092108</td>\n",
       "      <td>0.052184</td>\n",
       "      <td>0.099207</td>\n",
       "      <td>0.011452</td>\n",
       "      <td>0.004731</td>\n",
       "      <td>0.065721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.064436</td>\n",
       "      <td>0.148900</td>\n",
       "      <td>-0.170613</td>\n",
       "      <td>-0.155509</td>\n",
       "      <td>0.216841</td>\n",
       "      <td>0.002881</td>\n",
       "      <td>0.078774</td>\n",
       "      <td>-0.123730</td>\n",
       "      <td>-0.041174</td>\n",
       "      <td>2.200783</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.276544</td>\n",
       "      <td>0.022344</td>\n",
       "      <td>-0.029640</td>\n",
       "      <td>0.003638</td>\n",
       "      <td>0.070680</td>\n",
       "      <td>-0.000402</td>\n",
       "      <td>-0.029196</td>\n",
       "      <td>-0.033838</td>\n",
       "      <td>0.040326</td>\n",
       "      <td>0.118550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.029201</td>\n",
       "      <td>0.253583</td>\n",
       "      <td>-0.020073</td>\n",
       "      <td>0.035106</td>\n",
       "      <td>0.027043</td>\n",
       "      <td>-0.122413</td>\n",
       "      <td>0.076435</td>\n",
       "      <td>-0.253022</td>\n",
       "      <td>-0.044745</td>\n",
       "      <td>1.879360</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.070406</td>\n",
       "      <td>-0.012429</td>\n",
       "      <td>-0.075428</td>\n",
       "      <td>-0.106797</td>\n",
       "      <td>0.218251</td>\n",
       "      <td>-0.218904</td>\n",
       "      <td>-0.120035</td>\n",
       "      <td>-0.085280</td>\n",
       "      <td>0.002369</td>\n",
       "      <td>0.034781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.077888</td>\n",
       "      <td>0.213935</td>\n",
       "      <td>-0.108230</td>\n",
       "      <td>0.056640</td>\n",
       "      <td>0.071779</td>\n",
       "      <td>0.035684</td>\n",
       "      <td>-0.034920</td>\n",
       "      <td>-0.154641</td>\n",
       "      <td>-0.004615</td>\n",
       "      <td>1.916412</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.063743</td>\n",
       "      <td>0.002112</td>\n",
       "      <td>-0.026977</td>\n",
       "      <td>-0.102208</td>\n",
       "      <td>0.037013</td>\n",
       "      <td>-0.101183</td>\n",
       "      <td>-0.003163</td>\n",
       "      <td>0.025034</td>\n",
       "      <td>-0.087998</td>\n",
       "      <td>0.123148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0         1         2         3         4         5         6    \\\n",
       "0 -0.102596  0.089241 -0.236022  0.030403 -0.183757  0.188970 -0.057322   \n",
       "1 -0.063396  0.124589 -0.122292 -0.189419 -0.101237  0.002607  0.032192   \n",
       "2 -0.064436  0.148900 -0.170613 -0.155509  0.216841  0.002881  0.078774   \n",
       "3 -0.029201  0.253583 -0.020073  0.035106  0.027043 -0.122413  0.076435   \n",
       "4  0.077888  0.213935 -0.108230  0.056640  0.071779  0.035684 -0.034920   \n",
       "\n",
       "        7         8         9    ...       290       291       292       293  \\\n",
       "0 -0.194976 -0.025315  1.887160  ... -0.186171 -0.001768  0.088228  0.095176   \n",
       "1 -0.151461 -0.055412  1.796026  ... -0.183419 -0.016063 -0.005495 -0.045449   \n",
       "2 -0.123730 -0.041174  2.200783  ... -0.276544  0.022344 -0.029640  0.003638   \n",
       "3 -0.253022 -0.044745  1.879360  ... -0.070406 -0.012429 -0.075428 -0.106797   \n",
       "4 -0.154641 -0.004615  1.916412  ... -0.063743  0.002112 -0.026977 -0.102208   \n",
       "\n",
       "        294       295       296       297       298       299  \n",
       "0  0.195018 -0.085129 -0.113560 -0.108034  0.095647  0.274220  \n",
       "1  0.092108  0.052184  0.099207  0.011452  0.004731  0.065721  \n",
       "2  0.070680 -0.000402 -0.029196 -0.033838  0.040326  0.118550  \n",
       "3  0.218251 -0.218904 -0.120035 -0.085280  0.002369  0.034781  \n",
       "4  0.037013 -0.101183 -0.003163  0.025034 -0.087998  0.123148  \n",
       "\n",
       "[5 rows x 300 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment classification using average embeddings \n",
    "\n",
    "- What are the train and test accuracies with average word embedding representation? \n",
    "- The accuracy is a bit better with less overfitting. \n",
    "- Note that we are using **transfer learning** here.\n",
    "- The embeddings are trained on a completely different corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.81\n",
      "Test accuracy 0.81\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=1000)\n",
    "lgr.fit(X_train_embeddings, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(X_train_embeddings, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(X_test_embeddings, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment classification using advanced sentence representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Since representing documents is so essential for text classification tasks, there are more advanced methods for document representation. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "embedder = SentenceTransformer(\"paraphrase-distilroberta-base-v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768,)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_sents = embedder.encode(\"all empty promises\")\n",
    "emb_sents.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>-0.120494</td>\n",
       "      <td>0.250262</td>\n",
       "      <td>-0.022795</td>\n",
       "      <td>-0.116368</td>\n",
       "      <td>0.078650</td>\n",
       "      <td>0.037357</td>\n",
       "      <td>-0.251341</td>\n",
       "      <td>0.321429</td>\n",
       "      <td>-0.143984</td>\n",
       "      <td>-0.123487</td>\n",
       "      <td>...</td>\n",
       "      <td>0.199151</td>\n",
       "      <td>-0.150143</td>\n",
       "      <td>0.167078</td>\n",
       "      <td>-0.407671</td>\n",
       "      <td>-0.066161</td>\n",
       "      <td>0.049514</td>\n",
       "      <td>0.019385</td>\n",
       "      <td>-0.357601</td>\n",
       "      <td>0.125996</td>\n",
       "      <td>0.381073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>-0.182954</td>\n",
       "      <td>0.118282</td>\n",
       "      <td>0.066341</td>\n",
       "      <td>-0.136099</td>\n",
       "      <td>0.094947</td>\n",
       "      <td>-0.121303</td>\n",
       "      <td>0.069233</td>\n",
       "      <td>-0.097500</td>\n",
       "      <td>0.025739</td>\n",
       "      <td>-0.367980</td>\n",
       "      <td>...</td>\n",
       "      <td>0.113612</td>\n",
       "      <td>0.114661</td>\n",
       "      <td>0.049926</td>\n",
       "      <td>0.256736</td>\n",
       "      <td>-0.118687</td>\n",
       "      <td>-0.190720</td>\n",
       "      <td>0.011986</td>\n",
       "      <td>-0.141883</td>\n",
       "      <td>-0.230142</td>\n",
       "      <td>0.024899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>-0.032988</td>\n",
       "      <td>0.630251</td>\n",
       "      <td>-0.079516</td>\n",
       "      <td>0.148981</td>\n",
       "      <td>0.194709</td>\n",
       "      <td>-0.226264</td>\n",
       "      <td>-0.043630</td>\n",
       "      <td>0.217398</td>\n",
       "      <td>-0.010716</td>\n",
       "      <td>0.069644</td>\n",
       "      <td>...</td>\n",
       "      <td>0.676791</td>\n",
       "      <td>0.244484</td>\n",
       "      <td>0.051042</td>\n",
       "      <td>0.064099</td>\n",
       "      <td>-0.146945</td>\n",
       "      <td>0.090878</td>\n",
       "      <td>-0.090059</td>\n",
       "      <td>0.077212</td>\n",
       "      <td>-0.209226</td>\n",
       "      <td>0.308773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>-0.119259</td>\n",
       "      <td>0.172168</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.319859</td>\n",
       "      <td>0.415475</td>\n",
       "      <td>0.248360</td>\n",
       "      <td>-0.025923</td>\n",
       "      <td>0.385350</td>\n",
       "      <td>0.066414</td>\n",
       "      <td>-0.334289</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.128482</td>\n",
       "      <td>-0.232446</td>\n",
       "      <td>-0.077805</td>\n",
       "      <td>0.181328</td>\n",
       "      <td>0.123244</td>\n",
       "      <td>-0.143693</td>\n",
       "      <td>0.660456</td>\n",
       "      <td>-0.048714</td>\n",
       "      <td>0.204774</td>\n",
       "      <td>0.163496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>0.094240</td>\n",
       "      <td>0.360193</td>\n",
       "      <td>0.213747</td>\n",
       "      <td>0.363690</td>\n",
       "      <td>0.275521</td>\n",
       "      <td>0.134936</td>\n",
       "      <td>-0.276319</td>\n",
       "      <td>0.009336</td>\n",
       "      <td>-0.021523</td>\n",
       "      <td>-0.258992</td>\n",
       "      <td>...</td>\n",
       "      <td>0.474885</td>\n",
       "      <td>0.242125</td>\n",
       "      <td>0.294532</td>\n",
       "      <td>0.279014</td>\n",
       "      <td>0.037831</td>\n",
       "      <td>0.089761</td>\n",
       "      <td>-0.548748</td>\n",
       "      <td>-0.049258</td>\n",
       "      <td>0.154525</td>\n",
       "      <td>0.141268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5218</th>\n",
       "      <td>-0.204409</td>\n",
       "      <td>-0.145290</td>\n",
       "      <td>-0.064201</td>\n",
       "      <td>0.213571</td>\n",
       "      <td>-0.140225</td>\n",
       "      <td>0.338555</td>\n",
       "      <td>-0.148578</td>\n",
       "      <td>0.224516</td>\n",
       "      <td>-0.042963</td>\n",
       "      <td>0.075930</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.161949</td>\n",
       "      <td>0.040582</td>\n",
       "      <td>0.003971</td>\n",
       "      <td>-0.152549</td>\n",
       "      <td>-0.582907</td>\n",
       "      <td>-0.126527</td>\n",
       "      <td>0.060502</td>\n",
       "      <td>-0.111495</td>\n",
       "      <td>-0.097492</td>\n",
       "      <td>0.199321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12252</th>\n",
       "      <td>0.108408</td>\n",
       "      <td>0.438293</td>\n",
       "      <td>0.216812</td>\n",
       "      <td>-0.349289</td>\n",
       "      <td>0.422689</td>\n",
       "      <td>0.377761</td>\n",
       "      <td>0.045198</td>\n",
       "      <td>-0.034096</td>\n",
       "      <td>0.427570</td>\n",
       "      <td>-0.328272</td>\n",
       "      <td>...</td>\n",
       "      <td>0.257849</td>\n",
       "      <td>-0.032362</td>\n",
       "      <td>-0.275003</td>\n",
       "      <td>0.080452</td>\n",
       "      <td>-0.078975</td>\n",
       "      <td>-0.049972</td>\n",
       "      <td>-0.009762</td>\n",
       "      <td>-0.314754</td>\n",
       "      <td>-0.020774</td>\n",
       "      <td>0.268777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>0.068411</td>\n",
       "      <td>0.017591</td>\n",
       "      <td>0.236154</td>\n",
       "      <td>0.221446</td>\n",
       "      <td>-0.103567</td>\n",
       "      <td>0.055510</td>\n",
       "      <td>0.062909</td>\n",
       "      <td>0.067425</td>\n",
       "      <td>-0.003504</td>\n",
       "      <td>-0.157758</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007711</td>\n",
       "      <td>0.323297</td>\n",
       "      <td>0.334638</td>\n",
       "      <td>0.367042</td>\n",
       "      <td>-0.068821</td>\n",
       "      <td>0.063667</td>\n",
       "      <td>-0.329991</td>\n",
       "      <td>0.232331</td>\n",
       "      <td>-0.184768</td>\n",
       "      <td>-0.000683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11646</th>\n",
       "      <td>-0.091488</td>\n",
       "      <td>-0.155709</td>\n",
       "      <td>0.032391</td>\n",
       "      <td>0.018313</td>\n",
       "      <td>0.524998</td>\n",
       "      <td>0.563933</td>\n",
       "      <td>-0.080984</td>\n",
       "      <td>0.097983</td>\n",
       "      <td>-0.535285</td>\n",
       "      <td>-0.377195</td>\n",
       "      <td>...</td>\n",
       "      <td>0.428014</td>\n",
       "      <td>-0.144572</td>\n",
       "      <td>0.045296</td>\n",
       "      <td>-0.107935</td>\n",
       "      <td>-0.135673</td>\n",
       "      <td>-0.290019</td>\n",
       "      <td>-0.137200</td>\n",
       "      <td>-0.503395</td>\n",
       "      <td>-0.042567</td>\n",
       "      <td>-0.282591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3582</th>\n",
       "      <td>0.185626</td>\n",
       "      <td>0.092904</td>\n",
       "      <td>0.097085</td>\n",
       "      <td>-0.174650</td>\n",
       "      <td>-0.193584</td>\n",
       "      <td>0.047294</td>\n",
       "      <td>0.098216</td>\n",
       "      <td>0.332670</td>\n",
       "      <td>0.163098</td>\n",
       "      <td>-0.135102</td>\n",
       "      <td>...</td>\n",
       "      <td>0.078530</td>\n",
       "      <td>-0.030177</td>\n",
       "      <td>0.391598</td>\n",
       "      <td>0.073519</td>\n",
       "      <td>-0.454038</td>\n",
       "      <td>-0.244358</td>\n",
       "      <td>-0.790682</td>\n",
       "      <td>-0.607009</td>\n",
       "      <td>-0.255162</td>\n",
       "      <td>0.029779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11712 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "5789  -0.120494  0.250262 -0.022795 -0.116368  0.078650  0.037357 -0.251341   \n",
       "8918  -0.182954  0.118282  0.066341 -0.136099  0.094947 -0.121303  0.069233   \n",
       "11688 -0.032988  0.630251 -0.079516  0.148981  0.194709 -0.226264 -0.043630   \n",
       "413   -0.119259  0.172168  0.098698  0.319859  0.415475  0.248360 -0.025923   \n",
       "4135   0.094240  0.360193  0.213747  0.363690  0.275521  0.134936 -0.276319   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5218  -0.204409 -0.145290 -0.064201  0.213571 -0.140225  0.338555 -0.148578   \n",
       "12252  0.108408  0.438293  0.216812 -0.349289  0.422689  0.377761  0.045198   \n",
       "1346   0.068411  0.017591  0.236154  0.221446 -0.103567  0.055510  0.062909   \n",
       "11646 -0.091488 -0.155709  0.032391  0.018313  0.524998  0.563933 -0.080984   \n",
       "3582   0.185626  0.092904  0.097085 -0.174650 -0.193584  0.047294  0.098216   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "5789   0.321429 -0.143984 -0.123487  ...  0.199151 -0.150143  0.167078   \n",
       "8918  -0.097500  0.025739 -0.367980  ...  0.113612  0.114661  0.049926   \n",
       "11688  0.217398 -0.010716  0.069644  ...  0.676791  0.244484  0.051042   \n",
       "413    0.385350  0.066414 -0.334289  ... -0.128482 -0.232446 -0.077805   \n",
       "4135   0.009336 -0.021523 -0.258992  ...  0.474885  0.242125  0.294532   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5218   0.224516 -0.042963  0.075930  ... -0.161949  0.040582  0.003971   \n",
       "12252 -0.034096  0.427570 -0.328272  ...  0.257849 -0.032362 -0.275003   \n",
       "1346   0.067425 -0.003504 -0.157758  ...  0.007711  0.323297  0.334638   \n",
       "11646  0.097983 -0.535285 -0.377195  ...  0.428014 -0.144572  0.045296   \n",
       "3582   0.332670  0.163098 -0.135102  ...  0.078530 -0.030177  0.391598   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "5789  -0.407671 -0.066161  0.049514  0.019385 -0.357601  0.125996  0.381073  \n",
       "8918   0.256736 -0.118687 -0.190720  0.011986 -0.141883 -0.230142  0.024899  \n",
       "11688  0.064099 -0.146945  0.090878 -0.090059  0.077212 -0.209226  0.308773  \n",
       "413    0.181328  0.123244 -0.143693  0.660456 -0.048714  0.204774  0.163496  \n",
       "4135   0.279014  0.037831  0.089761 -0.548748 -0.049258  0.154525  0.141268  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5218  -0.152549 -0.582907 -0.126527  0.060502 -0.111495 -0.097492  0.199321  \n",
       "12252  0.080452 -0.078975 -0.049972 -0.009762 -0.314754 -0.020774  0.268777  \n",
       "1346   0.367042 -0.068821  0.063667 -0.329991  0.232331 -0.184768 -0.000683  \n",
       "11646 -0.107935 -0.135673 -0.290019 -0.137200 -0.503395 -0.042567 -0.282591  \n",
       "3582   0.073519 -0.454038 -0.244358 -0.790682 -0.607009 -0.255162  0.029779  \n",
       "\n",
       "[11712 rows x 768 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_train = embedder.encode(train_df[\"text\"].tolist())\n",
    "emb_train_df = pd.DataFrame(emb_train, index=train_df.index)\n",
    "emb_train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1671</th>\n",
       "      <td>-0.002864</td>\n",
       "      <td>0.217326</td>\n",
       "      <td>0.124350</td>\n",
       "      <td>-0.082548</td>\n",
       "      <td>0.709688</td>\n",
       "      <td>-0.582442</td>\n",
       "      <td>0.257897</td>\n",
       "      <td>0.169356</td>\n",
       "      <td>0.248880</td>\n",
       "      <td>-0.266686</td>\n",
       "      <td>...</td>\n",
       "      <td>0.501767</td>\n",
       "      <td>0.095387</td>\n",
       "      <td>0.340173</td>\n",
       "      <td>0.087452</td>\n",
       "      <td>-0.368359</td>\n",
       "      <td>0.276195</td>\n",
       "      <td>0.238676</td>\n",
       "      <td>-0.219546</td>\n",
       "      <td>0.066603</td>\n",
       "      <td>0.256149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10951</th>\n",
       "      <td>-0.141048</td>\n",
       "      <td>0.137934</td>\n",
       "      <td>0.131319</td>\n",
       "      <td>0.194774</td>\n",
       "      <td>0.868205</td>\n",
       "      <td>0.078791</td>\n",
       "      <td>-0.131656</td>\n",
       "      <td>0.036243</td>\n",
       "      <td>-0.215749</td>\n",
       "      <td>-0.291946</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.056256</td>\n",
       "      <td>-0.056040</td>\n",
       "      <td>0.147341</td>\n",
       "      <td>0.189665</td>\n",
       "      <td>-0.357366</td>\n",
       "      <td>0.061799</td>\n",
       "      <td>-0.161922</td>\n",
       "      <td>-0.278956</td>\n",
       "      <td>-0.173722</td>\n",
       "      <td>0.065324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5382</th>\n",
       "      <td>-0.252943</td>\n",
       "      <td>0.527507</td>\n",
       "      <td>-0.065608</td>\n",
       "      <td>0.013467</td>\n",
       "      <td>0.207989</td>\n",
       "      <td>0.003881</td>\n",
       "      <td>-0.066281</td>\n",
       "      <td>0.253166</td>\n",
       "      <td>0.021039</td>\n",
       "      <td>0.290956</td>\n",
       "      <td>...</td>\n",
       "      <td>0.180685</td>\n",
       "      <td>-0.042605</td>\n",
       "      <td>-0.173794</td>\n",
       "      <td>-0.079129</td>\n",
       "      <td>-0.169160</td>\n",
       "      <td>0.001317</td>\n",
       "      <td>-0.142593</td>\n",
       "      <td>-0.070816</td>\n",
       "      <td>-0.208826</td>\n",
       "      <td>0.400736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3954</th>\n",
       "      <td>0.054318</td>\n",
       "      <td>0.096738</td>\n",
       "      <td>0.113037</td>\n",
       "      <td>0.032039</td>\n",
       "      <td>0.493064</td>\n",
       "      <td>-0.641102</td>\n",
       "      <td>0.078760</td>\n",
       "      <td>0.402187</td>\n",
       "      <td>0.189743</td>\n",
       "      <td>-0.089538</td>\n",
       "      <td>...</td>\n",
       "      <td>0.123879</td>\n",
       "      <td>-0.285019</td>\n",
       "      <td>-0.297771</td>\n",
       "      <td>0.557171</td>\n",
       "      <td>0.076168</td>\n",
       "      <td>-0.029826</td>\n",
       "      <td>-0.076095</td>\n",
       "      <td>0.225454</td>\n",
       "      <td>0.002134</td>\n",
       "      <td>0.235429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11193</th>\n",
       "      <td>-0.065858</td>\n",
       "      <td>0.223270</td>\n",
       "      <td>0.507333</td>\n",
       "      <td>0.266193</td>\n",
       "      <td>0.104696</td>\n",
       "      <td>-0.219555</td>\n",
       "      <td>0.146247</td>\n",
       "      <td>0.315650</td>\n",
       "      <td>-0.126193</td>\n",
       "      <td>-0.435462</td>\n",
       "      <td>...</td>\n",
       "      <td>0.163994</td>\n",
       "      <td>0.207813</td>\n",
       "      <td>-0.001871</td>\n",
       "      <td>0.109391</td>\n",
       "      <td>-0.166779</td>\n",
       "      <td>-0.249199</td>\n",
       "      <td>-0.525419</td>\n",
       "      <td>-0.413066</td>\n",
       "      <td>0.119939</td>\n",
       "      <td>0.064297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5861</th>\n",
       "      <td>0.077512</td>\n",
       "      <td>0.322276</td>\n",
       "      <td>0.026697</td>\n",
       "      <td>-0.111392</td>\n",
       "      <td>0.174208</td>\n",
       "      <td>0.235201</td>\n",
       "      <td>0.053888</td>\n",
       "      <td>0.244941</td>\n",
       "      <td>0.181625</td>\n",
       "      <td>-0.226870</td>\n",
       "      <td>...</td>\n",
       "      <td>0.149843</td>\n",
       "      <td>0.311337</td>\n",
       "      <td>0.045975</td>\n",
       "      <td>-0.572319</td>\n",
       "      <td>-0.068256</td>\n",
       "      <td>0.217745</td>\n",
       "      <td>-0.056509</td>\n",
       "      <td>-0.355174</td>\n",
       "      <td>-0.028610</td>\n",
       "      <td>0.090676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3627</th>\n",
       "      <td>-0.173311</td>\n",
       "      <td>-0.023604</td>\n",
       "      <td>0.190388</td>\n",
       "      <td>-0.136543</td>\n",
       "      <td>-0.360269</td>\n",
       "      <td>-0.444687</td>\n",
       "      <td>0.056311</td>\n",
       "      <td>0.291941</td>\n",
       "      <td>-0.399719</td>\n",
       "      <td>-0.167930</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042209</td>\n",
       "      <td>-0.161905</td>\n",
       "      <td>-0.040535</td>\n",
       "      <td>-0.050516</td>\n",
       "      <td>-0.252020</td>\n",
       "      <td>-0.133981</td>\n",
       "      <td>0.155001</td>\n",
       "      <td>-0.154482</td>\n",
       "      <td>-0.060201</td>\n",
       "      <td>-0.126555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12559</th>\n",
       "      <td>-0.124636</td>\n",
       "      <td>-0.101799</td>\n",
       "      <td>0.129061</td>\n",
       "      <td>0.636908</td>\n",
       "      <td>0.681090</td>\n",
       "      <td>0.399300</td>\n",
       "      <td>-0.078321</td>\n",
       "      <td>0.221824</td>\n",
       "      <td>-0.277218</td>\n",
       "      <td>-0.178589</td>\n",
       "      <td>...</td>\n",
       "      <td>0.022364</td>\n",
       "      <td>-0.109274</td>\n",
       "      <td>-0.073540</td>\n",
       "      <td>-0.153336</td>\n",
       "      <td>-0.123705</td>\n",
       "      <td>-0.238896</td>\n",
       "      <td>0.296447</td>\n",
       "      <td>-0.116798</td>\n",
       "      <td>0.115076</td>\n",
       "      <td>-0.345925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8123</th>\n",
       "      <td>0.063508</td>\n",
       "      <td>0.332506</td>\n",
       "      <td>0.119605</td>\n",
       "      <td>-0.001362</td>\n",
       "      <td>-0.161801</td>\n",
       "      <td>-0.082302</td>\n",
       "      <td>-0.025883</td>\n",
       "      <td>0.048027</td>\n",
       "      <td>0.126974</td>\n",
       "      <td>-0.159802</td>\n",
       "      <td>...</td>\n",
       "      <td>0.002221</td>\n",
       "      <td>-0.093885</td>\n",
       "      <td>0.430285</td>\n",
       "      <td>-0.088562</td>\n",
       "      <td>0.321488</td>\n",
       "      <td>0.447437</td>\n",
       "      <td>0.292395</td>\n",
       "      <td>-0.188566</td>\n",
       "      <td>-0.272767</td>\n",
       "      <td>0.126173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>0.015537</td>\n",
       "      <td>0.425568</td>\n",
       "      <td>0.350672</td>\n",
       "      <td>0.113120</td>\n",
       "      <td>-0.128615</td>\n",
       "      <td>0.098112</td>\n",
       "      <td>0.222081</td>\n",
       "      <td>0.101654</td>\n",
       "      <td>0.224073</td>\n",
       "      <td>-0.341075</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100983</td>\n",
       "      <td>-0.008055</td>\n",
       "      <td>0.202025</td>\n",
       "      <td>0.029846</td>\n",
       "      <td>-0.019182</td>\n",
       "      <td>0.107063</td>\n",
       "      <td>0.002301</td>\n",
       "      <td>0.038213</td>\n",
       "      <td>-0.139270</td>\n",
       "      <td>-0.007586</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2928 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "1671  -0.002864  0.217326  0.124350 -0.082548  0.709688 -0.582442  0.257897   \n",
       "10951 -0.141048  0.137934  0.131319  0.194774  0.868205  0.078791 -0.131656   \n",
       "5382  -0.252943  0.527507 -0.065608  0.013467  0.207989  0.003881 -0.066281   \n",
       "3954   0.054318  0.096738  0.113037  0.032039  0.493064 -0.641102  0.078760   \n",
       "11193 -0.065858  0.223270  0.507333  0.266193  0.104696 -0.219555  0.146247   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5861   0.077512  0.322276  0.026697 -0.111392  0.174208  0.235201  0.053888   \n",
       "3627  -0.173311 -0.023604  0.190388 -0.136543 -0.360269 -0.444687  0.056311   \n",
       "12559 -0.124636 -0.101799  0.129061  0.636908  0.681090  0.399300 -0.078321   \n",
       "8123   0.063508  0.332506  0.119605 -0.001362 -0.161801 -0.082302 -0.025883   \n",
       "210    0.015537  0.425568  0.350672  0.113120 -0.128615  0.098112  0.222081   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "1671   0.169356  0.248880 -0.266686  ...  0.501767  0.095387  0.340173   \n",
       "10951  0.036243 -0.215749 -0.291946  ... -0.056256 -0.056040  0.147341   \n",
       "5382   0.253166  0.021039  0.290956  ...  0.180685 -0.042605 -0.173794   \n",
       "3954   0.402187  0.189743 -0.089538  ...  0.123879 -0.285019 -0.297771   \n",
       "11193  0.315650 -0.126193 -0.435462  ...  0.163994  0.207813 -0.001871   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5861   0.244941  0.181625 -0.226870  ...  0.149843  0.311337  0.045975   \n",
       "3627   0.291941 -0.399719 -0.167930  ...  0.042209 -0.161905 -0.040535   \n",
       "12559  0.221824 -0.277218 -0.178589  ...  0.022364 -0.109274 -0.073540   \n",
       "8123   0.048027  0.126974 -0.159802  ...  0.002221 -0.093885  0.430285   \n",
       "210    0.101654  0.224073 -0.341075  ...  0.100983 -0.008055  0.202025   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "1671   0.087452 -0.368359  0.276195  0.238676 -0.219546  0.066603  0.256149  \n",
       "10951  0.189665 -0.357366  0.061799 -0.161922 -0.278956 -0.173722  0.065324  \n",
       "5382  -0.079129 -0.169160  0.001317 -0.142593 -0.070816 -0.208826  0.400736  \n",
       "3954   0.557171  0.076168 -0.029826 -0.076095  0.225454  0.002134  0.235429  \n",
       "11193  0.109391 -0.166779 -0.249199 -0.525419 -0.413066  0.119939  0.064297  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5861  -0.572319 -0.068256  0.217745 -0.056509 -0.355174 -0.028610  0.090676  \n",
       "3627  -0.050516 -0.252020 -0.133981  0.155001 -0.154482 -0.060201 -0.126555  \n",
       "12559 -0.153336 -0.123705 -0.238896  0.296447 -0.116798  0.115076 -0.345925  \n",
       "8123  -0.088562  0.321488  0.447437  0.292395 -0.188566 -0.272767  0.126173  \n",
       "210    0.029846 -0.019182  0.107063  0.002301  0.038213 -0.139270 -0.007586  \n",
       "\n",
       "[2928 rows x 768 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_test = embedder.encode(test_df[\"text\"].tolist())\n",
    "emb_test_df = pd.DataFrame(emb_test, index=test_df.index)\n",
    "emb_test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.87\n",
      "Test accuracy 0.83\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=1000)\n",
    "lgr.fit(emb_train, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(emb_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(emb_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Some improvement over bag of words and average embedding representations! \n",
    "- But much slower ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Topic modeling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling motivation\n",
    "\n",
    "- Suppose you have a large collection of documents on a variety of topics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: A corpus of news articles \n",
    "\n",
    "![](img/TM_NYT_articles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_NYT_articles.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: A corpus of food magazines \n",
    "\n",
    "![](img/TM_food_magazines.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_food_magazines.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A corpus of scientific articles\n",
    "\n",
    "![](img/TM_science_articles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_science_articles.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->\n",
    "\n",
    "(Credit: [Dave Blei's presentation](http://www.cs.columbia.edu/~blei/talks/Blei_Science_2008.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling motivation\n",
    "\n",
    "- Humans are pretty good at reading and understanding a document and answering questions such as \n",
    "    - What is it about?  \n",
    "    - Which documents is it related to?     \n",
    "- But for a large collection of documents it would take years to read all documents and organize and categorize them so that they are easy to search.\n",
    "- You need an automated way\n",
    "    - to get an idea of what's going on in the data or \n",
    "    - to pull documents related to a certain topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling\n",
    "\n",
    "- Topic modeling gives you an ability to summarize the major themes in a large collection of documents (corpus). \n",
    "    - Example: The major themes in a collection of news articles could be \n",
    "        - **politics**\n",
    "        - **entertainment**\n",
    "        - **sports**\n",
    "        - **technology**\n",
    "        - ...\n",
    "- A common tool to solve such problems is unsupervised ML methods.\n",
    "- Given the hyperparameter $K$, the idea of topic modeling is to describe the data using $K$ \"topics\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Input and output\n",
    "\n",
    "- Input\n",
    "    - A large collection of documents\n",
    "    - A value for the hyperparameter $K$ (e.g., $K = 3$)\n",
    "- Output\n",
    "    1. Topic-words association \n",
    "        - For each topic, what words describe that topic? \n",
    "    2. Document-topics association\n",
    "        - For each document, what topics are expressed by the document? \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Example\n",
    "\n",
    "- Topic-words association \n",
    "    - For each topic, what words describe that topic?  \n",
    "    - A topic is a mixture of words. \n",
    "\n",
    "![](img/topic_modeling_word_topics.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/topic_modeling_word_topics.png\" height=\"1000\" width=\"1000\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Example\n",
    "\n",
    "- Document-topics association \n",
    "    - For each document, what topics are expressed by the document?\n",
    "    - A document is a mixture of topics. \n",
    "    \n",
    "![](img/topic_modeling_doc_topics.png)\n",
    "\n",
    "<!-- <center> -->    \n",
    "<!-- <img src=\"img/topic_modeling_doc_topics.png\" height=\"800\" width=\"800\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Input and output\n",
    "\n",
    "- Input\n",
    "    - A large collection of documents\n",
    "    - A value for the hyperparameter $K$ (e.g., $K = 3$)\n",
    "- Output\n",
    "    - For each topic, what words describe that topic?  \n",
    "    - For each document, what topics are expressed by the document?\n",
    "\n",
    "![](img/topic_modeling_output.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/topic_modeling_output.png\" height=\"800\" width=\"800\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Some applications\n",
    "\n",
    "- Topic modeling is a great EDA tool to get a sense of what's going on in a large corpus. \n",
    "- Some examples\n",
    "    - If you want to pull documents related to a particular lawsuit. \n",
    "    - You want to examine people's sentiment towards a particular candidate and/or political party and so you want to pull tweets or Facebook posts related to election.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling toy example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>fashion model pattern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>fresh fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>creative fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>probabilistic model pattern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>kiwi health nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>fresh apple health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>creative health nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>hidden markov model probabilistic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>apple kiwi health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>fresh kiwi health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    doc_id                                               text\n",
       "0        1                               famous fashion model\n",
       "1        2                             fashion model pattern \n",
       "2        3  fashion model probabilistic topic model confer...\n",
       "3        4                               famous fashion model\n",
       "4        5                                fresh fashion model\n",
       "5        6                               famous fashion model\n",
       "6        7                               famous fashion model\n",
       "7        8                               famous fashion model\n",
       "8        9                               famous fashion model\n",
       "9       10                             creative fashion model\n",
       "10      11                               famous fashion model\n",
       "11      12                               famous fashion model\n",
       "12      13  fashion model probabilistic topic model confer...\n",
       "13      14                          probabilistic topic model\n",
       "14      15                        probabilistic model pattern\n",
       "15      16                          probabilistic topic model\n",
       "16      17                          probabilistic topic model\n",
       "17      18                          probabilistic topic model\n",
       "18      19                          probabilistic topic model\n",
       "19      20                          probabilistic topic model\n",
       "20      21                          probabilistic topic model\n",
       "21      22  fashion model probabilistic topic model confer...\n",
       "22      23                               apple kiwi nutrition\n",
       "23      24                              kiwi health nutrition\n",
       "24      25                                 fresh apple health\n",
       "25      26                          probabilistic topic model\n",
       "26      27                          creative health nutrition\n",
       "27      28                          probabilistic topic model\n",
       "28      29                          probabilistic topic model\n",
       "29      30                  hidden markov model probabilistic\n",
       "30      31                          probabilistic topic model\n",
       "31      32                          probabilistic topic model\n",
       "32      33                               apple kiwi nutrition\n",
       "33      34                                  apple kiwi health\n",
       "34      35                               apple kiwi nutrition\n",
       "35      36                                  fresh kiwi health\n",
       "36      37                               apple kiwi nutrition\n",
       "37      38                               apple kiwi nutrition\n",
       "38      39                               apple kiwi nutrition"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df = pd.read_csv(\"data/toy_lda_data.csv\")\n",
    "toy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from gensim import corpora, matutils, models\n",
    "\n",
    "corpus = [doc.split() for doc in toy_df[\"text\"].tolist()]\n",
    "# Create a vocabulary for the lda (Latent Dirichlet Allocation) model\n",
    "dictionary = corpora.Dictionary(corpus)\n",
    "# Convert our corpus into document-term matrix for Lda\n",
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in corpus]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "\n",
    "# Train an lda model\n",
    "lda = models.LdaModel(\n",
    "    corpus=doc_term_matrix,\n",
    "    id2word=dictionary,\n",
    "    num_topics=3,\n",
    "    random_state=123,\n",
    "    passes=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.303*\"model\" + 0.296*\"probabilistic\" + 0.261*\"topic\" + 0.040*\"pattern\"'),\n",
       " (1, '0.245*\"kiwi\" + 0.219*\"apple\" + 0.218*\"nutrition\" + 0.140*\"health\"'),\n",
       " (2, '0.308*\"fashion\" + 0.307*\"model\" + 0.180*\"famous\" + 0.071*\"conference\"')]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Examine the topics in our LDA model\n",
    "lda.print_topics(num_words=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document:  ['famous', 'fashion', 'model']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic id</th>\n",
       "      <th>probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.828760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.087849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.083391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   topic id  probability\n",
       "2         2     0.828760\n",
       "0         0     0.087849\n",
       "1         1     0.083391"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Examine the topic distribution for a document\n",
    "print(\"Document: \", corpus[0])\n",
    "df = pd.DataFrame(lda[doc_term_matrix[0]], columns=[\"topic id\", \"probability\"])\n",
    "df.sort_values(\"probability\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "You can also visualize the topics using `pyLDAvis`. \n",
    "\n",
    "```\n",
    "pip install pyLDAvis\n",
    "\n",
    "```\n",
    "\n",
    "> Do not install it using `conda`. They have made some changes in the recent version and `conda` build is not available for this version yet. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Visualize the topics\n",
    "import pyLDAvis\n",
    "\n",
    "pyLDAvis.enable_notebook()\n",
    "import pyLDAvis.gensim_models as gensimvis\n",
    "\n",
    "vis = gensimvis.prepare(lda, doc_term_matrix, dictionary, sort_topics=False)\n",
    "vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Topic modeling pipeline \n",
    "\n",
    "- Preprocess your corpus. \n",
    "- Train LDA using `Gensim`.\n",
    "- Interpret your topics.     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is intelligence d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Unsupervised learning is a type of machine lea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey is a contact winter team sport play...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \n",
       "0  Artificial intelligence (AI) is intelligence d...  \n",
       "1  Unsupervised learning is a type of machine lea...  \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...  \n",
       "3  In many Commonwealth jurisdictions, the phrase...  \n",
       "4  Canadian constitutional law (French: droit con...  \n",
       "5  Ice hockey is a contact winter team sport play...  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import wikipedia\n",
    "\n",
    "queries = [\n",
    "    \"Artificial Intelligence\",\n",
    "    \"unsupervised learning\",\n",
    "    \"Supreme Court of Canada\",\n",
    "    \"Peace, Order, and Good Government\",\n",
    "    \"Canadian constitutional law\",\n",
    "    \"ice hockey\",\n",
    "]\n",
    "wiki_dict = {\"wiki query\": [], \"text\": []}\n",
    "for i in range(len(queries)):\n",
    "    wiki_dict[\"text\"].append(wikipedia.page(queries[i]).content)\n",
    "    wiki_dict[\"wiki query\"].append(queries[i])\n",
    "\n",
    "wiki_df = pd.DataFrame(wiki_dict)\n",
    "wiki_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing the corpus \n",
    "\n",
    "- **Preprocessing is crucial!**\n",
    "- Tokenization, converting text to lower case\n",
    "- Removing punctuation and stopwords\n",
    "- Discarding words with length < threshold or word frequency < threshold        \n",
    "- Possibly lemmatization: Consider the lemmas instead of inflected forms. \n",
    "- Depending upon your application, restrict to specific part of speech;\n",
    "    * For example, only consider nouns, verbs, and adjectives\n",
    "    \n",
    "We'll use [`spaCy`](https://spacy.io/) for preprocessing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\", disable=[\"parser\", \"ner\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(\n",
    "    doc,\n",
    "    min_token_len=2,\n",
    "    irrelevant_pos=[\"ADV\", \"PRON\", \"CCONJ\", \"PUNCT\", \"PART\", \"DET\", \"ADP\", \"SPACE\"],\n",
    "):\n",
    "    \"\"\"\n",
    "    Given text, min_token_len, and irrelevant_pos carry out preprocessing of the text\n",
    "    and return a preprocessed string.\n",
    "\n",
    "    Parameters\n",
    "    -------------\n",
    "    doc : (spaCy doc object)\n",
    "        the spacy doc object of the text\n",
    "    min_token_len : (int)\n",
    "        min_token_length required\n",
    "    irrelevant_pos : (list)\n",
    "        a list of irrelevant pos tags\n",
    "\n",
    "    Returns\n",
    "    -------------\n",
    "    (str) the preprocessed text\n",
    "    \"\"\"\n",
    "\n",
    "    clean_text = []\n",
    "\n",
    "    for token in doc:\n",
    "        if (\n",
    "            token.is_stop == False  # Check if it's not a stopword\n",
    "            and len(token) > min_token_len  # Check if the word meets minimum threshold\n",
    "            and token.pos_ not in irrelevant_pos\n",
    "        ):  # Check if the POS is in the acceptable POS tags\n",
    "            lemma = token.lemma_  # Take the lemma of the word\n",
    "            clean_text.append(lemma.lower())\n",
    "    return \" \".join(clean_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "wiki_df[\"text_pp\"] = [preprocess(text) for text in nlp.pipe(wiki_df[\"text\"])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "      <th>text_pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is intelligence d...</td>\n",
       "      <td>artificial intelligence intelligence demonstra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Unsupervised learning is a type of machine lea...</td>\n",
       "      <td>unsupervised learning type machine learning al...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "      <td>supreme court canada scc french cour suprême c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "      <td>commonwealth jurisdiction phrase peace order g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "      <td>canadian constitutional law french droit const...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey is a contact winter team sport play...</td>\n",
       "      <td>ice hockey contact winter team sport play ice ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \\\n",
       "0  Artificial intelligence (AI) is intelligence d...   \n",
       "1  Unsupervised learning is a type of machine lea...   \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...   \n",
       "3  In many Commonwealth jurisdictions, the phrase...   \n",
       "4  Canadian constitutional law (French: droit con...   \n",
       "5  Ice hockey is a contact winter team sport play...   \n",
       "\n",
       "                                             text_pp  \n",
       "0  artificial intelligence intelligence demonstra...  \n",
       "1  unsupervised learning type machine learning al...  \n",
       "2  supreme court canada scc french cour suprême c...  \n",
       "3  commonwealth jurisdiction phrase peace order g...  \n",
       "4  canadian constitutional law french droit const...  \n",
       "5  ice hockey contact winter team sport play ice ...  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training LDA with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html)\n",
    "\n",
    "To train an LDA model with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html), you need\n",
    "\n",
    "- Document-term matrix \n",
    "- Dictionary (vocabulary)\n",
    "- The number of topics ($K$): `num_topics`\n",
    "- The number of passes: `passes`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `Gensim`'s `doc2bow`\n",
    "- Let's first create a dictionary using [`corpora.Dictionary`](https://radimrehurek.com/gensim/corpora/dictionary.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4683</th>\n",
       "      <td>wrist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4684</th>\n",
       "      <td>yale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4685</th>\n",
       "      <td>youth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4686</th>\n",
       "      <td>zhenskaya</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4687</th>\n",
       "      <td>zhhl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4688 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Word\n",
       "0          0026\n",
       "1          0030\n",
       "2          0036\n",
       "3          0040\n",
       "4           007\n",
       "...         ...\n",
       "4683      wrist\n",
       "4684       yale\n",
       "4685      youth\n",
       "4686  zhenskaya\n",
       "4687       zhhl\n",
       "\n",
       "[4688 rows x 1 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = [doc.split() for doc in wiki_df[\"text_pp\"].tolist()]\n",
    "dictionary = corpora.Dictionary(corpus)  # Create a vocabulary for the lda model\n",
    "pd.DataFrame(\n",
    "    dictionary.token2id.keys(), index=dictionary.token2id.values(), columns=[\"Word\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `Gensim`'s `doc2bow`\n",
    "- Now let's convert our corpus into document-term matrix for LDA using [`dictionary.doc2bow`](https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.doc2bow).\n",
    "- For each document, it stores the frequency of each token in the document in the format (token_id, frequency). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(105, 1),\n",
       " (113, 1),\n",
       " (121, 1),\n",
       " (161, 1),\n",
       " (258, 2),\n",
       " (277, 1),\n",
       " (282, 1),\n",
       " (292, 2),\n",
       " (295, 4),\n",
       " (304, 1),\n",
       " (312, 2),\n",
       " (315, 1),\n",
       " (343, 13),\n",
       " (352, 1),\n",
       " (365, 6),\n",
       " (367, 1),\n",
       " (368, 1),\n",
       " (391, 3),\n",
       " (410, 4),\n",
       " (413, 1)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in corpus]\n",
    "doc_term_matrix[1][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we are ready to train an LDA model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "\n",
    "num_topics = 3\n",
    "\n",
    "lda = models.LdaModel(\n",
    "    corpus=doc_term_matrix,\n",
    "    id2word=dictionary,\n",
    "    num_topics=num_topics,\n",
    "    random_state=42,\n",
    "    passes=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examine the topics and topic distribution for a document in our LDA model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, '0.019*\"hockey\" + 0.011*\"ice\" + 0.010*\"player\" + 0.010*\"team\"'),\n",
       " (1, '0.037*\"court\" + 0.014*\"justice\" + 0.012*\"supreme\" + 0.011*\"law\"'),\n",
       " (2, '0.011*\"power\" + 0.010*\"law\" + 0.010*\"government\" + 0.009*\"provincial\"')]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.print_topics(num_words=4)  # Topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document:  Artificial Intelligence\n",
      "Topic assignment for document:  [(0, 0.9998983)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Document: \", wiki_df.iloc[0][0])\n",
    "print(\"Topic assignment for document: \", lda[doc_term_matrix[0]])  # Topic distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Visualize topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/pyLDAvis/_prepare.py:246: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only\n",
      "  default_term_info = default_term_info.sort_values(\n",
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  from imp import reload\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el7348856078391044998229239\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el7348856078391044998229239_data = {\"mdsDat\": {\"x\": [0.15260526131253155, -0.058827069776560006, -0.09377819153597156], \"y\": [0.013189887080643238, -0.09298041831048383, 0.07979053122984058], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [72.3415705327995, 9.429272312965818, 18.229157154234684]}, \"tinfo\": {\"Term\": [\"court\", \"law\", \"canada\", \"hockey\", \"justice\", \"case\", \"power\", \"provincial\", \"supreme\", \"government\", \"federal\", \"network\", \"year\", \"matter\", \"learning\", \"act\", \"ice\", \"appeal\", \"player\", \"decision\", \"team\", \"canadian\", \"league\", \"good\", \"doctrine\", \"parliament\", \"game\", \"judge\", \"constitution\", \"section\", \"hockey\", \"player\", \"ice\", \"league\", \"team\", \"game\", \"intelligence\", \"puck\", \"penalty\", \"retrieve\", \"archive\", \"goal\", \"sport\", \"nhl\", \"2019\", \"2015\", \"professional\", \"2020\", \"artificial\", \"2018\", \"minute\", \"original\", \"tournament\", \"university\", \"line\", \"knowledge\", \"play\", \"search\", \"logic\", \"agent\", \"woman\", \"stick\", \"world\", \"problem\", \"human\", \"system\", \"october\", \"machine\", \"include\", \"canada\", \"use\", \"national\", \"appeal\", \"appointment\", \"appoint\", \"clerk\", \"puisne\", \"superior\", \"tenure\", \"counsel\", \"minister\", \"prime\", \"lamer\", \"hear\", \"litigant\", \"override\", \"hearing\", \"judge\", \"building\", \"statue\", \"wagner\", \"scc\", \"appellate\", \"1949\", \"laskin\", \"website\", \"lawyer\", \"recommend\", \"dissent\", \"panel\", \"corporation\", \"justice\", \"supreme\", \"court\", \"territorial\", \"chief\", \"day\", \"french\", \"case\", \"law\", \"federal\", \"judicial\", \"year\", \"canada\", \"provincial\", \"decision\", \"charter\", \"parliament\", \"member\", \"canadian\", \"government\", \"reference\", \"power\", \"time\", \"act\", \"doctrine\", \"peace\", \"pogg\", \"substance\", \"pith\", \"variable\", \"purpose\", \"residual\", \"phrase\", \"immunity\", \"emergency\", \"boltzmann\", \"latent\", \"paramountcy\", \"interjurisdictional\", \"commonwealth\", \"authority\", \"unsupervised\", \"helmholtz\", \"federalism\", \"jurisprudence\", \"legislate\", \"enumerate\", \"branch\", \"cluster\", \"hopfield\", \"conflict\", \"competence\", \"invalid\", \"limitation\", \"matter\", \"section\", \"legislative\", \"legislation\", \"power\", \"constitution\", \"provincial\", \"government\", \"constitutional\", \"moment\", \"aspect\", \"jurisdiction\", \"act\", \"order\", \"federal\", \"law\", \"province\", \"good\", \"parliament\", \"network\", \"method\", \"learning\", \"training\", \"concern\", \"court\", \"canada\", \"right\", \"state\", \"canadian\", \"neural\"], \"Freq\": [95.0, 63.0, 90.0, 259.0, 32.0, 43.0, 50.0, 43.0, 29.0, 47.0, 40.0, 57.0, 30.0, 30.0, 56.0, 37.0, 157.0, 14.0, 139.0, 25.0, 135.0, 34.0, 124.0, 27.0, 16.0, 18.0, 118.0, 11.0, 19.0, 18.0, 258.28443202311325, 138.84887430571155, 156.7527974257029, 123.48401153366082, 135.09872815911805, 118.01577722775616, 108.0756723229997, 70.42239692077987, 67.72461286083131, 69.41427037486703, 64.95001026138907, 55.97528861384745, 51.49355293143989, 48.824066511694184, 39.852452641214846, 37.178379168477406, 36.26625633277471, 38.02098901160111, 88.3311147114373, 32.62330903067314, 29.94229658597721, 78.44401719415241, 25.483760556660418, 25.46203369144678, 25.458312793440054, 24.574899589262657, 109.90794371512754, 25.43746505931606, 24.550545062858564, 24.54431977665551, 47.07118617180574, 38.039194486416925, 42.54329772920492, 44.31273882264553, 53.306172013084094, 39.81894509499735, 37.13976876910858, 57.81840802945784, 43.46749483027905, 52.67099169075031, 38.959339798815904, 38.09000566817747, 13.390850073086451, 7.0511722314713, 5.991888045690977, 4.939795075430042, 4.405695841198676, 3.87847915033234, 3.877440235971716, 3.875496869914968, 3.351935829884927, 3.3516605619774116, 3.3507546303463016, 7.056569765786875, 2.824081287039348, 2.822739620264313, 2.821772799879467, 9.699801728942612, 6.523504371853321, 2.2937482759297074, 2.2926634835844975, 2.291458816505369, 2.291589050784194, 2.291399196267336, 2.2909146232688515, 1.7646488963410767, 1.7639451872549274, 1.7637834515028157, 1.7632890475076346, 1.763227524496047, 1.7627425286589953, 24.518896465136788, 21.87130968757321, 67.94487402460365, 3.356685998353487, 7.058781211495488, 7.578993158650754, 3.890170636726828, 18.744147863394772, 19.322993566205174, 13.997033134970142, 5.474333216094589, 10.25490081779589, 18.83025640065767, 10.844582181739932, 8.14238157478351, 5.476632612225266, 6.063811617526177, 4.946576137538845, 6.597164964524275, 7.203960719492437, 4.438221006658004, 4.581314229162351, 4.4361275329097545, 4.005575544146767, 15.982350205596749, 14.565930717624692, 9.098571864991245, 7.784943872097182, 7.782028833277275, 7.782084420108389, 7.091937943874324, 7.063192195107622, 7.059701669095001, 6.40730028609437, 6.375081177836421, 5.723973414839296, 5.724562390013088, 5.722155398480497, 5.721185898749591, 5.6931971118301234, 5.017284542901623, 13.276945674889053, 4.3514229722373345, 4.349153721602431, 4.33724097322259, 4.329891249714516, 4.324623171286334, 12.502773163798475, 3.6639924849205845, 3.663696158063541, 3.661506200408316, 3.661125267124503, 3.650662681074026, 3.645340241994828, 26.915835191138342, 15.963977123006662, 8.458613757626239, 11.905535988505035, 37.87298218275987, 15.305634488703987, 32.4665056243578, 34.43606149421466, 13.266413605299684, 7.0975800072324295, 7.0960783453389435, 11.194886963577165, 24.19166193039557, 15.277319537878421, 25.63245133909087, 35.95662944990662, 10.5155112533249, 17.265861580374423, 12.519049478420794, 27.076276769709096, 13.986547542013136, 22.34428428042123, 9.84779569835227, 10.479310329561688, 23.636377280978877, 19.468702301163784, 12.636123519281579, 11.918007875009614, 11.899208081745385, 9.915238056108539], \"Total\": [95.0, 63.0, 90.0, 259.0, 32.0, 43.0, 50.0, 43.0, 29.0, 47.0, 40.0, 57.0, 30.0, 30.0, 56.0, 37.0, 157.0, 14.0, 139.0, 25.0, 135.0, 34.0, 124.0, 27.0, 16.0, 18.0, 118.0, 11.0, 19.0, 18.0, 259.5041344231166, 139.6148890615649, 157.63439321219406, 124.26094460330101, 135.94996516738468, 118.82974628779006, 108.8745652711093, 71.04758417069948, 68.34565749172044, 70.1012037833671, 65.60563094972439, 56.5994839897733, 52.08976647291506, 49.39597430248892, 40.378729394953545, 37.684907272905896, 36.77919255802795, 38.56746291682266, 89.73343927327541, 33.156947562523264, 30.457696244325792, 79.82393307143205, 25.960493740205848, 25.953177340642377, 25.95262673515277, 25.05496729165476, 112.08304768619807, 25.942441210443867, 25.04775055638393, 25.04516878394897, 48.13332198686975, 39.2651891697231, 44.150329251679445, 46.251754608780615, 57.18712430136374, 41.96607692399352, 39.268909537848735, 65.99089997233493, 54.54435310872088, 90.96995039257177, 45.74522401333829, 49.127251121399595, 14.040644648853146, 7.637616533190348, 6.574391279541901, 5.50383491741461, 4.974257961451311, 4.4398108021836, 4.440328020900442, 4.441778000911138, 3.9049318476074073, 3.905243251761704, 3.9061121052568826, 8.324567516249678, 3.371107036043117, 3.3721465651492966, 3.372694830969668, 11.890369878452105, 8.003819968277782, 2.8394530140335186, 2.839823668838771, 2.8402854077949002, 2.8404567671036425, 2.840565750612257, 2.840912367127982, 2.306086895907014, 2.3067484154337197, 2.306950049633244, 2.3070711000138533, 2.3072012134985003, 2.3072053002618578, 32.1221663013713, 29.668275394749465, 95.08297925533795, 4.592915097879342, 11.295389193927678, 12.683454792995013, 5.813665579742924, 43.43100870759807, 63.83709882982558, 40.96174154338469, 10.598343394987515, 30.66341755831693, 90.96995039257177, 43.75557180145938, 25.402000797912812, 13.623504770664574, 18.96885836624335, 11.600597721797405, 34.13889363945781, 47.49391202668328, 18.64876787601272, 50.06687580463475, 31.3901829071091, 37.55530326051573, 16.56414911559203, 15.17142891481635, 9.661970721635488, 8.289434874820302, 8.288820819946865, 8.289082466464205, 7.601372271051188, 7.594287570285901, 7.597885372457066, 6.911835551997204, 6.907164706100892, 6.220886725486666, 6.221933331311336, 6.222369043628188, 6.222656074279122, 6.218649918375647, 5.529552040909347, 14.704468982783528, 4.843965172373244, 4.8440938557198345, 4.841639418112528, 4.840520543305193, 4.840625387321919, 14.164688352910328, 4.153796381998607, 4.154458970988824, 4.154999650322861, 4.154928596288244, 4.152134199060869, 4.151858039306166, 30.81845520612945, 18.5172107472706, 10.044225338835588, 14.556044957032952, 50.06687580463475, 19.067704734697983, 43.75557180145938, 47.49391202668328, 17.000181287079194, 8.5015808514805, 8.499555747550048, 14.601048425160066, 37.55530326051573, 21.598276121591717, 40.96174154338469, 63.83709882982558, 13.708216355445526, 27.096771693489163, 18.96885836624335, 57.34107869437999, 23.504002578139954, 56.106379273261794, 13.961363024846591, 15.704427081485562, 95.08297925533795, 90.96995039257177, 30.938503436457562, 30.445706546927184, 34.13889363945781, 37.39749653425975], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.988, -4.6086, -4.4873, -4.7259, -4.636, -4.7712, -4.8592, -5.2875, -5.3266, -5.3019, -5.3684, -5.5171, -5.6006, -5.6538, -5.8568, -5.9263, -5.9511, -5.9039, -5.0609, -6.057, -6.1427, -5.1796, -6.304, -6.3048, -6.305, -6.3403, -4.8424, -6.3058, -6.3413, -6.3415, -5.6904, -5.9034, -5.7915, -5.7507, -5.566, -5.8577, -5.9273, -5.4847, -5.77, -5.5779, -5.8795, -5.9021, -4.9099, -5.5512, -5.714, -5.9071, -6.0215, -6.149, -6.1493, -6.1498, -6.2949, -6.295, -6.2952, -5.5505, -6.4663, -6.4667, -6.4671, -5.2323, -5.629, -6.6742, -6.6747, -6.6752, -6.6752, -6.6753, -6.6755, -6.9365, -6.9369, -6.937, -6.9373, -6.9373, -6.9376, -4.305, -4.4193, -3.2857, -6.2935, -5.5502, -5.4791, -6.146, -4.5736, -4.5431, -4.8656, -5.8044, -5.1767, -4.569, -5.1208, -5.4074, -5.8039, -5.7021, -5.9057, -5.6178, -5.5298, -6.0142, -5.9824, -6.0147, -6.1167, -5.3922, -5.485, -5.9555, -6.1114, -6.1118, -6.1118, -6.2047, -6.2087, -6.2092, -6.3062, -6.3112, -6.419, -6.4189, -6.4193, -6.4195, -6.4244, -6.5507, -5.5776, -6.6931, -6.6937, -6.6964, -6.6981, -6.6993, -5.6377, -6.8651, -6.8652, -6.8658, -6.8659, -6.8687, -6.8702, -4.8709, -5.3933, -6.0285, -5.6866, -4.5294, -5.4354, -4.6834, -4.6245, -5.5784, -6.2039, -6.2041, -5.7482, -4.9776, -5.4373, -4.9198, -4.5813, -5.8108, -5.3149, -5.6364, -4.865, -5.5255, -5.0571, -5.8764, -5.8142, -5.0009, -5.1948, -5.6271, -5.6856, -5.6872, -5.8696], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.3191, 0.3183, 0.3182, 0.3175, 0.3175, 0.3169, 0.3164, 0.3149, 0.3146, 0.3139, 0.3137, 0.3127, 0.3123, 0.3121, 0.3107, 0.3102, 0.3097, 0.3095, 0.308, 0.3075, 0.3067, 0.3063, 0.3052, 0.3047, 0.3045, 0.3044, 0.3042, 0.3041, 0.3037, 0.3036, 0.3015, 0.292, 0.2867, 0.2809, 0.2535, 0.2713, 0.268, 0.1916, 0.0968, -0.2227, 0.1632, 0.0693, 2.314, 2.2815, 2.2686, 2.2532, 2.24, 2.2262, 2.2258, 2.225, 2.2086, 2.2085, 2.208, 2.1961, 2.1843, 2.1835, 2.183, 2.1577, 2.1568, 2.1479, 2.1473, 2.1466, 2.1466, 2.1465, 2.1462, 2.0938, 2.0931, 2.0929, 2.0926, 2.0925, 2.0922, 2.0912, 2.0564, 2.0253, 2.0478, 1.8912, 1.8464, 1.9596, 1.5211, 1.1663, 1.2876, 1.7007, 1.266, 0.7863, 0.9664, 1.2236, 1.45, 1.2209, 1.509, 0.7176, 0.4754, 0.9258, -0.03, 0.4046, 0.1232, 1.6664, 1.6614, 1.6421, 1.6394, 1.6391, 1.639, 1.6328, 1.6296, 1.6287, 1.6264, 1.622, 1.6189, 1.6188, 1.6183, 1.6181, 1.6139, 1.6049, 1.6, 1.5949, 1.5944, 1.5921, 1.5907, 1.5894, 1.5773, 1.5767, 1.5764, 1.5757, 1.5756, 1.5734, 1.572, 1.5667, 1.5538, 1.5303, 1.5011, 1.423, 1.4824, 1.4037, 1.3807, 1.4542, 1.5216, 1.5217, 1.4365, 1.2623, 1.3559, 1.2334, 1.1281, 1.437, 1.2515, 1.2866, 0.9518, 1.1831, 0.7815, 1.3531, 1.2976, 0.3102, 0.1604, 0.8067, 0.7643, 0.6482, 0.3746]}, \"token.table\": {\"Topic\": [2, 1, 1, 1, 1, 1, 2, 3, 1, 2, 2, 2, 2, 1, 1, 3, 1, 3, 3, 3, 2, 3, 1, 2, 1, 2, 3, 1, 2, 3, 1, 2, 3, 2, 3, 1, 2, 3, 2, 3, 3, 3, 1, 2, 3, 3, 2, 3, 2, 3, 2, 2, 1, 2, 3, 1, 2, 1, 2, 3, 2, 3, 3, 3, 1, 2, 3, 3, 2, 3, 1, 3, 1, 1, 2, 3, 1, 2, 3, 2, 3, 2, 3, 1, 2, 3, 3, 1, 2, 3, 1, 3, 3, 1, 2, 3, 1, 3, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 3, 2, 3, 1, 2, 2, 3, 1, 2, 3, 2, 1, 1, 3, 3, 2, 3, 2, 3, 3, 1, 2, 1, 1, 3, 1, 2, 3, 1, 2, 3, 1, 3, 2, 1, 1, 3, 1, 2, 3, 1, 3, 1, 3, 1, 1, 2, 1, 2, 3, 1, 3, 2, 2, 3, 2, 3, 3, 1, 3, 3, 1, 3, 1, 3, 1, 2, 3, 2, 1, 3, 1, 2, 3, 2, 3, 1, 2, 3, 2, 1, 2, 3, 3, 1, 1, 2, 3, 2, 1, 1, 2, 3, 1, 1, 3, 2, 1, 3, 3, 2, 1, 2, 3, 1, 2, 1, 2, 2, 3, 1, 2, 3, 1, 1, 3, 1, 1, 3, 1, 2, 3, 3, 2, 2, 1, 2, 1, 2, 1, 2], \"Freq\": [0.7040850927562297, 0.9818254223648224, 0.9952665255983738, 0.9906205717557601, 0.9852864857082642, 0.23964658033962066, 0.10650959126205363, 0.6390575475723217, 0.9981965071052779, 0.9258834138403932, 0.7041121073070795, 0.9126320209554177, 0.9165162940009498, 0.990768613288873, 0.9806823488844959, 0.011144117600960181, 0.11765320796775111, 0.8235724557742578, 0.9042323795867087, 0.964492726642698, 0.07059809401274518, 0.9177752221656873, 0.12494034148236527, 0.8745823903765569, 0.5826099692402136, 0.20886017765215206, 0.20886017765215206, 0.46867365325240545, 0.2050447232979274, 0.35150523993930405, 0.34537535383965917, 0.4374754481969016, 0.2302502358931061, 0.3670127536319784, 0.5872204058111654, 0.0885316993360081, 0.6197218953520567, 0.2655950980080243, 0.9084574801071098, 0.9629745014307592, 0.9648396482764607, 0.9627120917489058, 0.25470524835100317, 0.06367631208775079, 0.636763120877508, 0.9626956285517818, 0.15733409142532107, 0.7866704571266053, 0.1764687063825677, 0.7646977276577933, 0.8668496036191529, 0.9005402789557433, 0.04206851774446729, 0.715164801655944, 0.25241110646680376, 0.3942143589112224, 0.6307429742579558, 0.6298716438633708, 0.3149358219316854, 0.07873395548292135, 0.8669000274798598, 0.9659415577790839, 0.8686632294580118, 0.8263395077992194, 0.02441302450338564, 0.34178234304739896, 0.6347386370880267, 0.8257478321310515, 0.6880340716427787, 0.34401703582138937, 0.9930173520207598, 0.008415401288311523, 0.9894083135125116, 0.29523812247797204, 0.036904765309746505, 0.6273810102656906, 0.12633198117327224, 0.1473873113688176, 0.7158812266485426, 0.8408845247919363, 0.12012636068456234, 0.8894964265526164, 0.8257697687038172, 0.9942038132592363, 0.0038535031521675827, 0.0038535031521675827, 0.9628209179420393, 0.9267820448655802, 0.01748645367670906, 0.05245936103012718, 0.9959755406211379, 0.006343793252363936, 0.8680762085357014, 0.7883492524751367, 0.03666740709186682, 0.1650033319134007, 0.9919672214632358, 0.9642184829723349, 0.9633599995165669, 0.08410167305326759, 0.8410167305326759, 0.08410167305326759, 0.18870873734341348, 0.4717718433585337, 0.28306310601512025, 0.13697646509778455, 0.06848823254889227, 0.753370558037815, 0.8261664396229171, 0.7782787675479018, 0.2179180549134125, 0.997806131973157, 0.7680271121667429, 0.7039991881276852, 0.9643304870859873, 0.14098385053480963, 0.29763257335126475, 0.5639354021392385, 0.8670212956985843, 0.9898524463392215, 0.6059916972080059, 0.39211227466400383, 0.8263574060298748, 0.13739996035349372, 0.8243997621209623, 0.09955969388037729, 0.7964775510430183, 0.9634240771556959, 0.9632936293934965, 0.8899153802963465, 0.998093618974828, 0.8789090620724234, 0.12122883614792046, 0.0973442692028033, 0.032448089734267764, 0.8760984228252297, 0.5172147284036978, 0.43101227366974815, 0.08620245473394963, 0.3829135046288034, 0.5956432294225831, 0.7682592467876569, 0.9849727228003642, 0.11762518259481776, 0.8233762781637244, 0.7735014504698673, 0.040710602656308804, 0.1831977119533896, 0.5231851350389805, 0.4708666215350824, 0.7219734608509254, 0.26739757809293535, 0.9919836725951781, 0.9422212237479658, 0.0509308769593495, 0.18519996584362652, 0.09259998292181326, 0.6944998719135994, 0.9771505486982223, 0.012527571137156695, 0.8896410467458966, 0.8668511390765615, 0.964262961250121, 0.31630791290410476, 0.6853338112922269, 0.9887005425936556, 0.9949425098183843, 0.9213089770182046, 0.9651553790073705, 0.9814151405658594, 0.01784391164665199, 0.9955958202903864, 0.931486987416224, 0.15978628327472813, 0.09986642704670509, 0.7589848455549586, 0.7681979857840258, 0.951315260840869, 0.04324160276549405, 0.9788143103794738, 0.21884685229732778, 0.8024384584235352, 0.2513965547042198, 0.7313354318668212, 0.9852551753458282, 0.8041400407856899, 0.9208863545150349, 0.8669455154947796, 0.32173707345661146, 0.21449138230440762, 0.42898276460881524, 0.9217454481693366, 0.9842912286246877, 0.5171549436080929, 0.06464436795101161, 0.4201883916815755, 0.7041545876027758, 0.9636718378660347, 0.05400381373028322, 0.05400381373028322, 0.8640610196845315, 0.9790790677957503, 0.5912163664934457, 0.3941442443289638, 0.7043610125314054, 0.9677783503282171, 0.025467851324426766, 0.9650838833779274, 0.9009392918348478, 0.03370603739835092, 0.7415328227637203, 0.20223622439010555, 0.9531508049333665, 0.04765754024666832, 0.9930123912410345, 0.9008343485373521, 0.6531799382455755, 0.21772664608185852, 0.7964273439877956, 0.1274283750380473, 0.06371418751902365, 0.9630017152286168, 0.28650497754992316, 0.716262443874808, 0.9632731927913231, 0.06800653605178349, 0.8840849686731853, 0.8525480165673354, 0.0437204111060172, 0.0874408222120344, 0.9651249136881231, 0.7042690790790604, 0.8672700077129462, 0.97645452380829, 0.02077562816613383, 0.9739451716175891, 0.022649887712036954, 0.6522430176598284, 0.3261215088299142], \"Term\": [\"1949\", \"2015\", \"2018\", \"2019\", \"2020\", \"act\", \"act\", \"act\", \"agent\", \"appeal\", \"appellate\", \"appoint\", \"appointment\", \"archive\", \"artificial\", \"artificial\", \"aspect\", \"aspect\", \"authority\", \"boltzmann\", \"branch\", \"branch\", \"building\", \"building\", \"canada\", \"canada\", \"canada\", \"canadian\", \"canadian\", \"canadian\", \"case\", \"case\", \"case\", \"charter\", \"charter\", \"chief\", \"chief\", \"chief\", \"clerk\", \"cluster\", \"commonwealth\", \"competence\", \"concern\", \"concern\", \"concern\", \"conflict\", \"constitution\", \"constitution\", \"constitutional\", \"constitutional\", \"corporation\", \"counsel\", \"court\", \"court\", \"court\", \"day\", \"day\", \"decision\", \"decision\", \"decision\", \"dissent\", \"doctrine\", \"emergency\", \"enumerate\", \"federal\", \"federal\", \"federal\", \"federalism\", \"french\", \"french\", \"game\", \"game\", \"goal\", \"good\", \"good\", \"good\", \"government\", \"government\", \"government\", \"hear\", \"hear\", \"hearing\", \"helmholtz\", \"hockey\", \"hockey\", \"hockey\", \"hopfield\", \"human\", \"human\", \"human\", \"ice\", \"ice\", \"immunity\", \"include\", \"include\", \"include\", \"intelligence\", \"interjurisdictional\", \"invalid\", \"judge\", \"judge\", \"judge\", \"judicial\", \"judicial\", \"judicial\", \"jurisdiction\", \"jurisdiction\", \"jurisdiction\", \"jurisprudence\", \"justice\", \"justice\", \"knowledge\", \"lamer\", \"laskin\", \"latent\", \"law\", \"law\", \"law\", \"lawyer\", \"league\", \"learning\", \"learning\", \"legislate\", \"legislation\", \"legislation\", \"legislative\", \"legislative\", \"limitation\", \"line\", \"litigant\", \"logic\", \"machine\", \"machine\", \"matter\", \"matter\", \"matter\", \"member\", \"member\", \"member\", \"method\", \"method\", \"minister\", \"minute\", \"moment\", \"moment\", \"national\", \"national\", \"national\", \"network\", \"network\", \"neural\", \"neural\", \"nhl\", \"october\", \"october\", \"order\", \"order\", \"order\", \"original\", \"original\", \"override\", \"panel\", \"paramountcy\", \"parliament\", \"parliament\", \"peace\", \"penalty\", \"phrase\", \"pith\", \"play\", \"play\", \"player\", \"pogg\", \"power\", \"power\", \"power\", \"prime\", \"problem\", \"problem\", \"professional\", \"province\", \"province\", \"provincial\", \"provincial\", \"puck\", \"puisne\", \"purpose\", \"recommend\", \"reference\", \"reference\", \"reference\", \"residual\", \"retrieve\", \"right\", \"right\", \"right\", \"scc\", \"search\", \"section\", \"section\", \"section\", \"sport\", \"state\", \"state\", \"statue\", \"stick\", \"stick\", \"substance\", \"superior\", \"supreme\", \"supreme\", \"supreme\", \"system\", \"system\", \"team\", \"tenure\", \"territorial\", \"territorial\", \"time\", \"time\", \"time\", \"tournament\", \"training\", \"training\", \"university\", \"unsupervised\", \"unsupervised\", \"use\", \"use\", \"use\", \"variable\", \"wagner\", \"website\", \"woman\", \"woman\", \"world\", \"world\", \"year\", \"year\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 2, 3]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el7348856078391044998229239\", ldavis_el7348856078391044998229239_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el7348856078391044998229239\", ldavis_el7348856078391044998229239_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el7348856078391044998229239\", ldavis_el7348856078391044998229239_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x         y  topics  cluster       Freq\n",
       "topic                                                \n",
       "0      0.152605  0.013190       1        1  72.341571\n",
       "1     -0.058827 -0.092980       2        1   9.429272\n",
       "2     -0.093778  0.079791       3        1  18.229157, topic_info=          Term        Freq       Total Category  logprob  loglift\n",
       "744      court   95.000000   95.000000  Default  30.0000  30.0000\n",
       "1437       law   63.000000   63.000000  Default  29.0000  29.0000\n",
       "568     canada   90.000000   90.000000  Default  28.0000  28.0000\n",
       "4129    hockey  259.000000  259.000000  Default  27.0000  27.0000\n",
       "2998   justice   32.000000   32.000000  Default  26.0000  26.0000\n",
       "...        ...         ...         ...      ...      ...      ...\n",
       "568     canada   19.468702   90.969950   Topic3  -5.1948   0.1604\n",
       "2032     right   12.636124   30.938503   Topic3  -5.6271   0.8067\n",
       "2230     state   11.918008   30.445707   Topic3  -5.6856   0.7643\n",
       "2866  canadian   11.899208   34.138894   Topic3  -5.6872   0.6482\n",
       "1676    neural    9.915238   37.397497   Topic3  -5.8696   0.3746\n",
       "\n",
       "[185 rows x 6 columns], token_table=      Topic      Freq   Term\n",
       "term                        \n",
       "2794      2  0.704085   1949\n",
       "129       1  0.981825   2015\n",
       "136       1  0.995267   2018\n",
       "137       1  0.990621   2019\n",
       "139       1  0.985286   2020\n",
       "...     ...       ...    ...\n",
       "3213      2  0.020776  woman\n",
       "2537      1  0.973945  world\n",
       "2537      2  0.022650  world\n",
       "2547      1  0.652243   year\n",
       "2547      2  0.326122   year\n",
       "\n",
       "[239 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[1, 2, 3])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vis = gensimvis.prepare(lda, doc_term_matrix, dictionary, sort_topics=False)\n",
    "vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Topic modeling with `sklearn`\n",
    "\n",
    "- We are using `Gensim` LDA so that we'll be able to use `CoherenceModel` to evaluate topic model later. \n",
    "- Bit we can also train an LDA model with `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X = vec.fit_transform(wiki_df[\"text_pp\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "n_topics = 3\n",
    "lda = LatentDirichletAllocation(\n",
    "    n_components=n_topics, learning_method=\"batch\", max_iter=10, random_state=0\n",
    ")\n",
    "document_topics = lda.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lda.components_.shape: (3, 4696)\n"
     ]
    }
   ],
   "source": [
    "print(\"lda.components_.shape: {}\".format(lda.components_.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sorting = np.argsort(lda.components_, axis=1)[:, ::-1]\n",
    "feature_names = np.array(vec.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "topic 0       topic 1       topic 2       \n",
      "--------      --------      --------      \n",
      "hockey        intelligence  court         \n",
      "ice           artificial    law           \n",
      "player        original      provincial    \n",
      "team          retrieve      government    \n",
      "league        archive       power         \n",
      "game          machine       federal       \n",
      "play          human         canada        \n",
      "puck          10            justice       \n",
      "penalty       problem       supreme       \n",
      "canada        doi           case          \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/mglearn/plot_pca.py:7: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.\n",
      "You provided \"cachedir='cache'\", use \"location='cache'\" instead.\n",
      "  memory = Memory(cachedir=\"cache\")\n",
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.9/site-packages/mglearn/plot_nmf.py:7: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.\n",
      "You provided \"cachedir='cache'\", use \"location='cache'\" instead.\n",
      "  memory = Memory(cachedir=\"cache\")\n"
     ]
    }
   ],
   "source": [
    "import mglearn\n",
    "\n",
    "mglearn.tools.print_topics(\n",
    "    topics=range(3),\n",
    "    feature_names=feature_names,\n",
    "    sorting=sorting,\n",
    "    topics_per_chunk=5,\n",
    "    n_words=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic text preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Introduction \n",
    "- Why do we need preprocessing?\n",
    "    - Text data is unstructured and messy. \n",
    "    - We need to \"normalize\" it before we do anything interesting with it. \n",
    "- Example:     \n",
    "    - **Lemma**: Same stem, same part-of-speech, roughly the same meaning\n",
    "        - Vancouver's &rarr; Vancouver\n",
    "        - computers &rarr; computer \n",
    "        - rising &rarr; rise, rose, rises    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tokenization\n",
    "\n",
    "- Sentence segmentation\n",
    "    - Split text into sentences\n",
    "- Word tokenization \n",
    "    - Split sentences into words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tokenization: sentence segmentation\n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. MDS teaching team is truly multicultural!! Dr. George did his Ph.D. in Scotland. Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. Dr. Gelbart did his PhD in the U.S.\n",
    "</blockquote>\n",
    "\n",
    "- How many sentences are there in this text? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"MDS is a Master's program at UBC in British Columbia\", ' MDS teaching team is truly multicultural!! Dr', ' George did his Ph', 'D', ' in Scotland', ' Dr', ' Timbers, Dr', ' Ostblom, Dr', ' Rodríguez-Arelis, and Dr', ' Kolhatkar did theirs in Canada', ' Dr', ' Gelbart did his PhD in the U', 'S', '']\n"
     ]
    }
   ],
   "source": [
    "### Let's do sentence segmentation on \".\"\n",
    "text = (\n",
    "    \"MDS is a Master's program at UBC in British Columbia. \"\n",
    "    \"MDS teaching team is truly multicultural!! \"\n",
    "    \"Dr. George did his Ph.D. in Scotland. \"\n",
    "    \"Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. \"\n",
    "    \"Dr. Gelbart did his PhD in the U.S.\"\n",
    ")\n",
    "\n",
    "print(text.split(\".\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentence segmentation\n",
    "\n",
    "- In English, period (.) is quite ambiguous. (In Chinese, it is unambiguous.)\n",
    "    - Abbreviations like Dr., U.S., Inc.  \n",
    "    - Numbers like 60.44%, 0.98\n",
    "- ! and ? are relatively ambiguous.\n",
    "- How about writing regular expressions? \n",
    "- A common way is using off-the-shelf models for sentence segmentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"MDS is a Master's program at UBC in British Columbia.\", 'MDS teaching team is truly multicultural!!', 'Dr. George did his Ph.D. in Scotland.', 'Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada.', 'Dr. Gelbart did his PhD in the U.S.']\n"
     ]
    }
   ],
   "source": [
    "### Let's try to do sentence segmentation using nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "sent_tokenized = sent_tokenize(text)\n",
    "print(sent_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word tokenization\n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. \n",
    "</blockquote>\n",
    "\n",
    "- How many words are there in this sentence?  \n",
    "- Is whitespace a sufficient condition for a word boundary?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word tokenization \n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. \n",
    "</blockquote>\n",
    "\n",
    "- What's our definition of a word?\n",
    "    - Should British Columbia be one word or two words? \n",
    "    - Should punctuation be considered a separate word?\n",
    "    - What about the punctuations in `U.S.`?\n",
    "    - What do we do with words like `Master's`?\n",
    "- This process of identifying word boundaries is referred to as **tokenization**.\n",
    "- You can use regex but better to do it with off-the-shelf ML models.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Splitting on whitespace:  [['MDS', 'is', 'a', \"Master's\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia.'], ['MDS', 'teaching', 'team', 'is', 'truly', 'multicultural!!'], ['Dr.', 'George', 'did', 'his', 'Ph.D.', 'in', 'Scotland.'], ['Dr.', 'Timbers,', 'Dr.', 'Ostblom,', 'Dr.', 'Rodríguez-Arelis,', 'and', 'Dr.', 'Kolhatkar', 'did', 'theirs', 'in', 'Canada.'], ['Dr.', 'Gelbart', 'did', 'his', 'PhD', 'in', 'the', 'U.S.']]\n",
      "\n",
      "\n",
      "\n",
      "Tokenized:  [['MDS', 'is', 'a', 'Master', \"'s\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia', '.'], ['MDS', 'teaching', 'team', 'is', 'truly', 'multicultural', '!', '!'], ['Dr.', 'George', 'did', 'his', 'Ph.D.', 'in', 'Scotland', '.'], ['Dr.', 'Timbers', ',', 'Dr.', 'Ostblom', ',', 'Dr.', 'Rodríguez-Arelis', ',', 'and', 'Dr.', 'Kolhatkar', 'did', 'theirs', 'in', 'Canada', '.'], ['Dr.', 'Gelbart', 'did', 'his', 'PhD', 'in', 'the', 'U.S', '.']]\n"
     ]
    }
   ],
   "source": [
    "### Let's do word segmentation on white spaces\n",
    "print(\"Splitting on whitespace: \", [sent.split() for sent in sent_tokenized])\n",
    "\n",
    "### Let's try to do word segmentation using nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "word_tokenized = [word_tokenize(sent) for sent in sent_tokenized]\n",
    "# This is similar to the input format of word2vec algorithm\n",
    "print(\"\\n\\n\\nTokenized: \", word_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word segmentation \n",
    "\n",
    "For some languages you need much more sophisticated tokenizers. \n",
    "- For languages such as Chinese, there are no spaces between words.\n",
    "    - [jieba](https://github.com/fxsjy/jieba) is a popular tokenizer for Chinese. \n",
    "- German doesn't separate compound words.\n",
    "    * Example: _rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz_\n",
    "    * (the law for the delegation of monitoring beef labeling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Types and tokens\n",
    "- Usually in NLP, we talk about \n",
    "    - **Type** an element in the vocabulary\n",
    "    - **Token** an instance of that type in running text \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise for you \n",
    "\n",
    "<blockquote>    \n",
    "UBC is located in the beautiful province of British Columbia. It's very close \n",
    "to the U.S. border. You'll get to the USA border in about 45 mins by car.     \n",
    "</blockquote>  \n",
    "\n",
    "- Consider the example above. \n",
    "    - How many types? (task dependent)\n",
    "    - How many tokens? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other commonly used preprocessing steps\n",
    "\n",
    "- Punctuation and stopword removal\n",
    "- Stemming and lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Punctuation and stopword removal\n",
    "\n",
    "- The most frequently occurring words in English are not very useful in many NLP tasks.\n",
    "    - Example: _the_ , _is_ , _a_ , and punctuation\n",
    "- Probably not very informative in many tasks "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['than', 'these', 'didn', 'on', 'ours', 'most', 'they', 'more', \"it's\", 'couldn', 'this', 'does', 're', 'their', 'under', 'just', 'don', 'you', 'not', \"needn't\", 'wasn', 'will', 'of', 'him', \"hadn't\", 'll', 'shouldn', 'own', 'with', 'she', 'was', \"you're\", 'through', 'here', \"mustn't\", 'when', 'or', 'as', 'further', 'very', 'hasn', 'what', 'but', 'me', 'we', 'should', 'mustn', 'such', \"aren't\", 'same', \"weren't\", 'then', 'so', \"don't\", \"you'd\", 'both', 'myself', 'my', 's', 'an', 'them', 'have', \"wouldn't\", \"you'll\", 'below', 'in', 'up', 'having', \"you've\", 'nor', \"didn't\", 'had', 'off', \"should've\", 'until', \"couldn't\", 'been', 'his', 'themselves', \"hasn't\", 'while', 'down', 'are', 'be', 'a', 'o', 'any', 'all', \"doesn't\", 'because', 'about', 'against', 'again', \"that'll\", 'doing', 'to', 'won', \"won't\", 'd', 'herself', \"mightn't\", 'for', 'shan', 'doesn', 'by', 'm', 'wouldn', 'those', 'at', 'needn', 'yourselves', 'over', 'himself', 'too', 'it', 'why', 'yourself', 'is', \"shan't\", 've', 'and', 'there', 'out', 'during', 'aren', \"wasn't\", 'some', 'the', 'your', 'can', \"shouldn't\", 'where', 'no', 'other', 'into', 'he', 'whom', 'weren', \"isn't\", 'isn', 'who', 'yours', 'hadn', 'once', 'hers', 'y', 'how', 'ourselves', 'that', 'haven', 'which', 'were', 'each', 'being', 'now', 'i', 'ain', 'above', 'ma', 'our', 'did', 'am', 'after', 'do', 'has', 'theirs', 'if', \"haven't\", 'few', 'her', 't', \"she's\", 'itself', 'mightn', 'its', 'between', 'from', 'only', 'before', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\\\', ']', '^', '_', '`', '{', '|', '}', '~']\n"
     ]
    }
   ],
   "source": [
    "# Let's use `nltk.stopwords`.\n",
    "# Add punctuations to the list.\n",
    "stop_words = list(set(stopwords.words(\"english\")))\n",
    "import string\n",
    "\n",
    "punctuation = string.punctuation\n",
    "stop_words += list(punctuation)\n",
    "# stop_words.extend(['``','`','br','\"',\"”\", \"''\", \"'s\"])\n",
    "print(stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mds', 'master', \"'s\", 'program', 'ubc', 'british', 'columbia', 'mds', 'teaching', 'team', 'truly', 'multicultural', 'dr.', 'george', 'ph.d.', 'scotland', 'dr.', 'timbers', 'dr.', 'ostblom', 'dr.', 'rodríguez-arelis', 'dr.', 'kolhatkar', 'canada', 'dr.', 'gelbart', 'phd', 'u.s']\n"
     ]
    }
   ],
   "source": [
    "### Get rid of stop words\n",
    "preprocessed = []\n",
    "for sent in word_tokenized:\n",
    "    for token in sent:\n",
    "        token = token.lower()\n",
    "        if token not in stop_words:\n",
    "            preprocessed.append(token)\n",
    "print(preprocessed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Lemmatization \n",
    "\n",
    "- For many NLP tasks (e.g., web search) we want to ignore morphological differences between words\n",
    "    - Example: If your search term is \"studying for ML quiz\" you might want to include pages containing \"tips to study for an ML quiz\" or \"here is how I studied for my ML quiz\"\n",
    "- Lemmatization converts inflected forms into the base form. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /Users/kvarada/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download(\"wordnet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lemma of studying:  study\n",
      "Lemma of studied:  study\n"
     ]
    }
   ],
   "source": [
    "# nltk has a lemmatizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(\"Lemma of studying: \", lemmatizer.lemmatize(\"studying\", \"v\"))\n",
    "print(\"Lemma of studied: \", lemmatizer.lemmatize(\"studied\", \"v\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Stemming\n",
    "\n",
    "- Has a similar purpose but it is a crude chopping of affixes \n",
    "    * _automates, automatic, automation_ all reduced to _automat_.\n",
    "- Usually these reduced forms (stems) are not actual words themselves.  \n",
    "- A popular stemming algorithm for English is PorterStemmer. \n",
    "- Beware that it can be aggressive sometimes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before stemming:  UBC is located in the beautiful province of British Columbia... It's very close to the U.S. border.\n",
      "\n",
      "\n",
      "After stemming:  ubc is locat in the beauti provinc of british columbia ... It 's veri close to the u.s. border .\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "text = (\n",
    "    \"UBC is located in the beautiful province of British Columbia... \"\n",
    "    \"It's very close to the U.S. border.\"\n",
    ")\n",
    "ps = PorterStemmer()\n",
    "tokenized = word_tokenize(text)\n",
    "stemmed = [ps.stem(token) for token in tokenized]\n",
    "print(\"Before stemming: \", text)\n",
    "print(\"\\n\\nAfter stemming: \", \" \".join(stemmed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other tools for preprocessing \n",
    "\n",
    "- We used [Natural Language Processing Toolkit (nltk)](https://www.nltk.org/) above\n",
    "- Many available tools    \n",
    "- [spaCy](https://spacy.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [spaCy](https://spacy.io/)\n",
    "\n",
    "- Industrial strength NLP library. \n",
    "- Lightweight, fast, and convenient to use. \n",
    "- spaCy does many things that we did above in one line of code! \n",
    "- Also has [multi-lingual](https://spacy.io/models/xx) support. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# Load the model\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "text = (\n",
    "    \"MDS is a Master's program at UBC in British Columbia. \"\n",
    "    \"MDS teaching team is truly multicultural!! \"\n",
    "    \"Dr. George did his Ph.D. in Scotland. \"\n",
    "    \"Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. \"\n",
    "    \"Dr. Gelbart did his PhD in the U.S.\"\n",
    ")\n",
    "\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Tokens:  [MDS, is, a, Master, 's, program, at, UBC, in, British, Columbia, ., MDS, teaching, team, is, truly, multicultural, !, !, Dr., George, did, his, Ph.D., in, Scotland, ., Dr., Timbers, ,, Dr., Ostblom, ,, Dr., Rodríguez, -, Arelis, ,, and, Dr., Kolhatkar, did, theirs, in, Canada, ., Dr., Gelbart, did, his, PhD, in, the, U.S.]\n",
      "\n",
      "Lemmas:  ['MDS', 'be', 'a', 'Master', \"'s\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia', '.', 'MDS', 'teaching', 'team', 'be', 'truly', 'multicultural', '!', '!', 'Dr.', 'George', 'do', 'his', 'ph.d.', 'in', 'Scotland', '.', 'Dr.', 'Timbers', ',', 'Dr.', 'Ostblom', ',', 'Dr.', 'Rodríguez', '-', 'Arelis', ',', 'and', 'Dr.', 'Kolhatkar', 'do', 'theirs', 'in', 'Canada', '.', 'Dr.', 'Gelbart', 'do', 'his', 'phd', 'in', 'the', 'U.S.']\n",
      "\n",
      "POS:  ['PROPN', 'AUX', 'DET', 'PROPN', 'PART', 'NOUN', 'ADP', 'PROPN', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'NOUN', 'NOUN', 'AUX', 'ADV', 'ADJ', 'PUNCT', 'PUNCT', 'PROPN', 'PROPN', 'VERB', 'PRON', 'NOUN', 'ADP', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'CCONJ', 'PROPN', 'PROPN', 'VERB', 'PRON', 'ADP', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'VERB', 'PRON', 'NOUN', 'ADP', 'DET', 'PROPN']\n"
     ]
    }
   ],
   "source": [
    "# Accessing tokens\n",
    "tokens = [token for token in doc]\n",
    "print(\"\\nTokens: \", tokens)\n",
    "\n",
    "# Accessing lemma\n",
    "lemmas = [token.lemma_ for token in doc]\n",
    "print(\"\\nLemmas: \", lemmas)\n",
    "\n",
    "# Accessing pos\n",
    "pos = [token.pos_ for token in doc]\n",
    "print(\"\\nPOS: \", pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other typical NLP tasks \n",
    "In order to understand text, we usually are interested in extracting information from text. Some common tasks in NLP pipeline are: \n",
    "- Part of speech tagging\n",
    "    - Assigning part-of-speech tags to all words in a sentence.\n",
    "- Named entity recognition\n",
    "    - Labelling named “real-world” objects, like persons, companies or locations.    \n",
    "- Coreference resolution\n",
    "    - Deciding whether two strings (e.g., UBC vs University of British Columbia) refer to the same entity\n",
    "- Dependency parsing\n",
    "    - Representing grammatical structure of a sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting named-entities using spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    University of British Columbia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is located in the beautiful province of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    British Columbia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('University of British Columbia', 'ORG'), ('British Columbia', 'GPE')]\n",
      "\n",
      "ORG means:  Companies, agencies, institutions, etc.\n",
      "GPE means:  Countries, cities, states\n"
     ]
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "doc = nlp(\n",
    "    \"University of British Columbia \"\n",
    "    \"is located in the beautiful \"\n",
    "    \"province of British Columbia.\"\n",
    ")\n",
    "displacy.render(doc, style=\"ent\")\n",
    "# Text and label of named entity span\n",
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])\n",
    "print(\"\\nORG means: \", spacy.explain(\"ORG\"))\n",
    "print(\"GPE means: \", spacy.explain(\"GPE\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dependency parsing using spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"b85896da282b428c9522d2619ae1a70a-0\" class=\"displacy\" width=\"575\" height=\"224.5\" direction=\"ltr\" style=\"max-width: none; height: 224.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">like</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">cats</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b85896da282b428c9522d2619ae1a70a-0-0\" stroke-width=\"2px\" d=\"M70,89.5 C70,2.0 225.0,2.0 225.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b85896da282b428c9522d2619ae1a70a-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,91.5 L62,79.5 78,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b85896da282b428c9522d2619ae1a70a-0-1\" stroke-width=\"2px\" d=\"M245,89.5 C245,2.0 400.0,2.0 400.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b85896da282b428c9522d2619ae1a70a-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400.0,91.5 L408.0,79.5 392.0,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(\"I like cats\")\n",
    "displacy.render(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Many other things possible\n",
    "\n",
    "- A powerful tool \n",
    "- All my Capstone groups last year used this tool. \n",
    "- You can build your own rule-based searches. \n",
    "- You can also access word vectors using spaCy with bigger models. (Currently we are using `en_core_web_md` model.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}