docs/langchain-retrieval-augmentation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "f0Qrii3fJYbS"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "dQRA1HWOJYbU"
   },
   "source": [
    "#### [LangChain Handbook](https://pinecone.io/learn/langchain)\n",
    "\n",
    "# Retrieval Augmentation\n",
    "\n",
    "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, have no idea about recent world events.\n",
    "\n",
    "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n",
    "\n",
    "A solution to this problem is *retrieval augmentation*. The idea behind this is that we retrieve relevant information from an external knowledge base and give that information to our LLM. In this notebook we will learn how to do that.\n",
    "\n",
    "[![Open full notebook](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/full-link.svg)](https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb)\n",
    "\n",
    "To begin, we must install the prerequisite libraries that we will be using in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "0_4wHAWtmAvJ"
   },
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "  pinecone==5.4.2 \\\n",
    "  pinecone-datasets==1.0.2 \\\n",
    "  pinecone-notebooks==0.1.1 \\\n",
    "  langchain==0.3.20 \\\n",
    "  langchain-openai==0.3.9 \\\n",
    "  langchain-pinecone==0.2.3 \\\n",
    "  tqdm"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "🚨 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "RaVQA7J0JYbV"
   },
   "source": [
    "## Building the Knowledge Base\n",
    "\n",
    "We will download a pre-embedding dataset from `pinecone-datasets`. Allowing us to skip the embedding and preprocessing steps, if you'd rather work through those steps you can find the [full notebook here](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb).\n",
    "\n",
    "The dataset we will be working with in this demo contains 50K chunked wikipedia articles that have been embedded using OpenAI's `text-embedding-ada-002` embedding model. This model produces embeddings with a dimension of 1536."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 262
    },
    "id": "LeAmSrjvKJrV",
    "outputId": "2ec76d28-1f4b-4db4-a919-98ffa01fb8a0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading documents parquet files: 100%|██████████| 5/5 [01:21<00:00, 16.22s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>values</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1-0</td>\n",
       "      <td>[-0.011254455894231796, -0.01698738895356655, ...</td>\n",
       "      <td>{'chunk': 0, 'source': 'https://simple.wikiped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1-1</td>\n",
       "      <td>[-0.0015197008615359664, -0.007858820259571075...</td>\n",
       "      <td>{'chunk': 1, 'source': 'https://simple.wikiped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1-2</td>\n",
       "      <td>[-0.009930099360644817, -0.012211072258651257,...</td>\n",
       "      <td>{'chunk': 2, 'source': 'https://simple.wikiped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1-3</td>\n",
       "      <td>[-0.011600767262279987, -0.012608098797500134,...</td>\n",
       "      <td>{'chunk': 3, 'source': 'https://simple.wikiped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1-4</td>\n",
       "      <td>[-0.026462381705641747, -0.016362832859158516,...</td>\n",
       "      <td>{'chunk': 4, 'source': 'https://simple.wikiped...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                             values  \\\n",
       "0  1-0  [-0.011254455894231796, -0.01698738895356655, ...   \n",
       "1  1-1  [-0.0015197008615359664, -0.007858820259571075...   \n",
       "2  1-2  [-0.009930099360644817, -0.012211072258651257,...   \n",
       "3  1-3  [-0.011600767262279987, -0.012608098797500134,...   \n",
       "4  1-4  [-0.026462381705641747, -0.016362832859158516,...   \n",
       "\n",
       "                                            metadata  \n",
       "0  {'chunk': 0, 'source': 'https://simple.wikiped...  \n",
       "1  {'chunk': 1, 'source': 'https://simple.wikiped...  \n",
       "2  {'chunk': 2, 'source': 'https://simple.wikiped...  \n",
       "3  {'chunk': 3, 'source': 'https://simple.wikiped...  \n",
       "4  {'chunk': 4, 'source': 'https://simple.wikiped...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pinecone_datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset('wikipedia-simple-text-embedding-ada-002-50K')\n",
    "\n",
    "# We drop sparse_values and blob keys as they are not needed for this example\n",
    "dataset.documents.drop(['sparse_values'], axis=1, inplace=True)\n",
    "dataset.documents.drop(['blob'], axis=1, inplace=True)\n",
    "\n",
    "dataset.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "QPUmWYSA43eC"
   },
   "source": [
    "Now we move on to initializing our Pinecone vector database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing the Pinecone client\n",
    "\n",
    "Now the data is ready, we need to set up an index to store it.\n",
    "\n",
    "We begin by initializing our connection to Pinecone. To do this we need a [free API key](https://app.pinecone.io)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
    "    from pinecone_notebooks.colab import Authenticate\n",
    "    Authenticate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.12/site-packages/pinecone/data/index.py:1: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "from pinecone import Pinecone\n",
    "\n",
    "# Instantiate a Pinecone client\n",
    "pc = Pinecone(api_key=os.environ.get(\"PINECONE_API_KEY\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Pinecone Index\n",
    "\n",
    "When creating the index we need to define several configuration properties. \n",
    "\n",
    "- `name` can be anything we like. The name is used as an identifier for the index when performing other operations such as `describe_index`, `delete_index`, and so on. \n",
    "- `metric` specifies the similarity metric that will be used later when you make queries to the index.\n",
    "- `dimension` should correspond to the dimension of the dense vectors produced by your embedding model. In this quick start, we are using made-up data so a small value is simplest.\n",
    "- `spec` holds a specification which tells Pinecone how you would like to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects).\n",
    "\n",
    "There are more configurations available, but this minimal set will get us started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9pT9C4nW4vwo",
    "outputId": "6aea8f5a-9331-4c20-c967-989c668ac681"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"name\": \"langchain-retrieval-augmentation-fast\",\n",
       "    \"dimension\": 1536,\n",
       "    \"metric\": \"dotproduct\",\n",
       "    \"host\": \"langchain-retrieval-augmentation-fast-dojoi3u.svc.aped-4627-b74a.pinecone.io\",\n",
       "    \"spec\": {\n",
       "        \"serverless\": {\n",
       "            \"cloud\": \"aws\",\n",
       "            \"region\": \"us-east-1\"\n",
       "        }\n",
       "    },\n",
       "    \"status\": {\n",
       "        \"ready\": true,\n",
       "        \"state\": \"Ready\"\n",
       "    },\n",
       "    \"deletion_protection\": \"disabled\"\n",
       "}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pinecone import ServerlessSpec\n",
    "\n",
    "index_name = 'langchain-retrieval-augmentation-fast'\n",
    "\n",
    "if not pc.has_index(name=index_name):\n",
    "    pc.create_index(\n",
    "        name=index_name,\n",
    "        dimension=1536,  # dimensionality of text-embedding-ada-002\n",
    "        metric='dotproduct',\n",
    "        spec=ServerlessSpec(\n",
    "            cloud='aws',\n",
    "            region='us-east-1'\n",
    "        )\n",
    "    )\n",
    "\n",
    "pc.describe_index(name=index_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "YgPUwd6REY6z"
   },
   "source": [
    "## Upserting data into the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RFydARw4EcoQ",
    "outputId": "92cbb1cf-bb70-4b3a-a481-ede66a5dc7b0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'': {'vector_count': 50000}},\n",
       " 'total_vector_count': 50000}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instantiate an Index client\n",
    "index = pc.Index(name=index_name)\n",
    "\n",
    "index.describe_index_stats()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "0RqIF2mIDwFu"
   },
   "source": [
    "We should see that the new Pinecone index initially has a `total_vector_count` of `0`, as we haven't added any vectors yet.\n",
    "\n",
    "Now we upsert the data to Pinecone:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 99,
     "referenced_widgets": [
      "47188360f07a4c018bf7d2c5eb6f626c",
      "50d346f4ff614f8cb7793cce0a9c0916",
      "f06f0a851b6c434da5cd9ddb245de902",
      "3216e01669cc470894d25198a2099263",
      "47ecdc632edf4aeeb37239510aff8ced",
      "2ee36ac5673a4b74877dcf3fe63c5eb3",
      "f948308ac3414cfabbea8e2af917446f",
      "21045e189c5d49a082964eec2c5d8f6a",
      "4981784e865746a49a912188fb599985",
      "07c3ae73ab9b4c66813d1646ce6a320d",
      "6acfe82fd33043558667ac26141093f8",
      "75cef31e615d4851b029c3aacacf9c2d",
      "6b3b8cddec674c0bb00620e1c9128219",
      "54d772726617494d98e5ee1618db5cf0",
      "34b9b178e43a4cc1ba516c4f27ffd575",
      "0021290a0a3a44be911c4e94b7d25006",
      "7f7790181f72437cb10a88704d20f778",
      "62da3ff6ab334122aa0657a6f25417bb",
      "2d038c3eb946427297de35460a8d4ae2",
      "5f0fb5f8cfb7459d80f6a92bd0125dcb",
      "6e1f19542e2645ccad3a8c99bf6e030b",
      "7c834ff3591545f7809a46989c3a985b"
     ]
    },
    "id": "W-cIOoTWGY1R",
    "outputId": "0eaf686b-ee21-47ca-e818-f29fb768e3c1"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Upserting records batch: 100%|██████████| 500/500 [07:07<00:00,  1.17it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "batch_size = 100\n",
    "\n",
    "for start in tqdm(range(0, len(dataset.documents), batch_size), \"Upserting records batch\"):\n",
    "    batch = dataset.documents.iloc[start:start + batch_size].to_dict(orient=\"records\")\n",
    "    index.upsert(vectors=batch)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "XaF3daSxyCwB"
   },
   "source": [
    "We've now indexed everything. We can check the number of vectors in our index again using `describe_index_stats()`.\n",
    "\n",
    "We may see that the `total_vector_count` is a bit less than the total we expect (50K). This is because Pinecone is eventually consistent and not all vectors may be reflected in the index yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CaEBhsAM22M3",
    "outputId": "9718d994-c8e1-4ff3-ae1d-b57b9b1d3c46"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'index' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[2], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mindex\u001b[49m\u001b[38;5;241m.\u001b[39mdescribe_index_stats()\n",
      "\u001b[0;31mNameError\u001b[0m: name 'index' is not defined"
     ]
    }
   ],
   "source": [
    "index.describe_index_stats()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "-8P2PryCy8W3"
   },
   "source": [
    "## Creating a Langchain Vector Store and Querying\n",
    "\n",
    "Now that we've build our index we can switch over to LangChain. We need to initialize a LangChain vector store using the same index we just built. For this we will also need a LangChain embedding object, which we initialize like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "FvwwQA4qbcK9"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "# Get openai api key from platform.openai.com\n",
    "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY') or 'OPENAI_API_KEY'\n",
    "\n",
    "model_name = 'text-embedding-ada-002'\n",
    "\n",
    "embed = OpenAIEmbeddings(\n",
    "    model=model_name,\n",
    "    openai_api_key=OPENAI_API_KEY\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "jKuedXN8bcfA"
   },
   "source": [
    "Now initialize the vector store:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "qMXlvXOAyJHy"
   },
   "outputs": [],
   "source": [
    "from langchain_pinecone import PineconeVectorStore\n",
    "\n",
    "pinecone_vectorstore = PineconeVectorStore(\n",
    "    index_name=index_name, \n",
    "    embedding=embed, \n",
    "    text_key=\"text\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "h1Yg5mKse1bO"
   },
   "source": [
    "Now we can query the vector store directly using `pinecone_vectorstore.similarity_search`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "COT5s7hcyPiq",
    "outputId": "4373a9bb-682f-4000-a491-2c948b9b44f2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': '6754-0',\n",
      " 'metadata': {'chunk': 0.0,\n",
      "              'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini',\n",
      "              'title': 'Benito Mussolini',\n",
      "              'wiki-id': '6754'},\n",
      " 'page_content': 'Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 – '\n",
      "                 '28 April 1945) was an Italian politician and journalist. He '\n",
      "                 'was also the Prime Minister of Italy from 1922 until 1943. '\n",
      "                 'He was the leader of the National Fascist Party.\\n'\n",
      "                 '\\n'\n",
      "                 'Biography\\n'\n",
      "                 '\\n'\n",
      "                 'Early life\\n'\n",
      "                 'Benito Mussolini was named after Benito Juarez, a Mexican '\n",
      "                 'opponent of the political power of the Roman Catholic '\n",
      "                 'Church, by his anticlerical (a person who opposes the '\n",
      "                 'political interference of the Roman Catholic Church in '\n",
      "                 \"secular affairs) father. Mussolini's father was a \"\n",
      "                 'blacksmith. Before being involved in politics, Mussolini was '\n",
      "                 'a newspaper editor (where he learned all his propaganda '\n",
      "                 'skills) and elementary school teacher.\\n'\n",
      "                 '\\n'\n",
      "                 'At first, Mussolini was a socialist, but when he wanted '\n",
      "                 'Italy to join the First World War, he was thrown out of the '\n",
      "                 \"socialist party. He 'invented' a new ideology, Fascism, much \"\n",
      "                 'out of Nationalist\\xa0and Conservative views.\\n'\n",
      "                 '\\n'\n",
      "                 'Rise to power and becoming dictator\\n'\n",
      "                 'In 1922, he took power by having a large group of men, '\n",
      "                 '\"Black Shirts,\" march on Rome and threaten to take over the '\n",
      "                 'government. King Vittorio Emanuele III gave in, allowed him '\n",
      "                 'to form a government, and made him prime minister. In the '\n",
      "                 'following five years, he gained power, and in 1927 created '\n",
      "                 'the OVRA, his personal secret police force. Using the agency '\n",
      "                 'to arrest, scare, or murder people against his regime, '\n",
      "                 'Mussolini was dictator\\xa0of Italy by the end of 1927. Only '\n",
      "                 'the King and his own Fascist party could challenge his '\n",
      "                 'power.',\n",
      " 'type': 'Document'}\n",
      "\n",
      "{'id': '6754-1',\n",
      " 'metadata': {'chunk': 1.0,\n",
      "              'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini',\n",
      "              'title': 'Benito Mussolini',\n",
      "              'wiki-id': '6754'},\n",
      " 'page_content': 'Fascism as practiced by Mussolini\\n'\n",
      "                 'Mussolini\\'s form of Fascism, \"Italian Fascism\"- unlike '\n",
      "                 'Nazism, the racist ideology that Adolf Hitler followed- was '\n",
      "                 \"different and less destructive than Hitler's. Although a \"\n",
      "                 'believer in the superiority of the Italian nation and '\n",
      "                 'national unity, Mussolini, unlike Hitler, is quoted \"Race? '\n",
      "                 'It is a feeling, not a reality. Nothing will ever make me '\n",
      "                 'believe that biologically pure races can be shown to exist '\n",
      "                 'today\".\\n'\n",
      "                 '\\n'\n",
      "                 'Mussolini wanted Italy to become a new Roman Empire. In '\n",
      "                 '1923, he attacked the island of Corfu, and in 1924, he '\n",
      "                 'occupied the city state of Fiume. In 1935, he attacked the '\n",
      "                 'African country Abyssinia (now called Ethiopia). His forces '\n",
      "                 'occupied it in 1936. Italy was thrown out of the League of '\n",
      "                 'Nations because of this aggression. In 1939, he occupied the '\n",
      "                 'country Albania. In 1936, Mussolini signed an alliance with '\n",
      "                 'Adolf Hitler, the dictator of Germany.\\n'\n",
      "                 '\\n'\n",
      "                 'Fall from power and death\\n'\n",
      "                 'In 1940, he sent Italy into the Second World War on the side '\n",
      "                 'of the Axis countries. Mussolini attacked Greece, but he '\n",
      "                 'failed to conquer it. In 1943, the Allies landed in Southern '\n",
      "                 'Italy. The Fascist party and King Vittorio Emanuel III '\n",
      "                 'deposed Mussolini and put him in jail, but he was set free '\n",
      "                 'by the Germans, who made him ruler of the Italian Social '\n",
      "                 'Republic puppet state which was in a small part of Central '\n",
      "                 'Italy. When the war was almost over, Mussolini tried to '\n",
      "                 'escape to Switzerland with his mistress, Clara Petacci, but '\n",
      "                 \"they were both captured and shot by partisans. Mussolini's \"\n",
      "                 'dead body was hanged upside-down, together with his mistress '\n",
      "                 \"and some of Mussolini's helpers, on a pole at a gas station \"\n",
      "                 'in the village of Millan, which is near the border  between '\n",
      "                 'Italy and Switzerland.',\n",
      " 'type': 'Document'}\n",
      "\n",
      "{'id': '363-5',\n",
      " 'metadata': {'chunk': 5.0,\n",
      "              'source': 'https://simple.wikipedia.org/wiki/Italy',\n",
      "              'title': 'Italy',\n",
      "              'wiki-id': '363'},\n",
      " 'page_content': 'Veneto was made part of Italy in 1866 after a war with '\n",
      "                 'Austria. Italian soldiers won Latium in 1870. That was when '\n",
      "                 \"they took away the Pope's power. The Pope, who was angry, \"\n",
      "                 'said that he was a prisoner to keep Catholic people from '\n",
      "                 'being active in politics. That was the year of Italian '\n",
      "                 'unification.\\n'\n",
      "                 '\\n'\n",
      "                 'Italy participated in World War I. It was an ally of Great '\n",
      "                 'Britain, France, and Russia against the Central Powers. '\n",
      "                 \"Almost all of Italy's fighting was on the Eastern border, \"\n",
      "                 'near Austria. After the \"Caporetto defeat\", Italy thought '\n",
      "                 'they would lose the war. But, in 1918, the Central Powers '\n",
      "                 'surrendered. Italy gained the Trentino-South Tyrol, which '\n",
      "                 'once was owned by Austria.\\n'\n",
      "                 '\\n'\n",
      "                 'Fascist Italy \\n'\n",
      "                 'In 1922, a new Italian government started. It was ruled by '\n",
      "                 'Benito Mussolini, the leader of Fascism in Italy. He became '\n",
      "                 'head of government and dictator, calling himself \"Il Duce\" '\n",
      "                 '(which means \"leader\" in Italian). He became friends with '\n",
      "                 'German dictator Adolf Hitler. Germany, Japan, and Italy '\n",
      "                 'became the Axis Powers. In 1940, they entered World War II '\n",
      "                 'together against France, Great Britain, and later the Soviet '\n",
      "                 'Union. During the war, Italy controlled most of the '\n",
      "                 'Mediterranean Sea.',\n",
      " 'type': 'Document'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "documents = pinecone_vectorstore.similarity_search(\n",
    "    query=\"Who was Benito Mussolini?\",  # our search query\n",
    "    k=3  # return 3 most relevant docs\n",
    ")\n",
    "\n",
    "for doc in documents:\n",
    "    pprint(doc.__dict__)\n",
    "    print()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZCvtmREd0pdo"
   },
   "source": [
    "All of these are good, relevant results. But what can we do with this? There are many tasks, one of the most interesting (and well supported by LangChain) is called _\"Generative Question-Answering\"_ or GQA.\n",
    "\n",
    "## Generative Question-Answering\n",
    "\n",
    "In GQA we take the query as a question that is to be answered by a LLM, but the LLM must answer the question based on the information it is seeing being returned from the `vectorstore`.\n",
    "\n",
    "To do this we initialize a `RetrievalQA` object like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "moCvQR-p0Zsb"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "# Chat Completion LLM\n",
    "llm = ChatOpenAI(\n",
    "    openai_api_key=OPENAI_API_KEY,\n",
    "    model_name='gpt-4.5-preview',\n",
    "    temperature=0.0\n",
    ")\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=pinecone_vectorstore.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 106
    },
    "id": "KS9sa19K3LkQ",
    "outputId": "d7473cc3-6c7a-44c3-e219-12396eeb5580"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'Who was Benito Mussolini?',\n",
       " 'result': 'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and established a dictatorship in Italy, calling himself \"Il Duce\" (meaning \"leader\" in Italian). Initially a socialist, Mussolini was expelled from the socialist party due to his support for Italy\\'s involvement in World War I. He then created a new political ideology known as Fascism, which emphasized nationalism, authoritarianism, and strong centralized control.\\n\\nMussolini rose to power in 1922 after organizing the \"March on Rome,\" where his supporters, known as the \"Black Shirts,\" threatened to seize control of the government. King Vittorio Emanuele III appointed him Prime Minister, and by 1927 Mussolini had consolidated power, establishing a dictatorship enforced by his secret police, the OVRA.\\n\\nMussolini aimed to restore Italy to the glory of the ancient Roman Empire, leading aggressive military campaigns in places like Ethiopia (then Abyssinia), Albania, and Greece. He allied Italy with Nazi Germany and Japan during World War II, forming the Axis Powers. However, Italy suffered military defeats, and in 1943 Mussolini was removed from power and imprisoned. He was later rescued by German forces and installed as the leader of a puppet state in Northern Italy, the Italian Social Republic.\\n\\nIn April 1945, as World War II was ending, Mussolini attempted to flee to Switzerland but was captured and executed by Italian partisans along with his mistress, Clara Petacci. Their bodies were publicly displayed in Milan.'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.invoke(\"Who was Benito Mussolini?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "0qf5e3xf3ggq"
   },
   "source": [
    "We can also include the sources of information that the LLM is using to answer our question. We can do this using a slightly different version of `RetrievalQA` called `RetrievalQAWithSourcesChain`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "aYVMGDA13cTz"
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQAWithSourcesChain\n",
    "\n",
    "qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=pinecone_vectorstore.as_retriever(),\n",
    "    return_source_documents=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RXsVEh3S4ZJO",
    "outputId": "4150e3c6-3210-439b-9c85-389d8c20d02b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'Who was Benito Mussolini?',\n",
       " 'answer': 'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and established a dictatorship in Italy. Mussolini initially was a socialist but later created the ideology of Fascism, combining nationalist and conservative views. He rose to power through the March on Rome in 1922, eventually becoming dictator by 1927. Mussolini aimed to create a new Roman Empire, engaging in aggressive military actions such as invading Abyssinia (Ethiopia) and Albania. He allied with Adolf Hitler during World War II, forming part of the Axis Powers. Mussolini was deposed in 1943, briefly reinstated by the Germans as head of a puppet state, and was ultimately captured and executed by partisans in 1945.\\n\\n',\n",
       " 'sources': '',\n",
       " 'source_documents': [Document(id='6754-0', metadata={'chunk': 0.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, page_content='Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 – 28 April 1945) was an Italian politician and journalist. He was also the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.\\n\\nBiography\\n\\nEarly life\\nBenito Mussolini was named after Benito Juarez, a Mexican opponent of the political power of the Roman Catholic Church, by his anticlerical (a person who opposes the political interference of the Roman Catholic Church in secular affairs) father. Mussolini\\'s father was a blacksmith. Before being involved in politics, Mussolini was a newspaper editor (where he learned all his propaganda skills) and elementary school teacher.\\n\\nAt first, Mussolini was a socialist, but when he wanted Italy to join the First World War, he was thrown out of the socialist party. He \\'invented\\' a new ideology, Fascism, much out of Nationalist\\xa0and Conservative views.\\n\\nRise to power and becoming dictator\\nIn 1922, he took power by having a large group of men, \"Black Shirts,\" march on Rome and threaten to take over the government. King Vittorio Emanuele III gave in, allowed him to form a government, and made him prime minister. In the following five years, he gained power, and in 1927 created the OVRA, his personal secret police force. Using the agency to arrest, scare, or murder people against his regime, Mussolini was dictator\\xa0of Italy by the end of 1927. Only the King and his own Fascist party could challenge his power.'),\n",
       "  Document(id='6754-1', metadata={'chunk': 1.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, page_content='Fascism as practiced by Mussolini\\nMussolini\\'s form of Fascism, \"Italian Fascism\"- unlike Nazism, the racist ideology that Adolf Hitler followed- was different and less destructive than Hitler\\'s. Although a believer in the superiority of the Italian nation and national unity, Mussolini, unlike Hitler, is quoted \"Race? It is a feeling, not a reality. Nothing will ever make me believe that biologically pure races can be shown to exist today\".\\n\\nMussolini wanted Italy to become a new Roman Empire. In 1923, he attacked the island of Corfu, and in 1924, he occupied the city state of Fiume. In 1935, he attacked the African country Abyssinia (now called Ethiopia). His forces occupied it in 1936. Italy was thrown out of the League of Nations because of this aggression. In 1939, he occupied the country Albania. In 1936, Mussolini signed an alliance with Adolf Hitler, the dictator of Germany.\\n\\nFall from power and death\\nIn 1940, he sent Italy into the Second World War on the side of the Axis countries. Mussolini attacked Greece, but he failed to conquer it. In 1943, the Allies landed in Southern Italy. The Fascist party and King Vittorio Emanuel III deposed Mussolini and put him in jail, but he was set free by the Germans, who made him ruler of the Italian Social Republic puppet state which was in a small part of Central Italy. When the war was almost over, Mussolini tried to escape to Switzerland with his mistress, Clara Petacci, but they were both captured and shot by partisans. Mussolini\\'s dead body was hanged upside-down, together with his mistress and some of Mussolini\\'s helpers, on a pole at a gas station in the village of Millan, which is near the border  between Italy and Switzerland.'),\n",
       "  Document(id='363-5', metadata={'chunk': 5.0, 'source': 'https://simple.wikipedia.org/wiki/Italy', 'title': 'Italy', 'wiki-id': '363'}, page_content='Veneto was made part of Italy in 1866 after a war with Austria. Italian soldiers won Latium in 1870. That was when they took away the Pope\\'s power. The Pope, who was angry, said that he was a prisoner to keep Catholic people from being active in politics. That was the year of Italian unification.\\n\\nItaly participated in World War I. It was an ally of Great Britain, France, and Russia against the Central Powers. Almost all of Italy\\'s fighting was on the Eastern border, near Austria. After the \"Caporetto defeat\", Italy thought they would lose the war. But, in 1918, the Central Powers surrendered. Italy gained the Trentino-South Tyrol, which once was owned by Austria.\\n\\nFascist Italy \\nIn 1922, a new Italian government started. It was ruled by Benito Mussolini, the leader of Fascism in Italy. He became head of government and dictator, calling himself \"Il Duce\" (which means \"leader\" in Italian). He became friends with German dictator Adolf Hitler. Germany, Japan, and Italy became the Axis Powers. In 1940, they entered World War II together against France, Great Britain, and later the Soviet Union. During the war, Italy controlled most of the Mediterranean Sea.'),\n",
       "  Document(id='363-4', metadata={'chunk': 4.0, 'source': 'https://simple.wikipedia.org/wiki/Italy', 'title': 'Italy', 'wiki-id': '363'}, page_content='The head of government is Mario Draghi. He became Prime Minister on February 13, 2021. He succeeded Giuseppe Conte. Conte\\'s cabinet, fell after a political crisis caused by Italia Viva, a liberal political party.\\n\\nItaly was one of the first members of the European Union. In 2002 along with 11 other European countries, it changed to using the euro as its official currency. Before this, the Italian lira had been used since 1861.\\n\\nAnyone who wants to be President of Italy must have Italian citizenship, be at least 50 years old, and must be able to hold political and civil rights.\\n\\nHistory \\n\\nIn ancient times, the capital of Italy was Rome. Rome was founded in 753 BC. It  was a separate state well known as Roman Kingdom firstly, Roman Republic and Roman Empire later.\\n\\nBefore 1861, Italy was not a state.  The area included a group of separate states that were ruled by other countries (such as Austria, France, and Spain).  In the 1850s, the Earl of Camillo Benso, Count of Cavour was the head of government of the \"State of Sardinia\". He talked to the Austrians in Lombardy and Veneto and said they should create a Northern Italian state. This happened, but other Central and Southern Italian states also joined Piedmont to create a bigger state.\\n\\nKingdom of Italy \\n\\nIn 1860, Giuseppe Garibaldi took control of Sicily, creating the Kingdom of Italy in 1861. Victor Emmanuel II was made the king. In 1861, Latium and Veneto were still not part of Italy, because they were ruled by the Pope and Austrian Empire.')]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa_with_sources.invoke(\"Who was Benito Mussolini?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nMRk_P3Q7l5J"
   },
   "source": [
    "Now we answer the question being asked, *and* return the source of this information being used by the LLM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo cleanup\n",
    "\n",
    "Once you're done, you can free up resources by deleting your index. Deleting an index is a permanet operation and cannot be undone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vNkoIzQjjMTQ"
   },
   "outputs": [],
   "source": [
    "pc.delete_index(name=index_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ehJEn68qADoH"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0021290a0a3a44be911c4e94b7d25006": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "07c3ae73ab9b4c66813d1646ce6a320d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "21045e189c5d49a082964eec2c5d8f6a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2d038c3eb946427297de35460a8d4ae2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2ee36ac5673a4b74877dcf3fe63c5eb3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3216e01669cc470894d25198a2099263": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_07c3ae73ab9b4c66813d1646ce6a320d",
      "placeholder": "​",
      "style": "IPY_MODEL_6acfe82fd33043558667ac26141093f8",
      "value": " 30000/30000 [00:10&lt;00:00, 2929.37it/s]"
     }
    },
    "34b9b178e43a4cc1ba516c4f27ffd575": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6e1f19542e2645ccad3a8c99bf6e030b",
      "placeholder": "​",
      "style": "IPY_MODEL_7c834ff3591545f7809a46989c3a985b",
      "value": " 300/300 [00:00&lt;00:00, 2936.76it/s]"
     }
    },
    "47188360f07a4c018bf7d2c5eb6f626c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_50d346f4ff614f8cb7793cce0a9c0916",
       "IPY_MODEL_f06f0a851b6c434da5cd9ddb245de902",
       "IPY_MODEL_3216e01669cc470894d25198a2099263"
      ],
      "layout": "IPY_MODEL_47ecdc632edf4aeeb37239510aff8ced"
     }
    },
    "47ecdc632edf4aeeb37239510aff8ced": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4981784e865746a49a912188fb599985": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "50d346f4ff614f8cb7793cce0a9c0916": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2ee36ac5673a4b74877dcf3fe63c5eb3",
      "placeholder": "​",
      "style": "IPY_MODEL_f948308ac3414cfabbea8e2af917446f",
      "value": "sending upsert requests: 100%"
     }
    },
    "54d772726617494d98e5ee1618db5cf0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2d038c3eb946427297de35460a8d4ae2",
      "max": 300,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5f0fb5f8cfb7459d80f6a92bd0125dcb",
      "value": 300
     }
    },
    "5f0fb5f8cfb7459d80f6a92bd0125dcb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "62da3ff6ab334122aa0657a6f25417bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6acfe82fd33043558667ac26141093f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6b3b8cddec674c0bb00620e1c9128219": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7f7790181f72437cb10a88704d20f778",
      "placeholder": "​",
      "style": "IPY_MODEL_62da3ff6ab334122aa0657a6f25417bb",
      "value": "collecting async responses: 100%"
     }
    },
    "6e1f19542e2645ccad3a8c99bf6e030b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "75cef31e615d4851b029c3aacacf9c2d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6b3b8cddec674c0bb00620e1c9128219",
       "IPY_MODEL_54d772726617494d98e5ee1618db5cf0",
       "IPY_MODEL_34b9b178e43a4cc1ba516c4f27ffd575"
      ],
      "layout": "IPY_MODEL_0021290a0a3a44be911c4e94b7d25006"
     }
    },
    "7c834ff3591545f7809a46989c3a985b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7f7790181f72437cb10a88704d20f778": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f06f0a851b6c434da5cd9ddb245de902": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_21045e189c5d49a082964eec2c5d8f6a",
      "max": 30000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4981784e865746a49a912188fb599985",
      "value": 30000
     }
    },
    "f948308ac3414cfabbea8e2af917446f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}