project/misc_sampling_in_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "472b5244-ccc2-4ccd-ac90-ceb3e30fea92",
   "metadata": {},
   "source": [
    "## Sampling with weights in Pandas\n",
    "\n",
    "- sampling core utilities is based on numpy (see docstring)\n",
    "- [file](https://github.com/pandas-dev/pandas/blob/49d371364b734b47c85733aac74b03ac4400c629/pandas/core/sample.py) containing sampling functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5fbfc74-be62-4737-94c9-ae1ca70c52f6",
   "metadata": {},
   "source": [
    "## Some random data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bf16f56-2573-4154-a03e-be5895e6fd73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pimmslearn.utils import create_random_df\n",
    "X = create_random_df(100, 15, prop_na=0.1).stack().to_frame(\n",
    "    'intensity').reset_index()\n",
    "\n",
    "freq = X.peptide.value_counts().sort_index()\n",
    "freq.name = 'freq'\n",
    "\n",
    "X = X.set_index(keys=list(X.columns[0:2]))  # to_list as an alternative\n",
    "freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b780c2f0-27cf-4ef0-b7e3-7b37a9c6e962",
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75579e93-6867-45f0-be59-3dec87a882ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Based on total number of rows, 95% is roughly: {int(len(X) * 0.95)}\")\n",
    "print(\"Based on each sample's 95% obs, it is roughly: {}\".format(\n",
    "    X.groupby('Sample ID').apply(lambda df: int(len(df) * 0.95)).sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1827dbed-6978-4a41-84ef-c50ad820d523",
   "metadata": {},
   "source": [
    "## Samling using a column with the weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb600d35-c7c4-47a1-93e6-c47ca1df2aee",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.join(freq, on='peptide')\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cc9cced-609f-499b-99b7-4ca77c399c34",
   "metadata": {},
   "outputs": [],
   "source": [
    "t = X.groupby('Sample ID').get_group('sample_003')\n",
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8b56fa0-e2a0-4e36-b58b-adb10cc64e21",
   "metadata": {},
   "outputs": [],
   "source": [
    "t.sample(frac=0.75, weights='freq')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9511857f-68cb-4649-8db6-eb55b09605fb",
   "metadata": {},
   "source": [
    "Sampling the entire DataFrame based on the freq will normalize on N of all rows. The normalization leaves relative frequency the same (if no floating point unprecision is reached)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "469fe379-4182-4dd0-93d9-44cbb452eda7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of rows not the same as when using groupby (see above)\n",
    "X.sample(frac=0.95, weights='freq')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2521a3d-4f12-449b-9fbc-29d896c8743c",
   "metadata": {},
   "source": [
    "### Sampling fails with groupby, reindexing needed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "624276ec-687b-468f-832e-59289df76e6f",
   "metadata": {},
   "source": [
    "The above is not mapped one to one to the groupby sample method. One needs to apply it to every single df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9124121-3306-4930-bcc2-2c28fdd2d594",
   "metadata": {},
   "outputs": [],
   "source": [
    "# X.groupby('Sample ID').sample(frac=0.95, weights='freq') # does not work\n",
    "X.groupby('Sample ID').apply(\n",
    "    lambda df: df.reset_index(0, drop=True).sample(frac=0.95, weights='freq')\n",
    ").drop('freq', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25f7869d-f327-4c22-bf6b-9768cdfb7577",
   "metadata": {},
   "source": [
    "And passing a Series need the original X to be indexed the same (multi-indices are not supported)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8028508-e1a7-499f-a16b-e4b0bb0453f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i, t in X.groupby('Sample ID'):\n",
    "#     t = t.sample(frac=0.75, weights=freq)\n",
    "# t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "456d93ed-b9fe-4407-a293-1f8864900c03",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.reset_index('Sample ID')\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81a61f77-9c74-4a3d-9605-b2ec01ca2ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.groupby(by='Sample ID').sample(frac=0.95, weights=freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "874a8034-e621-428c-90b7-f4e3d08cacbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.groupby(by='Sample ID').get_group('sample_002')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8407ba71-2a4a-4dd2-b0cf-999bd52d53ef",
   "metadata": {},
   "source": [
    "## Sanity check: Downsampling the first feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bc83363-e9c0-4c59-b2cd-0948da3659ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "freq.loc['feat_00'] = 1  # none should be selected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efd28c48-46f3-4037-9058-4babd3f43e1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = freq / freq.sum()\n",
    "freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4eefda7-5c7e-4f83-a7d8-7d8dd1cdaad4",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.groupby(by='Sample ID').sample(\n",
    "    frac=0.5, weights=freq).sort_index().reset_index().peptide.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3623c8c4-dc79-48fe-ae1e-287ea94619d4",
   "metadata": {},
   "source": [
    "## Using a series\n",
    "\n",
    "- in the above approach, sampling weights might be readjusted based on the values present in `sample` as `NAN`s lead to the weights not summing up. Alteratively one could loop through the wide format rows and sample values from these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bdb02a1-de5b-4027-8121-2c3fcbf3e284",
   "metadata": {},
   "outputs": [],
   "source": [
    "freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6baa53bb-c1de-4b88-8cf9-87fded9521cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.drop('freq', axis=1).set_index(\n",
    "    'Sample ID', append=True).squeeze().unstack(0)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8810da03-1c34-413c-b5d1-04e5a5108367",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.iloc[0].sample(frac=0.8, weights=freq).sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3223d401-62a5-4b84-a575-652c1b427cf3",
   "metadata": {},
   "source": [
    "Sampling using the wide format would garuantee that the weights are not adjusted based on missing values, but that instead missing values are sample into on or the other set. Ultimately `NaN`s are dropped also in this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cad08c07-1804-4dd5-aeef-2d47a9af43d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = {}\n",
    "for row_key in X.index:\n",
    "    data[row_key] = X.loc[row_key].sample(frac=0.8, weights=freq)\n",
    "pd.DataFrame(data).stack()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)]"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}