nbs/diffedit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DiffEdit\n",
    "\n",
    "> A notebook for understanding and experimenting with the paper [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://arxiv.org/abs/2210.11427)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to read a paper\n",
    "\n",
    "Accoding to [Andrew Ng](https://www.youtube.com/watch?v=733m6qBH-jI&t=460s), You should not read a paper from start to finish, but do it in several passes, focusing on the parts that will give you an idea of what the paper is about.\n",
    "\n",
    "1. Title, Abstract, Figures\n",
    "2. Intro, Conclusion, Figures again, Skim the rest\n",
    "\n",
    "In this notebook, we will try to get an overview of the paper: [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://arxiv.org/abs/2210.11427)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  First pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Title: <u>DiffEdit: Diffusion-based <mark>semantic</mark> image editing with <mark>mask guidance</mark></u>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "So this paper is about something `semantic`, i.e. it is based on text. And then we edit images based on a `mask` to guide it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Semantic**\n",
    "\n",
    "> Relating to meaning in language or logic.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mask**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> An image that acts as a cookie cutter, with instructions on what to keep and what to remove."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![masking](images/masking_1.jpg)\n",
    "\n",
    "Source: [Shmu Perhiniak, yesimadesigner.com](https://yesimadesigner.com/our-ultimate-guide-to-masking-photoshop-cc-2019/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose <mark>DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query.</mark> Semantic image editing is an <u>extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image</u>. Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, <mark>our main contribution is able to automatically generate a mask</mark> highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. DiffEdit achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Figure 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure 1](images/diffedit1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In semantic image editing the goal is to <mark>modify an input image based on a textual query</mark>,\n",
    "while otherwise <mark>leaving the image as close as possible to the original</mark>. In our DIFFEDIT approach, <u>a\n",
    "mask generation module determines which part of the image should be edited</u>, and an encoder infers\n",
    "the latents, to provide inputs to a text-conditional diffusion model which produces the image edit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Second Pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task of <mark>semantic image editing consists in modifying an input image in accordance with a\n",
    "textual transformation query</mark>. For instance, given an image of a bowl of fruits and the query “fruits”\n",
    "→ “pears”, <mark>the aim is to produce a novel image where the fruits have been changed into pears</mark>,\n",
    "<u>while keeping the bowl and the background as similar as possible to the input image.</u>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text query\n",
    "can also be a more elaborate description like “A basket of fruits”. See the example edits obtained\n",
    "with DIFFEDIT in Figure 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Semantic image editing bears strong similarities with image generation and can be viewed as extending text-conditional image generation with an additional constraint: <mark>the\n",
    "generated image should be as close as possible to a given input image.</mark>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem - Existing approaces makes you loose data\n",
    "(I added this subheading, also you can skip this)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text-conditional image generation is currently undergoing a revolution, with DALL-E (Ramesh\n",
    "et al., 2021), Cogview (Ding et al., 2021), Make-a-scene (Gafni et al., 2022), Latent Diffusion Models (Rombach et al., 2022), DALL-E 2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022b),vastly improving state of the art in modelling wide distributions of images and allowing for unprecedented compositionality of concepts in image generation. Scaling these models is a key to\n",
    "their success. State-of-the art models are now trained on vast amounts of data, which requires large\n",
    "computational resources. Similarly to language models pretrained on web-scale data and adapted\n",
    "in downstreams tasks with prompt engineering, the generative power of these big generative models can be harnessed to solve semantic image editing, avoiding to train specialized architectures (Li\n",
    "et al., 2020a; Wang et al., 2022a), or to use costly instance-based optimization (Crowson et al., 2022;\n",
    "Couairon et al., 2022; Patashnik et al., 2021).\n",
    "Diffusion models are an especially interesting class of model for image editing because of their iterative denoising process starting from random Gaussian noise. This process can be guided through\n",
    "a variety of techniques, like CLIP guidance (Nichol et al., 2021; Avrahami et al., 2022; Crowson,\n",
    "2021), and inpainting by copy-pasting pixel values outside a user-given mask (Lugmayr et al., 2022). <mark>These previous works, however, lack two crucial properties for semantic image editing:</mark> (i) **inpainting discards information** about the input image that should be used in image editing (e.g. changing\n",
    "a dog into a cat should not modify the animal’s color and pose); (ii) **a mask must be provided** as\n",
    "input to <u>tell the diffusion model what parts of the image should be edited</u>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We believe that while\n",
    "drawing masks is common on image editing tools like Photoshop, language-guided editing offers a\n",
    "more intuitive interface to modify images that requires less effort from users.\n",
    "Conditioning a diffusion model on an input image <em>**can also be done without a mask**</em>, e.g. by considering the **distance to input image as a loss function** (Crowson, 2021; Choi et al., 2021), or by **using\n",
    "a noised version of the input image as a starting point** for the denoising process as in SDEdit (Meng\n",
    "et al., 2021). <mark>However, these editing methods tend to modify the entire image, whereas we aim for\n",
    "localized edits.</mark> Furthermore, **adding noise** to the input image **discards important information**, both\n",
    "inside the region that should be edited and outside."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution - Automatically paint what region needs to change and only change that\n",
    "(I added this subheading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To leverage the best of both worlds, we propose <mark>DIFFEDIT, an algorithm that automatically finds\n",
    "what regions of an input image should be edited given a text query.</mark> \n",
    "\n",
    "> **By <u>contrasting</u> the <em>predictions</em>\n",
    "of a <u>conditional</u> and <u>unconditional</u> diffusion model**, we are able to <mark>locate where editing is needed</mark>\n",
    "to match the text query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also show how using a <mark>reference text describing the input image</mark> **and similar to the query**, can help obtain **better masks**. Moreover, we demonstrate that using a <mark><u>reverse\n",
    "denoising model</u>, to <u>encode</u> the input image in latent space,</mark> rather than simply adding noise to it,\n",
    "allows to <mark>better integrate the edited region into the background</mark> and produces more subtle and natural\n",
    "edits. See Figure 1 for illustrations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We quantitatively evaluate our approach and compare to prior\n",
    "work using images of the ImageNet and COCO dataset, as well as a set of generated images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We introduced DIFFEDIT, a novel algorithm for semantic image editing based on diffusion models.\n",
    "<mark>Given a textual query, using the diffusion model, DIFFEDIT infers the relevant regions to be edited\n",
    "rather than requiring a user generated mask.</mark> Furthermore, in contrast to other diffusion-based methods, we initialize the generation process with a <mark>DDIM encoding of the input</mark>.We provide theoretical\n",
    "analysis that motivates this choice, and show experimentally that <mark>this approach conserves more appearance information from the input image, leading to lighter edits.</mark> Quantitative and qualitative\n",
    "evaluations on ImageNet, COCO, and images generated by Imagen, show that our approach leads\n",
    "excellent edits, improving over previous approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Figures againg\n",
    "### Figure 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure 2](images/diffedit2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 2: The three steps of DIFFEDIT. \n",
    "\n",
    "Step 1: we add noise to the input image, and denoise it:\n",
    "once conditioned on the query text, and once conditioned on a reference text (or unconditionally).\n",
    "<mark>We derive a mask based on the difference in the denoising results</mark>. \n",
    "\n",
    "Step 2: we <mark>encode</mark> the input\n",
    "image with <mark>DDIM</mark>, <u>**to estimate the latents** corresponding to the input image</u>. \n",
    "\n",
    "Step 3: we perform\n",
    "<mark>DDIM decoding</mark> <u>conditioned on the text query</u>, **using the inferred mask to replace the background**\n",
    "with pixel values coming from the encoding process at the corresponding timestep."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Figure 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure 5](images/diffedit3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 5: Edits obtained on ImageNet with DIFFEDIT and ablated models. \n",
    "\n",
    "Encode-Decode is\n",
    "DIFFEDIT without masking, and \n",
    "\n",
    "SDEdit is obtained when not using masking nor encoding. \n",
    "\n",
    "When\n",
    "not using masking (SDEdit and Encode-Decode) we observe undesired edits to the background,\n",
    "see e.g. the sky in the second column. When not using DDIM encoding (SDEdit and DiffEdit w/o\n",
    "Encode), appearance information from the input —such as pose— is lost, see last two columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skim the rest\n",
    "[Here you go](https://arxiv.org/pdf/2210.11427.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 SEMANTIC IMAGE EDITING WITH DIFFEDIT\n",
    "In many cases, semantic image edits can be restricted to only a part of the image, leaving other parts\n",
    "unchanged. However, the input text query does not explicitly identify this region, and a naive method\n",
    "could allow for edits all over the image, risking to modify the input in areas where it is not needed.\n",
    "To circumvent this, <mark>we propose DIFFEDIT, a method to leverage a text-conditioned diffusion model\n",
    "to infer a mask of the region that needs to be edited.</mark> Starting from a **DDIM encoding of the input\n",
    "image**, DIFFEDIT uses the **inferred mask** to guide the **denoising** process, minimizing edits outside\n",
    "the region of interest. Figure 2 illustrates the three steps of our approach, which we detail below.\n",
    "\n",
    "## Step 1: Computing editing mask.\n",
    "When the denoising an image, a text-conditioned diffusion\n",
    "model will yield different noise estimates given different text conditionings. <mark>We can consider where\n",
    "the estimates are different, which gives information about what image regions are concerned by\n",
    "the change in conditioning text.</mark> For instance, in Figure 2, the noise estimates conditioned to the\n",
    "query <mark>zebra and reference text horse\n",
    "are different on the body of the animal</mark>, where they will tend\n",
    "to decode different colors and textures depending on the conditioning. For the background, on the\n",
    "other hand, there is little change in the noise estimates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference between the noise estimates\n",
    "can thus be used to infer a mask that identifies what parts on the image need to be changed to\n",
    "match the query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In our algorithm, we use a <mark>Gaussian noise with strength 50%</mark> (see analysis in\n",
    "Appendix A.1), <mark>remove extreme values in noise predictions</mark> and stabilize the effect by <mark>averaging\n",
    "spatial differences over a set of n input noises</mark>, with `n= 10` in our default configuration. The result is\n",
    "then <mark>rescaled to the `range [0, 1]`</mark>, and <mark>binarized with a threshold</mark>, which we set to `0.5` by default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The\n",
    "masks generally somewhat overshoot the region that requires editing, this is beneficial as it allows it\n",
    "to be smoothly embedded in it’s context, see examples in Section 4 and Section A.5.\n",
    "\n",
    "## Step 2: Encoding.\n",
    "We encode the input image x0 in the implicit latent space at timestep r with\n",
    "the DDIM encoding function Er. This is done with the unconditional model, i.e. using conditioning\n",
    "text ∅, so no text input is used for this step.\n",
    "1We can also use an empty reference text, which we denote as Q = ∅.\n",
    "4\n",
    "Preprint. Under review.\n",
    "\n",
    "## Step 3: Decoding with mask guidance.\n",
    "After obtaining the latent xr, we decode it with our diffusion model conditioned on the editing text query Q, e.g. zebra in the example of Figure 2. We use our\n",
    "mask M to guide this diffusion process. Outside the mask M, the edited image should in principle\n",
    "be the same as the input image. We guide the diffusion model by replacing pixel values outside the\n",
    "mask with the latents xt inferred with DDIM encoding, which will naturally map back to the original\n",
    "pixels through decoding, unlike when using a noised version of x0 as typically done (Meng et al.,\n",
    "2021; Song et al., 2021). The mask-guided DDIM update can be written as ˜yt = Myt + (1−M)xt,\n",
    "where yt is computed from yt−dt with Eq. 2, and xt is the corresponding DDIM encoded latent.\n",
    "The encoding ratio r determines the strength of the edit: larger values of r allow for stronger edits\n",
    "that allow to better match the text query, at the cost of more deviation from the input image which\n",
    "might not be needed. We evaluate the impact of this parameter in our experiments. We illustrate the\n",
    "effect of the encoding ratio in Appendix A.5"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}