GANs Applications in CV

3.5 图像上色 Colourful Image Colorization

在计算机图形学领域中，一直存在两种为图片上色的方向：数据驱动的自动上色和用户交互的引导上色。

• Automatic Image Colorization
• User Guided Image Colorization
  • Based on color strokes
  • Based on reference color image
  • Based on color palette
  • Based on language(text)
• Video Colorization
  • Automatically
  • Based on reference

1. Automatic Image Colorization

Paper	Source	Code/Project Link
Learning Large-Scale Automatic Image Colorization	ICCV 2015	[project] [code]
Deep Colorization	ICCV 2015
Learning Representations for Automatic Colorization	ECCV 2016	[project] [code]
Colorful Image Colorization	ECCV 2016	[project] [code]
Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification	SIGGRAPH 2016	[project] [code]
Unsupervised Diverse Colorization via Generative Adversarial Networks	ECML-PKDD 2017	[code]
Learning Diverse Image Colorization	CVPR 2017	[code]
Structural Consistency and Controllability for Diverse Colorization	ECCV 2018
Pixelated Semantic Colorization	1901.10889
Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks	CVPR 2019

Colourful Image Colourization

2016 ECCV 里加州大学伯克利分校的一篇文章介绍的方法。这个方法与之前方法的不同之处在于，它把照片上色看成是一个分类问题——预测三百多种颜色在图片每一个像素点上的概率分布。这种方法tackle了这个任务本身的不确定性，例如，当你看到一个黑白的苹果时，你可能会觉得它是红色的，但如果这个苹果是青色的，其实也并没有多少违和感。大家也可以到作者的网站网站来试用他们的demo。 https://richzhang.github.io/colorization/

Colornet

Github 地址：https://github.com/pavelgonchar/colornet

2. User Guided Image Colorization

这种方法是由 Levin 等人在 2004 年开创的，用户通过彩色画笔在灰度图像中进行引导性上色，随后优化算法会生成符合用户逻辑的上色结果。这种方法可以保留人工上色的部分性质，因而经常会有绝佳的表现，但往往需要密集的用户交互次数（有时超过五十次）。随着技术进步，现在的交互次数慢慢减少。

2.1 Based on color strokes

Image Type	Paper	Source	Code/Project Link
Manga	Manga colorization	SIGGRAPH 2006
Line art / Sketch	Outline Colorization through Tandem Adversarial Networks	1704.08834	[Demo] [code]
Line art / Sketch	Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks	1705.01908	[code]
Natural Gray-Scale	Real-Time User-Guided Image Colorization with Learned Deep Priors	SIGGRAPH 2017	[project] [code1] [code2]
Sketch	Scribbler: Controlling Deep Image Synthesis with Sketch and Color	CVPR 2017
Natural Gray-Scale	Interactive Deep Colorization Using Simultaneous Global and Local Inputs (also palette based)	ICASSP 2019

Real-Time User-Guided Image Colorization with Learned Deep Priors

UC Berkeley SIGGRAPH 2017 论文链接：https://arxiv.org/abs/1705.02999 Demo 和代码链接：https://richzhang.github.io/ideepcolor/

2.2 Based on reference color image

Image Type	Paper	Source	Code/Project Link
Line art	Style2paints V1 : Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN	ACPR 2017	[Code]
Manga	Comicolorization: Semi-Automatic Manga Colorization (also palette based)	SIGGRAPH Asia 2017	[code]
Sketch	TextureGAN: Controlling Deep Image Synthesis with Texture Patches	CVPR 2018	[code]
Natural Gray-Scale	Deep Exemplar-based Colorization	SIGGRAPH 2018	[code]
Natural Gray-Scale	Example-Based Colourization Via Dense Encoding Pyramids (also palette based)	Pacific Graphics 2018	[code]
Natural Gray-Scale	A Superpixel-based Variational Model for Image Colorization	TVCG 2019
Natural Gray-Scale	Automatic Example-based Image Colourisation using Location-Aware Cross-Scale Matching	TIP 2019

2.3 Based on color palette

Image Type	Paper	Source	Code/Project Link
Natural Image	Palette-based Photo Recoloring	SIGGRAPH 2015	[project]
Manga	Comicolorization: Semi-Automatic Manga Colorization (also reference based)	SIGGRAPH Asia 2017	[code]
Natural Gray-Scale	Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation (also text based)	ECCV 2018	[code]
Natural Gray-Scale	Example-Based Colourization Via Dense Encoding Pyramids (also reference based)	Pacific Graphics 2018	[code]
Natural Gray-Scale	Interactive Deep Colorization Using Simultaneous Global and Local Inputs (also strokes based)	ICASSP 2019

2.4 Based on language or text

Image Type	Paper	Source	Code/Project Link
Natural Gray-Scale / Sketch	Language-Based Image Editing with Recurrent Attentive Models	CVPR 2018	[code]
Natural Gray-Scale	Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation (also palette based)	ECCV 2018	[code]
Scene Sketch	LUCSS: Language-based User-customized Colorization of Scene Sketches	1808.10544	[code]

3. Video Colorization

3.1 Automatically

Paper	Source	Code/Project Link
Fully Automatic Video Colorization with Self-Regularization and Diversity	CVPR 2019

3.2 Based on reference

Paper	Source	Code/Project Link
Switchable Temporal Propagation Network	ECCV 2018
Tracking Emerges by Colorizing Videos	ECCV 2018	[code]
Deep Exemplar-based Video Colorization	CVPR 2019

DeOldify: Colorizing and Restoring Old Images and Videos with Deep Learning

老照片上色, 人脸处理，港星老照片：“你我当年”