Colorizing with Examples and Deep Learning
 |
| /via http://bruxy.regnet.cz/web/gfx/EN/colorize/ |
Colorization — adding colors to gray-scale images — is deceptively tricky. The underlying issue is that the resultant image needs to be perceptually meaningful, and visually appealing. Which is all well and good, but, well, meaningful and visually appealing to whom? In the image above,
• Is the fence painted? Or, if not, what shades of brown should the fence be?
• That grass— should it be green, blue-green, straw yellow, or brown?
• And the trees? And the leaves?
You get it — the point here being that there really is no uniquely correct solution, and so much of what happens is about the intent of the person doing the colorizing
• Is the fence painted? Or, if not, what shades of brown should the fence be?
• That grass— should it be green, blue-green, straw yellow, or brown?
• And the trees? And the leaves?
You get it — the point here being that there really is no uniquely correct solution, and so much of what happens is about the intent of the person doing the colorizing
Historically, this was all deeply manual, with the work consisting of literallypainting the image. Software did help, but there was still a lot of manual work involve — for example, humans would draw a boundary and tell the computer to fill in that boundary with a given color with shading, fades, and whatnot. However, the point still remains that it was, and is, a huge PITA.
Deep learning does help quite a bit here though, especially in the detection of boundaries within which colors should be applied. The usual DL technique is something like “Make the target image look like this reference image”. The problem, as you might imagine, is that the result depends on the reference image — it’s lighting, viewpoint, and content dramatically affecting the final result.
What’s worse, most of the current techniques are fixed in their abilities. The colorization they perform is entirely predicated upon the reference-images that they have learned from, and any kind of user-controllability is hard, if not impossible.
What’s even worser is that they can only colorize things they have seen — resulting in the reference image database needing to contain all possible objects that might need to be colorized. Ouch!
 |
| /via https://arxiv.org/pdf/1705.02999.pdf |
There are some “Human in the Loop” approaches out there that are effective, with Zhang et al. (•) being a prominent one. Their approach — an interesting mix of old school manual and DL — consists of adding color “hints” to the image to help guide the colorization process. It’s really pretty cool, with the caveat that it still requires manual effort across the entire image.
The example above shows how the hints can alter the final output — as you can imagine, the “Human in the Loop” can have a pronounced effect on the result!
The example above shows how the hints can alter the final output — as you can imagine, the “Human in the Loop” can have a pronounced effect on the result!
A new paper by He et al. (••) is a simplified “Human in the Loop” version, which throws a hefty amount of UX into the mix. It allows users to choose different colorization patterns by simply choosing different reference images. Oh, the underlying system is still trained on a huge set of reference images, but, and these are key
• The reference images don’t need to be directly related to the target image being colorized, and
• the reference image the user selects determines how the target image gets colorized
• The reference images don’t need to be directly related to the target image being colorized, and
• the reference image the user selects determines how the target image gets colorized
 |
| /via https://arxiv.org/pdf/1807.06587v2.pdf |
To put it differently, it works, kinda, as shown above. Given a grey-scale image (the one in the middle), the system tries to identify items in the image that it might have seen before, e.g., the Golden Gate bridge. It then uses the color information from the reference images it has of the Golden Gate bridge to colorize the entire picture.
OK, so maybe there is actually a bit more going on under the hood (this isDeep Learning, after all ). The Convolutional Neural Network (CNN) that the authors designed actually works in two steps. In the first step, it figures out how “accurate” the reference image is compared to the target image. In the next step, it actually works with two loss functions, one for Chrominance (to make sure that the colors are reasonably correct) and the other Perceptual(to figure out how to color things it doesn’t have in the reference image)
In short, it ends up optimizing the colorization scheme for color accuracy, across things it sees and doesn’t see in the reference image.
 |
| /via https://arxiv.org/pdf/1807.06587v2.pdf |
The results are quite remarkable, as you can see from the example above (original at the top left, reference at the bottom left). Incidentally, do note the different skin tones in the reference and target images don’t seem to cause problems in the result, a healthy step forward from some of the classification mis-steps of the recent past!
There is actually quite a bit more in this paper. They also have an algorithm to recommend reference images (“Here is somebody else posing in front of the Golden Gate!”), as well as the ability to colorize images that don’t occur in the real world (anime!).
For my money though, the coolest part is that this also works with colorizing video by mapping the above process to key frames. The example below shows this applied to Lucasta, with the reference image on the bottom left.
 |
| /via https://arxiv.org/pdf/1807.06587v2.pdf |
(•) “Real-Time User-Guided Image Colorization with Learned Deep Priors” — by Zhang et al.
(••) “Deep Exemplar-based Colorization” by He et al.
(••) “Deep Exemplar-based Colorization” by He et al.
Comments