Reversing Style Transfer — via Deep Learning

Style Transfer using #DeepLearning has been getting easier — it’s gone from “minutes per image” to “almost real time” in less than a year, and will probably be part of your camera app Any Day Now.
The reverse however, where you convert the stylized image back to the original, is something that not too many people are looking at (Anyone? Bueller? Bueller?).

Enter a new paper by Shiri et al., where they use an FDNN (Face Destylization Neural Network) to restore a photo-realistic face from a stylized one, as shown below

What the FDNN does is, basically
a) Identify specific facial components from the stylized images
b) Map these components to components in real/actual faces
c) Mush the result into a “real” face by smoothing out transitions, fitting, fixing, etc. (e.g. “No, the cheeks can’t be 70% of the face. Not even BadPlasticSurgery will do that”)

When it all comes together, it works remarkably well , but there is still a ways to go. For example, it currently only works with “full frontal” faces, the faces need to be aligned with templates, etc — nothing that can’t be fixed fairly easily however.

So why do this? Apart from the usual “because we can” bit? Well, reversing your awesome Samsung AR emoji might be fun, but a much more beneficial use-case is in significantly simplifying photo-realistic photo editing — what takes you months of learning and a Photoshop license will probably move into your Camera app!

And then there is the most probably use-case — that it’ll probably make identifying people from images much much simpler. Remember McPherson et al.’s paper from 2016, where they showed that you can uniquely identify mosaiced/blurred/anonymized people on TV using #DeepLearning? Their system did this by matching against blurred images, but it’s just a step from there to reconstructing the original faces using this new work by Shiri et al.

So much for privacy 🤷