Image Compression, Green-Screens, and Deep Learning

-
Image compression can get awfully tricky once you get down to really low bit-rates — think 0.1 bpp (bits per pixel) or less. The reigning champ is currently BPG (•), and even it breaks down at really low bpp.
This is where Deep Learning comes in, with a bunch of work done on doing adaptive compression by generating textural detail, suppressing artifacts, amongst others. These all work really quite well — 1.7x to 2.5x better than BPG/WebP/JPEG — so well in fact that Machine Learning seems to be the new , and hot!, frontier in image compression these days.
The issue though is that most (all?) of them tend to focus on fixing the artifacts that emerge from compression in one form or the other (not that that is a bad thing mind you!).
In a new paper, Augustsson et al. (••) have an intriguing new approach using Generative Adversarial Networks (GANs), where they focus on preserving the important parts of the image, and generating the “unimportant” parts, such tree-leaves, cloud-detail, and the like. They even go so far as to completely generating parts of the image from scratch, in effect green-screening it!
To be a bit more precise, they do one of two things
  • • Global Generative Compression (GC)Structure at different scales is generated programmatically (think tree-leaves, concrete patterns, building windows, etc). The key here is that if the synthesized background looks good, thats just about as good as having the actual background back there. In fact, it might be better, as it keeps our focus on the important stuff!
  • • Selective Generative Compression (SC)
    This is the full green-screen effect, where everything that isn’t “important” is generated from a semantic map. Think of the compressed image having labels like “Tree”, “Cloud”, etc. which get filled in as templates. Again, this beats the pants off of blurry blobs of green or white…
The results are pretty amazing — check them out here — and definitely more pleasing to the eye. From the paper — “users prefer the images produced by our method over BPG even when BPG uses more than double the bits”.
The key here is clearly in being able to identify what’s important and what’s important and what’s not. So, if the background is supposed to contain evidence of a crime, this may not be the best method, but then again, this is also we humans are not particularly good at identifying background stuff — this is how our eyes/brain works too!
(•) Check out this comparator for BPG vs JPB andWebP, it’s really quite nifty!
(••) Generative Adversarial Networks for Extreme Learned Image Compression by Augustsson et al.

Comments

Post a Comment

Popular posts from this blog

Erlang, Binaries, and Garbage Collection (Sigh)

-
Image
The nice thing about garbage-collection is that it is makes memory management absolutely transparent to you Of course, the above is true right up to the point when it stops being transparent, at which point it does a phase change from nice to clusterf**k of biblical proportions .  There are entire posts, books, and suicide notes that have been written about this, and I have no great desire to add to them, save with the minor comment that paranoia about garbage-collection is still justified , albeit only in the edge-cases for most people. That said, I'll give you that Erlang GC is in many ways much, much better positioned in this regard than, oh, Java, but it still rears up and bites you in the butt when you least expect it. ( More information about Erlang's GC vis-a-vis Java here , and Jesper's truly excellent description of Erlang's GC here ).  A story from the past should serve to illustrate this. Once upon a time, in a magical land far far away - well, Her...
Read more

Spot The Difference — Deep Learning Edition

-
Image
You’ve played  Spot The Difference  at some point, right? (Of course you did. We  all  did  ). Think about what you actually did when you were trying to Spot The Difference — in fact, just do it again with SpongeBob juggling above. You’re probably doing something like 1. Quick glance to the left. 2. Quick glance to the right 3. “ Spatula on the left. Wonder if there is one on the right? ” 4. “ Yup, there is. Sriracha on the right. Is that on the left? ” 5. “ Nope! It’s a ketchup bottle on the left! And there is nothing on the plate below it! ” etc. To summarize, what you’re basically doing is looking at each image alternately, identifying objects in each image, and trying to find the same object in the other image. As it turns out, what most Deep Learning systems do is something quite different. They basically do something like this 1. Figure out everything in the first image (“ ketchup bottle at 1 o-clock, empty plate at 3 o-clock, spatula at 1...
Read more