Painting with … Deep Learning

The catch in the title is that this is about painting (brush strokes, pressure, and whatnot), and not image creation (generating pixels on the screen), i.e., more about the painting process that resulted in the image. (•)
To put this differently, how do we teach robots to draw? I mean, yeah, we could have them become a high-speed Seurat, but that’s faking it, at least in my viewpoint 

The folks at DeepMind have worked on exactly this, and have a pretty good start. They used Reinforcement Learning on an agent to paint using MyPaint. It starts off just slapping stuff on, with the reinforcement part being used to tell it whether what it’s painting is “good” or “bad”. The results below really aren’t all that bad!

Technically speaking, this doesn’t really work like a Generative Adversarial Networks (GAN), since the agent isn’t being trained on the pixels, but on how it’s painting, e.g., if the brush stroke is too hard, the stroke is too wide, etc. It’s a pretty amazing leap forward — as the authors put it:

Recovering structured representations from raw sensations is an ability that humans readily possess and frequently use. In this work we show it is possible to guide artificial agents to produce similar representations by giving them access to the same tools that we use to recreate the world around us. In doing so they learn to produce visual programs that succinctly express the causal relationships that give rise to their observations.

Good stuff!

(•) Creating the image itself, with GANs, is pretty far along already. Check out this paper by Radford et al., and this excellent writeup by Aylien.