Deep Learning and Interpretability

One of the confounding things about Deep Learning is that we really don’t grok why it works. Oh, fine, we do, kinda, understand the “how” — Stochastic Gradient Descent, TensorFlow, NVIDIA, whatever — but “why it does what it does” is, well, beyond us.

When you look at a trained neural network, our human tendency to categorize things tends to come to the fore. We look at the model, with all it’s weights, and we look for some sort of order, a pattern that could explain things, and, inevitably, we find what we’re looking for.

Or, well, we kinda find it. In many many cases, we spot neurons that seem to have very specific functions. For example, there is the infamous “Jennifer Aniston neuron” that fires whenever she shows up on the screen. On a more relevant note, we can identify clusters of neurons that are associated with very specific tasks (identifying cats, for example).

The vast majority of the neurons, however, don’t actually seem to do anything that we can identify. They’re all just there as part of a confusing morass of neurons that are clearly necessary, but for what purpose?

/via https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/

The folks at DeepMind set out to study this, using tried-and-proven software-engineering techniques (aka: “Start deleting stuff, and see what breaks”). In this case, they started deleting individual neurons to see what impact if would have on performance.

Surprisingly, they found that “networks which generalise well were much more robust to deletions than networks which simply memorised images that were previously seen during training. In other words, networks which generalise better are harder to break”

The thing about the above is that “generalise” is a different way of saying “one of those confusing morass of neurons”. Or, to put all this in other words, “The more generalizable a network, the less interpretable it is”, a quite humbling result, really.

Coda: There may be parallels to our DNA here, where we’ve been discovering that all that “junk DNA” isn’t actually junk after all…