Calibration, Trust, and Deep Learning

You know how you really need to calibrate your drone before you start flying it? And if you don’t do that whole “spinny” thing with it, it really acts goofy? Or, sometimes Google Maps gets horked, and the blue dot shows you in the wrong place? And to fix it, you need to do that “figure 8” thing?
These are examples of calibration — where the output of an instrument is compared against a reference standard, to make sure that it is correct (and/or set it appropriately).

We rely on calibration all the time, heck, modern life would basically be epically disastrous if it weren’t happening behind the scenes. Everything from the standardization of tape measures (think of how that results in the building you are in not collapsing), to voltage regulation (your iPhone is still working, right?), to, well, everything, assumes that instruments are calibrated!

Great, but what, you might ask, does this have to do with Deep Learning?
The answer is in the form of a question that Guo et al. asked themselves in a recent paper (•), viz., How do you know that the outputs of your model are still correct?

Think about that for a moment. You put together your model to recognize cats, trained it, and hey, you’re done right? You know this, because it works for you against your validation set. But (!), what about the other people who start using it? How do they know that it works for them? How can hey calibrate the network against their usage?

This isn’t just an abstract question — if your job depends on accurately recognizing that cat, would you leave it to the model? Or, on a more serious note, if this model was designed to recognize tumors, wouldn’t you like it to say “Hey, my accuracy on these kind of tumors is kinda poor, could one of you humans please look at this?”

When Guo et al. started looking at this, they discovered that

neural networks today are undoubtedly more accurate than they were a decade ago, we discover with great surprise that modern neural networks are no longer well-calibrated.

Or, to put it differently, while neural networks are way more accurate these days, the confidence level in their results is also way lower! Which is also (kinda) ok, as long as there is some way in which the results could come with a “confidence score”. You know, something like “I’m almost certain this is a cat” vs. “Yeah, I *think* this is a cat, but, boy howdy, I really wouldn’t bet on me being right!”
The bottom line is that confidence estimates provide valuable information for the user, allowing them to establish trust. The user may not trust the results, but they will trust the system enough to tell them if the results are trustworthy!

Anyhow, back to Guo et al., who discovered that confidence lags accuracy in neural networks these days. They found that this lack of confidence is clearly correlated with the increased width and depth of models, batch normalization, regularization, not using weight decay (go figure!), and pretty much everything that makes up a modern deep learning neural network.

They don’t really know why (there are some ideas in the paper though), but they do have an approach to providing a useful confidence metric — specifically Platt Scaling (••), where they train the model against the validation set to return a probability that the output (in the real world) is correct. Or, to put it a wee bit more technically, “The nonprobabilistic predictions of a classifier are used as features for a logistic regression model, which is trained on the validation set to return probabilities”.

For such a simple approach, it turns out to be remarkably effective — more computationally complex approaches like matrix scaling and binning, at best, perform as well, and sometimes perform far worse!

This is one of those things that really should be done by default — the classifier should always output a probability that the predicted label reflects the ground truth. It is pretty much the only way that we can reason about outputs when we start stitching these models together in the real world!

(•) “On Calibration of Modern Neural Networks” — Guo et al.

(••) “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods” — John C. Platt