Ignorance Is NOT Bliss — Flaky Tests Edition

We’ve all dealt with flaky tests, right? Tests that don’t consistently pass or fail, but are, instead, nondeterministic?They’re deeply annoying, end up being a huge time-suck, and inevitably end up occupying the bulk of your “productive” time.

I’ve written about the common reasons for flakiness before, viz., External Components, Infrastructure, Setup/Teardown, and, Complexity (read this post for the details — it’s a very short read!).

The big takeaway from that article though is that flakiness never goes away, it just moves up the food-chain! For example, as you clean up your Infrastructureissues, you’ll start running into issues with Setup/Teardown. Fix those, and you’re now dealing with Complexity. And there is, of course, a special place of pain involved with anything involving distributed systems, what with consensus, transactions, and whatnot.

There are many many ways of dealing with flakiness, I’m not going to get into that here. The thing though is that whatever you do, ignoring flaky tests should never be an option! See, by ignoring them, what you’re actually saying one of

1. 1. You’ve got Bugs that you don’t care about.
2. 2. You don’t quite understand how the system works
3. 3. Both -1- and -2-

/via
http://geekandpoke.typepad.com/geekandpoke/2009/07/the-art-of-programming-part-2.html

Here’s the important bit, y’see, -2- above is the real killer.
After all, if you don’t actuallyunderstand how your system works, you are, at best, doing a whole bunch of blackbox testingto ensure correct functionality.
And that means that the regression tests that are built to ensure that the system continues operating correctly are built on top of those blackbox tests. And the same with the integration tests, etc.
In short, you’re building on quicksand!

The bottom line here is that even though flakiness is here to stay, you need to make sure that it is about bugs that you haven’t found (or don’t care about), and not about your lack of knowledge about your system.