Continuous Testing and Development Hygiene

Continuous Development (and Continuous Testing) is increasingly common out there. The process of pushing a commit, and having it kick off a run on Travis / CircleCI/ Jenkins is so frightfully mundane that one does it without actually thinking about it much these days.
That said, here is (highly unscientific!) study of errors that I've found in CD environments
1. Compilation Errors. Yes. Seriously. And the ridiculous thing is that pretty much every damn IDE these days supports some form of "compile on save" option, no? 
2. Missing Code. There's literally a gaping hole where an entire module should be. This could be because the developer forgot to implement it, or that the developer pushed the wrong branch, but still, come on…
3. Wrong Tests. The developer cut/paste tests from a different module as a starting point, and then happily forgot to actually implement them correctly.
4. Missing Data. The test harness depends on data that isn't checked in. Whoops!
5. Timeouts. This one is pernicious, by far the most painful, and usually there because either
• the module depends on an (unmocked) resource that isn't available in the CD environment (network stack, external service, whatever). The test just hangs till it times out. Or
• the test is just dumb, with nested iterators that won't complete looping till the heat-death of the universe
In either case, you end up with a 10 minute wait (Travis/CircleCI defaults) before you know something bad happened!

The problem with all of the above is that they add friction. The friction can be as small as the round-trip time ("Damn it. Compilation error. Damn it" followed by git commit --amend), and as large as 10 minutes of thumb twiddling before a timeout.
But wait, that's not all! The odds are that you don't have an unlimited amount of testing concurrency, and every time you do something goofy, you're using up one of the slots. Add in enough people being goofy, and the next thing you know, entirely sane jobs are being queued up because of the goofiness.
The point here, in case it isn't painfully clear, is that you need to make sure that there is some minimal level of development hygiene that is embedded in the CD pipeline. There are plenty of tools available out there, make use of them. (•)
And yes, you're not just restricted to software tools, you have behavioral psychology at your disposal - "XXX's job barfed with a Compilation Error on Travis" tossed into a group slack channel might do wonders 😆

Comments

Popular posts from this blog

Cannonball Tree!

Erlang, Binaries, and Garbage Collection (Sigh)

Visualizing Prime Numbers