And you are here why?

So you’re going to revolutionize ice-cream delivery, through the powers of your iOS/Android app. ( Work with me here. Pretend that you’ve actually invented a better mouse-trap ) Your back-end runs on an AWS/EC2 instance, which is fine while you’re developing, but as you start thinking about real live customers,  reliability  comes to mind, and you split the back-end up so that it’s running on  TWO  AWS/EC2 instances. Great, right? If one goes down, the other takes over the load, and everything is hunky dory (as long as you the combined load is less than the capacity of any one of the servers, but let’s ignore that for now). Except, what if  AWS itself goes down ? As  June 29, 2012  taught a lot of us, the un-imagineable  does  happen every now and then. The solution, of course, is to deploy your instances across multiple availability zones, or maybe even multiple regions, or heck, while you’re at it, across multiple cloud providers....

A friend of mine has a restaurant in the East Village ( Grape & Grain  — you should go!). It’s a small thing, seating maybe two dozen people. The fascinating part is how the chef — Adam Rule — generates consistently great dishes despite his “kitchen” consisting of 3 induction plates and a convection oven, all tucked into a corner. I’ve seen him operating when • A group books the entire restaurant for dinner, and pre-selects the menu (“ Could you serve everyone clams for starters, then pasta, the sea-bass, and pavlova for dessert? ” • There are never more than a dozen or so in the restaurant, but as one couple leaves another shows up. This from 6pm to midnight • A dozen people suddenly show up, and proceed to order  everything  off the menu. And in all these cases, the food always shows up like clockwork, the entire party gets their pasta at the same time, newcomers don’t sit around waiting, etc. You get the point, the service is exactly what you would expect at...

Way back in 2014 Yuan et al conducted  an analysis of production failures that brought down distributed systems  (•). They went through 198 random failures in well known distributed systems like Cassandra, Hadoop, and whatnot, and, well, the results were remarkably depressing. Basically, they found that for most of the  catastrophic  failures — where the entire system went down — the root cause was ridiculously simple. How simple you ask? Well, it turns out that around 90% of them boiled down to non-fatal errors that weren’t handled correctly. Yeah, I’ll let that sink in for a bit. Bad error handling. That’s it. Ok, now that it’s sunk in, let’s make it worse. 35% of the catastrophic failures boiled down to one of the following three scenarios 1.  The error-handler catches an overly general exception (e.g. an  Exception or  Throwable  in Java), and then shuts the whole damn thing down. BOOM. 2.  Or, better yet, the error-handler a...

So yeah, this was years ago, before AWS was a thing. We provided business phone services (•), and had all our servers at a hosting facility (With cages. Where we had our own servers. Remember, this was before AWS!). This was a  real  hosting facility, with batteries, and a generator to back the whole thing up. And we were a  real  phone company, with tens of thousands of customers. All good fun — for a given definition of fun, mind you. Until the one day when there was a massive snow-storm, and a huge percent of D.C. lost power. Including our hosting facility. Which was fine, because the batteries took over immediately. And the generator kicked in, because the batteries were only good for, like, a minute or two of power. And after two minutes, all our servers crashed. Because, as it turned out, during routine maintenance, some dude had disconnected the generator from the circuits,  and never reconnected them . The good news was that a decent chunk ...

The good news around metrics is that it is increasingly easy to instrument and monitor your code, with plenty of usable software out there like  Prometheus ,  TICK ,  OpenTracing , and whatnot. The bad news is, well, pretty much the same — it’s so easy to instrument and monitor your code that the sheer amount of stuff that you can capture is mind-boggling. It is  so  easy to get caught up in all the charts and graphs that you will find yourself constantly losing sight of the the forest for the trees. All that data ticking around on the screen can, counter-intuitively, mean that it is actually  harder  to actually get any information (let alone knowledge!) from the metrics. Latency Latency is a particularly good example of this. Most everyone out there measures this as an  average , and while that’s better than nothing, it can easily hide issues. The fault, as always, lies in us — in that we tend to think about our systems in terms of steady-...

I already covered  “ Does It Work? ” from the customer’s perspective , looking at  R ate ( Does it scale? ),  E rrors ( Are these the responses the customer needs? ), and  D uration ( How quick was the response? ) which together go by the acronym  RED . The thing is  these metrics are great for the customers currently using the service, but they give you limited information about potential customers!  Think of it this way — if your system is right at it’s limit (whatever that means), then your current customers might be very happy, but the next one that hops on might get terrible service. Or worse, the service or  everybody  might crap out!. From an  observability  perspective, you should be looking — at the very least — at • Utilization  (aka “ How busy is it? ”): The percentage of time the service is busy doing stuff, and conversely, the percentage of time it has available to do more stuff. • Saturation  (aka: “...

There is a world of a difference between  “It’s up”  and  “It works” .  I could go on about this for hours, but, it basically boils down to your customers — they care about the quality of the experience, and  only  the quality of the experience. It may be up as far as you’re concerned, but if interacting with it sux, well, from the customer’s perspective, does it actually work?   Take the above comic. Your implicit assumption might be something like “ It’s a stupid UI ” or some such. Much as we’d like to blame everything on the poor folks in UX, it really could be as much about poor performance aspects, you know? Think of all the times that you’ve hit “Submit”, and gotten back a server error, searched for something with no result when you  know  it’s there, and so on. You need to make sure that  those  aspects of the Customer Experience are covered, as follows — • Request Rate  (aka “ Does it scale? ”): How many reque...