And you are here why?

So you’re going to revolutionize ice-cream delivery, through the powers of your iOS/Android app. ( Work with me here. Pretend that you’ve actually invented a better mouse-trap ) Your back-end runs on an AWS/EC2 instance, which is fine while you’re developing, but as you start thinking about real live customers,  reliability  comes to mind, and you split the back-end up so that it’s running on  TWO  AWS/EC2 instances. Great, right? If one goes down, the other takes over the load, and everything is hunky dory (as long as you the combined load is less than the capacity of any one of the servers, but let’s ignore that for now). Except, what if  AWS itself goes down ? As  June 29, 2012  taught a lot of us, the un-imagineable  does  happen every now and then. The solution, of course, is to deploy your instances across multiple availability zones, or maybe even multiple regions, or heck, while you’re at it, across multiple cloud providers....

Way back in 2014 Yuan et al conducted  an analysis of production failures that brought down distributed systems  (•). They went through 198 random failures in well known distributed systems like Cassandra, Hadoop, and whatnot, and, well, the results were remarkably depressing. Basically, they found that for most of the  catastrophic  failures — where the entire system went down — the root cause was ridiculously simple. How simple you ask? Well, it turns out that around 90% of them boiled down to non-fatal errors that weren’t handled correctly. Yeah, I’ll let that sink in for a bit. Bad error handling. That’s it. Ok, now that it’s sunk in, let’s make it worse. 35% of the catastrophic failures boiled down to one of the following three scenarios 1.  The error-handler catches an overly general exception (e.g. an  Exception or  Throwable  in Java), and then shuts the whole damn thing down. BOOM. 2.  Or, better yet, the error-handler a...

So yeah, this was years ago, before AWS was a thing. We provided business phone services (•), and had all our servers at a hosting facility (With cages. Where we had our own servers. Remember, this was before AWS!). This was a  real  hosting facility, with batteries, and a generator to back the whole thing up. And we were a  real  phone company, with tens of thousands of customers. All good fun — for a given definition of fun, mind you. Until the one day when there was a massive snow-storm, and a huge percent of D.C. lost power. Including our hosting facility. Which was fine, because the batteries took over immediately. And the generator kicked in, because the batteries were only good for, like, a minute or two of power. And after two minutes, all our servers crashed. Because, as it turned out, during routine maintenance, some dude had disconnected the generator from the circuits,  and never reconnected them . The good news was that a decent chunk ...

Migrating data is  never  fun. It is, at best, a PITA, and at worst, a reason to go find a different job. But hey, sometimes you have no choice, and you just have to Go There, and that’s where  Scientist  comes in. Scientist ? Yeah,  Scientist . But do wait, we’ll get to it in just a moment. First, let’s just look at what you’re probably going to be doing to migrate your data between, oh, whatever it is — databases, DB versions, applications, whatever. Basically, you’ll be doing some variation fo the  Dual Write  approach  (•), which, roughly, translates to something like 1. Do all your  Writes  (Actually,  Create / Update / Delete ) to both the  Old , and the  New  targets, while serving  Reads  only from the  Old  target. 2. Start serving some of the  Reads  from the  New  target. 3. Eventually serve all the  Reads  from the  New  target. 4. Tu...

“ Netsplits are rare, so I don’t think about them ”  —  #CowboyDeveloper The thing about the above statement is that even if you aren’t a   #CowboyDeveloper , it’s not necessarily wrong .   That’s for a given value of “rare” mind you, and   you ignore it at your own risk . The question you should ask yourself before making the above statement is “ What is the risk associated with a netsplit? ”, and the answer to that should inform your engineering decisions (•). And that brings us to the main point here — this is not just engineering decisions about network partitions — it’s   all   your engineering decisions! So great — you assess the risks, plan / design accordingly, and it’s all copacetic, right? Well, no, and that’s because “assess the risks” is carrying a lot of water. The issue here is that the   people   implementing the systems are, well, human, and as humans, they are quite likely to end up with some combination of 1. “ If I don’t kn...

You know what a bulkhead is, right? You know — in ships, where they create watertight compartments so that in case there is a leak, it is isolated to just one part of the ship, instead of the whole thing pulling a Titanic? Well, pretty much the same applies to software, where “bulkheading” is where you isolate parts of the system so that if one part of it barfs, it doesn’t infect and/or clobber the other parts of the system. It’s one of the core-precepts of building fault-tolerant systems, viz.,   Error Encapsulation . From a “big-picture” architectural perspective, yes, this does mean all the usual stuff about “loose-coupling”, “component-driven architecture”, “microservices”, and pretty much any other buzz-word in this vein that you can think of. However, the actual   functionality   associated with this is something that you should keep in mind — and towards that, let’s look at a specific example You’ve got a connection pool, serving two sets of back end se...