And you are here why?

Way back in 2014 Yuan et al conducted  an analysis of production failures that brought down distributed systems  (•). They went through 198 random failures in well known distributed systems like Cassandra, Hadoop, and whatnot, and, well, the results were remarkably depressing. Basically, they found that for most of the  catastrophic  failures — where the entire system went down — the root cause was ridiculously simple. How simple you ask? Well, it turns out that around 90% of them boiled down to non-fatal errors that weren’t handled correctly. Yeah, I’ll let that sink in for a bit. Bad error handling. That’s it. Ok, now that it’s sunk in, let’s make it worse. 35% of the catastrophic failures boiled down to one of the following three scenarios 1.  The error-handler catches an overly general exception (e.g. an  Exception or  Throwable  in Java), and then shuts the whole damn thing down. BOOM. 2.  Or, better yet, the error-handler a...

“ I already have enough logging, tracing is just redundant ”  —  #CowboyDeveloper   Logging works — oh yes it does — but only up to a point. The moment you start dealing with interactions that span systems and services all hell breaks loose.  Causality  becomes increasingly hard to track down ( which request caused this response? ), and “ before-after ” relationships are just a PITA. With tracing, you can — trivially — isolate the service and the relevant span, allowing you to drill into the behavior of your system without worrying about other concurrent requests cluttering your analysis. (And no, don’t get started with timestamps. These are distributed systems we are talking about!) Enter  OpenTracing , with which you can track the journey taken by the request as it bounces through your increasingly elaborate service. The general idea started with the  Dapper  project at Google, which inspired the OSS  Zipkin , and has ended with  it be...

“ What happens after May 25th, 2018 to data on blockchains? ” The context being around Article 17, and the "right to be forgotten" . The answer, of course is some combination of “ It’s tricky ”, “ It depends ”, and “ It’s complicated ”. For the nonce, I’m going to ignore Personally Identifiable Information (PII) that’s actually placed on public blockchains. If you did that, or are doing that, well, good luck — you’re going to need it. On the private/permissioned blockchain side, there are a couple of options out there ( Note: By “private/permissioned”, I mean that the blockchains are deployed in an internal/controlled environment ) 1. Store PII elsewhere : This is the most obvious route, where you store PII in a separate database / storage / whatever, and you effectively do a JOIN on this data whenever you do blockchain stuff. The upside is that when you get an Article 17 request, you can nuke the data in this separate store and you’re done. The downside is, well ...

Debugging distributed services is Teh Sux0r”  — this is a   very   common complaint from people new to the world of distributed computing. Mind you, it’s true, but it also kind of elides the point. “Old School” monolithic single-node apps may support things like setting breakpoints, stepping through code, following (and changing) variables on the fly, and so forth, but once you start thinking in distributed terms (Consistency, Latency, Resilience, etc.), the very act of “debugging” can interfere with the thing being debugged. This, however, does not mean that there is   no   room for debugging. The more you focus on debugging the components — instead of the interactions between the components — the more value there is in “old school” debugging, and thats where   Squash   comes in ( https://github.com/solo-io/squash ) From the docs “Squash brings the power of modern popular debuggers to developers of micro- services apps that run on container orchestra...

Reliability is Hard. Oh, there are tools to make it easier, but “easier” is not the same as “easy”. And Reliability is, certainly, not Easy. When you factor in the (geometrical) complexity of distributed components, and the (geometrical) complexities that get layered on top when these components span nodes, and then networks, well,   “Reliability is Hard”   gets close to being the Understatement of the Year. And that’s where rigor comes in, rigor in the form of empirical tests, rigor in the form of mathematical certainties, rigor in the form of proofs, models, and battle-testing, in short   RIGOR . Do you, genuinely, understand what your system does? Can you explain this to others, simply, in a form they can understand (a-la Feynman)? Most importantly, is there any place in the system where you up with “I hand-tuned this till it worked” “I implemented my own   <something in common use, that I didn’t know existed> ” “It doesn’t matter, it works, and ...

The world of Distributed Systems is as much about philosophy as it is about implementation — the specific context, architecture, tooling, and developers all come together to make your way different from everybody else’s. And yet   yet , there are overarching themes and patterns that hold true, that you ignore at your own peril. The most important of these is the necessity to architect for failure — everything springs from this. (°) Complexity grows geometrically with the number of components, and no amount of fakes, strict contracts, mocks, and the like can reduce this. If anything, poor practices will only   increase   this complexity The key is to accept this, and align the risk associated with a given service/component with the risk the business is willing to take. In short,   make the component reliable, but no more reliable than necessary” As a developer, what this means is that you   must • Understand the operational semantics of the system as ...

Rolling Your Own Consensus  is clearly the blockchain version of  Rolling Your Own Crypto. Reasons for doing so that I have seen include, • I have a Better Way To Do Things  (No. You don’t.) • It shouldn’t be hard, therefore it isn’t  (Why? also, Yes, it is.) • What could possibly go wrong?  (Yup) • We’re on AWS, so net-splits aren’t a thing  (Oh yes they are) • Or outages  (Ha. ha. ha. ha. ha) • Who cares what the academics think, this is the real world  (Yikes) • We use redis for co-ordination  (Sigh) • Actually, redis-cluster  (Double sigh) Next time you’re faced with one of these, I recommend running away. Immediately. With vigor. Mind you, if you  can’t  run away, try asking some combination of 1) Is it proven? 2) Really? 3) With Math? 4) And peer-reviewed papers? For extra credit, ask if they redid Paxos or Raft,  without the un-necessary parts. And  then , run away. Immediately. With v...