And you are here why?

There are known and proven ways to build fault-tolerant systems (erlang!  ). An underlying issue with most of these, is that they tend to assume that components are either  up  or  down . Gray Failure  is a real thing however — where the component is “failing” (quotes intentional), but the manifestation of this failure isn’t easily categorizable. Think “ flaky I/O ”, “ bufferbloat ”, “ packet loss ”, etc., things that may or may not be considered a failure based upon the impact to the user, interactions with other components, etc. The issue with gray failures is that they increase as the system scales out — in fact,  “the complex interactions, interference, and dependencies among cloud components … in a multi-tenancy environment”  (•), and the emergent behavior thereof, are the perfect petri-dish for gray failures. A common characteristic of all gray failures is  Differential Observability , viz., the behavior of the component varies...

#Wallaroo   pipelines (°) can distribute/fan-out their data-processing to multiple back-end nodes — in a map-reduce kinda. A (the?) cool aspect to this is that the processing can be stateful . The way it works is surprisingly simple — they Break “global state” into non-intersecting partitions, based on keys Map keys to partitions with a “partition function” Distributed partitions to nodes (redundantly, for reliability) Send input data to partition/node based on the input key Rebalance partitions when a node is added or removed It’s a clever trick, check out the details at   https://goo.gl/vAUEeS (°) Wallaroo applications are composed of one or more pipelines. A pipeline starts from a source, a point where data is ingested into the application. It is then composed of zero or more computations or state computations. Finally, it can optionally terminate in a sink, a point where data is emitted to an external system.

The failure rates for startups is 30 - 90% , depending on how you define "failure".  That said, the failure rate for tech / VC based startups is really pretty far along on the 90% end of the scale .  The point here? MongoDB is remarkably good, useful, and effective at rapid prototyping, and quick development.  Oh, it may have any number of issues at scaling, data integrity, etc., but it is - really - way cool when it comes to Just Getting Something Out There. Putting the two together MongoDB - because if your startup is destined to fail, you may as well Fail Fast

There is a paper out by Google on how they migrated from MySQL to F1 for AdSense . Its fascinating reading on its own, but its got some key take-aways for scaling and reliability.  I'd recommend going and reading it, but these two are worth emphasizing Scaling Eliminate the R in ORM .  In short, instead of building out a nice layer (hibernate, whatever) between your DB and your code, go the other way, and deliberately expose the workings of your DB to your application developers.  This actually makes tremendous sense - in many ways the advent of NoSQL has been due to people specifically picking data stores that map to their application requirements .  Need a key-value store? Use Riak .  Need a document store wht guaranteed writes?  Use CouchDB .  Hiding the specifics of your data store behind an ORM layer is - increasingly - becoming irrelevant, and Google quite probably just made it official. Fault-Tolerance Use 5 replicas. 5? Why 5...