And you are here why?

You may have heard of Project ORCA the Romney Campaign's massive high-tech Get Out The Vote effort in 2012, which failed, and spectacularly at that.  As John Ekdal put it By 2PM, I had completely given up. I finally got ahold of someone at around 1PM and I never heard back. From what I understand, the entire system crashed at around 4PM... So, the end result was that 30,000+ of the most active and fired-up volunteers were wandering around confused and frustrated when they could have been doing anything else to help From all appearances, it was a combination of lack of robustness , lack of scalability , and a pernicious consultant culture that doomed it. This wasn't the first time I'dseen problems of this type, and it almost certainly won't be the last.  The underlying issue was pretty standard. - basically, the word "Robust" takes on a whole new meaning when failure is not  an option. This is really important, and worth repeating loudly -   ...

So you've got a cloud-based service.  Good for you!  I bet its brilliant,  is brilliantly fault-tolerant 'cos of Erlang/OTP, and is going to make you Pots o' Money!  That is, of course, assuming that you have prepared yourself for the inevitable "Oh S**t!" moment, when It Goes Down.  Herewith a couple of pointers to help you prepare for everything not working exactly the way it is supposed to (Servers die?  really?)… 1) Make sure that you understand your entire system .  Done?  Good, now make sure that all of your developers understand the entire system.  Specialization might be good (and don't get me wrong, weed-whackers are *very* useful little tools, especially when you get into the weeds :-)  ), but if they don't grok the big picture, well, you're hosed.  Capturing vast quantities of scoring data could be really useful for GUI updates, and might be easy to implement from an API perspective, but if your persistence layer...