We’ve All Been There…

/via http://www.commitstrip.com/en/2013/10/09/la-pire-sensation-du-codeur/

It was a glorious Friday night in Hoboken, back in 2008. Ok, this is Hoboken we’re talking about, so it wasn’t all that glorious, but we’d been out for dinner with friends, friends who appreciated wine, and, thanks to the wine (and friends!) it was a glorious night.

Our taxi had just pulled up by our house, and everyone was trooping up for MOAR WINE YES LETS SEE WHAT WINES THERE ARE YES, when I got a call from one of the SysAdmins.

Him: “Uh, dude, I think I did something”
Me: “????”
Him: “I may have shutdown our primary DB”
Me: “!!!!”
Him: “I shutdown the DB on my desktop when I left, but now that I think of it, I may have done it in the wrong window”
Me: “!!!!”
Him: “And I’m on the freeway, which is backed up seven ways to Sunday…”
Me: “!!!!”
Him: “And I don’t think the secondary took over, because Customer Support is getting slammed”
Me: “!!!!”

We had many many protections against BadThings™, but, somehow, we’d never thought to protect against shutdown now . Go figure…

(Incidentally, yes, I managed to to deal with the — thankfully trivial — glitch that had caused the secondary to not take over…)