Life on the Edge

So yeah, this was years ago, before AWS was a thing. We provided business phone services (•), and had all our servers at a hosting facility (With cages. Where we had our own servers. Remember, this was before AWS!).

This was a real hosting facility, with batteries, and a generator to back the whole thing up. And we were a real phone company, with tens of thousands of customers. All good fun — for a given definition of fun, mind you.

Until the one day when there was a massive snow-storm, and a huge percent of D.C. lost power. Including our hosting facility.
Which was fine, because the batteries took over immediately.
And the generator kicked in, because the batteries were only good for, like, a minute or two of power.

And after two minutes, all our servers crashed.

Because, as it turned out, during routine maintenance, some dude had disconnected the generator from the circuits, and never reconnected them.

The good news was that a decent chunk of our customers were from the D.C. area, and were not actually at work (remember, massive snow-storm).
The bad news, was that they were, theoretically, relying on our oh-so-cool VoIP system to route their calls to their home.
The good news was that since we weren’t up, they didn’t get any calls, and they thought it was just because of the snow.
The bad news was that they eventually figured it out.

I still break out in a cold sweat when I think of that day…

(•) Incidentally, if anybody ever suggests that you get a job in Telephony, back away slowly without making eye-contact, and at the first available opportunity run.