And you are here why?

Ever been a Systems Administrator? I have (first job I ever had!), and I have to tell ya, it’s a pretty thankless gig. I mean yes, there is the satisfaction of a job well done, the joy of knowing that everything is up and working to spec, etc., but, behind all of this, you are left with the knowledge that  Nobody Cares Till It Breaks  😔. Mind you, it’s not that they don’t care at all. It’s more like, from their perspective, “It’s Working” is normalcy, steady-state, the way things should be. You don’t get into a Toyota Camry, and say to yourself “ Hmm, I wonder whether the engine-block will fall out of the car on the way to work ”. Engine-blocks remaining in the car is the default state of existence, and if one  does  happen to fall out of the car, then hey, it’s definitely Bizarro-City! And that, my friends, is what SysAdmins — and software engineers in general — deal with all the time. As far as users are concerned, systems are Toyota Camrys, and are supposed...

/via http://www.commitstrip.com/en/2013/10/09/la-pire-sensation-du-codeur/ It was a glorious Friday night in Hoboken, back in 2008. Ok, this  is  Hoboken we’re talking about, so it wasn’t all that glorious, but we’d been out for dinner with friends, friends who appreciated wine, and, thanks to the wine (and friends!) it was a glorious night. Our taxi had  just  pulled up by our house, and everyone was trooping up for  MOAR WINE YES LETS SEE WHAT WINES THERE ARE YES , when I got a call from one of the SysAdmins. Him: “ Uh, dude, I think I did something ” Me: “ ???? ” Him: “ I may have shutdown our primary DB ” Me: “ !!!! ” Him: “ I  shutdown  the DB on my desktop when I left, but now that I think of it, I may have done it in the wrong window ” Me: “ !!!! ” Him: “ And I’m on the freeway, which is backed up seven ways to Sunday… ” Me: “ !!!! ” Him: “ And I don’t think the secondary took over, because Customer Support is getting slammed ” Me: “ !...

/via http://www.commitstrip.com/en/2012/03/14/whatever-it-takes/ I was supposed to be on vacation. Well, technically I  was on vacation, but the point here is that I should have been doing the stuff one does on vacation, like eating (and drinking!) too much, hiking, waking up late, and so on. Instead of what I actually  was  doing, which was trying to figure out how to bring our servers back up from where I was in the middle of nowhere. Let’s rewind this story a little, and put some context out there. At the time, I was the CTO of a phone company¹. Not just  any  phone company, we were one of the CoolAndInnovative™ ones — so cool that I had been named one of the “Top Influencers in VoIP” (which basically meant that people laughed openly at me, instead of behind my back). Anyhow, we had a bajillion VoIP phones scattered around the country, each of which connected to our data-center over the InterTubes.  Our  data-center mind you, as this was be...

It happened back in the early 2000s— the VoIP-based phone company I had founded back in 1999 had ( barely ) recovered from it’s near-death experience after the  Telecom Crash , and we had managed to get our heads (again,  barely ) above the water. I had taken our postgres server offline for reasons, and — horror of horrors! — postgres wouldn’t come back up! I know what you’re thinking, but you do need to remember a few things This was pre-AWS, we hosted everything ourselves We were  really  cash-strapped. Like, “making payroll for everyone out of savings” level cash-strapped We, literally, couldn’t afford a backup database server (they were  expensive! ) So yeah, postgres wouldn’t start up — it kept barfing with some seriously obscure error involving a corrupted file header, we had passed our downtime window, and customers were hammering our cell-phones like you wouldn’t believe. If I couldn’t get things back up, well, my company was toast. I...

/via http://www.commitstrip.com/en/2018/06/22/are-you-any-good-at-it/ I still remember discovering perl back in 1991. I was one of two Unix Guys at Notre Dame back then, and we were responsible for managing a metric ton of SPARCstations, AFS, remote booting, and all sorts of weird-ass s**t that, luckily, most of us just don’t have to worry about anymore. And yeah,  so  much of everything was run using a cobbled together collection of  csh  ( bash wasn’t huge back then). Perl was a godsend, simplifying  so  much of the nasty text processing that was intrinsic to the sysadmin stuff we used to do back then. Mind you, most of this was because it beat the pants off of  csh , and that was good enough for us! Years went by, and perl just became a core part of my toolkit — to the point that we pretty much ran a telco for over 10 years with most of the management/automation (what we’d call DevOps these days!) done in perl,  and it worked . In...

/via https://xkcd.com/705/ One of the side-effects of the  DevOps  craze is the refrain that  SysAdmins  are no longer relevant, that the world of Container Orchestrations (and Continuous Delivery! and Serverless!) make  Operations  a thing of the past. Lets play with this for a moment. Take a look at your environment — your DevOps-fu is strong, you have the ultimate in Continuous Delivery, and whenever you merge into  master , stuff ends up in Production. Cool, isn’t it? A little twist here though, turns out that you’ve got a bunch of assets that are best served from a CDN ( Cloudflare   perhaps). Which leads to a whole thicket of issues —  • Who gets to decide whether the changes you want to make to it are viable? • What if those changes conflict with somebody else’s needs? • What happens if that other team has a problem — do they call you? • Heck, what if you want to switch to  CloudFront ? There are all sorts of ways you c...

/via CommitStrip

Began my work life as a sysadmin 24 years ago. Have done many things since then, but at heart, the sysadmin still abides, As usual, User Friendly and Dork Tower say it best