Blue-Green Deployments … and Persistence-Hell

Blue-Green deployments are what you’re most likely doing in the early stages of your product (and, quite possibly, later on too!). Basically, you have two “production” environments — call them “Blue” and “Green” — with only one of them live at any point in time (say, “Blue”). You test out your new release on the “Green” environment, and when you’re good to go, switch over the traffic in one shot. 

There are about a jillion variations of this, and names too (heck, Netflix calls itRed/Black), but the underlying philosophy is just about the same. (•)

The thing is, this is remarkably simple and straightforward if each deployment is a “change the world” scenario — you’re shoving out the latest documentation, or deploying a self-contained app, or other suchlike.

But what if you do need to remember the past? Deal with Persistence as it were?
The answer, as with most things, is “it depends”.

1. 1. Is there any change in the way the new version works with PersistenceAssuming the answer is “No”, then you’re good to go. Cut over when you’re ready, and you should be fine.
2. 2. Are you sure there are no changes?Remember, “works with persistence” covers everything. CRUD of course, but also any new data formats that are being introduced. The context being that you need to be able to roll-back to a previous version, and, well, are you sure that the old-version won’t barf with any new data that is written?
3. 3. What about stuff that barfs during cut-over?
   See, if your old code dealt with failed transactions one way (e.g by reconstructing them), and the new code deals with it differently (by ignoring them), then any transactions that are in flight when you cutover run the risk of being lost. Are you ok with that?
   The point here being that you need to think of not just the physical data involved with persistence, but with the business-cases associated with the data, and how that gets affected.
4. 4. Oh, wait a minute, I just remembered there is a change in the data formats!
   Well, you can deal with this in many ways, but the safe way is to do this in three steps.
   a) Update your old code so that it can deal with both the old and new data formats.
   b) Update the data to the new format.
   c) Update your code again to where it needed to get to.
   The key here is that you only move between islands of stability, and never put yourself in a place where things are unclear or uncertain (“Is it barfing because of bad code? Or bad data?”)
   Mind you, strictly speaking -b- above is not necessary. It is, however, something that you might want to do if you can — it’ll help reduce the problem surface with the new code. Remember—always move between islands of stability!

By the way, the above rules don’t just apply to Blue/Green deployments — though they are easy to envision here — they’re valid for most all software-update scenarios. You really should, always validate that there are no physical data changes, and no changes in how the business processes deal with the data. If there are any changes, you should make sure that you incrementally update the rules so that you are always in an “island of stability”.
Not following these rules will lead to a world of hurt — trust me, I’ve been there!

(•) See Martin Fowler’s excellent writeup for all the gory details