And you are here why?

It happened back in the early 2000s— the VoIP-based phone company I had founded back in 1999 had ( barely ) recovered from it’s near-death experience after the  Telecom Crash , and we had managed to get our heads (again,  barely ) above the water. I had taken our postgres server offline for reasons, and — horror of horrors! — postgres wouldn’t come back up! I know what you’re thinking, but you do need to remember a few things This was pre-AWS, we hosted everything ourselves We were  really  cash-strapped. Like, “making payroll for everyone out of savings” level cash-strapped We, literally, couldn’t afford a backup database server (they were  expensive! ) So yeah, postgres wouldn’t start up — it kept barfing with some seriously obscure error involving a corrupted file header, we had passed our downtime window, and customers were hammering our cell-phones like you wouldn’t believe. If I couldn’t get things back up, well, my company was toast. I...

It only matters whe it affects me  — Every #CowboyDeveloper Ever Isolation Levels You know about that the default isolation level in Postgres is “ Read Committed ”, right? You do? Well good for you Genius, ‘cos I sure as hell didn’t. But first, a digression. Isolation Levels , in a database, refer to the various ways in which concurrent transactions can impact each other. Say I’m  read -ing a bunch-a data from the database, and database, and during this you  update  one of the records, what happens? • Do I see the data  before  your  update ? • Do I see the data  after  your  update ? • If my  read  is actually a couple of table scans inside a single transaction, what happens? Do I get  both  the  before  and  after  version of the data? • Does my  read  just bork with a “ somebody is updating this table ” error? • Do I get a  NullPointerException ? (Sorry. PoorJava humor. M...

More  details on the whole thing by Adyen here , but the key part is below (Note, this assumes that you can take the whole DB offline. If not, well, don’t do the below!) Stop traffic to the database cluster. Run the PostgreSQL upgrade on the Master server, using a script to automate as many steps as possible. The advantages of this approach are to speed up the process and make it easily repeatable. ( 3–5 minutes ) Stop PostgreSQL on the upgraded master and create a snapshot of the volumes. ( Up to 10 minutes ) Copy the snapshot of the upgraded master to the storage device of the slaves. ( 2–5 minutes ) Restart the master. Import the snapshot to the slave server, mount the (already upgraded) master volume on the slave server. Reconnect the slave server to the master, by simply putting in the correct recovery.conf file. Only when there are at least two slaves online, connected, and up-to-date, we allow access to the cluster again. So there is never a moment w...

Historically, we have treated the distribution of data in a database as opaque for the purposes of indexing.(°) I mean, yes, this is where hand-tuning comes in, but that is, by definition, hard to automate. If, OTOH, our friendly   #MachineLearning   system could analyze the data, and tune the index on the fly, well, that’s a game-changer, no? Herewith a paper that goes into pretty much exactly this —  https://arxiv.org/abs/1712.01208 . The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or existence of records. Read on! (°) The point here is that you need to know what the data is. Is the key you are looking at • a UUID? • an integer? • a sequential integer? • Is it unique?