And you are here why?

From the   AWS S3 documentation , about adding randomness to key-names to avoid “hot-partition” / sharding issues. If you anticipate that your workload will consistently exceed 100 requests per second, you should avoid sequential key names. If you must use sequential numbers or date and time patterns in key names, add a random prefix to the key name. The randomness of the prefix more evenly distributes key names across multiple index partitions. An oldie but goodie, this one is. I’ve been bit by it, on average, about once a year or so.   Shopify got nailed recently too , and a quick Google search reveals that there is a fairly constant stream of people running into this. It’s not just an S3 thing, it’s there in   the docs for Google Cloud Storage   too… The issue at hand is, that is the the type of thing that • you experience rarely,   but • when you   do   experience it,   it hurts ,   but • it’s not the kind of thing that you walk a...

I f you’ve lived in a place with ants (which is, oh, most of the world), then you’ve seen how ants build “bridges” with their own bodies. (And yeah, if you haven’t, just take a look at the video below)   It's a pretty amazing thing, and even more so given that ants are, well, not exactly the smartest creatures around. Mind you, thats “individually smart” — when you put them together you get  swarm intelligence , emergent behavior that arises from adherence to a few simple rules. In this case, it turns out that ants build bridges by following two basic rules 1. If an ant is walking on top of you, you freeze in place 2. If the number of ants walking over you dips below a certain rate, you start moving again. Yeah, there are some details like timeouts, etc, but the rules themselves are pretty simple. The fascinating part is that this also takes into account all sorts of tradeoffs, e.g.  — number of ants foraging vs “building bridges”  —  where  to build br...

More  details on the whole thing by Adyen here , but the key part is below (Note, this assumes that you can take the whole DB offline. If not, well, don’t do the below!) Stop traffic to the database cluster. Run the PostgreSQL upgrade on the Master server, using a script to automate as many steps as possible. The advantages of this approach are to speed up the process and make it easily repeatable. ( 3–5 minutes ) Stop PostgreSQL on the upgraded master and create a snapshot of the volumes. ( Up to 10 minutes ) Copy the snapshot of the upgraded master to the storage device of the slaves. ( 2–5 minutes ) Restart the master. Import the snapshot to the slave server, mount the (already upgraded) master volume on the slave server. Reconnect the slave server to the master, by simply putting in the correct recovery.conf file. Only when there are at least two slaves online, connected, and up-to-date, we allow access to the cluster again. So there is never a moment w...

With a sufficient number of users of an API, it does not matter what you promise in the contract, all observable behaviors of your system will be depended on by somebody.  — —  Hyrum’s Law Or,  to put it differently , “ given enough consumers, the interface will eventually exactly match the implementation ”. A quick story here — a developer I know (of eponymous  #CowboyDeveloper fame) built out a complicated  thing  whose main job was to process incoming messages. There was an entire eco-system of queues — queues to handle messages, queues to handle ACKs, queues to deal with transactions, queues to deal with queues, and lord knows what else, but the whole thing was wrapped in a fairly simple interface. As fate would have it, this  thing  ended up in widespread use, as part of a larger mission-critical system. If you registered the “ lord knows what else ” bit above, you probably spotted the problem, that nobody other than the #CowboyDevelope...

It’s amazing how frequently I see this, y’know? Where “ passing tests ” is considered   proof , as if it has validated the bug-free nature of your code, when all it means is that   your code passed the regression tests . Thats it. No more (and no less, mind you). Are there still bugs in there? —  Yeah! Will your users find them for you? —  Hell yeah! Will they be found 2am Saturday? —  Certamente! Mind you, if your organization has leveled up, and you’ve built out your deployment pipelines, and you   understand   that your regression tests are your last barrier, the “ it’s the best we can do at this point in time ” validation of your system, then, well, you still have bugs, but at the very least you   understand that you still have bugs. Tragically, in this world of   #CowboyDevelopers , that is not the case. I mean, how do you argue with someone who genuinely believes the following? “I don’t see why I need to write tests for it — *I* wrote this...

“ Locking is easy — just use a mutex!”         --#CowboyDeveloper One of the surest “tells” of a  #CowboyDeveloper  is their behavior around a shared resources. Odds are they’ll slap a mutex around it (or around  something ), and keep going. Oh, if you really poke at them, they might be able to ( intelligently? maybe not ) say  something  about  mutexes vs semaphores vs spin-locks vs … , but, seriously, in their mind, “I t’s all so simple! You just lock/unlock it! Like, with mutexes! ” The above, mind you, as contrasted with the  excellent work by Willy Tarreau (most definitely  not  a #CowboyDeveloper) on  Progressive Locks . “What are they?” you ask? Well, if you’ve got a resource (tree, list, whatever) where you’re usually reading the resource, but, sometimes, you want to write  only  if your Read succeeded. In short, you want to be able to “upgrade” from a Read to a Write. S...

How do you  guarantee  that the data in your database is immutable? In particular, I’m talking about data that you  don’t want to change — things like call-records, historical data, transaction logs, etc. Of course, you could use something like  Datomic , or you could architect your application/data-structures/tables/… from the ground up to be immutable, but that could be … problematic. Especially problematic, in fact, given that I’m referring to your  existing  databases that have all this stuff in it — good luck trying to convince the business side of the house that you need to re-architect  everything ! Theoretically , this data should already be immutable, but, the reality is that there are  so many  attack vectors. 1. One of your developers could go in and tweak the data — you’d never know 2. Oh, only the DBA has access? What if  they  tweak it? 3. Worse, what if somebody hacks into your system? 4. Even worse, ...