Posts

Showing posts with the label s3

Passing on what you know — S3 Edition

Image
From the   AWS S3 documentation , about adding randomness to key-names to avoid “hot-partition” / sharding issues. If you anticipate that your workload will consistently exceed 100 requests per second, you should avoid sequential key names. If you must use sequential numbers or date and time patterns in key names, add a random prefix to the key name. The randomness of the prefix more evenly distributes key names across multiple index partitions. An oldie but goodie, this one is. I’ve been bit by it, on average, about once a year or so.   Shopify got nailed recently too , and a quick Google search reveals that there is a fairly constant stream of people running into this. It’s not just an S3 thing, it’s there in   the docs for Google Cloud Storage   too… The issue at hand is, that is the the type of thing that • you experience rarely,   but • when you   do   experience it,   it hurts ,   but • it’s not the kind of thing that you walk a...

Things the documentation doesn't make quite clear - Part 78

Image
So yeah, per the AWS ECR docs , EC2 stores your images in S3.  What they don't tell you, unless you really go hunting , is that This isn't just a feature, but something really relevant to you because  You'll need to be able to get to S3 to push/pull image Which may not matter till you, till you end up spending five hours trying to figure out WhyTF your wholly private VPC can't pull images from ECR. Oh yes, you got the NAT gateway set up, and can do pretty much everything  else (including pulling from DockerHub ), but not from ECR. Sigh. It's good to know that the quality of missing information in AWS Documentation remains as high as it used to be. P.S. The fix is easy - a VPCE.  But that's once you know the answer...