Raise the MiniBatch Size!
/via https://xkcd-excuse.com/ “ Training takes time ” ← That’s just the way it is, and the way it’s always been. The thing is, it’s not just the time you spend waiting for that bloody model to train, it’s the knock-on effects thereof. It is exceedingly common to spin off multiple workflows at a time, each with it’s own tweak, and that leads to any amount of chaos when it comes to keeping track of what exactly changed. Reproducibility is, well, a bit of a PITA in this field . In fact, you can even get tooling just to work around the chaos of tracking all these pipelined workflows . Ouch! The underlying issue here is that typically training is done in one of two ways 1) Throw everything at a single server/GPU (what you do if that’s what you’ve got) 2) Throw it at a server farm (AWS, GCE, whatever) with their concomitant GPUs/TPUs/whatever Take ResNet-50 . You’d typically break the images up into “minibatches” of 256 images each, and then have...