And you are here why?

/via https://xkcd-excuse.com/ “ Training takes time ” ← That’s just the way it is, and the way it’s always been. The thing is, it’s not just the time you spend waiting for that bloody model to train, it’s the knock-on effects thereof. It is exceedingly common to spin off multiple workflows at a time, each with it’s own tweak, and that leads to any amount of chaos when it comes to keeping track of what exactly changed.  Reproducibility is, well, a bit of a PITA in this field   . In fact,  you can even get tooling just to work around the chaos of tracking all these pipelined workflows . Ouch! The underlying issue here is that typically training is done in one of two ways 1)  Throw everything at a single server/GPU (what you do if that’s what you’ve got) 2)  Throw it at a server farm (AWS, GCE, whatever) with their concomitant GPUs/TPUs/whatever Take  ResNet-50 . You’d typically break the images up into “minibatches” of 256 images each, and then have...

/via https://xkcd-excuse.com/ If you’ve built out anything with Deep Learning, you pretty much know how the drill goes right? You 1.  Start off with a basic workflow. Or, alternatively, copy over a workflow from something you did previously, and mess with it for a while before you realize that it doesn’t quite match what you’re doing, throw it away, and then, well,  Start off with a basic workflow  . 2.  Start tweaking the workflow . There’s a step over  here  that takes way too long and needs to be optimized, there’s a step over  there  where you need to change a parameter, and so on. 3.  Wait a  lot . Seriously, that’s pretty much 99% of what you’re doing. Waiting. - 3-  is really quite the killer. Every time you tweak the workflow,  every time , you run the whole damn thing again. Which is really quite ridiculous, because, really, the entire pipeline can take quite a while to run. It’s actually quite a bit wor...