Deep Learning and Video — Dealing with the Overhead
So you’ve got an AI thing that you’re doing with video. Fun, right? Probably involves cats. In fact, it’s probably something like “GANs to locate and identify Cats on TV shows”, and is going to end up being demoed at https://cats_on_friends.io
Thing is, as you’re going down this road, at some point you’re going to have to train your model, and that is going to involve a s**t-ton of video. Video that is, by definition, data-heavy — all those 30 minute episodes at 720p add up.
What’s worse is that you’re likely to randomly sample these video clips, a minute here, a minute there, and so on.
What’s worse is that you’re likely to randomly sample these video clips, a minute here, a minute there, and so on.
Video Codecs and CompressionWait, you know how video codecs works, right? They, effectively, store the whole screen every now and then, with all the stuff between these screenshots just being the data that changed between these screenshots. These screenshots are called keyframes or intra-frames, and when you want to work with a video segment at time 2:11, what you actually need is the keyframe at or before 2:11, so that you can actually recreate the screen at 2:11. (•)
On Noes! The Loads!So yeah, back to Cats On Friends. When you actually get around to the training part, you’ll discover that you’re
a) Storing vast amounts of data (all those video clips!), and
b) Moving all this data into your GPU’s memory every time you want to do anything with it.
What’s more, since you’re randomly sampling this data, you’ve got to seek through it to find the key-frames, which basically means Moar Work. Yeah, you could compress the video, but then you need to decompress it before you process it, which means Even Moar Work.
Ouch! You are now using a ridonculous amount of CPU and Memory, before you even begin training your model!
a) Storing vast amounts of data (all those video clips!), and
b) Moving all this data into your GPU’s memory every time you want to do anything with it.
What’s more, since you’re randomly sampling this data, you’ve got to seek through it to find the key-frames, which basically means Moar Work. Yeah, you could compress the video, but then you need to decompress it before you process it, which means Even Moar Work.
Ouch! You are now using a ridonculous amount of CPU and Memory, before you even begin training your model!
NVIDIA’s NVVL to the Rescue!Never fear though, it turns out that NVIDIA has you covered with their Video Loader (NVVL). It basically uses the built in decoding hardware on their GPUs to offload/accelerate this whole process, and will even do fun stuff to the frames like scaling, cropping, and whatnot.
It’s pretty damn efficient too, with CPU loads going down 40–50%, at data compressions of over 38x.
Mind you, it only works with PyTorch right now, unless you want to work with the low-level libraries directly. That said, I suspect it’ll end up in the other frameworks pretty damn soon — it’s really way too useful to be ignored!
So yeah, go check out NVVL!
So yeah, go check out NVVL!
Coda — KeyframesOh yeah, keyframes. Thing is, if you want any kind of acceptable (let alone good!) performance when working with random chunks of video, you’re going to need to re-encode the video. As NVIDIA puts it “to get good performance when randomly reading short sequences from a video file, it is necessary to encode the file with frequent key frames. We’ve found setting the keyframe interval to the length of the sequences you will be reading provides a good compromise between filesize and loading performance.”
Comments