Residual flows [1, 2] are a special class of normalizing flows whose building blocks are residual layers of the form $ x \to x + g_\theta(x) $. In this post, we’ll discuss the math behind residual flows, how they’re implemented in practice, and their connection to flow-based generative models.

Convolutional neural networks (CNN’s) are ubiquitous in machine learning today. This popularity stems from one remarkable feature of CNN’s: translational equivariance. Basically, this means that if I first translate an image and then pass it through a CNN, I get the same result as if I had first passed it through the CNN, and then translated it. It would be a mistake to conclude that this is the whole picture. Concealed by the familiar scenery of flat, two-dimensional, pixelated images lies a siginficantly more general framework, of which “conventional” CNN’s are only a special case. Consider:

I was poking around the PyTorch internals a few weeks ago, and thought it’d be fun to write my own (very simplified) deep learning framework ¹. This is the first in what I hope will be a several posts documenting my progress on the project, and thoughts along the way. Coral can be found on GitHub here.