In 2007, a very nice implementation of a variational implementation of optical flow was described in: A Duality Based Approach for Realtime TV-L1 Optical Flow by C. Zach et. al. I won’t get into the details too much, but the formulation is described by this equation:

E = int_?{?|I0(x) ? I1(x + u(x))| + |?u| dx}

If you are familiar with the seminal work of Horn and Schunck, you will notice it is fairly similar to their variational formulation:

min_u{ int_?{ (|?u1|^2 + |?u2|^2) d? } + ? int_?{ ((I1(x + u(x)) ? I0(x))^2) d? }

And although it looks incredibly simple now, it is in fact fairly difficult computationally since now both terms are not continuously differentiable. To overcome this difficulty, they follow the work of Rudin-Osher-Fatemi energy for total variation image denoising.

Another big contribution comes in their implementation on the GPU. By linearization of the generally non-convex energy functional shown above, the problem is reduced to a pixel-wise convex energy minimization problem. Additionally, by employing coarse-to-fine image pyramids, they are able to account for both small and large movements. Luckily, graphics cards are great at doing both of these sorts of computations very quickly. You … Continue reading...