We learn to compute optical flow by combining a classical spatial-pyramidformulation with deep learning. This estimates large motions in acoarse-to-fine approach by warping one image of a pair at each pyramid level bythe current flow estimate and computing an update to the flow. Instead of thestandard minimization of an objective function at each pyramid level, we trainone deep network per level to compute the flow update. Unlike the recentFlowNet approach, the networks do not need to deal with large motions; theseare dealt with by the pyramid. This has several advantages. First, our SpatialPyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in termsof model parameters. This makes it more efficient and appropriate for embeddedapplications. Second, since the flow at each pyramid level is small (< 1pixel), a convolutional approach applied to pairs of warped images isappropriate. Third, unlike FlowNet, the learned convolution filters appearsimilar to classical spatio-temporal filters, giving insight into the methodand how to improve it. Our results are more accurate than FlowNet on moststandard benchmarks, suggesting a new direction of combining classical flowmethods with deep learning.