Lie Group Transformation Models for Predictive Video Coding

Abstract

We propose a new method for modeling the temporal correlation in videos, based on local transforms realized by Lie group operators. A large class of transforms can be theoretically described by these operators; however, we propose to learn from natural movies a subset of transforms that are statistically relevant for video representation. The proposed transformation modeling is further exploited to remove inter-view redundancy, i.e., as the prediction step of video encoding. Since the Lie group transformation coefficients are continuous, a quantization step is necessary for each transform. Therefore, we derive theoretical bounds on the distortion due to coefficient quantization. The experimental results demonstrate that the new prediction method with learned transforms leads to better rate-distortion performance at higher bit-rates, and competitive performance at lower bit-rates, compared to the standard prediction based on block-based motion estimation.

An Unsupervised Algorithm For Learning Lie Group Transformations

“…For this purpose the additional cost of encoding the transformation coefficients (µ and σ) needs to be accounted for and weighed against the gains in PSNR of the predicted frame. This tradeoff between encoding and reconstruction cost is explored, and a rate-distortion analysis performed, in a separate paper [Wang et al, 2011].…”

“…Unlike previous Lie group implementations, we demonstrate an ability to work simultaneously with multiple transformations and large inter-frame differences during both inference and learning. In another paper we additionally show the utility of this approach for video compression [Wang et al, 2011].…”

Symmetry-adapted representation learning

“…The more general problem of learning symmetries has been previously approached as estimating the infinitesimal generators of Lie groups generating data transformations [41][42][43][44] . Symmetries in learning have been used in the context of categorizing symmetry groups (mirror, roto-translation) in random patterns [45] .…”

Transformation Properties of Learned Visual Representations

“…A generative neural network model maps each transformed latent representation to an image. Unlike previous work on learning group representations (Rao & Ruderman, 1999;Miao & Rao, 2007;Sohl-Dickstein et al, 2010;Wang et al, 2011;Cohen & Welling, 2014), our model does not assume a linear action of the group in the input space, but instead acts linearly on a latent representation of the 3D scene. Furthermore, our model is the first learned Lie group model that can properly deal with non-commutative transformations.…”