Steered Mixture-of-Experts Approximation of Spherical Image Data

Abstract

Steered Mixture-of-Experts (SMoE) is a novel framework for approximating multidimensional image modalities. Our goal is to provide full Six Degrees-of-Freedom capabilities for camera captured content. Previous research concerned only limited translational movement for which the 4D light field representation is sufficient. However, our goal is to arrive at a representation that allows for unlimited translational-rotational freedom, i.e. our goal is to approximate the full 5D plenoptic function. Until now, SMoE was only applied on Euclidean spaces. However, the plenoptic function contains two spherical coordinate dimensions. In this paper, we propose a methodology to extend the SMoE framework to spherical dimensions. Furthermore, we propose a method to reduce the parameter space to the same two dimensional Euclidean space as for planar 2D images by using a projection of the covariance matrices onto tangent spaces perpendicular to the unit sphere. Finally, we propose a novel training technique for spherical dimensions based on these observations. Experiments performed on omnidirectional 360°images show that the introduction of the dimensionalityreduction projection step results in very low quality loss.

Steered mixture-of-experts for light field video coding

“…Evidently, SMoE has proven its potential by presenting beneficial properties for the distribution of 6DoF visual content compared to traditional image coding methods. [7][8][9][10][11] For instance, in the case of rendering it has three important properties. Firstly, view-rendering is very lightweight and pixels are coded independent from one another.…”