Self-supervised optical flow derotation network for rotation estimation of a spherical camera

Photometric Visual Gyroscope for Full-View Spherical Camera

“…VG algorithms consider two kinds of information, direct or indirect. An Indirect VG (IVG) leverages image features, either handcrafted as patches around image points [5,8] or learnt to estimate the optical flow [7]. Instead, a Direct VG (DVG) almost considers directly pixel brightness of the whole image as input of a 3D rotation optimization method [11].…”

“…Despite the absence of features, DVGs could recently reach very large estimation domains such as up to 360 deg in static environment [3]. But the rotation estimation error is still of the order of 1 deg (for real-time or close to real-time setups) whereas Neural Networks could recently reach 0.3 deg [7], though within a much tighter estimation domain of [−5; 5] deg around each axis. This error level is still problematic as it gets to be accumulated over time within 360 deg video stabilization [8] or visual odometry [12].…”

“…Hence, this article proposes a new DVG to improve the rotation estimation accuracy by directly considering pixel brightness of captured images lifted on a sphere, named PVG (Photometric Visual Gyroscope). PVG implements a new hybrid multi-resolution/sub-sampling scheme in two levels allowing to decrease significantly the best known estimation errors [7] together with a much larger convergence domain, as assessed by the evaluation made on many images available in the PanoraMIS dataset [1].…”