Rigid 3-D motion estimation using neural networks and initially estimated 2-D motion data

Abstract

The present paper extends a known e cient t e c hnique for rigid 3D motion estimation so as to make it applicable to motion estimation problems occuring in image sequence coding applications. The known technique estimates 3D motion using previously evaluated 3D correspondence. However, in image sequence coding applications 3D correspondence is unknown and only 2D motion vectors are usually initially available. The novel neural network (NN) introduced in this paper uses initially estimated 2D motion vectors to estimate 3D rigid motion and is therefore suitable for image sequence coding applications. Moreover, it is shown that the NN introduced in this paper performs extremely well even in cases where 3D correspondence is known with accuracy. Experimental results are presented for the evaluation of the proposed scheme. I IntroductionObject based coding has long attracted considerable attention as a promising alternative to block-based encoding, achieving excellent performance, and producing fewer blocking artifacts than those commonly occuring in block-based hybrid DCT coders at moderate and low bit rates 1, 2]. In addition, the ability o f object-based coding techniques to describe a scene in a structural way, i n c o n trast to traditional waveformbased coding techniques, opens new areas of applications 3]. Video production, realistic computer graphics, multimedia interfaces and medical visualization are some of the applications that may b e n e t by exploiting the potential of object based schemes.Object based codecs consist of an analysis part in which image pixels are grouped into objects described by their shape (2D silhouette and 3D structure), 3D motion and texture. Objects are then synthesized at the decoder using the transmitted parameters 1].In all model-based and object-based image sequence coding schemes, motion estimation and motion This work was partly supported by the EU CEC ACTS projects VIDAS and PANORAMA.

Rigid and non-rigid 3D motion estimation from multiview image sequences

“…In this case, initial 2D vectors are available at the projections of the 3D nodes on all the image planes of a multiview camera geometry. This is seen to improve significantly the results in [1], in all examined cases, even in the presence of measurement noise. The rigid 3D motion of each articulated object in the scene, is estimated using a neural network based on the available 2D motion information on the image planes of the multiview camera geometry.…”

“…The initial 2D motion field was obtained by a simple block matching motion estimation between the consecutive frames. The technique in [1] was seen to improve the 3D motion estimates of [22,23], even in cases where 3D correspondences were known with accuracy.…”

“…Depth understanding is an important element of enhanced perception and tele-presence in image communication [1][2][3]. A direct way of inferring the depth information is provided by stereo and multi-ocular vision [4,5].…”

“…In [1] a neural network approach was introduced for estimating the 3D motion parameters of the rigid 3D scene objects, from monoscopic image sequences using initial 2D motion vectors on the camera image plane. The authors adapted the results of [22] for the solution of coding-oriented motion estimation problems, where only 2D motion vectors rather than 3D correspondences are initially available.…”

“…The weights of the neural network are updated using nonlinear error minimization techniques. The technique in [1] is also extended by developing a novel approach for flexible 3D motion estimation of each node of the object model. The performance of the rigid and non-rigid 3D motion estimation techniques is evaluated experimentally on both synthetic and real 3D object motion, assuming stereo and trinocular camera setups.…”