Real-Time Diminished Reality for Dynamic Scenes

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

“…Various types of images are used as follows: Internet photos [13], X-ray images [8,24], image sets actively [25,26], streaming image sequences from surveillance cameras [27], multi-view cameras [6,[28][29][30], and RGB-D cameras [31,32]. One could store a sufficient number of viewpoints when one carefully captures a scene during enough time [13,25,26].…”

“…However, one will suffer from the geometric or photometric differences between the real and virtual scenes in the DR results due to the intervals between the preliminary observation and the current DR experience. In the case of real-time observation that can handle dynamic backgrounds [6,27,[29][30][31][32], because the number of installed cameras is physically limited, the number of viewpoints is sometimes insufficient. In addition, temporal inconsistency occurs when camera synchronization is not performed properly.…”

“…Most DR methods require the camera pose or the relative pose of the camera to the scene as an input to recover the hidden background in the current view. Positioning sensors [33,34], visual simultaneous localization and mapping (vSLAM) methods [14,23,27,35,36,43], pre-calibration [6,37], and fidicial markers [31,38] are available. 3.…”

“…In see-through, image-based methods are preferred for seamless real-virtual composition. Homography transformations of images [13,19,35,36,41,42], rendering texture 3D models [6,31,43], rendering voxels [37], and image-based rendering (IBR) [25,26,30,44] are the major approaches. In inpaint, pixels or image patches are synthesized in the ROI to fill in the region with plausible pixels [9,10,21,38].…”

“…However, multi-view based 3D reconstruction is time-consuming. Thus, Meerits and Saito used an RGB-D observer camera for fast 3D reconstruction of hidden backgrounds as 3D polygon mesh in real time [31]. Ienaga et al reported an example implementation of a multiple RGB-D camera system to remove the viewer's body in an AR-based mirror system to improve the efficiency of teaching anatomy [32].…”