MR360: Mixed Reality Rendering for 360° Panoramic Videos

Rhee, Taehyun; Petikam, Lohit; Allen, Benjamin; Chalmers, Andrew

doi:10.1109/tvcg.2017.2657178

Cited by 99 publications

(79 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, we will extend our system to include more visual effects, such as shadow rendering based on point-light detection [20] and camera effects as proposed by Murray and Klein [18] [12]. These features will supplement our work, building a toolbox for photo-realistic MR applications.…”

Section: Discussionmentioning

confidence: 94%

Learning Lightprobes for Mixed Reality Illumination

Mandl

Mohr

et al. 2017

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

View full text Add to dashboard Cite

Figure 1: Coherent Illumination. The real scene consists of an action figure of The Hulk and a toy car. To demonstrate the result of our system, we place a 3D scan next to the real action figure and display it using Mixed Reality: (left) The 3D reconstruction is rendered without real world light estimation. (right) Our system estimates the current lighting from a single input image. The estimated lighting is used to illuminate the 3D reconstruction. Note that we only register the real world lighting and do not consider any camera effects such as exposure or blur. ABSTRACTThis paper presents the first photometric registration pipeline for Mixed Reality based on high quality illumination estimation by convolutional neural network (CNN) methods. For easy adaptation and deployment of the system, we train the CNN using purely synthetic images and apply them to real image data. To keep the pipeline accurate and efficient, we propose to fuse the light estimation results from multiple CNN instances, and we show an approach for caching estimates over time. For optimal performance, we furthermore explore multiple strategies for the CNN training. Experimental results show that the proposed method yields highly accurate estimates for photo-realistic augmentations.

show abstract

Section: Discussionmentioning

confidence: 94%

Learning Lightprobes for Mixed Reality Illumination

Mandl

Mohr

et al. 2017

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

View full text Add to dashboard Cite

show abstract

“…This was achieved by re-using 3D datasets with both synthetic and real-world scanned indoors scenes and synthesizing a 360 o dataset via rendering. 360 o depth information can be useful for a variety of tasks, like in adding automation in the composition of 3D elements within spherical content [72].…”

Section: Discussionmentioning

confidence: 99%

OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas

Zioulis

Karakottas

Zarpalas

et al. 2018

Lecture Notes in Computer Science

169

251

View full text Add to dashboard Cite

Recent work on depth estimation up to now has only focused on projective images ignoring 360 o content which is now increasingly and more easily produced. We show that monocular depth estimation models trained on traditional images produce sub-optimal results on omnidirectional images, showcasing the need for training directly on 360 o datasets, which however, are hard to acquire. In this work, we circumvent the challenges associated with acquiring high quality 360 o datasets with ground truth depth annotations, by re-using recently released large scale 3D datasets and re-purposing them to 360 o via rendering. This dataset, which is considerably larger than similar projective datasets, is publicly offered to the community to enable future research in this direction. We use this dataset to learn in an end-to-end fashion the task of depth estimation from 360 o images. We show promising results in our synthesized data as well as in unseen realistic images.

show abstract

“…The proposed method is suitable for use in a wide range of applications, including image‐based measurements of reflectance [GGG*16]; in rendering, to provide accurate input data for spectral pre‐filtering and spectral upsampling of input textures; additional applications include characterization of HDR imaging [KK08], to provide more accurate illumination estimation [Deb98, LYL*16], and in augmented and mixed reality applications [RPAC17]. Industrial applications include the use as an analysis tool to compare the raw performance of commercial cameras [MPVC03]; in colorimetry, to perform colour characterization of displays and projectors, and, in general, the absolute measurement of non‐homogeneous samples, given the possibility of simultaneously measuring the whole visual field.…”

Section: Discussionmentioning

confidence: 99%

Turning a Digital Camera into an Absolute 2D Tele‐Colorimeter

Guarnera

Bianco

Schettini

2018

Computer Graphics Forum

View full text Add to dashboard Cite

We present a simple and effective technique for absolute colorimetric camera characterization, invariant to changes in exposure/aperture and scene irradiance, suitable in a wide range of applications including image‐based reflectance measurements, spectral pre‐filtering and spectral upsampling for rendering, to improve colour accuracy in high dynamic range imaging. Our method requires a limited number of acquisitions, an off‐the‐shelf target and a commonly available projector, used as a controllable light source, other than the reflected radiance to be known. The characterized camera can be effectively used as a 2D tele‐colorimeter, providing the user with an accurate estimate of the distribution of luminance and chromaticity in a scene, without requiring explicit knowledge of the incident lighting power spectra. We validate the approach by comparing our estimated absolute tristimulus values (XYZ data in cd/m2) with the measurements of a professional 2D tele‐colorimeter, for a set of scenes with complex geometry, spatially varying reflectance and light sources with very different spectral power distribution.

show abstract

MR360: Mixed Reality Rendering for 360° Panoramic Videos

Cited by 99 publications

References 35 publications

Learning Lightprobes for Mixed Reality Illumination

Learning Lightprobes for Mixed Reality Illumination

OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas

Turning a Digital Camera into an Absolute 2D Tele‐Colorimeter

Contact Info

Product

Resources

About