Environment-Aware Rendering and Interaction in Web-Based Augmented Reality

Abstract

This work presents a novel framework for web-based environment-aware rendering and interaction in augmented reality based on WebXR and three.js. It aims at accelerating the development of device-agnostic Augmented Reality (AR) applications. The solution allows for a realistic rendering of 3D elements, handles geometry occlusion, casts shadows of virtual objects onto real surfaces, and provides physics interaction with real-world objects. Unlike most existing state-of-the-art systems that are built to run on a specific hardware configuration, the proposed solution targets the web environment and is designed to work on a vast range of devices and configurations. Our solution can use monocular camera setups with depth data estimated by deep neural networks or, when available, use higher-quality depth sensors (e.g., LIDAR, structured light) that provide a more accurate perception of the environment. To ensure consistency in the rendering of the virtual scene a physically based rendering pipeline is used, in which physically correct attributes are associated with each 3D object, which, combined with lighting information captured by the device, enables the rendering of AR content matching the environment illumination. All these concepts are integrated and optimized into a pipeline capable of providing a fluid user experience even on middle-range devices. The solution is distributed as an open-source library that can be integrated into existing and new web-based AR projects. The proposed framework was evaluated and compared in terms of performance and visual features with two state-of-the-art alternatives.

MolecularWebXR: Multiuser discussions about chemistry and biology in immersive and inclusive VR

“…All developers of VR and AR headsets, from major players like Apple, Meta and Microsoft to smaller, highly specialized companies, have included web browsers that support the standard, ensuring that developments for the web should work out of the box in all of them. [15][16][17] From our side, next in the line is the release of a web app for immersive molecular simulations that will enable educational and research activities like those offered by moleculARweb's virtual modeling kits 19 but in multiuser, immersive VR that runs, like all of MolecularWebXR's content, on VR and non-VR capable devices. A growing universe of tools based on WebXR has the potential to transform the way how chemistry and biology, and science in general, are taught, learned, communicated and discussed, leaving no one behind.…”

“…15 WebXR-based solutions bring two advantages of unparalleled relevance to adoption and democratization of access to immersive content: (i) supreme portability across devices and operating systems, from laptops and smartphones to high-end AR/VR headsets; and (ii) zero barriers to deployment and availability as all content runs inside web browsers hence requires no installs or updates and is instantly available upon internet connection. [15][16][17] With this philosophy in mind, 3 years ago we released moleculARweb, a free platform that offers several activities and playgrounds for chemistry and structural biology education in commodity AR, that is AR that runs on non-specialized devices such as smartphones, computers, and laptops. [18][19][20] Next, building on the WebXR standard we extended moleculARweb with PDB2AR, a web tool that allows users to create web-based AR and VR sessions for any kind of consumer device.…”

Augmented Reality dengan Model Generate Target dalam Visualisasi Objek Digital pada Media Pembelajaran

“…Mobile augmented Reality (MAR) dipilih sebagai media proyeksi AR karena dianggap sebagai aplikasi yang inovatif, menjanjikan dan memiliki potensi untuk meningkatkan motivasi, partisipasi, dan peningkatan pengetahuan siswa [14] . Selain itu MAR merupakan alternatif dalam pengembangan AR karena perkembangan daya komputasi dari perangkat seluler mengalami peningkatan yang sangat signifikan selama beberapa tahun terakhir, sehingga menjadi pilihan utama dalam pengembangan produk AR [15]. Penerapan Model Target Generation bertujuan untuk memaksimalkan deteksi terhadap objek nyata sehingga informasi virtual dapat berperan sebagai informasi pendukung untuk meningkatkan transfer pengetahuan.…”

MolecularWebXR: Multiuser discussions about chemistry and biology in immersive and inclusive VR

“…15 WebXR-based solutions bring two advantages of unparalleled relevance to adoption and democratization of access to immersive content: (i) supreme portability across devices and operating systems, from laptops and smartphones to high-end AR/VR headsets; and (ii) zero barriers to deployment and availability as all content runs inside web browsers hence requires no installs or updates and is instantly available upon internet connection. [15][16][17] With this philosophy in mind, 3 years ago we released moleculARweb, a free platform that offers several activities and playgrounds for chemistry and structural biology education in commodity AR, that is AR that runs on non-specialized devices such as smartphones, computers, and laptops. [18][19][20] Next, building on the WebXR standard we extended moleculARweb with PDB2AR, a web tool that allows users to create web-based AR and VR sessions for any kind of consumer device.…”

“…16 WebXR-based solutions bring two advantages of unparalleled relevance to adoption and democratization of access to immersive content: (i) high portability across devices and operating systems, from laptops and smartphones to high-end AR/VR headsets; and (ii) low barriers to deployment and availability as all content runs inside web browsers hence requires no installs or updates and is instantly available upon internet connection. [16][17][18] Plus, by not requiring any login or registration, the website is used under full privacy (and no files loaded by the users are stored in our server).…”