Multi-view Data Capture using Edge-synchronised Mobiles

Abstract

Multi-view data capture permits free-viewpoint video (FVV) content creation. To this end, several users must capture video streams, calibrated in both time and pose, framing the same object/scene, from different viewpoints. New-generation network architectures (e.g. 5G) promise lower latency and larger bandwidth connections supported by powerful edge computing, properties that seem ideal for reliable FVV capture. We have explored this possibility, aiming to remove the need for bespoke synchronisation hardware when capturing a scene from multiple viewpoints, making it possible through off-the-shelf mobiles. We propose a novel and scalable data capture architecture that exploits edge resources to synchronise and harvest frame captures. We have designed an edge computing unit that supervises the relaying of timing triggers to and from multiple mobiles, in addition to synchronising frame harvesting. We empirically show the benefits of our edge computing unit by analysing latencies and show the quality of 3D reconstruction outputs against an alternative and popular centralised solution based on Unity3D.

An open-source mobile-based system for synchronised multi-view capture and dynamic object reconstruction

“…In industry, our system can promote digital creativity, enabling game development or the creation of social networks based on immersive videos [23]. For example, the results we presented in our prior works [7,8] show that our setup can be used as a 3D motion capture system with consumer mobiles and with a minimal hardware setup. Because our system relies on the SLAM system of Android ARCore [20], which is a continuously evolving Augmented Reality platform maintained by Google, we are confident that the mobile localisation accuracy will also improve with time, thus offering higherquality data captures.…”

“…Immersive computing is the next step in human interactions and the potential for immersive digital content is broad, impacting upon entertainment, advertising, gaming, mobile telepresence, and tourism [1][2][3][4][5][6][7][8]. But for immersive computing to become mainstream, systems to easily create large quantities of immersive content are needed.…”

“…In this paper, we present the first open-source mobile-based system to capture nearly-synchronous frame streams from multiple handheld mobiles and a software to reconstruct a dynamic object. A thorough description and assessment of all the modules and algorithms of our system can be found in our previous works [7,8]. To facilitate the use of our reconstruction algorithms we collected an outdoor dataset, namely 4DM (4DMobile), which involves an actor performing sports actions that we recorded using six handheld Augmented Reality mobiles (Fig.…”

Real-time image processing for augmented reality on mobile devices

“…In order to realize the three-dimensional reconstruction of dynamic objects, the paper entitled "Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobiles" [10] proposes a system that obtains almost synchronized frame streams from multiple mobile handheld devices. The system uses a decentralized trigger strategy and a data relay architecture to collect frames and movement gestures in real-time.…”

Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobiles

“…Dynamic objects recorded from multiple viewpoints can be digitally reconstructed to produce free-viewpoint videos (FVV), or 4D videos, which can then be experienced in Augmented or Virtual Reality [34,38,33]. Multi-view 3D/4D reconstruction and understanding of dynamic objects is challenging because it requires cameras to be calibrated in space and time [26,22,14,31,32,11]. The 4D reconstruction task in uncontrolled scenarios for real-time applications (e.g.…”