Fused Twins: A Cognitive Approach to Augmented Reality Media Architecture

Abstract

a) Entering the augmented lobby in the Fused Twins. (b) Coming down the stairs into the augmented lobby.Fig. 1. The Fused Twins are used to augment the building's lobby with visualizations of sensors and agents through a HoloLens 2.Recent advances in Augmented Reality (AR), the Internet of Things (IoT), cloud computing, and Digital Twins transform the types, rates, and volume of information generated in buildings as well as the mediums through which they can be perceived by users. These advances push the standard approach of media architecture to embed screens in the built environment to its limits because screens lack the immersive capacity that newer media afford. To bridge this gap, we propose a novel AR approach to media architecture that uses a Digital Twin as a platform for structuring and accessing data from various sources, including IoT and simulations. Our technical contribution to media architecture is threefold. First, we extend the possibilities of media architecture beyond embedded screens to three dimensions by presenting a Digital Twin using AR with a head-mounted display. This approach results in a shared and consistent augmented experience across large architectural spaces. Second, we use the Digital Twin to integrate and visualize real physical sensor information. Third, we make artificial occupancy simulations accessible to everyday users by presenting them within their natural context in the Digital Twin. Observing the Digital Twin in situ of the Physical Twin also has applications beyond media architecture.Fusing the two twins using AR can reduce the cognitive load of users from consuming big and complex information sources and enhance their experience. We present two use cases of the proposed Fused Twins in a university building at ETH Zürich. In the first use case, we visualize a dense indoor sensor network (DISN) with 390 IoT sensors that collected data from March 2020 to May 2021. In the second use case, we immerse visitors in agent-based simulations to enable insights into the real and projected uses of space. This

The Hitchhiker’s Guide to Fused Twins: A Review of Access to Digital Twins In Situ in Smart Cities

“…Second, the benefit of the third dimension is difficult to grasp and even more difficult to implement. Most modern immersive analytics actually rely on embedding 2D displays into the real world [17,[99][100][101]. Visualisations in 3D are typically derived from the previous 2D media, but there is a trend towards exploring new possibilities.…”

“…DTs are similar to Ubiquitous Computing [46] because they attach computations to real-world objects, but without the FTs paradigm, DTs are ultimately not Ubiquitous Computing because the computations are detached from the embodiment of the represented object. Fusing the PT and DT [17,101] joins CPS properties with the PT to produce truly Ubiquitous Computing as envisioned by Weiser [46].…”

“…At best, DTs are remotely displayed in city administration offices to obtain insights [9,16]. However, DTs have the potential beyond administration to inform decision-making on all levels from citizens to politicians and planners [17]. To realise such a wide-ranging implementation of a DT for a Smart City, its data must be accessible in situ within its physical context.…”

“…We argue that there is a need to clarify the definitions of DTs, Situated Analytics, and Smart Cities with an emphasis on the visualisation and development of a practical interface. The FTs concept aims to embed the DT in its Physical Twin (PT), fusing them via Extended Reality (XR) applications [17]. Throughout this paper, we follow the working hypothesis that DTs combined with Situated Analytics enable lower cognitive load, enhanced cognition, and easier embodiment; and therefore, FTs can make Smart Cities more accessible to users ranging from managers and policy makers to citizens.…”

“…The white areas visualise the areas of observation and the coloured spheres encode the type of sensor that is located at the respective position in the building. (b) The FTs [17] are visualised in Mixed Reality (MR) [22] by embedding the DT in situ in its PT, thus fusing the two. The fused representation enables Situated Analytics [19] which facilitates the user's understanding of analysis through spatial cues of the environment.…”