Integrating Mixed Reality and Internet of Things as an Assistive Technology for Elderly People Living in a Smart Home

Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review

“…An interaction device is a platform that visualizes AR content to the agent and it may also receive input from the agent. The interaction device can have many forms, such as a mobile device (e.g., smartphone, tablet PC) [10,11,[33][34][35][36][37]39,40,42,44,46,48,[50][51][52], HMD [12,38,41,45,49], PC/TV monitor [41,43,47], and projector [41].…”

“…The agent can obtain detailed information regarding the IoT devices that the application recognizes by touching the graphs. As an example of fourth interaction, four out of five studies [12,41,45,49] that used an HMD as an interaction device utilized it to identify the hand gestures of an agent. The exception was a study by Muthanna et al [46], who used a mobile device instead of an HMD to detect the agent's hand positions to display the data that were collected from IoT devices on the interaction device when the agent's hands were placed at a specific location.…”

“…The first type of commonly used interaction is that in which another entity is used as an identifier for AR. This can be implemented with either a printed AR marker [34,37,39,40], image/object recognition [38,41,42,47,49], or spatial data [45]. Another type of interaction is the manipulation of another entity by the agent.…”

“…In our literature search, we found 10 studies involving interactions between agents and other entities. Of these studies, only three used spatial data of a certain place to provide an indoor navigation feature [34,45,49]. Moreover, only one study used other entities (i.e., crops in soil) as either objects to monitor conditions (i.e., the location of crops, soil moisture, temperature, water level, nutrients, and luminance) or objects that can be managed by the agent with physical contact [47].…”

“…In order to visualize AR content, an agent moves the camera of an interaction device to locate an identifier within the camera view. The identifier can be either printed images used as AR markers [10,[34][35][36]39,40,48], physical objects that can be recognized using object/image recognition [11,12,33,34,37,38,41,42,44,46,[49][50][51][52], or environments for using spatial data [39,45]. The second identified input source is the camera view for capturing an agent's hands.…”