3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review

Abstract

Handheld mobile devices referring to mobile phones and tablets become the major output medium for augmented reality (AR), which have seen significant growth in popularity and usage among the public due to the growing release of consumer-oriented communication products nowadays, especially touchscreen smartphones. Unlike traditional desktop or tabletop AR (large display-based) and head-mounted display-based AR systems, small display-based AR (handheld mobile display) requires different interaction techniques that mostly utilize single-hand interaction as well as the limitation of small screen display and limited activity time due to the battery operation hour of handheld mobile devices. However, in handheld mobile AR, research is still lacking, especially research that focuses on 3D interaction in handheld mobile AR for virtual object manipulation. Thus, this paper provides an overview of 3D interaction techniques in handheld mobile AR with critical analysis. First, we describe three main interaction technique categories that applicable in handheld mobile AR, which is touch-based interaction, mid-air gestures-based interaction, and device-based interaction techniques, of their basic concepts on 3D object manipulation. Then, we classify and systematize the highlighted techniques and discuss the advantages and drawbacks of each. Previous studies for widely used techniques have been studied comprehensively. We then draw up a comparison among the different techniques based on the important elements considered in handheld mobile AR. The aim of this paper is to provide researchers with background information on AR and those who are interested in the field of 3D interaction, realizing each technique category. INDEX TERMS 3D object manipulation, device-based interaction technique, mid-air gestures-based interaction technique, handheld mobile AR, touch-based interaction technique. I. INTRODUCTIONAccording to Milgram and Kishino [1], augmented reality (AR) refers to all cases in which the display of an otherwise real environment is augmented by means of virtual (computer graphic) objects. Extended from this definition, Azuma [2] defined that AR is the variation of virtual environments or virtual reality; however, unlike virtual reality that immerses a user inside a synthetic environment and the user cannot see the real world around him/her, AR allows the user to see the real world, with virtual objects composited with the real world. Beyond that, some researchers have doneThe associate editor coordinating the review of this manuscript and approving it for publication was Waleed Alsabhan. a number of surveys or reviews about AR. For example, Krevelen and Poelman [3], drawing up conclusions from several previous works, defined the AR system as: 1) combining real and virtual objects in a real environment; 2) registering real and virtual objects with each other and 3) running interactively, in 3D and in real-time. From another point of view, Mekni and Lemieux [4] proposed to define AR as systems that have the following charact...

Interaction with Virtual Content using Augmented Reality

“…Another important component of any AR guidance system is the type of interaction used to complete the assembly procedures. Interaction can be provided through: touch gestures, device movement, mid-air gestures, controllers, gloves or tangible mechanisms [20].…”

“…The usefulness of AR is directly related to the user interfaces and interaction methods AR can provide. In this context, the study of innovative interfaces and interaction methods is of utmost importance since it directly affects the effectiveness and efficiency of users [20,21,27]. Currently, it is possible to identify at least six different categories of AR-based interfaces: traditional 3D user interfaces, natural user interfaces, tangible AR interfaces, emerging multimodal AR interfaces, collaborative AR interfaces and finally hybrid AR interfaces [6,27,50].…”

Wand-Like Interaction with a Hand-Held Tablet Device—A study on Selection and Pose Manipulation Techniques

“…There are three classes of pose manipulation techniques for hand-held AR: on-screen multi-touch gestures, mid-air wand or gesture action, and device-based interaction. For an overview of techniques and their characteristics, we refer to a comprehensive review [21] by Goh et al Pose manipulation in hand-held AR, with touch-enabled devices, is commonly done using multi-touch on-screen gestures, see for example [7,8,22,23]. While the multi-touch interaction is a useful technique, it has some issues.…”

Design of Interactions for Handheld Augmented Reality Devices Using Wearable Smart Textiles: Findings from a User Elicitation Study

“…In addition, the various built-in sensors in the handheld devices-such as cameras, GPS, compass, accelerometers, and gyroscope-enable to precisely determine the position and orientation of the device in the real world (e.g., [8][9][10]). Furthermore, the device's camera is used to naturally capture the user's mid-air hand movements while holding the device [11,12].Like in HMD AR, manipulations such as selecting and moving virtual 3D information are primary interactions in HAR devices [13]. The existing HAR interaction methods, such as touch input, offer promising solutions to manipulate virtual content (e.g., [14]).…”

“…Like in HMD AR, manipulations such as selecting and moving virtual 3D information are primary interactions in HAR devices [13]. The existing HAR interaction methods, such as touch input, offer promising solutions to manipulate virtual content (e.g., [14]).…”