Extending drag-and-drop to new interactive environments: A multi-display, multi-instrument and multi-user approach

Abstract

a b s t r a c tDrag-and-drop is probably one of the most successful and generic representations of direct manipulation in today's WIMP interfaces. At the same time, emerging new interactive environments such as distributed display environments or large display surface environments have revealed the need for an evolution of drag-and-drop to address new challenges. In this context, several extensions of drag-and-drop have been proposed over the past years. However, implementations for these extensions are difficult to reproduce, integrate and extend. This situation hampers the development or integration of advanced drag-and-drop techniques in applications.The aim of this paper is to propose a unifying implementation model of drag-and-drop and of its extensions. This model-called M-CIU-aims at facilitating the implementation of advanced drag-and-drop support by offering solutions to problems typical of new emerging environments. The model builds upon a synthesis of drag-and-drop implementations, an analysis of requirements for meeting new challenges and a dedicated interaction model based on instrumental interaction. By using this model, a programmer will be able to implement advanced drag-and-drop supporting (1) multi-display environments, (2) large display surfaces and (3) multi-user systems. Furthermore by unifying the implementation of all existing drag-and-drop approaches, this model also provides flexibility by allowing users (or applications) to select the most appropriate drag-and-drop technique depending on the context of use. For example, a user might prefer to use pick-and-drop when interacting with multiple displays attached to multiple computers, push-and-throw or drag-and-throw when interacting with large displays and possibly standard drag-and-drop in a more traditional context. Finally, in order to illustrate the various benefits of this model, we provide an API called PoIP which is a Java-based implementation of the model that can be used with most Java-based applications. We also describe Orchis, an interactive graphical application used to share bookmarks and that uses PoIP to implement distributed drag-and-drop like interactions.

Cross-device gaze-supported point-to-point content transfer

“…PhoneTouch enabled the precise selection and manipulation of content using mobile sensing to issue actions, and computer vision to detect a mobile's location on a rear-projected multitouch surface [Schmidt et al 2010]. In scenarios where close-proximity to a target is not achievable, drag-and-drop extensions have enabled out-of-reach targeting across multiple wall sized displays [Baudisch et al 2003;Collomb and Hascoët 2008].…”

“…Many methods are designed to support point-to-point transfer, allowing for the fine-grained positioning of acquired data [Baudisch et al 2003;Hinckley et al 2004;Collomb and Hascoët 2008]. However, there is currently no clear understanding as to the effect of switching from gaze to an alternate input modality on users' logical understanding and positioning success.…”

A cross-device drag-and-drop technique

“…Other work has explored techniques for settings in which the target is out of reach. This includes Drag-and-Pop and Drag-andPick, which have been investigated for use on wall-size displays but also demonstrated for interaction across multiple touchscreens [1,3]. Our technique exploits two-handedness in an asymmetric manner.…”

Improving modularity and usability of interactive systems with Malai

“…M-CIU is an instrument-based model that proposes to unify DnD interaction techniques [8]. Both VIGO and M-CIU do not separate the action, interaction and instrument concepts.…”