Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces

Abstract

Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this paper, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.

Pareto Optimal Layouts for Adaptive Mixed Reality

“…Commonly, these optimization-based adaptation approaches treat layout adaptation as a single-objective optimization problem, combining multiple objectives for multiple elements into a single cost to be minimized or a single utility score to be maximized and returning a single solution. Most frequently, this global criterion takes the form of a static weighted sum (e.g., [3,11,16,20,27,32,41]), which can be optimized using commercial solvers for linear programs (e.g., [11,27]) or using approximate techniques like simulated annealing (e.g., [3]). In contrast to these previous efforts, our approach treats layout adaptation as an online multi-objective optimization problem that returns a set of optimal adaptations.…”

Towards Flexible and Robust User Interface Adaptations With Multiple Objectives

“…Applic. Domain [5] A priori* Offline Text entry [6] A priori* Offline Text entry [14] A priori* Offline Text entry [24] A priori* Offline Text entry [40] A priori* Offline Text entry [44] A priori* Offline Text entry [45] A priori* Offline Text entry [50] A priori* Offline Text entry [2] A priori* Offline 2D UI layouts [20] A priori* Offline 2D UI layouts [21] A priori* Offline 2D UI layouts [3] A priori* Online 3D UI layouts [13] A priori* Online 3D UI layouts [18] A priori* Online 3D UI layouts [22] A priori* Online 3D UI layouts [31] A priori* Online 3D UI layouts [37] A priori* Online 3D UI layouts [49] A priori* Online 3D UI layouts [11] A posteriori Offline 3D touch interaction [27] A posteriori Offline Text entry [28] A posteriori Offline Physical input [29] A posteriori Offline Haptic feedback [43] A posteriori Offline 2D UI layouts [51] A posteriori…”

“…Pioneering work proposed novel objective functions and used existing optimization techniques to optimize keyboard layouts [10,30,52] (see [16] for an overview) and menu designs [1,25,32,34]. Since then, multi-objective optimization approaches have been applied to the design and adaptation of, for example, text entry methods (e.g., [6,14,24,40,44,45]) and 2D and 3D UI layouts (e.g., [13,18,22,31,37,47,49]). A key work in this area was the multiobjective optimization method proposed by Smith et al [44] which utilized a weighted sum-based scalarization with varying weights to generate Pareto optimal keyboard layouts.…”

Model-driven development for interactive surface prototypes

“…The hardware components were then used to define stereotypes (column 2) from which Device metaclass (column 1) was derived. Depth Sensor [7, 10, 16, 29, 31, 38-40, 42, 46] Infrared Sensor [27,36] Self-built Sensor [24] Imaging Device Projector [7,11,15,16,27,29,31,32,36,37,[39][40][41][42]46] Display [11,41] Actuator Device Actuator [15,27,32,36,40,42] Human Interface Device Mobile Device [16,24,29,31,32,37,40,42] Peripheral Device Phicon [36] Marker [24,27,32,47] Infrared lamp [27,36,37] Mirror [11,27,36] Audio Device Transducer [11]…”