Context-Aware Multimodal Human–Computer Interaction

Design Issues for Pen-Centric Interactive Maps

“…The research in ICIS/CHIM has explored different input modalities [22], including hand/arm gestures [53], speech [17,52], facial expressions [17], or pen gestures and handwriting [70] to communicate with the system. Recognition and interpretation of these various forms of user input is performed by specialized modules, using pattern recognition techniques to transform raw data inputs (like speech signals, pen input trajectories, or hand movements) to a ranked list of potential classification results.…”

“…It requires specialized skills and knowledge from a multi-disciplinary research team, involving significant collaborative software engineering efforts to Fig. 1 Different modalities involved in multimodal interaction in the ICIS project [22,70]. Input modalities comprise pen, hand gestures, speech, and facial expressions.…”

“…In the remainder of this chapter, the reader is provided with a worked-out case study of the design of a specific component of the pen gesture recognition module from the ICIS/CHIM multimodal system [22]. Our approach tries to bridge the gap between allowing unconstrained user inputs and limiting the user through constrained gesture repertoires.…”

“…Each feature is computed on the complete gesture trajectory (see Figure 9). The g-28 set contains the following 28 features (for details, the reader is referred to [71][72][73]): (1) length of pen stream, (2) area of convex hull, (3) compactness, (4) eccentricity, (5) ratio of the coordinate axes, (6) closure, (7) circular variance, (8) curvature, (9) average curvature, (10) perpendicularity, (11) average perpendicularity, (12) centroid offset along major axes, (13) length along first principle axis, (14) rectangularity, (15) initial horizontal offset, ( 16) final horizontal offset, (17) initial vertical offset, (18) final vertical offset, (19) straight line count, (20) largest straight line ratio, (21) straight line ratio, (22) largest straight line, ( 23) macro perpendicularity, (24) average macro perpendicularity, and ( 25) ratio of the principal axes. Also included are three features based on the pressure data, which is available in online data: (26) average pressure, ( 27) pendown count, and ( 28) penup/down ratio.…”