SketchSPORE: A Sketch Based Domain Separation and Recognition System for Interactive Interfaces

Abstract

Multimodal interfaces are used to interact with devices and automata using different channels of communication. In this context, the sketch modality plays a key role since it allows users to convey concepts and/or commands using freehand drawing (graphical domain) and/or handwriting (textual domain). The acquisition of the sketch modality can be performed using touch (e.g., touchscreen) or touchless (e.g., RGB-D camera) tools supporting the development of versatile and powerful interactive interfaces. Domain separation and sketch recognition are two fundamental issues of these interfaces. This paper presents SketchSPORE a novel framework designed both to automatically distinguish graphical from textual elements within the same sketch and to recognize freehand drawing as well as handwriting. The recognition processes support both on-line and off-line modes, moreover their processing can be suitably stored within an XML file to provide a means to maintain the compatibility between the framework and service and/or application targets. Extensive experiments showing the effectiveness of the proposed method are reported and discussed.

Crosside: A Cross-Surface Collaboration by Sketching Design Space

“…Sketching covers many domains of human activity and inside these domains, there are several works exploiting sketching for one or many sub-activities such as for example: problem analysis in general [44], computer science [17] (e.g., user experience support [8], [18], user interface design, prototyping, and recognition [23], [13], [14], [39], [3], [2], [47], cross-device UI design [25], user-centered design in agile projects [32], [7], system walkthrough [24]), system development (e.g., QUILL [16] for model-based design of web applications), flexible modelling [40] (e.g., FlexiSketch [54], [55] for model sketching), RAPIDO [35] for web API development, sketching UML models (e.g., TAHUTI for sketching UML Class diagrams [19] and SketchML for various UML diagrams [3]), distributed software design [29], [33], task modelling [41], notation creation [55]), computer-supported collaborative work [21] (e.g., stakeholders' meetings [20], collocated tables for meetings [51] and interactive design spaces [6]), product and service design (e.g., sketching in design [21], extreme designing [15], industrial design [52], shape-changing products [42]), public displays [36], learning (e.g., classroom design studio [26], teaching geometry [11]), ideation [4] and concept generation…”

“…The word "gambit" is used in chess in reference to a movement in which the player sacrifices a piece, usually a pawn, in exchange to some future advantage; in user interface design the gambit would be made by the designer who sacrifices sketches in exchange to know in advance how the interface would function in the hands of the end-users, allowing her to learn from the eventual observation of this usage 2. Accessible at http://gambitsketch.appspot.com/…”

Crosside: A Cross-Surface Collaboration by Sketching Design Space

“…Sketching covers many domains of human activity and inside these domains, there are several works exploiting some form sketching for one or many sub-activities such as for example: problem analysis in general [13], computer science [2] (e.g., user experience support [14,15], user interface design, prototyping, and recognition [16,17,18,19,20,21,22], cross-device UI design [4], user-centered design in agile projects [23,24], system walkthrough [25]), system development (e.g., QUILL [26] for model-based design of web applications), flexible modelling [27] (e.g., FlexiSketch [28,29] for model sketching), RAPIDO [30] for web API development, sketching UML models (e.g., TAHUTI for sketching UML Class diagrams [31] and SketchML for various UML diagrams [20]), distributed software design [32,33], task modelling [34], notation creation [29]), computer-supported collaborative work [35] (e.g., stakeholders' meetings [36], collocated tables for meetings [37] and interactive design spaces [38]), product and service design (e.g., sketching in design [35], extreme designing [39], industrial design [5], shape-changing products [40]), public displays [41], learning (e.g., classroom design studio [42], teaching geometry [43]), ideation [44] and concept generation …”