Keyboardless visual programming using voice, handwriting, and gesture

Abstract

Visual programming languages have facilitated the application development process, improving our ability to express programs, as well as our ability to view, edit, and interact with them. Yet even in visual programming environments, productivity is often restricted by the primary input sources: the mouse and the keyboard. As an alternative, we investigate a program development interface which responds to the most natural human communication technologies: voice, handwriting, and gesture. Speech-and penbased systems have yet to find broad acceptance in everyday life because they are insufficiently advantageous to overcome problems with reliability. However, we believe that a visual programming environment with a multimodal user interface properly constrained so as not to exceed the limits of the current technology has the potential to increase programming productivity for not only those people who are manually or visually impaired, but for the general population as well. In this paper we report on such a system.

End-User Service Computing: Spreadsheets as a Service Composition Tool

“…The spreadsheet paradigm, however, includes not only commercial spreadsheet systems such as MS Excel and OppenOffice.org CALC but also a number of research languages that extend the paradigm with features such as gestural formula specification [9], [33], graphical types [57], visual matrix manipulation [3], [55], high-quality visualizations of complex data [12], and specification of graphical user interfaces (GUIs) [35]. Most of the current research on spreadsheets is focused on reducing the errors introduced by users [7], [19], [53].…”

Scaling up a "What you see is what you test" methodology to spreadsheet grids

“…fact that the user has no control over which cell is scheduled last for execution. In the execution tracking task, Rij.Trace would accumulate the formula graph nodes that were covered by the most recent evaluation of any cell in region R; yet, in the validation task, the Rij.Trace information would treat this information as representative of the most recent evaluation of every cell in region R. For example, if the last cell to execute in Figure 5 were P [1,1], then validating any cell in P would not record coverage of du-association (7,12), even though it executed, because the stored trace would be the one reflecting P[1,1]'s execution of (6,11). Hence, depending upon the evaluation's scheduling strategies, some du-associations that are feasible may never be included in a validation.…”

“…The second possibility has more than one problem, but the worst is that even infeasible du-associations could be validated. Here, the trace stored for P's single region would be {10,11,12} and for M's would be {5,6,7}, implying that (6,11), (7,11), (6,12), and (7,12) have been covered. Yet two of these- (7,11) and (6,12)-are infeasible, because j cannot be both less and greater than 3.…”

“…Here, the trace stored for P's single region would be {10,11,12} and for M's would be {5,6,7}, implying that (6,11), (7,11), (6,12), and (7,12) have been covered. Yet two of these- (7,11) and (6,12)-are infeasible, because j cannot be both less and greater than 3. Hence, it is not possible in either case to share execution traces at the granularity of regions while still maintaining du-adequacy as the test criterion.…”

“…The spreadsheet paradigm includes not only commercial spreadsheet systems, but also a number of research languages that extend the paradigm with explicitly visual features, such as support for gestural formula specification [1,7], graphical types [1,17], visual matrix manipulation [14], high-quality visualizations of complex data [2], and specifying GUIs [8]. In this paper, we use the term spreadsheet languages to describe all such systems following the spreadsheet paradigm.…”