A Speech-Centric Perspective for Human-Computer Interface: A Case Study

Abstract

Speech technology has been playing a central role in enhancing human-machine interactions, especially for small devices for which graphical user interface has obvious limitations. The speech-centric perspective for human-computer interface advanced in this paper derives from the view that speech is the only natural and expressive modality to enable people to access information from and to interact with any device. In this paper, we describe some recent work conducted at Microsoft Research, aimed at the development of enabling technologies for speechcentric multimodal human-computer interaction. In particular, we present a case study of a prototype system, called MapPointS, which is a speech-centric multimodal map-query application for North America. This prototype navigation system provides rich functionalities that allow users to obtain map-related information through speech, text, and pointing devices. Users can verbally query for state maps, city maps, directions, places, nearby businesses and other useful information within North America. They can also verbally control applications such as changing the map size and panning the map moving interactively through speech. In the current system, the results of the queries are presented back to users through graphical user interface. An overview and major components of the MapPointS system will be presented in detail first. This will be followed by software design engineering principles and considerations adopted in developing the MapPointS system, and by a description of some key robust speech processing technologies underlying general speech-centric human-computer interaction systems.

Handbook of Research on User Interface Design and Evaluation for Mobile Technology

“…This feature makes the late fusion easier to scale up to more modalities in the future than the early fusion. The architecture shown in Figure 1 utilizes the late fusion approach that has been widely adopted, for example, by a variety of systems including Put-That-There (Bolt, 1980), MapPointS (Deng & Yu, 2005), MiPad (Huang et al, 2001), ShopTalk (Cohen, et al, 1989), QuickSet (Cohen, Johnston, McGee, Oviatt, Pittman, Smith, et al, 1997), CUBRICON (Neal & Shapiro, 1991), Virtual World (Codella, Jalili, Koved, Lewis, Ling, Lipscomb, et al, 1992), Finger-Pointer (Fukumoto et al, 1994), VisualMan (Wang, 1995), and Jeanie (Vo & Wood, 1996).…”

“…An even more effective language model would weight different names differently, depending on the frequencies the user exchanged e-mail with the person, and the recentness of the interaction (Yu, Wang, Mahajan, Mau, & Acero, 2003). Another example of constructing the language model based on the context and user information is described in the speech enabled MapPoint (Deng & Yu, 2005). Without context information, the speech recognizer needs to load all location names and business names in the North America.…”

“…The research on multimodal human-computer interaction, however, became active only after Bolt (1980) proposed his original concept of "Put That There." Since then, a great amount of research has been carried out in this area (Bregler, Manke, Hild, & Waibel 1993;Codella, Jalili, Koved, Lewis, Ling, Lipscomb, et al, 1992;Cohen, Dalrymple, Moran, Pereira, Sullivan, Gargan, et al, 1989;Cohen, Johnston, McGee, Oviatt, Pittman, Smith, et al, 1997;Deng & Yu, 2005;Fukumoto, Suenga, & Mase, 1994;Hsu, Mahajan, & Acero 2005;Huang, Acero, Chelba, Deng, Droppo, Duchene, et al, 2001;Neal & Shapiro, 1991;Pavlovic, Berry, & Huang, 1997;Pavlovic & Huang, 1998;Vo, Houghton, Yang, Bub, Meier, Waibel, et al, 1995;Vo & Wood, 1996;Wang, 1995). Importantly, the body of this research work pointed out that MUIs can support flexible, efficient, and powerful human-computer interaction.…”

“…Second, different modalities can compensate for each other's limitations and thus provide users with more desirable experience (Deng & Yu, 2005;Oviatt, Bernard, & Levow, 1999;Oviatt & vanGent, 1996;Suhm, 1998). For example, the accuracy of a resource-constrained, midsized vocabulary speech recognizer is low given the current speech technology.…”

“…Many MUIs in mobile devices are speech centric, where speech is the central and main modality. In this chapter, we will focus on main issues on the design of effective speech centric MUIs in mobile devices based on our research and experience in developing MapPointS (Deng & Yu, 2005) and MiPad (Deng, Wang, Acero, Hon, Droppo, Boulis, et al, 2002;Huang, Acero, Chelba, Deng, Droppo, Duchene, et al, 2001). In Section 2, we describe a generic MUI architecture in mobile setting that consists of various recognizers for different input modalities, semantic parsers, a discourse manager, and a response manager.…”