Using redundant speech and handwriting for learning new vocabulary and understanding abbreviations

Kaiser, Ed

doi:10.1145/1180995.1181060

Cited by 14 publications

(8 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the ACI we have implemented for testing SHACER, the perceived interactions occur in public spaces that are shared by the participants -e.g., (a) a shared interactive whiteboard or a piece of digital paper for public sketching and handwriting [5], (b) a shared conversational space for speech captured by close-talking microphones [18]. Participants in this shared public space can be co-located or remotely distributed.…”

Section: Multimodal Understanding Of Human-human Interactionmentioning

confidence: 99%

“…Neither of these handwritten abbreviations (CB [17] that combining information from redundant handwriting and speech is significantly more reliable for the recognition of Gantt chart labels than depending on either mode alone.…”

Section: Study Implicationsmentioning

confidence: 99%

“…For the cases shown in Figure 10 the nearby handwritten labels are firstletter abbreviations, FG and CB. The WPS spoken recognition, together with the nearby first-letter abbreviation matches, triggers the association to spoken semantics [18]. We call this associative process Multimodal Semantic Acquisition, because learned semantics carried in one mode -WPS speech recognition of Fred Green or Cindy Black -are dynamically acquired by new symbols in another mode (Fig.…”

Section: Study Implicationsmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal redundancy across handwriting and speech during computer mediated human-human interactions

Kaiser¹,

Barthelmess²,

Erdmann³

et al. 2007

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

Lecturers, presenters and meeting participants often say what they publicly handwrite. In this paper, we report on three empirical explorations of such multimodal redundancy -during whiteboard presentations, during a spontaneous brainstorming meeting, and during the informal annotation and discussion of photographs. We show that redundantly presented words, compared to other words used during a presentation or meeting, tend to be topic specific and thus are likely to be out-of-vocabulary. We also show that they have significantly higher tf-idf (term frequency-inverse document frequency) weights than other words, which we argue supports the hypothesis that they are dialogue-critical words. We frame the import of these empirical findings by describing SHACER, our recently introduced Speech and HAndwriting reCognizER, which can combine information from instances of redundant handwriting and speech to dynamically learn new vocabulary.

show abstract

Section: Multimodal Understanding Of Human-human Interactionmentioning

confidence: 99%

Section: Study Implicationsmentioning

confidence: 99%

Section: Study Implicationsmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal redundancy across handwriting and speech during computer mediated human-human interactions

Kaiser¹,

Barthelmess²,

Erdmann³

et al. 2007

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Once a term has been introduced in full (via handwriting and speech), the system is able to determine both its spelling and pronunciation. Further spoken references in the temporal vicinity of a an abbreviated handwritten term can then be recovered via the application of heuristics (see [13] for details of this mechanism).…”

Section: Propagation Via Abbreviationsmentioning

confidence: 99%

Collaborative multimodal photo annotation over digital paper

Barthelmess

Kaiser

Xuefeng

et al. 2006

Proceedings of the 8th International Conference on Multimodal Interfaces

View full text Add to dashboard Cite

The availability of metadata annotations over media content such as photos is known to enhance retrieval and organization, particularly for large data sets. The greatest challenge for obtaining annotations remains getting users to perform the large amount of tedious manual work that is required. In this paper we introduce an approach for semi-automated labeling based on extraction of metadata from naturally occurring conversations of groups of people discussing pictures among themselves. As the burden for structuring and extracting metadata is shifted from users to the system, new recognition challenges arise. We explore how multimodal language can help in 1) detecting a concise set of meaningful labels to be associated with each photo, 2) achieving robust recognition of these key semantic terms, and 3) facilitating label propagation via multimodal shortcuts. Analysis of the data of a preliminary pilot collection suggests that handwritten labels may be highly indicative of the semantics of each photo, as indicated by the correlation of handwritten terms with high frequency spoken ones. We point to initial directions exploring a multimodal fusion technique to recover robust spelling and pronunciation of these high-value terms from redundant speech and handwriting.

show abstract

“…handwriting and speech), which naturally occur redundantly, we can apply cross-stream correlation at the recognizer level to improve the systems understanding, as evidenced by the significant 37.5% relative reduction in abbreviation labeling error seen in the held-out test set [20].…”

Section: Applicationsmentioning

confidence: 99%

Human-centered collaborative interaction

Barthelmess

Kaiser

Lunsford

et al. 2006

Proceedings of the 1st ACM International Workshop on Human-Centered Multimedia

View full text Add to dashboard Cite

Recent years have witnessed an increasing shift in interest from single user multimedia/multimodal interfaces towards support for interaction among groups of people working closely together, e.g. during meetings or problem solving sessions. However, the introduction of technology to support collaborative practices has not been devoid of problems. It is not uncommon that technology meant to support collaboration may introduce disruptions and reduce group effectiveness.Human-centered multimedia and multimodal approaches hold a promise of providing substantially enhanced user experiences by focusing attention on human perceptual and motor capabilities, and on actual user practices. In this paper we examine the problem of providing effective support for collaboration, focusing on the role of human-centered approaches that take advantage of multimodality and multimedia. We show illustrative examples that demonstrate human-centered multimodal and multimedia solutions that provide mechanisms for dealing with the intrinsic complexity of human-human interaction support.

show abstract

Using redundant speech and handwriting for learning new vocabulary and understanding abbreviations

Cited by 14 publications

References 37 publications

Multimodal redundancy across handwriting and speech during computer mediated human-human interactions

Multimodal redundancy across handwriting and speech during computer mediated human-human interactions

Collaborative multimodal photo annotation over digital paper

Human-centered collaborative interaction

Contact Info

Product

Resources

About