Developing multimodal interfaces: A theoretical framework and guided propagation networks

Martin, Jean‐Claude; Veldman, R.; Béroule, Dominique

doi:10.1007/bfb0052318

Cited by 18 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…First, multimodal input systems provide disambiguation when multiple different input types (e.g., speech and gesture) are combined [9]. For example, the TYCOON system [16] provided a fusion method that computed input likelihoods based on temporal proximity of fused events. Additionally, XWand [25] uses a Bayesian network to fuse multiple input sources, as well as a user's previous actions, to approximate a user's intent.…”

Section: Framework For Handling Inputs With Uncertaintymentioning

confidence: 99%

An Architecture for Generating Interactive Feedback in Probabilistic User Interfaces

Schwarz

Mankoff

Hudson

2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Increasingly natural, sensed, and touch-based input is being integrated into devices. Along the way, both custom and more general solutions have been developed for dealing with the uncertainty that is associated with these forms of input. However, it is difficult to provide dynamic, flexible, and continuous feedback about uncertainty using traditional interactive infrastructure. Our contribution is a general architecture with the goal of providing support for continual feedback about uncertainty.Our architecture is based on prior work in modeling uncertainty using Monte Carlo sampling, and tracks multiple interfaces -one for each plausible and differentiable sequence of input that the user may have intended. Importantly, it considers how the presentation of uncertainty can be organized and implemented in a general way. Our primary contribution is a method for reducing the number of alternative interfaces and fusing possible interfaces into a single interface that both communicates uncertainty and allows for disambiguation. We demonstrate the value of this result through a collection of 11 new and existing feedback techniques along with two applications demonstrating the use of the feedback architecture.

show abstract

Section: Framework For Handling Inputs With Uncertaintymentioning

confidence: 99%

An Architecture for Generating Interactive Feedback in Probabilistic User Interfaces

Schwarz

Mankoff

Hudson

2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…The typology we described above was initially developed for studying multimodal human-computer interfaces (HCIs) and multimodal human-human communication (Martin et al 1998). As suggested by Hoc (2002), the use of concepts borrowed from human-human co-operation is justified because it has been proved that humans can transfer co-operative attitudes to machines.…”

Section: Application To Human Multimodal Collaborative Communication mentioning

confidence: 99%

On the use of the multimodal clues in human behaviour for the modelling of agent co-operative behaviour

Martin

2002

Connection Science

View full text Add to dashboard Cite

Although there are different definitions of autonomous agents, one of their features should probably be the flexibility of choice an autonomous agent should have on how and with which other agents it should co-operate. The TYCOON framework is introduced for studying co-operation between agents. The framework includes on one hand a typology including six types of co-operation between agents (transfer, equivalence, specialization, redundancy, 'complementarity' and concurrency), and on the other hand the associate algorithms for computing metrics of co-operative behaviour. It is described how the framework can be applied to two different problem-solving tasks: a simple variable binding task and a more complex multimodal collective presentation task. I believe that these TYCOON low-level specifications of co-operation between agents might be useful as building blocks for bridging the gap between experimental human behaviour analysis and the modelling of higher-level co-operation-related attitudes related to autonomy, delegation and control.

show abstract

“…The synchrony temporal coding has been proposed for dynamically binding units [45] a method used for instance between slots and values in frames symbolic structures [46]. Synchrony between pulse signals has been incorporated with the GPN approach by J. C. Martin for dealing with multimodal commands [47], [48].…”

Section: Gpns For Natural Language Processingmentioning

confidence: 99%

An Instance of Coincidence Detection Architecture Relying on Temporal Coding

Béroule

2004

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

Although time and space are interrelated in every occurrence of real-world events, only spatial codes are used at the basic level of most computational architectures. Inspired by neurobiological facts and hypotheses that assign a primordial coding role to the temporal dimension, and developed to address both cognitive and engineering applications, guided propagation networks (GPNs) are aimed at a generic real-time machine, based on time-space coincidence testing. The involved temporal parameters are gradually introduced, in relation with complementary applications in the field of human-machine communication: sensori-motor modeling, pattern recognition and natural language processing.

show abstract

Developing multimodal interfaces: A theoretical framework and guided propagation networks

Cited by 18 publications

References 19 publications

An Architecture for Generating Interactive Feedback in Probabilistic User Interfaces

An Architecture for Generating Interactive Feedback in Probabilistic User Interfaces

On the use of the multimodal clues in human behaviour for the modelling of agent co-operative behaviour

An Instance of Coincidence Detection Architecture Relying on Temporal Coding

Contact Info

Product

Resources

About