Multimodal Interfaces of Human–Computer Interaction

Karpov, Alexey; Yusupov, Rafael

doi:10.1134/s1019331618010094

Cited by 55 publications

(27 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Robots that communicate with their operators "in a human-like way" [17] have appeared in research labs since at least the 1970s, when WABOT-1 was completed at Waseda University, Japan [33]. Social robots could utilize physical contact, gesture, speech and facial expressions as means of communication rather than the more traditional screen-based interfaces [32]. These benefits, of both natural communication, and improved human abilities, currently remain in the confines of the research labs, however, as social robots are scarce in consumer markets worldwide.…”

Section: Introductionmentioning

confidence: 99%

Long-Term Value of Social Robots through the Eyes of Expert Users

Dereshev

Kirk

Matsumura

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Socially-enabled digital technologies have attracted academic interest for decades, with recent commercial examples of Siri and Alexa, capturing public attention. However, despite ubiquitous visions of a robotic future, very few fully-fledged social robots are currently available to consumers. To improve their designs, studies of their long-term use are particularly valuable, but are currently unavailable. To address this gap, we report on interviews with four long-term users of Pepper-a social robot introduced in 2014. Our thematic analysis elicited insights across three kinds of value Pepper brought to its users: utilitarian functionality; the community that formed around Pepper; and a personal value of affection. We focus on two contributions those values bring to social robot design: social robots as social proxies, alleviating disabilities or acting akin to social media profiles; and robot nurturing as a design construct, going beyond purely utilitarian or hedonistic perspectives on robots.

show abstract

Section: Introductionmentioning

confidence: 99%

Long-Term Value of Social Robots through the Eyes of Expert Users

Dereshev

Kirk

Matsumura

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…This preference can be used to improve recognition [12]. End-users feel that interactions with a system are more natural when they can chose input modalities based on their preference [4,25]. By leveraging this preference and multimodal inputs, many benefits can be realized.…”

Section: Motivationmentioning

confidence: 99%

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

Williams

Ortega

2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems. CCS Concepts: • Human-centered computing → Human computer interaction (HCI); User studies; Mixed / augmented reality; Interaction techniques; Empirical studies in HCI; User centered design.

show abstract

“…Let ψ (t) be the impulse response of the basilar membrane of cochlear ψ (t) ∈ L 2 (R), in which the function ψ (t) satisfies the following conditions: 1 . It integrates to zero:…”

Section: ) Auditory Transformmentioning

confidence: 99%

“…Its main purpose is to communicate with a computer, so that the computer can convert the speech signal into corresponding text or commands through the process of understanding and recognition. Interpretation of human spoken language through technology has a diverse range of applications including in air transport, intelligent homes, disaster rescue, medical diagnostics, and other human-computer interaction fields [1]. At present, research into speech recognition is mainly focused on feature…”

Section: Introductionmentioning

confidence: 99%

Fusion Feature Extraction Based on Auditory and Energy for Noise-Robust Speech Recognition

et al. 2019

View full text Add to dashboard Cite

Environmental noise can pose a threat to the stable operation of current speech recognition systems. It is therefore essential to develop a front feature set that is able to identify speech under low signalto-noise ratio. In this paper, a robust fusion feature is proposed that can fully characterize speech information. To obtain the cochlear filter cepstral coefficients (CFCC), a novel feature is first extracted by the power-law nonlinear function, which can simulate the auditory characteristics of the human ear. Speech enhancement technology is then introduced into the front end of feature extraction, and the extracted feature and their first-order difference are combined in new mixed features. An energy feature Teager energy operator cepstral coefficient (TEOCC) is also extracted, and combined with the above-mentioned mixed features to form the fusion feature sets. Principal component analysis (PCA) is then applied to feature selection and optimization of the feature set, and the final feature set is used in a non-specific persons, isolated words, and smallvocabulary speech recognition system. Finally, a comparative experiment of speech recognition is designed to verify the advantages of the proposed feature set using a support vector machine (SVM). The experimental results show that the proposed feature set not only display a high recognition rate and excellent anti-noise performance in speech recognition, but can also fully characterize the auditory and energy information in the speech signals.INDEX TERMS Cochlear filter cepstral coefficients, Teager energy operators cepstral coefficients, principal component analysis, speech recognition.

show abstract

Multimodal Interfaces of Human–Computer Interaction

Cited by 55 publications

References 17 publications

Long-Term Value of Social Robots through the Eyes of Expert Users

Long-Term Value of Social Robots through the Eyes of Expert Users

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

Fusion Feature Extraction Based on Auditory and Energy for Noise-Robust Speech Recognition

Contact Info

Product

Resources

About