A Pattern Recognition-Based Multi-Modal HCI Using 3 Sensory Channels

Abstract

This paper describes a PDA-based MMCR (Multi-Modal Command Recognizer for PDA control and handling) using double-touching with a finger by coupling embedded speech and KSSL recognizer, and suggests an improved synchronization method between multi-modalities for simultaneous multi-modality, for a pattern recognition-based neo multi-modal HCI. The MMCR fuses and recognizes 16 word-based command models that are represented by stylus, speech and KSSL (Korean Standard Sign Language), and then translates the recognition result into synthetic speech and visual illustration, for multi-modal PDA handling and interaction. Fifth International Conference on Software Engineering Research, Management and Applications 0-7695-2867-8/07 $25.00