Automatic hand trajectory segmentation and phoneme transcription for sign language

Abstract

This paper presents an automatic approach to segment 3-D hand trajectories and transcribe phonemes based on them, as a step towards recognizing American sign language (ASL). We first apply a segmentation algorithm which detects minimal velocity and maximal change of directional angle to segment the hand motion trajectory of naturally signed sentences. This yields oversegmented trajectories, which are further processed by a trained naïve Bayesian detector to identify true segmented points and eliminate false alarms. The above segmentation algorithm yielded 88.5% true segmented points and 11.8% false alarms on unseen ASL sentence samples. These segmentation results were refined by a simple majority voting scheme, and the final segments obtained were used to transcribe phonemes for ASL. This was based on clustering PCA-based features extracted from training sentences. We then trained Hidden Markov Models (HMMs) to recognize the sequence of phonemes in the sentences. On the 25 test sentences containing 157 segments, the average number of errors obtained was 15.6.

Sign Language Recognition Using Sub-units

“…Yin et al (2009) used an accelerometer glove to gather information about a sign, they then applied discriminative feature extraction and 'similar state tying' algorithms, to decide sub-unit level segmentation of the data. Whereas Kong and Ranganath (2008) and Han et al (2009) looked at automatic segmentation of sign motion into sub-units, using discontinuities in the trajectory and acceleration to indicate where segments begin and end. These were then clustered into a code book of possible exemplar trajectories using either Dynamic Time Warping (DTW) distance measures Han et al or Principal Component Analysis (PCA) Kong and Ranganath. Traditional sign recognition systems use tracking and data driven approaches (Han et al, 2009;Yin et al, 2009).…”

Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training

“…Then, using vector-quantified histograms of motion directions, the authors were able to successfully identify basic gestures, such as drawing a square or a circle on a plane, in real time. In simpler contexts with controlled motions, other authors have also used different representations, such as zero-velocity crossing [21], and velocity and direction [22]. However, these individual representations do not provide enough information.…”

May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora

“…Quite similarly, Kong and Ranganath [11] perform a segmentation of data provided by a Polhemus tracker. A Naive Bayes classifier, trained with manual annotation, is used to find false boundary points.…”