A System for Large Vocabulary Sign Search

Abstract

Abstract. A method is presented to help users look up the meaning of an unknown sign from American Sign Language (ASL). The user submits a video of the unknown sign as a query, and the system retrieves the most similar signs from a database of sign videos. The user then reviews the retrieved videos to identify the video displaying the sign of interest. Hands are detected in a semi-automatic way: the system performs some hand detection and tracking, and the user has the option to verify and correct the detected hand locations. Features are extracted based on hand motion and hand appearance. Similarity between signs is measured by combining dynamic time warping (DTW) scores, which are based on hand motion, with a simple similarity measure based on hand appearance. In user-independent experiments, with a system vocabulary of 1,113 signs, the correct sign was included in the top 10 matches for 78% of the test queries.

Local frame match distance: A novel approach for exemplar gesture recognition

“…One popular intuitive method is to segment a sign into motion or other types of sub-units and then use an HMM to model the temporal changes in sub-units throughout each sign [28], [29]. Dynamic Time Warping has also been used for action and gesture recognition [7], [8], [30], [31] and it has shown its superiority over LSTM and HMM models [31]. Some of these works approach the idea of class variability modeling [30], [31].…”

“…To represent a hand gesture, we use the feature vector introduced in [8]. The feature vector based on 2D hand position information is built for each video frame in order to describe what is occurring at every point in time.…”

“…In such cases, Dynamic Time Warping (DTW) [3] is frequently used on hand location or other information to generate scores that serve as a measure of similarity to training examples [4], [5], [6], [7]. DTW has been improved with the use of a well-designed feature vector that includes more than hand positions to represent the state of a gesture at each point in time [8].…”

Sign Spotting Using Hierarchical Sequential Patterns with Temporal Intervals

“…Approaches can be divided between tracking-based, sub-unit classifiers [5], and more data driven approaches. As examples of the latter, Wang et al [11], created an American Sign Language (ASL) dictionary based on similarity between signs using a Dynamic Space-Time Warping (DSTW) approach and Gavrilov et al [4] proposed a data mining approach for detecting reduplications in signs. One prevalent family of methods for SLR are Hidden Markov Models (HMMs).…”

Sign Language Recognition using Sequential Pattern Trees

“…They used an exemplar, sign level approach and did not use Hidden Markov Models (HMMs) due to the high quantities of training data required. They present results for a dictionary containing 1113 signs [12]. More recently, Pitsikalis et al [9] proposed a method which uses linguistic labelling to split signs into sub-units.…”