Training CNNs for 3-D Sign Language Recognition With Color Texture Coded Joint Angular Displacement Maps

Kumar, E. Kiran; Kishore, P. V. V.; Sastry, Ananth; Kumar, M. Teja Kiran; Kumar, D. Anil

doi:10.1109/lsp.2018.2817179

Cited by 70 publications

(36 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From any representation of angular joint relations,

can be computed as spatio-temporal representation

around a window

of neighboring frames. Joint relations shown to provide reasonable segmentation results are the combination of angular and distance features between line segments of pairs of joints introduced in [ 33 ], or the joint angular displacement transformation by Kumar et al [ 34 ]. Another possible variation could be to utilize the point of intersection between selected line segments as reference point for a subsequent computation of joint or inter-segment angular displacements.…”

Section: Cslr System Pipelinementioning

confidence: 99%

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Brock

Farag

Nakadai

2020

Sensors

View full text Add to dashboard Cite

The quality of recognition systems for continuous utterances in signed languages could be largely advanced within the last years. However, research efforts often do not address specific linguistic features of signed languages, as e.g., non-manual expressions. In this work, we evaluate the potential of a single video camera-based recognition system with respect to the latter. For this, we introduce a two-stage pipeline based on two-dimensional body joint positions extracted from RGB camera data. The system first separates the data flow of a signed expression into meaningful word segments on the base of a frame-wise binary Random Forest. Next, every segment is transformed into image-like shape and classified with a Convolutional Neural Network. The proposed system is then evaluated on a data set of continuous sentence expressions in Japanese Sign Language with a variation of non-manual expressions. Exploring multiple variations of data representations and network parameters, we are able to distinguish word segments of specific non-manual intonations with 86% accuracy from the underlying body joint movement data. Full sentence predictions achieve a total Word Error Rate of 15.75%. This marks an improvement of 13.22% as compared to ground truth predictions obtained from labeling insensitive towards non-manual content. Consequently, our analysis constitutes an important contribution for a better understanding of mixed manual and non-manual content in signed communication.

show abstract

“…From any representation of angular joint relations,

can be computed as spatio-temporal representation

around a window

Section: Cslr System Pipelinementioning

confidence: 99%

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Brock

Farag

Nakadai

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Hence, modeling joints as distance maps will bring the entire action sequence into a RGB image of the skeletal data. This map representation has improved both the recognition accuracy as well as training time latency [14,15]. These distance maps produce patterns on the image representing the underlying changes between Joint distances during an action sequence.…”

Section: Literature Reviewmentioning

confidence: 99%

“…These J joints form a joint set with position vectors Finally, the color-coding is done simply by using the 'jet' color map to encode the JCDs with the following standard mapping procedure in [15] to create JCDI spatial color maps for the CNN. Fig.4 shows some color coded JCDIs of common actions across datasets.…”

Section: Fig3 Skeleton Joint Representationmentioning

confidence: 99%

See 1 more Smart Citation

Convolutional Long Short-Term Memory Hybrid Networks for Skeletal Based Human Action Recognition

Prasad*,

Kishore,

Rao

2020

IJITEE

View full text Add to dashboard Cite

The objective is to develop a time series image representation of the skeletal action data and use it for recognition through a convolutional long short-term deep learning framework. Consequently, Kinect captured human skeletal data is transformed into a Joint Change Distance Image (JCDI) descriptor which maps the time changes in the joints. Subsequently, JCDIs are decoded spatially well with a Convolutional (CNN). Temporal decomposition is executed on long short term memory (LSTM) with data changes along x , y and z position vectors of the skeleton. We propose a combination of CNN and LSTM which maps the spatio temporal information to generate a generalized time series features for recognition. Finally, scores are fused from spatially vibrant CNNs and temporally sound LSTMs for action recognition. Publicly available action datasets such as NTU RGBD, MSR Action, UTKinect and G3D were used as test inputs for experimentation. The results showed a better performance due to spatio temporal modeling at both the representation and the recognition stages when compared to other state-of-the-arts

show abstract

“…18 And it has achieved breakthrough performance in gesture recognition tasks. 11,23,36,39 The first layer of the network often learns the local structure of images, such as texture and gradient information. 36 In the high layers, the network has learned discriminative high-level abstract features.…”

Section: Introductionmentioning

confidence: 99%

Student body gesture recognition based on Fisher broad learning system

Shi

Wei

Pan

et al. 2019

Int. J. Wavelets Multiresolut Inf. Process.

View full text Add to dashboard Cite

Observing student body gesture has been widely used to assess teaching effectiveness over the past few decades. However, manual observation is not suitable for the automatic data analysis in the field of learning analytics. Consequently, a student body gesture recognition method based on Fisher Broad Learning System (FBLS) and Local Log-Euclidean Multivariate Gaussian (L 2 EMG) is proposed in this paper. FBLS is designed by introducing the discriminative information into the hidden layer of Broad Learning System (BLS) and reducing the dimensionality of hidden-layer representations. FBLS has superiorities in accuracy and speed. In addition, L 2 EMG, which is a highly distinctive descriptor, characterizes the local image with a multivariate Gaussian distribution. So L 2 EMG features are fed into the FBLS for recognition in this paper. Extensive † Corresponding author. 1950001-1 Int. J. Wavelets Multiresolut Inf. Process. 2019.17. Downloaded from www.worldscientific.com by 44.224.250.200 on 07/03/20. Re-use and distribution is strictly not permitted, except for Open Access articles. Y. Shi et al.experimental results on self-built dataset show that the proposed student body gesture recognition method obtains better results than other benchmarking methods.

show abstract

Training CNNs for 3-D Sign Language Recognition With Color Texture Coded Joint Angular Displacement Maps

Cited by 70 publications

References 16 publications

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Convolutional Long Short-Term Memory Hybrid Networks for Skeletal Based Human Action Recognition

Student body gesture recognition based on Fisher broad learning system

Contact Info

Product

Resources

About