Can Gaze Inform Egocentric Action Recognition?

Zhang, Zehua; Crandall, David J.; Proulx, Michael J.; Talathi, Sachin S.; Sharma, Abhishek

doi:10.1145/3517031.3529628

Cited by 7 publications

(2 citation statements)

References 43 publications

(72 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Canedo et al (2018) presented a model that was theoretically capable of tracking the students' attention and gave an overview of the state-of-the-art on computer vision techniques for monitoring classrooms. Zhang et al (2022). constructed a neural fusion approach known as cross-modality attention blocks to capture gaze-signal for action recognition during inference.…”

Section: Students' Attention In Classroom Teachingmentioning

confidence: 99%

Analyzing students' attention by gaze tracking and object detection in classroom teaching

Zhang

Sun

et al. 2023

DTA

View full text Add to dashboard Cite

PurposeAttention is one of the most important factors to affect the academic performance of students. Effectively analyzing students' attention in class can promote teachers' precise teaching and students' personalized learning. To intelligently analyze the students' attention in classroom from the first-person perspective, this paper proposes a fusion model based on gaze tracking and object detection. In particular, the proposed attention analysis model does not depend on any smart equipment.Design/methodology/approachGiven a first-person view video of students' learning, the authors first estimate the gazing point by using the deep space–time neural network. Second, single shot multi-box detector and fast segmentation convolutional neural network are comparatively adopted to accurately detect the objects in the video. Third, they predict the gazing objects by combining the results of gazing point estimation and object detection. Finally, the personalized attention of students is analyzed based on the predicted gazing objects and the measurable eye movement criteria.FindingsA large number of experiments are carried out on a public database and a new dataset that is built in a real classroom. The experimental results show that the proposed model not only can accurately track the students' gazing trajectory and effectively analyze the fluctuation of attention of the individual student and all students but also provide a valuable reference to evaluate the process of learning of students.Originality/valueThe contributions of this paper can be summarized as follows. The analysis of students' attention plays an important role in improving teaching quality and student achievement. However, there is little research on how to automatically and intelligently analyze students' attention. To alleviate this problem, this paper focuses on analyzing students' attention by gaze tracking and object detection in classroom teaching, which is significant for practical application in the field of education. The authors proposed an effectively intelligent fusion model based on the deep neural network, which mainly includes the gazing point module and the object detection module, to analyze students' attention in classroom teaching instead of relying on any smart wearable device. They introduce the attention mechanism into the gazing point module to improve the performance of gazing point detection and perform some comparison experiments on the public dataset to prove that the gazing point module can achieve better performance. They associate the eye movement criteria with visual gaze to get quantifiable objective data for students' attention analysis, which can provide a valuable basis to evaluate the learning process of students, provide useful learning information of students for both parents and teachers and support the development of individualized teaching. They built a new database that contains the first-person view videos of 11 subjects in a real classroom and employ it to evaluate the effectiveness and feasibility of the proposed model.

show abstract

Section: Students' Attention In Classroom Teachingmentioning

confidence: 99%

Analyzing students' attention by gaze tracking and object detection in classroom teaching

Zhang

Sun

et al. 2023

DTA

View full text Add to dashboard Cite

show abstract

“…Outdoor multi-human datasets like 3DPW [110] and MuPoTS [83] have constrained human activities and lack egocentric annotations [5,108], or are limited in diversity [109]. Existing egocentric datasets primarily focus on hand-object interactions and action recognition [3,19,20,27,53,54,56,61,67,85,87,92,101,104,118,128]. Recent datasets like Mo2Cap2 [115], You2Me [86], HPS [35] and EgoBody [123] focus on 3D human pose annotations -but are limited to one or two human subjects and indoor settings.…”

Section: Related Workmentioning

confidence: 99%

Occluded Human Mesh Recovery

Khirodkar

Tripathi

2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Figure 1: Left: Our proposed capture setup consists of multiple egocentric cameras from wearable glasses and stationary secondary cameras. This setup is flexible and mobile, allowing us to generate high-quality multi-human 3D annotations for diverse in-the-wild settings. Center: Multiple synchronized egocentric views while playing soccer. Right: Synchronized secondary views (cropped) from the stationary cameras. All cameras are spatiotemporally localized in the world coordinate.

show abstract

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Zhang

Zhi-yin

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Can Gaze Inform Egocentric Action Recognition?

Cited by 7 publications

References 43 publications

Analyzing students' attention by gaze tracking and object detection in classroom teaching

Analyzing students' attention by gaze tracking and object detection in classroom teaching

Occluded Human Mesh Recovery

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Contact Info

Product

Resources

About