Dynamic Face Video Segmentation via Reinforcement Learning

Wang, Yujiang; Dong, Mingzhi; Shen, Jie; Wu, Yang; Cheng, Shiyang; Pantić, Maja

doi:10.1109/cvpr42600.2020.00699

Cited by 26 publications

(17 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is unnecessary to re-compute the whole current frame given that we have already obtained the features of previous frames: we can use the features in the previous frames to accelerate the analysis of the current frame. Such techniques have been studied in the field of video segmentation, and we have adopted the Deep Feature Flow [9] framework following [20], [16] to address temporal redundancy. Specifically, if a current frame is determined as a non-key frame with low resolution, we use FlowNet [8] to obtain the optical flow between the current frame and the last key frame, and the computed optical flow is used to propagate features from the last key frame into the current one such that the performance drop caused by lowresolution frames can be compensated for.…”

Section: B Dynamics Analysismentioning

confidence: 99%

“…if at = a 1 then if key action 16: Yt ← {bt, ct, st, dt} put together current predictions 28: end for Output: Energy-efficient video analytics results {Y0, Y1, ..., YN } state P sensor,idle is 141.8 mW and that in P sensor,active is 8.27 mW/MP•R + 17.364 mW + 113.03 mW. We use a T exp of 20 ms in the following study.…”

Section: B Experimental Settings 1) Evaluation Platformmentioning

confidence: 99%

See 1 more Smart Citation

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

Zhao

Dong

Wang

et al. 2022

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Deep-learning-based video processing has yielded transformative results in recent years. However, the video analytics pipeline is energy-intensive due to high data rates and reliance on complex inference algorithms, which limits its adoption in energy-constrained applications. Motivated by the observation of high and variable spatial redundancy and temporal dynamics in video data streams, we design and evaluate an adaptive-resolution optimization framework to minimize the energy use of multitask video analytics pipelines. Instead of heuristically tuning the input data resolution of individual tasks, our framework utilizes deep reinforcement learning to dynamically govern the input resolution and computation of the entire video analytics pipeline. By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency. Governed by reinforcement learning, optical flow is incorporated into the framework to minimize unnecessary spatio-temporal redundancy that leads to re-computation, while preserving accuracy. The proposed framework is applied to video instance segmentation which is one of the most challenging computer vision tasks, and achieves better energy efficiency than all baseline methods of similar accuracy on the YouTube-VIS dataset.

show abstract

Section: B Dynamics Analysismentioning

confidence: 99%

Section: B Experimental Settings 1) Evaluation Platformmentioning

confidence: 99%

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

Zhao

Dong

Wang

et al. 2022

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is inspired by [3], a work showing that statistical eye information such as pupil size can help to improve the emotion recognition accuracy. Denoting the pupil size information as ps 𝑡 ∈ R 2 , we treat ps 𝑡 as an expert information, and following [60,68], we concatenate this expert information ps 𝑡 with f 𝑡 𝑒 , which can be written as…”

Section: Extracting Eye Featuresmentioning

confidence: 99%

Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices

Zhao,

Chang,

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Emotion recognition in smart eyewear devices is highly valuable but challenging. One key limitation of previous works is that the expression-related information like facial or eye images is considered as the only emotional evidence. However, emotional status is not isolated; it is tightly associated with people's visual perceptions, especially those sentimental ones. However, little work has examined such associations to better illustrate the cause of different emotions. In this paper, we study the emotionship analysis problem in eyewear systems, an ambitious task that requires not only classifying the user's emotions but also semantically understanding the potential cause of such emotions. To this end, we devise EMOShip, a deep-learning-based eyewear system that can automatically detect the wearer's emotional status and simultaneously analyze its associations with semantic-level visual perceptions. Experimental studies with 20 participants demonstrate that, thanks to the emotionship awareness, EMOShip not only achieves superior emotion recognition accuracy over existing methods (80.2% vs. 69.4%), but also provides a valuable understanding of the cause of emotions. Pilot studies with 20 participants further motivate the potential use of EMOShip to empower emotion-aware applications, such as emotionship self-reflection and emotionship life-logging.CCS Concepts: • Human-centered computing → Mobile devices.

show abstract

“…A following fully-connected layer FC0 gradually reduces the channels into two representing the probability of the binary action . In addition, inspired by the work of Wang et al [46], we append the likelihood L of historical eye movement types with the input tensor of FC0 layer to supervise the learning during training and help with the inference during testing.…”

Section: Pipeline Of the Tva Networkmentioning

confidence: 99%

MemX

Chang

Zhao

Dong

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

Self Cite

View full text Add to dashboard Cite

This work presents MemX: a biologically-inspired attention-aware eyewear system developed with the goal of pursuing the long-awaited vision of a personalized visual Memex. MemX captures human visual attention on the fly, analyzes the salient visual content, and records moments of personal interest in the form of compact video snippets. Accurate attentive scene detection and analysis on resource-constrained platforms is challenging because these tasks are computation and energy intensive. We propose a new temporal visual attention network that unifies human visual attention tracking and salient visual content analysis. Attention tracking focuses computation-intensive video analysis on salient regions, while video analysis makes human attention detection and tracking more accurate. Using the YouTube-VIS dataset and 30 participants, we experimentally show that MemX significantly improves the attention tracking accuracy over the eye-tracking-alone method, while maintaining high system energy efficiency. We have also conducted 11 in-field pilot studies across a range of daily usage scenarios, which demonstrate the feasibility and potential benefits of MemX.

show abstract

Dynamic Face Video Segmentation via Reinforcement Learning

Cited by 26 publications

References 37 publications

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices

MemX

Contact Info

Product

Resources

About