Forecasting Future Action Sequences with Neural Memory Networks

Gammulle, Harshala; Denman, Simon; Sridharan, Sridha; Fookes, Clinton

doi:10.48550/arxiv.1909.09278

Cited by 3 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An extension of the initial work [100] used the temporal point process for the same task [105], where the intensity parameter λ is dynamically calculated from past observations as Though the Poisson process has been known for modelling time arrival events, not all researchers adopt it. An alternative research direction modeled time as a real-valued output directly from regression models, e.g., using an MLP to generate the output [1,2,30,61,101] as shown in Figure 3.8: For any nextstep action, the "length of future action" and "remaining length of current action" will be regressed from fully-connected layers. In comparison to previous Poisson methods [100,105], these approaches choose deterministic time models rather than the Poisson distribution and therefore their outputs for a given input will be fixed.…”

Section: Add_milk Crack_egg Add_buttermentioning

confidence: 99%

“…To obtain the whole scope of the future, results of next-step prediction are reused as input for the next-next. This procedure repeats until it reaches the end of sequence (e.g., [30,101,105,110]). Alternatively, other approaches generate the rest of the sequence in one shot (e.g., [1,183]).…”

Section: Add_milk Crack_egg Add_buttermentioning

confidence: 99%

“…For singleton activity prediction, as seen in Table 3.4, the current performance is far from satisfactory and thus definitely deserves more effort. For the joint anticipation of activity and time duration, the performance produced from methods that take groundtruth (action, time) observations as input achieves decent results even for remote time indices (e.g., 75% accuracy for 50% future video horizon predictions for the Breakfast dataset [30], as seen in Table 3.7). However, given (action, time) inputs from off-theself algorithms (Table 3.6 and 3.8), which do not guarantee noisy-free estimation, performance on both datasets drops dramatically.…”

Section: Mpii-cookingmentioning

confidence: 99%

See 2 more Smart Citations

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Zhao¹,

Wildes²

2021

Preprint

View full text Add to dashboard Cite

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the causal relationship between the past, current and potential future information is the main focus.Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks.However, these approaches face challenges such as the curse of dimensionality, poor generalization and constraints from domain specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, e.g., reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

show abstract

Section: Add_milk Crack_egg Add_buttermentioning

confidence: 99%

Section: Add_milk Crack_egg Add_buttermentioning

confidence: 99%

Section: Mpii-cookingmentioning

confidence: 99%

See 1 more Smart Citation

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Zhao¹,

Wildes²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Action recognition can be broadly grouped into two main categories: Discrete (Delaitre et al, 2010;Simonyan and Zisserman, 2014;Gammulle et al, 2017Gammulle et al, , 2019b and continuous fine-grained action recognition (Lea et al, 2017;Ni et al, 2014;Zhou et al, 2014Zhou et al, , 2015Gammulle et al, 2020Gammulle et al, , 2019a approaches. Discrete action recognition is performed on pre-segmented video sequences containing one action per video while continuous fine-grained models are evaluated on untrimmed video sequences containing more than one action per video.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Attention Network for Action Segmentation

Gammulle

Denman

Sridharan

et al. 2020

Pattern Recognition Letters

Self Cite

View full text Add to dashboard Cite

Temporal segmentation of events is an essential task and a precursor for the automatic recognition of human actions in the video. Several attempts have been made to capture frame-level salient aspects through attention but they lack the capacity to effectively map the temporal relationships in between the frames as they only capture a limited span of temporal dependencies. To this end we propose a complete end-to-end supervised learning approach that can better learn relationships between actions over time, thus improving the overall segmentation performance. The proposed hierarchical recurrent attention framework analyses the input video at multiple temporal scales, to form embeddings at frame level and segment level, and perform fine-grained action segmentation. This generates a simple, lightweight, yet extremely effective architecture for segmenting continuous video streams and has multiple application domains. We evaluate our system on multiple challenging public benchmark datasets, including MERL Shopping, 50 salads, and Georgia Tech Egocentric datasets and achieves state-of-the-art performance. The evaluated datasets encompass numerous video capture settings which are inclusive of static overhead camera views and dynamic, ego-centric headmounted camera views, demonstrating the direct applicability of the proposed framework in a variety of settings.

show abstract

“…Despite the widespread use of these tools in the behavioral neuroscience community, there is no existing research attempting to forecast future behaviors from behavioral video data. There is a large machine learning literature dedicated to time series forecasting in the context of human activity, from forecasting of future human poses [33, 34, 35] to prediction of specific activities [36, 37, 38, 39, 40, 41, 42, 43, 44, 45], as well as prediction of future semantic information in video [46]. Given that locomotion patterns are implicated in – and defining criteria of – neurobehavioral disorders [47, 48], it is useful to predict future behaviors from behavioral video, both in order to understand the antecedent patterns that yield adverse (or favorable) behaviors and to automatically intervene in the experiment in real-time.…”

Section: Introductionmentioning

confidence: 99%

FABEL: Forecasting Animal Behavioral Events with Deep Learning-Based Computer Vision

Catto,

O’Connor,

Braunscheidel

et al. 2024

Preprint

View full text Add to dashboard Cite

Behavioral neuroscience aims to provide a connection between neural phenomena and emergent organism-level behaviors. This requires perturbing the nervous system and observing behavioral outcomes, and comparing observed post-perturbation behavior with predicted counterfactual behavior and therefore accurate behavioral forecasts. In this study we present FABEL, a deep learning method for forecasting future animal behaviors and locomotion trajectories from historical locomotion alone. We train an offline pose estimation network to predict animal body-part locations in behavioral video; then sequences of pose vectors are input to deep learning time-series forecasting models. Specifically, we train an LSTM network that predicts a future food interaction event in a specified time window, and a Temporal Fusion Transformer that predicts future trajectories of animal body-parts, which are then converted into probabilistic label forecasts. Importantly, accurate prediction of food interaction provides a basis for neurobehavioral intervention in the context of compulsive eating. We show promising results on forecasting tasks between 100 milliseconds and 5 seconds timescales. Because the model takes only behavioral video as input, it can be adapted to any behavioral task and does not require specific physiological readouts. Simultaneously, these deep learning models may serve as extensible modules that can accommodate diverse signals, such as in-vivo fluorescence imaging and electrophysiology, which may improve behavior forecasts and elucidate invervention targets for desired behavioral change.

show abstract

Forecasting Future Action Sequences with Neural Memory Networks

Cited by 3 publications

References 17 publications

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Hierarchical Attention Network for Action Segmentation

FABEL: Forecasting Animal Behavioral Events with Deep Learning-Based Computer Vision

Contact Info

Product

Resources

About