Script Data for Attribute-Based Recognition of Composite Activities

Rohrbach, Marcus; Regneri, Michaela; Andriluka, Mykhaylo; Amin, Sikandar; Pinkal, Manfred; Schiele, Bernt

doi:10.1007/978-3-642-33718-5_11

Cited by 157 publications

(103 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This includes both discriminative [1], [13], [14], [15] and generative models [16], [17]. Another approach is to represent activities as collections of semantic attributes [18], [19], [20], [21]. These methods use an intermediate level of representation such as the presence or absence of semantic concepts (e.g., scene types, actions, objects, etc.)…”

Section: Related Workmentioning

confidence: 99%

Anticipating Human Activities using Object Affordances for Reactive Robotic Response

Koppula

Saxena

2013

Robotics: Science and Systems IX

159

214

View full text Add to dashboard Cite

Abstract-An important aspect of human perception is anticipation, which we use extensively in our day-to-day activities when interacting with other humans as well as with our surroundings. Anticipating which activities will a human do next (and how) can enable an assistive robot to plan ahead for reactive responses. Furthermore, anticipation can even improve the detection accuracy of past activities. The challenge, however, is two-fold: We need to capture the rich context for modeling the activities and object affordances, and we need to anticipate the distribution over a large space of future human activities. In this work, we represent each possible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations through object affordances. We then consider each ATCRF as a particle and represent the distribution over the potential futures using a set of particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, we first show that anticipation improves the state-ofthe-art detection results. We then show that for new subjects (not seen in the training set), we obtain an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of 84.1%, 74.4% and 62.2% for an anticipation time of 1, 3 and 10 seconds respectively. Finally, we also show a robot using our algorithm for performing a few reactive responses. †

show abstract

Section: Related Workmentioning

confidence: 99%

Anticipating Human Activities using Object Affordances for Reactive Robotic Response

Koppula

Saxena

2013

Robotics: Science and Systems IX

159

214

View full text Add to dashboard Cite

show abstract

“…The average number of words per summary is 67, the average number of words per sentence is 10 with stopwords and the average number of descriptions per video is eight. The recent data set [24] is also about cooking but it has a fixed scene and no object annotations.…”

Section: Datasets and Features Trecvid Med12 Datasetmentioning

confidence: 99%

A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching

Das

Doell

et al. 2013

2013 IEEE Conference on Computer Vision and Pattern Recognition

256

182

View full text Add to dashboard Cite

The problem of describing images through natural language has gained importance in the computer vision community. Solutions to image description have either focused on a top-down approach of generating language through combinations of object detections and language models or bottom-up propagation of keyword tags from training images to test images through probabilistic or nearest neighbor techniques. In contrast, describing videos with natural language is a less studied problem. In this paper, we combine ideas from the bottom-up and top-down approaches to image description and propose a method for video description that captures the most relevant contents of a video in a natural language description. We propose a hybrid system consisting of a low level multimodal latent topic model for initial keyword annotation, a middle level of concept detectors and a high level module to produce final lingual descriptions. We compare the results of our system to human descriptions in both short and long forms on two datasets, and demonstrate that final system output has greater agreement with the human descriptions than any single level.

show abstract

“…Sadanand and Corso [22] have shown substantial improvements over standard benchmarks by using a bank of action detectors sampled broadly across semantic and viewpoint spaces. Rohrbach et al [21] augmented video with text-script data and modeled activities as common sets of attributes, defined in terms of basic actions and objects. Finally, Li and Vasconcelos [14] introduced a model (BDS) of the temporal structure of attributes.…”

Section: Related Workmentioning

confidence: 99%

“…The second, inspired by recent advances in image analysis, is to represent activities as collections of semantic attributes [15,22,21,6]. This entails an intermediate level of representation, where features are no longer visual, but identifiers of the occurrence of semantic concepts of interest, such as scene types, actions, objects, etc.…”

Section: Introductionmentioning

confidence: 99%

Recognizing Activities via Bag of Words for Attribute Dynamics

Sawhney

et al. 2013

2013 IEEE Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

In this work, we propose a novel video representation for activity recognition that models video dynamics with attributes of activities. A video sequence is decomposed into short-term segments, which are characterized by the dynamics of their attributes. These segments are modeled by a dictionary of attribute dynamics templates, which are implemented by a recently introduced generative model, the binary dynamic system (BDS). We propose methods for learning a dictionary of BDS's from a training corpus, and for quantizing attribute sequences extracted from videos into these BDS codewords. This procedure produces a representation of the video as a histogram of BDS codewords, which is denoted the bag-of-words for attribute dynamics (BoWAD). An extensive experimental evaluation reveals that this representation outperforms other state-of-the-art approaches in temporal structure modeling for complex activity recognition.

show abstract

Script Data for Attribute-Based Recognition of Composite Activities

Cited by 157 publications

References 26 publications

Anticipating Human Activities using Object Affordances for Reactive Robotic Response

Anticipating Human Activities using Object Affordances for Reactive Robotic Response

A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching

Recognizing Activities via Bag of Words for Attribute Dynamics

Contact Info

Product

Resources

About