The “Something Something” Video Database for Learning and Evaluating Visual Common Sense

Abstract

Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the "something-something" database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.

Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

“…Recently, crowd-acted and fine-grained datasets [8,19,4,7] receive more and more favor and attention. These newly collected datasets pose new challenges for action recognition.…”

“…Both of them use late a fusion strategy. Although these 2D networks perform well on datasets like UCF101 [21] or Kinetics [3], they show much less satisfactory results on datasets that require extensive temporal reasoning [8,13]. In another branch, 3D networks(e.g.…”

“…The two examples are similar at the Figure 1. Examples from the something-something dataset [8]. The groundtruth for the two videos are "throwing something in the air and catching it" and "throwing something in the air and letting it fall".…”

Reasoning About Human-Object Interactions Through Dual Attention Networks

“…The focus of attention is represented by a heatmap indicating the likelihood of where an action is taking place or where an object is being manipulated in each frame. These attention maps can enhance video representation and improve both action and object recognition, yielding very competitive performance on Something-something [11] dataset. We show that the attention maps are intuitive and interpretable, enabling better video understanding and model diagnosis.…”

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

“…• We show that multi-modal self-supervision, applied to both source and unlabelled target data, can be used for Figure 2: Fine-grained action datasets [8,17,26,28,38,42,46,47,50], x-axis: number of action segments per environment (ape), y-axis: dataset size divided by ape. EPIC-Kitchens [8] offers the largest ape relative to its size.…”