Scaling Egocentric Vision: The "Equation missing" Dataset

Abstract

First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

“…Dataset. Our previous work, EPIC Kitchens [8], offers a unique opportunity to test domain adaptation for finegrained action recognition, as it is recorded in 32 environments. Similar to previous works for action recognition [14,19], we evaluate on pairs of domains.…”

“…In testing, as in [58], we use an average over 5 temporal windows, equidistant within the segment. We use the RGB and Optical Flow frames provided publicly [8]. The output of F is the result of the final average pooling layer of I3D, with 1024 dimensions.…”

“…However, due to the difficulty in collecting and annotating such fine-grained actions, many datasets collect long untrimmed sequences. These contain several fine-grained actions from a single [43,50] or few [8,47] environments. Figure 2 shows the recent surge in large-scale finegrained action datasets.…”

“…• We show that multi-modal self-supervision, applied to both source and unlabelled target data, can be used for Figure 2: Fine-grained action datasets [8,17,26,28,38,42,46,47,50], x-axis: number of action segments per environment (ape), y-axis: dataset size divided by ape. EPIC-Kitchens [8] offers the largest ape relative to its size.…”

Seeing and Hearing Egocentric Actions: How Much Can We Learn?

“…In particular, our contributions can be summarized as follows: (i) We provide an extensive evaluation and comparison with published methods of the proposed multimodal architecture on the EPIC-Kitchens dataset [12] (ii) In addition to action performance, we provide for the first time a detailed results on the object and verb components. The rest of the paper is organized as follows.…”