TUHOI: Trento Universal Human Object Interaction Dataset

Abstract

This paper describes the Trento Universal Human Object Interaction dataset, TUHOI, which is dedicated to human object interactions in images. 1 Recognizing human actions is an important yet challenging task. Most available datasets in this field are limited in numbers of actions and objects. A large dataset with various actions and human object interactions is needed for training and evaluating complicated and robust human action recognition systems, especially systems that combine knowledge learned from language and vision. We introduce an image collection with more than two thousand actions which have been annotated through crowdsourcing. We review publicly available datasets, describe the annotation process of our image collection and some statistics of this dataset. Finally, experimental results on the dataset including human action recognition based on objects and an analysis of the relation between human-object positions in images and prepositions in language are presented.

Joint Detection and Recounting of Abnormal Events by Learning Deep Generic Knowledge

“…The events in (i) and (j) that correspond to 'throwing a bag' and 'pushing a bicycle' include the interaction of objects, which could be captured with our approach. Since large datasets for object interactions is available [5,21,24], our framework could be extended to learn such knowledge, and this could be another direction for future work.…”

Combining Geometric, Textual and Visual Features for Predicting Prepositions in Image Descriptions

“…Although they also present alternative models that use text corpora for descriptions that are more human-like, they are limited to verbs and do not cover prepositions. Le et al (2014) examine prepositions modifying human actions (verbs), and conclude that these relate to positional information to a certain extent. Other related work include training classifiers for prepositions with spatial relation features to improve image segmentation and detection (Fidler et al, 2013); this work is however limited to four prepositions.…”

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

“…As a secondary evaluation dataset for the Multimodal Translation task, we collected and translated a set of image descriptions that potentially contain ambiguous verbs. We based our selection on the VerSe dataset (Gella et al, 2016), which annotates a subset of the COCO (Lin et al, 2014) and TUHOI (Le et al, 2014) images with OntoNotes senses for 90 verbs which are ambiguous, e.g. play.…”