Semantic Understanding of Scenes Through the ADE20K Dataset

Abstract

Semantic understanding of visual scenes is one of the holy grails of computer vision. Despite efforts of the community in data collection, there are still few image datasets covering a wide range of scenes and object categories with pixel-wise annotations for scene understanding. In this work, we present a densely annotated dataset ADE20K, which spans diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. Totally there are 25k images of the complex everyday scenes containing a variety of objects in their natural spatial context. On average there are 19.5 instances and 10.5 object classes per image. Based on ADE20K, we construct benchmarks for scene parsing and instance segmentation. We provide baseline performances on both of the benchmarks and re-implement the state-ofthe-art models for open source. We further evaluate the effect of synchronized batch normalization and find that a reasonably large batch size is crucial for the semantic segmentation performance. We show that the networks trained on ADE20K are able to segment a wide variety of scenes and objects 1 .

Object representations in the human brain reflect the co-occurrence statistics of vision and language

“…To accomplish this, we needed a large dataset of natural images in which all object occurrences were labeled. We took advantage of the recently created ADE20K database, which contains 22,210 annotated scenes in which every object has been manually labeled by an expert human annotator 25 . One approach for characterizing the co-occurrence statistics of this dataset would be to simply construct a matrix of co-occurrence frequencies for all pairwise comparisons of objects.…”

“…In the field of computational linguistics, there is a long history of modeling word co-occurrence data in language corpora with dense, lower dimensional representations 26 . This modeling framework, known as distributional semantics, has proved highly useful because 25 , which contains 22,210 images in which every pixel is associated with an object label provided by an expert human annotator. An adaptation of the word2vec machine-learning algorithm for distributional semantics-which we call object2vec-was applied to this corpus of image annotations to model the statistical regularities of object-label cooccurrence in a large sample of real-world scenes.…”

“…Our work directly addresses this issue by explicitly modeling the statistics of object co-occurrence in the visual environment and relating these statistics to object-evoked fMRI responses. We overcame the challenge of objectively quantifying the real-world statistics of object context by leveraging a large dataset of densely labeled images that was originally developed for training semantic segmentation models in computer vision 25 . We analyzed the annotations for these images using object2vec, which is a modified version of the word2vec algorithm from computational linguistics 23,24 .…”

“…This dataset contains 22,210 scene images that have been densely annotated by human observers such that every object within every scene is segmented and labelled. Across the entire scene image set, there are 3,148 unique object segmentation labels 25 . ADE20K is ideal for our purposes because it provides a large and diverse set of image annotations that we can use to model the co-occurrence statistics of objects in natural images.…”

Task-specific vision models explain task-specific areas of visual cortex

“…134 Therefore, in this analysis, we investigate the case if units corresponding to the free 135 space show a higher correlation with the behavior and brain RDMs than the readout 136 layer of the VGG scene-parse . The readout layer of the VGG scene-parse consists of 151 137 channels with 150 channel each containing an output corresponding to a particular class 138 in the ADE20k [26] dataset and 1 channel corresponding to the background. Therefore, 139 it is straightforward to separate specific category activation from the readout layer.…”