Learning Structured Inference Neural Networks with Label Relations

Abstract

Images of scenes have various objects as well as abundant attributes, and diverse levels of visual categorization are possible. A natural image could be assigned with finegrained labels that describe major components, coarsegrained labels that depict high level abstraction, or a set of labels that reveal attributes. Such categorization at different concept layers can be modeled with label graphs encoding label information. In this paper, we exploit this rich information with a state-of-art deep learning framework, and propose a generic structured model that leverages diverse label relations to improve image classification performance. Our approach employs a novel stacked label prediction neural network, capturing both inter-level and intra-level label semantics. We evaluate our method on benchmark image datasets, and empirical results illustrate the efficacy of our model.

Floor-SP: Inverse CAD for Floorplans by Sequential Room-Wise Shortest Path

“…Graph reconstruction: Graph structure inference has been a popular field of study in Computer Vision, for instance, inferring a human body pose [3] or the semantic relationships of categories [14,28]. In these problems, the graph topology is defined over the label space, common to all the instances (e.g., a head is always connected to a body).…”

Multi-label Zero-Shot Learning with Structured Knowledge Graphs

“…The model is further extended to allow for soft or probabilistic relations between labels [13]. Later, [21] introduced Structured Inference Neural Network (SINN). Inspired by the idea of Recurrent Neural Network (RNN) [20,41], positive correlation and negative correlation between labels are derived for bidirectionally propagating information between concept layers, which further improves the classification performance; Focusing on single-label activity recognition, [12] view both activity of input image and actions of each person in that image as a graph, and utilize RNN to update the observed graph for activity prediction.…”

“…[10] introduced a graph representation that enforces certain relations between label concepts. [21] employed recurrent neural networks (RNN) [20,41] to model positive and negative correlations between different concept layers. More recently, [30] extended neural networks for graphs [40,27] to efficiently learn a model that reasons about different types of relationships between class labels by propagating information in a knowledge graph.…”

Feedback-Prop: Convolutional Neural Network Inference Under Partial Evidence

“…We use a CNN + Softmax classifier as our first Baseline, and as a second baseline a CNN + Softmax classifier that uses true values for coarse categories in the form of a binary indicator vector as additional input to the classifier (Baseline + PL). Similar baselines were used in Hu et al [17]. Additionally, we re-implement the Structured Inference Neural Network (SINN) of Hu et al [17] which outputs three lev-size of known labels as partial evidence (Conv5, 10, 13) against the amount of known labels in the COCO multi-label image annotation task: the more the labels, the higher the performance.…”

“…Similar baselines were used in Hu et al [17]. Additionally, we re-implement the Structured Inference Neural Network (SINN) of Hu et al [17] which outputs three lev-size of known labels as partial evidence (Conv5, 10, 13) against the amount of known labels in the COCO multi-label image annotation task: the more the labels, the higher the performance.…”