Knowledge-Embedded Routing Network for Scene Graph Generation

Abstract

To understand a scene in depth not only involves locating/recognizing individual objects, but also requires to infer the relationships and interactions among them. However, since the distribution of real-world relationships is seriously unbalanced, existing methods perform quite poorly for the less frequent relationships. In this work, we find that the statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous, and thus well address the unbalanced distribution issue. To achieve this, we incorporate these statistical correlations into deep neural networks to facilitate scene graph generation by developing a Knowledge-Embedded Routing Network. More specifically, we show that the statistical correlations between objects appearing in images and their relationships, can be explicitly represented by a structured knowledge graph, and a routing mechanism is learned to propagate messages through the graph to explore their interactions. Extensive experiments on the large-scale Visual Genome dataset demonstrate the superiority of the proposed method over current state-ofthe-art competitors.

Attentive Gated Graph Neural Network for Image Scene Graph Generation

“…After the definition of the scene graph, the possibility of generating a scene graph from an image I can be composed by three components as similar to [28]:…”

“…Ref. [28] uses a gated graph neural network to model the fully connected scene graph. Each node will be affected equally by all other nodes in the graph.…”

“…In this way, each node can aggregate messages from the other nodes and transfer its message to the other nodes in the meantime, enabling interactions among all nodes in the graph. After T timesteps, we can obtain the final hidden state for each node, which can be represented by a set of subnode [28], we use a fully connected layer that takes the initial hidden state and final hidden state as input to compute the output feature for each subnode…”

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition

“…Inspired by the current graph propagation works [18,1,27,3], we adopt a gated recurrent update mechanism to propagate message through the graph and learn contextualized node-level features. Specifically, for each node v c ∈ V, it has a hidden state h t c at timestep t. In this work, as each node corresponds to a specific category and our model aims to explore the interactions among the semanticspecific features, we initialize the hidden state at t = 0 with the feature vector that relates to the corresponding category, formulated as…”

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

“…2)MSDN[30], IMP[62], TFR[20], MOTIFS[73], Graph-RCNN[65], GPI[17], KER[6] are joint inference models, which adopt message passing to encode the context. All these models are optimized by XE based training objective.…”