LinkNet: Relational Embedding for Scene Graph

Abstract

Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.

SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering

“…Each scene graph encodes high-level visual concepts, including objects, attributes, and relations, as nodes and edges. This type of structured representation has led to many state-of-the-art models in image captioning [43], image retrieval [17], relationship modeling [40], and image generation [16]. In addition, it promotes the publishing of several datasets containing well-annotated scene graphs, such as GQA [14] and Visual Genome [19], to facilitate research on the use of scene graphs for visual and textual content understanding.…”