Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues

Abstract

It is a common practice for recent works in vision language cross-modal reasoning to adopt a binary or multi-choice classification formulation taking as input a set of source image(s) and textual query. In this work, we take a sober look at such an "unconditional" formulation in the sense that no prior knowledge is specified with respect to the source image(s). Inspired by the designs of both visual commonsense reasoning and natural language inference tasks, we propose a new task termed "Premise-based Multi-modal Reasoning" (PMR) where a textual premise is the background presumption on each source image. The PMR dataset contains 15,360 manually annotated samples which are created by a multi-phase crowd-sourcing process. With selected high-quality movie screenshots and human-curated premise templates from 6 pre-defined categories, we ask crowdsource workers to write one true hypothesis and three distractors (4 choices) given the premise and image through a cross-check procedure. Besides, we generate adversarial samples to alleviate the annotation artifacts and double the size of PMR. We benchmark various stateof-the-art (pretrained) multi-modal inference models on PMR and conduct comprehensive experimental analyses to showcase the utility of our dataset.

Examining Consistency of Visual Commonsense Reasoning based on Person Grounding

“…Person-Centric Vision-Language Task The person-centric vision-language task (Zellers et al, 2019;Dong et al, 2022;Cui et al, 2021;You et al, 2022), is mainly based on grounding references to a person; therefore, person-centric visual grounding ability is a crucial component. VCR (Zellers et al, 2019) is a task that answers commonsensical questions about the people depicted in an image.…”

“…In the fields of vision and language, several datasets (Zellers et al, 2019;Park et al, 2020;Lei et al, 2020;Dong et al, 2022;You et al, 2022) that focus on the reasoning about individuals using visual commonsense knowledge have been proposed. In one of the notable datasets, VCR (Zellers et al, 2019), the models are required to provide answers with justifications for commonsense questions related to the individuals in the given images.…”