Learning Functional Distributional Semantics with Visual Data

Abstract

Functional Distributional Semantics is a recently proposed framework for learning distributional semantics that provides linguistic interpretability. It models the meaning of a word as a binary classifier rather than a numerical vector. In this work, we propose a method to train a Functional Distributional Semantics model with grounded visual data. We train it on the Visual Genome dataset, which is closer to the kind of data encountered in human language acquisition than a large text corpus. On four external evaluation datasets, our model outperforms previous work on learning semantics from Visual Genome

Visual Instruction Tuning with Polite Flamingo

“…However, this is inefficient due to the low bandwidth of natural language communication: given the diversity of real-world visual tasks, describing all of the potential task-relevant information within a single image requires a huge amount of language tokens. Therefore, many efforts opt to connect compact latent visual representations through a dense connector by visual instruction tuning, such as MiniGPT-4 (Zhu et al 2023), LLaVA (Liu, Emerson, and Collier 2022), Multimodal-GPT (Gong et al 2023), LLaMA-Adapter (Zhang et al 2023), Otter ), mPLUG-Owl (Ye et al 2023), InstructBLIP (Dai et al 2023). These models use linear projectors or perceivers as the connector between visual models and LLM, thus having a much larger information bandwidth compared to those prompt-based natural language communications.…”

“…However, what data is optimal for training these connectors to ensure that they propagate visual information faithfully is unclear. Existing attempts include generating self-instruct (Wang et al 2022b) data (i.e., LLaVA (Liu, Emerson, and Collier 2022)), using image-text captioning datasets (e.g., COCO (Chen et al 2015), SBU (Ordonez, Kulkarni, and Berg 2011), CC-3M (Sharma et al 2018)), and unifying downstream visionlanguage datasets (e.g., VQA and visual reasoning datasets). Although GPT-4 generated LLaVA dataset enjoy very high quality, its scale remains insufficient, and it could not encourage fine-grained vision-language alignment, as it does not "make V in VQA matter" (Goyal et al 2017).…”

“…After carefully filtering out lowquality results and hallucinations, we obtain a high-quality yet large-scale visual instruction tuning dataset PF-1M, and use it to tune a multi-modal LLM. (Liu, Emerson, and Collier 2022) performs multi-modal self-instruct (Wang et al 2022b) using GPT-4, which has high API cost and limited visual groundedness; InstructBLIP (Dai et al 2023) directly uses learn raw annotations, and thus suffer from multi-modal alignment tax; M 3 IT ) and MIMIC-IT (Li et al 2023a) employed ChatGPT-based rewriters, while we train a Polite Flamingo to rewrite responses, which enjoys advantages of 1) multi-modality, 2) scalability, and 3) diversity.…”

“…We perform a comprehensive evaluation comparing the resulting visual instruction-tuned model, which we called "Clever Flamingo", with other multi-modal LLMs, including MiniGPT-4 (Zhu et al 2023), LLaVA (Liu, Emerson, and Collier 2022), InstructBLIP (Dai et al 2023), and Otter . In summary, Clever Flamingo outperforms all of these models on detailed image captioning tasks, and only underperforms the InstructBLIP series (Dai et al 2023) on VQA tasks (InstructBLIP uses a 3×heavier visual backbone, 8.6×larger pretraining dataset, and +0.6M more instruction samples).…”