Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

Abstract

Vision-language pre-training has been an emerging and fast-developing research topic, which transfers multi-modal knowledge from rich-resource pre-training task to limited-resource downstream tasks. Unlike existing works that predominantly learn a single generic encoder, we present a pre-trainable Universal Encoder-DEcoder Network (Uni-EDEN) to facilitate both vision-language perception (e.g., visual question answering) and generation (e.g., image captioning). Uni-EDEN is a two-stream Transformer-based structure, consisting of three modules: object and sentence encoders that separately learns the representations of each modality and sentence decoder that enables both multi-modal reasoning and sentence generation via inter-modal interaction. Considering that the linguistic representations of each image can span different granularities in this hierarchy including, from simple to comprehensive, individual label, a phrase, and a natural sentence, we pre-train Uni-EDEN through multi-granular vision-language proxy tasks: Masked Object Classification, Masked Region Phrase Generation, Image-Sentence Matching, and Masked Sentence Generation. In this way, Uni-EDEN is endowed with the power of both multi-modal representation extraction and language modeling. Extensive experiments demonstrate the compelling generalizability of Uni-EDEN by fine-tuning it to four vision-language perception and generation downstream tasks.

Defending Privacy Inference Attacks to Federated Learning for Intelligent IoT with Parameter Compression

“…Deep learning (DL) [1], as one of the most popular machine learning methods driven by big data, has been widely studied and employed in various felds and different scenarios such as face detection [2], social networks [3,4], natural language process [5,6], speech technology [7][8][9], detection of network anomalies [10,11], and multimodal learning [12][13][14].…”

A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning

“…Recent strides in vision-language pretraining have exerted a profound impact on image captioning research [28][29][30]. Zhou et al [28] present a unified vision-language pre-training (VLP) model for image captioning, employing a Transformer network for both encoding and decoding, with pre-training on large image-text pairs.…”

“…This novel approach, leveraging textual augmentation, demonstrates improved performance in various vision-language tasks, notably in image captioning, by refining representation quality and model convergence. Li et al [30] introduce Uni-EDEN, a Universal Encoder-Decoder Network for vision-language tasks, focusing on multi-granular vision-language pre-training. This approach notably enhances multimodal reasoning and language modeling capabilities, advancing both perception and generation aspects in image captioning.…”

SNP-S3: Shared Network Pre-Training and Significant Semantic Strengthening for Various Video-Text Tasks

“…Owing to successful applications of pre-training methods in NLP [7,43] and CV [5,22], more and more researchers attempt to explore this "Pre-training & Finetuning" paradigm in the video-text field [25,33], which has achieved remarkable performance gain in various downstream video understanding tasks, such as video-text re- trieval [10,38,53], video question answering [44,55,59], and video reasoning [6,15,42,54,57]. There are two mainstream paradigms in current video-text pre-training methods: the feature-level paradigm and the pixel-level one.…”