Studying the impact of the Full-Network embedding on multimodal pipelines

Abstract

The current state of the art for image annotation and image retrieval tasks is obtained through deep neural network multimodal pipelines, which combine an image representation and a text representation into a shared embedding space. In this paper we evaluate the impact of using the Full-Network embedding (FNE) in this setting, replacing the original image representation in four competitive multimodal embedding generation schemes. Unlike the one-layer image embeddings typically used by most approaches, the Full-Network embedding provides a multi-scale discrete representation of images, which results in richer characterisations. Extensive testing is performed on three different datasets comparing the performance of the studied variants and the impact of the FNE on a levelled playground, i.e., under equality of data used, source CNN models and hyper-parameter tuning. The results obtained indicate that the Full-Network embedding is consistently superior to the one-layer embedding. Furthermore, its impact on performance is superior to the improvement stemming from the other variants studied. These results motivate the integration of the Full-Network embedding on any multimodal embedding generation scheme.

Network representation learning method embedding linear and nonlinear network structures

“…(2) Global structural information: global network embedding can obtain more abundant node characteristics [29]. In a realistic network, many legitimate edges are not observed.…”

Text Classification with Noisy Class Labels

“…Text pre-processing is the first step in the text classification pipeline, and can have a big impact on the final performance of the classifier [117]. The common steps of text pre-processing a text document into a clean, standardized form includes lowering cases of all characters, removing certain characters, stemming and lemmatizing words.…”