CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval

Yu, Licheng; Sinha, Animesh A.; Wang, Mengjiao MJ; Chen, Hugo; Berg, Tamara L.; Zhang, Ning

doi:10.48550/arxiv.2202.07247

Cited by 3 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The basic architecture of vision-language pre-training (VLP) models is usually composed of a visual embedding module, a textual embedding module and a fusion encoder [16,17]. Of these architectures the image is usually encoded with an off-the-shelf ResNet [5], or Faster-RCNN [13], or Visual Transformer [3] model.…”

Section: Contrastive Loss Unitmentioning

confidence: 99%

Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrieval

Chen¹,

Jin²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Same-style products retrieval plays an important role in e-commerce platforms, aiming to identify the same products which may have different text descriptions or images. It can be used for similar products retrieval from different suppliers or duplicate products detection of one supplier. Common methods use the image as the detected object, but they only consider the visual features and overlook the attribute information contained in the textual descriptions, and perform weakly for products in image less important industries like machinery, hardware tools and electronic component, even if an additional text matching module is added. In this paper, we propose a unified vision-language modeling method for e-commerce same-style products retrieval, which is designed to represent one product with its textual descriptions and visual contents. It contains one sampling skill to collect positive pairs from user click logs with category and relevance constrained, and a novel contrastive loss unit to model the image, text, and image+text representations into one joint embedding space. It is capable of cross-modal productto-product retrieval, as well as style transfer and user-interactive search. Offline evaluations on annotated data demonstrate its superior retrieval performance, and online testings show it can attract more clicks and conversions. Moreover, this model has already been deployed online for similar products retrieval in alibaba.com, the largest B2B e-commerce platform in the world. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval.

show abstract

Section: Contrastive Loss Unitmentioning

confidence: 99%

Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrieval

Chen¹,

Jin²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The newer methods in this domain uses graph neural networks (GNNs) (Liu et al, 2023a(Liu et al, , 2023b for feature aggregation by improving the embeddings, multi-modal representation learning with omni retrieval from text and images (Yu et al, 2022) and performing classification using Bayesian inference for GNNs (Liu et al, 2022). However, the goal of this study is to present a novel architecture that can improve the performance of BERT model for sentence representation learning owing to text classification.…”

Section: Related Workmentioning

confidence: 99%

A novel multi-layer feature fusion-based BERT-CNN for sentence representation learning and classification

Alyoubi,

Alotaibi,

Kumar

et al. 2023

RIA

View full text Add to dashboard Cite

Purpose The purpose of this paper is to describe a new approach to sentence representation learning leading to text classification using Bidirectional Encoder Representations from Transformers (BERT) embeddings. This work proposes a novel BERT-convolutional neural network (CNN)-based model for sentence representation learning and text classification. The proposed model can be used by industries that work in the area of classification of similarity scores between the texts and sentiments and opinion analysis. Design/methodology/approach The approach developed is based on the use of the BERT model to provide distinct features from its transformer encoder layers to the CNNs to achieve multi-layer feature fusion. To achieve multi-layer feature fusion, the distinct feature vectors of the last three layers of the BERT are passed to three separate CNN layers to generate a rich feature representation that can be used for extracting the keywords in the sentences. For sentence representation learning and text classification, the proposed model is trained and tested on the Stanford Sentiment Treebank-2 (SST-2) data set for sentiment analysis and the Quora Question Pair (QQP) data set for sentence classification. To obtain benchmark results, a selective training approach has been applied with the proposed model. Findings On the SST-2 data set, the proposed model achieved an accuracy of 92.90%, whereas, on the QQP data set, it achieved an accuracy of 91.51%. For other evaluation metrics such as precision, recall and F1 Score, the results obtained are overwhelming. The results with the proposed model are 1.17%–1.2% better as compared to the original BERT model on the SST-2 and QQP data sets. Originality/value The novelty of the proposed model lies in the multi-layer feature fusion between the last three layers of the BERT model with CNN layers and the selective training approach based on gated pruning to achieve benchmark results.

show abstract

“…In these schemes, the basic instance is directly the inputted image. To better understand multi-media information like advertisement or commerce topics, recent works [20,32,43] augment image-text pairs or even a flexible combination of text, image and other valuable data like query and click. Com-pared with approaches that exploit the raw input (e.g., image, image-text pairs), the basic instance in DocReL is a semantic entity.…”

Section: Pre-training Tasks In Vrdsmentioning

confidence: 99%

“…(a)) or visual-text pairs[32,43] (Fig. 2 (b)), VRDs offer multi-modal triplets that consist of visual, text, and layout, as shown in Fig.…”

mentioning

confidence: 99%

Relational Representation Learning in Visually-Rich Documents

Li¹,

Zheng²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relational understanding is critical for a number of visually-rich documents (VRDs) understanding tasks. Through multi-modal pre-training, recent studies provide comprehensive contextual representations and exploit them as prior knowledge for downstream tasks. In spite of their impressive results, we observe that the widespread relational hints (e.g., relation of key/value fields on receipts) built upon contextual knowledge are not excavated yet. To mitigate this gap, we propose DocReL, a Document Relational Representation Learning framework. The major challenge of DocReL roots in the variety of relations. From the simplest pairwise relation to the complex global structure, it is infeasible to conduct supervised training due to the definition of relation varies and even conflicts in different tasks. To deal with the unpredictable definition of relations, we propose a novel contrastive learning task named Relational Consistency Modeling (RCM), which harnesses the fact that existing relations should be consistent in differently augmented positive views. RCM provides relational representations which are more compatible to the urgent need of downstream tasks, even without any knowledge about the exact definition of relation. DocReL achieves better performance on a wide variety of VRD relational understanding tasks, including table structure recognition, key information extraction and reading order detection.* Equal contribution. † Contact person.

show abstract

CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval

Cited by 3 publications

References 32 publications

Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrieval

Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrieval

A novel multi-layer feature fusion-based BERT-CNN for sentence representation learning and classification

Relational Representation Learning in Visually-Rich Documents

Contact Info

Product

Resources

About