Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining

Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval

“…Specifically, a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP) model is proposed for instance-level commodity retrieval, that explicitly injects entity knowledge in both node-based and subgraph-based ways into the multi-modal networks via a self-supervised hybrid-stream transformer, which could reduce the confusion between different object contents, thereby effectively guiding the network to focus on entities with real semantic. Experimental results well verify the efficacy and generalizability of our EGE-CMP, outperforming several SOTA cross-modal baselines like CLIP [1], UNITER [2] and CAPTURE [3].…”

“…The results of experiments on both multi-product retrieval and identical-product retrieval tasks show the superiority of our EGE-CMP over the SOTA cross-modal baselines, such as ViLBERT [7], CLIP [1], UNITER [2], CAPTURE [3] and so on, on all major criteria by a large margin. Moreover, extensive ablation experiments are conducted to demonstrate the generalizability of EGE-CMP and investigate various essential factors of our proposed task.…”

“…With a wide variety of downstream E-commerce applications in the real world, we focus solely on the pretraining of multi-modality data of products. In the general scenarios, multi-modal visionlanguage pre-training model [1][2][3][7][8][9][10][11][12] such as CLIP [1] and VilBert [7] have shown superior performances on the diverse downstream tasks, such as zero-shot classification [13], crossmodal retrieval [14] and open-world detection [15]. They learn visual and textual embedding features from a huge number of imagetext pairs acquired from different sources and have quite generalization capability and robustness.…”

“…We devise a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP) 3 framework to learn the instancelevel feature representations via injecting the entity knowledge with real semantic information to the visual-text alignment. We find our entity-graph significantly benefits the text-image cross-modal alignment.…”

“…The study builds on our prior conference presentation, CAP-TURE [3]. In the version, we add the following features to the conference version: (1) To be suggested, a new goal, identicalproduct retrieval, must be adequately specified.…”