A Hybrid Model Words-Driven Approach for Web Product Duplicate Detection

Abstract

The detection of product duplicates is one of the challenges that Web shop aggregators are currently facing. In this paper, we focus on solving the problem of product duplicate detection on the Web. Our proposed method extends a state-of-the-art solution that uses the model words in product titles to find duplicate products. First, we employ the aforementioned algorithm in order to find matching product titles. If no matching title is found, our method continues by computing similarities between the two product descriptions. These similarities are based on the product attribute keys and on the product attribute values. Furthermore, instead of only extracting model words from the title, our method also extracts model words from the product attribute values. Based on our experimental results on real-world data gathered from two existing Web shops, we show that the proposed method, in terms of F1-measure, significantly outperforms the existing state-of-the-art title model words method and the well-known TF-IDF method.

Multi-component similarity method for web product duplicate detection

“…When duplicates are indeed detected, the algorithm directly proceeds to the next product combination. However, in the evaluation of the algorithms [3], it is shown that TMWM has only a precision of 0.556, whereas HSM has a precision of 0.741. TMWM thus declares a false duplicate in 44% of the cases, and HSM further improves this to 26% for the set of products declared non-duplicates by TMWM.…”

“…The Hybrid Similarity Method (HSM) in [3] builds on TMWM of [15] by including additional information given by the product attributes. As a first step, TMWM is used to determine if the two products under consideration are duplicates based on the product title.…”

“…In this paper, we also aim to both extend and improve a state-of-the-art duplicate detection algorithm for products on the Web, called the Hybrid Similarity Method [3]. This method builds on a procedure that uses model words in the product titles to detect duplicates, called the Title Model Words Method [15], and is designed to cluster products from only two Web shops.…”

“…API's and techniques such as text mining could be used to extract useful information from the Web, to be employed in the duplicate detection process. Currently, the field of duplicate detection on the Web is mostly focused on pair-wise duplicate detection [3,15]. However, since the Web is crowded with Web shops selling identical items, future duplicate detection algorithms must be able to handle more than two products.…”

Enriching Product Ads with Metadata from HTML Annotations

“…However, in real applications the problem of products matching is highly imbalanced. The approach is first extended in [1], using a hybrid similarity method. Later, the method is extended in [2], where hierarchical clustering is used for matching products from multiple web shops, using the same hybrid similarity method.…”