Semantic Data Matching: Principles and Performance

Abstract

Automated and real-time management of customer relationships requires robust and intelligent data matching across widespread and diverse data sources. Simple string matching algorithms, such as dynamic programming, can handle typographical errors in the data, but are less able to match records that require contextual and experiential knowledge. Latent Semantic Indexing (LSI) (Berry et al. 1995;Deerwester et al. 1990) is a machine intelligence technique that can match data based upon higher order structure, and is able to handle difficult problems, such as words that have different meanings but the same spelling, are synonymous, or have multiple meanings. Essentially, the technique matches records based upon context, or mathematically quantifying when terms occur in the same record.A study of LSI indicated that it could handle abbreviations, short record sizes, and typographical errors in business name data, as well as match synonyms and reveal higher order structure in the data, such as matching financial institutions though no words are shared. It was found that shared terms among documents affected LSI performance negatively. Thus, preprocessing to remove shared terms, which tend to be common or generic words, numerals, or abbreviations, improves LSI performance for semantic data matching. Some of these shared terms, however, contain information that can be used for record classification, which is revealed by representing these terms as a network. Thus, though LSI can match records based on contextual semantic information, for short records, like business Y.

Hybrid Approaches for Data Cleaning in Data Warehouse

“…In this approach we will be using three techniques. They are: 1) PNRS (Near Miss Strategy) 2) Transitive Closure 3) Semantic Approach based on Learning Systems Here we will be using Semantic Data Cleaning algorithm [6] along with modified versions of PNRS and Transitive Closure. The Semantic Approach will be using the learning system to clean the data.…”

A Comparative Analysis of Data Cleaning Approaches to Dirty Data

“…Semantic Data Matching Principles [11] can be applied to the data along with the above to get better results. The work in [5] clearly explains how this problem can be avoided by keeping a unique consistent name for the city based on the semantic similarity between the attribute values e.g.…”