ERBlox: Combining Matching Dependencies with Machine Learning for Entity Resolution

Bahmani, Zeinab; Bertossi, Leopoldo; Vasiloglou, Nikolaos

doi:10.1007/978-3-319-23540-0_27

Cited by 8 publications

(11 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The database community has proposed declarative matching and resolution rules to express the domain knowledge about matching and resolution [5,7,9,13,24,26,32,33,51]. Matching dependencies (MD) are a popular type of such declarative rules, which provide a powerful method of expressing domain knowledge on matching values [8,10,23,24,38]. Let S be the schema of the original database and R 1 and R 2 two distinct relations in S. Attributes A 1 and A 2 from relations R 1 and R 2 , respectively, are comparable if they share the dame domain.…”

Section: Matching Dependenciesmentioning

confidence: 99%

Learning Over Dirty Data Without Cleaning

Picado

Davis

Termehchy

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Real-world datasets are dirty and contain many errors. Examples of these issues are violations of integrity constraints, duplicates, and inconsistencies in representing data values and entities. Learning over dirty databases may result in inaccurate models. Users have to spend a great deal of time and effort to repair data errors and create a clean database for learning. Moreover, as the information required to repair these errors is not often available, there may be numerous possible clean versions for a dirty database. We propose DLearn, a novel relational learning system that learns directly over dirty databases effectively and efficiently without any preprocessing. DLearn leverages database constraints to learn accurate relational models over inconsistent and heterogeneous data. Its learned models represent patterns over all possible clean instances of the data in a usable form. Our empirical study indicates that DLearn learns accurate models over large real-world databases efficiently.

show abstract

Section: Matching Dependenciesmentioning

confidence: 99%

Learning Over Dirty Data Without Cleaning

Picado

Davis

Termehchy

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…We equip LACE with a 'dynamic' and 'global' semantics. In line with approaches to ER based on matching dependencies (MDs) [9,20,22] and extensions thereof, such as relational MDs [4,6], LACE adopts a dynamic semantics in which rule bodies are evaluated on induced instances resulting from applying the already 'derived' merges. It is thanks to the dynamic nature of the semantics that we obtain a collective yet justifiable framework, in which merges can trigger further merges, possibly in a recursive fashion, while still being able to trace back the origins of each merge.…”

Section: Introductionmentioning

confidence: 99%

LACE: A Logical Approach to Collective Entity Resolution

Cima

Gutiérrez-Basulto

2022

Proceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems

View full text Add to dashboard Cite

In this paper, we revisit the problem of entity resolution and propose a novel, logical framework, LACE, which mixes declarative and procedural elements to achieve a number of desirable properties. Our approach is fundamentally declarative in nature: it utilizes hard and soft rules to specify conditions under which pairs of entity references must or may be merged, together with denial constraints that enforce consistency of the resulting instance. Importantly, however, rule bodies are evaluated on the instance resulting from applying the already 'derived' merges. It is the dynamic nature of our semantics that enables us to capture collective entity resolution scenarios, where merges can trigger further merges, while at the same time ensuring that every merge can be justified. As the denial constraints restrict which merges can be performed together, we obtain a space of (maximal) solutions, from which we can naturally define notions of certain and possible merges and query answers. We explore the computational properties of our framework and determine the precise computational complexity of the relevant decision problems. Furthermore, as a first step towards implementing our approach, we demonstrate how we can encode the various reasoning tasks using answer set programming.

show abstract

“…ER is a significant and common data cleaning problem, and it consists of detecting data duplicate representations for the same external entities, and merging them into single representations [ 43 ]. This problem can be applied to many different domains, such as deduplication in databases [ 44 ], duplicate detection in

data or hierarchical data [ 45 ], cross-document co-reference resolution methods and tools [ 46 ], blocking techniques [ 43 , 47 ], bug reports [ 48 ], customer recognition [ 31 ], and E-health [ 49 ]. Most of the existing studies have been validated using real-world datasets, but very few of them have applied their proposal in a real case in the industry [ 42 ].…”

Section: Introductionmentioning

confidence: 99%

“…Authors in [ 31 ] used Levenshtein Edit Distance for feature selection in combination with weights based on the Inverse Document Frequency (IDF) of terms. Matching dependencies, a new class of semantic constraints for data quality and cleaning, has been shown to be profitably integrated with traditional machine learning methods for developing classification models for ER [ 43 ].…”

Section: Introductionmentioning

confidence: 99%

Entropic Statistical Description of Big Data Quality in Hotel Customer Relationship Management

Serrano

Ballestero

Muñoz-Romero

et al. 2019

Entropy

View full text Add to dashboard Cite

Customer Relationship Management (CRM) is a fundamental tool in the hospitality industry nowadays, which can be seen as a big-data scenario due to the large amount of recordings which are annually handled by managers. Data quality is crucial for the success of these systems, and one of the main issues to be solved by businesses in general and by hospitality businesses in particular in this setting is the identification of duplicated customers, which has not received much attention in recent literature, probably and partly because it is not an easy-to-state problem in statistical terms. In the present work, we address the problem statement of duplicated customer identification as a large-scale data analysis, and we propose and benchmark a general-purpose solution for it. Our system consists of four basic elements: (a) A generic feature representation for the customer fields in a simple table-shape database; (b) An efficient distance for comparison among feature values, in terms of the Wagner-Fischer algorithm to calculate the Levenshtein distance; (c) A big-data implementation using basic map-reduce techniques to readily support the comparison of strategies; (d) An X-from-M criterion to identify those possible neighbors to a duplicated-customer candidate. We analyze the mass density function of the distances in the CRM text-based fields and characterized their behavior and consistency in terms of the entropy and of the mutual information for these fields. Our experiments in a large CRM from a multinational hospitality chain show that the distance distributions are statistically consistent for each feature, and that neighbourhood thresholds are automatically adjusted by the system at a first step and they can be subsequently more-finely tuned according to the manager experience. The entropy distributions for the different variables, as well as the mutual information between pairs, are characterized by multimodal profiles, where a wide gap between close and far fields is often present. This motivates the proposal of the so-called X-from-M strategy, which is shown to be computationally affordable, and can provide the expert with a reduced number of duplicated candidates to supervise, with low X values being enough to warrant the sensitivity required at the automatic detection stage. The proposed system again encourages and supports the benefits of big-data technologies in CRM scenarios for hotel chains, and rather than the use of ad-hoc heuristic rules, it promotes the research and development of theoretically principled approaches.

show abstract

ERBlox: Combining Matching Dependencies with Machine Learning for Entity Resolution

Cited by 8 publications

References 43 publications

Learning Over Dirty Data Without Cleaning

Learning Over Dirty Data Without Cleaning

LACE: A Logical Approach to Collective Entity Resolution

Entropic Statistical Description of Big Data Quality in Hotel Customer Relationship Management

Contact Info

Product

Resources

About