An extended vector space model for information retrieval with generalized similarity measures : theory and applications.

Abstract

We present an Extended Vector Space Model (EVSM) for information retrieval, endowed with a new set of similarity functions. Our model considers records as multisets of tokens. A token weight function maps records into a real vectors. Using this vector representation we define a p-norm of a record and pairwise conjunction and disjunction operations on records. These operations prompt consistent extensions of published set-based similarity functions and yield new p distance-based similarities. We demonstrate that some well-known similarities form a subset of the new functions resulting from particular choices of token weights and p-values. In so doing, 3 we establish the equivalence of the corresponding information retrieval models with a properly augmented vector space model. The performance of the extended similarity measures is compared by solving an entity matching (EM) problem for two types of benchmark datasets. Among other things, our results show that the new similarity functions perform particularly well on tasks involving matching of records by keywords.The EVSM served as foundation for mathematically rigorous definition of EM problem. We developed a supervised EM framework that interprets the EM as the combinatorial optimization problem of finding the maximum weight matching in a weighted bipartite graph connecting records from two databases, also known as Linear Sum Assignment Problem (LSAP). Casting of EM problems into LSAP offers valuable practical and theoretical advantages. There are efficient algorithms that solve LSAP in polynomial time. Availability of such algorithms reduces the task of solving the EM problem to computing weights for the edges of the bipartite graph connecting the records from the databases. This allowed focusing efforts on the development of robust and flexible methodologies for the estimation of the similarity between records and led to the notion of an optimal similarity function (OSF) for MMIR problems. The OSF is sought as a linear combination of similarity functions for the common relation attributes. Solution of a suitably defined quadratic program using training data defines the weights in the linear combination. Computational studies using the Abt-Buy e-commerce set and publication databases comprising of research articles in cloud computing, antennas and information retrieval areas confirm the robustness of our approach.4