A scaleable technique for best-match retrieval of sequential information using metrics-guided search

Abstract

A new technique is described for retrieving information by finding the best match or matches between a textual 'query' and a textual database. The technique uses principles of beam search with a measure of probability to guide the search and prune the search tree. Unlike many methods for comparing strings, the method gives a set of alternative matches, graded by the 'quality' of the matching achieved.For any one sequence of hits between a query and a database, the probability measure is an estimate of the probability that the observed configuration, or better, could have occurred by chance. This probability is an inverse measure of the redundancy between the query and the database. The new technique is embodied in a software simulation called SP21 which runs on a conventional computer.Examples are presented showing best-match retrieval of information from a textual database.Analytic and empirical evidence is presented showing that, in a serial processing environment, the search technique has time complexity of O( Q : D ) where Q is the size of the query in characters and D is the size of the database in characters.The technique lends itself well to parallel processing. In that kind of environment, the time complexity of the process should approach O( Q ), depending on how well the parallelism is applied. The space complexity of the process is O( D ).Planned developments of these ideas in the future are described and discussed.

Towards an intelligent database system founded on the SP theory of computing and cognition

“…At the heart of the SP system is a version of dynamic programming (see Sankoff and Kruskall (1983)) that allows the system to find 'good' full and partial matches between patterns (Wolff, 1994). 3 This allows the system to recognise objects and patterns in a 'fuzzy' manner and to retrieve stored information without the need for an exact match between the retrieval query and any item to be retrieved.…”

Neural Mechanisms for Information Compression by Multiple Alignment, Unification and Search

“…The dynamic programming built into both computer models [Wolff, 1994] gives them a human-like capability for recognising patterns despite errors of omission, commission or substitution.…”

Parallel computing in information retrieval – an updated review

“…In a system without parallelism, pattern matching normally involves the sequential scanning of every document in the system: no index is used. Methods include left hand truncation, variable length don't care (VLDC), a proposed implementation of the "computing as compression theory" SP [49], proximity searches and pre-computed patterns [42,50]. We describe below parallel methods which have been implemented or are proposed for pattern matching algorithms.…”

“…The SP pattern match [49] would use a completely different method. The SP algorithm works by broadcasting each character in the query, from left to right, to each character in the text corpus to make a true or false match.…”