Implementation of the substring test by hashing

Abstract

A technique is described for implementing the test which determines if one string is a substring of another. When there is low probability that the test will be satisfied, it is shown how the operation can be speeded up considerably if it is preceded by a test on appropriately chosen hash codes of the strings. . This work was done under AEC contract number AT (30-1) 1480 V.This note describes a fast implementation of the test which determines if one string contains a specified substring. The scheme makes use of hashing techniques and of the ability to do many Boolean operations in parallel on a standard computer. It is most useful when the same strings are being tested repeatedly, and when the probability of finding the substring is small.The scheme makes use of three ideas. The first is that if the search is likely to be unsuccessful, it can usefully be preceded by a computationally faster test for necessary but not sufficient conditions that the substring be found. A simple example of such a test is the comparison of the lengths of the strings, but this is usually too weak a test to be useful.The second is that a string can be represented by the set of its substrings, and in particular by the set of its substrings of a specified length. In general such a representation is not unique, but it does preserve the substring property in the sense that, if one string has another string as a substring, the set of substrings of the first will include the set of substrings of the second. Because of the lack of uniqueness, the reverse is not true, of course.The third is that a set S can be represented by a binary string blb2b3 ... bm in which a value of 1 for b~ indicates that S contains at least one element of the set E~. In general such a representation is not unique, unless each E~ contains exactly one element and each possible element is contained in some E~. However, it preserves the subset property in the sense that, if set $1 is a subset of set $2, the binary string representing $2 will have ones in all positions where the string representing $1 has ones. Accordingly, consider a string S to be represented by a binary string blb2b3 • • • bm constructed as follows:

A signature file scheme based on multiple organizations for indexing very large text databases

“…As a result, signature file methods have been proposed for applications for which these structures occur, such as prolog databases. Signature file methods have been proposed for multikey retrieval (Roberts, 1979;Sacks-Davis & Ramamohanarao, 1983;Sacks-Davis, 1985), text retrieval (Faloutsos & Christodoulakis, 1985;Faloutsos, 1985;Harrison, 1971) prolog systems (Colomb, 1985;Ramamohanarao & Shepherd, 1986;Wada, Morita, Yamazaki, Yamashita, Miyazaki, & Itoh, 1987;Wise & Powers, 1984), office systems Faloutsos & Christodoulakis, 1987;Rabitti & Zizka, 1984), statistical databases (Wong, Liu, Olken, Rotem, & Wong, 1985), and filtering methods (Bloom, 1970;McIiroy, 1982). Signature file methods are well suited to hardware implementation (Al!en et al, 1986;Berra et al, 1987).…”

An n-gram hash and skip algorithm for finding large numbers of keywords in continuous text streams

“…Harrison 41 introduced the notion of signatures for locating a passage of text containing a given keyword. In advance of any queries, the text is preprocessed, building a signature (based on n-grams) for each convenient block of text, such as a line.…”

Hashing — the key to rapid pattern matching

“…A signature is some encoding of the salient structural features of an object to provide an economical test of whether t matches s. Signatures have been used to enhance searching in text-editors [9].…”

“…As will be explained in the following sections, we use "hashing" techniques in a variety of ways [8,9]: to characterize features of patterns and structures to be matched, so as to decrease the number of matches to be attempted; and to store and retrieve only a single unique copy of any structure, decreasing A template is an arbitrary object "t" used to represent a class of objects. A template is a generator of a set of object strings S={s}, each element, "s", of which is formed by replacing the marker(s) in "t" by arbitrary object(s); we say "t generates s".…”