Database tomography for information retrieval

Abstract

Database tomography is an information extraction and analysis system which operates on textual databases. Its primary use to date has been to identify pervasive technical thrusts and themes, and the interrelationships among these themes and sub-themes, which are intrinsic to large textual databases. Its two main algorithmic components are multiword phrase frequency analysis and phrase proximity analysis. This paper shows how database tomography can be used to enhance information retrieval from large textual databases through the newly developed process of simulated nucleation. The principles of simulated nucleation are presented, and the advantages for information retrieval are delineated. An application is described of developing, from Science Citation Index and Engineering Compendex, a database of journal articles focused on near-Earth space science and technology.

Systematic acceleration of radical discovery and innovation in science and technology

“…The parent article to the present Appendix (Kostoff, 1997a) focused on the use of computational linguistics imbedded in an iterative relevance feedback procedure. The final product is a query that will retrieve documents with two aggregate characteristics; the maximum number of relevant documents will be retrieved, and the ratio of relevant to non-relevant documents will be very large; i.e., the signalto-noise ratio will be large.…”

“…Three iterative steps were required; each step required the technical expert(s) to read many hundreds of the retrieved records in order to identify those that were relevant and non-relevant . Then, computational linguistics analyses (Kostoff, 1997a) were performed on both the relevant and non-relevant records to identify phrase patterns and relationships characteristic of the relevant records and the non-relevant records. Substantial time and judgement were required to select the appropriate phrases unique to the relevant records and the non-relevant records, and then modify the query accordingly using the key phrases identified.…”

“…Simulated Nucleation (Kostoff et al, 1997a), the name given to the form of Database Tomography adapted to information retrieval, derives in concept from the growth of materials. A core nucleus is developed, the properties of this nucleus are identified, and then similar material is added onto the nucleus as time develops until the desired amount of material is obtained.…”

“…The second is a semi-automated approach, using computer-based clustering techniques for separating the relevant from non-relevant records (e.g., see Hearst (1996) and Zamir (1999) for examples of clustering approaches to separate relevant from non-relevant documents), but still requiring manual identification of candidate query terms from computational linguistics analysis of each separate document category. The initial reported effort was on the first approach (Kostoff et al, 1997a); recent efforts have focused on the second approach (Kostoff et al, 2005a). …”

Duplicate publication and ‘paper inflation’ in the fractals literature

“…For the present study, the SCI database (including both the Science Citation Index and the Social Science Citation Index) was used. The approach used for query development was the first author's iterative relevance feedback concept of Simulated Nucleation [Kostoff et al, 1997].…”