Semantic indexing for a complete subject discipline

Abstract

As part of the Illinois Digital Library Initiative (DLI) project we developed "scalable semantics" technologies. These statistical techniques enabled us to index large collections for deeper search than word matching. Through the auspices of the DARPA Information Management program, we are developing an integrated analysis environment, the Interspace Prototype, that uses "semantic indexing" as the foundation for supporting concept navigation. These semantic indexes record the contextual correlation of noun phrases, and are computed generically, independent of subject domain.Using this technology, we were able to compute semantic indexes for a subject discipline. In particular, in the summer of 1998, we computed concept spaces for 9.3M MEDLINE bibliographic records from the National Library of Medicine (NLM) which extensively covered the biomedical literature for the period from 1966 to 1997. In this experiment, we first partitioned the collection into smaller collections (repositories) by subject, extracted noun phrases from titles and abstracts, then performed semantic indexing on these subcollections by creating a concept space for each repository. The computation required 2 days on a 128-node SGI/CRAY Origin 2000 at the National Center for Supercomputer Applications (NCSA). This experiment demonstrated the feasibility of scalable semantics techniques for large collections. With the rapid increase in computing power, we believe this indexing technology will shortly be feasible on personal computers.

Transaction clustering of web log data files using genetic algorithm

“…In the exclusive set, the largest domain is Artificial Intelligence with 856375 web pages while the largest domain in the inclusive set is Operating Systems with 634544 web pages. A similar exercise was performed in the biological sciences in [8]. In our experiment, the exclusive partition set was used to build the subject repositories.…”

Meta-data Extraction and Query Translation. Treatment of Semantic Heterogeneity

“…[6,7,8] Such methods have also been investigated in the "Interspace" 4 prototype. [9,10] The main principles of this approach are discussed in section 3.2.…”

“…[14] Full-text terms are obtained by tokenising the full-text of an Internet document, eliminating stop words, and stemming the remaining terms using a Porter stemmer. 9 For weighting the terms the inverse document frequency is used. [15] The full-text is then indexed with full-text terms having a weight greater than a certain minimum threshold.…”

Monitoring of Quality of Life in Congestive Heart Failure Populations

“…Locating similar cohorts requires the use of sophisticated statistical matching techniques, such as self-organizing maps 64 or concept co-occurrence. 65 However, the health care providers need only use the matching software, not understand the algorithms that are implemented. The underlying technology would be embedded within the health care infrastructure for Internet health monitors.…”

“…A major use of such a national database is to support physicians in prescribing treatments for a patient, by examining treatment patterns of cohorts of similar patients in the database. Locating similar cohorts requires the use of sophisticated statistical matching techniques, such as self‐organizing maps 64 or concept co‐occurrence 65 . However, the health care providers need only use the matching software, not understand the algorithms that are implemented.…”