Subject indexing and citation indexing— part II: An evaluation and comparison

Shaw, Jr. W. M.

doi:10.1016/0306-4573(90)90047-6

Cited by 13 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent research also draws similar conclusions that retrieval performance could be improved if both subject and citation searching were used (Chapman & Subramanyam, 1981;McCain, 1989;McCain, White, & Griffith, 1987;Pao, 1986;Shaw, 1990;White, Griffith, Cowen, Selinger, & Steere, 1984). Nevertheless, previous works comparing the relative retrieval effectiveness of citation and subject approaches were either based on small samples of searches or conducted on specially constructed document collections of limited size.…”

Section: Introductionmentioning

confidence: 75%

“…These files were usually narrowly focused in terms of subjects. Furthermore, investigations on the use of citation as a retrieval strategy were often based on experimental algorithms unavailable to the average online searcher (Bichteler & Eaton, 1980;Chapman & Subramanyam, 1981;Pao & Fu, 1985;Shaw, 1990;White et al, 1984). A review of relevant studies has been included in the pilot study report (Pao & Worthen, 1989).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Term and citation retrieval: A field study

Pao

1993

Information Processing & Management

View full text Add to dashboard Cite

Abstract-Therelative efficacy of searching by terms and by citations is investigated with real searches collected in health sciences libraries. The objective is to seek evidence to confirm or refute findings from a controlled pilot study, and to understand the factors at work in operational search environments.Overall confirmation was found. In both the pilot and field studies, the improvement of the odds that overlap items retrieved would be relevant or partially relevant was truly astounding. If an item was retrieved from both MEDLINE@ and SCISEARCHB, it was six times more likely that it would be relevant or partially relevant as opposed to being not relevant, and 8.4 times more likely for definitely relevant retrievals. In the field setting, citation searching was able to add an average of 24% recall to traditional subject retrieval. Term or citation searching from the open literature produced lower precision results. Attempts to identify distinguishing characteristics in queries which might benefit most from additional citation searches proved to be inconclusive.In spite of the obvious gain shown by citation searching, online access of citation databases has been hampered by their relative high cost.The reported work is an extension of a pilot study of the characteristics and retrieval effectiveness of two subject searching modes. The two search approaches available on commercial bibliographic databases are semantic retrieval based on text words, assigned keywords and descriptors, and pragmatic retrieval based on citations. The earlier study was an experiment performed on a data file with narrow subject focus (Pao & Worthen, 1989). The database was constructed such that its documents content was retrievable by descriptors and text words as well as by cited references. Thus, direct comparison of the retrieval results was possible from parallel searches using appropriate terms and citations on identical queries.Citation searching in the control setting was found to add an average of 14% of relevant documents to a search. This confirms earlier indications from other researchers that citation searching complements searching by terms (McCain, 1989;Pao, 1986;Salton, 1971). A logical follow-up question is whether these results could be of practical use to the online searcher. Can the searcher expect similar results when searches are done on commercially available databases? For example, what is the percentage of MEDLINE@ search topics which could benefit from a citation search? One also wonders how generalizable and how stable the findings are if a sample of real searches was collected from libraries where a wide variety of topics was searched. While only a few overlap items were found, these common documents tended to be highly relevant to the search topic. What are the odds that retrieved items derived from both types of search methods are relevant? Is a higher yield from a citation search related to specific types of topics? Obviously, knowledge of this type could be useful to the online searcher.This second stu...

show abstract

Section: Introductionmentioning

confidence: 75%

Section: Introductionmentioning

confidence: 99%

Term and citation retrieval: A field study

Pao

1993

Information Processing & Management

View full text Add to dashboard Cite

show abstract

“…representations and distinguish between the representations that include subject descriptions and the representation composed of references and citations. Similar patterns characterize the constituent subject and citation representations (Shaw, 1990~) and an automatically generated subject description of documents in a distinct environment (Shaw, 1990a). occurs when all documents are connected and the entire collection is retrieved for each query.…”

Section: Effectiveness Measuresmentioning

confidence: 87%

“…In a previous investigation, measures of clusterbased retrieval effectiveness were computed for four subject representations and two citation representations in the cystic fibrosis (CF) Database (Shaw, 1990~). Results reveal consistent patterns from which optimal performance values can be identified, when single-link clustering is employed.…”

Section: Introductionmentioning

confidence: 85%

“…The magnitude of DWO varies as a function of the representation and identifies an optimal compromise between complete descriptions of the documents and descriptions that discriminate between documents relevant to different queries. The trend reverses for DW > DWO and reveals an optimal performance level for one partition of the document subject representations (Shaw, 1990~). These distinctions are influenced by collection statistics associated with the constituent representations.…”

Section: Effectivenessmentioning

confidence: 93%

See 1 more Smart Citation

Subject and citation indexing. Part II: The optimal, cluster-based retrieval performance of composite representations

Shaw

1991

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

Measures of cluster-based retrieval effectiveness are computed for five composite representations in the cystic fibrosis (CF) Document Collection. The composite representations are constructed from combinations of two subject representations, based on Medical Subject Headings and subheadings, and two citation representations, consisting of the complete list of cited references and a comprehensive list of citations for each document. Experimental retrieval results are presented as a function of the exhaustivity and similarity of the composite representations and reveal consistent patterns from which optimal performance levels can be identified. The optimal performance values provide an assessment of the absolute capacity of each composite representation to associate documents relevant to the same query and discriminate between documents relevant to different queries in single-link hierarchies. The optimal performance values for all composite representations are completely comparable and are superior to the optimal performance of constituent representations. Optimal performance consistently occurs at low levels of exhaustivity. Exhaustive composite representations that include subject descriptions produce the lowest levels of performance; retrieval results derived from random structures are comparable to the observed results. The effectiveness of the exhaustive representation composed of references and citations is materially superior to the effectiveness of exhaustive composite representations that include subject descriptions.

show abstract

The Monte Carlo method and the evaluation of retrieval system performance

Burgin

1999

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

The ability to distinguish between acceptable and unacceptable levels of retrieval performance and the ability to distinguish between significant and non‐significant differences between retrieval results are important to traditional information retrieval experiments. The Monte Carlo method is shown to represent an attractive alternative to the hypergeometric model for identifying the levels at which random retrieval performance is exceeded in retrieval test collections and for overcoming some of the limitations of the hypergeometric model. The Monte Carlo method produces low performance thresholds for the individual test collections that are very similar to the thresholds derived by the hypergeometric model, both at the test collection level and at the individual query level. In addition, the Monte Carlo method is much less computer‐intensive than the hypergeometric model, can be used with measures of retrieval effectiveness that take the rank order of the retrieved documents into consideration, can be used to derive the probability of obtained results, and can be used to determine the statistical significance of difference between two or more retrieval results. The ability to use the Monte Carlo method to derive the probability of obtained results and to compare two or more retrieval results makes it possible to determine more accurately how well retrieval systems operate under specific conditions and, in conjunction with the presentation of individual query results, makes it possible to determine whether relationships between query characteristics and retrieval system performance exist. Understanding these relationships should lead to improvements in the effectiveness of retrieval systems.

show abstract

Subject indexing and citation indexing— part II: An evaluation and comparison

Cited by 13 publications

References 29 publications

Term and citation retrieval: A field study

Term and citation retrieval: A field study

Subject and citation indexing. Part II: The optimal, cluster-based retrieval performance of composite representations

The Monte Carlo method and the evaluation of retrieval system performance

Contact Info

Product

Resources

About