Constructing an index of search goal redefinition through transaction log analysis

Abstract

Purpose -The purpose of this research is to discuss the methodological aspects of a study, the results of which are detailed in a second paper. Design/methodology/approach -Indices of search goal revision (or redefinition) were constructed based on the chronological coding of queries logged on the OCLC FirstSearch service. Queries within search sessions were coded according to their conceptual relationships, based on a set of rules designed to identify possible revisions of search goals, as well as strategic reformulations. Given the same user interface, similar types of user, and large datasets, it was assumed that the number of strategic reformulations would be reasonably uniform across databases, so that the indices could be regarded as indicators of relative amounts of goal revision. Findings -The reliability of the coding used to construct the indices of goal redefinition was tested through parallel coding, and confirmed. Differences between the indices for various databases, on the other hand, were found to be statistically significant, allowing for the conclusion that the indices were able to show differences in the amount of goal redefinition occurring, given the assumption of similar levels of strategic reformulations across databases. Originality/value -Transaction log analysis has mostly focused on syntactic issues; few analyses have examined the semantic content of queries or contextualised individual queries as part of an evolving search session. However, this paper shows that although transaction log analysis is a relatively indirect means of examining users' thoughts and intentions during their interaction with an information retrieval system, logs that provide sufficiently rich data may, in certain controlled situations, be used to provide deeper insights into information retrieval behaviour.

Information activities and tasks

“…This may be exacerbated by the particular situation in which the worker is involved and the technologies that are provided to facilitate the work. Rather than the assembly line process we noted in the beginning, task completion in information work is fluid, with changing goals (see for example, Xie, 1997;Pharo, 2004;Hider, 2007), that affect how and when the particular task is finished.…”

Search goal redefinition through user‐system interaction

“…As such, the indices may be accepted as indices of redefinition. Further discussion as to the indices' validity and reliability is provided in the first part of this report (Hider, 2007).…”

“…Ultimately, all analyses of thought processes require interpretation of the data derived from those thought processes. This particular TLA technique has been discussed at greater length in a separate paper (Hider, 2007).…”

“…This paper forms the second part of a report on a study examining the extent and some possible reasons why such goal redefinition occurs when users interact with a document retrieval system. A fuller account of the methodological aspects of the study has published as the first part of this report (Hider, 2007).…”

Constructing Record Quality Measures Based on Catalog Use

“…When systematic evaluations are reported, the measures of quality are usually based on fairly crude error counts, in much the same way as OPAC studies have often defined search success in terms of numbers of hits (regardless of precision and recall ratios). 1,2 The two most commonly used measures of cataloging quality are "level," that is, the amount of data contained in records, or claimed to be contained, and error rate, that is, the number of errors, or types of error, found per record. Omissions, and partial omissions, are usually counted as errors, so that the error rate can then be determined against a particular standard, for example, the data that should be present in a "full level" record for a particular resource, where "full level" is defined ultimately by the cataloging agency, but usually based on external standards, such as those set out by a cooperative cataloging network.…”