Abstract. IR research has a strong tradition of laboratory evaluation of systems. Such research is based on test collections, pre-defined test topics, and standard evaluation metrics. While recent research has emphasized the user viewpoint by proposing user-based metrics and non-binary relevance assessments, the methods are insufficient for truly user-based evaluation. The common assumption of a single query per topic and session poorly represents real life. On the other hand, one well-known metric for multiple queries per session, instance recall, does not capture early (within session) retrieval of (highly) relevant documents. We propose an extension to the Discounted Cumulated Gain (DCG) metric, the Session-based DCG (sDCG) metric for evaluation scenarios involving multiple query sessions, graded relevance assessments, and open-ended user effort including decisions to stop searching. The sDCG metric discounts relevant results from later queries within a session. We exemplify the sDCG metric with data from an interactive experiment, discuss how the metric might be applied, and present research questions for which the metric is helpful.
Digital libraries allow information access to be integrated into work processes rather than separated from them, but also have the potential to overwhelm users with excessive or irrelevant information, impairing their performance rather than improving it. With the opportunity to create new models of what a library is and how it can be used comes the challenge of improving our understanding of its patrons, their work, and the circumstances under which they perform it. In this article we offer an overview of our experiences using observational methods to learn about one class of users, expert clinicians treating patients in hospital settings. We describe the evolution of our understanding of the users and their informational tasks, and how this evolving understanding is guiding our efforts to create digital library technology. The multidisciplinary composition of our team has enriched our observations and improved the validity of our analysis and interpretations. The multiple observation methods we have employed, including "thinkaloud" scenarios in the laboratory, participant observation in the field, key informant interviews, and focus group sessions, have enabled us to enrich the data set, gain greater insight, and verify findings with informants. The relatively tight cycle of observation, analysis, development, and repeat observation has enabled us to iteratively and more rapidly refine our "user model" and "task model," improving, we hope, the usefulness of the technologies we are developing.
Web applications called mash-ups combine information of varying granularity from different, possibly disparate, sources. We describe Mash-o-matic, a utility that can extract, clean, and combine disparate information fragments, and automatically generate data for mash-ups and the mash-ups themselves. As an illustration, we generate a mash-up that displays a map of a university campus, and outline the potential benefits of using Mash-o-matic.Mash-o-matic exploits superimposed information (SI), which is new information and structure created in reference to fragments of existing information. Masho-matic is implemented using middleware called the Superimposed Pluggable Architecture for Contexts and Excerpts (SPARCE), and a query processor for SI and referenced information, both parts of our infrastructure to support SI management. We present a high-level description of the mash-up production process and discuss in detail how Mash-o-matic accelerates that process.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.