Formative design evaluation of superbook

Abstract

SuperBook is a hypertext browsing system designed to improve the usability of conventional documents. Successive versions of SuperBook were evaluated in a series of behavioral studies. Students searched for information in a statistics text. presented either in conventional printed form or in SuperBook form. The best version of SuperBook enabled students to answer search questions more quickly and accurately than they could with the conventional text. Students wrote higher quality "open-book" essays using SuperBook than they did with the conventional text, and their subjective ratings of the documentation strongly favored SuperBook.This work is a case study of formative design-evaluation. Behavioral evaluation of the first version of SuperBook showed how design factors and user strategies affected search and established baseline performance measures with printed text. The second version of SuperBook was implemented with the goal of improving search accuracy and speed. User strategies that had proved effective in the first study were made very easy and attractive to use. System response time for common operations was greatly improved. Behavioral evaluation of the new SuperBook demonstrated its superiority to printed text and suggested additional improvements that were incorporated into "MiteyBook," a SuperBook implementation for PC-size screens. Search with MiteyBook proved to be approximately 25 percent faster and 25 percent more accurate than that obtained with a conventional printed book.

Scratchable Devices: User-Friendly Programming for Household Appliances

“…Using this kind of formative design-evaluation [7], our interface will be made entirely with the average user in mind. We are particularly interested in verifying that users with no programming experience can, with minimum time and effort, learn to program devices in ways that add novel operations.…”

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…These end products, in turn, are studied along with the subject's interactions with the system. Example studies that have used such methods include Egan et al [83], Halttunen and Järvelin [116], Kelly et al [166], Marchionini and Crane [192] and Vakkari [280].…”

“…This is not technically a self-report measure, but such assessments are usually generated by experts or other subjects who use standardized instruments to make assessments. As noted earlier, example studies that have attempted to assess final products include Egan et al [83], Kelly et al [166], Marchionini and Crane [192] and Vakkari [280].…”

“…Summarization technology is developing rapidly and there is no reason to expect that interactive, user-centered evaluations will not be of interest to researchers in this community. Finally, the success of a recent SIGIR workshop on book search [160] has demonstrated a renewed interest in a domain that is rooted in interactive IR evaluation [83].…”

From paragraph to graph: Latent semantic analysis for information visualization

“…A more optimistic view is that the application of more user testing to understand what does and doesn't help people do what, will steer innovations in more effective directions. Precedent exists, for example in Bellcore's SuperBook, for turning novel information search devices from useless as first designed to order-of-magnitude more effective through iterative empirical usability analysis and redesign (12).…”