Task-based interaction with an integrated multilingual, multimedia information system

Abstract

This paper describes a formative evaluation of an integrated multilingual, multimedia information system, a series of user studies designed to guide system development. The system includes automatic speech recognition for English, Chinese, and Arabic, automatic translation from Chinese and Arabic into English, and query-based and profile-based search options. The study design emphasizes repeated evaluation with the same (increasingly experienced) participants, exploration of alternative task designs, rich qualitative and quantitative data collection, and rapid analysis to provide the timely feedback needed to support iterative and responsive development. Results indicate that users presented with materials in a language that they do not know can generate remarkably useful work products, but that integration of transcription, translation, search and profile management poses challenges that would be less evident were each technology to be evaluated in isolation.

Extending sense‐making models with ideas from cognition and learning theories

“…We had available data from participants performing assigned intelligence analysis tasks (information tasks to support someone else's work task) in the context of the formative evaluation of the Rosetta system. This is a news (print, radio and television) retrieval system covering both broadcast and Web‐based news in several languages (Arabic, Chinese, English, and Spanish), automatically transcribed and translated into English as necessary (Zhang, et al, 2007). Automatic translation and transcription introduced a number of errors, which made intelligence analysis challenging.…”

Cross-language Information Retrieval

“…The problem of interaction of IR and the user also exists in CLIR. There has been far less reported research on interaction design for CLIR than on monolingual retrieval, and the work that has been done (e.g., [135,143,201,97]) points to a number of open research questions. Interactive CLIR systems could, for example, use information about preferred languages of the original documents, in which the user might put more weight on different languages, based on the topic of interest.…”

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…However, crosslanguage retrieval is an important and relevant task to many, most notably government intelligence officers. In addition, there are research programs whose goal is to bring together numerous technologies, including summarization, multimedia and cross-language into a single system and some preliminary reports of user-centered evaluations of these systems [300]. This paper did not address evaluation issues associated with adaptive systems and other systems designed to personalize interactions.…”