To be most useful, evaluation requires detailed observation and effective analysis of a full spectrum of system use. We have developed an approach and architecture for in-depth data collection and analysis of all use of a visualization system. User interface components in a large visualization and analysis platform automatically record user actions, and can restore previous system states on demand. Audio and text annotations are collected and indexed to states, allowing users to find a comment and restore the system state in which they made it; then explore actions before and after. History is visible as data; so a variety of visual displays and analysis techniques may be used to develop insights about the user's experience. States of any part of the interface may be analyzed separately. Actions are categorized in a taxonomy as the user interface is built, allowing comparison of similar patterns in all tools. History data can co-exist with other data during data exploration, supporting further individual or group data exploration.
To be most useful, evaluation metrics should be based on detailed observation and effective analysis of a full spectrum of system use. Because observation is costly, ideally we want a system to provide in-depth data collection with allied analyses of the key user interface elements. We have developed a visualization and analysis platform [1] that automatically records user actions and states at a high semantic level [2 and 3], and can be directly restored to any state. Audio and text annotations are collected and indexed to states, allowing users to comment on their current situation as they work, and/or as they review the session. These capabilities can be applied to usability evaluation of the system, describing problems they encountered, or to suggest improvements to the environment. Additionally, computed metrics are provided at each state [3, 4, and 5]. We believe that the metrics and the associated history data will allow us to deduce patterns of data exploration, to compare users, to evaluate tools, and to understand in a more automated approach the usability of the visualization system as a whole.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.