AppMonitor: A tool for recording user actions in unmodified Windows applications

Abstract

Interface designers and human-computer interaction (HCI) researchers need to understand how users interact with systems in order to improve them. Several strategies are commonly used to gain such insights, including controlled experiments run in research labs, field studies, and contextual inquiry (Holtzblatt & Jones, 1993). Client-side logging of user actions is another technique that has several advantages over other methods: It is highly scalable, since software is easily disseminated over the Web; it is low-cost to the participants and experimenters, because, once it has been installed, the logs are generated and stored automatically; it readily allows longitudinal analysis of weeks, months, or years of use; and it allows finegrain analysis of subsecond events and low-level actions. The two primary disadvantages of client-side logging are, first, that it cannot record the user's high-level goals or context; and second, that developing logging software can be prohibitively complex.The difficulty in implementing logging software has discouraged and impaired its use. For example, in analyzing Web navigation behavior, participants have been asked to abandon their preferred proprietary Web browsers in favor of customized logging versions of open-source software (e.g., Tauscher & Greenberg, 1997) or to use "roll your own" systems (e.g., Kellar, Hawkey, Inkpen, & Watters, in press). Clearly, logging analyses should measure interaction with the users' preferred software rather than with a surrogate that supports a subset of the real system features or that presents the features in a different manner.In this article, we describe AppMonitor, a tool that allows logging of user interaction with unmodified applications running under Microsoft Windows. Once installed, AppMonitor requires no intervention from the user. It silently records all user actions of interest (from low-level mouse movements to the selection of items in combo/ dialog boxes), and periodically uploads them to a remote Web server. AppMonitor's logs provide a terse but semantically rich description of interaction. As well as capturing a description of low-level user events, such as mouse movement and button-or keypresses, it can also record semantic aspects of interaction, discriminating between the different interface components used and their modes. This capability distinguishes AppMonitor's logs from the video logs generated by screen recorders (such as TechSmith's Camtasia Studio, www.techsmith.com), because video logs require human interpretation to determine the semantics connecting their key logs and mouse logs with the video.To assist researchers in developing their own logging software, we explain how we implemented AppMonitor. The following section reviews related work on tools supporting event logging. We then describe AppMonitor's underlying architecture and implementation and give an example of the output it generates. Finally, we provide some preliminary analyses of the total output from our six beta testers, who used AppMonitor to record ...

Supporting Novice to Expert Transitions in User Interfaces

“…Typically, these APIs form part of the accessibility framework that assists development of software for users with special needs. Several researchers describe tools that assist with monitoring and logging interaction with unaltered software systems (e.g., Kukreja et al [2007], Alexander et al [2008], and Morgan et al [2013]), and these methods can also be extended to enable some control over proprietary systems, as demonstrated by the Blur and Skillometer systems (Section 5.1), which ran on Microsoft Windows and Apple Mac OSX. However, these techniques have several limitations, including access to only a subset of controls, reduced system responsiveness, and increased frequency of software crashes, which make them impractical for studies of real use.…”

Plug-in Tutor Agents: Still Pluggin’

“…The AppMonitor system (Alexander et al 2008) uses a combination of accessibility interfaces and monitoring of low-level device drivers to monitor applications. Patina (Matejka et al 2013) uses similar inputs, as well as window position information, to understand and present a summary of user actions.…”

Artivity: Documenting Digital Art Practice

“…It can also be done by tracking the use of software commands. For example this was shown by Alexander, Cockburn & Lobb (2008). The latter option allows easier reconstruction of the interaction in terms of the process used to reach a result, although the meaning of that interaction is again lost.…”