Design and evaluation of a command recommendation system for software applications

Abstract

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

An Investigation of Metrics for the In Situ Detection of Software Expertise

“…This metric considers the frequency of actual usage. Command frequency has been used previously to profile user expertise (Li, Matejka, Grossman, Konstan, & Fitzmaurice, 2011;Matejka, Li, Grossman, & Fitzmaurice, 2009;Rooke, Grossman, & Fitzmaurice, 2011). For UI Expertise, a higher frequency of usage with modifier hotkeys may be a sign of a higher level of expertise (Grossman, Dragicevic, & Balakrishnan, 2007).…”

“…Research on complex applications has shown that any individual user may only use a subset of a software application's commands (McGrenere & Moore, 2000), and some commands have higher relevance to users than others (Li et al, 2011;Matejka et al, 2009). We define component relevance as the level of importance that a component of a software application has to a user, based to that user's day-to-day tasks.…”

“…The collection and analysis of usage metrics from search engines and web browsers are commonly used to understand how people use online systems (Adar, Teevan, & Dumais, 2008;e.g., Teevan, Dumais, & Liebling, 2008, 2010. Numerous recent research projects have also looked at capturing usage logs during software application usage (Akers, Simpson, Jeffries, & Winograd, 2009;Lafreniere, Bunt, Whissell, Clarke, & Terry, 2010;Li et al, 2011; and lower level input activities (Chapuis, Blanch, & Beaudouin-Lafon, 2007;Evans & Wobbrock, 2012;Gajos, Reinecke, & Herrmann, 2012;Hurst, Mankoff, & Hudson, 2008). These previous projects have discussed the benefits and challenges of capturing data from in situ usage logs.…”

Deploying CommunityCommands: A Software Command Recommender System Case Study

“…Determining the right length of this training period is difficult -we wanted the recommendations to start as soon as possible, but only after we reliably know what commands the user is already aware of. To minimize the time needed for training, we ran a pilot test by analyzing data from 27 users (Li et al 2011). On a daily interval, we measured the rate at which new commands were used (had not been previously observed for that user), across a period of 4 weeks ( figure 9).…”

“…This project continued to evolve and we explored the design space of a contextual software command recommender system and completed a six-week user study (Li et al 2011 Product Year assume such assistance would be readily available.…”