Investigating self-reporting behavior in long-term studies

Abstract

Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects' forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in order to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the subjects' reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly overestimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and burdensome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources. With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data collected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommendations for maximizing the reliability of results when conducting large-scale, long-term app usage studies.

Contextual experience sampling of mobile application micro-usage

“…We did not find any potential link between participants' demographics and micro-usage behavior, but we must acknowledge that this might be due to our sample diversity. Despite our effort in designing the ESM to capture as much context as possible, the answer-choices could have limited users' micro-usage context [22]. Nonetheless, through our studies, we were able for the first time to specifically investigate micro-usage and the context in which it occurs.…”

Towards Leveraging Behavioral Economics in Mobile Application Design

“…Work by Birnbaum (1999) shows that cognitive biases distort people's behavior during user studies to a high degree, resulting in all unwanted type I, II and III errors. Recently, Möller et al (2013) showed that the design of a user study has a great impact on the behavior of the participants. Besides, the context we work in drives us to investigate how intuitive interaction can be explained through neurology and behavioral sciences.…”

Work or leisure? Designing a user-centered approach for researching activity “in the wild”

“…This may lead to participants not recording their activities every time they should. For example, Möller et al [71] reported that only 40-70 % of their self-report questionnaires were completed, depending on study design. Bolger et al [8] remark that participants are responsible for data gathering themselves and therefore need to be more involved and motivated.…”

“…Such methods have to meet special requirements with respect to the privacy of users [63], especially when they deal with tracking of a user's location [5]. While some researchers decided not to take privacy concerns into account [71], other approaches address social and ethical needs of users directly within the overall design process, e.g., value-sensitive design [33]. Khovanskaya et al [57] demonstrate within an empirical design study that current tools for data gathering ''do a poor job of communicating the scope of data collected […] or explicitly acknowledging the values embedded in decisions about which data are collected and how they are reflected back to the user'' [57].…”