An Investigation of Different Methodologies for Rating Product Satisfaction

Abstract

Usability researchers and product designers stand to benefit by minimizing costs (e.g., time, overhead, recruitment costs) related to usability research. Several potential discount methods for collecting subjective data exist. This study compared four methods of collecting satisfaction data using the After-Scenario Questionnaire (ASQ). Use-Then-Measure (rating immediately after use), was used as a comparison group for the performance of three other methods: Retrospective (delayed measurement based on past use), Prospective (judging a product before use), and Watching-Others (evaluating a product based on video footage). Users were asked to use three different products (a website, a can opener, and a digital timer) and rate their satisfaction with the products using the ASQ. The Retrospective method produced mean ASQ scores that were statistically indistinguishable from the average ASQ scores produced by Use-Then-Measure. Both the Prospective and Watching-Others conditions generated mean ASQ scores higher than Use-Than-Measure. Our results, consistent with previous research, support Retrospective assessment as an alternative for collecting subjective ratings of usability. While our results did not support Watching-Others or Prospective methods, more research is needed before completely ruling them out as viable methods.

Rating Product Satisfaction by Watching Others Using Video

“…Due to COVID-19 in-person research restrictions at the time of data collection, Use-Then-Measure benchmark data from Robertson and Kortum (2019) was used for the can opener and timer. Usability testing for the website was conducted remotely via Zoom.…”

“…Watching Others Using Video has been used to evaluate workload (Hale & Long, 2017), support cognitive walkthroughs (Gabrielli et al, 2005), and execute heuristic evaluations of robot interactions (Weiss et al, 2010). Though advantageous, previous work found that Watching Others Using Video results in positively biased usability ratings compared to Use-Then-Measure (Robertson & Kortum, 2019). That is, these usability ratings significantly deviate from the true score (as captured by the Use-Then-Measure method) and thus are not accurate representations of the products in question.…”

“…Product satisfaction is a primary concern in the design of consumer products. As satisfaction declines, so does product use; consumers will then seek alternative products (Kortum, 2016;Robertson & Kortum 2019). User satisfaction also influences and catalyzes brand loyalty and subsequent customer recommendations (Casaló et al, 2008;Leninkumar, 2017).…”

Developing and Validating Measurement Scales During Pandemic Conditions: A Case Study with the Scale for Habitat Usability

“…The major drawback to this approach is that prospective usability ratings tend to be higher on average than ratings based on actual use (Robertson & Kortum, 2019;Robertson & Kortum, 2020). However, this drawback has little bearing on this study.…”

“…Additionally, it has been suggested that prior experience with a product (e.g., expertise) may lessen this drawback (Robertson & Kortum, 2019;Robertson & Kortum, 2020). Visual media have also been found to improve the quality of inspection methods.…”