What Is the Best Size for Matrix-Style Questions in Online Surveys?

Abstract

Across two studies, we aimed to determine the row and column size in matrix-style questions that best optimizes participant experience and data quality for computer and mobile users. In Study 1 (N ¼ 2,492), respondents completed 20 questions (comprising four short scales) presented in a matrix grid (converted to item-by-item format on mobile phones). We varied the number of rows (5, 10, or 20) and columns (3, 5, or 7) of the matrix on each page. Outcomes included both data quality (straightlining, item skip rate, and internal reliability of scales) and survey experience measures (dropout rate, rating of survey experience, and completion time). Results for row size revealed dropout rate and reported survey difficulty increased as row size increased. For column size, seven columns increased the completion time of the survey, while three columns produced lower scale reliability. There was no interaction between row and column size. The best overall size tested was a 5 Â 5 matrix. In Study 2 (N ¼ 2,570), we tested whether the effects of row size replicated when using a single 20-item scale that crossed page breaks and found that participant survey ratings were still best in the five-row condition. These results suggest that having around five rows or potentially fewer per page, and around five columns for answer options, gives the optimal survey experience, with equal or better data quality, when using matrix-style questions in an online survey. These recommendations will help researchers gain the benefits of using matrices in their surveys with the least downsides of the format.

How to Design, Implement, and Analyse a Survey

“…The response options can include anything measured on an ordinal scale, such as frequency or importance, but matrix questions are not advised for more demanding count or financial data, except for a very short list of items. To reduce satisficing behaviour (section 3.8), the number of question items in a matrix in all delivery formats (printed, online, face-to-face and CATI) should be limited to no more than nine question items and preferably fewer (Grady et al, 2019).…”

“…The best completion rates (no item non-responses) and data quality for matrix questions viewed on a computer or tablet are obtained when there are less than ten question items and preferably no more than five to seven (Grady et al, 2019). Reassuringly, the type of device used to complete an online survey has very little effect on data quality (Sandorf et al, 2022;Tourangeau et al, 2018).…”

A comparison of question order effects on item-by-item and grid formats: visual layout matters

“…Grids can be very diverse in their visual appearance. Responses may be affected by the number of items (Grady et al, 2019;Mavletova et al, 2018;Toepoel et al, 2009), the length of the scale (Liu and Cernat, 2018;Mavletova et al, 2018), the labeling, the size of the grid, the horizontal-vertical layout (Revilla and Couper, 2017), and more. Clearly, no single study can involve all the possible combinations.…”

Using Eye-Tracking Methodology to Study Grid Question Designs in Web Surveys

“…The amount of information and the effort required to answer the items increases with the size of a grid. An increasing number of items in rows and more response entry fields in columns imply a larger matrix, making navigation more difficult (Couper et al 2013;Grady et al 2019). With a cognitively more demanding task, respondents may get more easily confused and distracted, thereby increasing their actual or perceived response burden (Couper et al 2013;Liu and Cernat 2018).…”

“…An alternative approach of presenting items sharing the same response scale is the item-by-item design, where items are presented as stand-alone questions (Couper et al 2013). Between those two extreme points of presenting a series of items are design choices that break the series of target items in smaller groups; for instance, by presenting a set of ten target items in two grids with five items (e.g., Couper et al 2001;Grady et al 2019). Each of these formats has benefits and drawbacks.…”