The Effect of Question Order on Evaluations of Test Performance: Can the Bias Dissolve?

Bard, Gabriele; Weinstein, Yana

doi:10.1080/17470218.2016.1225108

Cited by 7 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Question-order bias has been found in several studies across disciplines (see, e.g., Auh et al, 2003;Bard & Weinstein, 2017;Bowling & Windsor, 2008;Strack, 1992;Stark et al, 2020;Thau et al, 2021;Van de Walle & Van Ryzin, 2011). The extent to which the bias occurs and its size are described in the literature as dependent on attitude and belief accessibility as well as five other factors: recency of activation, frequency of activation, relation between the questions, ambiguity of the question, and background of the respondent (Thau et al, 2021;Tourangeau et al, 2000).…”

Section: Question-order Biasmentioning

confidence: 99%

“…Prior questions can affect how subsequent questions are answered, resulting in question‐order bias (Thau et al, 2021; Van de Walle & Van Ryzin, 2011). Question‐order bias has the potential to change the results of items themselves (levels) as well as their correlation with other items, even when the items do not seem connected in terms of topic (Bard & Weinstein, 2017; Hjortskov, 2017; Tourangeau et al, 1989). The effects are potentially large, perhaps even outshining the effects of actual public scandals (Thau et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The consistency of question‐order bias in a changing political context

Dooren

Hjortskov

Vadder

et al. 2023

Public Administration

View full text Add to dashboard Cite

Question‐order bias is a well‐known weakness of surveys commonly used in public administration research. However, most research on question‐order bias uses question‐order experiments that are relatively small, performed in one context, and rarely replicated. We carry out six question‐order experiments in six large‐scale Belgian surveys conducted during the COVID‐19 pandemic. All experiments vary whether the respondents see questions regarding the effectiveness of pandemic governance or trust in different actors first. Results show that question‐order effects are real and reasonably consistent across the high‐powered replications, despite the changing political context of the pandemic. However, the direction of the effects largely changes when we flip the order of the trust outcome questions in the last three experiments, which sheds light on an underappreciated point: question‐order bias also seems to exist within batteries of seemingly similar outcome questions.

show abstract

Section: Question-order Biasmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The consistency of question‐order bias in a changing political context

Dooren

Hjortskov

Vadder

et al. 2023

Public Administration

View full text Add to dashboard Cite

show abstract

“…Thus, objective measures may be unaffected by the placement of the perceptions-based questions. This follows a long literature that has studied the effects of question order in surveys across a wide range of fields (Bradburn and Mason, 1964;Bishop and Smith, 2001;Rohwedder et al, 2006;Holbrook et al, 2007;Lumsdaine and Exterkate, 2013;Bard and Weinstein, 2017).…”

Section: Introductionmentioning

confidence: 99%

A Cautionary Tale: An Experiment on the Stability of Business Environment Perceptions in a Firm Survey

Islam¹,

Meza²

2021

Policy Research Working Papers

View full text Add to dashboard Cite

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Policy Research Working Paper 9610Several studies in the literature have adopted attitude or perception-based survey questions to evaluate the business environment and its effect on firms. The Enterprise Surveys of the World Bank are not an exception. In the case of the Enterprise Surveys, these questions involve rating an element of the business environment at the end of each section of the survey instrument. Such questions are often used but sometimes are inconsistent with responses elicited on the experience of the firm over a specific timeframe-experience-based questions. The literature is mixed as to whether perception-based questions are susceptible to anchoring or context effects. In this study, an experiment is set up to explore whether perceptions of the business environment are stable or vulnerable to the ordering of questions in the Enterprise Surveys questionnaire. The experiment entails randomizing the placement order of the perception-based questions at the end of a section or at the beginning of the survey. Significant question-order effects are uncovered only for perceptions of corruption and business licensing and permits but not the other elements, after accounting for a variety of factors. The study recommends that analysis in these two areas should go beyond perception-based questions and verify their findings with experience-based questions.This paper is a product of the Global Indicators Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world.

show abstract

What's Context Got to Do with It? Comparative Difficulty of Test Questions Influences Metacognition and Corrected Scores for Formula‐scored Exams

Arnold

Graham

Hollingworth-Hughes

2017

Applied Cognitive Psychology

View full text Add to dashboard Cite

On formula-scored exams students receive points and penalties for correct and incorrect answers, respectively, but they can avoid the penalty by withholding incorrect answers. However, test-takers have difficulty strategically regulating their accuracy and often set an overly conservative metacognitive response bias (e.g., Higham, 2007). The current experiments extended these findings by exploring whether the comparative difficulty of surrounding test questions (i.e., easy vs. hard)-a factor unrelated to the knowledge being tested-impacts metacognitive response bias for medium-difficulty test questions. Comparative difficulty had no significant influence on participants' ability to choose correct answers for medium questions, but it did affect willingness to report answers and confidence ratings. This difference carried over to corrected scores (scores after penalties are applied) when comparative difficulty was manipulated within-subjects: Scores were higher in the hard condition. Results are discussed in terms of implications for interpreting formula-scored tests and underlying mechanisms of performance.

show abstract

The Effect of Question Order on Evaluations of Test Performance: Can the Bias Dissolve?

Cited by 7 publications

References 26 publications

The consistency of question‐order bias in a changing political context

The consistency of question‐order bias in a changing political context

A Cautionary Tale: An Experiment on the Stability of Business Environment Perceptions in a Firm Survey

What's Context Got to Do with It? Comparative Difficulty of Test Questions Influences Metacognition and Corrected Scores for Formula‐scored Exams

Contact Info

Product

Resources

About