Advantages, Challenges and Limitations of Audit Experiments with Constituents

Abstract

Audit experiments examining the responsiveness of public officials have become an increasingly popular tool used by political scientists. While these studies have brought significant insight into how public officials respond to different types of constituents, particularly those from minority and disadvantaged backgrounds, audit studies have also been controversial due to their frequent use of deception. Scholars have justified the use of deception by arguing that the benefits of audit studies ultimately outweigh the costs of deceptive practices. Do all audit experiments require the use of deception? This article reviews audit study designs differing in their amount of deception. It then discusses the organizational and logistical challenges of a UK study design where all letters were solicited from MPs’ actual constituents (so-called confederates) and reflected those constituents’ genuine opinions. We call on researchers to avoid deception, unless necessary, and engage in ethical design innovation of their audit experiments, on ethics review boards to raise the level of justification of needed studies involving fake identities and misrepresentation, and on journal editors and reviewers to require researchers to justify in detail which forms of deception were unavoidable.

Misinformation in Experimental Political Science

“…4 Concerns about subject disengagement and overextraction are particularly pertinent in the case of "elite field experiments" or "audit experiments," where potential participants are savvier to the deceptive practices used by researchers (Landgrave 2020, 490)not to mention, there are fewer of them. Misinformation is a common feature of such experiments (Bischof et al 2021) and may produce unique process costs that extend beyond the effects on the individual taking the survey (Desposato 2022). 5 Some theories emphasize "hot" or otherwise "affectively charged" cognition (Lodge and Taber 2013).…”

Narratives and expert information in agenda‐setting: Experimental evidence on state legislator engagement with artificial intelligence policy

“…In particular, we aimed to satisfy three main ethical principles: reducing deception, minimizing harm, and minimizing burden (Butler & Broockman, 2011). Regarding deception, Bischof et al (2022) urge researchers employing audit studies of public officials to minimize deception where possible, particularly the most problematic forms, 'identity' and 'misinformation' deception. In contrast, they note it is less feasible to avoid 'activity' and 'motivation' deception when the experimental design entails participants may not have a full understanding of study purposes.…”

Introduction to Symposium: Experiments with Politicians: Ethics, Power, and the Boundaries of Political Science

“…Bischof et al (2021) discuss the advantages and limitations of using confederates and eliminating identity deception, in detail. Confederates are MPs’ constituents who are recruited to the research project and express genuine opinions to politicians (see also Butler et al, 2012; Breunig et al, 2020; de Vries et al, 2016).…”

“…Precautions are likely to further increase in the wake of the murder of the MP David Amess in October 2021, as staff will be more cautious about hearing from people who are not constituents. As Bischof et al (2021) acknowledge, two other types of deception, motivation and activity deception, cannot be eliminated without increasing the probability of inducing the type of bias that audit experiments are meant to counter in the first place. One practical limitation of this method is that large numbers of confederates within each constituency are needed, which means that confederate studies are more costly to implement and difficult to scale up.…”

“…One practical limitation of this method is that large numbers of confederates within each constituency are needed, which means that confederate studies are more costly to implement and difficult to scale up. Rather than as a problem, some contributors to this symposium see increasing the costs of audit studies as an important means of improving ethical standards (Bischof et al 2021). Butler and Desposato (2021) propose offering politicians donations, either in money or time, to a pre-registered charity to offset some of the costs or harms caused by audit studies.…”