An experimental study of countermeasures against threats: real-world effects meet treatment effects

Abstract

The experimental study of positions on policies and measures against various new types of threat is fast becoming a mainstream research practice. In this article we argue as follows: in security studies in particular, there is a risk that the experimental treatment is contaminated by subjects’ previous experience of the real world (‘contamination’), and this may substantially complicate the assessment of the size of the experimental treatment’s causal effect. We discuss ways to decrease the risk of uncontrolled contamination. Using two experimental case studies we show two typical cases of contamination in security studies (one, where the contamination of all treatments was extremely high, and another, where the level of contamination was unknown and might have varied across the experimental groups) and consider what this implies for the substantive results of the experiments. An analysis of contamination should become a routine, especially when reporting security experiments. Supplementary Information The online version contains supplementary material available at 10.1007/s11135-022-01354-4.