Communicating evidence for policy makers in icons and tables- What works? preprint

Abstract

Policy decisions have vast consequences, but there is little empirical research on how best to communicate underlying evidence to decision makers. Groups in diverse fields (e.g., education, medicine, crime) use brief, graphical displays to list policy options, expected outcomes, and evidence quality, to make such evidence easy to assess. However, understanding of these representations is rarely studied. We surveyed experts and non-experts on what information they want and tested their objective comprehension of commonly used graphics. 252 UK residents from Prolific and 452 UK What Works Centre users interpreted the meaning of graphics shown without labels. Comprehension was low (often below 50 per cent). The best-performing graphics used unambiguous, relevant shapes, color cues, and indications of quantity. The participants also reported what types of evidence they wanted and in what detail (e.g., subgroups; different outcomes). Intervention effectiveness and quality were universally paramount, and policy makers also wanted to know the financial costs and other negative consequences. Comprehension and preferences were remarkably consistent between the samples. Groups communicating evidence about policy options can use these results to design summaries, toolkits, and reports for expert and non-expert audiences.

Risk communication in tables versus text: a registered report randomized trial on ‘fact boxes'

“…Third, participants often asked for more information than the data provided, such as the effects on different subgroups (gender, age) or outcome severity, and a few mentioned wanting to know about the population-level effects of individual choices (here: antibiotic resistance and herd immunity from vaccination). These comments are consistent with previous findings that participants regularly want more detail on presented evidence, but it remains unclear how to balance clarity and brevity with comprehensiveness [40]. Broadly, the current insights reinforce the importance of user-centred design during the development of communication materials, in order to find out what outcomes, subgroups and details are desired by the target audience.…”

“…Another option is using icons to graphically express the quality of evidence. A recent study examined which quality of evidence icons were best understood by an expert sample of policy makers and practitioners [40]. Expert and general public users had similar goals and comprehension of icons, and both groups widely misunderstood common icons, e.g.…”

Principles for the production of evidence‐based guidance for conservation actions

“…Writing evidence‐based guidance includes many challenges. Relaying complex and nuanced scientific and technical information into easy‐to‐use formats is complicated (Brick et al, 2018; Brick & Freeman, 2019) and there are a variety of ways in which guidance can be presented depending upon the intended audience. Ensuring guidance is accessible to all is essential.…”

Research Worth Using: (Re)Framing Research Evidence Quality for Educational Policymaking and Practice

“…Policymakers and practitioners face “differing incentives, goals, language, demands and time frames” (Tseng & Nutley, 2014, p. 171). Policymakers attend more closely to financial costs and harmful effects (Brick & Freeman, 2019) and “mobilize information to support an agenda” (Shonkoff, 2000, p. 181). School staff value actionability of concrete procedures and compatibility with local context (Hemsley-Brown & Sharp, 2003; Neal et al, 2018).…”

“…A different set of hidden costs may comprise side effects, induced costs, adverse consequences, or other possible harms (Brick & Freeman, 2019; Green et al, 2009; Zhao, 2017). Some possibilities include psychological constructs such as motivation, creativity, learning orientation, self-efficacy, or other socioemotional learning components (Zhao, 2017).…”