Defining and evaluating the Hawthorne effect in primary care, a systematic review and meta-analysis

Abstract

In 2015, we conducted a randomized controlled trial (RCT) in primary care to evaluate if posters and pamphlets dispensed in general practice waiting rooms enhanced vaccination uptake for seasonal influenza. Unexpectedly, vaccination uptake rose in both arms of the RCT whereas public health data indicated a decrease. We wondered if the design of the trial had led to a Hawthorne effect (HE). Searching the literature, we noticed that the definition of the HE was unclear if stated. Our objectives were to refine a definition of the HE for primary care, to evaluate its size, and to draw consequences for primary care research. We designed a Preferred Reporting Items for Systematic reviews and Meta-Analyses review and meta-analysis between January 2012 and March 2022. We included original reports defining the HE and reports measuring it without setting limitations. Definitions of the HE were collected and summarized. Main published outcomes were extracted and measures were analyzed to evaluate odds ratios (ORs) in primary care. The search led to 180 records, reduced on review to 74 for definition and 15 for quantification. Our definition of HE is “an aware or unconscious complex behavior change in a study environment, related to the complex interaction of four biases affecting the study subjects and investigators: selection bias, commitment and congruence bias, conformity and social desirability bias and observation and measurement bias.” Its size varies in time and depends on the education and professional position of the investigators and subjects, the study environment, and the outcome. There are overlap areas between the HE, placebo effect, and regression to the mean. In binary outcomes, the overall OR of the HE computed in primary care was 1.41 (95% CI: [1.13; 1.75]; I2 = 97%), but the significance of the HE disappears in well-designed studies. We conclude that the HE results from a complex system of interacting phenomena and appears to some degree in all experimental research, but its size can considerably be reduced by refining study designs.

Relationship between electronically monitored adherence to direct oral anticoagulants and ischemic or hemorrhagic events after an initial ischemic stroke—A case control study

“…One persistent concern with any indirect method of adherence measure is that the recording does no guarantee the ingestion of the medicine. Moreover, a Hawthorne effect (i.e., an “aware or unconscious complex behavior change in a study environment” [ 46 ]) cannot be excluded when individuals know that they are observed [ 31 , 47 ]. Although all medical research with observational design is prone to the Hawthorne effect, its significance disappears in well-designed RCT [ 46 ] because they are designed to reduce the risk of bias.…”

“…Moreover, a Hawthorne effect (i.e., an “aware or unconscious complex behavior change in a study environment” [ 46 ]) cannot be excluded when individuals know that they are observed [ 31 , 47 ]. Although all medical research with observational design is prone to the Hawthorne effect, its significance disappears in well-designed RCT [ 46 ] because they are designed to reduce the risk of bias. Moreover, it can be assumed that all groups are equally expose to the Hawthorne effect so that its impact on the main outcome may be reduced [ 48 ].…”

An Ergonomic Risk Assessment of Ophthalmology Residents Using the Rapid Entire Body Assessment (REBA) Scale

“…Residents were also not masked as to the purpose of the study, introducing a limitation of potential bias from the Hawthorne effect [ 21 ]. Due to their awareness of observation, residents may have adjusted their posture that does not normally occur during their clinic practice.…”

Association between participation in the Northern Finland Birth Cohort 1966 study and use of psychiatric care services

“…In the meta-analysis, the Hawthorne effect was defined as "an aware or unconscious complex behavioural change in a study environment, related to the interaction of four biases affecting the study subjects and investigators: selection bias, commitment and congruence bias, conformity and social desirability bias, and observation and measurement bias". The findings demonstrate that medical research is unavoidably prone to the Hawthorne effect, which restricts its external validity, beginning with the intentional or unintentional selection of the study population, resulting in blind spots in medical knowledge [4].…”