Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about treatment effects: one-year follow up of a randomised trial

Abstract

Introduction: Earlier, we designed and evaluated an educational mass media intervention for improving people's ability to think more critically and to assess the trustworthiness of claims (assertions) about the benefits and harms (effects) of treatments. The overall aims of this follow-up study were to evaluate the impact of our intervention 1 year after it was administered, and to assess retention of learning and behaviour regarding claims about treatments. Methods: We randomly allocated consenting parents to listen to either the Informed Health Choices podcast (intervention) or typical public service announcements about health issues (control) over 7-10 weeks. Each intervention episode explained how the trustworthiness of treatment claims can be assessed by using relevant key concepts of evidence-informed decision-making. Participants listened to two episodes per week, delivered by research assistants. We evaluated outcomes immediately, and a year after the intervention. Primary outcomes were mean score and the proportion with a score indicating a basic ability to apply the key concepts (> 11 out of 18 correct answers) on a tool measuring people's ability to critically appraise the trustworthiness of treatment claims. Skills decay/retention was estimated by calculating the relative difference between the follow-up and initial results in the intervention group, adjusting for chance. Statistical analyses were performed using R (R Core Team, Vienna, Austria; version 3.4.3). Results: After 1 year, the mean score for parents in the intervention group was 58.9% correct answers, compared to 52.6% in the control (adjusted mean difference of 6.7% (95% CI 3.3% to 10.1%)). In the intervention group, 47.2% of 267 parents had a score indicating a basic ability to assess treatment claims compared to 39.5% of 256 parents in the control (adjusted difference of 9.8% more parents (95% CI 0.9% to 18.9%). These represent relative reductions of 29% in the mean scores and 33% in the proportion of parents with a score indicating a basic ability to assess the trustworthiness of claims about treatment effects.

The Norwegian public’s ability to assess treatment claims: results of a cross-sectional study of critical health literacy

“…We compared Norwegian and Ugandan adults' understanding of the Key Concepts to that of Ugandan children in the intervention arm of our randomized trial. We compared mean test scores, and probabilities of achieving passing and mastery scores, using estimates from one year after the IHC primary school intervention for the Ugandan children and their teachers 51,52 and estimates from the control group for parents in the IHC podcast trial, one year after parents in the intervention group listened to the podcast 52,53 . We used predetermined thresholds of at least 13 of 24 questions answered correctly for a passing score and at least 20 of 24 for mastery.…”

Informed Health Choices media intervention for improving people’s ability to critically appraise the trustworthiness of claims about treatment effects: a mixed-methods process evaluation of a randomised trial in Uganda

“…The podcast intervention had a large effect initially, with almost twice as many parents in the intervention group having a passing score on the test used to measure their ability to assess treatment claims, compared with the parents in the control group 19. After 1 year, the proportion of parents with a passing score on the same test decreased by one third 21. We found a number of factors that help to explain the initial effectiveness of the intervention.…”

“…Many of the participants with tertiary education were teachers, and this might partially explain that finding. A large proportion of teachers in both intervention and control schools had passing scores on the test initially and after 1 year,22 compared with the parents overall 21…”

“…In that study, skills retention (or decay) was assessed by comparing the scores in the intervention group initially after the intervention and in the same group a year later. These results are reported in greater detail elsewhere 21. The overall goal of the process evaluation was to provide information that could be used to explain the results observed in the trials (impact) and identify other effects not reported in the trial.…”

“…Results from both trials initially showed a large improvement in participants’ ability to assess the trustworthiness of treatment claims. However, follow-up assessments (described elsewhere) revealed that parents’ critical appraisal skills decayed substantially over the following year,21 whereas the children’s or their teachers’ ability did not 22. In that study, skills retention (or decay) was assessed by comparing the scores in the intervention group initially after the intervention and in the same group a year later.…”