An online Paper Authoring Tool (PAT) to improve reporting of, and synthesis of evidence from, trials in behavioral sciences.

West, Robert

doi:10.1037/hea0000927

Cited by 10 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also advise that study authors use machine-readable tools (e.g. [ 98 ]) from the protocol stage to avoid inconsistent reporting within and across study outputs. Finally, valid, reliable and acceptable fidelity measures are needed to determine how school-based interventions are being implemented in practice.…”

Section: Discussionmentioning

confidence: 99%

Features of effective staff training programmes within school-based interventions targeting student activity behaviour: a systematic review and meta-analysis

Ryan

Alliott

Ikeda

et al. 2022

Int J Behav Nutr Phys Act

View full text Add to dashboard Cite

Background Evaluations of school-based activity behaviour interventions suggest limited effectiveness on students’ device-measured outcomes. Teacher-led implementation is common but the training provided is poorly understood and may affect implementation and student outcomes. We systematically reviewed staff training delivered within interventions and explored if specific features are associated with intervention fidelity and student activity behaviour outcomes. Methods We searched seven databases (January 2015–May 2020) for randomised controlled trials of teacher-led school-based activity behaviour interventions reporting on teacher fidelity and/or students’ device-measured activity behaviour. Pilot, feasibility and small-scale trials were excluded. Study authors were contacted if staff training was not described using all items from the Template for Intervention Description and Replication reporting guideline. Training programmes were coded using the Behaviour Change Technique (BCT) Taxonomy v1. The Effective Public Health Practice Project tool was used for quality assessment. Promise ratios were used to explore associations between BCTs and fidelity outcomes (e.g. % of intended sessions delivered). Differences between fidelity outcomes and other training features were explored using chi-square and Wilcoxon rank-sum tests. Random-effects meta-regressions were performed to explore associations between training features and changes in students’ activity behaviour. Results We identified 68 articles reporting on 53 eligible training programmes and found evidence that 37 unique teacher-targeted BCTs have been used (mean per programme = 5.1 BCTs; standard deviation = 3.2). The only frequently identified BCTs were ‘Instruction on how to perform the behaviour’ (identified in 98.1% of programmes) and ‘Social support (unspecified)’ (50.9%). We found moderate/high fidelity studies were significantly more likely to include shorter (≤6 months) and theory-informed programmes than low fidelity studies, and 19 BCTs were independently associated with moderate/high fidelity outcomes. Programmes that used more BCTs (estimated increase per additional BCT, d: 0.18; 95% CI: 0.05, 0.31) and BCTs ‘Action planning’ (1.40; 0.70, 2.10) and ‘Feedback on the behaviour’ (1.19; 0.36, 2.02) were independently associated with positive physical activity outcomes (N = 15). No training features associated with sedentary behaviour were identified (N = 11). Conclusions Few evidence-based BCTs have been used to promote sustained behaviour change amongst teachers in school-based activity behaviour interventions. Our findings provide insights into why interventions may be failing to effect student outcomes. Trial registration PROSPERO registration number: CRD42020180624

show abstract

Section: Discussionmentioning

confidence: 99%

Features of effective staff training programmes within school-based interventions targeting student activity behaviour: a systematic review and meta-analysis

Ryan

Alliott

Ikeda

et al. 2022

Int J Behav Nutr Phys Act

View full text Add to dashboard Cite

show abstract

“…There is no good reason why there should be such variability in the way that information is presented and a major effort will be needed in the coming years minimise this unnecessary variability. In many cases, this may be achieved by adopting authoring tools such as the Paper Authoring Tool (PAT) (30) developed as part of the HBCP which not only ensures that information is included in study reports in a form that can easily be extracted and compared across studies, but actually generates a machine readable version of the report dramatically reducing the need for either a human or NLP system to extract the information (31).…”

Section: Discussionmentioning

confidence: 99%

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour Change Project

West¹,

Bonin²,

Thomas³

et al. 2023

Preprint

View full text Add to dashboard Cite

Background and aims: Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour Change Project. Methods: Researchers manually annotated 70 items of information (‘entities’) in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the ‘FLAIR’ framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results: The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05 to 0.88) compared with an average human annotator’s score of 0.75 (SD=0.15, range 0.38 to 1.00). The algorithm for assigning entities to study arms (e.g., intervention or control) was not successful. The ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions: While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML.

show abstract

“…There is no good reason why there should be such variability in the way that information is presented, and a major effort will be needed in the coming years minimise this unnecessary variability. In many cases, this may be achieved by adopting authoring tools such as the Paper Authoring Tool (PAT) 31 developed as part of the HBCP, which not only ensures that information is included in study reports in a form that can easily be extracted and compared across studies, but actually generates a machine readable version of the report dramatically reducing the need for either a human or NLP system to extract the information 32 .…”

Section: Discussionmentioning

confidence: 99%

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project

West,

Bonin,

Thomas

et al. 2023

Wellcome Open Res

Self Cite

View full text Add to dashboard Cite

Background Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods Researchers manually annotated 70 items of information (‘entities’) in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the ‘FLAIR’ framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator’s score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms (e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper (1)).

show abstract

An online Paper Authoring Tool (PAT) to improve reporting of, and synthesis of evidence from, trials in behavioral sciences.

Cited by 10 publications

References 16 publications

Features of effective staff training programmes within school-based interventions targeting student activity behaviour: a systematic review and meta-analysis

Features of effective staff training programmes within school-based interventions targeting student activity behaviour: a systematic review and meta-analysis

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour Change Project

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project

Contact Info

Product

Resources

About