History and publication trends in the diffusion and early uptake of indirect comparison meta-analytic methods to study drugs: animated coauthorship networks over time

et al. 2023

Preprint

Background: Network meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee intended the RoB NMA tool to be used in combination with ROBIS (recommend as it was designed to assess biases only) or other similar quality appraisal tools (e.g., AMSTAR 2) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed, and results were presented, including the way in which the evidence was assembled and interpreted. Objectives: Conduct (a) a Delphi process to determine expert opinion on an item’s inclusion, and (b) a knowledge user survey to widen its impact. Design: Cross sectional survey and Delphi process Methods: Delphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users’ views and preferences about the importance, utility, and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks. Results: 22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2.Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organization produced NMAs, and 78% showedhigh interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors’ conclusions. Response bias in our sample is a major limitation as knowledge users working in higher income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines. Conclusions: This Delphi process and knowledge user survey informs the development of the RoB NMA tool. Registration: We uploaded our study protocol on the Open Science Framework at https://osf.io/da4uy/

“…We created an email list of journal editors publishing NMAs, using one bibliometric study of NMAs [26]. From this list, we extracted the journal names, and names of authors of NMAs.…”

Section: Email List Developmentmentioning

confidence: 99%

Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool)

et al. 2023

Preprint

“…We created an email list of journal editors publishing NMAs, using one bibliometric study of NMAs 19. From this list, we extracted the journal names, and names of authors of NMAs.…”

Section: Knowledge User Surveymentioning

confidence: 99%

“…We created an email list of journal editors publishing NMAs, using one bibliometric study of NMAs. 19 From this list, we extracted the journal names, and names of authors of NMAs. We also developed a list of organisations and institutions producing NMAs (online supplemental appendix D).…”

Section: Email List Developmentmentioning

confidence: 99%

Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool)

et al. 2022

BMJ EBM

BackgroundNetwork meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee recommended that the RoB NMA tool to be used in combination with the Risk of Bias in Systematic reviews (ROBIS) tool (i.e. because it was designed to assess biases only) or other similar quality appraisal tools (eg, A MeaSurement Tool to Assess systematic Reviews 2 [AMSTAR 2]) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed and results were presented, including the way in which the evidence was assembled and interpreted.ObjectivesConduct (a) a Delphi process to determine expert opinion on an item’s inclusion and (b) a knowledge user survey to widen its impact.DesignCross-sectional survey and Delphi process.MethodsDelphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users’ views and preferences about the importance, utility and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks.Results22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2. Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organisation produced NMAs, and 78% showed high interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors’ conclusions. Response bias in our sample is a major limitation as knowledge users working in high-income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines.ConclusionsThis Delphi process and knowledge user survey informs the development of the RoB NMA tool.

“…We created an email list of journal editors publishing NMAs, using one bibliometric study of NMAs [23]. From this list, we extracted the journals that publish NMAs, and names of authors of NMAs.…”

Section: Email List Developmentmentioning

confidence: 99%

Stakeholder opinions on the structure and development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool): a cross-sectional survey

et al. 2022

Preprint

Introduction: Network meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We engaged with stakeholders in early phases of the development of a new tool to assess risk of bias (RoB) in NMA to facilitate its later use and potentially widen its impact. Methods: We surveyed stakeholders’ views and preferences about the importance, utility, and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and followed a knowledge translation plan to disseminate the survey through social media and professional networks. Results: A total of 298 stakeholders participated in the survey (14% respondent rate) which took on average 2 minutes to complete. 75% of stakeholders indicated that their organization produced NMAs, and 78% of respondents showed high interest in the tool. Most stakeholders (84%) reported they would use the tool to assess an NMA if they had received adequate training, while half reported they preferred a tool to assess both bias in individual NMA results and authors’ conclusions. Twelve (8%) stakeholders said they would use the tool to conduct an overview of reviews, while 24 (17%) respondents indicated that they would use the tool to distinguish between NMAs at high and low risk of bias. Conclusions: This survey informs the development of a RoB tool for NMAs and is a pillar in an integrated knowledge translation approach. Stakeholders preferred a tool to assess both bias in individual NMA results and authors’ conclusions.