A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies

Gebrye, Tadesse; Fatoye, Francis; Mbada, Chidozie Emmanuel; Hakimi, Zalmaï

doi:10.1007/s00296-023-05354-x

Cited by 7 publications

(5 citation statements)

References 45 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The findings of a preliminary scoping review by Gebrye et al [ 18 ] suggested that the generic quality assessment tools are mostly used in systematic reviews and meta-analysis involving real-world studies, while no validated and reliable specific tool currently exist. A related quality assessment tool for real-world studies by Wylde et al [ 30 ] seems to suggest that each individual item rating is reported, rather than an overall score and it is non-summative.…”

Section: Discussionmentioning

confidence: 99%

“…This study was approved by the Health and Education Research Ethics and Governance Committee at The Manchester Metropolitan University (EthOS Reference Number: 56368). From an initial scoping review [ 18 ], the authors identified 16 quality assessment tools that were used to assess the quality of systematic reviews and meta-analyses involving real-world studies. Using Excel spreadsheet, the list of items used by the 16 quality assessment tools were listed and helped to develop themes.…”

Section: Methodsmentioning

confidence: 99%

“…A formal quality assessment of the individual studies included in SRs and MA of real-world studies is important to evaluate the overall quality of their results. A scoping review on quality assessment tools used in SRs and MA of real-world studies identified no validated and reliable tool that are specific to these types of study [ 18 ].…”

Section: Introductionmentioning

confidence: 99%

“…The adoption of a standardised QATSM-RWS may also avoid the use of different types of tools that could be a source of bias. Although several types of quality assessment tools are currently used [ 18 ], none of these have been systematically developed to evaluate SRs and MA involving real-world studies. This study was aimed to develop QATSM-RWS using a formal consensus method.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey

Gebrye,

Mbada,

Hakimi

et al. 2024

Rheumatol Int

Self Cite

View full text Add to dashboard Cite

The increasing adoption of real-world studies in healthcare for decision making and planning has further necessitated the need for a specific quality assessment tool for evidence synthesis. This study aimed to develop a quality assessment tool for systematic reviews (SR) and meta-analysis (MA) involving real-world studies (QATSM-RWS) using a formal consensus method. Based on scoping review, the authors identified a list of items for possible inclusion in the quality assessment tool. A Delphi survey was formulated based on the identified items. A total of 89 experts, purposively recruited, with research experience in real-world data were invited to participate in the first round of Delphi survey. The participants who responded in the first Delphi round were invited to participate (n = 15) in the phrasing of the items. Strong level of agreement was found on the proposed list of items after the first round of Delphi. A rate of agreement ≥ 0.70 was used to define which items to keep in the tool. A list of 14 items emerged as suitable for QATSM-RWS. The items were structured under five domains: introduction, methods, results, discussions, and others. All participants agreed with the proposed phrasing of the items. This is the first study that has developed a specific tool that can be used to appraise the quality of SR and MA involving real-world studies. QATSM-RWS may be used by policymakers, clinicians, and practitioners when evaluating and generating real-world evidence. This tool is now undergoing validation process.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey

Gebrye,

Mbada,

Hakimi

et al. 2024

Rheumatol Int

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other assessment tools contain similar indicators. For more examples and details of PSQ assessments, we refer readers to existing reviews (e.g., Buccheri & Sharifi, 2017;Gebrye et al, 2023;Ma et al, 2020;Quigley et al, 2019). If meta-analysts used any two of these assessment tools for the primary studies, it is likely that resulting PSQ scores would diverge substantially.…”

Section: Psq Indicators and Assessment Toolsmentioning

confidence: 99%

Quality Assessment in Meta-Analysis (QuAMA)

Scherer¹,

Emslander²

2023

Preprint

View full text Add to dashboard Cite

Evaluating the quality of primary studies is a key step in meta-analytic reviews to reduce the risk of bias and establish the validity of the meta-analytic inferences. However, the extant body of research offers little guidance on how to represent and incorporate primary study quality (PSQ) in meta-analyses, and some common procedures, such as creating sum scores from a set of quality indicators, often lack the backing from measurement models. Addressing these issues, we present a tutorial that guides meta-analysts in their analytic decisions and approaches to represent and incorporate PSQ. Specifically, we describe, review, and illustrate approaches to (a) select or create quality indicators or scores a priori or as part of the meta-analytic model; (b) examine the possible moderator effects of PSQ; and (c) test the sensitivity of moderator effects to PSQ. We illustrate these approaches with three examples and present a step-by-step tutorial with analytic code for researchers’ guidance. Overall, we argue for representing PSQ model-based if multiple quality indicators are available, the testing of moderator effects of PSQ on the effect sizes and their heterogeneity, and performing moderator sensitivity analyses.

show abstract

ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): a screening tool to assess the potential for real-world data sources to support creation of credible real-world evidence for regulatory decision-making

Berger,

Crown,

et al. 2023

Health Serv Outcomes Res Method

View full text Add to dashboard Cite

Adoption and use of real-world data (RWD) for decision-making has been complicated by concerns regarding whether RWD was fit-for-purpose or was of sufficient validity to support the creation of credible RWE. This has greater urgency as regulatory agencies begin to use real world evidence (RWE) to inform decisions about treatment effectiveness. Researchers need an efficient and systematic method to screen the quality of RWD sources considered for use in studies of effectiveness and safety. Based on a literature review we developed a listing of screening criteria that have been previously proposed to assess the quality of RWD sources. We also developed an additional criterion based on Modern Validity Theory. While there has occurred some convergence of conceptual frameworks to assess data quality (DQ) and there is much agreement on specific assessment criteria, consensus has yet to emerge on how to assess whether a specific RWD source is reliable and fit-for-purpose. To create a user-friendly tool to assess whether RWD sources may have sufficient quality to support a well-designed RWE study for submission to a regulatory authority, we grouped the quality criteria with a view to harmonize published frameworks and to be consistent with how researchers generally evaluate existing RWD sources for research that they intend to submit to regulatory agencies. Screening data quality criteria were grouped into five dimensions after a comprehensive literature review via PubMed: authenticity, transparency, relevance, accuracy, and track record. The resultant tool was tested for its response burden using a hypothetical administrative claims data source. Providing responses to the screening criteria required only few hours effort by an experienced data source manager. Thus, the tool should not be an onerous burden on data source providers if asked by prospective researchers to provide the required information. Assessing whether a particular data source is fit-for-purpose will be facilitated by the use of this tool, but it will not be sufficient by itself. Fit-for-purpose judgements will still require further careful consideration based on the context and the specific scientific question of interest. Unlike prior DQ frameworks (DQF), the track record dimension of the tool adds the consideration of experience with RWD sources consistent with Modern Validity Theory. However, the tool does not address issues of study design and analysis that are critical to regulatory agencies in evaluating the robustness and credibility of the real-world evidence generated.

show abstract

A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies

Cited by 7 publications

References 45 publications

Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey

Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey

Quality Assessment in Meta-Analysis (QuAMA)

ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): a screening tool to assess the potential for real-world data sources to support creation of credible real-world evidence for regulatory decision-making

Contact Info

Product

Resources

About