Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review

Abstract

Background: A relatively novel method of appraisal, methodological reviews (MRs) are used to synthesize information on the methods used in health research. There are currently no guidelines available to inform the reporting of MRs. Objectives: This pilot review aimed to determine the feasibility of a full review and the need for reporting guidance for methodological reviews. Methods: Search strategy: We conducted a search of PubMed, restricted to 2017 to include the most recently published studies, using different search terms often used to describe methodological reviews: "literature survey" OR "meta-epidemiologic* review" OR "meta-epidemiologic* survey" OR "methodologic* review" OR "methodologic* survey" OR "systematic survey." Data extraction: Study characteristics including country, nomenclature, number of included studies, search strategy, a priori protocol use, and sampling methods were extracted in duplicate and summarized. Outcomes: Primary feasibility outcomes were the sensitivity and specificity of the search terms (criteria for success of feasibility set at sensitivity and specificity of ≥ 70%). Analysis: The estimates are reported as a point estimate (95% confidence interval). Results: Two hundred thirty-six articles were retrieved and 31 were included in the final analysis. The most accurate search term was "meta-epidemiological" (sensitivity [Sn] 48.39; 95% CI 31.97-65.16; specificity [Sp] 97.56; 94.42-98.95). The majority of studies were published by authors from Canada (n = 12, 38.7%), and Japan and USA (n = 4, 12.9% each). The median (interquartile range [IQR]) number of included studies in the MRs was 77 (13-1127). Reporting of a search strategy was done in most studies (n = 23, 74.2%). The use of a pre-published protocol (n = 7, 22.6%) or a justifiable sampling method (n = 5, 16.1%) occurred rarely. Conclusions: Using the MR nomenclature identified, it is feasible to build a comprehensive search strategy and conduct a full review. Given the variation in reporting practices and nomenclature attributed to MRs, there is a need for guidance on standardized and transparent reporting of MRs. Future guideline development would likely include stakeholders from Canada, USA, and Japan.

A tutorial on methodological studies: the what, when, how and why

“…While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [9]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020).…”

“…The term "review" may point towards an appraisal or "review" of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [41,42]. The term "survey" is also in line with the approaches used in many methodological studies [9], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term "methodological study" is broad enough to capture most of the scenarios of such studies.…”

“…For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 – 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them.…”

A call for consensus guidelines on classification and reporting of methodological studies

“…This enterprise is registered on the EQUA-TOR network as the METhodological Review reportIng Checklist [4]. We have recently conducted a pilot study on a sample of methodological papers from 2017 and have identified many concerns, some of which were also raised in Dr. Puljak's letter and related commentary (e.g., issues with classification, reporting of sampling methods, and search strategies) [5].…”

Evaluating the quality of reported outcomes for microsurgical TESE in men with non‐obstructive azoospermia: A methodological analysis

“…Inspired by the IMPRINT review of reporting standards in prospective trials on infertility, this study aims to systematically review the methodological quality of published, largely retrospective studies on mTESE approaches and outcomes. Methodological analyses are designed to synthesize, analyze, and report on primary and secondary research and are highly useful in being able to demonstrate trends and inconsistencies in data reporting 15 . As noted by Li et al (2011), methodologic analysis also allows for identification of key areas of bias among non‐randomized trials and may provide the groundwork for standards of reporting 16 .…”