Agreement studies in radiology research

Farzin, Behzad; Gentric, Jean-Christophe; Pham, Mary; Tremblay-Paquet, Sophie; Brosseau, Lucie; Roy, Claudie; Jamali, Sarah; Chagnon, Miguel; Darsaut, T.E.; Guilbert, F; Naggara, Olivier; Raymond, Jean

doi:10.1016/j.diii.2016.05.014

Cited by 29 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the poor interobserver agreement was not influenced by readers’ experience. These findings add to other limitations of all DSA-based collateral grading, even those which demonstrated a higher interobserver agreement 29. The main limitation is that for patients with AIS-LVO, therapeutic decisions have to be made in advance of DSA.…”

Section: Discussionmentioning

confidence: 82%

“…The electronic survey used imaging data from the THRACE23 randomized trial database sent to a large number of readers with different levels of experience, and from various backgrounds, and institutions, as methodologically requested 29. In our opinion, an electronic survey accounts for non-response bias and maximizes the response rate, offering the possibility of having a large panel of readers.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Inter- and intraobserver reliability for angiographic leptomeningeal collateral flow assessment by the American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) scale

Hassen

Malley

Boulouis

et al. 2018

J NeuroIntervent Surg

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 82%

Section: Discussionmentioning

confidence: 99%

Inter- and intraobserver reliability for angiographic leptomeningeal collateral flow assessment by the American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) scale

Hassen

Malley

Boulouis

et al. 2018

J NeuroIntervent Surg

View full text Add to dashboard Cite

show abstract

“…These may be very informative, but they are rarely performed. [31,40,41] Better agreement can be expected when the same clinician responds twice to the same series of cases (typically weeks apart in patients presented in a different order to assure independence between judgments), but the risk here is that the clinician may reveal their own inconsistencies in decision-making. In the case of diagnostic tests, poor intra-rater agreement (across multiple raters) is evidence of the lack of reliability of the score/measurement/diagnostic categories, and a strong indication that the scale or categories should be modi ed.…”

Section: Spectrum Of Cliniciansmentioning

confidence: 99%

Measuring Clinical Uncertainty and Equipoise by Applying the Agreement Study Methodology to Patient Management Decisions

Fahed

Darsaut

Farzin³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Clinical uncertainty and equipoise are vague notions that play important roles in contemporary problems of medical care and research, including the design and conduct of pragmatic trials. Our goal was to show how the reliability study methods normally used to assess diagnostic tests can be applied to particular management decisions to measure the degree of uncertainty and equipoise regarding the use of rival management options.Methods: We first use thrombectomy in acute stroke as an illustrative example of the method we propose. We then review, item by item, how the various design elements of diagnostic reliability studies can be modified in order to measure clinical uncertainty.Results: The thrombectomy example shows sufficient disagreement and uncertainty to warrant the conduct of additional randomized trials. The general method we propose is that a sufficient number of diverse individual cases sharing a similar clinical problem and covering a wide spectrum of clinical presentations be assembled into a portfolio that is submitted to a variety of clinicians who routinely manage patients with the clinical problem. Clinicians are asked to independently choose one of the predefined management options, which are selected from those that would be compared within a randomized trial that would address the clinical dilemma. Intra-rater agreement can be assessed at a later time with a second evaluation. Various professional judgments concerning individual patients can then be compared and analyzed using kappa statistics or similar methods. Interpretation of results can be facilitated by providing examples or by translating the results into clinically meaningful summary sentences.Conclusions: Measuring the uncertainty regarding management options for clinical problems may reveal substantial disagreement, provide an empirical foundation for the notion of equipoise, and inform or facilitate the design/conduct of clinical trials to address the clinical dilemma.

show abstract

“…These may be very informative, but they are rarely performed. [18][19][20] Better agreement can be expected when the same clinician responds twice to the same series of cases (typically weeks apart in patients presented in a different order to assure independence between judgments), but the risk here is that the clinician may reveal their own inconsistencies in decision-making. In the case of diagnostic tests, poor intra-rater agreement (across multiple raters) is evidence of the lack of reliability of the score/measurement/ diagnostic categories, and a strong indication that the scale or categories should be modified.…”

Section: Spectrum Of Cliniciansmentioning

confidence: 99%

Measuring Clinical Uncertainty and Equipoise by Applying the Agreement Study Methodology to Patient Management Decisions

Fahed

Darsaut

Farzin³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Clinical dilemmas in the treatment of patients translate into disagreements in decision-making. Such disagreements can reveal clinical uncertainty that should be addressed through care research. Our goal was to explore the use of reliability study methods to measure the degree of clinical uncertainty and equipoise regarding the use of rival management options prior to the conduct of randomized trials. Methods: The study design resembles an inter-/intra-observer diagnostic reliability study. A portfolio of a sufficient number of diverse individual patients sharing a similar clinical problem and covering a wide spectrum of clinical presentations can be independently submitted to a variety of clinicians who manage that problem. Clinicians are asked to choose one of the predefined management options that are involved in the clinical dilemma. Intra-rater agreement can be assessed at a later time with a second evaluation. Results: Descriptive statistics are presented, and results analyzed using kappa statistics. Interpretation of results can be facilitated by providing examples or by translating the results into clinically meaningful summary sentences. Reporting should follow standard guidelines. Conclusions: Measuring the uncertainty regarding management options for clinical problems may reveal disagreements, provide an empirical foundation for the notion of equipoise, and inform or facilitate the design/conduct of clinical trials to address the clinical dilemma.

show abstract

Agreement studies in radiology research

Abstract: Agreement studies are preliminary and not adequately reported. Studies dedicated to agreement are infrequent. They are research opportunities that should be promoted.

Cited by 29 publications

References 16 publications

Inter- and intraobserver reliability for angiographic leptomeningeal collateral flow assessment by the American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) scale

Inter- and intraobserver reliability for angiographic leptomeningeal collateral flow assessment by the American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) scale

Measuring Clinical Uncertainty and Equipoise by Applying the Agreement Study Methodology to Patient Management Decisions

Measuring Clinical Uncertainty and Equipoise by Applying the Agreement Study Methodology to Patient Management Decisions

Contact Info

Product

Resources

About