A practical method to quantify knowledge‐based DVH prediction accuracy and uncertainty with reference cohorts

Abstract

The adoption of knowledge-based dose-volume histogram (DVH) prediction models for assessing organ-at-risk (OAR) sparing in radiotherapy necessitates quantification of prediction accuracy and uncertainty. Moreover, DVH prediction error bands should be readily interpretable as confidence intervals in which to find a percentage of clinically acceptable DVHs. In the event such DVH error bands are not available, we present an independent error quantification methodology using a local reference cohort of high-quality treatment plans, and apply it to two DVH prediction models, ORBIT-RT and RapidPlan, trained on the same set of 90 volumetric modulated arc therapy (VMAT) plans. Organ-atrisk DVH predictions from each model were then generated for a separate set of 45 prostate VMAT plans. Dose-volume histogram predictions were then compared to their analogous clinical DVHs to define prediction errors V clin,i À V pred,i (ith plan), from which prediction bias μ, prediction error variation σ, and root-mean-square error RMSE pred ≡ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 N ∑ i V clin,i À V pred,i À Á 2 r ≅ ffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi σ 2 þ μ 2 p could be calculated for the cohort. The empirical RMSE pred was then contrasted to the model-provided DVH error estimates. For all prostate OARs, above 50% Rx dose, ORBIT-RT μ and σ were comparable to or less than those of RapidPlan. Above 80% Rx dose, μ < 1% and σ < 3-4% for both models. As a result, above 50% Rx dose, ORBIT-RT RMSE pred was below that of RapidPlan, indicating slightly improved accuracy in this cohort. Because μ ≈ 0, RMSE pred is readily interpretable as a canonical standard deviation σ, whose error band is expected to correctly predict 68% of normally distributed clinical DVHs. By contrast, RapidPlan's provided error band, although described in literature as a standard deviation range, was slightly less predictive than RMSE pred (55-70% success), while the provided ORBIT-RT error band was confirmed to resemble an interquartile range (40-65% success) as described. Clinicians can apply this methodology using their own institutions' reference cohorts to (a) independently assess a knowledge-based model's predictive accuracy of local treatment plans, and (b) interpret from any error band whether further OAR dose sparing is likely attainable.

Using Scorecards to Tune Ethos Directive Templates: An Adaptive Radiation Therapy Individualized Approach-Cervix Dosimetric Planning Study

“…The scorecard-guided CDT (CDT-2) was then compared with CDT-1 with and without inclusion of RapidPlan DVHe from a publicly available knowledge-based planning model for gynecologic cancers. 5 CDT-1 was intended to be used with a DVHe model, but for the purposes of this study, the performance of both CDT-1 and CDT-2 were assessed with and without DVHe. After obtaining favorable results for 2 test cases, 10 additional cases were retrospectively identified and tested, both with DVHe.…”

Probabilistic Pareto plan generation for semiautomated multicriteria radiation therapy treatment planning

“…In this context, we showed (Zhang et al 2021) that probabilistic methods, which output predictive probability distributions expressing estimation uncertainties, may reduce the information loss between the prediction and mimicking parts. Indeed, much of other previous work (Fogliata et al 2019, Covele et al 2021, Nguyen et al 2021, Nilsson et al 2021 have already been directed toward precise quantification of predictive uncertainties for spatial dose or DVH statistics.…”

Technical note: Determining the applicability of a clinical knowledge‐based learning model via prospective outlier detection

“…Recently, researchers have started to study methods to quantify the overall prediction uncertainties of a KBP model. Covele et al 17 proposed to model DVH prediction uncertainties with prediction bias and prediction error variations calculated using a reference cohort. The prediction uncertainty information is then applied as an uncertainty band.…”