The VAMPIRE challenge: A multi‐institutional validation study of CT ventilation imaging

Abstract

Purpose CT ventilation imaging (CTVI) is being used to achieve functional avoidance lung cancer radiation therapy in three clinical trials (NCT02528942, NCT02308709, NCT02843568). To address the need for common CTVI validation tools, we have built the Ventilation And Medical Pulmonary Image Registration Evaluation (VAMPIRE) Dataset, and present the results of the first VAMPIRE Challenge to compare relative ventilation distributions between different CTVI algorithms and other established ventilation imaging modalities. Methods The VAMPIRE Dataset includes 50 pairs of 4DCT scans and corresponding clinical or experimental ventilation scans, referred to as reference ventilation images (RefVIs). The dataset includes 25 humans imaged with Galligas 4DPET/CT, 21 humans imaged with DTPA‐SPECT, and 4 sheep imaged with Xenon‐CT. For the VAMPIRE Challenge, 16 subjects were allocated to a training group (with RefVI provided) and 34 subjects were allocated to a validation group (with RefVI blinded). Seven research groups downloaded the Challenge dataset and uploaded CTVIs based on deformable image registration (DIR) between the 4DCT inhale/exhale phases. Participants used DIR methods broadly classified into B‐splines, Free‐form, Diffeomorphisms, or Biomechanical modeling, with CT ventilation metrics based on the DIR evaluation of volume change, Hounsfield Unit change, or various hybrid approaches. All CTVIs were evaluated against the corresponding RefVI using the voxel‐wise Spearman coefficient rS, and Dice similarity coefficients evaluated for low function lung (DSClow) and high function lung (DSChigh). Results A total of 37 unique combinations of DIR method and CT ventilation metric were either submitted by participants directly or derived from participant‐submitted DIR motion fields using the in‐house software, VESPIR. The rS and DSC results reveal a high degree of inter‐algorithm and intersubject variability among the validation subjects, with algorithm rankings changing by up to ten positions depending on the choice of evaluation metric. The algorithm with the highest overall cross‐modality correlations used a biomechanical model‐based DIR with a hybrid ventilation metric, achieving a median (range) of 0.49 (0.27–0.73) for rS, 0.52 (0.36–0.67) for DSClow, and 0.45 (0.28–0.62) for DSChigh. All other algorithms exhibited at least one negative rS value, and/or one DSC value less than 0.5. Conclusions The VAMPIRE Challenge results demonstrate that the cross‐modality correlation between CTVIs and the RefVIs varies not only with the choice of CTVI algorithm but also with the choice of RefVI modality, imaging subject, and the evaluation metric used to compare relative ventilation distributions. This variability may arise from the fact that each of the different CTVI algorithms and RefVI modalities provides a distinct physiologic measurement. Ultimately this variability, coupled with the lack of a “gold standard,” highlights the ongoing importance of further validation studies before CTVI can be widely translated from academic ce...

CT-based ventilation imaging in radiation oncology

“…The level of smoothing that is applied to the ventilation images varies by research group and implementation. 19 …”

Technical Note: Correction for the effect of breathing variations in CT pulmonary ventilation imaging

“…The voxel‐wise correlation coefficients ranged from 0.08 to 0.35 before correction, which are similar to those reported in the previous human studies comparing CT and ground truth ventilation images . Kipritidis et al observed median voxel‐wise correlations ranging from approximately 0 to 0.5 for 25 subjects with 68 Ga‐aerosol (Galligas) PET and 21 subjects with 99m Tc‐DTPA SPECT in a multi‐institutional study to evaluate 23 different methods of CT ventilation imaging (i.e., unique combinations of DIR and ventilation metric) …”

“…16 Tc-DTPA SPECT in a multi-institutional study to evaluate 23 different methods of CT ventilation imaging (i.e., unique combinations of DIR and ventilation metric). 23 The accuracy and precision of CT ventilation imaging may also be improved by strategies to reduce 4D-CT artifacts, such as improved sorting, [31][32][33][34][35] breathing guidance, 36,37 prospectively gated acquisition, [38][39][40] and image processing. 41,42 Combining the proposed correction method with such strategies may generate additional value for improving CT ventilation imaging.…”

“…As pulmonary ventilation is a function of depth of breathing, such inconsistencies deteriorate the accuracy and precision of CT ventilation imaging. Other major sources of errors in CT ventilation imaging include DIR algorithms, ventilation metrics, and perfusion‐induced lung density variations . In this study, we propose a method to correct for the effect of breathing variations based on external respiratory signals acquired throughout a scan.…”

Technical Note: On the spatial correlation between robust CT‐ventilation methods and SPECT ventilation

“…The resulting affine transformation was then applied to the CT‐ventilation image and the voxel‐wise Spearman correlation was computed at the resolution of the SPECT ventilation, after applying a median filter with structuring element to the SPECT image (as done in Ref. [20]).…”