A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets

Abstract

Expert landmark correspondences are widely reported for evaluating deformable image registration (DIR) spatial accuracy. In this report, we present a framework for objective evaluation of DIR spatial accuracy using large sets of expert-determined landmark point pairs. Large samples (>1100) of pulmonary landmark point pairs were manually generated for five cases. Estimates of inter- and intra-observer variation were determined from repeated registration. Comparative evaluation of DIR spatial accuracy was performed for two algorithms, a gradient-based optical flow algorithm and a landmark-based moving least-squares algorithm. The uncertainty of spatial error estimates was found to be inversely proportional to the square root of the number of landmark point pairs and directly proportional to the standard deviation of the spatial errors. Using the statistical properties of this data, we performed sample size calculations to estimate the average spatial accuracy of each algorithm with 95% confidence intervals within a 0.5 mm range. For the optical flow and moving least-squares algorithms, the required sample sizes were 1050 and 36, respectively. Comparative evaluation based on fewer than the required validation landmarks results in misrepresentation of the relative spatial accuracy. This study demonstrates that landmark pairs can be used to assess DIR spatial accuracy within a narrow uncertainty range.

Evaluation of the tool “Reg Refine” for user‐guided deformable image registration

“…The framework for validation relies upon the establishment a ground‐truth deformation. This takes several forms, including physical phantoms which can be deformed mechanically in known ways, 2 , 3 real patient images with identifiable landmarks which can be tracked between images, 4 , 5 and digital phantoms which can be deformed manually to create a linked pair of reference images (6) . While extensive time and effort is required to establish a ground‐ truth deformation using these methods, several authors have graciously performed this work and made such data publicly available.…”

“…While extensive time and effort is required to establish a ground‐ truth deformation using these methods, several authors have graciously performed this work and made such data publicly available. Examples include the Point‐validated Pixel‐based Breathing Thorax Model (POPI) for lung, (4) the DIR‐Lab Thoracic 4D CT model also for lung, (5) and the Deformable Image Registration Evaluation Project (DIREP) for head and neck (6) . These datasets are a great option for many clinics that may not have the capital to invest in validation‐specific software such as ImSimQA (Oncology Systems Limited, Shrewsbury, UK) or the resources to create their own phantoms.…”

Benchmarking of five commercial deformable image registration algorithms for head and neck patients

“…In an effort to provide consistent datasets for algorithm validation, several researchers have made their ground‐truth models publicly available. Examples include the extended cardiac–torso (XCAT) phantom, (3) the point‐validated pixel‐based breathing thorax model (POPI model), (4) the DIR‐Lab Thoracic 4D CT model, 5 , 6 and those provided as part of the EMPIRE 10 challenge (7) . In an effort to improve the correlation of computer‐based phantoms with the actual anatomical changes seen over a course of radiation therapy, the current authors previously developed synthetic datasets derived from clinical images of real head and neck patients (8) .…”

Deformable image registration and interobserver variation in contour propagation for radiation therapy planning

“…Studies have assessed the accuracy of DIR algorithms using digital phantoms, (10) physically deforming phantoms, (11) mathematical descriptors, 12 , 13 and clinical CT scans 14 , 15 , 16 . Digital (10) or physically deforming phantom studies, (11) while useful, may lack clinical complexity.…”

“…Landmark‐based quality assurance from clinical CT currently represents the most robust quantification of DIR accuracy. Castillo et al (14) demonstrate the efficacy of landmark pairs to assess DIR quality in thoracic CT imaging and suggest the technique could be used for routine DIR quality assurance. In a large multi‐institutional study, Brock et al (16) measure DIR error for intra‐ and intermodality DIR using landmarks and found DIR errors on the order of voxel size.…”