Performance Evaluation of Grid-Enabled Registration Algorithms Using Bronze-Standards

Abstract

Abstract. Evaluating registration algorithms is difficult due to the lack of gold standard in most clinical procedures. The bronze standard is a real-data based statistical method providing an alternative registration reference through a computationally intensive image database registration procedure. We propose in this paper an efficient implementation of this method through a grid-interfaced workflow enactor enabling the concurrent processing of hundreds of image registrations in a couple of hours only. The performances of two different grid infrastructures were compared. We computed the accuracy of 4 different rigid registration algorithms on longitudinal MRI images of brain tumors. Results showed an average subvoxel accuracy of 0.4 mm and 0.15 degrees in rotation. Performance evaluation using bronze standardsThe accuracy performances of registration algorithms are critical for many clinical procedures but quantifying them is difficult due to the lack of gold standard in most clinical applications. To analyze registration algorithms, one may consider them as black boxes that take images as input and output a transformation. The performance evaluation problem is to estimate the quality of the transformation. However, no registration algorithm will perform the same for all types of input data. For instance, one algorithm may perform very well for multimodal MR registration but poorly for SPECT/CT. This means that the evaluation data set has to be representative of the clinical application problem we are targeting: all sources of perturbation in the data should be represented, such as acquisition noise and artifacts, pathologies, etc, and that we cannot just conclude from one experiment that one algorithm is better than the others for all applications. Performance quantifiersAs far as the registration result is concerned, one can distinguish between gross errors (convergence to wrong local minima) and small errors around the exact transformation. The robustness can be quantified by the size of the basin of attraction of the right solution or by the probability of false positives. The small errors may be sorted into systematic biases, repeatability and accuracy [1]. The repeatability accounts for the errors due to internal parameters of the algorithm, mainly the initial transformation, and to the finite numerical accuracy of the optimization algorithm, while the external error accounts for the propagation

Bootstrap Resampling for Image Registration Uncertainty Estimation Without Ground Truth

“…More realistic but less widely applicable 'gold standard' approach is to use some independent and sufficiently accurate method to determine the true deformation, such as using special markers for validation which are not used for registration [23]- [25]. A "bronze standard" [26], [27] uses a robust mean of several registration algorithms as a reference. The registration accuracy can also be estimated indirectly, from ground truth segmentations [28], [29] or by its ability to create good generative models [30].…”

Virtual Resources Allocation for Workflow-Based Applications Distribution on a Cloud Infrastructure

“…The experiments are performed using the Bronze Standard (BS) a real workflow-based application from the area of medical image analysis [7]. The BS technique tackles the difficult problem of validating medical-image analysis tools.…”

A Scalable Security Model for Enabling Dynamic Virtual Private Execution Infrastructures on the Internet

“…This section reports on its porting for execution in a private VPXI infrastructure. The bronze standard [10] technique tackles the difficult problem of validating of medical image analysis procedures.…”