The 2016 QSM Challenge: Lessons learned and considerations for a future challenge design

Abstract

Purpose: The 4th International Workshop on MRI Phase Contrast and QSM (2016, Graz, Austria) hosted the first QSM Challenge. A single-orientation gradient recalled echo acquisition was provided, along with COSMOS and the χ 33 STI component as ground truths. The submitted solutions differed more than expected depending on the error metric used for optimization and were generally over-regularized. This raised (unanswered) questions about the ground truths and the metrics utilized. Methods: We investigated the influence of background field remnants by applying additional filters. We also estimated the anisotropic contributions from the STI tensor to the apparent susceptibility to amend the χ 33 ground truth and to investigate the impact on the reconstructions. Lastly, we used forward simulations from the COSMOS reconstruction to investigate the impact noise had on the metric scores. Results: Reconstructions compared against the amended STI ground truth returned lower errors. We show that the background field remnants had a minor impact in the errors. In the absence of inconsistencies, all metrics converged to the same regularization weights, whereas structural similarity index metric was more insensitive to such inconsistencies. Conclusion: There was a mismatch between the provided data and the ground truths due to the presence of unaccounted anisotropic susceptibility contributions and noise.

Comparison of parameter optimization methods for quantitative susceptibility mapping

“…Reconstruction quality metrics with respect to COSMOS are provided for reference in the Supporting Information Table S1. As reported in Milovic et al, 35 these metrics should not be considered as proper error metrics because both COSMOS and χ 33 (Susceptibility Tensor Imaging, STI) ground truths incorporate anisotropic and microstructural contributions that are not present in the single-orientation acquisition, creating significant discrepancies. [35][36][37] This results in overregularized reconstructions when using such metrics and ground-truth data sets for parameter optimization.…”

QSM Reconstruction Challenge 2.0: Design and Report of Results

“…If the input phase data contains only information compatible with the magnetic dipole convolutional model, it is to be expected that most of the global metrics tend to produce the same optimal reconstruction parameters for a given algorithm 16 . Phase data inconsistencies or external contributions lead to a disagreement of the optimal parameters 16 , as shown in RC1. Given that RC2 consists of phantom-based forward simulations, to avoid unnecessary complexity in the challenge design (winning categories), only RMSE-based metrics were chosen to officialy evaluate the global performance of the submissions.…”

“…Despite its success as a benchmark dataset, the challenge itself had limitations that limited its practical relevance. The limitations were discussed in the report paper, and analyzed more quantitatively in a separate manuscript 16 . In brief, it was concluded that the estimated susceptibility tensor component χ 33 that was used as the ground-truth removed anisotropic contributions found in single-orientation phase data, which resulted in an inconsistency between the provided field map and the groundtruth susceptibility.…”

QSM Reconstruction Challenge 2.0: a realistic in silico head phantom for MRI data simulation and evaluation of susceptibility mapping procedures

“…Meanwhile, only one field-map obtained from those twelve orientation acquisitions was provided to challenge participants. After completion of the challenge, it was observed (13) that a non-negligible discrepancy existed between the provided frequency map and the frequency map obtained when forward-simulating the field perturbations using the provided reference susceptibility maps. Part of the discrepancies could be explained by the presence of unaccounted microstructure effects on in vivo brain phase images (14).…”

“…The rationale of this approach was that RC1 would yield the most objective and meaningful results if algorithms were evaluated using real-world in vivo data. However, at the completion of RC1, it was observed (21) that a non-negligible discrepancy existed between the provided frequency map and the frequency map obtained when the field perturbation was forward-simulated based on the provided reference susceptibility maps. It was speculated that a part of the discrepancies were related to unaccounted microstructure effects on in vivo brain phase images (22).…”