A hybrid CNN-Swin Transformer network as deep learning model observer to predict human observer performance in 2AFC trial

Abstract

Model observers designed to predict human observers in detection tasks are important tools for assessing task-based image quality and optimizing imaging systems, protocol, and reconstruction algorithms. Linear model observers have been widely studied to predict human detection performance, and recently, deep learning model observers (DLMOs) have been developed to improve the prediction accuracy. Most existing DLMOs utilize convolutional neural network (CNN) architectures, which are capable of learning local features while not good at extracting long-distance relations in images. To further improve the performance of CNN-based DLMOs, we investigate a hybrid CNN-Swin Transformer (CNN-SwinT) network as DLMO for PET lesion detection. The hybrid network combines CNN and SwinT encoders, which can capture both local information and global context. We trained the hybrid network on the responses of 8 human observers including 4 radiologists in a two-alternative forced choice (2AFC) experiment with PET images generated by adding simulated lesions to clinical data. We conducted a 9-fold cross-validation experiment to evaluate the proposed hybrid DLMO, compared to conventional linear model observers such as a channelized Hotelling observer (CHO) and a non-prewhitening matched filter (NPWMF). The hybrid CNN-SwinT DLMO predicted human observer responses more accurately than the linear model observers and DLMO with only the CNN encoder. This work demonstrates that the proposed hybrid CNN-SwinT DLMO has the potential as an improved tool for task-based image quality assessment.