A machine and human reader study on AI diagnosis model safety under attacks of adversarial images

Abstract

While active efforts are advancing medical artificial intelligence (AI) model development and clinical translation, safety issues of the AI models emerge, but little research has been done. We perform a study to investigate the behaviors of an AI diagnosis model under adversarial images generated by Generative Adversarial Network (GAN) models and to evaluate the effects on human experts when visually identifying potential adversarial images. Our GAN model makes intentional modifications to the diagnosis-sensitive contents of mammogram images in deep learning-based computer-aided diagnosis (CAD) of breast cancer. In our experiments the adversarial samples fool the AI-CAD model to output a wrong diagnosis on 69.1% of the cases that are initially correctly classified by the AI-CAD model. Five breast imaging radiologists visually identify 29%-71% of the adversarial samples. Our study suggests an imperative need for continuing research on medical AI model’s safety issues and for developing potential defensive solutions against adversarial attacks.

Towards precision medicine based on a continuous deep learning optimization and ensemble approach

“…If the diagnostic performance is improved, a new model can be adopted so that the diagnostic performance of the CLS can be guaranteed to be stable or improve but not decrease. A study by Zhou et al 37 suggests an imperative need for research on medical AI model safety issues; thus, the use of optimal methods in this study ensured the safe operation of the model.…”

Cancer Informatics 2022: Real-World Data Yields Important Insights into the Conduct of Clinical Trials and Registries

“…Zhou, et al, [19] conducted an interesting study on the effect of adversarial images within the domain of breast cancer mammography. They found that human radiologists could identify the adversarial images at a generally higher rate of success than a deep learning-based computer-aided diagnosis system, illustrating the need for ongoing work in this area.…”

Artificial Intelligence in Nuclear Medicine: Opportunities, Challenges, and Responsibilities Toward a Trustworthy Ecosystem

“…There is limited awareness by the public and stakeholders about the level of evidence required to validate safety, security (39) and each specific clinical claim by AI software solutions, as well as the evidential requirements to verify the added-value for appropriate reimbursement.…”