Bias in AI-based models for medical applications: challenges and mitigation strategies

Abstract

Artificial intelligence systems are increasingly being applied to healthcare. In surgery, AI applications hold promise as tools to predict surgical outcomes, assess technical skills, or guide surgeons intraoperatively via computer vision. On the other hand, AI systems can also suffer from bias, compounding existing inequities in socioeconomic status, race, ethnicity, religion, gender, disability, or sexual orientation. Bias particularly impacts disadvantaged populations, which can be subject to algorithmic predictions that are less accurate or underestimate the need for care. Thus, strategies for detecting and mitigating bias are pivotal for creating AI technology that is generalizable and fair. Here, we discuss a recent study that developed a new strategy to mitigate bias in surgical AI systems.

Machine Learning and Bias in Medical Imaging: Opportunities and Challenges

“…To date, AI technologies have gained approval without necessarily having robust external validation, 96 and the only high-level guidance is provided to promote fairness without any specific requirements to ensure mitigation of potential bias. 59 Notwithstanding plans to establish processes for monitoring of AI model performance in realworld settings, 93 there are growing calls for the FDA to establish a more structured process for initial evaluation and approval of AI technologies that are ideally aligned with how drugs and devices are regulated. 97,98 For this reason, we offer a framework for considering approaches to bias mitigation that are based on existing workflows for building new AI technologies while emphasizing transparency and also approximating the conventional 4 phases of drug development and review (Table 4).…”

“…91 Upon completion of a final model, initial and then routine reporting of model performance across data sets can be facilitated using tools such as model cards for algorithms and data sheets for data sets-intended to provide optimal transparency and updated guidance on how to use, validate, and interpret results from a given model applied to a given data set. [78][79][80]92,93 In addition to routinely applied and updated evaluations of overall performance and fairness metrics, these same tools are likely to add value for ongoing assessments of how an algorithm operates in realworld settings. will also for interdisciplinary teams to iterate on and further optimize models across multiple use scenerios.…”

“…Transparency of differential performance will also for interdisciplinary teams to iterate on and further optimize models across multiple use scenerios. 92,93…”

“…will also for interdisciplinary teams to iterate on and further optimize models across multiple use scenerios. 92,93 Given that transparency is essential for bias mitigation, concerns have been raised regarding the black box nature of commercial data sets and algorithms. Ideally, even proprietary models will have accessible documentation regarding training and testing data set features, model assumptions, and metrics of performance and fairness assessed across relevant subgroups.…”

Generative artificial intelligence in mental health care: potential benefits and current challenges

“…Since current datasets used in AI models are trained on non‐psychiatric sources, today all major AI chatbots clearly state that their products must not be used for clinical purposes. Even with proper training, risks of AI bias must be carefully explored, given numerous recent examples of clear harm in other medical fields 6 . A rapid glance at images generated by an AI program when asked to draw “schizophrenia” 7 visualized the extent to which extreme stigma and harmful bias have informed what current AI models conceptualize as mental illness.…”