Background Artificial intelligence (AI) is rapidly expanding in medicine despite a lack of consensus on its application and evaluation. Objective We sought to identify current frameworks guiding the application and evaluation of AI for predictive analytics in medicine and to describe the content of these frameworks. We also assessed what stages along the AI translational spectrum (ie, AI development, reporting, evaluation, implementation, and surveillance) the content of each framework has been discussed. Methods We performed a literature review of frameworks regarding the oversight of AI in medicine. The search included key topics such as “artificial intelligence,” “machine learning,” “guidance as topic,” and “translational science,” and spanned the time period 2014-2022. Documents were included if they provided generalizable guidance regarding the use or evaluation of AI in medicine. Included frameworks are summarized descriptively and were subjected to content analysis. A novel evaluation matrix was developed and applied to appraise the frameworks’ coverage of content areas across translational stages. Results Fourteen frameworks are featured in the review, including six frameworks that provide descriptive guidance and eight that provide reporting checklists for medical applications of AI. Content analysis revealed five considerations related to the oversight of AI in medicine across frameworks: transparency, reproducibility, ethics, effectiveness, and engagement. All frameworks include discussions regarding transparency, reproducibility, ethics, and effectiveness, while only half of the frameworks discuss engagement. The evaluation matrix revealed that frameworks were most likely to report AI considerations for the translational stage of development and were least likely to report considerations for the translational stage of surveillance. Conclusions Existing frameworks for the application and evaluation of AI in medicine notably offer less input on the role of engagement in oversight and regarding the translational stage of surveillance. Identifying and optimizing strategies for engagement are essential to ensure that AI can meaningfully benefit patients and other end users.

BACKGROUND Artificial intelligence (AI) is rapidly expanding in medicine even while lacking formal oversight. OBJECTIVE We sought to identify and describe considerations for the oversight of AI in medicine. We also explored where along the translational process (i.e., AI development, reporting, evaluation, implementation, and surveillance) these considerations were targeted. METHODS We conducted a targeted review of frameworks for the oversight of AI in medicine. The search included key topics such as ‘artificial intelligence,’ ‘machine learning’, ‘guidance as topic’, ‘translational science’, ‘medical device legislation’, and ‘evaluation study,’ and spanned the time period 2014-2021. Frameworks were included if they described translational considerations for AI. The included frameworks were summarized descriptively. Content analysis was used to identify considerations for the oversight of AI in medicine. An evaluation matrix methodology was used to map each consideration across the different translational stages for each framework. RESULTS Six frameworks were featured in the review and included peer reviewed and white papers from consortium and professional organizations. Content analysis of the frameworks revealed five overarching considerations related to the oversight of AI in medicine, including: transparency, reproducibility, ethics, effectiveness, and engagement. All frameworks included discussions regarding transparency, reproducibility, ethics, and effectiveness, while only half of frameworks discussed engagement. The evaluation matrix revealed that frameworks were most likely to report AI considerations for the translational stage of development, and least likely to report considerations for the translational stage of surveillance. CONCLUSIONS Frameworks provided broad guidance for the oversight of AI in medicine, but notably offered less input on the role engagement approaches for oversight, and regarding the translational stage of surveillance. Identifying and optimizing strategies for engagement is essential to ensure that AI can meaningfully benefit patients and other end-users.