Embryo selection with artificial intelligence: how to evaluate and compare methods?

Abstract

Embryo selection within in vitro fertilization (IVF) is the process of evaluating qualities of fertilized oocytes (embryos) and selecting the best embryo(s) available within a patient cohort for subsequent transfer or cryopreservation. In recent years, artificial intelligence (AI) has been used extensively to improve and automate the embryo ranking and selection procedure by extracting relevant information from embryo microscopy images. The AI models are evaluated based on their ability to identify the embryo(s) with the highest chance(s) of achieving a successful pregnancy. Whether such evaluations should be based on ranking performance or pregnancy prediction, however, seems to divide studies. As such, a variety of performance metrics are reported, and comparisons between studies are often made on different outcomes and data foundations. Moreover, superiority of AI methods over manual human evaluation is often claimed based on retrospective data, without any mentions of potential bias. In this paper, we provide a technical view on some of the major topics that divide how current AI models are trained, evaluated and compared. We explain and discuss the most common evaluation metrics and relate them to the two separate evaluation objectives, ranking and prediction. We also discuss when and how to compare AI models across studies and explain in detail how a selection bias is inevitable when comparing AI models against current embryo selection practice in retrospective cohort studies.

Robust and generalizable embryo selection based on artificial intelligence and time-lapse image sequences

“…The sub-group analysis is an internal validation where the model is tested on a fraction of the training data [ 50 ]. Potentially this presents an optimistic performance estimate as the test data has the same patient population and distribution as in the training data.…”

“…Potentially this presents an optimistic performance estimate as the test data has the same patient population and distribution as in the training data. This issue can be addressed using a geographic internal validation (a clinic hold-out test) or even stronger by using an external validation, where the model is tested under new conditions such as new clinics, time-periods, procedures, or populations that were not used during model development [ 50 ].…”

The Blurring of Boundaries in Bioscientific Discourse

“…Such an incorporation of AI would also be useful for making the training of new professionals (collaboration) more effective, thus increasing the efficiency of the whole assisted reproduction process. Similar ways of applying AI can be imagined in other areas of assisted reproduction, although it is also clear that there will be fundamental reservations about these applications of AI as well, based on concerns regarding the safety of the technologies and procedures involved (Kragh and Karstoft 2021). Safety is the core reason for a third possible way of applying AI in assisted reproduction.…”

“…Proponents of the application of automation in assisted reproduction therefore usually argue in favour of using already proven advanced technologies that would be sensitively adapted to the possibilities and needs of specific interventions in human reproduction (Casciani et al 2021). Proponents of this idea often point to the need for a judicious consideration of the implications of changing standards in assisted reproduction (Kragh and Karstoft 2021). Outcomes are also a fundamental topic of debate about the future of assisted reproduction, as a growing number of actors in this debate assert that the proper automation of assisted reproduction could dramatically increase its success rate while significantly reducing the costs of application (Meseguer et al 2012;Varghese and Charalampos 2019;Casciani et al 2021).…”

“…Robotics in this area would allow for the reduction of the risk of potential damage to gametes and embryos, as it would reduce the number of steps currently required for the assisted reproduction procedures in question. Consequently, this procedure could also lead to the greater standardization of embryo and gamete handling (Kragh and Karstoft 2021). Furthermore, research into the application of robotics in assisted reproduction is not only taking place at the level of the "microcosm".…”