The COVID-19 crisis has exposed some of the most pressing challenges affecting healthcare and highlighted the benefits that robust integration of digital and AI technologies in the healthcare setting may bring. Although medical solutions based on AI are growing rapidly, regulatory issues and policy initiatives including ownership and control of data, data sharing, privacy protection, telemedicine, and accountability need to be carefully and continually addressed as AI research requires robust and ethical guidelines, demanding an update of the legal and regulatory framework all over the world. Several recently proposed regulatory frameworks provide a solid foundation but do not address a number of issues that may prevent algorithms from being fully trusted. A global effort is needed for an open, mature conversation about the best possible way to guard against and mitigate possible harms to realize the potential of AI across health systems in a respectful and ethical way. This conversation must include national and international policymakers, physicians, digital health and machine learning leaders from industry and academia. If this is done properly and in a timely fashion, the potential of AI in healthcare will be realized.