News: 3 questions: Finding the pulse of robust AI in health

Creating and applying machine learning to understand and improve human health in ways that are effective, private and fair.

Mindy Blodgett | IMES

Artificial intelligence in health is at the cutting-edge of improvements in healthcare detection, delivery, and treatments. But an obstacle is ensuring that the medical models do not learn biased rules or recommendations, that in the end, can harm minority populations. This is the heart of the research conducted by Marzyeh Ghassemi, Germeshausen Career Development Professor, core faculty member at IMES, associate professor of Electrical Engineering and Computer Science (EECS), and CIFAR AI Chair, Vector Institute. She leads the Healthy ML Lab, which tackles the many novel technical opportunities for machine learning in health, and works to improve how AI is deployed for health and medicine.

She has been named one of MIT Tech Review’s 35 Innovators Under 35, a Sloan Research Fellow, a CIFAR Azrieli Global Scholar and an AI2050 Fellow; she has also received the 2018 Seth J. Teller Award, the 2023 MIT Prize for Open Data, a 2024 NSF CAREER Award, and a Google Research Scholar Award. Prior to her PhD in Computer Science at MIT, Ghassemi received an MSc. degree in biomedical engineering from Oxford University as a Marshall Scholar, and BS degrees in computer science and electrical engineering as a Goldwater Scholar at New Mexico State University.

You lead the Healthy ML Lab, which builds machine learning algorithms to improve healthcare decisions—to make them more equitable and effective—and you study how to ensure that state-of-the art large language models are used for enhanced fairness, and enforced for practical health deployment settings. You’ve shown how important it is for reproducible and ethical machine learning in health, along with acknowledging and understanding that bias may be inherent in models that learned with medical images, clinical notes, or through process and devices. Can you tell us some more about the work at Healthy ML?

At Healthy ML, we focus on the full path from data through deployment - not only is a medical model is accurate, but whether it is reliable across hospitals and populations, whether it has learned harmful shortcuts, and whether it actually improves decisions when people use it. For example, strong average performance in chest X-ray models can hide underdiagnosis for Black and female patients. We develop methods to detect these failures, quantify uncertainty, reduce bias, and design safer ways for clinicians and patients to work with AI.

One of the projects at the Healthy ML Lab is around integrating multi-modal clinical data, and using recurrent and convolution neural networks to predict when patients will need important interventions. The problem you are working to solve is that this task is complicated by data sources that are sparse, noisy, heterogeneous, and with imbalanced outcomes. You have shown that your models are able to significantly outperform baselines for intervention prediction, and provide insights into model learning. Can you say more about this?

Health data really are unusually difficult: vital signs are measured continuously, laboratory tests are ordered irregularly, clinical notes appear at different times, some patients have genomics and environmental data, and that's only for those who are lucky enough to have regular access to the healthcare system. Our analyses have shown that the models identify and use clinically meaningful patterns, some of which are sub-clinical. For example, by predicting higher respiratory rates and lower oxygen saturation before ventilation, and increasing blood sugar levels before a diabetes diagnosis. The larger lesson we've learned across projects is that multimodal models improve prediction, but we also need to examine what they learned before placing them in clinical workflows.

A recent study you were a part of found that a standardized or generic approach to designing AI systems that aid in diagnosis of skin diseases wasn’t a winning strategy, that while AI improved the accuracy of the detection of problems, AI explainability methods had different impacts according to the underlying knowledge of users. How do the AI explainability methods trigger bias in humans, and how could this impact healthcare decisions as providers at different levels of experience apply AI to diagnosis and treatment?

Unfortunately we have found that an explanation can act as persuasion rather than transparency. In our dermatology study, a fluent LLM explanation made lay users more likely to follow the AI, whether it was right or wrong. When the AI was correct, LLM explanations improved their accuracy by 13.4 percent; when it was wrong, accuracy fell by 21.1 percent, which is the largest decline among the explanation methods we tested. Fortunately, primary care physicians were much more resistant to incorrect AI guidance, In practice, this means that the same interface may help an experienced clinician test a hypothesis but lead a less experienced clinician, or a patient, to form a mistaken belief with greater confidence. We have to develop human-first workflows, tailor assistance to the user’s expertise, and avoid assuming that a more polished explanation is necessarily safer.