Nov. 13, 2024 — Mass General Brigham AI is hosting the Healthcare AI Challenge, a multi-institutional virtual, interactive series of events where healthcare professionals can explore and assess the latest AI healthcare technologies in real-world healthcare scenarios.
The Healthcare AI Challenge Collaborative is launching with a diverse set of healthcare institutions and their healthcare professionals, including
- Mass General Brigham; Emory Healthcare;
- The Department of Radiology at the University of Wisconsin School of Medicine and Public Health; and
- The Department of Radiology at the University of Washington School of Medicine.
The American College of Radiology (ACR), a professional medical society representing radiologists, has also joined the Healthcare AI Challenge Collaborative as a founding member to ensure its 42,000 members have access to the Healthcare AI Challenge. Additional member institutions will be announced in subsequent phases.
“The velocity of AI innovations and breadth of their healthcare applications continues to increase. This unprecedented growth leaves clinicians struggling to determine the effectiveness of these innovations in safely delivering value to healthcare providers and our patients,” explained Keith Dreyer, DO, PhD, Chief Data Science Officer at Mass General Brigham, and leader of Mass General Brigham AI. “The Healthcare AI Challenge is a collective response to the complexities involved in advancing the responsible development and use of AI in healthcare. This new approach strives to put clinicians in the driver’s seat, allowing them to evaluate the utility of different AI technologies and ultimately, determine which solutions have the greatest promise to advance patient care.”
Participating healthcare professionals will be granted access to the Healthcare AI Challenge that features late breaking AI solutions they can assess for effectiveness on specific medical tasks, such as providing medical image interpretation, in a simulated environment. Participants with relevant healthcare credentials can then provide their feedback on the solutions’ performance and utility, which will generate publicly available insights and analytics. By crowdsourcing input from healthcare professionals, the Healthcare AI Challenge seeks to create continuous, consistent and reliable expert evaluations of AI solutions in medicine. Importantly, scaling the evaluation of these technologies and sharing the insights broadly and transparently can result in societal benefit for healthcare stakeholders and patients globally.
“We need to go beyond collaboratives that come to consensus on how to think about AI,” said Alistair Erskine, MD, chief information and digital officer at Emory Healthcare and Emory University. “We need healthcare delivery communities to provide real-world experience of the application of AI at the point of care. That is what the Healthcare AI Challenge is designed to do.”
Interactive and Data-Driven
Radiology was chosen as the first in a series of healthcare AI events given the historic use of AI solutions to make rapid and meaningful impact in the field against the wave of generative AI model proliferation.
Health care professionals at instituyions that are a part of the Collaborative who register can log onto the Healthcare AI Challenge, select one of several events — such as image interpretation — and choose from a series of challenges to assess any one of the multiple foundation models available on the platform. The image interpretation challenges include questions focused on draft report generation, key findings, differential diagnosis, among others. The expert then rates the clinical skill level of the foundation models’ responses, which contributes to the insights and analytics rankings. Only verified healthcare professionals can participate in challenges that contribute to the rankings. The results of the Healthcare AI Challenge can be followed by the general public at HealthcareAIChallenge.org.
“We are facing an overwhelming influx of FDA-approved AI tools in healthcare, especially in radiology. Forming an academic collaborative could play a crucial role in validating and selecting these tools, ensuring they adhere to the highest standards of efficacy and safety,” said Dushyant Sahani, MD, chair and professor of the Department of Radiology at the University of Washington School of Medicine.
The rankings generated by the Healthcare AI Challenge can serve to provide industry, healthcare stakeholders, and the public with a transparent analysis of AI solutions’ performance across a wide range of healthcare data and clinical scenarios. User activity and expert feedback are expected to provide valuable insights to help AI developers better understand how healthcare professionals consider AI healthcare solutions in the context of providing clinical value. In turn, developers can enhance these technologies so that they are fit-for-purpose, commercially viable and clinically relevant. This may foster collaborations and knowledge-sharing with industry to help ensure responsible adoption of AI in healthcare settings.
“The American College of Radiology is leading safe and effective radiological artificial intelligence implementation by offering real-world solutions that address the challenges radiologists face today,” said Christoph Wald, MD, PhD, MBA, FACR, vice chair of the ACR Board of Chancellors and chair of the ACR Commission on Informatics. “The ACR is strengthening care by promoting AI use to identify efficiencies, improve outcomes, build safeguards and optimize patient care. Participating in the Healthcare AI Challenge is another important step in ACR’s efforts to facilitate the safe, responsible and transparent evaluation of AI, especially as the use of newer foundation models is poised to unfold in healthcare.”
The Healthcare AI Challenge will continually add new AI solutions, events, multimodal diverse data, domain experts and specialties to its interactive environment, including pathology, genomics, waveform, as well as public and specialized text-based foundation models.
June 12, 2025 
