News | Artificial Intelligence | August 07, 2023

Scientists design new way to score accuracy of AI-generated radiology reports 

Scientists design new way to score accuracy of AI-generated radiology reports 

August 7, 2023 — AI tools that quickly and accurately create detailed narrative reports of a patient’s CT scan or X-ray can greatly ease the workload of busy radiologists. 

Instead of merely identifying the presence or absence of abnormalities on an image, these AI reports convey complex diagnostic information, detailed descriptions, nuanced findings, and appropriate degrees of uncertainty. In short, they mirror how human radiologists describe what they see on a scan. 

Several AI models capable of generating detailed narrative reports have begun to appear on the scene. With them have come automated scoring systems that periodically assess these tools to help inform their development and augment their performance. 

So how well do the current systems gauge an AI model’s radiology performance? 

The answer is good but not great, according to a new study by researchers at Harvard Medical School published Aug. 3 in the journal Patterns

Ensuring that scoring systems are reliable is critical for AI tools to continue to improve and for clinicians to trust them, the researchers said, but the metrics tested in the study failed to reliably identify clinical errors in the AI reports, some of them significant. The finding, the researchers said, highlights an urgent need for improvement and the importance of designing high-fidelity scoring systems that faithfully and accurately monitor tool performance. 

The team tested various scoring metrics on AI-generated narrative reports. The researchers also asked six human radiologists to read the AI-generated reports. 

The analysis showed that compared with human radiologists, automated scoring systems fared worse in their ability to evaluate the AI-generated reports. They misinterpreted and, in some cases, overlooked clinical errors made by the AI tool. 

“Accurately evaluating AI systems is the critical first step toward generating radiology reports that are clinically useful and trustworthy,” said study senior author Pranav Rajpurkar, assistant professor of biomedical informatics in the Blavatnik Institute at HMS. 

Improving the score 

In an effort to design better scoring metrics, the team designed a new method (RadGraph F1) for evaluating the performance of AI tools that automatically generate radiology reports from medical images. 

They also designed a composite evaluation tool (RadCliQ) that combines multiple metrics into a single score that better matches how a human radiologist would evaluate an AI model’s performance. 

Using these new scoring tools to evaluate several state-of-the-art AI models, the researchers found a notable gap between the models’ actual score and the top possible score. 

“Measuring progress is imperative for advancing AI in medicine to the next level,” said co-first author Feiyang ‘Kathy’ Yu, a research associate in the Rajpurkar lab. “Our quantitative analysis moves us closer to AI that augments radiologists to provide better patient care.” 

Long term, the researchers’ vision is to build generalist medical AI models that perform a range of complex tasks, including the ability to solve problems never before encountered. Such systems, Rajpurkar said, could fluently converse with radiologists and physicians about medical images to assist in diagnosis and treatment decisions. 

The team also aims to develop AI assistants that can explain and contextualize imaging findings directly to patients using everyday plain language. 

“By aligning better with radiologists, our new metrics will accelerate development of AI that integrates seamlessly into the clinical workflow to improve patient care,” Rajpurkar said. 

For more information: https://hms.harvard.edu/ 

 

Related Artificial Intelligence Content: 

AiMed Global Summit 2023 to Focus on “Changing Healthcare One Connection at a Time” in San Diego 

AiMed Global Summit’s Lineup Announced 

AiMed 2023: Changing Healthcare One Connection at a Time 

Find more AiMed23 conference coverage here 

The Pros and Cons of Using ChatGPT in Clinical Radiology: An Open Discussion 

ChatGPT Passes Radiology Board Exam 

RamSoft Harnesses ChatGPT to Supercharge Their Medical Imaging Patient App, Blume - Launch at HIMSS 2023 

JNM Explores Potential Applications for ChatGPT in Nuclear Medicine and Molecular Imaging 

New Research Suggests AI Image Generation Using DALL-E 2 has Promising Future in Radiology 


Related Content

News | Lung Imaging

April, 15, 2025 — Optellum has entered an agreement with Bristol Myers Squibb to leverage AI in early diagnosis and ...

Time April 17, 2025
arrow
News | Pediatric Imaging

April 10, 2025 — Cincinnati Children’s and GE HealthCare will form a strategic research program focused on driving ...

Time April 10, 2025
arrow
News | Breast Imaging

March 20, 2025 — GE HealthCare has launched Invenia Automated Breast Ultrasound (ABUS) Premium, the latest 3D ultrasound ...

Time March 21, 2025
arrow
News | X-Ray

March 18, 2025 — GE HealthCare recently announced a collaboration with NVIDIA expanding the existing relationship ...

Time March 19, 2025
arrow
News | SPECT Imaging

Feb. 5, 2025 — Serac Healthcare Ltd., a clinical radiopharmaceutical company developing an innovative molecular imaging ...

Time February 05, 2025
arrow
News | Ultrasound Imaging

Jan. 28, 2025 — GE HealthCare recently announced it has received 510(k) clearance from the United States Food and Drug ...

Time January 29, 2025
arrow
News | Breast Imaging

Dec.11, 2024 — iCAD, Inc., a provider of clinically proven AI-powered cancer detection solutions, recently announced ...

Time December 18, 2024
arrow
News | Computed Tomography (CT)

Dec. 3, 2024 — During RSNA '24, GE HealthCare announced the 510(k) submission to the U.S. Food and Drug Administration ...

Time December 18, 2024
arrow
News | SPECT Imaging

Dec. 2, 2024 — GE HealthCare has agreed to acquire full ownership of Nihon Medi-Physics Co., Ltd (NMP), by purchasing ...

Time December 05, 2024
arrow
News | Ultrasound Imaging

Dec. 1, 2024 — At RSNA 2024, Siemens Healthineers introduced Acuson Sequoia 3.51, a major software and hardware update ...

Time December 04, 2024
arrow
Subscribe Now