News | Mammography | May 22, 2023

Effects of human-machine interaction must be carefully considered to ensure safety and accuracy

Interface of the artificial intelligence–based diagnostic system. Images show interface appearance when (A) loading a new mammogram, (B) performing the image evaluation, (C) displaying the results of the image evaluation via a heatmap, and (D) displaying the results of the image evaluation with the heatmap turned off.

Interface of the artificial intelligence–based diagnostic system. Images show interface appearance when (A) loading a new mammogram, (B) performing the image evaluation, (C) displaying the results of the image evaluation via a heatmap, and (D) displaying the results of the image evaluation with the heatmap turned off. Image courtesy of RSNA.

May 22, 2023 — Incorrect advice by an AI-based decision support system could seriously impair the performance of radiologists at every level of expertise when reading mammograms, according to a new study published in Radiology, a journal of the Radiological Society of North America (RSNA).

Often touted as a “second set of eyes” for radiologists, AI-based mammographic support systems are one of the most promising applications for AI in radiology. As the technology expands, there are concerns that it may make radiologists susceptible to automation bias—the tendency of humans to favor suggestions from automated decision-making systems. Several studies have shown that the introduction of computer-aided detection into the mammography workflow could impair radiologist performance. However, no studies have looked at the influence of AI-based systems on the performance of accurate mammogram readings by radiologists.

Researchers from institutions in Germany and the Netherlands set out to determine how automation bias can affect radiologists at varying levels of experience when reading mammograms aided by an AI system.

In the prospective experiment, 27 radiologists read 50 mammograms. They then provided their Breast Imaging Reporting and Data System (BI-RADS) assessment assisted by an AI system. BI-RADS is a standard system used by radiologists to describe and categorize breast imaging findings. While BI-RADS categorization is not a diagnosis, it is crucial in helping doctors determine the next steps in care.

Researchers presented the mammograms in two randomized sets. The first was a training set of 10 in which the AI suggested the correct BI-RADS category. The second set contained incorrect BI-RADS categories, purportedly suggested by AI, in 12 of the 40 mammograms.

The results showed that the radiologists were significantly worse at assigning the correct BI-RADS scores for the cases in which the purported AI suggested an incorrect BI-RADS category. For example, inexperienced radiologists assigned the correct BI-RADS score in almost 80% of cases in which the AI suggested the correct BI-RADS category. When the purported AI suggested the wrong category, their accuracy fell to less than 20%. Experienced radiologists—those with more than 15 years of experience on average—saw their accuracy fall from 82% to 45.5% when the purported AI suggested the incorrect category.

“We anticipated that inaccurate AI predictions would influence the decisions made by radiologists in our study, particularly those with less experience,” said study lead author Thomas Dratsch, M.D., Ph.D., from the Institute of Diagnostic and Interventional Radiology, at University Hospital Cologne in Cologne, Germany. “Nonetheless, it was surprising to find that even highly experienced radiologists were adversely impacted by the AI system’s judgments, albeit to a lesser extent than their less seasoned counterparts.”

The researchers said the results show why the effects of human-machine interaction must be carefully considered to ensure safe deployment and accurate diagnostic performance when combining human readers and AI.

“Given the repetitive and highly standardized nature of mammography screening, automation bias may become a concern when an AI system is integrated into the workflow,” Dr. Dratsch said. “Our findings emphasize the need for implementing appropriate safeguards when incorporating AI into the radiological process to mitigate the negative consequences of automation bias.”

Possible safeguards include presenting users with the confidence levels of the decision support system. In the case of an AI-based system, this could be done by showing the probability of each output. Another strategy involves teaching users about the reasoning process of the system. Ensuring that the users of a decision support system feel accountable for their own decisions can also help decrease automation bias, Dr. Dratsch said.

The researchers plan to use tools like eye-tracking technology to better understand the decision-making process of radiologists using AI.

“Moreover, we would like to explore the most effective methods of presenting AI output to radiologists in a way that encourages critical engagement while avoiding the pitfalls of automation bias,” Dr. Dratsch said.

For more information:

Related Breast Imaging Content: 

VIDEO: Research and Advancements in Breast Imaging Technology 

VIDEO: FDA Update on the US National Density Reporting Standard - A Discussion on the Final Rule 

One on One … with Wendie Berg, MD, PhD, FACR, FSBI 

Creating Patient Equity: A Breast Density Legislative Update 

FDA Needs to Ensure that Information on Dense Breast Notifications are Clear and Understandable to all Members of the Public 

AI Provides Accurate Breast Density Classification 

VIDEO: The Impact of Breast Density Technology and Legislation 

VIDEO: Personalized Breast Screening and Breast Density 

VIDEO: Breast Cancer Awareness - Highlights of the NCoBC 2016 Conference 

Fake News: Having Dense Breast Tissue is No Big Deal 

The Manic World of Social Media and Breast Cancer: Gratitude and Grief 

Single vs. Multiple Architectural Distortion on Digital Breast Tomosynthesis 

Today's Mammography Advancements  

Digital Breast Tomosynthesis Spot Compression Clarifies Ambiguous Findings  

AI DBT Impact on Mammography Post-breast Therapy  

ImageCare Centers Unveils PINK Better Mammo Service Featuring Profound AI  

Radiologist Fatigue, Experience Affect Breast Imaging Call Backs  

Fewer Breast Cancer Cases Between Screening Rounds with 3-D Mammography 

Study Finds Racial Disparities in Access to New Mammography Technology 

American College of Radiology (ACR) Launches Contrast-Enhanced Mammography Imaging Screening Trial (CMIST) in Collaboration With GE Healthcare and the Breast Cancer Research Foundation 

Related Content

Feature | Radiology Imaging | By Melinda Taschetta-Millane

It's been a fruitful month for! Here's a look at what ITN viewers found to be most interesting during the ...

Time June 01, 2023
News | Breast Density

May 25, 2023 — Researchers from the Mayo Clinic (Rochester, MN) and University of California, San Francisco confirmed ...

Time May 25, 2023
News | Breast Imaging

May 25, 2023 — GE HealthCare, a leader in breast cancer care technology and diagnostics, has today announced that NCCN ...

Time May 25, 2023
News | Ultrasound Women's Health

May 25, 2023 — According to an accepted manuscript published in ARRS’ own American Journal of Roentgenology (AJR), deep ...

Time May 25, 2023
News | Pediatric Imaging

May 24, 2023 — A new advanced form of computed tomography (CT) imaging called photon-counting computed tomography (PCCT) ...

Time May 24, 2023
News | Radiation Oncology

May 22, 2023 — Physicians and scientists from the UCLA Jonsson Comprehensive Cancer Center will discuss the latest ...

Time May 22, 2023
News | PET Imaging

May 22, 2023 — New research finds that the brains of otherwise healthy military personnel who are exposed to explosions ...

Time May 22, 2023
News | Artificial Intelligence

May 19, 2023 — The latest version of ChatGPT passed a radiology board-style exam, highlighting the potential of large ...

Time May 19, 2023
News | Radiology Imaging

May 19, 2023 — Asymptomatic adults with a high accumulation of fat in their muscles, known as myosteatosis, are at an ...

Time May 19, 2023
News | Computed Tomography (CT)

May 18, 2023 — Royal Philips, a global leader in health technology, announced the launch of the Philips CT 3500, a new ...

Time May 18, 2023
Subscribe Now