News | Mammography | May 22, 2023

Effects of human-machine interaction must be carefully considered to ensure safety and accuracy

Interface of the artificial intelligence–based diagnostic system. Images show interface appearance when (A) loading a new mammogram, (B) performing the image evaluation, (C) displaying the results of the image evaluation via a heatmap, and (D) displaying the results of the image evaluation with the heatmap turned off.

Interface of the artificial intelligence–based diagnostic system. Images show interface appearance when (A) loading a new mammogram, (B) performing the image evaluation, (C) displaying the results of the image evaluation via a heatmap, and (D) displaying the results of the image evaluation with the heatmap turned off. Image courtesy of RSNA.

May 22, 2023 — Incorrect advice by an AI-based decision support system could seriously impair the performance of radiologists at every level of expertise when reading mammograms, according to a new study published in Radiology, a journal of the Radiological Society of North America (RSNA).

Often touted as a “second set of eyes” for radiologists, AI-based mammographic support systems are one of the most promising applications for AI in radiology. As the technology expands, there are concerns that it may make radiologists susceptible to automation bias—the tendency of humans to favor suggestions from automated decision-making systems. Several studies have shown that the introduction of computer-aided detection into the mammography workflow could impair radiologist performance. However, no studies have looked at the influence of AI-based systems on the performance of accurate mammogram readings by radiologists.

Researchers from institutions in Germany and the Netherlands set out to determine how automation bias can affect radiologists at varying levels of experience when reading mammograms aided by an AI system.

In the prospective experiment, 27 radiologists read 50 mammograms. They then provided their Breast Imaging Reporting and Data System (BI-RADS) assessment assisted by an AI system. BI-RADS is a standard system used by radiologists to describe and categorize breast imaging findings. While BI-RADS categorization is not a diagnosis, it is crucial in helping doctors determine the next steps in care.

Researchers presented the mammograms in two randomized sets. The first was a training set of 10 in which the AI suggested the correct BI-RADS category. The second set contained incorrect BI-RADS categories, purportedly suggested by AI, in 12 of the 40 mammograms.

The results showed that the radiologists were significantly worse at assigning the correct BI-RADS scores for the cases in which the purported AI suggested an incorrect BI-RADS category. For example, inexperienced radiologists assigned the correct BI-RADS score in almost 80% of cases in which the AI suggested the correct BI-RADS category. When the purported AI suggested the wrong category, their accuracy fell to less than 20%. Experienced radiologists—those with more than 15 years of experience on average—saw their accuracy fall from 82% to 45.5% when the purported AI suggested the incorrect category.

“We anticipated that inaccurate AI predictions would influence the decisions made by radiologists in our study, particularly those with less experience,” said study lead author Thomas Dratsch, M.D., Ph.D., from the Institute of Diagnostic and Interventional Radiology, at University Hospital Cologne in Cologne, Germany. “Nonetheless, it was surprising to find that even highly experienced radiologists were adversely impacted by the AI system’s judgments, albeit to a lesser extent than their less seasoned counterparts.”

The researchers said the results show why the effects of human-machine interaction must be carefully considered to ensure safe deployment and accurate diagnostic performance when combining human readers and AI.

“Given the repetitive and highly standardized nature of mammography screening, automation bias may become a concern when an AI system is integrated into the workflow,” Dr. Dratsch said. “Our findings emphasize the need for implementing appropriate safeguards when incorporating AI into the radiological process to mitigate the negative consequences of automation bias.”

Possible safeguards include presenting users with the confidence levels of the decision support system. In the case of an AI-based system, this could be done by showing the probability of each output. Another strategy involves teaching users about the reasoning process of the system. Ensuring that the users of a decision support system feel accountable for their own decisions can also help decrease automation bias, Dr. Dratsch said.

The researchers plan to use tools like eye-tracking technology to better understand the decision-making process of radiologists using AI.

“Moreover, we would like to explore the most effective methods of presenting AI output to radiologists in a way that encourages critical engagement while avoiding the pitfalls of automation bias,” Dr. Dratsch said.

For more information:

Related Breast Imaging Content: 

VIDEO: Research and Advancements in Breast Imaging Technology 

VIDEO: FDA Update on the US National Density Reporting Standard - A Discussion on the Final Rule 

One on One … with Wendie Berg, MD, PhD, FACR, FSBI 

Creating Patient Equity: A Breast Density Legislative Update 

FDA Needs to Ensure that Information on Dense Breast Notifications are Clear and Understandable to all Members of the Public 

AI Provides Accurate Breast Density Classification 

VIDEO: The Impact of Breast Density Technology and Legislation 

VIDEO: Personalized Breast Screening and Breast Density 

VIDEO: Breast Cancer Awareness - Highlights of the NCoBC 2016 Conference 

Fake News: Having Dense Breast Tissue is No Big Deal 

The Manic World of Social Media and Breast Cancer: Gratitude and Grief 

Single vs. Multiple Architectural Distortion on Digital Breast Tomosynthesis 

Today's Mammography Advancements  

Digital Breast Tomosynthesis Spot Compression Clarifies Ambiguous Findings  

AI DBT Impact on Mammography Post-breast Therapy  

ImageCare Centers Unveils PINK Better Mammo Service Featuring Profound AI  

Radiologist Fatigue, Experience Affect Breast Imaging Call Backs  

Fewer Breast Cancer Cases Between Screening Rounds with 3-D Mammography 

Study Finds Racial Disparities in Access to New Mammography Technology 

American College of Radiology (ACR) Launches Contrast-Enhanced Mammography Imaging Screening Trial (CMIST) in Collaboration With GE Healthcare and the Breast Cancer Research Foundation 

Related Content

News | Artificial Intelligence

February 28, 2024 — iCAD, Inc., a global leader in clinically proven AI-powered solutions that enable medical providers ...

Time February 28, 2024
News | Women's Health

February 27, 2024 — Hologic, Inc. continues to deliver on its commitment to advancing women’s health by unveiling new ...

Time February 27, 2024
News | Artificial Intelligence

February 27, 2024 — As artificial intelligence (AI) is increasingly used in radiology, researchers caution that it’s ...

Time February 27, 2024
News | Breast Imaging

February 23, 2024 — ScreenPoint Medical is showcasing its industry leading Transpara Breast AI at the 2024 European ...

Time February 23, 2024
News | Breast Imaging

February 22, 2024 — The FAST-Forward randomized trial from the UK found that ultrahypofractionated whole breast ...

Time February 22, 2024
News | Computed Tomography (CT)

February 20, 2024 — Ultrahigh-spatial-resolution photon-counting detector CT improved assessment of coronary artery ...

Time February 20, 2024
News | Breast Imaging

February 20, 2024 — Annual breast cancer screening beginning at age 40 and continuing to at least age 79 results in the ...

Time February 20, 2024
Sponsored Content | Videos | Enterprise Imaging

Clinicians and referring physicians need fast, secure access to both patient data and their diagnostic or viewing tool ...

Time February 15, 2024
News | FDA

February 15, 2024 — Merit Medical Systems, Inc., a global leader of healthcare technology, has received US Food and Drug ...

Time February 15, 2024
Sponsored Content | Videos | Digital Radiography (DR)

Agfa Radiology Solutions is committed to enhancing clinical outcomes and operational efficiency, underscoring its ...

Time February 12, 2024
Subscribe Now