Using a standardized assessment, researchers in the UK compared the performance of a commercially available artificial intelligence (AI) algorithm with human readers of screening mammograms.

(A) Right mediolateral oblique unadulterated mammogram shows an 8-mm ill-defined mass (arrowhead), which, after biopsy, was determined to be a histologic grade 2 ductal carcinoma of no special type. (B) Mammogram shows findings by human readers (blue areas) and the Lunit INSIGHT MMG artificial intelligence (AI) algorithm (red cross). Each blue dot is a mark placed by an individual human reader on a perceived abnormality when the Personal Performance in Mammographic Screening (PERFORMS) case was read. A region of interest (pentagon) has been annotated by the PERFORMS scheme organizers and their expert radiology panel. AI has correctly marked the region of interest in the right breast for recall. Source: PERFORMS via Yan Chen. Image courtesy of RSNA


September 5, 2023 — Using a standardized assessment, researchers in the UK compared the performance of a commercially available artificial intelligence (AI) algorithm with human readers of screening mammograms. Results of their findings were published in Radiology, a journal of the Radiological Society of North America (RSNA).

Mammographic screening does not detect every breast cancer. False-positive interpretations can result in women without cancer undergoing unnecessary imaging and biopsy. To improve the sensitivity and specificity of screening mammography, one solution is to have two readers interpret every mammogram.

According to the researchers, double reading increases cancer detection rates by 6 to 15% and keeps recall rates low. However, this strategy is labor-intensive and difficult to achieve during reader shortages.

“There is a lot of pressure to deploy AI quickly to solve these problems, but we need to get it right to protect women’s health,” said Yan Chen, Ph.D., professor of digital screening at the University of Nottingham, United Kingdom.

Prof. Chen and her research team used test sets from the Personal Performance in Mammographic Screening, or PERFORMS, quality assurance assessment utilized by the UK’s National Health Service Breast Screening Program (NHSBSP), to compare the performance of human readers with AI. A single PERFORMS test consists of 60 challenging exams from the NHSBSP with abnormal, benign and normal findings. For each test mammogram, the reader’s score is compared to the ground truth of the AI results.

Breast imaging AI

Left mediolateral oblique mammogram. Unadulterated mammogram shows an asymmetric density (arrowhead) which, after biopsy, was determined to be a histologic grade 2 ductal carcinoma. (B) Artificial intelligence (AI) has correctly marked the region of interest in the left breast for recall (red cross) when set at a recall threshold of 2.91 or higher to match average human specificity, demonstrating a true-positive case. (C) AI has not marked the region of interest in the same breast when set at a recall threshold of 3.06 or higher, indicating a false-negative case. Blue dots indicate findings identified by the human readers. This shows how modifying the threshold for recall can impact the sensitivity of the AI model. Source: Personal Performance in Mammographic Screening via Yan Chen. Image courtesy of RSNA

 

“It’s really important that human readers working in breast cancer screening demonstrate satisfactory performance,” she said. “The same will be true for AI once it enters clinical practice.”

The research team used data from two consecutive PERFORMS test sets, or 120 screening mammograms, and the same two sets to evaluate the performance of the AI algorithm. The researchers compared the AI test scores with the scores of the 552 human readers, including 315 (57%) board-certified radiologists and 237 non-radiologist readers consisting of 206 radiographers and 31 breast clinicians.

“The 552 readers in our study represent 68% of readers in the NHSBSP, so this provides a robust performance comparison between human readers and AI,” Prof. Chen said.

Treating each breast separately, there were 161/240 (67%) normal breasts, 70/240 (29%) breasts with malignancies, and 9/240 (4%) benign breasts. Masses were the most common malignant mammographic feature (45/70 or 64.3%), followed by calcifications (9/70 or 12.9%), asymmetries (8/70 or 11.4%), and architectural distortions (8/70 or 11.4%). The mean size of malignant lesions was 15.5 mm.

No difference in performance was observed between AI and human readers in the detection of breast cancer in 120 exams. Human reader performance demonstrated mean 90% sensitivity and 76% specificity. AI was comparable in sensitivity (91%) and specificity (77%) compared to human readers.

“The results of this study provide strong supporting evidence that AI for breast cancer screening can perform as well as human readers,” Prof. Chen said.

Prof. Chen said more research is needed before AI can be used as a second reader in clinical practice.

“I think it is too early to say precisely how we will ultimately use AI in breast screening,” she said. “The large prospective clinical trials that are ongoing will tell us more. But no matter how we use AI, the ability to provide ongoing performance monitoring will be crucial to its success.”

Prof. Chen said it’s important to recognize that AI performance can drift over time, and algorithms can be affected by changes in the operating environment.

“It’s vital that imaging centers have a process in place to provide ongoing monitoring of AI once it becomes part of clinical practice,” she said. “There are no other studies to date that have compared such a large number of human reader performance in routine quality assurance test sets to AI, so this study may provide a model for assessing AI performance in a real-world setting.”

For more information: www.rsna.org

Related breast density content: 

VIDEO: FDA Update on the US National Density Reporting Standard - A Discussion on the Final Rule   

One on One … with Wendie Berg, MD, PhD, FACR, FSBI   

Task Force Issues New Draft Recommendation Statement on Screening for Breast Cancer   

Creating Patient Equity: A Breast Density Legislative Update   

FDA Needs to Ensure that Information on Dense Breast Notifications are Clear and Understandable to all Members of the Public   

AI Provides Accurate Breast Density Classification   

VIDEO: The Impact of Breast Density Technology and Legislation   

VIDEO: Personalized Breast Screening and Breast Density   

VIDEO: Breast Cancer Awareness - Highlights of the NCoBC 2016 Conference   

Fake News: Having Dense Breast Tissue is No Big Deal   

The Manic World of Social Media and Breast Cancer: Gratitude and Grief   

Related breast imaging content:  

Breast Cancer Risk Calculator Can Assess Risk of Advanced Breast Cancer  

Uncertainty About Breast Cancer Risk and Screening Choices and Perceived Risk Heighten with Breast Density Awareness Following Mammography  

Creating Patient Equity: A Breast Density Legislative Update  

FDA Needs to Ensure that Information on Dense Breast Notifications are Clear and Understandable to all Members of the Public  

AI Provides Accurate Breast Density Classification  

VIDEO: The Impact of Breast Density Technology and Legislation  

VIDEO: Personalized Breast Screening and Breast Density  

VIDEO: Breast Cancer Awareness - Highlights of the NCoBC 2016 Conference  

Fake News: Having Dense Breast Tissue is No Big Deal  

The Manic World of Social Media and Breast Cancer: Gratitude and Grief 


Related Content

News | RSNA

May 29, 2024 — The Radiological Society of North America (RSNA) has launched the 2024 RSNA Lumbar Spine Degenerative ...

Time May 29, 2024
arrow
News | Breast Imaging

May 28, 2024 — iCAD, Inc., a global leader in clinically proven AI-powered cancer detection solutions, announced a ...

Time May 28, 2024
arrow
News | Lung Imaging

May 24, 2024 — Smokers who have small abnormalities on their CT scans that grow over time have a greater likelihood of ...

Time May 24, 2024
arrow
News | FDA

May 22, 2024 — The U.S. Food and Drug Administration (FDA) has issued a recall of the Hologic Inc. BioZorb marker due to ...

Time May 22, 2024
arrow
News | Artificial Intelligence

May 22, 2024 — Lunit, a provider of Artificial intelligence (AI)-powered solutions for cancer diagnostics and ...

Time May 22, 2024
arrow
Sponsored Content | Case Study | Enterprise Imaging

Having the most efficient clinical workflows with enhanced diagnostic capabilities is a major goal for clinicians and ...

Time May 16, 2024
arrow
News | Prostate Cancer

May 13, 2024 — Avenda Health, an AI healthcare company creating the future of personalized prostate cancer care, unveils ...

Time May 13, 2024
arrow
News | Breast Imaging

May 10, 2024 — According to the Summa Cum Laude Award-Winning Online Poster presented during the 124th ARRS Annual ...

Time May 10, 2024
arrow
News | Radiology Business

May 6, 2024 — ScreenPoint Medical’s Board of Directors has announced the appointment of Peter Kroese as the new Chief ...

Time May 06, 2024
arrow
News | Mammography

May 6, 2024 — Enable Me, a VELA Medical company, cited major new research by Siemens Healthineers entitled, “The future ...

Time May 06, 2024
arrow
Subscribe Now