Locally run large language models (LLMs) may be a feasible option for extracting data from text-based radiology reports while preserving patient privacy

Getty Images


October 13, 2023 — Locally run large language models (LLMs) may be a feasible option for extracting data from text-based radiology reports while preserving patient privacy, according to a new study from the National Institutes of Health Clinical Center (NIH CC) published in Radiology, a journal of the Radiological Society of North America (RSNA). LLMs are deep-learning models trained to understand and generate text in a human-like way. 

Recently released LLM models such as ChatGPT and GPT-4 have garnered attention. However, they are not compatible with healthcare data due to privacy constraints. 

“ChatGPT and GPT-4 are proprietary models that require the user to send data to OpenAI sources for processing, which would require de-identifying patient data,” said senior author Ronald M. Summers, M.D., Ph.D., senior investigator in the Radiology and Imaging Sciences Department at the NIH. “Removing all patient health information is labor-intensive and infeasible for large sets of reports.” 

In this study, led by Pritam Mukherjee, Ph.D., staff scientist at the NIH CC, researchers tested the feasibility of using a locally run LLM, Vicuna-13B, to label key findings from chest X-ray reports from the NIH and the Medical Information Mart for Intensive Care (MIMIC) Database, a publicly available dataset of de-identified electronic health records. 

“Preliminary evaluation has shown that Vicuna, a free publicly available LLM, approaches the performance of ChatGPT in tasks such as multi-lingual question answering,” Dr. Summers said. 

The study dataset included 3,269 chest X-ray reports obtained from MIMIC and 25,596 reports from the NIH. 

Using two prompts for two tasks, the researchers asked the LLM to identify and label the presence or absence of 13 specific findings on the chest X-ray reports. Researchers compared the LLM’s performance with two widely used non-LLM labeling tools. 

A statistical analysis of the LLM output showed moderate to substantial agreement with the non-LLM computer programs. 

“Our study demonstrated that the LLM’s performance was comparable to the current reference standard,” Dr. Summers said. “With the right prompt and the right task, we were able to achieve agreement with currently used labeling tools.” 

Dr. Summers said LLMs that can be run locally will be useful in creating large data sets for AI research without compromising patient privacy. 

“LLMs have turned the whole paradigm of natural language processing on its head,” he said. “They have the potential to do things that we've had difficulty doing with traditional pre-large language models.” 

Dr. Summers said LLM tools could be used to extract important information from other text-based radiology reports and medical records, and as a tool for identifying disease biomarkers. 

“My lab has been focusing on extracting features from diagnostic images,” he said. “With tools like Vicuna, we can extract features from the text and combine them with features from images for input into sophisticated AI models that may be able to answer clinical questions. 

“LLMs that are free, privacy-preserving, and available for local use are game changers,” he said. “They're really allowing us to do things that we weren't able to do before.” 

For more information: www.rsna.org 

 

Related Artificial Intelligence Content: 

Broader Digital Health Trends 

AiMed Global Summit 2023 to Focus on “Changing Healthcare One Connection at a Time” in San Diego 

AiMed Global Summit’s Lineup Announced 

Making AI an Integral Part of the Solution 

AiMed 2023: Changing Healthcare One Connection at a Time 

Find more AiMed23 conference coverage here 

The Pros and Cons of Using ChatGPT in Clinical Radiology: An Open Discussion 

ChatGPT Passes Radiology Board Exam 

RamSoft Harnesses ChatGPT to Supercharge Their Medical Imaging Patient App, Blume - Launch at HIMSS 2023 

JNM Explores Potential Applications for ChatGPT in Nuclear Medicine and Molecular Imaging 

New Research Suggests AI Image Generation Using DALL-E 2 has Promising Future in Radiology 


Related Content

News | Lung Imaging

June 11, 2025 — To prepare healthcare workforces and providers for an AI-driven future, Qure.ai has expanded its Global ...

Time June 11, 2025
arrow
News | Radiology Imaging

June 10, 2025 — CIVIE has announced the official launch of RadPod, an AI-driven, on-demand radiology platform designed ...

Time June 10, 2025
arrow
News | Ultrasound Imaging

June 4, 2025 — RadNet, Inc., a provider of high-quality, cost-effective diagnostic imaging services and digital health ...

Time June 09, 2025
arrow
News | Mammography

June 9, 2025 — A new independent, peer-reviewed study published in the journal Clinical Breast Cancer reinforces the ...

Time June 09, 2025
arrow
News | Imaging Software Development

June 05, 2025 — Nano-X Imaging Ltd. has announced that its deep-learning medical imaging analytics subsidiary, Nanox AI ...

Time June 05, 2025
arrow
News | Prostate Cancer

June 5, 2025 – Artera, the developer of multimodal artificial intelligence (MMAI)-based prognostic and predictive cancer ...

Time June 05, 2025
arrow
News | Ultrasound Imaging

June 3, 2025 — In a collaborative study between the Departments of Radiology at the Children’s Hospital of Philadelphia ...

Time June 04, 2025
arrow
News | Breast Imaging

June 2, 2025 — Clairity, Inc., a digital health innovator advancing AI-driven healthcare solutions, has received U.S ...

Time June 02, 2025
arrow
News | Magnetic Resonance Imaging (MRI)

Hyperfine, Inc., producer of the world’s first FDA-cleared AI-powered portable MRI system for the brain — the Swoop ...

Time May 29, 2025
arrow
News | Imaging Software Development

May 27, 2025 — DeepLook Medical, a company advancing medical imaging through visual enhancement technology, recently ...

Time May 28, 2025
arrow
Subscribe Now