News | Radiology Imaging | January 12, 2022

Investigating Hallucinations in Medical Imaging

Beckman Institute researchers have mathematically defined a type of error that can arise during routine medical imaging, called hallucinations

From left-to-right are examples of a true object, a reconstructed estimate of the object produced by use of a U-Net from tomographic measurements, the total error map, the error in the null component of the reconstructed object, and the error in the measurement component of the reconstructed object. The two rows correspond to different objects. In each case, the true object is outside the respective training data distribution of the U-Net and phase noise was added to the measurements prior to image reconstr

From left-to-right are examples of a true object, a reconstructed estimate of the object produced by use of a U-Net from tomographic measurements, the total error map, the error in the null component of the reconstructed object, and the error in the measurement component of the reconstructed object. The two rows correspond to different objects. In each case, the true object is outside the respective training data distribution of the U-Net and phase noise was added to the measurements prior to image reconstruction. Image courtesy of IEEE Xplore

January 12, 2022 — Machine-learning methods are being actively developed for computed imaging systems like magnetic resonance imaging (MRI). However, these methods occasionally introduce false, unexplainable structures in images, known as hallucinations, that can lead to incorrect diagnoses.

Researchers at the Beckman Institute for Advanced Science and Technology and the Computational Imaging Science Laboratory have defined a mathematical framework for identifying hallucinations, a first step toward reducing their frequency.

This work, “On hallucinations in tomographic image reconstruction,” is published in IEEE Transactions on Medical Imaging in a special issue on machine learning methods for image reconstruction.

Most modern medical imaging devices — such as MRI, computed tomography (CT) and PET — do not record images directly. Instead, they employ a computational procedure known as image reconstruction. Researchers worldwide are seeking to develop improved image reconstruction methods that use deep learning to circumvent the limitations of traditional methods. Deep learning, like all machine-learning methods, allows algorithms to self-improve by learning from characterized datasets. Advanced DL-based methods can reduce imaging scan times and radiation doses. Although DL-based image reconstruction methods hold great potential for such purposes, they sometimes produce images that appear plausible, but contain false structures that may confound a medical diagnosis.

“This work is an excellent example of how our research group is making foundational contributions to the rapidly evolving field of imaging science,” said Mark Anastasio, the principal investigator of this study and head of the Department of Bioengineering at the University of Illinois Urbana-Champaign.

“Because our framework for analyzing hallucinations in biomedical images will enable researchers to quantitatively assess their image reconstruction methods in new and meaningful ways, we expect the impact of our study to be large."

Lead authors Varun Kelkar and Sayantan Bhadra, both graduate students in Anastasio’s group, anticipate that hallucination maps will help researchers and radiologists identify hallucinations and assess how detrimental they might be. Ultimately, this leads to a deeper understanding of how DL reconstruction methods should be trained to prevent patient misdiagnosis.

“An effective reconstruction method is based on understanding the physics of the imaging system, but also uses realistic assumptions about how an object should appear,” Bhadra said. “DL-based image reconstruction methods are about designing the interplay of these elements by … training deep neural networks on large databases of previously existing clinical images.”

Though DL-based image reconstruction methods are gaining popularity in the field, evidence suggests that they may be unstable, meaning small changes to collected measurements could cause large differences in the resulting image. These challenges are exacerbated when image sampling is incomplete (in other words, when fewer raw imaging measurements are collected to reduce time and costs).

“Although DL has been shown to be useful in many contexts, there’s a lot going on inside these methods that are … still viewed as a black box by many researchers,” Bhadra said. “That makes it hard to tell when a medical abnormality might actually be a false structure created by the machine-learning method. The novelty of our work is in mathematically defining those false structures, which we call hallucinations.”

Kelkar, Bhadra, and colleagues provide a formal definition of hallucinations based on fundamental principles of imaging science. Through numerical studies, they evaluated the presence and impact of hallucinations across both DL-based and non-data-driven reconstruction methods. These methods were used to reconstruct images of adult and pediatric brain tissue — with the DL-based method trained only on adult brain images — and evaluate the effect of bias from the training images.

“Our goal in mathematically defining hallucinations was to separate sources of systemic error, like measurement errors or noise, from errors that arise due to inaccurate assumptions about the to-be-imaged subject that are built into the image reconstruction method,” Kelkar said.

The impact of a generated hallucination is highly dependent on the type of tissue being imaged and the disease being monitored. Sophisticated image processing methods can be applied to identify how a particular diagnostic task may be affected by the presence of hallucinations.

“A hallucination could be something minor, such as a slight imperfection at the boundary of a tissue, or it could be something major, like introducing an entire fold of tissue in the brain,” Kelkar said.

For more information: www.beckman.illinois.edu/

Example of true object and reconstructed images along with error map and hallucination maps (null space) for OOD data with different reconstruction methods – U-Net (top), PLS-TV (middle) and DIP (bottom). Expanded regions are shown to the right of the reconstructed images. The specific error map (blue) and specific null space hallucinations map (red) are overlaid on the reconstructed images for each method. The image estimated by the U-Net method has some distinct false structures (region within red bounding box) that do not exist in the reconstructed images obtained by using PLS-TV and DIP. This region is also highlighted in the specific null space hallucination map for the U-Net method which suggests that the false structure is a hallucination. Image courtesy of IEEE Xplore
Scatter plots for centroids of localized regions in specific error maps and specific null space hallucination maps with different reconstruction methods for IND (top) and OOD (bottom) data. Note that for each type of data distribution and for all the reconstruction methods, the centroids of the regions detected from the error map have a higher variance compared to the hallucination map as well as some degree of non-overlap. Image courtesy of IEEE Xplore
Example of a true object and reconstructed images along with error maps and hallucination maps (null space) for IND data with different reconstruction methods – U-Net (top), PLS-TV (middle) and DIP (bottom). Expanded regions are shown to the right of the reconstructed images. The specific error map (blue) and specific null space hallucinations map (red) are overlaid on the reconstructed images for each method. The image estimated by the U-Net method has visibly lower hallucinations in the null space compared to PLS-TV and DIP. The region within the red bounding box is one of the locations that contains hallucinations for all the reconstruction methods. In this region, the U-Net method shows mild hallucinations compared to PLS-TV and DIP. Fine structures in this region appear to be oversmoothed in the image estimates obtained by use of PLS-TV and DIP. A false structure is also shown (within the blue bounding box region) that appears for all the reconstruction methods due to the phase noise and not due to the imposed prior, and hence cannot be classified as a hallucination. Image courtesy of IEEE Xplore
An error map, a null space hallucination map and a bias map for IND and OOD images estimated by use of the U-Net method. The corresponding true objects are shown in Figs. 2 and 1(b) respectively. The bias map was computed over a dataset of 100 images estimated from a single set of simulated measurements with fixed phase noise and different realizations of the iid Gaussian noise. The bias map contains contributions from both the model error, as well as inaccuracies in the prior. Image courtesy of IEEE Xplore