News | Artificial Intelligence | April 30, 2020

RSNA AI Challenge Breaks New Ground

A complex multicompartmental cerebral hemorrhage on a single axial CT image displayed using the annotation tool in a single portal window. Hemorrhage labels (left column) relevant to the image display on the bottom of the image once selected. ASNR = American Society of Neuroradiology RSNA = Radiological Society of North America.

A complex multicompartmental cerebral hemorrhage on a single axial CT image displayed using the annotation tool in a single portal window. Hemorrhage labels (left column) relevant to the image display on the bottom of the image once selected. ASNR = American Society of Neuroradiology RSNA = Radiological Society of North America. Image courtesy of RSNA

April 30, 2020 — An unprecedented collaboration among two medical societies and over 60 volunteer neuroradiologists has resulted in the generation of the largest public collection of expert-annotated brain hemorrhage computed tomography (CT) images, according to a report published in Radiology: Artificial Intelligence. Leaders of the project expect the dataset to help speed the development of machine learning (ML) algorithms to aid in the detection and characterization of this potentially life-threatening condition.

The creation of the dataset stems from the most recent edition of the Radiology Society of North America (RSNA) Artificial Intelligence (AI) Challenge. For the 2019 edition, participants were asked to create an ML algorithm that could assist in the detection and characterization of intracranial hemorrhage on brain CT. Accuracy in diagnosing the presence and type of intracranial hemorrhage is a vital part of effective treatment, as even a small hemorrhage can lead to death if it is in a critical location.

Rather than using an existing dataset, as had been done for the first two challenges, the competition's organizers set out to create one from scratch. They compiled the brain hemorrhage CT dataset from three institutions: Stanford University in Palo Alto, California, Universidade Federal de São Paulo in São Paulo, Brazil, and Thomas Jefferson University Hospital in Philadelphia, Pennsylvania.

"The value of this challenge is to create a dataset that might lead to a generalizable solution, and the best way to do that is to train a model from data originating from multiple institutions that use a variety of CT scanners from various manufacturers, scanning protocols and a heterogeneous patient population," said the paper's lead author, Adam E. Flanders, M.D., neuroradiologist and professor at Thomas Jefferson University Hospital. "In this case, we had data from three institutions and international participation. The dataset is unique, not only in terms of the volume of abnormal images but also the heterogeneity of where they all came from."

RSNA and the American Society of Neuroradiology (ASNR) collaborated to curate the dataset and organizers issued an open call for volunteers within the ASNR membership to annotate the images. A day-and-a-half later, they had 140 volunteers from which they selected 60 to annotate a vast trove of 874,035 brain hemorrhage CT images in 25,312 unique exams. The volunteers marked each image as normal or abnormal. For the abnormal images, they indicated the hemorrhage subtype.

"It was a nail-biter all the way along," Flanders said of the process. "We were building the airplane while it was in flight. When you consider the number of images that we had to de-identify locally, consume, curate, label, cross-check and then organize into just the right datasets to release to the contestants, there was a lot of work involved by the volunteer workforce, the RSNA Machine Learning Subcommittee, data scientists, contractors and RSNA staff."

The dataset's release attracted interest from far and wide. Organizers received more than 22,200 submissions from 1,787 individual competitors in 1,345 teams from 75 countries. Dr. Flanders was particularly struck by the international reach of the project and the level of enthusiasm even from people outside of the medical realm.

"The 10 top solutions came from all over the world," he said. "Some of the winners had absolutely no background in medical imaging."

The dataset was released under a non-commercial license, meaning it is freely available to the AI research community for non-commercial use and further enhancement.

Flanders said the objective of engaging with a subspecialty society to leverage their unique expertise in developing a high-quality dataset is an effective and useful pathway to follow for future collaborations. The model worked so well that organizers are using it again for this year's competition, a collaboration with the Society of Thoracic Radiology seeking improved detection and characterization of pulmonary embolism on chest CT.

"I was really impressed by the huge volunteer effort and the tremendous worldwide interest in this project," Flanders said. "The dataset we created for this challenge will endure as a valuable ML research resource for years to come."

For more information: www.rsna.org

Related Content

The FDA clearance, Quantib’s 6th to date, marks the first time a comprehensive AI prostate solution will be available to radiologists in the United States
News | Prostate Cancer | October 21, 2020
October 21, 2020 — Quantib, a market leader in...
According to an inquest, a man with a heart disorder and chest pain died two days after a doctor viewed the wrong scan and sent him home
News | Computed Tomography (CT) | October 21, 2020
October 21, 2020 — The BBC News
Flowchart of patient inclusion and exclusion.

Figure 1. Flowchart of patient inclusion and exclusion.

News | Coronavirus (COVID-19) | October 20, 2020
October 20, 2020 — A new multi-institutional study published in the journal ...
Rensselaer, First-Imaging, and GE Global researchers develop a deep neural network to perform nearly as well as more complex dual-energy CT imaging technology
News | Computed Tomography (CT) | October 20, 2020
October 20, 2020 — Bioimaging technologies are the eyes that allow doctors to see inside the body in order to diagnos
Ezra, a NY-based startup transforming early cancer screening using magnetic resonance imaging (MRI), announced that it has received FDA 510(k) premarket authorization for its Artificial Intelligence, designed to decrease the cost of MRI-based cancer screening, assisting radiologists in their analysis of prostate MRI scans. It is the first prostate AI to be cleared by the FDA.
News | Artificial Intelligence | October 20, 2020
October 20, 2020 — Ezra, a NY-based startup transforming early cancer screening using...
Carestream Health will offer insights on technology, innovative new products and research in medical imaging through a new webinar series titled Carestream Talks. #RSNA20
News | Radiology Education | October 14, 2020
October 14, 2020 — Carestrea