News | Artificial Intelligence | April 30, 2020

RSNA AI Challenge Breaks New Ground

A complex multicompartmental cerebral hemorrhage on a single axial CT image displayed using the annotation tool in a single portal window. Hemorrhage labels (left column) relevant to the image display on the bottom of the image once selected. ASNR = American Society of Neuroradiology RSNA = Radiological Society of North America.

A complex multicompartmental cerebral hemorrhage on a single axial CT image displayed using the annotation tool in a single portal window. Hemorrhage labels (left column) relevant to the image display on the bottom of the image once selected. ASNR = American Society of Neuroradiology RSNA = Radiological Society of North America. Image courtesy of RSNA

April 30, 2020 — An unprecedented collaboration among two medical societies and over 60 volunteer neuroradiologists has resulted in the generation of the largest public collection of expert-annotated brain hemorrhage computed tomography (CT) images, according to a report published in Radiology: Artificial Intelligence. Leaders of the project expect the dataset to help speed the development of machine learning (ML) algorithms to aid in the detection and characterization of this potentially life-threatening condition.

The creation of the dataset stems from the most recent edition of the Radiology Society of North America (RSNA) Artificial Intelligence (AI) Challenge. For the 2019 edition, participants were asked to create an ML algorithm that could assist in the detection and characterization of intracranial hemorrhage on brain CT. Accuracy in diagnosing the presence and type of intracranial hemorrhage is a vital part of effective treatment, as even a small hemorrhage can lead to death if it is in a critical location.

Rather than using an existing dataset, as had been done for the first two challenges, the competition's organizers set out to create one from scratch. They compiled the brain hemorrhage CT dataset from three institutions: Stanford University in Palo Alto, California, Universidade Federal de São Paulo in São Paulo, Brazil, and Thomas Jefferson University Hospital in Philadelphia, Pennsylvania.

"The value of this challenge is to create a dataset that might lead to a generalizable solution, and the best way to do that is to train a model from data originating from multiple institutions that use a variety of CT scanners from various manufacturers, scanning protocols and a heterogeneous patient population," said the paper's lead author, Adam E. Flanders, M.D., neuroradiologist and professor at Thomas Jefferson University Hospital. "In this case, we had data from three institutions and international participation. The dataset is unique, not only in terms of the volume of abnormal images but also the heterogeneity of where they all came from."

RSNA and the American Society of Neuroradiology (ASNR) collaborated to curate the dataset and organizers issued an open call for volunteers within the ASNR membership to annotate the images. A day-and-a-half later, they had 140 volunteers from which they selected 60 to annotate a vast trove of 874,035 brain hemorrhage CT images in 25,312 unique exams. The volunteers marked each image as normal or abnormal. For the abnormal images, they indicated the hemorrhage subtype.

"It was a nail-biter all the way along," Flanders said of the process. "We were building the airplane while it was in flight. When you consider the number of images that we had to de-identify locally, consume, curate, label, cross-check and then organize into just the right datasets to release to the contestants, there was a lot of work involved by the volunteer workforce, the RSNA Machine Learning Subcommittee, data scientists, contractors and RSNA staff."

The dataset's release attracted interest from far and wide. Organizers received more than 22,200 submissions from 1,787 individual competitors in 1,345 teams from 75 countries. Dr. Flanders was particularly struck by the international reach of the project and the level of enthusiasm even from people outside of the medical realm.

"The 10 top solutions came from all over the world," he said. "Some of the winners had absolutely no background in medical imaging."

The dataset was released under a non-commercial license, meaning it is freely available to the AI research community for non-commercial use and further enhancement.

Flanders said the objective of engaging with a subspecialty society to leverage their unique expertise in developing a high-quality dataset is an effective and useful pathway to follow for future collaborations. The model worked so well that organizers are using it again for this year's competition, a collaboration with the Society of Thoracic Radiology seeking improved detection and characterization of pulmonary embolism on chest CT.

"I was really impressed by the huge volunteer effort and the tremendous worldwide interest in this project," Flanders said. "The dataset we created for this challenge will endure as a valuable ML research resource for years to come."

For more information: www.rsna.org

Related Content

A World Health Organization (WHO) rapid advice guide on the use of chest imaging in the diagnosis and management of COVID-19 was published in the journal Radiology.

Getty Images

News | Coronavirus (COVID-19) | July 30, 2020
July 30, 2020 — A World Health Organization (WHO) rapid advice gui
It covers every major modality, including breast imaging/mammography, fixed and portable C-arms (cath, IR/angio, hybrid, OR), CT, MRI, nuclear medicine, radiographic fluoroscopy, ultrasound and X-ray
News | Radiology Imaging | July 29, 2020
July 29, 2020 — IMV Medical Information, part of Scien...
Left, a 3-D rendering of a heart from a cardiac CT exam. Right, a lung-CT exam showing the heart and ground glass lesions in the lungs of a COVID-19 patient. CT has become a front-line imaging modality in the COVID era because it offers both cardiac and lung information to help determine a patients disposition with chest pain, COVID-19 and COVID-caused myocarditis and pulmonary embolism. #COVID19 #CCTfirst #YesCCT

Left, a 3-D rendering of a heart from a cardiac CT exam. Right, a lung-CT exam showing the heart and ground glass lesions in the lungs of a COVID-19 patient. CT has become a front-line imaging modality in the COVID era because it offers both cardiac and lung information to help determine a patients disposition with chest pain, COVID-19 and COVID-caused myocarditis and pulmonary embolism.

Feature | Computed Tomography (CT) | July 28, 2020
July 28, 2020 — The use of cardiova...
 The Radiological Society of North America (RSNA) has been awarded a $750,000 grant by the Department of Energy’s National Nuclear Security Administration (NNSA) to develop a Global Learning Center (GLC) in Sub-Saharan Africa. The grant will help address the growing need in Sub-Saharan Africa for training in radiology.
News | Radiology Education | July 28, 2020
July 28, 2020 — The Radiological Society of North America (RSNA)
ams Commended by United Imaging for Accelerated Supply of Unprecedented Levels of Essential CT Detectors to Fight Covid-19 (SARS-CoV-2)
News | Coronavirus (COVID-19) | July 28, 2020
July 28, 2020 — ams, a leading worldwide supplier of high performance sensor solutions, has been thanked by long-term
Prostate biopsy with cancer probability (blue is low, red is high). This case was originally diagnosed as benign but changed to cancer upon further review. The AI accurately detected cancer in this tricky case. Image courtesy of Ibex Medical Analytics

Prostate biopsy with cancer probability (blue is low, red is high). This case was originally diagnosed as benign but changed to cancer upon further review. The AI accurately detected cancer in this tricky case. Image courtesy of Ibex Medical Analytics

News | Prostate Cancer | July 28, 2020
July 28, 2020 — A study published in 
A cardiac CT of a patient with pacemaker leads, which can be challenging to get good images due to metal artifact. This image was rendered from using Canon's AiCE AI-assisted interactive reconstruction with Global Illumination 3-D rendering from a scan on an Aquilion One Genesis SP system.

A cardiac CT of a patient with pacemaker leads, which can be challenging to get good images due to metal artifact. This image was rendered from using Canon's AiCE AI-assisted interactive reconstruction with Global Illumination 3-D rendering from a scan on an Aquilion One Genesis SP system.

Feature | Computed Tomography (CT) | July 27, 2020
There has been tremendous growth in the field of cardiovascular...
Zebra Medical Vision announced its sixth FDA 510(k) clearance for its mammography solution, HealthMammo, which has already received a CE mark. Zebra Medical’s algorithm empowers breast radiologists by prioritizing and identifying suspicious mammograms, providing a safety net for radiologists. The suspicious mammograms are identified faster and read earlier than the current “first-in first-out” standard of care. 
News | Breast Imaging | July 27, 2020
July 26, 2020 —  Zebra Medical Vision announced its sixth FDA 510