January 13, 2020 — Researchers at Radboud university medical center have developed a "deep learning" system that is better than most pathologists at determining the aggressiveness of prostate cancer. The AI system, which uses tissue samples to arrive at its diagnosis, taught itself to identify prostate cancer based on data from over 1,200 patients. The Radboud team is now working with researchers from the Karolinska Institute in Sweden and Kaggle, a Google subsidiary, with the intention to continue developing these methods as part of a major international competition.
Prostate cancer is a frequently occurring type of cancer, but not always aggressive: more men die with prostate cancer than from prostate cancer. However, its treatment has many consequences for the quality of life of patients, so determining aggressiveness is an important step in choosing a treatment. To determine the aggressiveness of the cancer, pieces of tissue (biopsies) are taken from the prostate, which are scored by a pathologist. This Gleason score is then used to classify biopsies into five groups — the Gleason Grade Groups — which indicate the risk of dying from prostate cancer. However, this is a subjective process; whether and how a patient is treated may depend on the pathologist who assesses the tissue.
Better than a Pathologist
The researchers at Radboudumc developed an AI system that examines those biopsies the same way a pathologist does. The AI system also determines the Gleason score, and then the system can classify a biopt according to the Gleason Grade Groups. By means of deep learning, the system examined thousands of images of biopsies to learn what a healthy prostate is, and what more or less aggressive prostate cancer tissue looks like. Researcher Wouter Bulten described this process: "The AI system has now been trained with 5,759 biopsies from more than 1,200 patients. When we compared the performance of the algorithm with that of 15 pathologists from various countries and with differing levels of experience, our system performed better than ten of them and was comparable to highly experienced pathologists." An additional advantage of such a computer system is that it is consistent and can be used anywhere; the treatment of a patient no longer depends on the pathologist looking at the tissue.
An International Competition
As 1.2 million men globally are diagnosed with prostate cancer every year, the development of an AI diagnostic system is interesting for many research groups and companies. "It is advantageous that we are an academic hospital," said Bulten. "We are close to the patient and the practitioner, and have our own database of biopsies." As a next step, the Radboud university medical center team — together with researchers from the Karolinska Institute in Sweden and Kaggle, a subsidiary of Google specialized in data science competitions — wants to hold an international competition in which participants try to beat the Radboudumc algorithm. The insights resulting from this competition will then be used to improve the algorithms.
Background: What is Deep Learning?
Deep learning is a term used for systems that learn in a way that is similar to how our brain works. It consists of networks of electronic 'neurons', each of which learns to recognize one aspect of the desired image. Then it follows the principles of learning by doing, and practice makes perfect. The system is fed more and more images that include relevant information saying — in this case —whether this is cancer or not, and if so, what the Gleason score is. The system then learns to recognize which characteristics belong to cancer, and the more pictures it sees, the better it can recognize those characteristics in undiagnosed images. A major advantage of these systems is also that they learn much faster than humans and can work 24 hours a day.
The Nijmegen study, authored by Wouter Bulten, Geert Litjens and others, has been published in The Lancet Oncology.
October 13, 2026 
