AI agents for oncology

From multimodal data to medical reasoning: Building autonomous AI agents for oncology

Clinical decision-making in oncology is challenging and requires the analysis of various data types – from medical imaging and genetic information to patient records and treatment guidelines. To effectively support medical practice, AI models must be capable of processing multimodal data and have reasoning and problem-solving capabilities that resemble those of humans.

To build an autonomous AI agent for precision medicine, the researchers enhanced the large language model GPT-4 with several digital tools – including radiology report generation from MRI and CT scans, medical image analysis, prediction of genetic alterations directly from histopathology slides, and search functions across platforms such as PubMed, Google and OncoKB. To ensure that decisions were grounded in current medical knowledge, the model was given access to around 6,800 documents compiled from official oncology guidelines and clinical resources.

Autonomous AI agents successfully tested on realistic, simulated patient cases

The AI agent was evaluated on 20 real-world, simulated patient cases using a two-step process: First, the system selected appropriate tools, then it retrieved relevant medical information to guide its reasoning. The outputs were reviewed by human medical experts for accuracy, completeness, and correct citation of sources. The AI agent reached correct clinical conclusions in 91% of cases and accurately cited relevant oncology guidelines in over 75% of its responses. Importantly, the use of specialized tools and medical information retrieval significantly improved the model’s performance. As a result, so-called “hallucinations” – seemingly plausible but incorrect statements – were significantly reduced. This improvement is particularly important in the sensitive area of healthcare.

Potential future support in everyday clinical practice

The study serves as proof of principle that AI agents can be designed to support oncologists in everyday clinical practice.  Despite the promising results, the researchers acknowledge current limitations of their study. The system was only tested on a small number of simulated cases and requires further validation. Future work will focus on integrating conversational capabilities with human feedback – so-called “human-in-the-loop” interactions – and ensuring data privacy through deployment on local servers.

To fully leverage the potential of AI agents in the future, it will be crucial to integrate them smoothly into routine clinical practice with minimal disruption,” says Prof. Jakob N. Kather, Professor of Clinical Artificial Intelligence at EKFZ for Digital Health at TU Dresden, and oncologist at Dresden University Hospital Carl Gustav Carus. “Challenges such as interoperability with existing systems, compliance with data privacy laws, the need for regulatory approval processes as medical devices, and ensuring accountability still need to be addressed.”

 

Long-term, the research team envisions that similar AI agents could be adapted for use in other medical fields – provided they are equipped with appropriate tools and data. Overall, the study highlights the considerable potential of large language models when combined with precision oncology and search tools, establishing a solid foundation for the future use of AI-driven, personalized support systems in clinical practice.

This publication in Nature Cancer was created in collaboration between researchers from the EKFZ for Digital Health at TU Dresden, University Hospital Dresden, NCT – National Center for Tumor Diseases Heidelberg, School of Computer Science at University of St Andrews (UK), University Hospital RWTH Aachen, University Medical Center Mainz, University Hospital rechts der Isar / Technical University of Munich (TUM), Google Cloud, Institute for Cancer Research and Royal Marsden Hospital (UK), Department of Epidemiology and Biostatistics at Memorial Sloan Kettering Cancer Center (US).