MediCARE: Medical Collaborative Agents REasoning over Interpretable Heterogeneous Graphs

In the era of Artificial Intelligence (AI), Large Language Models (LLMs) are increasingly being used as medical reasoning agents. However, the risk of LLM hallucinations, where the model generates incorrect or nonsensical responses, poses a significant challenge, especially in the medical domain. To address this, we propose the adoption of the Predict Interpret Explain (PIE) paradigm. This paradigm involves building a predictive model using high-quality and well-controlled training data, employing eXplainable AI (XAI) techniques to interpret the model, and using LLMs to process the predictions and their interpretations to provide an accurate, comprehensive report for medical specialists. We focus on the medication recommendation task, utilizing graph data structures to represent patient clinical data and Graph Neural Network (GNN) models to learn from these structures. We interpret the GNN model using GNNExplainer and Integrated Gradients methods. The outcomes are then examined by a pool of LLM-based collaborative agents, who generate, discuss, and revise comprehensive assessments to explain the predictions of the model. We test our framework, MediCARE, on the MIMIC-III database, demonstrating the impact of LLMs in providing accurate reports and improving prediction precision.