Explainable artificial intelligence in clinical healthcare: a systematic review, meta-analysis, and proposed clinxai framework (2017–2025)

Abstract

Background: AI models used in the clinic should be both accurate and explainable to the clinician, agency/regulatory officials, and patient. Despite a wide range of approaches developed in the field of Explainable AI (XAI) to explain models after the fact, create inherently interpretable models, and produce concept-based attributions, comprehensive evidence synthesis of the clinical performance and user acceptance of these methods is lacking. Objective: To comprehensively synthesize and meta-analyse studies of XAI methods for clinical healthcare AI from January 2017 to December 2025. Methods: We conducted a literature search in PubMed/MEDLINE, Embase, CINAHL, IEEE Xplore, and Scopus and found 104 eligible studies that were subject to qualitative synthesis (and meta-analysis of 78). Cochrane framework was used to assess the risk of bias. Results: SHAP and Grad-CAM are the most popular XAI methods used (41.3% and 28.8% of studies respectively). The highest scores of clinician agreement (pooled mean: 86.3%, 95% CI: 83.1–89.5) are obtained by prototype-based methods (ProtoPNet-Med). The proposed ClinXAI framework, which integrates concept bottleneck modelling and counterfactual clinical reasoning, has the best agreement scores of 86.5–92.1% in six clinical domains, and outperforms the state-of-the-art systems. Conclusion: XAI can be used to help build clinician trust in and increase diagnostic accuracy in AI-assisted contexts, but there was considerable methodological variation (I² = 68.7%) and no standardised clinician evaluation protocols. There are 7 priority research gaps identified: there is a need for prospective clinical trial evidence of the impact of XAI on patient outcomes.