An interpretable integer score for angiographic coronary artery disease in the Cleveland cohort: derivation and cross-validated comparison with machine-learning models
Keywords:
Coronary Artery Disease, Diagnostic Prediction, Clinical Risk Score, Interpretable Machine Learning, Random Forest, Calibration, Decision Curve Analysis, Cleveland Heart Disease Data.Abstract
Background: Machine-learning classifiers are often proposed for the non-invasive diagnosis of coronary artery disease (CAD), but they are rarely compared with simple scores that clinicians can compute by hand.
Objective: To compare a transparent integer points score with six alternative models for the diagnosis of angiographic CAD.
Methods: The public Cleveland Clinic angiography cohort was used (303 patients, 138 with at least 50% stenosis in a major coronary artery). Nine clinically dichotomised items were defined (male sex, age ≥55 years, asymptomatic chest pain, exerciseinduced angina, ST depression ≥1 mm, flat or downsloping ST segment, maximum heart rate <140 bpm, at least one vessel visible on fluoroscopy, and a thallium perfusion defect), weighted by rounding twice their logistic coefficients, and recalibrated to absolute risk. The score was compared with a unit-weight item count, logistic regression with all 13 predictors, L1-penalised logistic regression, random forest, gradient boosting and a support vector machine. Evaluation used 20 repeats of stratified five-fold cross-validation, with all derivation steps confined to training folds.
Results: The points score (range 0–19) achieved a mean area under the receiver operating characteristic curve (AUC) of 0.915, a Brier score of 0.113 and a calibration slope of 0.88. Full logistic regression performed similarly (AUC 0.912), whereas random forest (0.902), support vector machine (0.894) and gradient boosting (0.887) had lower AUC than the score in every one of the 20 repeats. Observed CAD prevalence rose from 0% (0/30) at 02 points to 96.8% (60/62) at ≥15 points. Decision curve analysis showed net benefit for the score comparable to or greater than that of the ensemble models across threshold probabilities of 0.10–0.70. The score was well calibrated in both women (observed-to-expected ratio 1.00) and men (1.00), whereas the random forest over-predicted risk in women.
Conclusions: In this dataset, a transparent 19-point score performed at least as well as machine-learning ensembles while offering advantages in interpretability and bedside use. External validation in contemporary cohorts is required before clinical application.
References
Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM, et al. Global burden of cardiovascular diseases and risk factors, 1990–2019: update from the GBD 2019 study. J Am Coll Cardiol. 2020;76(25):2982–3021. https://doi.org/10.1016/j.jacc.2020.11.010
Knuuti J, Wijns W, Saraste A, Capodanno D, Barbato E, Funck-Brentano C, et al. 2019 ESC Guidelines for the diagnosis and management of chronic coronary syndromes. Eur Heart J. 2020;41(3):407–477. https://doi.org/10.1093/eurheartj/ehz425
Gulati M, Levy PD, Mukherjee D, Amsterdam E, Bhatt DL, Birtcher KK, et al. 2021 AHA/ACC/ASE/CHEST/SAEM/SCCT/SCMR guideline for the evaluation and diagnosis of chest pain. Circulation. 2021;144(22):e368–e454. https://doi.org/10.1161/CIR.0000000000001029
Diamond GA, Forrester JS. Analysis of probability as an aid in the clinical diagnosis of coronary-artery disease. N Engl J Med. 1979;300(24):1350–1358. https://doi.org/10.1056/NEJM197906143002402
Detrano R, Janosi A, Steinbrunn W, Pfisterer M, Schmid JJ, Sandhu S, et al. International application of a new probability algorithm for the diagnosis of coronary artery disease. Am J Cardiol. 1989;64(5):304–310. https://doi.org/10.1016/0002-9149(89)90524-9
Genders TSS, Steyerberg EW, Alkadhi H, Leschka S, Desbiolles L, Nieman K, et al. A clinical prediction rule for the diagnosis of coronary artery disease: validation, updating, and extension. Eur Heart J. 2011;32(11):1316–1330. https://doi.org/10.1093/eurheartj/ehr014
Wilson PWF, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18):1837–1847. https://doi.org/10.1161/01.CIR.97.18.1837
Sullivan LM, Massaro JM, D'Agostino RB Sr. Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med. 2004;23(10):1631–1660. https://doi.org/10.1002/sim.1742
Six AJ, Backus BE, Kelder JC. Chest pain in the emergency room: value of the HEART score. Neth Heart J. 2008;16(6):191–196. https://doi.org/10.1007/BF03086144
Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2nd ed. Cham: Springer; 2019. https://doi.org/10.1007/978-3-030-16399-0
Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://doi.org/10.1136/bmj-2023-078378
Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004
Dawes RM. The robust beauty of improper linear models in decision making. Am Psychol. 1979;34(7):571–582. https://doi.org/10.1037/0003-066X.34.7.571
Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–215. https://doi.org/10.1038/s42256-019-0048-x
Ustun B, Rudin C. Supersparse linear integer models for optimized medical scoring systems. Mach Learn. 2016;102(3):349–391. https://doi.org/10.1007/s10994-015-5528-6
Shaw LJ, Bugiardini R, Merz CNB. Women and ischemic heart disease: evolving knowledge. J Am Coll Cardiol. 2009;54(17):1561–1575. https://doi.org/10.1016/j.jacc.2009.04.098
Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ. 2006;332(7549):1080. https://doi.org/10.1136/bmj.332.7549.1080
Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Series B Stat Methodol. 1996;58(1):267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
Breiman L. Random forests. Mach Learn. 2001;45(1):5–32. https://doi.org/10.1023/A:1010933404324
Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001;29(5):1189–1232. https://doi.org/10.1214/aos/1013203451
Cortes C, Vapnik V. Support-vector networks. Mach Learn. 1995;20(3):273–297. https://doi.org/10.1007/BF00994018
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830. https://doi.org/10.48550/arXiv.1201.0490
Riley RD, Ensor J, Snell KIE, Harrell FE, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441. https://doi.org/10.1136/bmj.m441
Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78(1):1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7
Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–574. https://doi.org/10.1177/0272989X06295361
Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209–212. https://doi.org/10.1080/01621459.1927.10502953
Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594. https://doi.org/10.1136/bmj.g7594
Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98(9):691–698. https://doi.org/10.1136/heartjnl-2011-301247
Published
How to Cite
Issue
Section
Copyright (c) 2026 The Author(s)

This work is licensed under a Creative Commons Attribution 4.0 International License.