Calibrated but not safe: Conformal prediction intervals for turbofan remaining useful life near end of life and under operating shift

Abstract

Machine learning models that predict the remaining useful life (RUL) of aircraft engines are increasingly paired with conformal prediction, which converts point predictions into intervals with a guaranteed average coverage. In a safety-critical maintenance setting, however, an average guarantee is not sufficient: what matters is whether the interval is reliable and informative for engines that are close to failure, and whether it remains valid when the fleet differs from the data used for calibration. This study evaluates five interval methods built on ridge regression, random forest, gradient boosting and multilayer-perceptron predictors using the NASA C-MAPSS turbofan benchmark (FD001–FD004, 707 test engines), with repeated engine-level calibration splits. Coverage, width and the rate at which the interval overestimated the true RUL were measured overall and for engines with a true RUL of 30 cycles or less. Global split conformal prediction met the nominal 90% coverage on average (89–92%) and covered almost every engine near failure, but with intervals about 44 cycles wide, which is uninformative when only 30 cycles remain; it also undercovered engines with 61–90 cycles of remaining life (68–76%). Calibrating separately within bands of predicted RUL (Mondrian conformal prediction) halved the width near failure for tree ensembles (from 43.7 to 20.7 cycles for gradient boosting, p = 3 × 10−8) while keeping near-failure coverage at 96.5–97.8%, but failed for ridge regression (77%). Uncalibrated quantile regression overestimated the remaining life of 20% of near-failure engines, and conformalised quantile regression restored average coverage, but its near-failure coverage fell as low as 80% and its near-failure intervals averaged over 90 cycles. When models calibrated on FD001 were applied to engines with an additional fault mode (FD003), near-failure coverage collapsed to 37–50%, and 95% of all miscoverage under shift was overestimation of remaining life, the unsafe direction. Conformal guarantees are therefore marginal and conditional on exchangeability; prognostic intervals should be evaluated near end of life and under realistic shift before they are used for maintenance decisions.