Same accuracy, different failures: Robustness of pixel-based, gradient-feature and augmented classifiers to common image corruptions

Abstract

Image classifiers are usually compared by their accuracy on clean test images, although deployed systems receive images degraded by noise, blur, poor contrast, compression and misalignment. This study examines whether clean accuracy predicts robustness to such corruptions and how two common remedies, hand-crafted gradient features and data augmentation, change the picture. Seven classifiers were trained on 20,000 real product photographs from the Fashion-MNIST dataset: logistic regression, random forest and multilayer perceptron (MLP) models on raw pixels; the same three models on histogram-of-oriented-gradients (HOG) features; and an MLP trained with noise and blur augmentation. Each was tested on 5,000 held-out images, both clean and under nine corruptions at three severities, with three training seeds. Clean accuracy varied only between 82.8% and 87.0%, whereas mean corrupted accuracy ranged from 48.6% to 71.0%. The pixel MLP and the random forest differed by 0.7 points on clean images but by 5.0 points under corruption, and the corruption to which each model was most vulnerable differed between models. Contrast reduction reduced the accuracy of pixel models to 12.6–34.3%, whereas HOG models retained 83.4–85.2%, because block normalisation makes HOG descriptors invariant to contrast scaling; the same features, however, were highly sensitive to noise. Augmentation with noise and blur raised the MLP’s mean corrupted accuracy from 64.2% to 71.0% without reducing clean accuracy, and it also improved robustness to related unseen corruptions (salt-and-pepper noise and motion blur) and, most strikingly, to contrast reduction (from 12.4% to 64.5% at the highest severity). No model was robust to rotation or translation, and augmentation slightly reduced accuracy under both. Clean accuracy is therefore an unreliable guide to robustness, and robustness should be evaluated separately for each type of corruption expected in deployment.