Learning before memorising: How far does early stopping protect multilayer perceptrons from label noise?

Abstract

Supervised neural networks are trained on labels that are frequently wrong, and over-parameterised models are able to fit such errors exactly. This study quantifies, on a controlled benchmark, how much of the resulting damage is prevented by early stopping when the stopping signal itself comes from a validation split carrying the same label noise. Using the UCI handwritten digits dataset, training labels were corrupted with symmetric noise (0–60%) and with structured pair-flip noise (10–40%), and a multilayer perceptron (MLP) trained without regularisation, with strong weight decay (MLP-L2) or with early stopping (MLP-ES) was compared with logistic regression, a support vector machine, a random forest and k-nearest-neighbour classifiers over five random splits. Per-epoch tracking showed a clear learning-before-memorising phase under symmetric noise: with 40% corrupted labels, the MLP fitted 95% of the correctly labelled training examples by epoch 14 while reproducing only 2.2% of the corrupted ones, whereas memorisation of the latter exceeded 50% only after about 89 epochs. Consequently, clean test accuracy fell from a peak of 0.928 to 0.646 when training continued, while early stopping on noisy validation data retained 0.911. Across symmetric noise levels, MLP-ES improved on the unregularised MLP by 0.239 in accuracy on average (30 of 30 paired comparisons, Holm-adjusted p < 0.001) and reached 0.873 at 60% noise. Under pair-flip noise, however, the learning and memorising phases overlapped, the benefit shrank to 0.084, and MLP-ES was no better than weight decay. A random forest reproduced every corrupted label yet remained accurate, showing that memorisation alone does not determine harm. Early stopping is therefore an effective and nearly free defence against random label noise, but not a substitute for noise-aware methods when errors are systematic.