When uncertainty sampling backfires: A benchmark of active learning query strategies on real image datasets

Abstract

Active learning promises to reduce annotation cost by letting a model choose which unlabelled examples should be labelled next, but published gains vary widely and some strategies have been reported to perform worse than random selection. This study compares six query strategies under identical conditions on three real image datasets: handwritten digits from MNIST, product photographs from Fashion-MNIST and the smaller UCI Digits collection. Starting from 20 randomly chosen labelled images, a logistic-regression classifier was retrained after each batch of queries until 1,000 images (400 for Digits) had been labelled, with five random seeds per condition. The strategies were random sampling, least-confidence, margin and entropy uncertainty sampling, k-center greedy diversity sampling, and margin sampling combined with k-means diversification of each batch. Margin sampling with batch diversification was the most label-efficient strategy on all three datasets: it reached 95% of the accuracy of a model trained on the full pool with 372, 452 and 82 labels on average, whereas random sampling needed 750 and 216 labels on MNIST and Digits and reached the target in only one of five runs on Fashion-MNIST (at 820 labels), and its area under the learning curve exceeded that of random sampling in all 15 runs. However, least-confidence and entropy sampling were worse than random sampling throughout the budget on MNIST and Fashion-MNIST, and k-center greedy sampling ended 16.3 and 9.9 percentage points below random sampling on these datasets. Inspection of the images queried by each strategy explained these failures: entropy sampling repeatedly selected visually similar ambiguous digits from a few classes, whereas k-center greedy selected atypical images, whose distance to their tenth-nearest neighbour was 1.38 and 1.82 times the pool median, and over-sampled particular classes. On the small Digits dataset all strategies outperformed random sampling. The value of active learning therefore depends strongly on the strategy and the data, and margin-based sampling with batch diversity is a robust default, whereas entropy-based and purely geometric diversity sampling should be validated before use.