Tuned on paper, not in practice: How much does hyperparameter tuning improve classifiers on small tabular datasets?

Abstract

Hyperparameter tuning is routinely reported as a source of improvement in applied machine learning, but the gain is usually judged by the same cross-validation scores that were used to select the hyperparameters. This study measures how much of that gain survives on data untouched by the search. Logistic regression, a radial-basis support vector machine, a random forest, histogram gradient boosting and a multilayer perceptron were evaluated on seven public tabular datasets with 208–1797 samples, comparing library default hyperparameters with configurations found by random search over 20 candidates. A nested design was used: tuning took place inside 3-fold cross-validation on the development data of each of five outer folds, and both the defaults and the selected configurations were then scored once on the held-out outer fold, giving 175 paired comparisons. Measured by the inner cross-validation, tuning improved balanced accuracy by 1.34 percentage points on average and appeared to beat the defaults in 66–100% of runs, depending on the model. On the held-out folds, the realised gain was 0.34 percentage points, tuning improved accuracy in only 76 of 175 comparisons (30 were unchanged), and no model showed a significant gain (Wilcoxon p ≥ 0.455). The apparent gain was therefore about four times the realised gain (p = 2.2 × 10−5). The discrepancy arose because the inner score of the selected configuration was optimistically biased (+0.26 points) while that of the defaults was pessimistically biased (−0.74 points). Increasing the search budget from 5 to 20 trials did not increase the realised gain consistently, and the best tuned model outperformed the best default model on only two of seven datasets. On small tabular datasets, defaults are strong baselines, and reported tuning gains should be measured on held-out data rather than on the scores used for selection.