Two AUCs from the Same Five Folds
Five folds of the same 2,617 ligands produce two AUCs, and they answer different questions. One is the average of five within-fold AUCs. The other is a single AUC on every out-of-fold score. A DeLong test and a bootstrap interval attach to the second number.
AUC is a rank statistic. It is the probability that a substrate, drawn at random, receives a higher score than a non-substrate drawn at random,
$$ P(\hat{y}_{+} > \hat{y}_{-}). $$Which pairs enter that probability is the difference between the two procedures.
Mean-of-folds computes an AUC inside each fold, on about 523 ligands, then averages the five numbers. A pair counts only when both ligands were held out together. Pooled out-of-fold scoring stacks all 2,617 scores into one vector and computes one AUC. A pair counts even when the two ligands were scored by different fold-models.
Take one substrate and one non-substrate. When every fold holds about the same number of each class, the non-substrate sits in the substrate’s fold one time in five. The other four comparisons cross a fold line. Mean-of-folds keeps the one. Pooled keeps all five, and those four cross-fold pairs are scored by two different models. They land in the right order only if a high score means the same thing in every fold.
A shared scale is part of the test
A model can order the ligands inside every fold and still lose rank once the scores are stacked. A substrate from a low-scoring fold falls below a non-substrate from a high-scoring fold. Mean-of-folds never makes that comparison, so the scale shift is invisible to it.
On a stratified random split, where the folds are meant to be copies of the same mixture, the pooled AUC is the stricter rank test. It asks for order inside each fold and for agreement among the five scales. Mean-of-folds lets each fold keep its own ruler.
Five numbers are a spread, not an interval
An AUC on about 523 ligands moves when the partition moves. The standard deviation of the five fold AUCs is the spread across those partitions. It is not a confidence interval for the pooled AUC. Dividing that standard deviation by $\sqrt{5}$ does not make it one. The folds share most of their training ligands, so the five AUCs are not five independent draws. The pooled AUC is the quieter number because it counts the cross-fold pairs as well as the pairs inside a fold of 523.
The test needs one vector
DeLong’s test compares two ROC curves built on the same ligands. The curves are correlated, and the test uses that covariance [1]. A bootstrap interval resamples one vector of scores and labels. The average of five AUCs has neither the paired rows nor the single vector.
On these 2,617 out-of-fold scores, the pooled vector carries a bootstrap interval of 0.717 to 0.754 for the pooled AUC, and a paired difference of 0.038 between two such vectors, with a DeLong p-value of $6.7 \times 10^{-8}$. A split of the same rows by ligand efficiency is the same kind of calculation. It needs one row per ligand.
The QSAR validation note is where the Q² and R² cuts for a paper live. Those cuts do not choose between these two AUCs.
When the mean is the number you want
Use the pooled AUC when the folds are exchangeable and a later step will treat the scores as one list. A stratified random split is that case. So is any threshold you intend to apply to more than one fold-model, and so is any comparison that needs a p-value or an interval.
Use the mean of the within-fold AUCs when the folds are different populations. A scaffold split, a split by protein target, or a split by assay batch can put few substrates in one fold and many in another. A model that is calibrated inside each group then scores the high-rate group higher. Cross-fold pairs mix the ranking with that shift in base rate. The geometry is Figure 2, and the shift belongs to the groups. The within-fold AUC is the rank inside the group. The glossary note is the scaffold split itself.
Leave-one-out is the small extreme. Each fold has one ligand, so a within-fold AUC is not defined. A fold that contains no substrates has the same hole. Pooling is what puts every substrate back into one test.
| Need | Report |
|---|---|
| A DeLong test or a bootstrap interval | Pooled AUC |
| One threshold applied to every fold-model | Pooled AUC |
| Scaffolds, targets, or batches with different positive rates | Mean of the within-fold AUCs |
| Whether one fold is an odd series | The five AUCs and their spread |
| Leave-one-out, or a fold with no substrates | Pooled AUC |
When the folds are exchangeable, the inferential number is the pooled AUC, with the interval and the test. The mean of the five fold AUCs sits beside it to show whether one partition is carrying the result.
References
- DeLong, E. R.; DeLong, D. M.; Clarke-Pearson, D. L. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 1988, 44, 837–845. doi:10.2307/2531595.
