Does data leakage really inflate binding-affinity GNNs? A laptop-scale reproduction

Motivation

If you train a deep model to predict protein–ligand binding affinity on PDBbind and test on the CASF benchmark, you will probably get an impressive number. A line of recent work argues that a large part of that number is data leakage: the standard train/test splits contain complexes so similar that the model can essentially recognise the test set rather than generalise to it. When the leakage is removed — e.g. with LP-PDBBind or the GEMS CleanSplit — reported performance drops, sometimes markedly.

I wanted to feel this effect myself rather than take it on faith. So I built a small, honest pipeline on a 4 GB laptop GPU and asked a simple question:

If I train the same model under a naive split and under a leakage-controlled split, how big is the gap?

The answer I got was not the one I expected — and chasing down why turned out to be the interesting part.

⚠️ Framing up front. This is a deliberately small-scale reproduction, not a new method. The leakage phenomenon is real and well-documented in the references below. My contribution here is a careful, reproducible negative result and an explanation for it. All code is in the accompanying repository.

The setup

A minimal but faithful design:

• Data: PDBbind v2020 refined set (~5,300 usable complexes), held-out test = CASF-2016 core (266 complexes present in the refined set).
• Splits: the naive split vs. the leakage-controlled CleanSplit (both from the GEMS release).
• Model: a compact E(3)-equivariant GNN (EGNN) over the binding pocket (heavy atoms within 6 Å of the ligand), distance featurised with a radial basis, mean+sum readout. Deliberately modest: ~96 hidden units, 4 layers.
• Ablations: full complex vs. ligand_only vs. pocket_only — to test whether the model uses the interaction or just one partner.
• Uncertainty: 3-model deep ensembles, evaluated with calibration (ENCE) and selective prediction.

The target is the usual

Everything was sized to run on the laptop, which matters for the conclusion.

Result 1 — there is no leakage gap

Training the same EGNN ensemble under both splits, across all three ablations:

The leakage-controlled split was supposed to expose inflated performance. Instead, the gap is within noise in every ablation. My first instinct was that I had broken something — most likely by capping the training set to 1,500 complexes for speed. So I checked.

Digging in: two diagnostics

Diagnostic 1 — the split structure was preserved

CleanSplit works by removing training complexes that are too similar to the test set. So I asked how many of those leakage-prone complexes actually survived my training-set cap:

full train sets:   naive = 3953   clean = 3369   leaky (removed by CleanSplit) = 1270 (32%)
capped to 1500:    naive_capped = 32.5% leaky    clean_capped = 0% leaky

The capped naive set still contained 32.5% leakage-prone complexes and the clean set zero — the exact contrast the experiment needed. The cap was not the culprit; the null result is real.

Diagnostic 2 — similar ligands are no easier

A more direct test of the mechanism. Leakage inflates scores because the model can memorise a near-duplicate of a test compound it saw in training. So: do test complexes whose ligand is nearly identical to a training ligand get predicted better?

For each of the 266 test complexes I computed the maximum Tanimoto similarity (Morgan fingerprints) to any training ligand, and plotted it against the trained model’s absolute error.

There are 24 test ligands with a near-duplicate (Tanimoto ≥ 0.9) in training, and the model is no better on them than on novel chemistry. Splitting the test set by similarity:

The “leaky” complexes are essentially as hard as the novel ones (and actually less correlated). There is no memorisation to exploit.

Why? Leakage needs a model strong enough to memorise

Two independent analyses agree, so the explanation has to account for both. The simplest one that does:

Leakage only inflates performance for models with enough capacity to memorise the leaked examples.

My EGNN is small, regularised, and trained on ~1,500–4,000 complexes. It learns broad structure–activity trends (Pearson ~0.66) but it cannot store and replay individual complexes. With nothing memorised, a near-duplicate in the test set is worth nothing — so removing those near-duplicates (CleanSplit) costs nothing either.

The published leakage effects, by contrast, come from higher-capacity models (e.g. GNNs augmented with protein-language-model embeddings) trained on the full general set (~17–19k complexes). Those models can memorise, so they do benefit from leakage — and lose that benefit under a clean split.

The takeaway isn’t “leakage isn’t real.” It’s that leakage-sensitivity scales with model capacity: the inflation you suffer is a function of how much your model can memorise. A weak model is, ironically, robust to leakage by being too weak to cheat.

Two things that did reproduce

The ablation behaves exactly as the literature predicts. Averaging over splits:

The model does use the interaction (full beats either partner alone), but pocket-alone is much closer to full than ligand-alone — consistent with the well-known result that affinity models lean heavily on protein/pocket identity.

Calibration tracks input richness. A small bonus from the uncertainty analysis:

Honest limitations

This is a small study and the conclusion deserves its caveats:

• Ligand-only similarity. I measured ligand Tanimoto, not protein/pocket structural similarity (the other leakage vector). A TM-score analysis would make the null airtight. (Planned follow-up.)
• Refined set, capped training, small model, single fold. All push toward the low-capacity regime where I claim leakage is invisible — which is exactly the point, but it means I have not shown leakage is absent at scale, only that it disappears for a weak model.
• This is a reproduction, not a refutation. The cited works demonstrate leakage convincingly at their scale; my result is a boundary condition, not a contradiction.

What I took away

1. Reproduce before you believe — including your own expectations. I expected a clean leakage gap and got a flat one; the value was in finding out why.
2. A “failed” reproduction can be the better result. “Leakage-sensitivity scales with model capacity” is a more useful thing to internalise than another confirmation.
3. Evaluation rigor is cheap and high-leverage. The two diagnostics took far less compute than the training, and they’re what turned a confusing null into an explanation.

The full pipeline — data loading, pocket-graph construction, the EGNN, the three UQ methods, the splits, and both diagnostics — is in the repository, and the whole thing runs on a 4 GB laptop GPU.

References

• Graber, D., Stockinger, P., Meyer, F., Mishra, S., Horn, C., & Buller, R. (2025). Resolving data bias improves generalization in binding affinity prediction (GEMS / PDBbind CleanSplit). Nature Machine Intelligence. https://www.nature.com/articles/s42256-025-01124-5
• Li, J., Guan, X., Zhang, O., Sun, K., Wang, Y., Bagni, D., & Head-Gordon, T. (2024). Leak Proof PDBBind: A Reorganized Dataset of Protein–Ligand Complexes for More Generalizable Binding Affinity Prediction. J. Phys. Chem. B. https://pubs.acs.org/doi/10.1021/acs.jpcb.5c08598 (arXiv:2308.09639)
• Satorras, V. G., Hoogeboom, E., & Welling, M. (2021). E(n) Equivariant Graph Neural Networks. ICML.
• Levi, D., Gispan, L., Giladi, N., & Fetaya, E. (2022). Evaluating and Calibrating Uncertainty Prediction in Regression Tasks (ENCE). Sensors, 22(15), 5540.