Reproducible ≠ Robust: Why One UMAP Seed Isn't Enough to Trust a Split

Tue, 23 Jun 2026 00:00:00 +0000

Here’s a small disagreement worth unpacking, because both sides of it are right — about different things.

The claim: “Your UMAP split is fine; it’s deterministic. You pinned random_state=42, so the split is perfectly reproducible every time.”

The objection: “That’s the problem. Seed 42 gives you one split. What if that split just happened to flatter the model?”

Both statements are true simultaneously. Resolving the apparent contradiction comes down to separating two ideas that get conflated constantly: reproducibility and robustness. They are not the same property, and a workflow can have the first in full while completely lacking the second.

A pinned seed (left) gives the identical answer on every run — repeatable, but a single point. Robustness (right) asks what the whole distribution over seeds looks like, and whether your one seed is typical or a lucky draw.

Why the “it’s reproducible” argument is technically correct

UMAP is a stochastic algorithm. Its low-dimensional embedding depends on a random initialization and on randomized negative sampling during optimization; run it twice without fixing the seed and you get two different layouts. The same is true of the K-means step that carves that embedding into clusters — its centroid initialization is random too.

So when a script like umap_split_rigorous.py sets

reducer = umap.UMAP(n_components=2, random_state=42)
kmeans  = KMeans(n_clusters=2, random_state=42, n_init=10)

it pins the seed for both the projection and the clustering. The consequence is genuine and valuable: the split is now deterministic. Anyone who runs that script gets byte-for-byte the same train/test partition, the same numbers, the same figures. That is real reproducibility, and it is a precondition for honest science — without it, no result can be checked.

So, narrowly: yes, the split is reproducible, and the person pointing that out is correct.

Why the “one seed isn’t enough” argument is scientifically correct

The trouble is that reproducibility answers the question “will I get the same answer if I run it again?” — not the question we actually care about, which is “is this answer representative of how the method behaves?”

Because UMAP’s embedding depends so heavily on its seed, each seed produces a different geometry of chemical space, and therefore a different boundary between the training and test domains. Seed 42 induces one specific domain shift. Seed 7 induces another. Seed 2024, another still.

The same molecules embedded under three seeds. The clusters land in different places and the train/test boundary cuts through chemical space differently each time — so each seed is effectively a different generalization test.

This is what turns a reproducibility win into an evaluation risk. When you evaluate, say, a GNN on the seed-42 split and report its performance, you are reporting how it does on one particular partition of chemical space — chosen, however innocently, by an arbitrary integer. A skeptical reviewer’s objection writes itself:

“What if seed 42 happened to produce a split where the held-out ’test’ molecules sit close to the training set, making the domain shift mild and the GNN’s job easy? Pick a different seed and the gap could vanish — or reverse.”

This isn’t paranoia; it’s a textbook selection effect. Even with no intent to cherry-pick, reporting a single arbitrary split lets luck masquerade as signal. The split is reproducible, but the conclusion you draw from it may not generalize to the next seed — and generalization across domain shifts is the entire point of using a UMAP split in the first place.

It’s the same trap as the “single fold” caveat I flagged in — one split is an anecdote, not an estimate.

The resolution: treat the seed as a nuisance parameter

The fix is conceptually simple. The seed is not a meaningful scientific variable — it’s a nuisance parameter you happen to have to set. So you should marginalize over it: run the whole split-and-evaluate procedure across many seeds and report the distribution of outcomes, not a single draw.

Formally, instead of reporting a point estimate $r_{42}$, you report the mean and spread over $N$ seeds:

$$ \bar{r} = \frac{1}{N}\sum_{i=1}^{N} r_{s_i}, \qquad \text{SD} = \sqrt{\frac{1}{N-1}\sum_{i=1}^{N}\left(r_{s_i}-\bar{r}\right)^2}. $$

Test performance across 30 seeds. The honest summary is the mean ± spread (≈ 0.60 ± 0.06), not the single value seed 42 happened to give (0.72) — which here sits at the optimistic tail.

The picture above shows exactly why this matters. If seed 42 lands at the top of the distribution, the single-seed report overstates the model. If it lands at the bottom, you’ve understated it. Only the distribution tells you which, and only the distribution lets a reader judge whether a difference between two models is real or within seed-to-seed noise.

What to do in practice

import numpy as np
import umap
from sklearn.cluster import KMeans

def evaluate_split(seed):
    reducer = umap.UMAP(n_components=2, random_state=seed)
    emb = reducer.fit_transform(X_descriptors)

    labels = KMeans(n_clusters=2, random_state=seed,
                    n_init=10).fit_predict(emb)

    train_idx = np.where(labels == 0)[0]
    test_idx  = np.where(labels == 1)[0]

    model.fit(X[train_idx], y[train_idx])
    preds = model.predict(X[test_idx])
    return pearsonr(preds, y[test_idx])[0]

seeds = range(30)                       # marginalize over the nuisance param
scores = np.array([evaluate_split(s) for s in seeds])

print(f"Test r = {scores.mean():.3f} ± {scores.std(ddof=1):.3f} "
      f"(n={len(scores)}, min={scores.min():.3f}, max={scores.max():.3f})")

A few refinements worth adding:

Report the full distribution, not just mean ± SD — a violin or strip plot reveals skew and outliers a summary statistic hides.
Use a bootstrap or percentile confidence interval if you want an honest uncertainty band on the mean.
Keep determinism inside each run. Pinning the seed per iteration (seed=s) is still good practice — it means each individual split remains reproducible. You’re not abandoning reproducibility; you’re sampling reproducible splits.
Compare models on the same set of seeds (paired comparison) so the seed isn’t confounded with the model choice. A paired test across seeds is far more convincing than two independent single-seed numbers.
Watch the cost. Thirty UMAP fits plus thirty model trainings is real compute; budget for it, or use a smaller seed set with a stated caveat.

The takeaway

The two positions reconcile cleanly once you name what each is about:

	Pinned single seed	Many seeds
Reproducible?	✅ Yes — identical every run	✅ Yes — each split is still seeded
Robust?	❌ No — one arbitrary domain shift	✅ Yes — a distribution over shifts
Answers	“same result again?”	“is the result typical?”
Fails when	the seed is lucky/unlucky	(mainly: more compute)