A Chemist's Glossary of Neural Network and GNN Training Terms

The words in a training script are the steps of one loop. An epoch, a batch, a loss, and a scaffold split are not four topics. They are where you are in that loop, and what you are allowed to look at while you are there.

This note is the vocabulary. The DGL beginner’s guide and the protein–ligand DGL note are the hands-on graph code. The history note is how those models arrived in drug discovery. The leakage note and the QSAR validation note are why the split decides whether a good number means anything.

Epoch, batch, iteration

Epoch. One complete pass of the training set through the model. Twenty epochs means every training molecule is seen twenty times. Seeing a molecule again is the point: the first pass rarely fits a nonlinear model.

Batch, or mini-batch. The subset processed together before the weights change. A batch of 32 molecular graphs is one input to the forward pass, not 32 separate trainings.

Batch size. How many samples are in that subset. It is a hyperparameter. It is not learned.

Iteration, or step. One trip around the top row of Figure 1. One batch in, one weight update out. An epoch is a pile of iterations.

Learning rate. How far the optimizer moves each weight. Too large and the loss jumps or diverges. Too small and the same epoch count barely moves the fit.

Weights, or parameters. The numbers inside the model that training changes. In a GNN the matrices that transform atom features at each layer are weights. They start random. A seed makes that start repeatable.

Hyperparameters. Choices fixed before training: learning rate, batch size, depth, hidden size, dropout. They are not updated by loss.backward().

Activation. A nonlinear function after a layer. Without it, stacking layers is still one linear map. ReLU replaces negative values with zero and leaves positive values alone. It is a common choice, not a law.

Gradient. How the loss changes if one weight is nudged. Backpropagation computes one gradient per weight [1]. A large positive gradient means increasing that weight would increase the loss, so the optimizer moves the other way.

Forward, loss, backward, step

Forward pass. The batch goes through the model and comes out as predictions. For a regression GNN that is one predicted pIC50 per molecule. The model does not know that a higher pIC50 is more potent. It is fitting the number you handed it.

Loss. One number that says how wrong those predictions are. Training minimizes it. A lower training loss means a closer fit to the training labels, including whatever noise is in them.

Mean squared error. The average squared gap between prediction and target.

Suppose three pIC50 values are 5, 5, and 5, and the model says 5, 5, and 8. The errors are 0, 0, and 3. MSE is $(0 + 0 + 9) / 3 = 3$. The one miss dominates because the error was squared. That is why MSE is a common default for pIC50 or solubility, and why a few bad labels can steer it.

Mean absolute error. The average of the absolute gaps, $|y - \hat{y}|$. On the same three points it is $(0 + 0 + 3) / 3 = 1$. The outlier still hurts, but it is not squared. MAE is the loss to reach for when a few noisy assays should not run the fit.

Binary cross-entropy. The loss for a yes/no label. The model offers a probability of “active”, and the loss compares it with 0 or 1. Some implementations want that probability. PyTorch’s BCEWithLogitsLoss wants the raw score and applies the sigmoid itself. Mixing those two double-applies the sigmoid.

Categorical cross-entropy. The same idea for more than two exclusive classes. One compound, one class, a predicted distribution over the classes.

Backward pass. The gradient of that one loss number is computed for every weight, by the chain rule run from the loss back to the inputs [1]. In PyTorch, loss.backward() does this. It adds the new gradients onto whatever is already stored.

Optimizer step. The optimizer turns those gradients into a change in the weights. optimizer.step() does that. optimizer.zero_grad() has to come first. Skip it and the second batch’s update still contains the first batch’s gradient.

Loss is not the metric

The loss is what the optimizer sees. A metric is what you use to decide the model is worth keeping. They are often different functions, and the metric has to be computed on molecules the weights did not train on. The QSAR validation note is that rule in full.

Accuracy. The fraction of labels that match. It is a bad compass on a screen. If 95 of 100 compounds are inactive, predicting “inactive” for all of them scores 0.95 and finds no active.

Precision. Of the compounds called active, the fraction that are active. Precision 0.90 means most of the shortlist is real. The cost of a false active is a synthesis or a purchase that fails.

Recall. Of the true actives, the fraction the model names. Recall 0.10 with precision 0.90 is a careful model that misses almost everything.

F1. The harmonic mean of precision and recall:

With $P = 0.90$ and $R = 0.10$, the ordinary average is 0.50 and $F_1 = 0.18$. The harmonic mean refuses to be impressed by one of the two being high.

For regression the usual reported numbers are RMSE, MAE, and $R^2$. RMSE is the square root of MSE, in the same units as pIC50. $R^2$ compares the errors with the errors of predicting the training mean. It is not a percent correct.

Optimizers

Optimizer. The rule that converts gradients into the step.

SGD. Subtract learning rate times the mini-batch gradient from each weight. Often used with momentum, which keeps a running average of recent gradients so one noisy batch does not reverse the step.

RMSProp. Gives each weight its own step size from a running average of squared gradients. Useful when some weights see huge gradients and others see tiny ones.

Adam. Momentum and a per-weight step size together [2]. It is a common default for GNNs because it often trains without a long search over the learning rate. “Often” is not “for every dataset”.

Learning-rate scheduler. A rule that changes the learning rate as training goes. A typical one lowers it after the validation loss has stalled.

When training goes wrong

Overfitting. Training loss keeps falling while validation loss rises. The model is fitting the training set’s noise. Early stopping, dropout, weight decay, or more diverse molecules are the usual responses. A bigger model is not.

Underfitting. Both losses stay high. The model is too small, the training is too short, or the learning rate is too low to move.

Dropout. During training, a random fraction of the activations is set to zero, so the network cannot depend on one path [3]. At evaluation that randomness is turned off. In PyTorch, model.train() uses dropout and model.eval() disables it. Scoring the validation set in train mode makes the number noisy for no reason.

Weight decay. A penalty on large weights, the L2 penalty, set with weight_decay on most PyTorch optimizers. It is not the same knob as dropout. Dropout hits activations. Weight decay hits the parameters.

Normalization. Putting numbers on a stable scale. Standardizing the input descriptors, or the pIC50 values, is the chemist’s version. Batch normalization does a related rescaling inside the network, using the mean and variance of the current batch [4]. Because that mean depends on the batch, batch normalization also cares about train() versus eval().

Early stopping. Stop when validation performance has not improved for a set number of epochs, and keep the weights from the best epoch, not the last one.

Seed. A random seed fixes the split, the initial weights, the shuffle, and the dropout masks. One seed makes a run repeatable. It does not make the number typical. The seed note is that argument in full.

Tuning, on the right set

Grid search. Every combination from a short list. Three learning rates and three batch sizes are nine trainings.

Random search. Draw combinations from ranges. Bergstra and Bengio showed that, for the same number of trainings, random search beats a grid when only a few of the hyperparameters actually matter [5].

Bayesian optimization. Fit a model of how the validation score depends on the hyperparameters, and use it to pick the next trial [6]. It earns its cost when one training run is expensive.

All three read the validation score. Using the test score to pick a learning rate turns the test set into a second validation set. The number you then “report” has already been used to choose the model.

Graphs, in the same loop

Nothing above changes when the input is a graph. The forward pass does.

Graph. Nodes and edges. For a molecule the usual choice is atoms as nodes and bonds as edges. Ethanol drawn without hydrogens is three nodes and two edges, C–C–O. Including hydrogens is a different graph and a different model.

Node features. The vector on each atom: element, formal charge, aromaticity, hybridization, hydrogen count. The network does not see the periodic table except through this vector.

Edge features. The vector on each bond: order, conjugation, ring membership.

Message passing. A node builds a new vector from its own features and its neighbors’ features. Gilmer and colleagues put a large family of molecular GNNs into this one pattern [7]. After one round, a carbon’s vector contains information about the atoms bonded to it, and nothing further.

Aggregation. How the neighbors’ messages become one vector: sum, mean, or max. Sum can tell a carbon with three neighbors from a carbon with one. Mean divides that difference out.

Depth. The number of message-passing layers. Each layer adds one bond of reach. Three layers mean an atom can be influenced by atoms three bonds away, not by the whole molecule.

Oversmoothing. As layers repeat, the node vectors move toward each other and become less useful as a description of one atom [8]. When a deeper GNN gets worse, this is one of the first suspects, ahead of “needs more epochs”.

Readout, or pooling. The step that turns every atom vector into one molecule vector, often by a sum or a mean. A molecular property such as pIC50 is predicted from that vector, not from a single atom. The protein–ligand note is where readout sits in a DGL model.

Splits

Training, validation, and test. Train on the training set. Choose hyperparameters and the stopping epoch on the validation set. Touch the test set once, at the end. An 80/10/10 cut is a common starting point, not a justified one. The QSAR note separates internal fit, cross-validation, and an external check.

Random split. Each molecule is assigned independently. Close analogues of a test compound usually land in training. The model can look strong by recognizing a near neighbor.

Scaffold split. Molecules are grouped by core, and a whole core goes to one side. MoleculeNet’s split uses the Bemis–Murcko scaffold so that the test molecules are new chemotypes relative to training [9]. The Chemprop paper evaluates property models on that kind of split as well as on a random one [10]. It is harder, and it is closer to “would this work on a series we have not made”.

Data leakage. Anything from the evaluation molecules that is allowed to influence training. A near-duplicate compound is the obvious case. The same protein, or a label derived from the test rows, is the less obvious one. The binding-affinity note is a laptop-scale check of how large that effect is for one GNN, and when it is not.

Regression target. A continuous label: pIC50, logS, a computed free energy. State the units.

Classification target. A class, often active or inactive at a potency cutoff. The cutoff is part of the model. A different cutoff is a different dataset, and it changes precision and recall even when the molecules do not change.

The loop in PyTorch

The sketch below is the top row of Figure 1, then the validation step. MyModel and the two loaders are stand-ins. DGL and PyTorch Geometric supply loaders that batch graphs into one disconnected graph. The order of the seven calls does not change.

import torch

model = MyModel()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
loss_fn = torch.nn.MSELoss()

for epoch in range(20):
    model.train()
    for x, y in train_loader:
        optimizer.zero_grad()
        pred = model(x)
        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()

    model.eval()
    with torch.no_grad():
        totals = [loss_fn(model(x), y).item() for x, y in val_loader]
    val_loss = sum(totals) / len(totals)
    print(f"epoch {epoch + 1}: validation loss {val_loss:.3f}")

torch.no_grad() stops PyTorch from storing the graph of operations it would need for a backward pass. Evaluation does not take a step, so that graph is wasted memory. The printed val_loss averages one number per batch, so a final batch of 8 counts as much as a batch of 32. A mean over molecules weights each compound once. Either is fine if you say which one you plotted.

Cheat sheet

References

1. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature 1986;323:533–536. doi:10.1038/323533a0
2. Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv:1412.6980. arxiv.org/abs/1412.6980
3. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014;15:1929–1958. jmlr.org/papers/v15/srivastava14a.html
4. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167. arxiv.org/abs/1502.03167
5. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res 2012;13:281–305. jmlr.org/papers/v13/bergstra12a.html
6. Snoek J, Larochelle H, Adams RP. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25, 2012. arXiv:1206.2944. arxiv.org/abs/1206.2944
7. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 2017. proceedings.mlr.press/v70/gilmer17a.html
8. Li Q, Han Z, Wu XM. Deeper insights into graph convolutional networks for semi-supervised learning. AAAI 2018. arXiv:1801.07606. arxiv.org/abs/1801.07606
9. Wu Z, Ramsundar B, Feinberg EN, Gomes J, Geniesse C, Pappu AS, Leswing K, Pande V. MoleculeNet: a benchmark for molecular machine learning. Chem Sci 2018;9:513–530. doi:10.1039/C7SC02664A
10. Yang K, Swanson K, Jin W, Coley C, Eiden P, Gao H, Guzman-Perez A, Hopper T, Kelley B, Mathea M, Palmer A, Settels V, Jaakkola T, Jensen K, Barzilay R. Analyzing learned molecular representations for property prediction. J Chem Inf Model 2019;59:3370–3388. doi:10.1021/acs.jcim.9b00237