Reliability
Among predictions assigned 80% confidence, roughly 80% should be correct. Reliability diagrams compare confidence and empirical accuracy by bin.
gap(B) = |acc(B) − conf(B)|
0.082ECE
0.161Brier score
78%accuracy
64%coverage
Accuracy asks who was right. Calibration asks who knew.
ECE
Expected calibration error weights each bin’s confidence-accuracy gap by its sample share. It is intuitive but depends on binning.
ECE = Σ |B|/n · gap(B)
Brier score
The Brier score is a proper scoring rule: squared error between predicted probabilities and outcomes. Lower is better.
BS = 1/n Σ(pᵢ − yᵢ)²
Temperature scaling changes certainty without changing class order.
A positive scalar divides logits before softmax. On a held-out calibration set it can reduce overconfidence while preserving argmax accuracy, but it cannot repair missing features or systematic class confusion.
Selective prediction trades coverage for reliability.
An abstaining system acts only above a confidence threshold. Higher thresholds usually improve accepted-case accuracy while reducing coverage. Evaluate risk-coverage curves, not one convenient threshold.
Calibration drifts with the deployment population.
A model calibrated on one domain can become miscalibrated after class prevalence, input difficulty, sensors, or user behavior changes. Monitor by cohort and over time.
Evaluation questions
Can a perfectly calibrated model be inaccurate?
Yes. A model predicting 50% for every balanced example can be calibrated but unhelpful. Calibration and discrimination measure different properties.
Does low ECE guarantee safe confidence?
No. Aggregate bins can hide class or subgroup failures, and ECE estimates vary with sample size and binning. Inspect classwise and adaptive-bin measures.
Should temperature be fitted on the test set?
No. Fit calibration parameters on a held-out validation set and reserve the test set for unbiased evaluation.
Primary sources
Guo et al. (2017) studies calibration of modern neural networks and temperature scaling. Brier (1950) introduced the probability score. Naeini et al. and later calibration work motivates better calibration estimation. Ovadia-style distribution-shift evaluation emphasizes uncertainty under changing data.