Proper scoring rules

A scoring rule S(p, y) rewards a probabilistic forecast once the outcome y is seen. It is strictly proper if the forecaster's expected reward is uniquely optimised by reporting their true belief — the property that makes honest reporting a dominant strategy. The three classics, in loss form:

$S_{\log}=-\log p_y, \qquad S_{\text{Brier}}=\textstyle\sum_i (p_i-\mathbf 1\{i=y\})^2, \qquad S_{\text{sph}}=1-\dfrac{p_y}{\lVert p\rVert_2}$

For point forecasts, the pinball loss $L_\tau(z,y)=(y-z)(\tau-\mathbf 1\{y<z\})$ elicits the $\tau$-quantile, and the interval score rewards narrow, well-calibrated prediction intervals. For full distributional forecasts the CRPS and its multivariate generalisation the energy score $\mathrm{ES}=\mathbb E\lVert X-y\rVert-\tfrac12\mathbb E\lVert X-X'\rVert$ are the gold standard — the energy score is what scores sample-based forecasts in contests like monteprediction.com.

Code: mechanisms/scoring_rules.py · Demo: examples/sim_scoring_rules.py · Research: parimutuel-and-scoring-rules.md