← Catalog

Scoring point-cloud submissions

Many contests elicit a forecast as a cloud of Monte-Carlo samples rather than a formula: monteprediction's eleven-dimensional clouds, the nearest-the-pin parimutuel, generative-model evaluation. The cloud must be turned into a score at the realised outcome $z$, and how that is done decides whether the contest rewards your beliefs or a distortion of them.

Two ways to score a cloud

Density route. Smooth the cloud with a Gaussian KDE of bandwidth $h$ and log-score the density at $z$ (the pot-split of the nearest-the-pin pool pays the exponentiated form). Projection route. Project the cloud and $z$ onto random directions and average the one-dimensional CRPS; by the projection identity this recovers the multivariate energy score, a strictly proper score built from cheap 1-D pieces — the robust choice in high dimension.

The deconvolution incentive

The density route has a flaw when the outcome is used raw: the mechanism scores the smoothed submission $\rho * \varphi_h$, so the optimal cloud is drawn from the belief deconvolved by the kernel, not from the belief. For a Gaussian belief $N(\mu,\tau^2)$:

The truthfulness gap is $\mathrm{KL}(p^*\Vert p^* * \varphi_h) \approx h^4/(4\tau^4)$: small, but a slope any optimising submitter walks down. When $\tau^2 \le h^2$ the deconvolution degenerates toward point masses — the Theis, van den Oord & Bethge (2016) exploit, where a cloud of k-means centroids out-scores the true distribution.

The repair: jitter the pin

If you smooth the forecasts, smooth the outcome too. Settling at $z' = z + j\varepsilon$ with jitter matched to the bandwidth ($j = h$, same kernel) makes the score strictly proper for the pre-smoothing cloud, because Gaussian convolution is injective — its characteristic function never vanishes. The calibration is a single formula: the score peaks at

so $j=0$ pays shaving, $j>h$ pays padding, and $j=h$ — alone — pays the truth. One requirement: the bandwidth must be fixed in advance — a bandwidth computed from the participant's own cloud puts the smoothing channel under the participant's control and voids the guarantee. Run the repaired rung at several smoothing scales and de Bruijn's identity splits the total log-score edge into Fisher-divergence rungs (shape) plus a coarse-scale term (between-mode mass): the heat-ladder pool of the companion paper.

In this repo

mechanisms/nearest_the_pin.py provides kde_density, pot_split (the pool), energy_score_via_projection (the projection route), mollified_log_score (the jittered-pin repair), and the closed-form expected scores gaussian_expected_raw_kde_log_score / gaussian_expected_mollified_log_score used by the demo below and parity-tested against the JavaScript. Tests verify the incentive in both directions: raw settlement prefers the $h^2$-shaved cloud; the jittered pin prefers the truth.

The race: shave against the truth

A Galton board drops the truth: sixteen rows of pegs, so the pin lands standard-normally. Three players submit clouds of $m=150$ samples each round: truthful ($v=1$), shaver ($v=1-h^2$) and padder ($v=1+h^2$), with $h=0.7$. The pot splits by the density each cloud's KDE puts at the pin. With the raw pin ($j=0$) the shaver's wealth pulls ahead, exactly as Theorem 1 predicts. Slide the jitter to $j=h$ (settlement then averages the log density over sixteen jittered pins, the mollified score of §4 of the paper, drawn as ghost ticks) and the truthful player wins.

Try it: how much should we jitter?

Truth is $N(0,1)$. The mechanism smooths your cloud with KDE bandwidth $h$ and jitters the pin with s.d. $j$. The curves are exact expected scores; the score peaks at $v^\ast = 1 + j^2 - h^2$. With $j=0$ (red) the mechanism pays you to shave $h^2$ off your variance; with $j>h$ it pays you to pad. Truth-telling sits at $v^\ast=1$ iff $j=h$: jitter exactly as much as you smooth. Slide $j$ and watch the green peak cross the truth line.

Code: mechanisms/nearest_the_pin.py · Paper: Scoring Point-Cloud Distributional Submissions · Pool: the nearest-the-pin parimutuel (§2 of the paper) · Related: scoring rules, local (Hyvärinen) scoring