Peer prediction

A scoring rule needs an outcome to settle against. Peer prediction is what you do when there is no such outcome, when the thing being elicited is never verified. Instead of comparing a report to ground truth, it scores a report by its statistical relationship to other agents' reports, and is designed so that truthful reporting is an equilibrium. The technical work is entirely in defeating the obvious bad equilibrium: paying for agreement invites everyone to agree on nothing in particular.

Eliciting truth without ground truth

Many valuable judgements are never settled by an observable outcome: subjective quality ratings, peer review, the accuracy of a field report, or a forecast about a counterfactual that will not be run. A proper scoring rule cannot be applied: there is no realised value to plug in. What is available is the joint distribution of the agents' private signals. If agents share a common prior and each draws a private signal, then one agent's signal is informative about another's report; a mechanism can exploit that correlation to make truth-telling pay, even though it never learns the truth itself. Peer-prediction mechanisms are the family of payment rules built on exactly this idea, and they sit alongside aggregation as the two halves of working with crowd judgements: aggregation combines reports, peer prediction incentivises them.

Output agreement and its degenerate equilibrium

The naive baseline simply pays two agents when their reports match. With reports $a$ and $b$ the reward is

$\mathrm{OA}(a,b) = \mathbf{1}[\,a = b\,].$

Truth-telling is indeed an equilibrium of this game, if I expect you to report truthfully and our signals are correlated, matching your likely answer means reporting my own. But it is not the only one, and not the attractive one. The constant strategy "always report option 0," played by everyone, scores a perfect $1$ on every pairing while revealing no private information at all. This degenerate equilibrium dominates: it is risk-free and requires no thought. Any mechanism that rewards raw agreement is therefore exploitable by coordinated, uninformative reporting.

In code: output_agreement(report_a, report_b) in mechanisms/peer_prediction.py returns $1$ on a match and $0$ otherwise, included to show precisely the failure that the Bayesian Truth Serum repairs.

Prelec's Bayesian Truth Serum

Prelec (2004) breaks the degenerate equilibrium by asking each respondent for two things rather than one. Respondent $i$ supplies an information report $x_i$, their own answer, an index in $0,\dots,n-1$, and a prediction report $y_i$, a full distribution over the answers estimating how the rest of the population will answer. The mechanism then rewards answers that are more common than the population collectively predicted them to be, "surprisingly common" answers. Truthful respondents are surprisingly common because they have private knowledge that others, when predicting, cannot fully anticipate.

Writing $\bar x_k$ for the empirical frequency of answer $k$ across all information reports, and $\bar y_k$ for the geometric mean of the predicted frequencies for $k$, the BTS score for respondent $i$ who chose answer $k = x_i$ is

$\displaystyle \mathrm{score}_i = \log\frac{\bar x_k}{\bar y_k} \;+\; \alpha \sum_m \bar x_m \log\frac{y_{i,m}}{\bar x_m}.$

The weight $\alpha \ge 0$ trades off the two halves; with $\alpha = 1$ the score is Prelec's original. The key quantities are population-level: no single respondent can move $\bar x$ or $\bar y$ much, so the incentive each faces is, in the large-population limit, against a fixed empirical distribution.

Information score and prediction score

The two terms reward different things. The first, $\log(\bar x_k / \bar y_k)$, is the information score. It is positive exactly when answer $k$ occurs more often than the crowd's predictions implied it would, the "surprisingly common" criterion. An agent maximises it by reporting the answer their own private signal makes genuinely more likely than others realise, not by chasing the popular answer.

The second term is the prediction score,

$\displaystyle \alpha \sum_m \bar x_m \log\frac{y_{i,m}}{\bar x_m} = -\,\alpha\, D_{\mathrm{KL}}\!\big(\bar x \,\|\, y_i\big),$

the negative Kullback–Leibler divergence between the empirical answer distribution $\bar x$ and the respondent's own prediction $y_i$. It is maximised (at $0$) when the prediction report exactly matches the realised population frequencies, and penalises miscalibrated forecasts. This is the same logarithmic, strictly proper structure that drives the scoring-rules family, here applied not to an outcome but to the crowd's own distribution. Together the two terms make it best to report both a truthful answer and a truthful belief about everyone else.

Equilibrium guarantee and limitations

Prelec's theorem is that, under a common-prior model where each agent's signal shifts their posterior over the population distribution, truthful reporting of both $x_i$ and $y_i$ is a Bayesian Nash equilibrium, and truthful, well-calibrated respondents earn the highest expected score. Crucially the information score has positive expected value precisely because truthful answers are the surprisingly common ones, which is what dismantles the constant-report equilibrium that defeats output agreement.

The guarantees come with assumptions. BTS leans on a common prior and, in its original single-question form, on a large population so that empirical frequencies are stable; it is an equilibrium result, not a dominant strategy, so other equilibria can still exist. The multi-task line of work, Dasgupta–Ghosh and especially the Correlated Agreement mechanism of Shnayder, Agarwal, Frongillo and Parkes (2016), relaxes these. By scoring agents across many tasks and rewarding agreement only beyond what correlation alone would predict, Correlated Agreement attains informed truthfulness: no strategy beats truthful reporting, and truthful reporting strictly beats every uninformative one, without the mechanism needing to know the prior in advance.

Correlated Agreement, and enforced (stochastic-dominance) truthfulness

Correlated Agreement needs no prediction report at all, only each agent's signal on a shared pool of tasks. It scores agent $i$ by agreement with peers on the same task, beyond the chance agreement expected from unrelated tasks, read through a sign matrix $S = \operatorname{sign}(\Delta)$, $\Delta_{ab} = P_{\text{same}}(a,b) - P(a)P(b)$:

$\operatorname{score}_i = \operatorname{mean}_{j\neq i,\,t} S[r_{i,t}, r_{j,t}] \;-\; \operatorname{mean}_{j\neq i,\,t}\ \mathbb{E}_{b}\, S[r_{i,t}, b].$

The baseline subtraction is what kills the degenerate equilibrium: a constant report agrees with peers but agrees equally on unrelated tasks, so it nets zero. In the binary case truthful reporting is moreover stochastically-dominant truthful (the "Enforced Agreement" property of Schoenebeck et al.): its entire score distribution lies to the right of any deviation's, so truthfulness wins for every risk-averse, monotone utility, not just in expectation. The demo shows the truthful 10th percentile beating 100% of a constant deviator's outcomes.

Try it

The simplest peer mechanism: output agreement pays 1 when two agents' reports match, else 0. It rewards truth-telling when private signals are positively correlated, but note the failure mode it motivates the richer mechanisms above: agents can also collude on a constant report and always score 1 without revealing anything.

agent A reports 1 0

agent B reports 1 0

Code: mechanisms/peer_prediction.py (bayesian_truth_serum, correlated_agreement) · Demo: examples/sim_peer_prediction.py, examples/sim_correlated_agreement.py · Related: scoring rules, aggregation · Research: gaps-and-roadmap.md