← Catalog

Peer prediction

A scoring rule needs an outcome to settle against. Peer prediction is what you do when there is no such outcome, when the thing being elicited is never verified. Instead of comparing a report to ground truth, it scores a report by its statistical relationship to other agents' reports, and is designed so that truthful reporting is an equilibrium. The technical work is entirely in defeating the obvious bad equilibrium: paying for agreement invites everyone to agree on nothing in particular.

Eliciting truth without ground truth

Many valuable judgements are never settled by an observable outcome: subjective quality ratings, peer review, the accuracy of a field report, or a forecast about a counterfactual that will not be run. A proper scoring rule cannot be applied: there is no realised value to plug in. What is available is the joint distribution of the agents' private signals. If agents share a common prior and each draws a private signal, then one agent's signal is informative about another's report; a mechanism can exploit that correlation to make truth-telling pay, even though it never learns the truth itself. Peer-prediction mechanisms are the family of payment rules built on exactly this idea, and they sit alongside aggregation as the two halves of working with crowd judgements: aggregation combines reports, peer prediction incentivises them.

Output agreement, and two ways it fails

The naive baseline simply pays two agents when their reports match. With reports $a$ and $b$ the reward is

The familiar failure is the uninformative equilibrium. The constant strategy "always report option 0," played by everyone, scores a perfect $1$ on every pairing while revealing no private information at all. It is risk-free and requires no thought. This is not special to output agreement: an elicitation game without verification has an uninformative equilibrium exactly when its reward matrix, read as a two-player complete-information game, has one (Waggoner & Chen 2014, Theorem 3).

The less obvious failure is that truth-telling need not be an equilibrium at all, and correlated signals are not enough to make it one. Take symmetric binary signals with $P(0,0) = 0.50$, $P(0,1) = P(1,0) = 0.20$, $P(1,1) = 0.10$. These are positively correlated, since $P(1,1) - P(A{=}1)P(B{=}1) = 0.10 - 0.09 > 0$. But an agent who observes a $1$ computes $P(B{=}1 \mid A{=}1) = 0.10/0.30 = 1/3$ and maximises expected agreement by reporting $0$. Truth is not even a best response on that signal.

The condition that does deliver truthfulness is self-dominance: $P(S_k = x \mid S_i = x) > P(S_k = \tilde x \mid S_i = x)$ for every $\tilde x \neq x$, that is, whatever you saw is what you believe your peer most likely saw (Radanovic & Faltings 2013, Proposition 1). Under it, output agreement is the unique truthful mechanism up to positive affine transformation (Frongillo & Witkowski 2016). Do not confuse it with the weaker self-predicting condition $P(S_k = x \mid S_i = x) > P(S_k = x \mid S_i = x')$, which compares down a column rather than across a row, is equivalent to positive correlation in the binary case, and belongs to the Peer Truth Serum instead. And even where truth is an equilibrium, no mechanism without verification can guarantee a strict equilibrium more informative than common knowledge (Waggoner & Chen 2014, Theorem 2).

In code: output_agreement(report_a, report_b) in mechanisms/peer_prediction.py returns $1$ on a match and $0$ otherwise, and output_agreement_is_truthful(joint) checks self-dominance for a joint signal distribution.

Prelec's Bayesian Truth Serum

Prelec (2004) breaks the degenerate equilibrium by asking each respondent for two things rather than one. Respondent $i$ supplies an information report $x_i$, their own answer, an index in $0,\dots,n-1$, and a prediction report $y_i$, a full distribution over the answers estimating how the rest of the population will answer. The mechanism then rewards answers that are more common than the population collectively predicted them to be, "surprisingly common" answers. Truthful respondents are surprisingly common because they have private knowledge that others, when predicting, cannot fully anticipate.

Writing $\bar x_k$ for the empirical frequency of answer $k$ across all information reports, and $\bar y_k$ for the geometric mean of the predicted frequencies for $k$, the BTS score for respondent $i$ who chose answer $k = x_i$ is

The weight $\alpha \ge 0$ trades off the two halves; with $\alpha = 1$ the score is Prelec's original. The key quantities are population-level: no single respondent can move $\bar x$ or $\bar y$ much, so the incentive each faces is, in the large-population limit, against a fixed empirical distribution.

Information score and prediction score

The two terms reward different things. The first, $\log(\bar x_k / \bar y_k)$, is the information score. It is positive exactly when answer $k$ occurs more often than the crowd's predictions implied it would, the "surprisingly common" criterion. An agent maximises it by reporting the answer their own private signal makes genuinely more likely than others realise, not by chasing the popular answer.

The second term is the prediction score,

the negative Kullback–Leibler divergence between the empirical answer distribution $\bar x$ and the respondent's own prediction $y_i$. It is maximised (at $0$) when the prediction report exactly matches the realised population frequencies, and penalises miscalibrated forecasts. This is the same logarithmic, strictly proper structure that drives the scoring-rules family, here applied not to an outcome but to the crowd's own distribution. Together the two terms make it best to report both a truthful answer and a truthful belief about everyone else.

Equilibrium guarantee and limitations

Prelec's theorem is that, under a common-prior model where each agent's signal shifts their posterior over the population distribution, truthful reporting of both $x_i$ and $y_i$ is a Bayesian Nash equilibrium, and truthful, well-calibrated respondents earn the highest expected score. Crucially the information score has positive expected value precisely because truthful answers are the surprisingly common ones, which is what dismantles the constant-report equilibrium that defeats output agreement.

The guarantees come with assumptions, and they are strong ones. BTS needs a common prior, conditionally independent signals, and, in the original, an infinite population; Prelec notes it survives for suitably large finite $n$ with a prior-dependent threshold. Small populations break it outright rather than degrading: Witkowski and Parkes (2012, Theorem 2) show BTS is neither Bayes-Nash incentive compatible nor interim individually rational at $n = 3$, with an explicit counterexample, and its score is unboundedly negative for prediction reports at $0$ or $1$. Their Robust BTS repairs both for every $n \ge 3$ and every admissible prior, at the cost of binary signals; Radanovic and Faltings (2013) extend it to non-binary signals under self-dominance.

Truthful reporting is also not the unique equilibrium. What it has beyond existence is payoff dominance: no other equilibrium earns a higher expected information score, and uninformative ones earn strictly less (Kong & Schoenebeck 2019, Theorem 7.8). The multi-task line of work, Dasgupta–Ghosh and especially the Correlated Agreement mechanism of Shnayder, Agarwal, Frongillo and Parkes (2016), relaxes the prior requirement by scoring agents across many tasks.

Correlated Agreement, and what it does and does not guarantee

Correlated Agreement needs no prediction report at all, only each agent's signal on a shared pool of tasks. It scores agent $i$ by agreement with peers on the same task, beyond the chance agreement expected from unrelated tasks, read through a sign matrix $S = \operatorname{Sign}(\Delta)$, which is $1$ where $\Delta > 0$ and $0$ otherwise, $\Delta_{ab} = P_{\text{same}}(a,b) - P(a)P(b)$:

The baseline subtraction is what kills the uninformative equilibrium: a constant report agrees with peers but agrees equally on unrelated tasks, so it nets zero.

What CA guarantees is informed truthfulness, and it holds for all worlds (their Theorem 5.2): no strategy profile earns more than truthful, and a profile can only tie if both agents play informed strategies. Three qualifications belong with that. It is a strict correlated equilibrium, not a dominant strategy. Truthful is only weakly highest-paying, and the ties are characterised: a non-permutation strategy can tie exactly when signals are clustered (two identical rows or columns in $S$), and asymmetric permutations can tie exactly when the distribution has paired permutations. Absent both, CA is strongly truthful, which the authors report holds in roughly half their empirical worlds.

The third qualification corrects what this page previously said. CA is not stochastic-dominance truthful. That property belongs to a different mechanism, Enforced Agreement (Zhang, Xu, Pennock & Schoenebeck 2025), which is output agreement preceded by forcing each agent's report marginal to a target, so over-reporting the majority signal buys nothing because the surplus is simply flipped back. Their Theorem 6.1 gives SD-truthfulness for binary self-predicting signals; their Example 4.1 exhibits a world where a constant strategy is not stochastically dominated under CA, which needs their partition-rounding reduction to get there. First-order stochastic dominance means the truthful score distribution is preferred under every increasing utility, which is stronger than an expectation comparison and stronger than the risk-aversion framing this page used before.

One more caveat the demo makes concrete. CA assumes $S$ is fixed, not estimated from the reports it scores. With two agents that assumption fails loudly: a constant reporter is half the data, flips the estimated $S$ from the identity to its opposite, and is then paid by the matrix they inverted. The detail-free variant estimates $S$ on a held-out half of the tasks, which does not help when a single agent is half the population.

Try it

The simplest peer mechanism: output agreement pays 1 when two agents' reports match, else 0. It rewards truth-telling when signals are self-dominant, which is stronger than positive correlation, and it is exploitable regardless: agents can settle on a constant report and always score 1 without revealing anything.

Code: mechanisms/peer_prediction.py (bayesian_truth_serum, correlated_agreement) · Demo: examples/sim_peer_prediction.py, examples/sim_correlated_agreement.py · Related: scoring rules, aggregation · Research: gaps-and-roadmap.md