Peer prediction
When there is no verifiable ground truth, reward a report by its statistical relationship to others'. Naive output-agreement collapses to a degenerate equilibrium (everyone reports the same thing). Prelec's Bayesian Truth Serum fixes this by asking for both an answer and a prediction of the population distribution, rewarding answers that are surprisingly common:
$\text{score}_i=\log\dfrac{\bar x_k}{\bar y_k}
+\alpha\textstyle\sum_m \bar x_m\log\dfrac{y_{i,m}}{\bar x_m}$
Code: mechanisms/peer_prediction.py ·
Demo: examples/sim_peer_prediction.py