Forecast aggregation

Suppose $m$ forecasters each report a probability distribution over the same outcomes. We want one consensus distribution. An opinion pool is a rule that maps the panel of forecasts to that single distribution. The two canonical pools, linear and logarithmic, differ in whether they average probabilities or average evidence, and that one choice fixes almost everything else about how the consensus behaves.

The problem of combining forecasts

Take a categorical outcome with $K$ classes. Forecaster $i$ submits a probability vector $p_i = (p_{i1}, \dots, p_{iK})$ with $p_{ik} \ge 0$ and $\sum_k p_{ik} = 1$, and carries a weight $w_i \ge 0$ with $\sum_i w_i = 1$. A pool is a map $(p_1, \dots, p_m) \mapsto \bar p$ returning another probability vector. The defining survey is Genest & Zidek (1986), Combining Probability Distributions: A Critique and an Annotated Bibliography, which frames the question axiomatically: which natural properties, preservation of unanimity, the external Bayesianity of commuting with a shared likelihood update, marginalisation, can a pool satisfy, and which are mutually exclusive. No single pool satisfies all of them, so the operator is a modelling decision, not a default.

A panel disagrees in two ways that a pool must handle differently: truthful spread of opinion, which we want to summarise, and a few badly wrong or adversarial reports, which we want to discount. The first three pools below treat all reports as truthful; the fourth defends against the second case.

The linear opinion pool

The linear pool is the weighted arithmetic mean of the forecasts:

$\bar p_{\,\mathrm{lin}} = \sum_{i=1}^{m} w_i \, p_i.$

It is the most transparent pool: the consensus probability of an outcome is just the weighted fraction of belief the panel placed there. It preserves the support: if any single forecaster assigns positive probability to an outcome, so does the pool, which makes it robust to confident dissent and the natural pool when forecasters are exchangeable. The price of that robustness is under-confidence: averaging probabilities pulls mass toward the centre and inflates the variance of the consensus, so a linear pool of sharp, agreeing forecasters is typically less sharp than its members. It is also generically multimodal: it can reproduce a panel's clusters as separate bumps rather than reconciling them. In code, linear_opinion_pool(forecasts, weights) in mechanisms/aggregation.py; with $w_i = 1/m$ it is the plain average.

The logarithmic opinion pool

The logarithmic pool is the normalised weighted geometric mean, a weighted average in log-space, renormalised to sum to one:

$\bar p_{\,\mathrm{log},k} = \dfrac{\prod_{i} p_{ik}^{\,w_i}}{\sum_{j} \prod_{i} p_{ij}^{\,w_i}}.$

Where the linear pool averages probabilities, the log pool multiplies evidence: adding log-probabilities is combining independent log-likelihoods, so the log pool is the consensus you reach by treating each forecaster as carrying separate evidence about the outcome. It is correspondingly sharper than its members: unanimous mass reinforces multiplicatively, and it is unimodal under mild conditions. It is also externally Bayesian: pooling and then applying a shared likelihood update gives the same answer as updating first and then pooling. The flip side is decisiveness: because any zero probability annihilates the product, the log pool zeroes out any outcome that any forecaster rules out (the implementation clips to a small floor before taking logs to keep this numerically stable).

The log pool is intimately tied to scoring. Its log-domain form is exactly the operator that, given the panel, minimises the weighted sum of Kullback–Leibler divergences $\sum_i w_i \, D_{\mathrm{KL}}(\bar p \,\|\, p_i)$: the consensus is the distribution closest to the whole panel in total KL. That is the same relative-entropy geometry that makes the logarithmic score strictly proper, so the log pool is the aggregation counterpart of log-score elicitation. In code, logarithmic_opinion_pool(forecasts, weights).

Robust pooling against adversarial nodes

Both pools above are linear or log-linear in the reports, so a single extreme forecast can drag the consensus arbitrarily far, fine for a truthful panel, dangerous for an open network where some nodes may be broken, stale, or adversarial. The depth-trimmed mean hardens the linear pool by discarding outliers first. Each forecast is scored by its outlyingness, here the $L_1$ distance from the coordinate-wise median forecast, a simple proxy for low statistical depth, and the most outlying $\mathrm{trim}$ fraction is dropped before averaging the rest:

$\bar p_{\,\mathrm{trim}} \;\propto\; \mathrm{mean}\big\{\, p_i : i \in \mathcal{D}_{1-\mathrm{trim}} \,\big\},$

where $\mathcal{D}_{1-\mathrm{trim}}$ is the deepest $1 - \mathrm{trim}$ fraction of the panel and the result is renormalised. With $\mathrm{trim} = 0$ this is exactly the linear pool; raising it trades a little efficiency for a bounded influence per node, so a minority of bad forecasts cannot move the consensus past the trimming threshold. The median anchor is itself a deep, hard-to-manipulate summary, which is what makes the trim adversarially robust rather than merely tidy. In code, depth_trimmed_mean(forecasts, trim).

Choosing the weights

Every pool above takes the weights $w_i$ as given, but choosing them is the substantive question. Equal weights are the truthful default when nothing distinguishes the forecasters; performance weights, set from past scores, so that consistently accurate forecasters count for more, are the usual refinement, and tie the pool back to the scoring rule used to evaluate the panel.

A different lens applies when the reports are point forecasts of the same quantity: the weights that minimise the variance of the combined forecast are the classic result of Bates & Granger (1969). With forecast-error covariance $\Sigma$, the optimal weights are $w \propto \Sigma^{-1}\mathbf{1}$, normalised by $\mathbf{1}^{\top}\Sigma^{-1}\mathbf{1}$, exactly the minimum-variance portfolio, with forecasts in place of assets and forecast errors in place of returns. The whole difficulty then collapses to estimating the precision matrix $\Sigma^{-1}$ from short, noisy histories: estimate it badly and the “optimal” weights underperform a flat average, the forecast-combination puzzle (Timmermann 2006; Smith & Wallis 2009). The microprediction precise package is a battery of exactly these covariance / precision estimators and the portfolio and aggregation rules built on them.

The most interesting case makes the weights endogenous: rather than assigning weights administratively, let forecasters stake on their own reports and read the weights off the resulting allocation. This is the wealth-weighted regime, where a forecaster's influence is the capital they are willing to risk and the pool is realised as a settlement rule rather than a formula imposed from outside. The nearest-the-pin parimutuel pool develops exactly this idea: a density-based pot-splitting mechanism in which stakes determine weights, the consensus emerges from the allocation, and proper-scoring incentives keep truthful reporting optimal, aggregation and reward fused into a single mechanism.

Try it

Two categorical forecasts over outcomes A/B, equally weighted. The linear pool is the arithmetic mean; the logarithmic pool is the normalised geometric mean, sharper, and it pulls toward agreement.

Code: mechanisms/aggregation.py · Demo: examples/sim_aggregation.py · Related: scoring rules