Why does Bayesian model comparison avoid the look-elsewhere effect when scanning doesn't?

This is a great question about statistical inference, though it touches foundational statistics rather than the frontier physics that dominates this archive. Let me explain the key insight using the preprints available.

The core difference hinges on what gets penalized for complexity.

When you scan many hypotheses (the "look-elsewhere effect"), you're testing each one independently against the data. The more scans you run, the higher the chance that *random noise* will produce a false positive in at least one of them — like flipping a coin 1000 times and being surprised to get 10 heads in a row in one stretch. Traditional hypothesis testing doesn't automatically account for this multiple-testing problem.

Bayesian model comparison, by contrast, embeds complexity penalty into the calculation of *evidence* — the total probability the model assigns to the observed data, integrated over its parameter space [AG-2025.06-064]. A model with more free parameters spreads its predictive probability over a wider range of possible outcomes. If the data don't specifically align with those extra parameters, the evidence *automatically decreases* relative to a simpler model. This is sometimes called "Occam's razor" in Bayesian form: you pay a price in evidence for flexibility you don't actually use [AG-2024.05-316].

The key is that Bayesian comparison compares *complete models* (prior + likelihood) rather than testing isolated parameter values. When you compare a simple model to a complex one using Bayes factors, you're asking: "Given the observed data, how much more probable is model A than model B?" That single question naturally down-weights overfitting [AG-2024.01-1092].

However, this protection is not absolute. The preprints reveal real pitfalls: if model components interact poorly (what [AG-2025.02-241] calls "category II" comparison), systematics in one part can be absorbed by another, and even Bayesian ranking fails—requiring specialized validation frameworks like BaNTER. Similarly, [AG-2024.09-163] warns that misuse of posterior distributions as priors in sequential analyses can break the protection entirely.

So the honest answer: Bayesian model comparison doesn't magically avoid look-elsewhere by some abstract principle—it does so by forcing you to commit to a full model (including a prior) and then comparing predictive adequacy. But if you abuse the procedure (setting priors from previous results, using composite models with absorbing systematics), you lose that protection and create the same biases as naive scanning.