What happens to Bayes factors when two competing models share absorbing systematic uncertainties?

This is a sharp technical question about Bayesian model comparison under a specific pathology — and the archive has real insight.

When two competing models both fail to account for the same systematic uncertainty, that uncertainty can "absorb" into the parameters of whichever model is more flexible, allowing both to fit the data equally well. The consequence is subtle but severe: the Bayes factor becomes insensitive to model truth [AG-2025.02-241].

Here's why it matters: Bayes factors (evidence ratios) are meant to reward models that predict data without overfitting. But if a systematic error is invisible to both models — say, a calibration bias affecting all measurements equally — then a more flexible model can simply adjust its parameters to hide the problem, and the likelihood will look just as good. Both models achieve similar evidence, so the Bayes factor hovers near 1, telling you nothing about which is actually correct.

This is called a category II model comparison problem in the literature [AG-2025.02-241]. The danger is real in composite models (e.g., foreground + signal in 21-cm astronomy, or one model component + another). A poor fit in one piece gets absorbed by flexibility in another, yielding unbiased aggregate predictions but biased component estimates and spurious Bayes factor contrasts.

The solution is component-level validation before comparing composites. The BaNTER framework [AG-2025.02-241] introduces "null-test evidence ratios" — you first validate each model component separately on data it wasn't fit to, then use that information to disambiguate the composite comparison. Likewise, when systematic uncertainties are known, explicitly modeling them as part of the framework — rather than assuming they vanish — improves both parameter recovery and model discrimination [AG-2025.04-1538].

The deeper lesson: Bayes factors are only as good as your model's treatment of uncertainty. If you ignore correlated systematics, the factor is blind.