How would you validate a surrogate's accuracy in rare processes it hasn't seen much training data for?

This is a sophisticated question about a real problem in computational physics: how do you know a neural network surrogate—a fast approximation trained on simulated data—actually works when it encounters rare events or regions of parameter space it saw little training data for?

The archive offers several complementary approaches:

Direct Validation Against Known Benchmarks

The most practical strategy is to validate against a "reference" surrogate you trust. The SURF method [AG-2025.11-1399] does exactly this: it trains your target surrogate on samples from *another* surrogate (trained on real data), then runs exact statistical tests to see if your model matches that reference distribution. This lets you spot when a surrogate is hallucinating—the paper gives a concrete example where autoregressive models incorrectly exaggerate physics signals, revealing they've learned something unphysical.

Similarly, surrogate models for ATLAS analyses [AG-2025.02-1260] were validated by comparing their predicted event selection probabilities against the original analysis, checking whether rare signal events were handled correctly.

Uncertainty Quantification as Your Canary

A surrogate that *knows what it doesn't know* is safer in rare regimes. Two approaches stand out:

Bayesian uncertainties: Train the surrogate to output both a prediction *and* a confidence interval [AG-2024.12-1502]. Then test the calibration using "pull distributions"—if the surrogate says it's 95% confident, does the true answer actually fall in that range 95% of the time? This catches overconfident models before they mislead you.

Conformal prediction [AG-2025.12-1734] is even more powerful: it wraps any pre-trained model in a distribution-free calibration layer that gives you rigorous finite-sample coverage guarantees—no retraining needed. The catch is it widens uncertainty bands, but it does so honestly.

Testing on Out-of-Distribution Data

For rare processes, you need to explicitly test the surrogate on data it wasn't trained on. The NPLM method [AG-2025.11-1255] uses a learning-based goodness-of-fit test inspired by hypothesis testing: train a separate "detector" network to distinguish real rare events from surrogate-generated ones. If the detector succeeds, your surrogate is missing something about those rare events.

Alternatively, generative surrogates can be extended with Bayesian uncertainties [AG-2024.02-1165] to flag when they encounter "unknown inputs"—inputs far from the training distribution—and automatically indicate reduced validity.

The Bottom Line

Validating surrogates on rare processes requires *three* things working together:

1. Reference benchmarks (SURF method) to catch systematic biases

2. Calibrated uncertainties (Bayesian or conformal) that honestly reflect what the surrogate doesn't know

3. Out-of-distribution detection (NPLM or Bayesian flagging) to warn when you're in a regime the training data didn't cover

No single approach is foolproof, but combining them catches most failure modes.