Can transformer models generalize to gravitational waves not present in their training data?

# Generalization of Transformers to Unseen Gravitational Waves

Your question hits on a central challenge: whether transformers trained on known gravitational wave signals can handle novel ones they've never encountered.

The short answer is yes, but with important limits — and the archive shows this in multiple ways.

Out-of-distribution robustness

The strongest evidence comes from transfer learning and domain adaptation. GraviBERT [AG-2025.12-634] was pretrained on one set of detector configurations and waveform models, then fine-tuned on new ones. The result: when facing detector noise profiles it had never seen, the pretrained model converged up to 15× faster and reduced estimation errors by ~47%. Critically, this wasn't memorization — it was learning *generalizable* features that transferred to genuinely different conditions.

Similarly, a transfer-learning approach achieved 99%+ accuracy detecting gravitational waves even when the noise characteristics shifted over time, adapting without retraining from scratch [AG-2024.10-284]. This suggests transformers learn noise-robust representations that apply beyond their training distribution.

Architectural advantages for generalization

Transformers seem particularly suited to this problem because they excel at capturing long-range temporal structure. When applied to overlapping signals — a scenario that mimics the challenge of truly novel, complex signals — transformer-based networks (combined with normalizing flows) maintained accuracy across different correlation levels in the data, achieving chirp mass and coalescence time estimates within 10–20% of true values [AG-2025.05-459].

The Vision Transformer on continuous gravitational wave searches approached matched-filter sensitivity on *directed* and *all-sky* benchmarks despite never training on those specific parameter spaces [AG-2025.09-278]. This is a proxy for handling unseen signals: all-sky searches are the broadest possible generalization task.

Caveats

These results assume the unseen signals belong to the same *class* as training data (e.g., binary black hole mergers in the Einstein Telescope's band). A transformer trained on neutron star mergers might struggle with exotic sources like core-collapse supernovae. Generalization is strong *within* the trained signal morphology space, but there's no evidence it extends to fundamentally different physics.

Also, these studies mostly measure generalization via noise robustness or parameter shifts, not via signals with genuinely novel waveform shapes outside their training parameter ranges.

The provided preprints don't directly test how transformers perform on gravitational wave signals *qualitatively different* from those in training data—for instance, signals from exotic compact objects with no training examples.