Can transformer models generalize to gravitational waves not present in their training data?

# Transformers and Generalization to Unseen Gravitational Waves

The short answer: yes, transformers can generalize beyond their training data—but with important caveats about what "beyond" means in practice.

Transfer Learning and Domain Adaptation

The most direct evidence comes from models designed explicitly to handle shifts in detector noise and waveform characteristics. GraviBERT demonstrates that transformer-based systems can adapt to *new detector configurations* and *different waveform models* not seen during pretraining [AG-2025.12-634]. When the detector noise profile changes, a pretrained transformer converges up to 15× faster on small target datasets and reduces estimation errors by up to 47%—suggesting the model has learned generalizable features rather than memorizing specific training examples [AG-2025.12-634]. Similarly, transfer learning approaches achieve 99%+ accuracy detecting signals even when noise characteristics shift over time, adapting without full retraining [AG-2024.10-284].

Why Transformers Generalize Better

Transformers excel at capturing *long-range temporal dependencies* in waveforms, which appears to be the key. When asked to handle overlapping signals—a genuine out-of-distribution challenge for next-generation detectors—transformers combined with normalizing flows maintain accuracy across different correlation levels in the data, estimating chirp mass within 10–20% of true values [AG-2025.05-459]. This suggests the model learns underlying signal structure rather than surface patterns.

A single flexible transformer model called Dingo-T1 can analyze real gravitational-wave events under a *wide range of analysis configurations*—different detector setups, frequency ranges, and data cuts—all without retraining [AG-2025.12-052]. This is a strong form of generalization: the same weights handle variations the model never explicitly trained on.

The Limits

However, generalization isn't magic. Most work trains on *simulated* waveforms from numerical relativity models (e.g., the NRHybSur3dq8 surrogate) [AG-2024.09-091]. A model trained on one waveform approximant transfers reasonably well to another, but with some loss: cross-waveform transfer achieves up to 44% MAE reductions compared to training from scratch, but doesn't eliminate error entirely [AG-2025.12-634]. The provided preprints don't directly test whether a transformer trained on, say, binary black hole mergers could reliably detect a neutron star–black hole merger or some exotic signal outside its training family.

**The archive doesn't directly address whether transformers can generalize to *fundamentally new classes* of gravitational waves absent from training.** Only whether they adapt to noise shifts, detector changes, and overlapping signals within a known signal family.