Can transformer models generalize to gravitational waves not present in their training data?

# Generalization of Transformers to Unseen Gravitational Waves

This is an excellent question, and the archive shows transformers excel at it—though with important nuances.

Transfer Learning Adapts Beyond Training Data

The clearest evidence comes from two directions. First, [AG-2025.12-634] demonstrates that GraviBERT—a transformer pretrained on gravitational wave data in a self-supervised phase—can generalize dramatically to *new detector configurations and waveform models it never saw during training*. When the detector noise profile changes, the pretrained model converges 15× faster on small new datasets and cuts estimation errors by up to 47%. This matters because real detectors drift; a model that adapts without full retraining is operationally invaluable.

Second, [AG-2024.10-284] shows transformers handle shifts in noise characteristics that occur *within* a single observing run. The model maintains 99%+ accuracy even when the noise power spectral density (the statistical fingerprint of detector noise) fluctuates over short timescales—something the training data didn't explicitly represent.

Architectural Flexibility Enables Generalization

[AG-2025.12-052] pushes this further with a model called Dingo-T1, which uses a single flexible transformer to analyze real gravitational-wave events under *different frequency ranges, detector configurations, and analysis settings* all at inference time—without retraining. This is conceptually similar to asking a language model trained on English text to suddenly work on French: the architecture's learned structure generalizes across domains.

The Mechanism: Self-Attention

Why transformers specifically? The self-attention mechanism (the core innovation in transformer architecture) learns long-range dependencies in time-series data directly from examples. Because it doesn't rely on hand-crafted signal-processing features, [AG-2024.12-621] found that Whisper—a transformer pretrained on *audio*, not gravitational waves at all—could be fine-tuned for gravitational wave detection, achieving reliable classification of real signals versus instrumental glitches. That cross-domain transfer suggests the learned representations are robust.

Caveats

Generalization isn't perfect or unlimited. [AG-2025.05-459] applied transformers to overlapping binary black hole mergers (a harder regime) and achieved 10–20% errors on key parameters—acceptable for a first pass, but not negligible. The broader pattern suggests transformers generalize well to *variations within the physics they've learned*, but there's no evidence they extrapolate far beyond their training distribution (e.g., to entirely new classes of sources).

The provided preprints focus on detector adaptation and noise robustness rather than generalization to gravitationally-wave signals of truly different physical types, so evidence for broader source-type generalization isn't directly available here.