QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Shuxiang Cao; Zijian Zhang; Abhishek Agarwal; Grace Bratrud; Niyaz R. Beysengulov; Daniel C. Cole; Alejandro Gómez Frieiro; Elena O. Glen; Hao Hsu; Gang Huang; Raymond Jow; Greshma Shaji; Tom Lubowe; Ligeng Zhu; Luis Mantilla Calderón; Nicola Pancotti; Joel Pendleton; Brandon Severin; Charles Etienne Staub; Sara Sussman; Antti Vepsäläinen; Neel Rajeshbhai Vora; Yilun Xu; Varinia Bernales; Daniel Bowring; Elica Kyoseva; Ivan Rungger; Giulia Semeghini; Sam Stanwyck; Timothy Costa; Alán Aspuru-Guzik; Krysta Svore

doi:10.48550/arXiv.2604.25884

← Recent

AG-2026.04-1734·quant-ph·cross-listed: cs.CV

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Authors

Shuxiang Cao
Zijian Zhang
Abhishek Agarwal
Grace Bratrud
Niyaz R. Beysengulov
Daniel C. Cole
Alejandro Gómez Frieiro
Elena O. Glen
Hao Hsu
Gang Huang
Raymond Jow
Greshma Shaji
Tom Lubowe
Ligeng Zhu
Luis Mantilla Calderón
Nicola Pancotti
Joel Pendleton
Brandon Severin
Charles Etienne Staub
Sara Sussman
Antti Vepsäläinen
Neel Rajeshbhai Vora
Yilun Xu
Varinia Bernales

Daniel Bowring

Elica Kyoseva

Ivan Rungger

Giulia Semeghini

Sam Stanwyck

Timothy Costa

Alán Aspuru-Guzik

Krysta Svore

Abstract

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Submitted

28 April 2026yesterday

Version

v1

License

CC-BY-4.0

DOI

10.48550/arXiv.2604.25884

Cite this preprint

BibTeX RIS

Imports into BibLaTeX, Zotero, Mendeley, EndNote.

PDF

Open PDF

Opens in a new tab · v1.

Chat with this PDF

Ask questions, probe assumptions, request a plain-English summary. Answers cite sections from the preprint itself.

Community

Questions and answers about this paper from other readers. No formal peer review — just a place to think out loud.