Does the CritPt benchmark distinguish between missing reasoning steps versus genuinely impossible reconstructions from context?

Your question is about benchmarking methodology, not frontier science itself. The preprints provided do address AI evaluation in physics, but *none of them* directly compare missing reasoning steps versus genuinely impossible reconstructions—that's a subtle epistemological distinction in benchmark design that these papers don't explicitly probe.

That said, I can point toward what the benchmarks *do* reveal about this problem:

CritPt's design [AG-2025.09-1139] creates "guess-resistant" research challenges, but the paper doesn't distinguish between whether a model fails because it skips intermediate steps versus because the task genuinely exceeds its capability ceiling. It measures *outcome* (6% success on full problems), not the mechanism of failure.

The most relevant work here is AG-2026.04-565, which directly addresses your concern. It introduces a five-level grading rubric that separates "statement correctness" from "tacit step reconstruction" and notes "systematic degradation when tasks require reconstruction of omitted reasoning steps or reorganization of representations under global consistency constraints" [AG-2026.04-565]. This rubric *attempts* to distinguish incomplete reasoning chains from conceptual impossibility—but the paper doesn't explain how to tell a model that *skipped* a step from one that *cannot compute* it.

TPBench [AG-2025.02-240] documents "common failure modes" but doesn't categorize them along the axis you're asking about. Similarly, the test-time scaling work [AG-2025.06-1078] uses symbolic verification to catch errors, not to diagnose *why* they occur.

The honest answer: none of these benchmarks have a principled method to distinguish missing reasoning steps (a shortcut or lazy search problem) from genuinely impossible reconstructions (a fundamental capability gap). That's an open methodological question.