Do LVLMs Know When They Are Wrong? — Trajectory-Based Correctness Estimation

Abstract

Large vision-language models (LVLMs) exhibit strong multimodal reasoning capabilities, yet reliably estimating whether a generated response is correct remains a fundamental challenge. Existing confidence estimation methods commonly rely on repeated querying, self-consistency sampling, or layer-wise probing, which introduce additional inference overhead, self-preference bias, or costly full-depth analysis. In this work, we propose a trajectory-based framework for estimating LVLM response correctness from a single forward generation pass. Instead of treating internal representations as isolated layer-wise signals, we model response generation as a structured trajectory through the LVLM, capturing the evolution of visual evidence in the vision encoder, geometric transformations induced by cross-modal projection, and probability dynamics within the language decoder. Our key insight is that response correctness is encoded in the geometry and dynamics of these component-specific trajectories. Based on this observation, we derive lightweight trajectory descriptors for each major LVLM module and integrate them through a calibrated correctness estimator trained with as few as 500 labeled examples. At test time, the proposed estimator requires no additional inference passes or ground-truth labels. Extensive experiments across multiple LVLM architectures and benchmark datasets demonstrate consistent improvements over existing baselines, including up to 16.44% higher AUROC on Qwen3-VL-2B over VizWiz and 15.61% higher AUPR on LLaVA-Next-7B over ScienceQA compared to the strongest competing methods.

Correctness is a trajectory, not a snapshot

Prior work treats confidence as a property of a single layer, a final logit, or a re-sampled answer. We instead look at how representations move across normalized depth τ ∈ [0,1] — and find that each component of an LVLM leaves a distinct, structured signature of correctness along the way.

Line charts showing top-token probability trajectories across language model layers for ScienceQA, OCRBench, InfoVQA, and MME, with correct and incorrect generations diverging in the middle layers.

Language decoder

Confidence emerges in three phases

At shallow depth the top-token probability gap between correct and incorrect generations is negligible — the residual stream hasn't committed yet.

Through the middle layers the two trajectories diverge, and that separation is then preserved and reinforced as logits crystallize near τ ≈ 1.

Vision encoder

Evidence accumulates, unevenly

Visual-evidence trajectories are non-monotonic — discriminative regions appear intermittently, interleaved with layers where separation weakens or reverses.

For visually demanding tasks, correct trajectories often start weak and only become discriminative in deeper layers — evidence accumulates rather than starting strong.

Projection layer

Signal hides in low-variance directions

Global projection magnitude looks the same for correct and incorrect generations — but it isn't uninformative, it's just measured wrong.

Low-index (dominant) PCA components are shared by both classes; higher-index, low-variance components carry the real geometric separation.

A trajectory-scoring pipeline

Each transformer layer update is treated as one step of a continuous residual-stream trajectory, \( \frac{d\mathbf{h}}{d\tau} = F(\mathbf{h}(\tau), \tau; \theta) \), following the Neural-ODE view of residual networks. Three lightweight, component-specific descriptors are computed from that trajectory in a single pass and combined into one correctness estimate.

Diagram of the multimodal trajectory-scoring pipeline: vision transformer blocks feed a projection module, then decoder blocks, with logit-probability calculators (LPC) at each stage producing vision-encoder, projection-subspace, and language-model trajectory scores. — Projection modules (Proj) and logit probability calculators (LPC) estimate the vision-encoder (S_ve), projection-subspace (S_proj), and language-model (S_lm) trajectory scores from evolving multimodal representations — all within one forward pass.

Language-model score

causal depth integral

\[ S_{lm}[p] = \int_0^1 \tau \big(x_p(\tau) - w_p(\tau)\big)\, d\tau \]

A depth-weighted integral of the top-token probability gap, so deeper, more decisive layers count more than early, uncommitted ones.

Vision-encoder score

conditional depth integral

\[ S_{ve}[\mathbf{h}] = \frac{\displaystyle\int_0^1 \chi(\tau)\big(x_{ve}(\tau)-w_{ve}(\tau)\big)\,d\tau}{\displaystyle\int_0^1 \chi(\tau)\,d\tau} \]

A gate χ(τ) restricts integration to depth intervals where correct and incorrect evidence actually separate, avoiding dilution from ambiguous regions.

Projection-subspace score

discriminative subspace integral

\[ S_{proj}[\phi] = \frac{\displaystyle\int_0^1 I(s)\,\hat{\gamma}\big(\cos\phi_x(s)-\cos\phi_w(s)\big)\,ds}{\displaystyle\int_0^1 I(s)\,ds} \]

An indicator I(s) selects the low-variance PCA directions where cross-modal alignment differs most between correct and incorrect trajectories.

\[ S(\mathbf{h},p,\phi) = \alpha\, S_{ve}[\mathbf{h}] + \beta\, S_{lm}[p] + \gamma\, S_{proj}[\phi] \]

α, β, γ are calibrated per task from as few as 250 correct + 250 incorrect examples. Higher S means the sample's trajectory looks more like a correct generation — computed with zero extra inference passes.

Results

Evaluated across 9 LVLM variants and 8 benchmarks spanning document VQA, chart reasoning, OCR / scene-text, science reasoning, and real-world VQA, against single-pass and multi-pass self-evaluation baselines.

+16.44%

AUROC gain, Qwen3-VL-2B on VizWiz, vs. strongest baseline

+15.61%

AUPR gain, LLaVA-Next-7B on ScienceQA, vs. strongest baseline

4.6 ms

inference latency per sample, after calibration

500

labeled examples needed to calibrate α, β, γ

AUROC (%), negative-class — LLaVA-Next-7B, all eight benchmarks
Method	SQA	InfoVQA	ChartQA	TextVQA	DocVQA	OCRBench	MME	VizWiz
CoE_R	59.35	53.02	58.24	52.12	50.26	51.09	52.59	57.41
CoE_C	59.04	54.28	61.62	52.76	57.83	63.08	55.03	56.10
INSIDE	70.80	59.07	60.43	59.48	60.10	59.25	66.92	60.94
Logical Closed Loop †*	54.10	51.26	58.74	54.25	52.35	61.56	49.66	51.51
Devils in Middle Layers *	33.94	50.63	56.38	49.12	57.68	54.04	48.30	34.04
Semantic Uncertainty †	50.74	53.74	56.07	60.52	62.20	57.09	64.41	50.45
Calib-Corr	73.73	56.54	54.43	61.53	62.67	62.34	66.82	60.28
VL-Calibration *	53.30	53.21	54.19	54.38	50.75	53.13	53.51	42.03
Ours *	74.80	61.14	62.35	62.80	61.68	65.25	67.47	61.25

† requires multiple inference passes to determine correctness · * uses both vision and text to determine correctness. Full AUPR/F1 breakdown and results for LLaVA-Next-13B, Qwen3-VL-2B/4B, InstructBLIP-XL/XXL, InternVL2-4B/8B, and ShareGPT4V are in the paper.

Cheap to calibrate, free at inference

Vision-encoder calibration (α)608.98 s / 250 samples
Language-model calibration (β)146.38 s / 250 samples
Projection calibration (γ)56.23 s / 250 samples
Total one-time calibration≈ 13.5 minutes
Ensemble / scoring at test time4.60 ms / sample

All timings are measured on OCRBench with LLaVA-Next-7B. Calibration is a one-time cost per task — after that, the pipeline requires no additional forward passes, sampling, or re-prompting to score a new generation, unlike self-consistency or multi-pass judge baselines.

Where this doesn't reach yet

Abstractive answers

The framework assumes a response maps to a discrete answer token or small token group — it doesn't yet cover open-ended captions or free-form explanations.

Compositional reasoning

Visual evidence is aggregated per-patch independently, which can limit cases needing relational reasoning — comparing positions, counting, or object interactions.

Refusals & hedges

Scoring assumes the model commits to an answer at generation time; abstentions and hedged responses aren't currently assigned a score.

Long reasoning chains

Analysis targets the final answer token's trajectory. Intermediate reasoning tokens before that answer may follow different dynamics.

Citation

This paper is currently under anonymous review. Please check back for the final venue, author list, and BibTeX key once the review process concludes.

@misc{trajectory-lvlm-correctness,
  title = {Do LVLMs Know When They Are Wrong? Trajectory-Based Correctness Estimation from a Single Forward Pass},
  author = {Anonymous Author(s)},
  note   = {Under anonymous review at EMNLP},
  year   = {2026}
}