๐Ÿ† NeurIPS 2026  ยท  Accepted, Main Track
Multimodal Reliability AI Alignment & Safety Trustworthy AI

Do robust LVLMs hallucinate more?

Uncovering Robustness-Induced Hallucination in Large Vision-Language Models

Md Zarif Hossain Awal Ahmed Fime Ahmed Imteaj
๐Ÿ“„ Paper arXiv Code
The phenomenon, live

Adversarial training collapses visual tokens โ€” and the decoder falls back on its priors.

Robust vision encoders raise inter-token cosine similarity by up to 38%: tokens from distinct spatial regions become nearly interchangeable. We call this spatial token homogenization. Drag ฮฑ to homogenize the tokens yourself and watch the decoder abandon what it sees for what it expects.

Visual tokens of one image

Each point is a projected visual token, colored by its spatial region (sky, snow, skier, trees).   \(\tilde{z}_i(\alpha) = (1-\alpha)\,z_i + \alpha\,\bar{z}\)

0.00
token cosine similarity โ‰ˆ 0.49 regime: vanilla CLIP
A blue pumpkin resting on a white dining chair beside a wooden table
Prompt: The color of the pumpkin on the chair is
blue
โˆ’2.5
orange
โˆ’8.2
The decoder answers blue โ€” grounded in the pixels.

The image shows a blue pumpkin. As tokens homogenize, probability shifts to the language prior: pumpkins are โ€œsupposed to beโ€ orange.

00

Abstract

Robustness and accurate alignment with visual content are fundamental to the reliable deployment of large vision-language models (LVLMs). In this work, we uncover a previously overlooked trade-off: improving adversarial robustness can inadvertently increase hallucination in LVLMs. Through extensive evaluations on both open-ended generation and discriminative benchmarks, we reveal that adversarially trained LVLMs consistently produce outputs that are less aligned with the visual input, despite their improved robustness. To understand this trade-off, we analyze the visual token embeddings of robust encoders and find that adversarial training increases inter-token similarity by up to 38%. We refer to this phenomenon as spatial token homogenization. We further demonstrate that spatial token homogenization degrades visual information and amplifies decoder's reliance on language priors. Motivated by our findings, we propose Homogenization-Aware Latent Steering (HALS), a training-free inference-time method that steers decoder hidden states away from homogenization-induced hallucination without modifying the robust vision encoder. Extensive experiments across five hallucination benchmarks show that HALS consistently outperforms existing hallucination mitigation methods on robust LVLMs, reducing CHAIRS by 31.3 points on average, improving POPE accuracy by 2.6 points and achieving a 30.6% reduction in hallucination on AMBER. Moreover, robustness evaluations confirm that HALS preserves adversarial performance, demonstrating that robust LVLMs can be made visually grounded without sacrificing their performance.

Findings at a glance

Adversarial robustness degrades visual grounding. HALS recovers it โ€” at inference, for free.

+38%
inter-token similarity

Adversarially trained encoders (TeCoA, FARE, SLADE, TGA-ZSR) homogenize visual tokens, erasing the fine-grained detail the decoder needs to stay grounded.

โˆ’31.3
CHAIRS on average

HALS steers decoder hidden states away from prior-driven generation, outperforming VCD, OPERA, PAI, VTI, and VISTA across every robust encoder.

ยฑ0.0
robustness cost

Clean and robust accuracy on COCO and OKVQA under APGD attack are preserved โ€” grounding is restored without touching the robust encoder.

Figure 1: CHAIR and POPE evaluation showing robust vision encoders exacerbate hallucination on LLaVA-v1.5 and ShareGPT4V
Figure 1: Robust vision encoders exacerbate hallucination in LVLMs. CHAIR evaluation on LLaVA-v1.5 and ShareGPT4V shows that adversarially trained encoders yield higher CHAIR scores, while POPE evaluation confirms higher object hallucination compared to the base LVLM.
Figure 2: Relationship between visual token cosine similarity and hallucination across robust encoders
Figure 2: Visual token similarity tracks hallucination. Adversarially trained encoders produce visual tokens with higher cosine similarity, which increases further after projection; higher similarity correlates with higher CHAIR scores and lower POPE performance.
Figure 3: t-SNE visualization showing robust encoder tokens collapsing into compact clusters compared to vanilla CLIP
Figure 3: Qualitative visualization of spatial token homogenization. t-SNE of projected visual tokens: vanilla CLIP tokens stay well dispersed, while robust-encoder tokens collapse into compact image-wise clusters, indicating increased token similarity.
Why it happens

From visual token homogenization to language-prior dominance

a

Visual tokens homogenize

Mean pairwise cosine similarity rises sharply, before and even more after cross-modal projection. In t-SNE, tokens from distinct regions collapse into a single compact cluster. Higher similarity tracks higher CHAIR and lower POPE, and a larger training budget (ฮต = 4/255 vs 2/255) homogenizes more aggressively.

b

The decoder leans on language priors

With less discriminative visual evidence, generation drifts toward what is statistically plausible rather than visually true โ€” a blue pumpkin becomes orange, empty tables acquire spoons.

Results

Consistent gains across five hallucination benchmarks

Across CHAIR, POPE, AMBER, MMHal-Bench, and MME โ€” and across LLaVA-v1.5, LLaVA-Next, ShareGPT4V, and MiniGPT-4 โ€” the same pattern holds: robust encoders degrade the vanilla baseline, and HALS recovers and surpasses it.

Object hallucination โ€” CHAIR & POPE

CHAIRS โ†“ and POPE accuracy โ†‘ on LLaVA-v1.5, baseline vs. HALS.

Encoder CHAIRS โ†“ (base) CHAIRS โ†“ (HALS) POPE Acc โ†‘ (base) POPE Acc โ†‘ (HALS)
Vanilla CLIP 46.4 15.4 84.8 85.2
SLADE 50.6 16.0 78.6 82.2
FARE 52.4 13.6 77.8 80.2
TeCoA 56.4 19.8 76.5 79.3
TGA-ZSR 60.2 23.2 75.1 77.4

HALS outperforms DOLA, VCD, PAI, OPERA, VTI, and VISTA on CHAIR across every robust encoder, with average CHAIRS reductions of 27.9 (SLADE), 31.3 (FARE), and 34.6 (TeCoA) points.

Open-ended generation โ€” AMBER

CHAIR โ†“, Hallucination โ†“, and Cognition โ†“ on LLaVA-v1.5, baseline โ†’ HALS.

Encoder CHAIR โ†“ Hal โ†“ Cog โ†“
Vanilla CLIP 11.2 โ†’ 7.3 37.6 โ†’ 19.5 7.2 โ†’ 4.2
SLADE 13.9 โ†’ 9.6 40.6 โ†’ 24.9 7.3 โ†’ 4.8
FARE 13.5 โ†’ 9.4 36.4 โ†’ 22.8 6.9 โ†’ 4.5
TeCoA 16.6 โ†’ 12.4 48.7 โ†’ 28.6 7.6 โ†’ 4.9

On ShareGPT4V + TeCoA, where hallucination is most severe, HALS cuts CHAIR from 41.1 to 22.4 (โˆ’18.7) and Hal from 73.0 to 17.3 (โˆ’55.7). The Cog improvement indicates HALS also reduces hallucinations driven by common scene-level priors.

Beyond object hallucination โ€” MMHal-Bench & MME

ShareGPT4V across vision encoders, baseline โ†’ HALS.

Encoder MMHal Avg. โ†‘ MMHal Hall. Rate โ†“ MME Score โ†‘
Vanilla CLIP 2.35 โ†’ 3.05 0.57 โ†’ 0.46 615.6 โ†’ 635.2
SLADE 2.05 โ†’ 2.78 0.61 โ†’ 0.46 598.4 โ†’ 612.4
FARE 2.01 โ†’ 2.66 0.62 โ†’ 0.48 582.6 โ†’ 616.6
TeCoA 1.98 โ†’ 2.58 0.63 โ†’ 0.50 552.6 โ†’ 590.6

On average, HALS improves the MMHal-Bench score by 0.66 points, reduces the hallucination rate by 0.14, and improves MME by 28.7 points โ€” outperforming VTI and VCD across all robust encoders.

Robustness is preserved

Clean / robust performance on COCO (CIDEr) and OKVQA (accuracy) under APGD attack (ฮต = 4/255), LLaVA-v1.5.

Encoder COCO Clean โ†‘ COCO Robust โ†‘ OKVQA Clean โ†‘ OKVQA Robust โ†‘
TeCoA 115.6 79.5 44.4 20.5
TeCoA + HALS 116.3 (+0.7) 79.5 (+0.0) 44.4 (+0.0) 20.3 (โˆ’0.2)
FARE 123.5 86.4 46.4 22.1
FARE + HALS 123.4 (โˆ’0.1) 87.2 (+0.8) 46.5 (+0.1) 22.1 (+0.0)
SLADE 124.0 94.6 48.4 23.4
SLADE + HALS 125.2 (+1.2) 94.0 (โˆ’0.6) 48.4 (+0.0) 23.4 (+0.0)

Changes are negligible across all encoders โ€” hallucination is reduced without sacrificing the adversarial robustness the encoders were trained for.

Generalization โ€” transformation-based defenses

CHAIRS โ†“ and POPE accuracy โ†‘ under test-time defenses (TTC, AOM) with MiniGPT-4, a non-CLIP-based LVLM.

Defense Method CHAIRS โ†“ CHAIRI โ†“ POPE Acc โ†‘ F1 โ†‘
None โ€” 36.8 14.5 76.8 76.3
TTC VTI 38.1 15.5 75.9 74.7
HALS 28.2 14.7 75.5 75.8
AOM VTI 37.4 14.7 76.4 74.9
HALS 31.6 14.0 76.4 75.3

HALS also holds up beyond greedy decoding โ€” it reduces CHAIR under nucleus sampling and beam search alike โ€” and is the fastest mitigation method compared, at 10.2 tokens/s, since the steering vector is pre-computed once rather than extracted per input. Full tables and ablations (steering strength ฮป, training budget ฮต, decoding strategies, qualitative examples) are in the paper.

Acknowledgements

Lab Logo
NSF Logo
NAIRR Logo