Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

Suhyeong Park1,3, Junha Jung2,3, Jungwoo Park2,3, Jaewoo Kang2,3,*
1The Catholic University of Korea   2Korea University   3AIGEN Sciences Inc.
arXiv preprint 2026
*Corresponding Author
Paper Code arXiv

Abstract

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer (Semantic-aware Merging), an object-aware token merging framework that compresses image-side post-projector tokens into \(K\) representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With \(K=64\), SaMer removes more than 93% of image-side tokens and reduces ColPali storage by \(16.09\times\), while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.

Method

SaMer method overview: feature-spatial merging, object-aware assignment, and projection-only adaptation.

SaMer compresses post-projector visual tokens with an object-aware feature-spatial merge, then scores with the original MaxSim objective — at inference it is entirely bbox-free.

SaMer (Semantic-aware Merging) keeps the original late-interaction interface intact and replaces only the image side with a small set of compressed representatives, through three coordinated steps:

Feature-Spatial Merging

Compresses the many post-projector visual tokens into a small set of representative centroids by soft assignment over a distance that combines feature similarity and 2D spatial coherence.

\[ d(i,k)=\bigl(1-v_i^\top\mu_k\bigr)+\gamma\lVert p_i-s_k\rVert^2 \]

Object-Aware Merge Prior

During training only, object annotations add an instance-inconsistency penalty that discourages merging tokens from different object instances — no auxiliary grounding loss, no boxes at inference.

\[ \begin{gathered} P_k(b)=\frac{\sum_{i=1}^{N}\mathbf{1}[c_i=k]\,\mathbf{1}[b_i=b]}{\sum_{i=1}^{N}\mathbf{1}[c_i=k]+\epsilon}, \\[4pt] P_{\mathrm{inst}}(i,k)=1-P_k(b_i). \end{gathered} \]

Projection-Only Adaptation

Freezes the vision encoder and language backbone and trains only the shared projection layer with a contrastive retrieval objective, pulling each query toward its positive images and pushing away the other images in the batch so the compressed representatives stay aligned with MaxSim scoring.

\[ \mathcal{L}_{\mathrm{ret}}=-\log\frac{\sum_{I^{+}\in\mathcal{P}(q)}\exp\!\bigl(S_K(q,I^{+})/\tau\bigr)}{\sum_{I\in\mathcal{B}}\exp\!\bigl(S_K(q,I)/\tau\bigr)} \]

Main Results

Table 1: Retriever-only results across four benchmarks

Analysis

Tables

Auto-advances every few seconds — use the ‹ › arrows to browse one table at a time.

Table 2: Adapted compression at K=64 under projection-only adaptation
Table 3: Compression budget analysis as K varies
Table 4: Feature, spatial, and object-aware component ablation
Table 5: Storage, MaxSim ops, and QPS efficiency

Figures

Auto-advances every few seconds — use the ‹ › arrows to browse one figure at a time.

Figure 2: Token budget vs R@5 trade-off
Figure 3: Grounding comparison (BoxMass, RegionHit, CoverageIoU)
Figure 4a: Qualitative grounding vs single-vector baselines and full multi-vector retrievers
Figure 4b: Qualitative grounding vs compression methods applied to multi-vector retrievers

BibTeX

@article{park2026samer,
  title   = {Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval},
  author  = {Park, Suhyeong and Jung, Junha and Park, Jungwoo and Kang, Jaewoo},
  journal = {arXiv preprint arXiv:2607.04605},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04605}
}