Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
Figure 1 by Park et al., CC BY 4.0. Cropped to remove the original caption and converted to WebP. Source ↗ · License ↗
Abstract
SaMer compresses the visual representations used by multi-vector image retrievers while trying to keep the small objects and regions a future query might mention. It merges nearby and semantically similar tokens after projection, using object annotations to guide the merge during training. With ColPali and 64 visual tokens per image, SaMer reduced estimated embedding storage by 16.09× and improved top-1 retrieval on Flickr30K and MSCOCO in the reported experiments. Paper
The retrieval problem
Multi-vector retrievers keep an embedding for each visual token. During search, MaxSim matches each text token against the image tokens and uses the strongest match to score the image. This makes it possible to find a small object or attribute mentioned in a query, but searching a large collection requires storing and comparing many vectors. A compact image representation still needs to preserve evidence for queries that have yet to be written. Paper, §§1–3
How SaMer works
- Merge after projection. SaMer softly assigns visual tokens to a smaller set of centroids using both embedding similarity and spatial proximity. The query tokens and MaxSim scoring stay the same.
- Use object annotations during training. An instance-consistency penalty discourages grouping tokens from different objects into one centroid. At inference, only the image features and token positions are needed.
- Train the shared projector. A contrastive retrieval objective adapts the layer that puts visual and textual features in a shared space. Both encoders remain frozen.
Method and training details in the paper
Main results
With ColPali, SaMer reduces the visual representation from 1,030 to 64 tokens per image. Estimated FP16 embedding storage for one million images drops from 263.7 GB to 16.4 GB, a 16.09× reduction. Paper, Table 8
| Benchmark | ColPali R@1 | ColPali + SaMer R@1 |
|---|---|---|
| Flickr30K | 77.0 | 82.4 |
| MSCOCO | 47.4 | 51.6 |
| ImageCoDe | 5.4 | 5.9 |
| DocVQA | 51.0 | 45.7 |
On DocVQA, the fixed token budget lowers retrieval accuracy. The paper identifies broad textual and spatial coverage as important for that benchmark. Paper, Table 1
Ablation and phrase grounding
What does object supervision add?
At 64 tokens, adding object-instance supervision to the feature-and-spatial merge improves both retrieval and grounding. BoxMass measures how much phrase relevance falls within the annotated region. Coverage IoU measures how closely the highlighted area overlaps that region. Paper, Table 4 and grounding analysis
| Merge components | Flickr30K R@1 | MSCOCO R@1 | BoxMass | Coverage IoU |
|---|---|---|---|---|
| Feature similarity | 80.7 | 49.8 | 47.6 | 13.0 |
| Feature similarity + position | 80.4 | 49.8 | 47.8 | 13.1 |
| Feature similarity + position + object prior | 82.4 | 51.6 | 54.2 | 16.4 |
Where does the token budget matter?
The 64-token setting removes 93.79% of ColPali’s visual tokens. Increasing the budget from 64 to 512 raises R@5 from 96.3 to 96.8 on Flickr30K and from 75.5 to 76.1 on MSCOCO. The DocVQA result in the main table shows a different trade-off on text-heavy images. Paper, Table 3
Resources and citation
BibTeX
@misc{park2026visualtokensmatterequally,
title = {Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval},
author = {Park, Suhyeong and Jung, Junha and Park, Jungwoo and Kang, Jaewoo},
year = {2026},
eprint = {2607.04605},
archivePrefix = {arXiv},
primaryClass = {cs.IR},
url = {https://arxiv.org/abs/2607.04605}
}