Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval
Problem Statement
Enterprise RAG pipelines must retrieve from large collections of visually rich documents such as PDFs and slides. OCR-based text extraction there is lossy and adds pipeline complexity. VLM-based embedding models keep layout and visual information, but getting top accuracy requires many training-recipe choices, and late interaction brings large compute and storage costs that limit production deployment.
Key Novelty
- A family of late-interaction visual document retrievers built on three VLM backbones (Eagle 2 with Llama 3.2 3B, Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct); the 8B model ranks #1 on the ViDoRe V3 leaderboard as of Feb 3, 2026.
- A training recipe that combines cluster-based data sampling, hard-negative mining, converting causal VLM decoders to bidirectional attention, and post-training model merging.
- An engineering analysis of late-interaction storage and compute costs, with experiments on reduced-dimension multi-vector embeddings that trade accuracy against index size.
Evaluation Highlights
- The 8B variant achieves an average NDCG@10 of 63.42 on ViDoRe V3, ranking first on the leaderboard as of Feb 3, 2026.
- The authors report state-of-the-art performance across ViDoRe benchmarks, plus accuracy-vs-storage trade-off experiments using lower-dimensional embeddings. The abstract does not give specific numbers for the 3B/4B variants, the baselines, or the size of the improvement.
Signal Assessment
Methodology
- Start from pre-trained VLMs (Eagle 2/Llama 3.2 3B, Qwen3-VL 4B/8B) and switch the language decoder from causal to bidirectional attention to produce better token-level representations.
- Curate training data with cluster-based sampling for diversity and balance, and mine hard negatives to sharpen contrastive learning.
- Train with a ColBERT-style late-interaction (MaxSim) objective, producing multi-vector embeddings for query tokens and document image patches/tokens.
- Post-train by merging multiple checkpoints or model variants to improve robustness and generalization.
- Study embedding dimensionality reduction to balance retrieval accuracy against index storage and compute cost.
System Components
Pre-trained vision-language model (Eagle 2 with Llama 3.2 3B, or Qwen3-VL-4B/8B-Instruct) that encodes document page images and text queries into token-level hidden states.
Replaces causal masking so every token attends to the full context, giving richer contextualized embeddings for retrieval.
Keeps one embedding per token; relevance is the sum, over query tokens, of each token's maximum similarity to any document token. This enables fine-grained matching.
Groups training data into clusters and samples across them to keep diversity and avoid domain over-representation.
Picks challenging non-relevant documents (presumably with a teacher or retriever model) as contrastive negatives to improve discrimination.
Combines weights from multiple trained models or checkpoints after training to improve generalization without extra inference cost.
Reduces per-token embedding dimension to cut the large storage footprint of multi-vector indexes, with measured accuracy trade-offs.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| ViDoRe V3 avg NDCG@10 (8B) | Prior leaderboard entries (not specified in abstract) | 63.42 | Ranked #1 as of Feb 3, 2026 |
| ViDoRe V3 (4B, Qwen3-VL) | Prior models of similar size (not specified) | Competitive / strong (not specified) | Not specified in abstract |
| ViDoRe V3 (3B, Eagle 2/Llama 3.2) | Prior models of similar size (not specified) | Competitive / strong (not specified) | Not specified in abstract |
| Storage vs. accuracy (reduced dims) | Full-dimension multi-vector embeddings | Lower-dim embeddings with modest accuracy loss | Large storage reduction (qualitative) |
Key Takeaways
- For RAG over PDFs and slides, VLM-based late-interaction retrievers can replace OCR pipelines, simplifying indexing while preserving visual and layout signals. Checkpoints at 3B, 4B, and 8B let you choose a latency/accuracy point.
- Converting a VLM decoder to bidirectional attention, then adding hard-negative mining and model merging, is a reusable recipe for turning generative VLMs into strong retrievers.
- Multi-vector indexes are expensive to store and serve. Plan for dimension reduction (and possibly token pooling or quantization), and benchmark the accuracy loss on your own corpus before production rollout.
- Data curation (cluster-based sampling for diversity) matters as much as model scale for top benchmark performance; apply similar sampling when fine-tuning domain-specific retrievers.
Abstract
Retrieval-Augmented Generation (RAG) systems have been popular for generative applications, powering language models by injecting external knowledge. Companies have been trying to leverage their large catalog of documents (e.g. PDFs, presentation slides) in such RAG pipelines, whose first step is the retrieval component. Dense retrieval has been a popular approach, where embedding models are used to generate a dense representation of the user query that is closer to relevant content embeddings. More recently, VLM-based embedding models have become popular for visual document retrieval, as they preserve visual information and simplify the indexing pipeline compared to OCR text extraction. Motivated by the growing demand for visual document retrieval, we introduce Nemotron ColEmbed V2, a family of models that achieve state-of-the-art performance on the ViDoRe benchmarks. We release three variants - with 3B, 4B, and 8B parameters - based on pre-trained VLMs: NVIDIA Eagle 2 with Llama 3.2 3B backbone, Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct, respectively. The 8B model ranks first on the ViDoRe V3 leaderboard as of February 03, 2026, achieving an average NDCG@10 of 63.42. We describe the main techniques used across data processing, training, and post-training - such as cluster-based sampling, hard-negative mining, bidirectional attention, late interaction, and model merging - that helped us build our top-performing models. We also discuss compute and storage engineering challenges posed by the late interaction mechanism and present experiments on how to balance accuracy and storage with lower dimension embeddings.