← Back to Paper Radar

Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval

G. Moreira, Ronay Ak, Mengyao Xu, Oliver Holworthy, Benedikt Schifferer, Zhiding Yu, Yauhen Babakhin, Radek Osmulski, Jiarui Cai, Ryan Chesler, Bo Liu, Even Oldridge
arXiv.org | 2026
Nemotron ColEmbed V2 is a family of VLM-based late-interaction (ColBERT-style) embedding models at 3B, 4B, and 8B parameters for visual document retrieval. It reaches state-of-the-art results on the ViDoRe benchmarks by combining careful data curation, hard-negative mining, bidirectional attention, and model merging.

Problem Statement

Enterprise RAG pipelines must retrieve from large collections of visually rich documents such as PDFs and slides. OCR-based text extraction there is lossy and adds pipeline complexity. VLM-based embedding models keep layout and visual information, but getting top accuracy requires many training-recipe choices, and late interaction brings large compute and storage costs that limit production deployment.

Key Novelty

  • A family of late-interaction visual document retrievers built on three VLM backbones (Eagle 2 with Llama 3.2 3B, Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct); the 8B model ranks #1 on the ViDoRe V3 leaderboard as of Feb 3, 2026.
  • A training recipe that combines cluster-based data sampling, hard-negative mining, converting causal VLM decoders to bidirectional attention, and post-training model merging.
  • An engineering analysis of late-interaction storage and compute costs, with experiments on reduced-dimension multi-vector embeddings that trade accuracy against index size.

Evaluation Highlights

  • The 8B variant achieves an average NDCG@10 of 63.42 on ViDoRe V3, ranking first on the leaderboard as of Feb 3, 2026.
  • The authors report state-of-the-art performance across ViDoRe benchmarks, plus accuracy-vs-storage trade-off experiments using lower-dimensional embeddings. The abstract does not give specific numbers for the 3B/4B variants, the baselines, or the size of the improvement.

Signal Assessment

5/10 This is a strong engineering and empirical contribution with leaderboard-topping, openly released models and practical deployment insights. However, the techniques (late interaction, hard-negative mining, bidirectional attention, model merging) are known components combined well, not a new paradigm.

Methodology

  1. Start from pre-trained VLMs (Eagle 2/Llama 3.2 3B, Qwen3-VL 4B/8B) and switch the language decoder from causal to bidirectional attention to produce better token-level representations.
  2. Curate training data with cluster-based sampling for diversity and balance, and mine hard negatives to sharpen contrastive learning.
  3. Train with a ColBERT-style late-interaction (MaxSim) objective, producing multi-vector embeddings for query tokens and document image patches/tokens.
  4. Post-train by merging multiple checkpoints or model variants to improve robustness and generalization.
  5. Study embedding dimensionality reduction to balance retrieval accuracy against index storage and compute cost.

System Components

VLM backbone

Pre-trained vision-language model (Eagle 2 with Llama 3.2 3B, or Qwen3-VL-4B/8B-Instruct) that encodes document page images and text queries into token-level hidden states.

Bidirectional attention

Replaces causal masking so every token attends to the full context, giving richer contextualized embeddings for retrieval.

Late interaction (MaxSim) scoring

Keeps one embedding per token; relevance is the sum, over query tokens, of each token's maximum similarity to any document token. This enables fine-grained matching.

Cluster-based sampling

Groups training data into clusters and samples across them to keep diversity and avoid domain over-representation.

Hard-negative mining

Picks challenging non-relevant documents (presumably with a teacher or retriever model) as contrastive negatives to improve discrimination.

Model merging

Combines weights from multiple trained models or checkpoints after training to improve generalization without extra inference cost.

Low-dimensional embedding projection

Reduces per-token embedding dimension to cut the large storage footprint of multi-vector indexes, with measured accuracy trade-offs.

Results

Metric/Benchmark Baseline This Paper Delta
ViDoRe V3 avg NDCG@10 (8B) Prior leaderboard entries (not specified in abstract) 63.42 Ranked #1 as of Feb 3, 2026
ViDoRe V3 (4B, Qwen3-VL) Prior models of similar size (not specified) Competitive / strong (not specified) Not specified in abstract
ViDoRe V3 (3B, Eagle 2/Llama 3.2) Prior models of similar size (not specified) Competitive / strong (not specified) Not specified in abstract
Storage vs. accuracy (reduced dims) Full-dimension multi-vector embeddings Lower-dim embeddings with modest accuracy loss Large storage reduction (qualitative)

Key Takeaways

  • For RAG over PDFs and slides, VLM-based late-interaction retrievers can replace OCR pipelines, simplifying indexing while preserving visual and layout signals. Checkpoints at 3B, 4B, and 8B let you choose a latency/accuracy point.
  • Converting a VLM decoder to bidirectional attention, then adding hard-negative mining and model merging, is a reusable recipe for turning generative VLMs into strong retrievers.
  • Multi-vector indexes are expensive to store and serve. Plan for dimension reduction (and possibly token pooling or quantization), and benchmark the accuracy loss on your own corpus before production rollout.
  • Data curation (cluster-based sampling for diversity) matters as much as model scale for top benchmark performance; apply similar sampling when fine-tuning domain-specific retrievers.

Abstract

Retrieval-Augmented Generation (RAG) systems have been popular for generative applications, powering language models by injecting external knowledge. Companies have been trying to leverage their large catalog of documents (e.g. PDFs, presentation slides) in such RAG pipelines, whose first step is the retrieval component. Dense retrieval has been a popular approach, where embedding models are used to generate a dense representation of the user query that is closer to relevant content embeddings. More recently, VLM-based embedding models have become popular for visual document retrieval, as they preserve visual information and simplify the indexing pipeline compared to OCR text extraction. Motivated by the growing demand for visual document retrieval, we introduce Nemotron ColEmbed V2, a family of models that achieve state-of-the-art performance on the ViDoRe benchmarks. We release three variants - with 3B, 4B, and 8B parameters - based on pre-trained VLMs: NVIDIA Eagle 2 with Llama 3.2 3B backbone, Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct, respectively. The 8B model ranks first on the ViDoRe V3 leaderboard as of February 03, 2026, achieving an average NDCG@10 of 63.42. We describe the main techniques used across data processing, training, and post-training - such as cluster-based sampling, hard-negative mining, bidirectional attention, late interaction, and model merging - that helped us build our top-performing models. We also discuss compute and storage engineering challenges posed by the late interaction mechanism and present experiments on how to balance accuracy and storage with lower dimension embeddings.

Generated automatically from available metadata and abstract on 2026-11-09.