Co-LMLM: Continuous-Query Limited Memory Language Models
Problem Statement
Conventional LLMs store factual knowledge inside their weights, making it costly to update, hard to control, and impossible to attribute. The recently introduced LMLM paradigm externalizes knowledge to a KB but has so far depended on relational KBs, rigid symbolic queries, and Wikipedia-only annotation, which caps flexibility and scale. Co-LMLM removes these constraints by enabling continuous vector-based querying and generalizing knowledge annotation to arbitrary text corpora.
Key Novelty
- Continuous-key KB architecture that pairs dense vector keys with textual knowledge values, departing from prior relational KB and symbolic query designs in LMLMs
- Flexible, low-cost vector query generation at inference time while preserving human-readable, attributable retrieved knowledge
- A free-form annotation pipeline that tags factual spans in arbitrary text, lifting the prior restriction of LMLM training data to Wikipedia and enabling pretraining on corpora like FineWeb-Edu
- Demonstrated scaling behavior showing gains hold across model sizes and pretraining corpora, not just a single controlled setting
Evaluation Highlights
- Perplexity across Wikipedia and FineWeb-Edu pretraining at multiple model scales, outperforming both prior LMLMs and vanilla LLM baselines
- At 360M parameters, achieves lower perplexity than models trained on roughly 40x more data
- SimpleQA-verified factual precision at 360M scale is on par with GPT-4o-mini and exceeds Claude Sonnet 4.5
Signal Assessment
Methodology
- Replace relational/symbolic KB entries with a KB storing textual knowledge values indexed by continuous vector keys
- Train the language model to emit flexible continuous query vectors during generation for efficient KB lookup
- Build an annotation pipeline that automatically tags free-form factual spans in arbitrary text to construct training data beyond Wikipedia
- Pretrain models at multiple scales on Wikipedia and FineWeb-Edu, fetching and integrating retrieved textual knowledge into generation while retaining attribution
- Compare against prior LMLMs and vanilla LLMs using perplexity and SimpleQA-based factual precision, including against much larger and higher-data-volume models
System Components
Stores factual knowledge as human-readable text values paired with continuous vector keys, replacing the relational/symbolic KB structure used in prior LMLMs.
Model module that produces flexible continuous query vectors during generation to retrieve relevant KB entries at minimal computational cost.
Automatically tags factual spans in arbitrary text (not just Wikipedia) to create training signal for knowledge externalization on diverse corpora such as FineWeb-Edu.
Fetches KB knowledge during generation and weaves it into the model's output while keeping retrieved content human-readable and attributable.
Results
| Metric/Benchmark | Baseline | Co-LMLM | Delta |
|---|---|---|---|
| Perplexity (Wikipedia & FineWeb-Edu, multi-scale) | Prior LMLMs / vanilla LLMs | Lower perplexity across scales | Outperforms both baseline types |
| Perplexity vs. data volume (360M scale) | LLMs pretrained on ~40x more data | Co-LMLM (360M) | Lower perplexity despite far less training data |
| SimpleQA factual precision (360M scale) | GPT-4o-mini | Co-LMLM (360M) | Roughly on par |
| SimpleQA factual precision (360M scale) | Claude Sonnet 4.5 | Co-LMLM (360M) | Co-LMLM higher |
Key Takeaways
- Small models (360M params) with externalized knowledge can match or exceed much larger frontier LLMs on factual QA precision, pointing to knowledge externalization as a path toward compute-efficient, factually grounded models
- Continuous vector queries/keys can replace relational/symbolic KBs in memory-augmented LMs without sacrificing retrieval interpretability or attribution, useful for building auditable and updatable knowledge systems
- Automated factual-span annotation generalizes knowledge-externalization training beyond Wikipedia to broad web corpora (e.g., FineWeb-Edu), broadening applicability to real-world pretraining pipelines
- Knowledge externalization appears to substantially improve data efficiency, suggesting reduced pretraining data/compute needs for comparable or better perplexity and factuality
Abstract
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.