← Back to Paper Radar

Panini: Continual Learning in Token Space via Structured Memory

Shreyas Rajesh, Pavan Holur, M. Y. Turali, Chenda Duan, V. Roychowdhury
2026
Panini treats continual learning as a non-parametric memory problem. The base LLM stays frozen, and each new document is integrated at write time into a structured, entity- and event-aware network of QA pairs called a Generative Semantic Workspace (GSW). At query time the system retrieves reasoning-grounded inference chains from that network instead of raw text chunks.

Problem Statement

Standard RAG stores verbatim chunks and makes the LLM re-reason over the same raw documents on every query, which wastes test-time compute. Chunk retrieval also injects irrelevant context that increases unsupported or hallucinated answers. Parametric alternatives such as fine-tuning on new knowledge are costly and prone to forgetting, so a frozen-model approach that consolidates knowledge reliably and efficiently is needed.

Key Novelty

  • Frames continual learning as accumulation and consolidation in token space: the frozen LLM's knowledge is updated by evolving an external semantic memory state, not its weights.
  • Introduces Generative Semantic Workspaces (GSW), an entity- and event-aware network of QA pairs. The network is meant to be sufficient for an LLM to reconstruct experienced situations and to surface latent knowledge.
  • A retrieval procedure that traverses only the continually updated GSW, never the verbatim documents, and returns the most likely reasoning-grounded inference chains. This shifts reasoning cost from read time to write time.

Evaluation Highlights

  • Highest average performance across six QA benchmarks, 5–7% above competitive baselines (the comparators are not named in the abstract but plausibly include chunk RAG and graph/structured RAG variants).
  • Uses 2–30x fewer answer-context tokens than baselines.
  • Fewer unsupported answers on curated unanswerable queries.
  • Supports fully open-source pipelines.

Signal Assessment

6/10 Write-time structuring into QA-pair networks with inference-chain retrieval is a well-motivated, practical step beyond chunk-RAG and existing graph-RAG approaches, with clear efficiency and reliability gains. It extends the established structured-memory/GraphRAG line rather than shifting the paradigm, and the 5–7% average gains are solid but not transformative.

Methodology

  1. Write time: each incoming document is processed by an LLM into a GSW representation, extracting entities, events, and their relations as atomic question-answer pairs.
  2. Consolidation: the new GSW fragment is merged into the persistent global memory network. Entities and events are linked across documents so the memory accumulates and self-consolidates continually.
  3. Read time: given a query, the system traverses the GSW network to build candidate reasoning-grounded inference chains (multi-hop paths of QA pairs).
  4. The system ranks the chains and retrieves the most likely ones, passing only this compact chain context to a frozen LLM to generate the answer, or to abstain when support is lacking.

System Components

Generative Semantic Workspace (GSW)

Structured memory representation: an entity- and event-centric network of QA pairs that encodes the situations described in documents in compact, reasoning-ready form.

Write-time Memory Builder

LLM-driven pipeline that converts each new document into GSW QA pairs and entity/event nodes, front-loading comprehension cost once per document.

Continual Consolidation Module

Integrates new experiences into the existing memory state, linking and reconciling entities and events across documents so knowledge accumulates without changing model weights.

Inference-Chain Retriever

Traverses the GSW network for a query and selects the most likely multi-hop chains of QA pairs, replacing verbatim chunk retrieval.

Frozen Reader LLM

Fixed base model (can be open-source) that reasons over the compact retrieved chains to produce answers, with fewer unsupported generations.

Results

Metric/Benchmark Baseline This Paper Delta
Average QA performance (6 benchmarks) Competitive RAG / structured-memory baselines Highest average +5% to +7%
Answer-context tokens Chunk-based / baseline retrieval contexts Compact inference chains 2–30x fewer tokens
Unsupported answers on curated unanswerable queries Higher rate (chunk retrieval injects irrelevant context) Reduced Qualitative reduction (exact numbers not in abstract)
Pipeline openness Often relies on proprietary LLMs Fully open-source pipeline supported Qualitative

Key Takeaways

  • Moving reasoning to write time, by structuring documents into QA-pair networks once, can cut read-time token costs substantially (2–30x). This matters for high-query-volume RAG systems where the same corpus is queried repeatedly.
  • Retrieving compact, reasoning-grounded chains instead of raw chunks improves accuracy and lowers hallucination on unanswerable queries. Context quality matters more than context quantity.
  • Agent memory systems can do continual learning without fine-tuning by keeping a consolidating external semantic memory. This avoids catastrophic forgetting and suits evolving or user-specific knowledge, though practitioners should weigh the upfront ingestion cost and possible extraction errors in the write-time LLM pipeline.

Abstract

Language models are increasingly used to reason over content they were not trained on, such as new documents, evolving knowledge, and user-specific data. A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time for an LLM to reason over. However, this results in inefficient usage of test-time compute (LLM repeatedly reasons over the same documents); moreover, chunk retrieval can inject irrelevant context that increases unsupported generation. We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state that accumulates and consolidates itself continually. We present Panini, which realizes this by representing documents as Generative Semantic Workspaces (GSW) -- an entity- and event-aware network of question-answer (QA) pairs, sufficient for an LLM to reconstruct the experienced situations and mine latent knowledge via reasoning-grounded inference chains on the network. Given a query, Panini only traverses the continually-updated GSW (not the verbatim documents or chunks), and retrieves the most likely inference chains. Across six QA benchmarks, Panini achieves the highest average performance, 5%-7% higher than other competitive baselines, while using 2-30x fewer answer-context tokens, supports fully open-source pipelines, and reduces unsupported answers on curated unanswerable queries. The results show that efficient and accurate structuring of experiences at write time -- as achieved by the GSW framework -- yields both efficiency and reliability gains at read time. Code is available at https://github.com/roychowdhuryresearch/gsw-memory.

Generated automatically from available metadata and abstract on 2026-11-09.