← Back to Paper Radar

Synthesizing question answering data from financial documents: An End-to-End Multi-Agent Approach

Chetan Harsha, Karmvir Singh Phogat, Sridhar Dasaratha, Shashishekar Ramakrishna
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track) | 2026
A modular, end-to-end multi-agent pipeline can turn unstructured financial documents into synthetic numerical-reasoning QA pairs. Small language models fine-tuned on this data match manually annotated data in-distribution (for at least one model) and generalize better out-of-distribution.

Problem Statement

Numerical reasoning over financial documents is hard because the relevant facts are scattered across text and tables. LLMs handle it well but are often too costly and slow for enterprise deployment. SLMs are a cheaper option, but they need high-quality domain-specific QA data for fine-tuning, and producing that data today requires expert annotation, which limits adoption.

Key Novelty

  • An end-to-end agentic pipeline that goes from raw, unstructured financial documents to fine-tuning-ready QA pairs, without relying on pre-curated tables or passages.
  • A modular multi-agent design that separates content extraction and relevance selection from QA generation, so each stage can be scaled or swapped independently.
  • Evidence that synthetic, pipeline-generated data can substitute for manually curated expert datasets (e.g., FinQA-style data) when fine-tuning SLMs, with gains in generalization.

Evaluation Highlights

  • In-distribution: one of the SLMs fine-tuned on the synthetic data was competitive with the same model fine-tuned on prior manually annotated data. The wording implies the other models fell short of the manual-data baseline in-distribution.
  • Out-of-distribution: every tested SLM generalized better when trained on the pipeline's synthetic data than when trained on the manually created data.

Signal Assessment

5/10 This is a solid industry-track contribution that combines known ideas (agentic synthetic data generation, distilling into SLMs) into a practical financial QA pipeline. The generalization gains are useful, but the in-distribution results are mixed and the approach is incremental rather than a fundamental advance.

Methodology

  1. Ingest unstructured financial documents (e.g., annual reports and filings) and use extraction agents to parse text and tabular content.
  2. Apply selection agents to identify content that is relevant and information-dense enough to support multi-step numerical reasoning questions.
  3. Use generation agents (LLM-based) to produce question-answer pairs, likely with reasoning steps or programs, grounded in the selected content.
  4. Fine-tune several SLMs on the synthetic QA data and compare them with the same models fine-tuned on existing manually annotated datasets.
  5. Evaluate on in-distribution test sets (matching the manual dataset) and on out-of-distribution financial QA benchmarks to measure generalization.

System Components

Document Extraction Agent

Parses raw, unstructured financial documents into usable text and table segments.

Content Selection Agent

Filters and ranks extracted segments to find content suitable for complex, numerically grounded questions.

QA Generation Agent

Generates question-answer pairs, potentially with reasoning chains, from the selected content to serve as SLM fine-tuning data.

Orchestration Layer

Coordinates the agents in a modular, scalable end-to-end workflow so stages can be replaced or scaled independently.

SLM Fine-tuning & Evaluation

Trains small language models on the synthetic data and benchmarks them against models trained on manual annotations.

Results

Metric/Benchmark Baseline This Paper Delta
In-distribution accuracy (best SLM) SLM fine-tuned on manually annotated data SLM fine-tuned on synthetic pipeline data Competitive (comparable performance)
In-distribution accuracy (other SLMs) SLMs fine-tuned on manually annotated data SLMs fine-tuned on synthetic data Not reported in abstract; wording implies lower
Out-of-distribution generalization (all SLMs) SLMs fine-tuned on manually annotated data SLMs fine-tuned on synthetic data Better for all tested models (exact margin not reported)
Manual annotation effort Expert annotation required Automated agentic generation Substantially reduced (qualitative)

Key Takeaways

  • For domain-specific numerical reasoning, agentic synthetic data generation is a viable way to fine-tune cheap SLMs, which reduces reliance on costly expert annotation.
  • Synthetic data drawn from diverse raw documents can generalize better than narrow manual datasets, so it is worth evaluating out-of-distribution rather than only on in-distribution benchmarks.
  • Keeping extraction, selection and generation as separate agents makes the pipeline easier to debug, scale, and adapt to other document-heavy enterprise domains such as legal, insurance or healthcare.
  • In-distribution parity is model-dependent, so practitioners should test several SLM backbones and consider mixing synthetic data with a small amount of manual data where in-distribution accuracy matters.

Abstract

Answering complex questions that require numerical reasoning over financial documents is challenging due to the diverse and scattered nature of relevant information. While large language models (LLMs) excel at financial reasoning, their enterprise deployment is often limited by cost and latency. Small language models (SLMs) present a cost-effective alternative but need to be fine-tuned with high-quality, domain-specific question-answer (QA) data. Acquiring such data requires manual expert annotation, presenting a bottleneck to the wider application of SLMs. This work introduces a modular, scalable end-to-end agentic pipeline that extracts and selects relevant content from unstructured financial documents and then generates QA pairs from the selected content for SLM fine-tuning. Compared to the same models trained on previous manually generated data for the task, one of the models trained on our pipeline-produced synthetic data achieved competitive in-distribution performance, and all tested models demonstrated superior generalization. The framework thus demonstrates considerable potential to accelerate the deployment of smaller, cost-effective models by reducing manual data creation efforts.

Generated automatically from available metadata and abstract on 2026-11-09.