← Back to Paper Radar

PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature

Daoyu Wang, Mingyue Cheng, Qi Liu, Shuo Yu, Zirui Liu, Ze Guo
arXiv.org | 2025
PaperArena is a benchmark and standardized execution platform for testing LLM agents on research questions that require reasoning across multiple scientific papers and coordinating external tools (multimodal parsing, retrieval, computation). It shows that even the strongest current agents perform poorly on these tasks.

Problem Statement

Existing scientific-literature benchmarks mostly test tool-free question answering within a single paper. Real research instead requires synthesizing evidence across papers, parsing figures and tables, and running computations. Without a benchmark for cross-paper reasoning and multi-tool orchestration in realistic settings, progress on scientific agents is hard to measure, and their failure modes go undiagnosed.

Key Novelty

  • A benchmark of research questions that require integrating information across multiple papers, not single-document QA.
  • A standardized agent execution platform with a modular tool environment covering multimodal parsing, context retrieval, and programmatic computation, so agents can be compared under the same conditions.
  • Difficulty stratification, including a hard subset, plus reasoning-trace analysis that diagnoses how agents fail at planning and tool use.

Evaluation Highlights

  • The best-performing LLM in a well-established agentic workflow reaches only 38.78% average accuracy.
  • On the hard subset, the best accuracy falls to 18.47%, showing large headroom for cross-paper, multi-tool reasoning.

Signal Assessment

5/10 This is a solid, timely benchmark that fills a real gap (cross-paper, tool-augmented scientific reasoning) and gives a standardized platform. Benchmarks for agentic science and deep research are a crowded area, though, and the contribution is evaluative rather than methodological.

Methodology

  1. Collect scientific papers and build research questions whose answers require integrating evidence from multiple papers, often including figures, tables, or numerical computation.
  2. Provide a standardized agent platform with a modular tool suite (multimodal PDF/figure parsing, context retrieval, code execution) that agents call while planning and executing.
  3. Evaluate a range of LLMs within established agentic workflows, report accuracy overall and on a hard subset, and analyze reasoning traces to identify behavioral failure modes.

System Components

Cross-paper QA benchmark

Research questions that require multi-hop reasoning and evidence integration across several papers, stratified by difficulty, including a hard subset.

Multimodal parsing tools

Extract text, tables, and figures from scientific papers so agents can ground answers in non-textual content.

Context retrieval tools

Let agents locate relevant papers and passages within a large literature corpus.

Programmatic computation tools

Code execution environment for numerical analysis or derivations needed to answer quantitative questions.

Agent execution platform

Standardized harness where agents form reasoning plans, interact with papers, and invoke tools, enabling reproducible comparison.

Trace diagnosis analysis

Inspection of agent reasoning trajectories to find failure patterns in planning, tool selection, and evidence integration.

Results

Metric/Benchmark Baseline This Paper Delta
Average accuracy (best LLM + agentic workflow) N/A (new benchmark) 38.78% ~61% headroom to perfect
Hard-subset accuracy (best LLM + agentic workflow) N/A (new benchmark) 18.47% ~20-point drop vs. average
Task scope vs. prior benchmarks Single-paper, tool-free QA Cross-paper, multi-tool agentic reasoning Qualitatively harder, more realistic setting

Key Takeaways

  • Current frontier LLM agents are far from reliable research assistants. Expect sub-40% accuracy on realistic cross-paper questions, and under 20% on hard ones, so plan human verification accordingly.
  • Orchestrating multiple tools and planning over many documents is a core bottleneck, not just model knowledge. Investing in planning, tool selection, and evidence-aggregation strategies may pay off more than swapping base models.
  • PaperArena's standardized tool environment offers a reproducible testbed for comparing scientific-agent designs (RAG pipelines, multimodal parsers, code tools). Use its trace diagnostics to target specific failure modes.

Abstract

Understanding and reasoning on the large-scale scientific literature is a crucial touchstone for large language model (LLM) based agents. However, existing works are mainly restricted to tool-free tasks within single papers, largely due to the lack of a benchmark that evaluates cross-paper reasoning and multi-tool orchestration in authentic research scenarios. In this work, we propose PaperArena, a benchmark to evaluate LLM-based agents on questions that require integrating information across multiple papers with the assistance of external tools. Given a research question, agents should formulate a reasoning plan, interact with multiple papers, and invoke appropriate tools to produce a well-grounded answer. To support standardized evaluation, we provide a platform for agent execution, offering a modular tool environment including multimodal parsing, context retrieval, and programmatic computation. Experiments reveal that even the leading LLM powering a well-established agentic workflow achieves merely 38.78% average accuracy, while on the hard subset, accuracy drops to only 18.47%. We also analyze reasoning traces and diagnose agent behavior, providing the community with insights to develop and evaluate more capable scientific agents.

Generated automatically from available metadata and abstract on 2026-11-09.