PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature
Problem Statement
Existing scientific-literature benchmarks mostly test tool-free question answering within a single paper. Real research instead requires synthesizing evidence across papers, parsing figures and tables, and running computations. Without a benchmark for cross-paper reasoning and multi-tool orchestration in realistic settings, progress on scientific agents is hard to measure, and their failure modes go undiagnosed.
Key Novelty
- A benchmark of research questions that require integrating information across multiple papers, not single-document QA.
- A standardized agent execution platform with a modular tool environment covering multimodal parsing, context retrieval, and programmatic computation, so agents can be compared under the same conditions.
- Difficulty stratification, including a hard subset, plus reasoning-trace analysis that diagnoses how agents fail at planning and tool use.
Evaluation Highlights
- The best-performing LLM in a well-established agentic workflow reaches only 38.78% average accuracy.
- On the hard subset, the best accuracy falls to 18.47%, showing large headroom for cross-paper, multi-tool reasoning.
Signal Assessment
Methodology
- Collect scientific papers and build research questions whose answers require integrating evidence from multiple papers, often including figures, tables, or numerical computation.
- Provide a standardized agent platform with a modular tool suite (multimodal PDF/figure parsing, context retrieval, code execution) that agents call while planning and executing.
- Evaluate a range of LLMs within established agentic workflows, report accuracy overall and on a hard subset, and analyze reasoning traces to identify behavioral failure modes.
System Components
Research questions that require multi-hop reasoning and evidence integration across several papers, stratified by difficulty, including a hard subset.
Extract text, tables, and figures from scientific papers so agents can ground answers in non-textual content.
Let agents locate relevant papers and passages within a large literature corpus.
Code execution environment for numerical analysis or derivations needed to answer quantitative questions.
Standardized harness where agents form reasoning plans, interact with papers, and invoke tools, enabling reproducible comparison.
Inspection of agent reasoning trajectories to find failure patterns in planning, tool selection, and evidence integration.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Average accuracy (best LLM + agentic workflow) | N/A (new benchmark) | 38.78% | ~61% headroom to perfect |
| Hard-subset accuracy (best LLM + agentic workflow) | N/A (new benchmark) | 18.47% | ~20-point drop vs. average |
| Task scope vs. prior benchmarks | Single-paper, tool-free QA | Cross-paper, multi-tool agentic reasoning | Qualitatively harder, more realistic setting |
Key Takeaways
- Current frontier LLM agents are far from reliable research assistants. Expect sub-40% accuracy on realistic cross-paper questions, and under 20% on hard ones, so plan human verification accordingly.
- Orchestrating multiple tools and planning over many documents is a core bottleneck, not just model knowledge. Investing in planning, tool selection, and evidence-aggregation strategies may pay off more than swapping base models.
- PaperArena's standardized tool environment offers a reproducible testbed for comparing scientific-agent designs (RAG pipelines, multimodal parsers, code tools). Use its trace diagnostics to target specific failure modes.
Abstract
Understanding and reasoning on the large-scale scientific literature is a crucial touchstone for large language model (LLM) based agents. However, existing works are mainly restricted to tool-free tasks within single papers, largely due to the lack of a benchmark that evaluates cross-paper reasoning and multi-tool orchestration in authentic research scenarios. In this work, we propose PaperArena, a benchmark to evaluate LLM-based agents on questions that require integrating information across multiple papers with the assistance of external tools. Given a research question, agents should formulate a reasoning plan, interact with multiple papers, and invoke appropriate tools to produce a well-grounded answer. To support standardized evaluation, we provide a platform for agent execution, offering a modular tool environment including multimodal parsing, context retrieval, and programmatic computation. Experiments reveal that even the leading LLM powering a well-established agentic workflow achieves merely 38.78% average accuracy, while on the hard subset, accuracy drops to only 18.47%. We also analyze reasoning traces and diagnose agent behavior, providing the community with insights to develop and evaluate more capable scientific agents.