← Back to Paper Radar

Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling

Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, Estevam Hruschka
arXiv.org | 2025
FuncBenchGen recasts multi-step tool use as traversal over a hidden function-dependency DAG. Because tasks are generated synthetically, the benchmark has no contamination and lets researchers set difficulty precisely. Evaluations with it show that LLMs struggle mainly with dependency depth, connected distractors and tracking state across turns.

Problem Statement

Existing benchmarks for tool-augmented LMs use fixed, often publicly available task sets. That makes them vulnerable to pretraining and test-time leakage, and it prevents systematic control over task difficulty. As a result, it is hard to tell whether models are actually reasoning through multi-step function calls or recalling memorized patterns, and hard to pinpoint which structural factors cause failures.

Key Novelty

  • A contamination-free generative benchmark (FuncBenchGen) that builds synthetic multi-step function-calling tasks from a hidden function-dependency DAG; the model must infer the call sequence needed to compute a target value.
  • Explicit difficulty knobs, including graph size, dependency depth, and distractor functions. Distractors come in two kinds: connected (type-compatible with relevant functions) and unconnected.
  • A failure analysis showing that strong models produce syntactically valid calls but propagate stale or incorrect argument values, plus a simple mitigation that restates prior variable values at every step.

Evaluation Highlights

  • Reasoning-optimized models consistently outperform general-purpose models, and GPT-5 significantly outperforms all other models evaluated. Performance drops sharply as dependency depth increases.
  • Connected distractors are especially harmful. Explicitly restating variable values raises GPT-5's success rate from 62.5% to 81.3%, with substantial gains for other models as well.

Signal Assessment

6/10 This is a solid, methodologically useful contribution: a controllable, contamination-resistant benchmark generator with insightful diagnostics and a cheap, effective mitigation. Synthetic DAG-based task generation is a known idea, though, and the scope is evaluation rather than a new modeling capability.

Methodology

  1. Generate a random function-dependency DAG in which each function's inputs are typed variables produced by other functions, and designate a target value to compute.
  2. Expose a subset of functions (relevant ones plus connected or unconnected distractors) to the TaLM as tools, without revealing the graph structure.
  3. Control difficulty by varying graph size, dependency depth, and the number and type of distractors.
  4. Run models in a multi-turn function-calling loop and measure success at computing the correct target value.
  5. Analyze failure modes such as invalid calls, wrong arguments and stale values, then test a state-restatement intervention that re-presents known variable values at each step.

System Components

Hidden Function-Dependency DAG

Latent graph structure defining which functions produce the inputs required by others. It determines the correct call sequence to reach the target value.

Synthetic Task Generator

Procedurally creates novel functions, variables and values, so tasks cannot appear in pretraining data and can be regenerated freshly to prevent test-time leakage.

Difficulty Controls

Parameters for graph size, dependency depth and distractor configuration that allow systematic stress-testing.

Connected vs. Unconnected Distractors

Irrelevant functions that either share type-compatible variables with relevant functions (connected, hard to rule out) or are fully disjoint (unconnected).

Variable-Value Restatement Mitigation

A lightweight prompting intervention that explicitly restates previously obtained variable values to the agent at each step, reducing state-tracking errors.

Results

Metric/Benchmark Baseline This Paper Delta
GPT-5 success rate (with vs. without variable restatement) 62.5% 81.3% +18.8 pts
Other models' success rate with restatement Standard multi-turn prompting Restated variable values Substantial gains (qualitative)
Reasoning-optimized vs. general-purpose models General-purpose models Reasoning-optimized models Consistently higher success
Effect of increasing dependency depth Shallow DAGs Deep DAGs Sharp performance decline
Connected vs. unconnected distractors Unconnected distractors Connected distractors Notably harder for models

Key Takeaways

  • In multi-step agent pipelines, explicitly restate or maintain a structured record of intermediate variable values in the context at each step. It is a near-free change that substantially improves reliability, even for frontier models like GPT-5.
  • Tool catalogs containing type-compatible but irrelevant functions are a major failure source. Tool retrieval and filtering that prunes semantically or type-similar distractors can meaningfully improve agent accuracy.
  • Use procedurally generated, parameterized benchmarks like FuncBenchGen to evaluate agents without contamination. Sweeping dependency depth separately from tool-set size helps locate where a model's planning and state tracking break down.

Abstract

Existing benchmarks for tool-augmented language models (TaLMs) lack fine-grained control over task difficulty and remain vulnerable to data contamination. We present FuncBenchGen, a unified, contamination-free framework that evaluates TaLMs by generating synthetic multi-step tool-use tasks to stress-test TaLMs. The key idea is to cast tool use as traversal over a hidden function-dependency DAG where models must infer the correct sequence of calls to compute a target value. FuncBenchGen allows precise control over task difficulty (e.g., graph size, dependency depth, and distractor functions) while avoiding pretraining/test-time leakage. Our evaluation demonstrates reasoning-optimized models consistently outperform general-purpose models with GPT-5 significantly outperforming other available models. Performance declines sharply as dependency depth increases. Furthermore, connected distractors -- irrelevant functions sharing type-compatible variables with relevant functions -- prove especially difficult to handle. Also, strong models often make syntactically valid function calls but propagate incorrect or stale argument values across steps, revealing brittle state tracking by LLMs in multi-turn tool use. Motivated by this observation, we introduce a simple mitigation strategy that explicitly restates prior variable values to the agent at each step. Surprisingly, this lightweight change yields substantial gains across models. e.g., yielding an improvement in success rate from 62.5% to 81.3% for GPT-5.

Generated automatically from available metadata and abstract on 2026-11-09.