← Back to Paper Radar

Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents

Zeping Li, Hongru Wang, Yiwen Zhao, Guanhua Chen, Yixia Li, Keyang Chen, Yixin Cao, Guangnan Ye, Hongfeng Chai, Zhenfei Yin
arXiv.org | 2026
When an LLM agent makes a high-quality tool call, its uncertainty (entropy) tends to drop afterwards. The paper uses this entropy reduction as a supervisory signal in RL, with a sparse outcome-level reward that improves tool-call efficiency and a dense process-level reward that improves task performance.

Problem Statement

Over long trajectories, tool-using LLM agents often make excessive, redundant, or low-quality tool calls. This raises latency and cost and can hurt final answer quality. Existing RL rewards are mostly outcome-correctness signals that say little about whether a given tool call was useful, so tool-use behavior is hard to control without costly step-level annotation.

Key Novelty

  • Pilot experiments showing a strong positive correlation between post-call entropy reduction and the quality of a tool call, which yields an intrinsic, annotation-free signal of tool-call utility.
  • A sparse, trajectory-level outcome reward built on entropy reduction, aimed at efficiency by discouraging tool calls that do not reduce uncertainty.
  • A dense, step-level process reward built on per-call entropy reduction, giving fine-grained credit assignment to individual tool calls to improve task performance.
  • A reframing of entropy from an exploration-regularization quantity to a measure of the information gained from tool interactions.

Evaluation Highlights

  • The sparse outcome reward reduces tool calls by 72.07% relative to the average of the baselines.
  • The dense process reward improves task performance by 22.27%, likely relative to the baseline average (the abstract does not state the comparison point).
  • Gains hold across diverse domains, including mathematical reasoning and multi-hop QA.

Signal Assessment

6/10 Using entropy reduction as a cheap, intrinsic proxy for tool-call quality is a useful contribution, and the efficiency gains are large. However, entropy- and uncertainty-based rewards are already an active line of work in LLM RL, so this is a strong extension rather than a paradigm shift.

Methodology

  1. Run pilot studies measuring model entropy before and after each tool call, then correlate entropy reduction with tool-call quality and downstream correctness.
  2. Define entropy reduction (the drop in policy uncertainty after the tool response is incorporated) as the supervisory signal.
  3. Design a sparse outcome reward that aggregates entropy-reduction information at the trajectory level to favor efficient, high-utility tool use.
  4. Design a dense process reward that assigns per-step credit to each tool call according to its entropy reduction.
  5. Train agents with RL under each reward design and evaluate on math reasoning and multi-hop QA benchmarks, tracking accuracy and tool-call counts against baselines.

System Components

Entropy-based pilot analysis

Empirical study linking post-call reductions in token-level predictive entropy to high-quality, useful tool calls, which motivates the reward design.

Entropy reduction signal

An intrinsic, annotation-free measure of how much a tool call's result reduces the agent's uncertainty, used as a proxy for tool-call utility.

Sparse outcome reward

Coarse, trajectory-level reward that favors efficient trajectories and penalizes redundant or uninformative calls, mainly cutting tool-call volume and latency.

Dense process reward

Fine-grained, step-level reward that credits each tool call by its entropy reduction, improving credit assignment and task accuracy.

RL training framework for tool-using agents

Policy optimization loop, likely GRPO/PPO-style, that incorporates either reward variant to shape tool-use behavior.

Results

Metric/Benchmark Baseline This Paper Delta
Number of tool calls (sparse outcome reward) Average of baselines Substantially fewer calls -72.07%
Task performance (dense process reward) Average of baselines Higher accuracy +22.27%
Cross-domain generality (math reasoning, multi-hop QA) Standard RL tool-use agents Consistent improvements across domains Qualitative gain

Key Takeaways

  • Entropy reduction after a tool call is a cheap, model-internal signal of tool-call usefulness, and it can replace or complement costly step-level annotations or learned process reward models.
  • Choose the reward granularity by objective: use sparse trajectory-level entropy rewards to cut latency and cost, and dense step-level rewards when accuracy matters most.
  • Agents trained only on outcome correctness tend to over-call tools. Adding an information-gain-style signal can sharply reduce redundant calls, which matters for production agents with expensive APIs.
  • Treating entropy as a measure of information acquired from the environment, not just an exploration regularizer, offers a general design pattern for agentic RL beyond tool use.

Abstract

Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.

Generated automatically from available metadata and abstract on 2026-11-09.