Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents
Problem Statement
Over long trajectories, tool-using LLM agents often make excessive, redundant, or low-quality tool calls. This raises latency and cost and can hurt final answer quality. Existing RL rewards are mostly outcome-correctness signals that say little about whether a given tool call was useful, so tool-use behavior is hard to control without costly step-level annotation.
Key Novelty
- Pilot experiments showing a strong positive correlation between post-call entropy reduction and the quality of a tool call, which yields an intrinsic, annotation-free signal of tool-call utility.
- A sparse, trajectory-level outcome reward built on entropy reduction, aimed at efficiency by discouraging tool calls that do not reduce uncertainty.
- A dense, step-level process reward built on per-call entropy reduction, giving fine-grained credit assignment to individual tool calls to improve task performance.
- A reframing of entropy from an exploration-regularization quantity to a measure of the information gained from tool interactions.
Evaluation Highlights
- The sparse outcome reward reduces tool calls by 72.07% relative to the average of the baselines.
- The dense process reward improves task performance by 22.27%, likely relative to the baseline average (the abstract does not state the comparison point).
- Gains hold across diverse domains, including mathematical reasoning and multi-hop QA.
Signal Assessment
Methodology
- Run pilot studies measuring model entropy before and after each tool call, then correlate entropy reduction with tool-call quality and downstream correctness.
- Define entropy reduction (the drop in policy uncertainty after the tool response is incorporated) as the supervisory signal.
- Design a sparse outcome reward that aggregates entropy-reduction information at the trajectory level to favor efficient, high-utility tool use.
- Design a dense process reward that assigns per-step credit to each tool call according to its entropy reduction.
- Train agents with RL under each reward design and evaluate on math reasoning and multi-hop QA benchmarks, tracking accuracy and tool-call counts against baselines.
System Components
Empirical study linking post-call reductions in token-level predictive entropy to high-quality, useful tool calls, which motivates the reward design.
An intrinsic, annotation-free measure of how much a tool call's result reduces the agent's uncertainty, used as a proxy for tool-call utility.
Coarse, trajectory-level reward that favors efficient trajectories and penalizes redundant or uninformative calls, mainly cutting tool-call volume and latency.
Fine-grained, step-level reward that credits each tool call by its entropy reduction, improving credit assignment and task accuracy.
Policy optimization loop, likely GRPO/PPO-style, that incorporates either reward variant to shape tool-use behavior.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Number of tool calls (sparse outcome reward) | Average of baselines | Substantially fewer calls | -72.07% |
| Task performance (dense process reward) | Average of baselines | Higher accuracy | +22.27% |
| Cross-domain generality (math reasoning, multi-hop QA) | Standard RL tool-use agents | Consistent improvements across domains | Qualitative gain |
Key Takeaways
- Entropy reduction after a tool call is a cheap, model-internal signal of tool-call usefulness, and it can replace or complement costly step-level annotations or learned process reward models.
- Choose the reward granularity by objective: use sparse trajectory-level entropy rewards to cut latency and cost, and dense step-level rewards when accuracy matters most.
- Agents trained only on outcome correctness tend to over-call tools. Adding an information-gain-style signal can sharply reduce redundant calls, which matters for production agents with expensive APIs.
- Treating entropy as a measure of information acquired from the environment, not just an exploration regularizer, offers a general design pattern for agentic RL beyond tool use.
Abstract
Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.