R2IF: Aligning Reasoning with Decisions via Composite Rewards for Interpretable LLM Function Calling

R2IF is a reasoning-aware reinforcement learning framework that trains LLMs for function calling using a composite reward—combining format/correctness checks, a Chain-of-Thought Effectiveness Reward (CER), and a Specification-Modification-Value (SMV) reward—optimized via GRPO to ensure the model's stated reasoning actually drives its tool-call decisions.

Problem Statement

Current RL-trained function-calling LLMs often generate reasoning traces that are decoupled from their final tool-call decisions, producing plausible-looking Chain-of-Thought text that doesn't causally explain the chosen action. This undermines trust, debuggability, and safety in tool-augmented agentic deployments, where practitioners need to verify that a model's reasoning genuinely justifies its API calls rather than serving as post-hoc rationalization.

Key Novelty

Chain-of-Thought Effectiveness Reward (CER) that explicitly measures and optimizes for causal alignment between the reasoning process and the final tool-call decision
Specification-Modification-Value (SMV) reward that jointly evaluates adherence to tool specifications, appropriateness of parameter modifications, and correctness of assigned values
A composite reward architecture combining format/correctness constraints with CER and SMV inside a GRPO optimization loop, explicitly targeting interpretability as a first-class training objective alongside accuracy

Evaluation Highlights

Up to 34.62% relative improvement over baselines on BFCL with Llama3.2-3B
Positive Average CoT Effectiveness score (0.05 for Llama3.2-3B), indicating reasoning traces measurably contribute to correct decisions
Consistent gains validated across both BFCL and ACEBench benchmarks

Signal Assessment

5/10 The work is a solid, well-motivated contribution that introduces a novel reward-engineering approach (CER + SMV) to a real and underexplored problem—reasoning/decision misalignment—but relies on established RL machinery (GRPO) rather than introducing a fundamentally new training paradigm.

Methodology

Frame function calling as an RL problem where the LLM policy jointly generates a reasoning trace (CoT) and a structured tool-call decision
Construct a composite reward signal: format/correctness constraints for output validity, CER for reasoning-decision alignment, and SMV for specification/parameter/value correctness
Optimize the policy using GRPO (Group Relative Policy Optimization), a critic-free RL algorithm suited for LLM fine-tuning
Evaluate trained models (e.g., Llama3.2-3B) on BFCL and ACEBench, measuring both function-calling accuracy and CoT Effectiveness as an interpretability metric

System Components

CER (Chain-of-Thought Effectiveness Reward)

Rewards reasoning traces whose content causally supports and explains the final tool-call decision, penalizing disconnected or decorative CoT

SMV (Specification-Modification-Value) Reward

Evaluates whether the model correctly follows the tool's API specification, makes appropriate modifications to parameters, and assigns correct argument values

Format/Correctness Constraints

Base-level reward enforcing valid structured output format and correctness of the final function call

GRPO Optimizer

Group Relative Policy Optimization used to train the policy on the composite reward without a separate value/critic network

Results

Metric/Benchmark	Baseline	This Paper	Delta
BFCL Accuracy (Llama3.2-3B)	Standard RL/SFT baseline	R2IF (composite reward + GRPO)	Up to +34.62% relative improvement
Average CoT Effectiveness (Llama3.2-3B)	Not explicitly positive/aligned	0.05 (positive)	Reasoning shown to meaningfully influence decisions
ACEBench Performance	Baseline approaches	R2IF	Outperforms baselines (exact figures not given in abstract)

Key Takeaways

Explicitly rewarding reasoning-decision alignment (via CER) can improve both accuracy and interpretability simultaneously, rather than trading one off for the other
Composite, multi-faceted reward design (format + causal alignment + specification correctness) is an effective pattern for GRPO-based fine-tuning in structured agentic tasks like function calling
Interpretability metrics like Average CoT Effectiveness should be tracked alongside accuracy when validating tool-augmented LLMs for production deployment
Gains on small models (Llama3.2-3B) suggest this reward strategy is a cost-effective way to improve reliability of lightweight agentic models without scaling parameters

Abstract

Function calling empowers large language models (LLMs) to interface with external tools, yet existing RL-based approaches suffer from misalignment between reasoning processes and tool-call decisions. We propose R2IF, a reasoning-aware RL framework for interpretable function calling, adopting a composite reward integrating format/correctness constraints, Chain-of-Thought Effectiveness Reward (CER), and Specification-Modification-Value (SMV) reward, optimized via GRPO. Experiments on BFCL/ACEBench show R2IF outperforms baselines by up to 34.62% (Llama3.2-3B on BFCL) with positive Average CoT Effectiveness (0.05 for Llama3.2-3B), enhancing both function-calling accuracy and interpretability for reliable tool-augmented LLM deployment.