OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning
Problem Statement
Multi-agent LLM reasoning systems usually rely on fixed topologies, majority voting, or round-table debate. These setups can drown out correct minority opinions and let dominant but wrong agents prevail. Recent graph-based multi-agent optimizers tune for end-task performance alone and ignore whether the agents' interactions are actually productive. That leaves the communication process, a likely driver of collective reasoning quality, unoptimized.
Key Novelty
- Frames multi-agent collaboration-structure design as a verbal reinforcement learning problem, with explicit action spaces for dynamically constructing and refining which agents communicate and how.
- Introduces a feedback signal that scores interaction quality (communication robustness and coherence across debate rounds) in addition to agent output correctness.
- Tests the hypothesis that debate quality, not just individual agent capability, is a key driver of multi-agent reasoning performance.
Evaluation Highlights
- The abstract reports that OPTAGENT significantly outperforms single-agent prompting methods (e.g., CoT-style baselines) and state-of-the-art multi-agent frameworks; no specific numbers are given.
- It is evaluated on mathematical reasoning, scientific reasoning, creative writing, and numerical sorting, which suggests the gains hold beyond purely mathematical domains.
Signal Assessment
Methodology
- Model the multi-agent system as a graph of LLM agents and define an action space for modifying the collaboration structure, such as adding, removing, or reweighting communication edges and changing debate participation.
- Run multi-round debates over the current structure while a feedback mechanism evaluates each interaction for robustness and coherence, producing verbal (natural-language) reward signals.
- Iteratively refine the collaboration structure using this verbal RL feedback, aggregate the final answer by majority vote over all agents, and benchmark against single-agent and multi-agent baselines on reasoning tasks.
System Components
Dynamic graph representation of which LLM agents exchange messages during debate. This graph is the object being optimized.
The set of operations the optimizer can apply to construct or refine the collaboration topology during or across debates.
An evaluator, likely LLM-based, that judges communication robustness and coherence throughout the debate and emits verbal feedback used as the RL signal.
A gradient-free optimization loop that uses natural-language feedback to iteratively improve the collaboration structure.
Produces the final answer by voting over all agents' outputs after the optimized debate.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Mathematical reasoning accuracy | Single-agent prompting / SOTA multi-agent frameworks | OPTAGENT | Significant improvement (no numbers in abstract) |
| Scientific reasoning accuracy | Single-agent prompting / SOTA multi-agent frameworks | OPTAGENT | Significant improvement (no numbers in abstract) |
| Creative writing quality | Single-agent prompting / SOTA multi-agent frameworks | OPTAGENT | Outperforms (no numbers in abstract) |
| Numerical sorting accuracy | Single-agent prompting / SOTA multi-agent frameworks | OPTAGENT | Outperforms (no numbers in abstract) |
Key Takeaways
- When building multi-agent debate systems, optimize the communication topology itself rather than fixing it ahead of time. Who talks to whom can matter as much as which agents you use.
- Process-level rewards on debate coherence and robustness can supplement outcome-only rewards and may help keep correct minority views from being suppressed.
- Verbal RL is a gradient-free way to optimize agentic system structure with closed or API-only models. Practitioners should still budget for the extra inference cost of repeated debates and LLM-based evaluation, and check that gains justify that compute against strong self-consistency baselines.
Abstract
Large Language Models (LLMs) have shown remarkable reasoning capabilities in mathematical and scientific tasks. To enhance complex reasoning, multi-agent systems have been proposed to harness the collective intelligence of LLM agents. However, existing collaboration structures are either predefined or rely on majority voting or round-table debates, which can suppress correct but less dominant agent contributions. Recent approaches model multi-agent systems as graph networks but optimize purely for agent performance, neglecting the quality of interactions. We hypothesize that effective agent communication is crucial for multi-agent reasoning and that debating quality plays a significant role. To address this, we propose $\ours$, a multi-agent verbal reinforcement learning algorithm that dynamically constructs and refines multi-agent collaboration structures. Our method defines action spaces and a feedback mechanism that evaluates communication robustness and coherence throughout the debate. The final decision is achieved through a majority vote over all the agents. We assess $\ours$ on various reasoning tasks, including mathematical reasoning, creative writing, scientific reasoning, and numerical sorting. Results demonstrate that our approach significantly outperforms single-agent prompting methods and state-of-the-art multi-agent frameworks on diverse tasks.