← Back to Paper Radar

OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning

Zhenyu Bi, Meng Lu, Yang Li, Swastik Roy, Weijie Guan, Morteza Ziyadi, Xuan Wang
IJCNLP-AACL | 2025
OPTAGENT treats the collaboration structure of a multi-agent LLM system as something to optimize. It uses verbal reinforcement learning, meaning natural-language feedback instead of gradient updates, to build and refine the communication graph between agents, rewarding the quality of their debate (robustness and coherence) and not just final-answer accuracy.

Problem Statement

Multi-agent LLM reasoning systems usually rely on fixed topologies, majority voting, or round-table debate. These setups can drown out correct minority opinions and let dominant but wrong agents prevail. Recent graph-based multi-agent optimizers tune for end-task performance alone and ignore whether the agents' interactions are actually productive. That leaves the communication process, a likely driver of collective reasoning quality, unoptimized.

Key Novelty

  • Frames multi-agent collaboration-structure design as a verbal reinforcement learning problem, with explicit action spaces for dynamically constructing and refining which agents communicate and how.
  • Introduces a feedback signal that scores interaction quality (communication robustness and coherence across debate rounds) in addition to agent output correctness.
  • Tests the hypothesis that debate quality, not just individual agent capability, is a key driver of multi-agent reasoning performance.

Evaluation Highlights

  • The abstract reports that OPTAGENT significantly outperforms single-agent prompting methods (e.g., CoT-style baselines) and state-of-the-art multi-agent frameworks; no specific numbers are given.
  • It is evaluated on mathematical reasoning, scientific reasoning, creative writing, and numerical sorting, which suggests the gains hold beyond purely mathematical domains.

Signal Assessment

5/10 Optimizing interaction quality rather than only outcomes is a sensible and useful extension of graph-based multi-agent optimization. It combines established ideas, verbal RL (Reflexion-style) and agent-graph optimization, and still ends in majority voting, so it is a solid incremental contribution rather than a major advance.

Methodology

  1. Model the multi-agent system as a graph of LLM agents and define an action space for modifying the collaboration structure, such as adding, removing, or reweighting communication edges and changing debate participation.
  2. Run multi-round debates over the current structure while a feedback mechanism evaluates each interaction for robustness and coherence, producing verbal (natural-language) reward signals.
  3. Iteratively refine the collaboration structure using this verbal RL feedback, aggregate the final answer by majority vote over all agents, and benchmark against single-agent and multi-agent baselines on reasoning tasks.

System Components

Agent Collaboration Graph

Dynamic graph representation of which LLM agents exchange messages during debate. This graph is the object being optimized.

Structure Action Space

The set of operations the optimizer can apply to construct or refine the collaboration topology during or across debates.

Interaction-Quality Feedback Mechanism

An evaluator, likely LLM-based, that judges communication robustness and coherence throughout the debate and emits verbal feedback used as the RL signal.

Verbal Reinforcement Learning Loop

A gradient-free optimization loop that uses natural-language feedback to iteratively improve the collaboration structure.

Majority-Vote Aggregator

Produces the final answer by voting over all agents' outputs after the optimized debate.

Results

Metric/Benchmark Baseline This Paper Delta
Mathematical reasoning accuracy Single-agent prompting / SOTA multi-agent frameworks OPTAGENT Significant improvement (no numbers in abstract)
Scientific reasoning accuracy Single-agent prompting / SOTA multi-agent frameworks OPTAGENT Significant improvement (no numbers in abstract)
Creative writing quality Single-agent prompting / SOTA multi-agent frameworks OPTAGENT Outperforms (no numbers in abstract)
Numerical sorting accuracy Single-agent prompting / SOTA multi-agent frameworks OPTAGENT Outperforms (no numbers in abstract)

Key Takeaways

  • When building multi-agent debate systems, optimize the communication topology itself rather than fixing it ahead of time. Who talks to whom can matter as much as which agents you use.
  • Process-level rewards on debate coherence and robustness can supplement outcome-only rewards and may help keep correct minority views from being suppressed.
  • Verbal RL is a gradient-free way to optimize agentic system structure with closed or API-only models. Practitioners should still budget for the extra inference cost of repeated debates and LLM-based evaluation, and check that gains justify that compute against strong self-consistency baselines.

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities in mathematical and scientific tasks. To enhance complex reasoning, multi-agent systems have been proposed to harness the collective intelligence of LLM agents. However, existing collaboration structures are either predefined or rely on majority voting or round-table debates, which can suppress correct but less dominant agent contributions. Recent approaches model multi-agent systems as graph networks but optimize purely for agent performance, neglecting the quality of interactions. We hypothesize that effective agent communication is crucial for multi-agent reasoning and that debating quality plays a significant role. To address this, we propose $\ours$, a multi-agent verbal reinforcement learning algorithm that dynamically constructs and refines multi-agent collaboration structures. Our method defines action spaces and a feedback mechanism that evaluates communication robustness and coherence throughout the debate. The final decision is achieved through a majority vote over all the agents. We assess $\ours$ on various reasoning tasks, including mathematical reasoning, creative writing, scientific reasoning, and numerical sorting. Results demonstrate that our approach significantly outperforms single-agent prompting methods and state-of-the-art multi-agent frameworks on diverse tasks.

Generated automatically from available metadata and abstract on 2026-11-09.