Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates

ReViewGraph improves automatic paper review by first simulating multi-round reviewer-author debates with LLM-based multi-agent collaboration, then encoding the resulting argumentative exchanges (e.g., acceptance, rejection, clarification, compromise) as typed edges in a heterogeneous graph that a GNN reasons over to produce the final review decision.

Problem Statement

Automatic paper review tools are increasingly needed as submission volumes grow, but current approaches either rely on shallow manuscript features or directly query LLMs, which are prone to hallucination, biased or inconsistent scoring, and shallow reasoning. Critically, these methods ignore the iterative negotiation dynamic of real peer review—where reviewers and authors exchange rebuttals, clarifications, and compromises—resulting in decisions that fail to reflect the nuanced argumentative structure of genuine review discussions.

Key Novelty

Simulates realistic multi-round reviewer-author debates via LLM-based multi-agent collaboration, rather than generating a single static review
Explicitly extracts and types diverse opinion relations (acceptance, rejection, clarification, compromise) as structured edges rather than treating debate text as unstructured input
Introduces a heterogeneous interaction graph formulation combined with graph neural network reasoning to model fine-grained argumentative dynamics for review decision-making

Evaluation Highlights

Evaluated on three distinct paper-review datasets against strong existing baselines
Achieves an average relative improvement of 15.73% over baselines

Signal Assessment

6/10 The work is a thoughtful, well-motivated combination of established techniques (LLM multi-agent debate simulation and heterogeneous GNN reasoning) applied to a novel structured formulation of automatic peer review, yielding solid empirical gains but not introducing a fundamentally new learning paradigm.

Methodology

Simulate multi-round reviewer-author interactions using LLM-based multi-agent role play (reviewer critiques, author rebuttals, follow-up clarifications)
Parse the simulated dialogue to identify and classify opinion relations between statements (e.g., acceptance, rejection, clarification, compromise)
Construct a heterogeneous interaction graph where these relations become typed edges connecting nodes representing reviewer/author arguments
Apply graph neural networks to propagate and aggregate information across the typed graph, capturing argumentative dynamics
Derive the final review decision/score from the learned graph representation

System Components

Multi-Agent Debate Simulator

Uses LLMs to role-play reviewers and authors across multiple rounds, generating realistic critique-rebuttal-clarification exchanges

Opinion Relation Extractor

Analyzes debate turns to identify and label argumentative relations such as acceptance, rejection, clarification, and compromise

Heterogeneous Interaction Graph

Structures the debate as a graph with typed nodes and edges encoding the diverse relations between reviewer and author statements

Graph Neural Network Reasoner

Performs message passing over the heterogeneous graph to capture fine-grained argumentative dynamics for downstream review reasoning

Review Decision Module

Consumes the graph-reasoned representation to output the final automated review judgment/score

Results

Metric/Benchmark	Baseline	ReViewGraph	Delta
Average across 3 review datasets	Strong existing baselines (feature-based & direct LLM methods)	ReViewGraph (heterogeneous graph reasoning)	+15.73% relative improvement

Key Takeaways

Explicitly modeling the negotiation/debate process—rather than treating review generation as a single-pass text task—can meaningfully improve automated review quality
Typed heterogeneous graphs are an effective way to encode diverse argumentative relations for GNN-based reasoning in dialogue/debate-centric tasks
LLM-based multi-agent simulation can serve as a practical data-generation step to create structured training signals for downstream graph-based models
The debate-graph-reasoning paradigm may generalize beyond peer review to other domains involving negotiation or argumentation, such as legal analysis or structured decision-making

Abstract

Existing paper review methods often rely on superficial manuscript features or directly on large language models (LLMs), which are prone to hallucinations, biased scoring, and limited reasoning capabilities. Moreover, these methods often fail to capture the complex argumentative reasoning and negotiation dynamics inherent in reviewer-author interactions. To address these limitations, we propose ReViewGraph (Reviewer-Author Debates Graph Reasoner), a novel framework that performs heterogeneous graph reasoning over LLM-simulated multi-round reviewer-author debates. In our approach, reviewer-author exchanges are simulated through LLM-based multi-agent collaboration. Diverse opinion relations (e.g., acceptance, rejection, clarification, and compromise) are then explicitly extracted and encoded as typed edges within a heterogeneous interaction graph. By applying graph neural networks to reason over these structured debate graphs, ReViewGraph captures fine-grained argumentative dynamics and enables more informed review decisions. Extensive experiments on three datasets demonstrate that ReViewGraph outperforms strong baselines with an average relative improvement of 15.73%, underscoring the value of modeling detailed reviewer–author debate structures.