MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

MCPAgentBench is a benchmark that evaluates LLM agents' ability to select and invoke tools via the Model Context Protocol (MCP) using real-world-derived tasks, simulated (non-live) MCP tools, and a dynamic sandbox that injects distractor tools to test selection and discrimination skills.

Problem Statement

As LLM agents increasingly rely on MCP to interface with external tools and services, existing evaluation sets are hampered by dependence on live external MCP servers (causing reproducibility, cost, and reliability issues) and by a lack of difficulty stratification across tasks. This makes it hard to rigorously and fairly compare agents' tool-use capabilities as MCP adoption grows, motivating a controlled, reproducible, and difficulty-aware benchmark.

Key Novelty

Grounds the benchmark in authentic real-world MCP tool definitions but replaces live services with simulated tools, removing external dependencies while preserving realism
Introduces a dynamic sandbox that presents candidate tool lists with distractors, explicitly testing tool selection/discrimination rather than just execution correctness
Proposes comprehensive metrics jointly measuring task completion rate and execution efficiency, rather than binary success/failure

Evaluation Highlights

Multiple latest mainstream LLMs evaluated on authentic multi-step tool-use tasks within the sandbox
Results reported via task completion rate and execution efficiency metrics, revealing large performance gaps between models

Signal Assessment

4/10 This is a well-motivated, practically useful benchmark that fills a specific gap in MCP evaluation, but its core design elements (simulated tools, distractor injection, sandboxing) build on established tool-use benchmark patterns rather than introducing fundamentally new evaluation paradigms.

Methodology

Curate authentic, real-world tasks paired with real MCP tool/service definitions to ground the benchmark in practical use cases
Construct simulated (mock) MCP tool implementations that emulate real tool behavior without requiring live external services
Build a dynamic sandbox environment that presents agents with candidate tool lists augmented with distractor tools to probe selection accuracy
Define metrics capturing both task completion correctness and execution efficiency (e.g., steps/calls taken)
Run benchmark evaluations across a range of current mainstream LLM agents and analyze multi-step tool invocation performance

System Components

Real-world task dataset

Collection of authentic tasks derived from practical scenarios requiring MCP tool invocation

Simulated MCP tools

Mocked implementations of real MCP tool definitions that remove reliance on live external services

Dynamic sandbox environment

Execution environment that injects distractor tools into candidate lists to test tool selection and discrimination

Evaluation metric suite

Combined metrics for task completion rate and execution efficiency to give a difficulty-aware performance profile

Results

Aspect	Prior MCP Benchmarks	MCPAgentBench	Improvement
Tool environment	Depends on live external MCP services	Simulated tools in a sandbox	Removes external dependency, improves reproducibility
Task design	Limited difficulty awareness	Difficulty-aware, real-world-derived tasks	Enables finer-grained capability differentiation
Tool selection testing	Rarely tested explicitly	Candidate lists with distractor tools	Directly measures tool discrimination ability
Model comparison	Not standardized	Significant performance gaps found across mainstream LLMs on complex multi-step tasks	Surfaces concrete weaknesses in current agents

Key Takeaways

Even leading mainstream LLMs show substantial weaknesses in complex, multi-step MCP tool invocation, so teams should benchmark specific models before deploying MCP-based agentic workflows
Tool discrimination amid distractors is a distinct and important capability separate from raw execution accuracy, and should be tested explicitly in agent evaluations
Reproducible agent benchmarking is achievable via simulated tool environments, avoiding the cost and instability of relying on live external MCP services
Execution efficiency (not just task success) is a meaningful axis for comparing agents, relevant to latency- and cost-sensitive production deployments

Abstract

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at Github.