← Back to Paper Radar

MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)

Aaron Steiner, R. Peeters, Christian Bizer
arXiv.org | 2025
This is the first controlled, same-task comparison of four agent-to-web interfaces: HTML browsing, RAG over pre-crawled content, MCP-based Web APIs, and NLWeb natural-language querying. Structured or retrieval-based interfaces (RAG, MCP, NLWeb) beat raw HTML browsing on both task effectiveness and efficiency.

Problem Statement

LLM web agents increasingly automate e-commerce tasks such as search, comparison and checkout. Practitioners and site owners can choose among very different interaction architectures, but no one had benchmarked these on identical tasks in one environment. Without such a comparison, design choices about how to expose websites to agents, and which interface agents should use, rest on anecdote rather than evidence.

Key Novelty

  • A controlled testbed of four simulated e-shops, each exposing the same product catalog via HTML, MCP and NLWeb interfaces, plus pre-crawled content for RAG
  • Specialized agents for each of the four interfaces, evaluated on identical task sets ranging from simple product search to complementary/substitute product queries and multi-step checkout
  • A cross-model analysis (GPT-4.1, GPT-5, GPT-5 mini, Claude Sonnet 4) quantifying effectiveness (F1, completion rate), token usage, runtime and API cost trade-offs per interface

Evaluation Highlights

  • Averaged over all tasks, F1 rises from 0.67 (HTML) to 0.75–0.77 (RAG/MCP/NLWeb). Tokens per task fall from about 241k to 47k–140k, and runtime per task falls from 291s to 50–62s.
  • The best configuration is RAG + GPT-5 (F1 0.87, completion rate 0.79). RAG + GPT-5 mini is identified as the best cost/performance compromise.

Signal Assessment

5/10 This is a useful and timely empirical benchmark that fills a clear gap in agent interface design. However, it introduces no new algorithmic method, and its simulated e-shop setting may limit how far the results generalize to the real web.

Methodology

  1. Build four simulated e-shops with product catalogs, each served through an HTML front-end, an MCP server exposing Web API tools, and an NLWeb natural-language endpoint; pre-crawl the content to build a RAG index
  2. Implement an interface-specific agent for each architecture (HTML browser agent, RAG agent, MCP tool-calling agent, NLWeb query agent)
  3. Define a shared task suite: product search, price/offer comparison, complementary/substitute product discovery, and checkout
  4. Run every agent–interface combination with GPT-4.1, GPT-5, GPT-5 mini and Claude Sonnet 4 as backbone LLMs
  5. Measure F1, task completion rate, token consumption, runtime and API cost, and compare across interfaces and models

System Components

Simulated e-shop testbed

Four controlled online shops with identical product data exposed through multiple interfaces, enabling apples-to-apples comparison

HTML agent

Baseline agent that navigates and parses raw web pages, mimicking traditional browsing agents

RAG agent

Agent that retrieves from a pre-crawled index of shop content instead of browsing live pages

MCP agent

Agent that calls shop Web APIs exposed as tools via the Model Context Protocol

NLWeb agent

Agent that queries shops through the NLWeb natural-language interface, receiving structured (e.g., schema.org-style) responses

Task suite & metrics

Tasks from simple lookups to checkout, evaluated with F1, completion rate, tokens, runtime and cost

Results

Metric/Benchmark Baseline This Paper Delta
Avg F1 (all tasks) 0.67 (HTML) 0.75–0.77 (RAG/MCP/NLWeb) +0.08 to +0.10
Tokens per task ~241k (HTML) 47k–140k ~42%–80% reduction
Runtime per task 291 s (HTML) 50–62 s ~79%–83% reduction
Best config F1 0.67 (HTML avg) 0.87 (RAG + GPT-5) +0.20
Best config completion rate Not reported for HTML 0.79 (RAG + GPT-5) Qualitatively higher
Cost-efficient config HTML agents (high token cost) RAG + GPT-5 mini Good performance at much lower API cost

Key Takeaways

  • Avoid raw HTML browsing when a structured alternative exists. RAG, MCP or NLWeb interfaces give higher accuracy with roughly 4–5x faster runs and far fewer tokens.
  • For e-commerce agents, RAG over pre-crawled content with a strong model (GPT-5) was the top performer. GPT-5 mini with RAG is a pragmatic choice for cost-sensitive deployments.
  • Site operators who want agent traffic should expose MCP or NLWeb endpoints. Interface choice affects agent success and cost as much as the choice of backbone LLM.
  • Evaluate agents on joint effectiveness–efficiency metrics (F1, completion, tokens, latency, cost), because interface choices shift all of them substantially.

Abstract

Large language model agents are increasingly used to automate web tasks such as product search, offer comparison, and checkout. Current research explores different interfaces through which these agents interact with websites, including traditional HTML browsing, retrieval-augmented generation (RAG) over pre-crawled content, communication via Web APIs using the Model Context Protocol (MCP), and natural-language querying through the NLWeb interface. However, no prior work has compared these four architectures within a single controlled environment using identical tasks. To address this gap, we introduce a testbed consisting of four simulated e-shops, each offering its products via HTML, MCP, and NLWeb interfaces. For each interface (HTML, RAG, MCP, and NLWeb) we develop specialized agents that perform the same sets of tasks, ranging from simple product searches and price comparisons to complex queries for complementary or substitute products and checkout processes. We evaluate the agents using GPT 4.1, GPT 5, GPT 5 mini, and Claude Sonnet 4 as underlying LLM. Our evaluation shows that the RAG, MCP and NLWeb agents outperform HTML on both effectiveness and efficiency. Averaged over all tasks, F1 rises from 0.67 for HTML to between 0.75 and 0.77 for the other agents. Token usage falls from about 241k for HTML to between 47k and 140k per task. The runtime per task drops from 291 seconds to between 50 and 62 seconds. The best overall configuration is RAG with GPT 5 achieving an F1 score of 0.87 and a completion rate of 0.79. Also taking cost into consideration, RAG with GPT 5 mini offers a good compromise between API usage fees and performance. Our experiments show the choice of the interaction interface has a substantial impact on both the effectiveness and efficiency of LLM-based web agents.

Generated automatically from available metadata and abstract on 2026-11-09.