Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applications

The paper introduces a two-stage framework that jointly tackles domain customization and inference efficiency for LLM-based multi-agent systems, using compact fine-tuned models paired with speculative decoding and FP8 quantization to make agentic AI viable for enterprise production deployment.

Problem Statement

LLM-based multi-agent systems show strong reasoning and task-execution capabilities but remain difficult to deploy in production due to the need for domain-specific customization and the high latency/inference costs inherent in multi-step agentic workflows. This matters because these deployment barriers currently prevent enterprises from translating research-level agentic performance into cost-effective, real-world business applications. The paper addresses both the adaptation gap (generic models underperforming on specialized domains) and the serving cost gap (expensive, slow inference at scale) in a single unified pipeline.

Key Novelty

A unified end-to-end framework that couples model customization and inference optimization specifically for multi-agent systems, rather than treating training and deployment as separate concerns
'Agentic Model Customization' pipeline combining continual pretraining, supervised fine-tuning, and preference optimization to adapt compact models to specialized domains while explicitly preserving agentic (tool-use/reasoning/planning) capabilities
Inference Optimization stage combining speculative decoding with FP8 quantization and targeted calibration tailored to agentic workflows, minimizing quality loss while cutting serving costs

Evaluation Highlights

4.48x throughput speedup in inference serving compared to baseline deployment
Maintained task performance and improved robustness on long-tail/edge-case scenarios after optimization
Demonstrated rapid domain adaptation across multiple real enterprise workloads

Signal Assessment

4/10 The paper delivers a valuable, well-engineered production pipeline combining established techniques (continual pretraining, SFT, preference optimization, speculative decoding, FP8 quantization) rather than introducing fundamentally new algorithms, placing it as a solid applied/systems contribution rather than a research breakthrough.

Methodology

Stage 1: Apply continual pretraining on domain-specific enterprise data to a compact base model
Stage 1: Perform supervised fine-tuning to instill task-specific agentic behaviors (tool use, multi-step reasoning)
Stage 1: Apply preference optimization to align model outputs with desired agentic response quality
Stage 2: Apply speculative decoding to accelerate token generation during multi-agent inference
Stage 2: Apply FP8 quantization with targeted calibration to reduce compute/memory footprint with minimal accuracy degradation
Evaluate the combined pipeline across enterprise workloads for throughput, task performance, and long-tail robustness

System Components

Agentic Model Customization

Training pipeline combining continual pretraining, SFT, and preference optimization to adapt a compact model to a specialized enterprise domain while retaining multi-agent reasoning and tool-use capabilities

Inference Optimization

Serving pipeline integrating speculative decoding and FP8 quantization with targeted calibration to reduce latency and inference cost with minimal quality loss

Compact Domain-Adapted Model

A smaller, efficient LLM backbone customized for enterprise-specific agentic tasks, replacing larger general-purpose models to reduce serving cost

Targeted Calibration

A calibration procedure applied during FP8 quantization to preserve model quality, particularly on rare or long-tail scenarios

Results

Metric/Benchmark	Baseline	This Paper	Delta
Inference Throughput	Standard (uncompressed) serving	Optimized serving pipeline	4.48x speedup
Task Performance	Pre-optimization baseline	Post customization + optimization	Maintained (no significant degradation)
Long-tail Scenario Robustness	Standard model	Customized + optimized model	Improved (qualitative)
Domain Adaptation Effort	Manual/slow adaptation	Framework-driven adaptation	Faster/rapid (qualitative)

Key Takeaways

Pairing domain-specific model customization with inference optimization can yield compounding benefits, making multi-agent LLM deployment both accurate and cost-efficient in production
Compact models, when adapted via continual pretraining, SFT, and preference optimization, can retain strong agentic capabilities—reducing reliance on large, expensive foundation models
Speculative decoding combined with FP8 quantization (with careful calibration) can deliver substantial throughput gains (~4.5x) without sacrificing quality, offering a practical recipe for teams optimizing agent-serving infrastructure
Robustness on long-tail scenarios can be improved alongside efficiency gains, suggesting the customization stage helps address edge cases that pure inference optimization alone would not

Abstract

Large language model (LLM)-based multi-agent systems demonstrate strong performance on complex reasoning and task execution, enabling broad enterprise applications. However, production deployment remains challenging due to domain-specific customization requirements and high latency and inference costs in agentic workflows. We propose a unified framework for customization and efficient deployment of multi-agent systems in real-world settings. The first stage, Agentic Model Customization, combines continual pretraining, supervised fine-tuning, and preference optimization to adapt a compact model to specialized domains while retaining strong agentic capabilities. The second stage, Inference Optimization, integrates speculative decoding and FP8 quantization with targeted calibration to enable cost-efficient serving with minimal quality loss. Across enterprise workloads, our framework enables rapid domain adaptation and achieves a 4.48x speedup in throughput while maintaining performance and improving robustness on long-tail scenarios.