Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs

AsyncFC is a training-free, execution-layer framework that introduces 'futures'—symbolic placeholders for unresolved results—into LLM function calling, letting model decoding overlap with tool execution and independent tool calls run in parallel without any changes to the model or the function-calling protocol.

Problem Statement

Function calling/tool use is central to LLM agents, but current implementations are strictly synchronous: decoding halts until each function call returns, serializing execution and inflating end-to-end latency in multi-step agentic workflows (e.g., coding agents, research agents). Prior fixes to this latency problem typically require fine-tuning specialized models or altering the tool-calling protocol, limiting deployability; this work targets a solution that works with existing models and tools out of the box.

Key Novelty

Introduces 'futures'—symbolic references to unresolved execution results—as a mechanism for LLMs to reason about pending tool outputs, adapting a classic concurrent-programming abstraction to model-tool interaction
Proposes a purely execution-layer (middleware/runtime) framework that requires no model fine-tuning, no modification of function implementations, and no change to the standard synchronous function-calling protocol
Enables both decode-execution overlap and dependency-aware inter-function parallelism within a single unified scheduling framework
Empirically demonstrates that LLMs possess an emergent, native ability to reason over and manipulate symbolic futures without explicit training for asynchronous interaction

Evaluation Highlights

Significant reductions in end-to-end task completion latency on standard function-calling benchmarks (e.g., BFCL-style evaluations)
Reduced completion time on adapted software engineering benchmarks while task accuracy is preserved relative to the synchronous baseline

Signal Assessment

6/10 The paper delivers a practically impactful, model-agnostic systems contribution (a drop-in async execution layer) plus a genuinely interesting empirical finding that LLMs can natively reason over futures, but it is a runtime/orchestration innovation rather than a new architecture or training paradigm.

Methodology

Represent unresolved function call outputs as symbolic 'future' tokens/objects that are injected into the model's context in place of actual results
Decouple the decoding loop from the execution loop so the LLM can continue generating (e.g., issuing further calls or reasoning) while dispatched functions execute asynchronously in the background
Construct a dependency-aware scheduler that identifies which function calls are independent and can run in parallel versus which must wait on a prior future's resolution
Resolve futures transparently when their values are needed downstream, substituting real results back into context while preserving the standard synchronous function-calling protocol format for compatibility
Evaluate against synchronous baselines on function-calling benchmarks and adapted software engineering benchmarks, measuring latency and accuracy

System Components

AsyncFC Runtime

Execution-layer framework sitting between the LLM and tool implementations that orchestrates asynchronous function dispatch and result handling

Future Objects

Symbolic placeholders representing unresolved function call results that are inserted into model context so the LLM can reference pending outputs

Dependency-aware Scheduler

Determines which function calls can be executed concurrently versus which have data dependencies requiring sequential resolution

Decode-Execution Decoupler

Allows LLM token generation to proceed in parallel with in-flight function executions rather than blocking

Protocol Compatibility Layer

Translates the internal asynchronous execution back into the standard synchronous function-calling interface, requiring no changes to existing models or function implementations

Results

Benchmark Type	Synchronous Baseline	AsyncFC	Delta
Standard function-calling benchmarks (e.g., BFCL-style)	Full latency from sequential call execution	Overlapped decode/execution + parallel independent calls	Significant latency reduction, accuracy preserved
Adapted software engineering benchmarks	Baseline end-to-end task completion time	Reduced completion time via async tool execution	Faster task completion with comparable task accuracy

Key Takeaways

AsyncFC can be adopted as a drop-in runtime layer to cut agent latency without retraining models or modifying existing tool/function code, making it low-friction for production agent stacks
Substantial wall-clock savings are achievable in tool-heavy agentic workflows (e.g., coding agents) by exploiting independent tool-call parallelism and decode-execution overlap
The finding that LLMs can natively reason about symbolic futures suggests agent frameworks can build richer asynchronous orchestration (e.g., speculative execution, pipelining) without specialized fine-tuning
Latency optimization for agentic LLM systems can be pursued at the execution/runtime layer as a complement to—or substitute for—model-side or training-side improvements

Abstract

Function calling, also known as tool use, is a core capability of modern LLM agents but is typically constrained by synchronous execution semantics. Under these semantics, LLM decoding is blocked until each function call completes, resulting in increasing end-to-end latency. In this work, we introduce AsyncFC, a pure execution-layer framework that decouples LLM decoding from function execution, enabling overlap between model decoding and function execution as well as inter-function parallelism when dependencies permit. AsyncFC layers over existing models and unmodified function implementations, requiring no fine-tuning or changes to the standard synchronous function-calling protocol. Across standard function-calling benchmarks and adapted software engineering benchmarks, AsyncFC significantly reduces end-to-end task completion time while preserving task accuracy. Furthermore, these results reveal that LLMs possess a native capability to reason over symbolic futures that represent unresolved execution results, enabling an asynchronous paradigm for model-tool interaction.