Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning

Tool-use knowledge (schemas, planning heuristics, execution patterns) can be distilled from prompt context into the weights of small LLMs via QLoRA fine-tuning, enabling accurate agentic planning without including tool catalogs in the prompt at inference time.

Problem Statement

Agentic LLM pipelines typically require full tool schemas to be injected into every prompt, which inflates token counts, increases latency/cost, and disproportionately burdens smaller models with limited context efficiency. This makes small LMs impractical as planners in production tool-use systems despite their appeal for low-cost, on-device deployment. The paper asks whether this prompt-time overhead can be eliminated by internalizing tool knowledge directly into model parameters.

Key Novelty

Description-free inference paradigm: fine-tuning small LLMs so tool schemas/knowledge live in weights rather than being re-transmitted in every prompt
Purpose-built ~1,700-example fine-tuning corpus spanning tool knowledge, question-to-plan mappings, and execution-style traces derived from AssetOpsBench
Head-to-head QLoRA comparison of two small backbones (Gemma3 4B/E4B and Qwen3-4B) evaluating not just accuracy but memory, latency, and catastrophic forgetting trade-offs
LoRA rank ablation revealing a tunable quality-vs-retention dial (r=32 maximizes planning quality; smaller ranks better preserve general knowledge)

Evaluation Highlights

AT-F1 of 0.65 (best Gemma fine-tune) vs. 0.47 for an informed baseline given full tool descriptions
Overall LLM-judge planning score of 3.88 (Gemma) and 3.78 (Qwen3-4B) vs. 2.88 for the informed baseline, alongside an 82.6% reduction in input length

Signal Assessment

4/10 The work applies existing QLoRA techniques to a well-motivated, practical problem (removing tool-schema overhead for small-model agentic planning) with solid empirical validation and useful ablations, but it is an applied engineering contribution rather than a new algorithmic or architectural breakthrough.

Methodology

Curate ~1,700 tool-use examples from AssetOpsBench covering tool knowledge, question-to-plan mappings, and execution-style traces
Fine-tune Gemma3 4B (E4B) and Qwen3-4B using 8-bit QLoRA (quantized base weights + low-rank adapters) to internalize tool schemas and planning behavior
Train and evaluate under a description-free protocol where inference prompts omit the tool catalog entirely
Benchmark against an 'informed' unfine-tuned baseline that receives full tool descriptions in-context
Measure structural correctness (AT-F1), LLM-judge planning quality, input token reduction, and system resource usage (memory, inference speed)
Run LoRA rank ablations and general multiple-choice benchmarks to quantify the trade-off between planning quality and catastrophic forgetting

System Components

AssetOpsBench

Primary benchmark providing tool-use tasks, schemas, and evaluation scaffolding for asset-operations planning scenarios

QLoRA fine-tuning

8-bit quantized base model combined with low-rank adapters, used to parameter-efficiently teach tool knowledge to small LLMs

Tool-use training corpus (~1,700 examples)

Curated dataset spanning tool knowledge, question-to-plan mappings, and execution-style traces used for internalization

Description-free inference protocol

Evaluation setup where prompts exclude the tool catalog, testing whether models can plan using only internalized knowledge

Informed baseline

Unfine-tuned model given full tool descriptions in-context, serving as the comparison point for overhead and quality

AT-F1 metric

Structural metric assessing correctness of predicted tool/action sequences relative to ground-truth plans

LLM-judge scoring

Automated LLM-based evaluator producing an overall quality score for generated plans

LoRA rank ablation

Systematic variation of adapter rank (e.g., r=32) to characterize the trade-off between tool-planning quality and retention of general knowledge

Results

Metric	Informed Baseline	Fine-Tuned Model	Delta
AT-F1 (best Gemma run)	0.47	0.65	+0.18 (+38%)
Overall LLM-judge score (best Gemma run)	2.88	3.88	+1.00 (+35%)
Input length / token overhead	Full tool catalog in-context	Tool catalog omitted	-82.6%
Qwen3-4B vs. Gemma efficiency	n/a	Judge score 3.78	62% less memory, 2.5x faster than Gemma (with more catastrophic forgetting)

Key Takeaways

For fixed, known tool catalogs, QLoRA fine-tuning can move tool knowledge from prompt context into model weights, cutting token overhead by over 80% while matching or beating prompt-based baselines.
Small (4B-scale) fine-tuned models can outperform larger-context 'informed' prompting on both structural and LLM-judged planning quality, making them viable low-cost planners for agentic systems.
Model and hyperparameter choice matters beyond raw accuracy: Qwen3-4B offers major memory/speed advantages but forgets general knowledge faster, while LoRA rank (e.g., r=32) lets practitioners explicitly tune the quality-vs-retention trade-off.
This internalization approach is best suited to stable, domain-specific tool sets (e.g., AssetOps-style operations) rather than rapidly changing or open-ended tool ecosystems, since retraining is needed when the tool catalog changes.

Abstract

Large language models are increasingly used as planning components in agentic systems, but current tool-use pipelines often require full tool schemas to be included in every prompt, creating substantial token overhead and limiting the practicality of smaller models. This paper investigates whether tool-use knowledge can be internalized into small language models through parameter-efficient fine-tuning, enabling structured planning without explicit tool descriptions at inference time. Using AssetOpsBench as the primary benchmark, we fine-tune Gemma 4 E4B and Qwen3-4B with 8-bit QLoRA on approximately 1,700 tool-use examples spanning tool knowledge, question-to-plan mappings, and execution-style traces. We evaluate the resulting models under description-free inference, where the prompt omits the tool catalog entirely. The fine-tuned models outperform an informed unfine-tuned baseline that receives full tool descriptions, reducing input length by 82.6\% while improving structural and LLM-judge planning scores. In the best Gemma run, the model achieves an AT-F1 of 0.65 and an overall judge score of 3.88, compared with 0.47 and 2.88 for the informed baseline. Qwen3-4B achieves a strong overall judge score of 3.78 while using 62\% less memory and running 2.5$\times$ faster than Gemma, though it also exhibits greater catastrophic forgetting on general multiple-choice benchmarks. Additional ablations show that LoRA rank controls a quality--retention trade-off, with $r=32$ maximizing planning quality and smaller ranks preserving more general knowledge. These results suggest that, for fixed tool catalogs, QLoRA fine-tuning can shift tool knowledge from prompt context into model weights, substantially reducing inference overhead while maintaining or improving tool-planning quality.