OASIF: An Efficient Obfuscation-Aware Self-Improving Framework for LLM-Based Assembly Code Instruction Following and Comprehension

OASIF is a framework that combines a token-efficient assembly encoder with a three-phase training pipeline (feature alignment, instruction tuning, and self-evolving reinforcement learning) to make code LLMs robust to commercial-grade binary obfuscation while staying within a bounded context budget.

Problem Statement

LLMs applied to binary/assembly analysis are effective on clean code but degrade sharply on commercial VM-based obfuscators (e.g., Themida, VMProtect, Code Virtualizer) that are pervasive in real-world malware and software protection, largely because obfuscated code is extremely long and token-inefficient to represent in-context. Existing fine-tuning pipelines also rely heavily on costly manual verification/labeling, limiting scalability and continual adaptation to new obfuscation variants.

Key Novelty

A token-efficient assembly encoder paired with a lightweight projector that compresses long obfuscated assembly into a bounded-context representation consumable by a pretrained code LLM
A three-phase training recipe (feature-space alignment → supervised instruction fine-tuning → online self-evolving RL with hybrid rewards) tailored to obfuscated code comprehension
A self-evolving RL loop that enables continual model adaptation with minimal manual verification, reducing reliance on labeled obfuscated-code data
Introduction of VMISA-Bench, a challenging out-of-distribution benchmark suite built from three commercial VM-based obfuscators for rigorous generalization testing

Evaluation Highlights

Qwen2.5-Coder-Instruct-14B gains of +15.9pp, +5.8pp, and +16.9pp Success Rate on Code Virtualizer, Themida (v3.0.7), and VMProtect (v3.5) respectively on VMISA-Bench
+9.8 point average improvement on OASIF-Bench across open-source backbones
Stable gains across seven standard Binary Code Similarity Detection (BCSD) benchmarks
Preserved general and domain-relevant capabilities on HumanEval, VulBench, and HumanEval-Decompile, indicating no catastrophic forgetting

Signal Assessment

6/10 The work assembles established techniques (encoder-projector alignment, SFT, RL with hybrid rewards) into a well-engineered, effective system for a genuinely hard and underexplored problem (commercial obfuscation robustness), plus contributes new benchmarks, but does not introduce a fundamentally new learning paradigm.

Methodology

Encode long, obfuscated assembly sequences with a specialized token-efficient encoder to fit a bounded context budget
Project encoder outputs into the pretrained code LLM's embedding space using a lightweight projector (feature-space alignment phase)
Apply supervised instruction fine-tuning to teach instruction-following behavior for assembly comprehension tasks
Run online self-evolving reinforcement learning with hybrid reward signals to continually adapt the model with minimal human labeling
Evaluate generalization on out-of-distribution commercial obfuscators (VMISA-Bench), standard BCSD benchmarks, and general/domain coding benchmarks to check capability retention

System Components

Token-efficient assembly encoder

Compresses long, obfuscated assembly instruction sequences into compact latent representations to fit within a bounded LLM context budget

Lightweight projector

Maps encoder outputs into the pretrained code LLM's input/embedding space, bridging the assembly-representation and language-model modalities

Feature-space alignment (Phase 1)

Aligns the encoder-projector representations with the code LLM's semantic space prior to instruction tuning

Supervised instruction fine-tuning (Phase 2)

Trains the model on instruction-following tasks over assembly comprehension using labeled/curated data

Online self-evolving RL with hybrid rewards (Phase 3)

Continually improves the model via reinforcement learning using combined reward signals, minimizing the need for manual verification

VMISA-Bench

New out-of-distribution benchmark suite built from three commercial VM-based obfuscators (Code Virtualizer, Themida v3.0.7, VMProtect v3.5) for stress-testing robustness

OASIF-Bench

Aggregate benchmark used to report overall average performance gains across obfuscation and comprehension tasks

Results

Benchmark/Obfuscator	Baseline (Qwen2.5-Coder-Instruct-14B)	OASIF	Delta
Code Virtualizer (VMISA-Bench, Success Rate)	Baseline SR	Baseline SR + 15.9pp	+15.9 pp
Themida v3.0.7 (VMISA-Bench, Success Rate)	Baseline SR	Baseline SR + 5.8pp	+5.8 pp
VMProtect v3.5 (VMISA-Bench, Success Rate)	Baseline SR	Baseline SR + 16.9pp	+16.9 pp
OASIF-Bench (average)	Baseline avg	Baseline avg + 9.8	+9.8 pp
Seven standard BCSD benchmarks	Baseline	Consistently improved	Stable positive gains
HumanEval / VulBench / HumanEval-Decompile	Baseline capability	Preserved capability	No significant degradation

Key Takeaways

Pairing an efficient encoder-projector front-end with staged alignment→SFT→RL training is an effective recipe for extending LLM context efficiency to long, dense, security-relevant inputs like obfuscated assembly
Self-evolving RL with hybrid rewards can substantially cut manual annotation costs for continual adaptation to new obfuscator variants, a pattern reusable in other low-label specialized domains
Robustness to commercial-grade obfuscation can be achieved without sacrificing general coding or decompilation capabilities, making the approach practical for real-world reverse-engineering and security tooling
New OOD benchmarks like VMISA-Bench highlight that commercial obfuscators remain a meaningful stress test for code LLMs and should be a standard evaluation axis for binary-analysis research

Abstract

Large Language Models (LLMs) have recently shown promise in automated binary analysis, yet they remain brittle under commercial-grade obfuscation. We present OASIF, an Obfuscation-Aware Self-evolving Instruction-Following framework for obfuscated assembly comprehension. OASIF couples a token-efficient assembly encoder with a lightweight projector to expose long obfuscated code to a pretrained code LLM under a bounded context budget and follows a three-phase training: (i) feature-space alignment, (ii) supervised instruction fine-tuning, and (iii) online self-evolving reinforcement learning with hybrid rewards, enabling continual adaptation with minimal manual verification. On VMISA-Bench, a challenging out-of-distribution suite featuring three commercial VM-based obfuscators, OASIF consistently improves open-source backbones; Qwen2.5-Coder-Instruct-14B attains Success Rate gains of +15.9, +5.8, and +16.9 percentage points (pp) on Code Virtualizer, Themida (v3.0.7), and VMProtect (v3.5), respectively, and improves the OASIF-Bench average by +9.8. OASIF further delivers stable gains across seven standard BCSD benchmarks while preserving general and domain-relevant capabilities on HumanEval, VulBench, and HumanEval-Decompile.