← Back to Paper Radar

UI-UG: A Unified MLLM for UI Understanding and Generation

Hao Yang, Weijie Qiu, Ru Zhang, Zhou Fang, Ruichao Mao, Xiao Lin, Maji Huang, Zhaosong Huang, Teng Guo, Shuoyang Liu, Hai Rao
arXiv.org | 2025
UI-UG is a single, moderately sized MLLM that handles both UI understanding (fine-grained element grounding and parsing) and UI generation (producing renderable UIs via an LLM-friendly DSL). Its training pipeline combines SFT, GRPO, and DPO. The authors report that joint training on understanding and generation improves both capabilities.

Problem Statement

General-purpose MLLMs still struggle with domain-specific UI tasks. They lack fine-grained understanding of modern, dense, complex interfaces, and they generate UIs that are low quality or do not match what humans prefer. Existing UI-specialized models usually address only understanding or only generation. Industrial deployment also needs a practical end-to-end workflow covering representation format, training, rendering, and evaluation, which most prior work does not provide.

Key Novelty

  • A unified MLLM that performs both UI understanding and UI generation, with evidence that joint training benefits both tasks.
  • A task-specific post-training recipe: SFT plus GRPO (RL with verifiable rewards) for fine-grained UI understanding, and DPO for aligning UI generation with human preferences.
  • An industrial workflow built around an LLM-friendly domain-specific language (DSL) for UIs, together with matching training strategies, a rendering pipeline, and evaluation metrics.

Evaluation Highlights

  • Reports SOTA on UI understanding tasks, outperforming both larger general-purpose MLLMs and similarly sized UI-specialized models.
  • UI generation quality is reported as on par with much larger MLLMs at a fraction of the compute cost; ablations indicate that joint understanding+generation training improves both tasks.

Signal Assessment

5/10 The work is a solid, practically oriented contribution that integrates established techniques (SFT, GRPO, DPO, a DSL representation) into a unified UI model with strong efficiency. It is not methodologically new, but the evidence of synergy between understanding and generation, plus the open release, make it useful for practitioners.

Methodology

  1. Design an LLM-friendly DSL that represents UI structure compactly and is easy to render, serving as the generation target and intermediate representation.
  2. Curate modern, complex UI datasets covering understanding tasks (e.g., element detection/grounding, attribute recognition) and generation tasks (e.g., producing DSL from instructions or screenshots).
  3. Run SFT on a base MLLM over the mixed understanding and generation data to build unified capabilities.
  4. Apply GRPO with rule-based or verifiable rewards (e.g., localization and classification accuracy) to sharpen fine-grained UI understanding.
  5. Apply DPO on human-preferred versus dispreferred generated UIs to improve generation quality and aesthetics.
  6. Render the generated DSL into actual UIs and evaluate with domain-specific metrics for understanding accuracy and generation quality.

System Components

Unified UI MLLM backbone

A moderately sized multimodal LLM fine-tuned to perform both UI understanding and UI generation within one model.

LLM-friendly UI DSL

A compact domain-specific language for describing UI layouts and components. It is designed for efficient token generation and deterministic rendering.

SFT stage

Supervised fine-tuning on mixed understanding and generation data to establish the base UI capabilities.

GRPO for understanding

Group Relative Policy Optimization using verifiable rewards to improve fine-grained UI element understanding on complex screens.

DPO for generation

Direct Preference Optimization on preference pairs that steers the model toward human-preferred UI outputs.

Rendering pipeline

Converts the generated DSL into visual UIs for inspection, preference collection, and evaluation.

Evaluation metrics

Domain-specific metrics for assessing understanding accuracy and generated-UI quality in an industrial setting.

Results

Metric/Benchmark Baseline This Paper Delta
UI understanding accuracy Larger general-purpose MLLMs and similarly sized UI-specialized models SOTA Outperforms both baseline groups (exact numbers not in abstract)
UI generation quality Larger general-purpose MLLMs On par Comparable quality
Computational cost (generation) Larger MLLMs Fraction of the cost Substantially cheaper inference
Joint vs. separate training Single-task training Unified understanding+generation Improved accuracy and quality on both tasks

Key Takeaways

  • For structured generation domains like UI, designing a compact, LLM-friendly DSL as the output target can greatly improve generation reliability and efficiency compared with raw HTML/CSS.
  • Match the post-training method to the task: GRPO with verifiable rewards suits perception and grounding tasks, while DPO suits subjective, preference-driven generation quality.
  • Co-training related understanding and generation tasks in one model can produce mutual gains. A specialized mid-sized model can then rival much larger general MLLMs at lower serving cost, which is relevant for GUI agents and design-to-code tools.

Abstract

Although Multimodal Large Language Models (MLLMs) have been widely applied across domains, they are still facing challenges in domain-specific tasks, such as User Interface (UI) understanding accuracy and UI generation quality. In this paper, we introduce UI-UG (a unified MLLM for UI Understanding and Generation), integrating both capabilities. For understanding tasks, we employ Supervised Fine-tuning (SFT) combined with Group Relative Policy Optimization (GRPO) to enhance fine-grained understanding on the modern complex UI data. For generation tasks, we further use Direct Preference Optimization (DPO) to make our model generate human-preferred UIs. In addition, we propose an industrially effective workflow, including the design of an LLM-friendly domain-specific language (DSL), training strategies, rendering processes, and evaluation metrics. In experiments, our model achieves state-of-the-art (SOTA) performance on understanding tasks, outperforming both larger general-purpose MLLMs and similarly-sized UI-specialized models. Our model is also on par with these larger MLLMs in UI generation performance at a fraction of the computational cost. We also demonstrate that integrating understanding and generation tasks can improve accuracy and quality for both tasks. Code and Model: https://github.com/neovateai/UI-UG

Generated automatically from available metadata and abstract on 2026-11-09.