UI-UG: A Unified MLLM for UI Understanding and Generation
Problem Statement
General-purpose MLLMs still struggle with domain-specific UI tasks. They lack fine-grained understanding of modern, dense, complex interfaces, and they generate UIs that are low quality or do not match what humans prefer. Existing UI-specialized models usually address only understanding or only generation. Industrial deployment also needs a practical end-to-end workflow covering representation format, training, rendering, and evaluation, which most prior work does not provide.
Key Novelty
- A unified MLLM that performs both UI understanding and UI generation, with evidence that joint training benefits both tasks.
- A task-specific post-training recipe: SFT plus GRPO (RL with verifiable rewards) for fine-grained UI understanding, and DPO for aligning UI generation with human preferences.
- An industrial workflow built around an LLM-friendly domain-specific language (DSL) for UIs, together with matching training strategies, a rendering pipeline, and evaluation metrics.
Evaluation Highlights
- Reports SOTA on UI understanding tasks, outperforming both larger general-purpose MLLMs and similarly sized UI-specialized models.
- UI generation quality is reported as on par with much larger MLLMs at a fraction of the compute cost; ablations indicate that joint understanding+generation training improves both tasks.
Signal Assessment
Methodology
- Design an LLM-friendly DSL that represents UI structure compactly and is easy to render, serving as the generation target and intermediate representation.
- Curate modern, complex UI datasets covering understanding tasks (e.g., element detection/grounding, attribute recognition) and generation tasks (e.g., producing DSL from instructions or screenshots).
- Run SFT on a base MLLM over the mixed understanding and generation data to build unified capabilities.
- Apply GRPO with rule-based or verifiable rewards (e.g., localization and classification accuracy) to sharpen fine-grained UI understanding.
- Apply DPO on human-preferred versus dispreferred generated UIs to improve generation quality and aesthetics.
- Render the generated DSL into actual UIs and evaluate with domain-specific metrics for understanding accuracy and generation quality.
System Components
A moderately sized multimodal LLM fine-tuned to perform both UI understanding and UI generation within one model.
A compact domain-specific language for describing UI layouts and components. It is designed for efficient token generation and deterministic rendering.
Supervised fine-tuning on mixed understanding and generation data to establish the base UI capabilities.
Group Relative Policy Optimization using verifiable rewards to improve fine-grained UI element understanding on complex screens.
Direct Preference Optimization on preference pairs that steers the model toward human-preferred UI outputs.
Converts the generated DSL into visual UIs for inspection, preference collection, and evaluation.
Domain-specific metrics for assessing understanding accuracy and generated-UI quality in an industrial setting.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| UI understanding accuracy | Larger general-purpose MLLMs and similarly sized UI-specialized models | SOTA | Outperforms both baseline groups (exact numbers not in abstract) |
| UI generation quality | Larger general-purpose MLLMs | On par | Comparable quality |
| Computational cost (generation) | Larger MLLMs | Fraction of the cost | Substantially cheaper inference |
| Joint vs. separate training | Single-task training | Unified understanding+generation | Improved accuracy and quality on both tasks |
Key Takeaways
- For structured generation domains like UI, designing a compact, LLM-friendly DSL as the output target can greatly improve generation reliability and efficiency compared with raw HTML/CSS.
- Match the post-training method to the task: GRPO with verifiable rewards suits perception and grounding tasks, while DPO suits subjective, preference-driven generation quality.
- Co-training related understanding and generation tasks in one model can produce mutual gains. A specialized mid-sized model can then rival much larger general MLLMs at lower serving cost, which is relevant for GUI agents and design-to-code tools.
Abstract
Although Multimodal Large Language Models (MLLMs) have been widely applied across domains, they are still facing challenges in domain-specific tasks, such as User Interface (UI) understanding accuracy and UI generation quality. In this paper, we introduce UI-UG (a unified MLLM for UI Understanding and Generation), integrating both capabilities. For understanding tasks, we employ Supervised Fine-tuning (SFT) combined with Group Relative Policy Optimization (GRPO) to enhance fine-grained understanding on the modern complex UI data. For generation tasks, we further use Direct Preference Optimization (DPO) to make our model generate human-preferred UIs. In addition, we propose an industrially effective workflow, including the design of an LLM-friendly domain-specific language (DSL), training strategies, rendering processes, and evaluation metrics. In experiments, our model achieves state-of-the-art (SOTA) performance on understanding tasks, outperforming both larger general-purpose MLLMs and similarly-sized UI-specialized models. Our model is also on par with these larger MLLMs in UI generation performance at a fraction of the computational cost. We also demonstrate that integrating understanding and generation tasks can improve accuracy and quality for both tasks. Code and Model: https://github.com/neovateai/UI-UG