FaceLLM: A Multimodal Large Language Model for Face Understanding

FaceLLM is a multimodal large language model specialized for facial image understanding, trained on FairFaceGPT—a synthetic instruction dataset generated by prompting ChatGPT with attribute-aware prompts over FairFace images.

Problem Statement

General-purpose MLLMs underperform on facial analysis tasks (expression, demographics, pose, forensic cues) because they are trained on generic vision-language corpora that lack fine-grained facial annotations. The core bottleneck is the absence of large-scale, high-quality face image-text datasets needed to teach MLLMs domain-specific facial reasoning.

Key Novelty

A weakly supervised data generation pipeline that uses ChatGPT with attribute-aware prompts to convert FairFace's demographic labels into rich, diverse question-answer pairs
FairFaceGPT, a new publicly released face-centric visual instruction-tuning dataset covering expression, pose, skin texture, and forensic attributes
FaceLLM, the first MLLM explicitly specialized for facial understanding, achieved via domain-specific fine-tuning rather than architectural changes
A demonstrated blueprint for using LLM-generated synthetic supervision to build domain-specialized MLLMs in data-scarce visual domains

Evaluation Highlights

FaceLLM outperforms baseline/generic MLLMs across multiple face-centric understanding tasks
Reported state-of-the-art performance on facial attribute and expression reasoning benchmarks (exact figures not specified in abstract)

Signal Assessment

4/10 The paper applies a well-established recipe (LLM-generated synthetic instruction data + visual instruction tuning) to a new, underexplored domain (faces), producing a useful public resource and solid empirical gains rather than a fundamentally new modeling or training paradigm.

Methodology

Collect face images with demographic/attribute labels from the FairFace dataset
Design attribute-aware prompts that condition ChatGPT to generate diverse, high-quality question-answer pairs per image (expression, pose, skin texture, forensic details)
Aggregate generated QA pairs into the FairFaceGPT corpus for visual instruction tuning
Fine-tune a base MLLM on FairFaceGPT to produce FaceLLM
Evaluate FaceLLM against generic MLLMs on face-centric understanding tasks

System Components

FairFace dataset

Source of face images with balanced demographic annotations (age, gender, race) used as the visual backbone for data generation

Attribute-aware prompting pipeline

A weakly supervised procedure that queries ChatGPT with structured, attribute-conditioned prompts to produce facial QA pairs

FairFaceGPT dataset

The resulting corpus of synthetic image-question-answer triples covering expression, pose, skin texture, and forensic attributes, released publicly

FaceLLM

An MLLM fine-tuned on FairFaceGPT to specialize in facial structure, expression, emotion, and demographic reasoning

Results

Task/Benchmark	Generic MLLM (Baseline)	FaceLLM	Delta
Facial attribute reasoning	Limited domain-specific accuracy	Improved accuracy	State-of-the-art gain (qualitative)
Expression/emotion understanding	Weak generic performance	Enhanced performance	Notable improvement (qualitative)
Demographic and forensic attribute QA	Not specialized for these cues	Specialized handling via FairFaceGPT training	New capability enabled

Key Takeaways

Domain adaptation of MLLMs via LLM-generated synthetic instruction data is a practical, low-cost strategy when large annotated multimodal datasets don't exist
Attribute-aware prompting of a general LLM (ChatGPT) can effectively bootstrap high-quality QA supervision from existing labeled image datasets like FairFace
Publicly released FairFaceGPT and FaceLLM checkpoints lower the barrier for building or benchmarking face-centric multimodal systems
Because the pipeline relies on FairFace demographic labels, practitioners should scrutinize fairness, bias, and privacy implications before deploying FaceLLM in sensitive human-centric applications

Abstract

Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale an-notated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. To construct the training data, we propose a novel weakly supervised pipeline that uses ChatGPT with attribute-aware prompts to generate high-quality question-answer pairs based on images from the FairFace dataset. The resulting corpus, called FairFaceGPT, covers a diverse set of attributes including expression, pose, skin texture, and forensic information. Our experiments demonstrate that FaceLLM improves the performance of MLLMs on various face-centric tasks and achieves state-of-the-art performance. This work high-lights the potential of synthetic supervision via language models for building domain-specialized MLLMs, and sets a precedent for trustworthy, human-centric multimodal AI systems. FairFaceGPT dataset and pretrained FaceLLM models are publicly available in the project page.