Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
Problem Statement
LLM safety guardrails are largely developed and validated in English, and they degrade sharply when confronted with Chinese linguistic and cultural evasion strategies, leaving deployed models vulnerable to jailbreaks in real-world Chinese-language contexts. There is a shortage of culturally grounded, richly annotated benchmarks covering high-stakes domains (self-harm, drugs, fraud, satire) needed to rigorously test and improve non-English safety alignment.
Key Novelty
- First large-scale human-annotated Chinese adversarial safety benchmark explicitly targeting culturally-specific evasion tactics (Pinyin romanization, character decomposition, slang, hedging tone)
- A novel nine-category obfuscation taxonomy that systematically classifies Chinese-language evasion techniques for red-teaming and analysis
- Fine-grained annotation schema combining a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), risk-level ratings, and annotator rationale rather than simple binary safe/unsafe labels
- Explicit framing of open methodological tensions (train/eval data blurring, real-world domain coverage, scale vs. cultural expertise) that are broadly relevant to safety benchmark design
Evaluation Highlights
- Dataset scale: 1,897 adversarial Chinese prompts across 4 high-stakes domains, with 1,544 carrying complete gold-standard annotations
- Annotation richness: every fully-annotated item includes a 3-class response label, one of nine obfuscation categories, a risk-level rating, and rationale text, enabling multi-dimensional benchmarking of LLM safety behavior
Signal Assessment
Methodology
- Curate adversarial Chinese prompts across four high-stakes domains (self-harm/violence, drug/illicit trade, fraud, satire) reflecting real-world risk scenarios
- Develop a nine-category obfuscation taxonomy capturing Chinese-specific evasion techniques (e.g., Pinyin romanization, character decomposition, internet slang, hedging tone)
- Conduct human annotation assigning a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), obfuscation category, risk-level rating, and written rationale per prompt
- Apply quality control to distinguish fully gold-standard annotated entries (1,544) from the broader raw prompt pool (1,897)
System Components
Core benchmark of 1,897 adversarial Chinese prompts spanning self-harm/violence, drug/illicit trade, fraud, and satire domains
Classification scheme for Chinese-specific evasion techniques used to bypass safety filters
REFUSE, SAFE-REDIRECT, RESPOND labels capturing nuanced model behavior beyond binary safe/unsafe judgments
Severity annotation indicating the real-world harm potential of each prompt
Free-text justification accompanying each gold-standard label to support interpretability and downstream analysis
Results
| Aspect | Prior/English-centric Benchmarks | ChiSafe-PAS | Delta |
|---|---|---|---|
| Cultural/Linguistic Coverage | Predominantly English-centric safety evaluation | Chinese-specific adversarial prompts with cultural evasion tactics | Fills a major cross-lingual safety gap |
| Dataset Size | Not specified (varies by benchmark) | 1,897 prompts total, 1,544 fully gold-annotated | New dedicated resource |
| Domain Coverage | Often narrow or generic risk categories | 4 high-stakes domains (self-harm/violence, drugs, fraud, satire) | Broader real-world risk grounding |
| Annotation Granularity | Typically binary safe/unsafe labels | 3-class label + 9-category taxonomy + risk rating + rationale | Substantially richer, more interpretable labeling |
Key Takeaways
- Safety alignment validated only in English cannot be assumed robust in Chinese; practitioners must explicitly test against Pinyin romanization, character decomposition, slang, and hedging-based evasion
- The nine-category obfuscation taxonomy offers a reusable framework for red-teaming and diagnosing failure modes of multilingual safety filters
- Fine-grained labels (REFUSE/SAFE-REDIRECT/RESPOND plus risk level) enable evaluation of over-refusal and under-refusal behavior, not just binary jailbreak success
- High-quality human annotation with cultural expertise and rationale is positioned as difficult to substitute with scale alone, a caution for teams relying purely on synthetic or LLM-generated safety data
Abstract
When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural bound-aries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.