URDUMMLU: A Massive Multitask Benchmark for Urdu Language Understanding Ahmer Tabassum∗1 Sarfraz Ahmad∗1 Hasan Iqbal∗1 Owais Aijaz1 Momina Ahsan1 Preslav Nakov1 1 MBZUAI {ahmer.tabassum, sarfraz.ahmad, hasan.iqbal}@mbzuai.ac.ae ç Project Ɓ UrduMMLU a Code Ô Leaderboard
arXiv:2606.07167v1 [cs.CL] 5 Jun 2026
Abstract Meaningful multilingual evaluation must test models in the target language and educational context. Urdu, spoken by more than 230 million people, lacks a broad MMLU-style benchmark built from native educational sources. We introduce URDUMMLU, a benchmark of 26,431 Urdu MCQs across 26 subjects and five domains, collected from native Urdu MCQ banks and public examination PDFs. Unlike translation-based resources, URDUMMLU covers both standard academic subjects and Urdu- and region-specific content. We label the exam-derived portion through dual human annotation with strict consensus filtering. We evaluate 30 LLMs under English and Urdu prompts, yielding 60 zero-shot evaluations, and further evaluate four open-source LLMs under multiple few-shot settings across both prompt languages. Gemini-3.5-Flash performs best, reaching 90.20% and 90.34% accuracy, while no other model exceeds 85%. The strongest open-source model trails by 7.79 and 8.92 points, and many models lose 25 to 40 points on Urdu-centered Humanities subjects compared with STEM. Few-shot prompting yields only modest gains. URDUMMLU shows that Urdu knowledge remains uneven in current LLMs, especially for regionally grounded content.
1
Introduction
Evaluating the knowledge and reasoning abilities of Large Language Models (LLMs) has become central to Natural Language Processing (NLP). Benchmarks such as MMLU (Hendrycks et al., 2021) and MMLU-PRO (Wang et al., 2024) are widely used for this purpose, but they are in English and largely reflect English-language educational and cultural contexts. This limits their ability to test whether model competence transfers across language, script, and regional knowledge. As a result, these benchmarks provide only a partial view of model performance in multilingual and culturally diverse settings. ∗
Equal contribution.
The issue is especially important for Urdu, a language spoken by over 230 million people, with a long literary and educational tradition, but limited broad-coverage evaluation resources. Existing Urdu benchmarks focus mainly on reading comprehension, syntactic diagnostics, task-level NLP evaluation, or translated reasoning benchmarks (Kazi and Khoja, 2026; Kazi et al., 2025; Adeeba et al., 2025; Tahir et al., 2025; Shafique et al., 2026). Multilingual benchmarks that include Urdu, such as MMLU-PROX (Xuan et al., 2025), GLOBALMMLU (Singh et al., 2025), and INDICMMLUPRO (KJ et al., 2025), also rely mainly on translated questions. As a result, they only partially capture knowledge grounded in Urdu-medium education, Urdu literature, local history, religious studies, and civic curricula. Recent language-specific benchmarks such as ARABICMMLU (Koto et al., 2024), CMMLU (Li et al., 2024), INDOMMLU (Koto et al., 2023), KMMLU (Son et al., 2025), and KAZMMLU (Togmanov et al., 2025) highlight the importance of evaluation grounded in local educational material. Following this direction, we introduce URDUMMLU, the first broad-coverage, natively written MMLU-style benchmark for Urdu. URDUMMLU contains 26,431 MCQs across 26 subjects and five domains, collected from Urdu MCQ banks and public SSC/HSSC examination PDFs, and combines answer-labeled questions with examderived questions annotated through dual human annotation and strict consensus filtering, and covers both standard academic subjects and Urdu- and region-specific content. Figure 1 summarizes the resulting subject distribution. We evaluate 30 open-source and closed-source LLMs on URDUMMLU under English and Urdu prompts, yielding 60 zero-shot evaluations, and further evaluate four open-source LLMs in 1-, 3, and 5-shot settings. Gemini-3.5-Flash (Google DeepMind, 2026) achieves the highest accuracy at 90.20% and 90.34%, while the strongest opensource model trails by 7.79 and 8.92 points.
C
15
Blank Normalization
Schema Canonicalization
Consolidation
Exam past-paper PDFs via Gemini vision extraction
Normalize all blank variants to exactly four underscores
Unify correct_option (full text) and correct_key (A–D)
Merge scraped + annotated MCQs into one provenance-tagged set
03
06
Gen e
11
14
04
05
ci
al
Sc
ie n c e s
ic
ation Educ
Keep dual-consensus only; prefer longer edits on conflict
He
Batches assigned to exactly two rolematched annotators
Economics
NFKC plus Arabic letters remapped to Urdu ( ك,)ي
So cio log y
iv
PE h
&
C
Character Normalization
ar
Curly to straight quotes, collapse runs, repair missing openers
m
alt
13
Annotation Consolidation
G
12
Annotation Assignment
n ista Pak dies Stu
Quotation Normalization
Is St lam ud ic ies
du
s
Two pass: exact match, then nword-Jaccard fuzzy match
Urdu Language
m
Deduplication
Stratified cap per subdomain × source × length tier
e merc s Com ffair tl. A & In y ent r r log Cu ho c y y Ps ph ra og Ge
ra
Subsampling
Map ASCII punctuation to Urdu equivalents ( ۔،؟، ،، …)
Chemistry
Ur
Punctuation Normalization
Add RTL marks; wrap Latin/digit fragments in LRI…PDI markers
Scie nce
Biology
So
Bidirectional Text
ral
ci en ce
ati cs
ities man Hu
OCR Documents
rS
STE M
10
n Professio
07
er Oth
02
pu te
th em
ent
Ma
om
velopm
Anonymize, renumber, export slim schema to Hugging Face
Prof. Studies
Finalization
Drop malformed MCQs; normalize option lists to A–D
ics
Structural Cleanup
Strip redundant ' ')اﻟﻒor '1.' prefixes from option text
Li Ur te d u ra tu re
16
ys Ph
Parse websites into structured MCQ records
09
al ner e Ge ledg ow Kn
Website Scrapping
08
Option Prefix Stripping
Prof. De
01
Figure 1: The 16-stage UrduMMLU construction pipeline (left) and the resulting 26,431-MCQ benchmark broken down by 5 domains and 26 subdomains (right); wedge size is proportional to MCQ count.
Across models, performance remains substantially higher on STEM subjects than on Urducentered Humanities, with many systems losing 25 to 40 points on Urdu literature, Urdu language, and Islamic studies. These results show that strong English-centered benchmark performance does not reliably transfer to Urdu educational and cultural knowledge. They also highlight the need for benchmarks that better capture linguistic and cultural diversity beyond English. The main contributions of this work are: • We introduce URDUMMLU, a natively written Urdu MMLU-style benchmark with 26,431 MCQs across 26 subjects and five domains, covering both standard academic subjects along with Urdu- and region-specific knowledge. • We produce human-annotated gold answers for the exam-derived portion of the benchmark using dual annotation and strict consensus filtering. • We conduct 60 zero-shot evaluations across 30 open-source and closed-source LLMs under English and Urdu prompt settings, and 24 additional few-shot evaluations across four open-source LLMs. • We release the dataset and evaluation code to support future work on Urdu-capable language models.
2 Related Work Urdu evaluation resources: Existing Urdu resources cover reading comprehension, crosslingual question answering , syntax, and tasklevel NLP. UQUAD+ (Kazi and Khoja, 2026) provides annotated Urdu reading comprehension, while Kazi et al. (2025) study Urdu-English QA with UQUAD1.0 (Kazi and Khoja, 2021) and SQUAD2.0 (Rajpurkar et al., 2018). URBLIMP (Adeeba et al., 2025) evaluates Urdu syntax via minimal pairs, and Tahir et al. (2025) benchmark models across Urdu NLP tasks. For reasoning, URDUBENCH (Shafique et al., 2026) translates MGSM (Shi et al., 2023), COMMONSENSEQA (Talmor et al., 2019), OPENBOOKQA (Mihaylov et al., 2018), and MATH-500 (Lightman et al., 2024) into Urdu, and URDUFACTCHECK (Ahmad et al., 2025) targets factual QA. These resources remain taskspecific, diagnostic, or translation-derived. In contrast, URDUMMLU evaluates broad educational knowledge using questions originally written for Urdu-speaking educational settings. Multilingual benchmarks: MMLU (Hendrycks et al., 2021) and MMLU-PRO (Wang et al., 2024) are widely used for evaluating general knowledge and reasoning. Several multilingual extensions adapt these benchmarks through translation. MMLU-PROX (Xuan et al., 2025) extends MMLUPRO to 29 languages using LLM-based translation and expert review, while GLOBAL-MMLU (Singh et al., 2025) studies cultural and linguistic bias in multilingual evaluation.
INDICMMLU-PRO (KJ et al., 2025) adapts MMLU-PRO to nine Indic languages, including Urdu. Other multilingual exam-based resources, such as EXAMS (Hardalov et al., 2020), INCLUDE (Romanou et al., 2025), and MILU (Verma et al., 2025), collect examination questions across multiple languages and regions. However, Urdu still appears primarily in translated or cross-lingual settings rather than through a dedicated native benchmark, limiting fair knowledge assessment in cultural context. Localized MMLU-style benchmarks: Recent work increasingly builds MMLU-style benchmarks from local educational material instead of translating English benchmarks. ARABICMMLU (Koto et al., 2024), CMMLU (Li et al., 2024), INDOMMLU (Koto et al., 2023), KMMLU (Son et al., 2025), and KAZMMLU (Togmanov et al., 2025) show that language-specific curricula and regional cultural knowledge remain important for evaluating LLMs beyond English. URDUMMLU follows this direction for Urdu by combining regional SSC/HSSC examination material, native Urdu MCQ banks, human annotation for exam-derived questions, and broad coverage of both standard academic subjects and Urdu- and Pakistan-specific knowledge.
3
URDUMMLU
URDUMMLU is a broad-coverage benchmark for evaluating knowledge and reasoning in Urdu. Unlike translation-based multilingual benchmarks, URDUMMLU draws its questions directly from Urdu educational and examination material. The benchmark contains 26,431 MCQs across 26 subdomains and five domains, covering both standard academic subjects and Urdu- and region-specific content such as Urdu literature, Urdu language, Islamic studies, and Pakistan studies. Appendix A.1 and Figure 7 provide detailed benchmark statistics and subdomain distributions. We collect questions from Urdu MCQ banks and public SSC/HSSC examination PDFs, and produce gold answers for exam-derived questions through dual human annotation with strict consensus filtering. We design URDUMMLU around broad subject coverage, faithful representation of Urdu educational material, and reliable multiple-choice evaluation through clean text extraction, normalized metadata, and verified gold labels. Figure 1 summarizes the overall construction pipeline.
3.1 Data Sources We collect candidate questions from two source families. The first consists of public SSC and HSSC examination PDFs from Pakistan covering school- and high school-level subjects such as mathematics, physics, chemistry, biology, computer science, Urdu, Islamic studies, Pakistan studies, and economics. The second consists of native Urdu MCQ websites that publish answerlabeled questions for examination preparation. Together, these sources allow URDUMMLU to cover both globally shared academic subjects and region-specific educational content taught in Urdumedium curricula. We treat all collected items as candidates and include them in the final benchmark only after cleaning, answer annotation or verification, deduplication, and release packaging. 3.2 Raw MCQ Extraction For PDF-based sources, we use a multi-stage extraction pipeline to recover Urdu MCQs from heterogeneous examination layouts. We first convert each PDF into page images and use Claude Opus 4.7 (Anthropic, 2026) as OCR to classify each page, filtering out English-only pages, non-MCQ pages, answer keys, and unrelated material. For the remaining pages, we extract question stems, answer options, source metadata, and page-level provenance using a vision-language OCR procedure. We design the extraction prompt specifically for Urdu examination documents. The prompt preserves Urdu question text, answer options, poetry, quotations, and other context required to answer the question correctly. In bilingual material, we ignore English text unless it forms a structural part of the Urdu question, and we discard unreadable questions rather than reconstructing missing content. For web-based sources, we directly scrape question stems, answer options, category labels, and answer keys when available. 3.3 Metadata and Schema Normalization The collected sources use heterogeneous category names, grade labels, and answer formats, so we normalize all examples into a unified representation. We map source-specific labels to a controlled set of subdomains. For example, we map variants such as Everyday Science and General Science to general science, and mathematics-related labels such as maths, General Mathematics, and riazi to mathematics.
For curriculum-derived material, we normalize grade labels into regional examination levels: Grade 9 to SSC-I, Grade 10 to SSC-II, Grade 11 to HSSC-I, and Grade 12 to HSSC-II. Table 13 in Appendix C.1 summarizes the final domain hierarchy, subdomains, acronyms, and examination levels covered in URDUMMLU. We also canonicalize the MCQ schema to support consistent evaluation. Each released item stores a question, four answer options, normalized domain and subdomain labels, academic level, source metadata, and answer annotations. We remove ambiguous index-based answer fields because different sources follow different option-ordering conventions. 3.4
Cleaning and Quality Control
We apply several cleaning and validation steps to reduce noise from OCR, web scraping, and heterogeneous source formatting. First, we normalize Urdu text representation through right-to-left display normalization, punctuation and quote normalization, standardization of fill-in-the-blank markers, and Unicode normalization for visually similar Arabic and Urdu codepoints. We then enforce structural validity by removing items with missing, empty, duplicate, or malformed answer options, discarding examples with invalid option counts, and standardizing option fields into a consistent schema format. Next, we deduplicate the candidate pool. We merge exact duplicates with consistent answers while preserving source provenance and discard duplicate groups with conflicting labels. To handle OCR and wording variations, we additionally apply conservative near-duplicate filtering based on high question-token overlap together with answer-option overlap. Finally, we remove residual non-Urdu artifacts, including a small number of English OCR artifacts that survived earlier filtering stages. We use the resulting cleaned pool for annotation, answer verification, and final benchmark construction. 3.5
Human Annotation
The exam-derived portion of URDUMMLU did not include answer keys, so we produced gold labels through annotation. We organized annotation batches by subdomain and assigned each item to two annotators with relevant subject familiarity. Annotators selected the correct answer, marked questions as unsure, flagged problematic items, and could suggest light corrections to question text, answer options, and subdomain labels.
Provenance
Count
%
Answer-labeled web MCQs Exam-extracted annotated MCQs
13,672 12,759
51.7 48.3
Total
26,431
100.0
Table 1: Composition of the final URDUMMLU benchmark by source type. Web-derived questions use human-validated published answers, while examderived questions use dual human annotation with strict consensus filtering.
Seventeen annotators participated in the process; 94.1% identified Urdu as their native language, and most held either a bachelor’s degree (47.1%) or a master’s degree (41.2%). Appendix B.1 reports full demographic and satisfaction details. We applied a strict consensus rule and retained an item only when both annotators selected the same valid answer without flags or unsure labels. This process helped ensure high annotation quality and label reliability. In total, 17,565 exam-extracted MCQs entered annotation, and 14,459 satisfied the consensus criteria. The main exclusion reasons included answer disagreement (1,611 items), flags (1,247), unsure selections (243), and incomplete annotations (5). Annotators also corrected 141 domain labels during the process. Overall observed agreement reached 89.98%, with simplified Cohen’s κ = 0.8663. After verification, deduplication, and release packaging, the final benchmark retained 12,759 human-annotated exam-derived questions. Annotators additionally verified the correctness of pre-existing answer labels for web-derived MCQs. 3.6 Final Benchmark The final release of URDUMMLU contains 26,431 Urdu MCQs after cleaning, annotation, answer verification, deduplication, and release packaging. Answer-labeled web sources contribute 13,672 questions, while exam-derived sources contribute 12,759 questions annotated through dual human labeling and strict consensus filtering (Table 1). Appendix A reports statistics for the larger cleaned candidate pool before annotation and final selection. URDUMMLU spans 26 subdomains grouped into five domains: STEM, Humanities, Social Sciences, Profession, and Other. Humanities and Social Sciences constitute the largest portions of the benchmark, reflecting strong coverage of Urdu language, Urdu literature, Islamic studies, Pakistan studies, and related educational content.
Group
Question
Answer
Domain
Humanities Social Sciences STEM Other Profession
41.8 43.0 46.0 48.7 41.3
10.2 8.6 7.9 9.7 8.2
Level
SSC-I SSC-II HSSC-I HSSC-II
43.5 44.0 41.8 42.8
9.3 9.0 9.2 9.1
Table 2: Average character length of questions and correct answers in URDUMMLU, grouped by domain and academic level. Values denote mean character counts. Domain
Questions
Subdomains
STEM Humanities Social Sciences Profession Other
5,113 11,010 7,968 975 1,365
6 6 11 2 1
Total
26,431
26
Table 3: Distribution of questions and subdomains across the five domains in URDUMMLU.
Table 3 summarizes the domain-level composition, while Table 13 lists the corresponding subdomains and academic levels. Table 2 reports average question and answer lengths, and Appendix C describes the dataset schema.
4
Experiments
We evaluate URDUMMLU with generation-based protocols that require each model to select an answer option for an Urdu MCQ. We run a large zero-shot evaluation across 30 open- and closedsource LLMs using both English and Urdu instruction prompts. We also run a focused few-shot study on four open-source LLMs using 1-, 3-, and 5-shot settings in both prompt languages. All evaluations use the same benchmark format and accuracy metric, which allows direct comparison across model families, prompt languages, and shot settings. 4.1
Models, Prompting, and Decoding
We evaluate 30 LLMs spanning a broad range of model sizes, access regimes, and training backgrounds, including proprietary API systems, open-weight multilingual instruction-tuned models, compact models, mixture-of-experts architectures, reasoning-oriented variants, and Urdu- or regionally specialized models.
Table 14 in Appendix D.1 lists the full model roster. This setup allows us to compare open-source and closed-source systems and examine transfer to native Urdu educational content. We evaluate each model with English and Urdu prompt templates while keeping the Urdu question stem, answer options, and response format fixed. The two settings differ only in the instruction language and field labels. Appendix D.2 provides the prompt templates in Figures 16 and 17. We use temperature 0 whenever deterministic decoding is available and otherwise follow provider-specific reasoning settings. We set the maximum output length to 4096 tokens, batch API requests with a concurrency of 10, and decode locally evaluated Hugging Face models greedily. 4.2 Evaluation Protocols Zero-shot evaluation: We use a generationbased zero-shot protocol in which each input contains the domain, subdomain, academic level, Urdu question, and labeled answer options. We evaluate all 30 models under both English and Urdu prompt templates, resulting in 60 zero-shot runs. Since the question and answer options remain unchanged across settings, this protocol isolates the effect of instruction language on the same Urdu MCQs. Few-shot evaluation: We conduct a controlled few-shot study on four open-source LLMs under 1-, 3-, and 5-shot settings with both English and Urdu prompts, yielding 24 runs. We reserve 200 validated MCQs as a demonstration pool for lmevaluation-harness (Gao et al., 2024), which we use only for prompt construction, demonstration sampling, and execution management. All models generate structured answers that we parse and compare against the gold labels. 4.3 Evaluation Measure We use accuracy as the primary evaluation measure by comparing the generated answer with the gold label. Alongside accuracy, we report the invalid-output rate, defined as the percentage of unparsable, malformed, or error outputs. We further analyze results by domain, subdomain, academic level, prompt language, and model category to examine performance differences across standard academic and Urdu- or region-specific subjects.
English Prompt (Accuracy % ↑)
Model STEM
H
SS
P
O
Urdu Prompt (Accuracy % ↑)
Overall
Inv% ↓
H
SS
P
O
Overall
Inv% ↓
67.32 57.73 63.25 56.10 55.71 58.12 56.55 63.25
88.95 75.41 82.06 73.43 74.55 84.43 72.49 81.30
89.44 77.82 84.10 71.28 74.46 84.50 72.62 79.59
84.88 71.79 78.39 71.65 70.55 77.99 69.01 80.81
81.42 70.23 76.39 67.00 69.70 75.46 68.20 75.83
0.3 <0.1 <0.1 <0.1 0 0.4 <0.1 <0.1
24.53 28.74 26.56 29.24 69.02 51.79 37.24 46.49 57.25 71.37 37.08 33.83 68.75 74.37
25.83 28.76 27.70 30.88 48.08 38.27 29.32 37.61 43.07 45.74 28.70 27.25 43.00 30.87
25.34 33.84 30.35 33.35 60.53 48.69 39.97 49.85 52.07 62.74 38.97 31.75 53.30 56.54
23.49 32.60 32.64 33.45 62.15 50.15 38.36 49.54 52.26 61.54 38.67 34.22 53.23 57.48
24.47 27.51 26.75 30.73 55.82 46.23 38.02 44.98 48.86 57.00 35.85 30.57 47.69 49.38
25.27 30.57 28.62 31.36 56.80 44.88 34.85 43.84 49.16 56.99 34.15 30.31 51.70 48.97
<0.1 24.8 74.2 34.6 0 0 0 0 <0.1 0 <0.1 0.4 0 0.5
0.1 0 <0.1 0.1 0 0
91.96 96.26 97.09 97.81 88.25 97.40
59.31 72.69 74.38 85.31 62.35 74.82
77.06 87.53 90.10 92.14 78.45 89.37
78.26 87.18 90.36 91.38 79.59 87.08
74.29 85.86 85.57 91.72 75.09 83.74
72.45 82.94 84.68 90.34 73.51 84.53
<0.1 0 <0.1 <0.1 0 0.4
0 0.6
36.26 33.27
32.72 29.00
37.78 36.26
39.34 37.67
42.55 42.93
35.52 32.68
11.3 12.6
STEM
Open-source Models: > 25B Parameters DeepSeek-V4-Flash Gemma-4-26B-A4B-IT Gemma-4-31B-IT LLaMA-3.3-70B Qwen3.6-27B Qwen3.6-35B-A3B LLaMA-4-Scout-17B-16E LLaMA-4-Maverick-17B-128E
97.49 85.92 93.33 81.32 90.46 88.77 84.14 91.98
68.71 57.43 63.62 56.30 55.95 56.81 57.03 63.27
90.05 73.83 81.70 75.45 73.19 73.52 71.20 80.11
89.85 75.49 83.69 76.51 74.77 75.49 74.15 81.64
86.44 70.55 79.49 73.77 69.60 69.82 69.52 80.66
BLOOMZ-1.1B BLOOMZ-1.7B BLOOMZ-3B BLOOMZ-7B Gemma-2-9B-IT Gemma-3-4B-IT LLaMA-3.2-3B LLaMA-3.1-8B Ministral-3-3B Ministral-3-8B Phi-4-mini Phi-3.5-mini Qwen3-4B-Instruct-2507 Qwen3-8B
23.52 28.94 27.89 27.75 67.08 49.97 37.12 46.96 55.04 67.81 37.07 33.76 68.61 70.70
27.21 27.70 30.25 27.81 47.21 37.87 26.78 36.38 43.69 45.27 28.85 22.89 42.33 39.21
24.39 33.33 33.26 33.56 58.40 47.68 38.06 49.66 48.99 58.99 38.10 30.90 51.48 53.99
26.06 37.17 32.34 31.83 60.00 47.79 36.00 48.51 49.08 59.59 40.41 32.21 52.62 56.62
25.63 31.43 28.39 28.11 54.58 45.57 35.60 44.54 47.91 54.43 32.67 28.28 47.84 50.18
Claude-Haiku-4.5 Claude-Sonnet-4.6 Gemini-3.1-Flash-Lite Gemini-3.5-Flash GPT-5.4-mini GPT-5.4
90.49 96.34 96.85 97.75 88.34 95.13
58.57 72.69 74.20 84.98 62.82 69.29
75.86 87.36 90.01 92.15 77.52 86.35
75.90 87.18 90.26 92.10 79.08 85.85
74.14 86.01 86.08 91.43 75.24 84.10
Qalb-1.0-8B Alif-1.0-8B
38.18 41.09
29.99 25.74
37.68 41.40
40.31 39.87
39.56 41.04
82.41 69.23 76.38 68.56 69.22 69.39 67.82 75.47
0.1 <0.1 <0.1 0 0 0 0 <0.1
97.57 87.21 93.86 78.39 91.12 96.32 85.59 92.38
Open-source Models: ≤ 25B Parameters 25.52 30.19 30.68 29.73 55.28 43.93 32.98 43.30 47.90 54.77 33.85 28.03 50.84 50.97
0.5 2.5 6.5 11.2 0 0 0 0 <0.1 0 <0.1 0 0 0
Proprietary Models 71.40 82.91 84.56 90.20 73.43 80.81
Urdu Models 34.77 34.72
Table 4: Model performance on URDUMMLU. Accuracy (%) under English and Urdu prompts across five domains and overall average. Inv% denotes the percentage of unparsable or malformed outputs (lower is better). Boxed values mark the best overall score per column, while bold values indicate the best score within each model group.
5
Results
We first analyze overall zero-shot performance across all evaluated models and then examine how performance changes across domains, prompt languages, model scales, and few-shot settings. Table 4 summarizes zero-shot accuracy on URDUMMLU under English and Urdu prompts, together with invalid-output rates. We focus on four main findings: a small set of models performs strongly, STEM transfers much better than Urducentered Humanities, prompt language has limited effect for most models, and few-shot prompting gives modest but insufficient gains. Appendix E.1 provides per-subdomain and per-level results. 5.1
Overall Model Performance
Gemini-3.5-Flash leads all models with 90.20% accuracy under the English prompt and 90.34% under the Urdu prompt, while no other model exceeds 85%. Gemini-3.1-Flash-Lite (Google, 2026a), GPT-5.4 (Singh et al., 2026), ClaudeSonnet-4.6 (Anthropic, 2026), and DeepSeek-V4Flash (DeepSeek-AI, 2026) are the next-best.
With DeepSeek-V4-Flash giving the strongest open-source result at 82.41% under English prompt and 81.42% under Urdu prompt. Even so, it trails Gemini-3.5-Flash by 7.79 and 8.92 points. Performance drops sharply outside this toptier. In the ≤ 25B open-source group, Gemma-29B-IT (Team et al., 2024) and Ministral-3-8B (Liu et al., 2026) lead at roughly 55–57%, while Qwen34B and Qwen3-8B (Yang et al., 2025) remain near 50%. BLOOMZ (Muennighoff et al., 2023) models stay close to the 25% random baseline despite multilingual pretraining that includes Urdu. The two Urdu-specific models, Qalb-1.0-8B (Hassan et al., 2026) and Alif-1.0-8B (Shafique et al., 2025), also remain below 36%, showing that Urdufocused tuning alone does not produce strong broad-coverage Urdu knowledge. 5.2 Domain-Level Performance Domain-level results reveal the clearest pattern in URDUMMLU. Nearly every model that performs above chance scores highest on STEM and lowest on Humanities.
100
97.8
97.6
100
STEM Humanities
90
85.3
English prompt Urdu prompt
90.3 81.4
80 67.3
70
69.0
60 48.1
50 40
36.3
32.7
30
70 60
10
DeepSeek-V4-Flash
Gemma-2-9B-IT
Qalb-1.0-8B
Open-source > 25B
Open-source ≤ 25B
Urdu-targeted
34.8
35.5
30 20
Proprietary
56.8
40
10
Gemini-3.5-Flash
55.3
50
20
0
90.2
82.4
80
Overall accuracy (%)
Accuracy (%) under Urdu prompt
90
0
Gemini-3.5-Flash
DeepSeek-V4-Flash
Gemma-2-9B-IT
Qalb-1.0-8B
Proprietary
Open-source > 25B
Open-source ≤ 25B
Urdu-targeted
Figure 2: STEM and Humanities accuracy on URDUMMLU under the Urdu prompt for top representative models from each model group. All models score lower on Humanities.
Figure 3: Overall accuracy on URDUMMLU under English and Urdu prompts for representative models from each model group. Prompt language has only a small effect on overall performance.
Under the Urdu prompt, Gemini-3.5-Flash scores 97.81% on STEM and 85.31% on Humanities, a gap of 12.50 points, while DeepSeek-V4Flash drops from 97.57% to 67.32%. GPT-5.4 and Claude-Sonnet-4.6 lose more than 22 points, and several Qwen models lose more than 35 points between the two domains. Figure 2 illustrates this trend for representative top-performing models from each section of Table 4. This pattern highlights the main challenge that URDUMMLU exposes. STEM questions rely on scientific and mathematical concepts that transfer more consistently across languages, whereas the Humanities domain requires stronger coverage of Urdu literature, Urdu language, Islamic studies, ethics, and other culturally grounded subjects. Many models can process Urdu well enough to answer science questions, but they struggle on Urdu literary, linguistic, and religious content. Social Sciences generally falls between STEM and Humanities, reflecting a mix of globally shared and region-specific knowledge.
A few models show larger prompt effects: Qwen3.6-35B-A3B gains 6.07 points with the Urdu prompt and GPT-5.4 gains 3.72, while Qwen3-8B and Alif-1.0-8B lose about two points. However, these shifts remain much smaller than the STEM-Humanities gaps. We therefore attribute the main difficulty of URDUMMLU to Urduspecific content instead of instruction language.
5.3
Prompt-Language Effects
Changing the prompt language usually has little effect on overall accuracy. Figure 3 compares representative models from each group in Table 4. The English and Urdu prompt results nearly overlap for all four: Gemini-3.5-Flash changes by +0.14 points, DeepSeek-V4-Flash by −0.99, Gemma-29B-IT by +1.52, and Qalb-1.0-8B by +0.75. The full table shows the same general pattern. Most models move by less than one point when the prompt changes from English to Urdu.
5.4 Invalid-Output Rates Invalid-output rates provide a useful complement to accuracy. Most modern proprietary and open-source models follow the required response format, with invalid-output rates below 0.1%. This includes Gemini, Claude, GPT, Gemma, LLaMA (Meta, 2025), Ministral, and most larger Qwen models. Smaller and weaker models show a different pattern. Under the Urdu prompt, BLOOMZ-3B returns invalid answers for 74.2% of examples, BLOOMZ-7B for 34.6%, and BLOOMZ-1.7B for 24.8%. The Urdu-targeted models also degrade under the Urdu prompt: Qalb1.0-8B reaches an invalid-output rate of 11.3%, and Alif-1.0-8B reaches 12.6%. These failures matter because accuracy over parseable outputs can hide severe formatting breakdowns. Reporting invalid outputs separately shows which models can both answer Urdu questions and follow Urdu evaluation instructions reliably; Appendix F gives one real example of each failure mode. 5.5 Few-Shot Performance Table 5 and Figure 4 summarize few-shot evaluation for LLaMA-3.1-8B, Gemma-3-4B-IT, Qwen38B, and Qwen3-4B-Instruct-2507.
English Prompt (Accuracy % ↑)
Model
Urdu Prompt (Accuracy % ↑)
0-shot
1-shot
3-shot
5-shot
0-shot
1-shot
3-shot
5-shot
LLaMA-3.1-8B Gemma-3-4B-IT Qwen3-8B Qwen3-4B-Instruct-2507
43.30 43.93 50.97 50.84
44.93 (+1.63) 45.91 (+1.98) 50.23 (−0.74) 52.56 (+1.72)
46.09 (+2.79) 46.07 (+2.14) 52.27 (+1.30) 53.40 (+2.56)
46.59 (+3.29) 46.27 (+2.34) 53.21 (+2.24) 53.65 (+2.81)
43.84 44.88 48.97 51.70
45.08 (+1.24) 46.40 (+1.52) 51.86 (+2.89) 52.06 (+0.36)
46.32 (+2.48) 46.36 (+1.48) 53.24 (+4.27) 52.89 (+1.19)
46.10 (+2.26) 45.99 (+1.11) 53.49 (+4.52) 52.94 (+1.24)
Mean
47.26
48.41 (+1.15)
49.46 (+2.20)
49.93 (+2.67)
47.35
48.85 (+1.50)
49.70 (+2.35)
49.63 (+2.28)
Table 5: Few-shot performance on UrduMMLU. Accuracy (%) at 0-, 1-, 3-, and 5-shot settings under English and Urdu instruction prompts. Coloured deltas in parentheses are relative to the 0-shot baseline of the same model under the same prompt; green indicates a gain and red indicates a loss. The Mean row aggregates the four evaluated models per shot setting.
LLaMA-3.1-8B (EN) LLaMA-3.1-8B (UR) Gemma-3-4B-IT (EN) Gemma-3-4B-IT (UR)
55
6 Conclusion and Future Work
Qwen3-8B (EN) Qwen3-8B (UR) Qwen3-4B-Instruct-2507 (EN) Qwen3-4B-Instruct-2507 (UR)
54 53 52
Accuracy (%)
51 50 49 48 47 46 45 44 43 42
0-shot
1-shot
3-shot
5-shot
Number of shots
Figure 4: Few-shot accuracy on URDUMMLU for LLaMA-3.1-8B (Grattafiori et al., 2024), Gemma-3-4BIT, Qwen3-8B, and Qwen3-4B-Instruct-2507 under English (solid) and Urdu (dotted) prompts. Accuracy generally improves from zero-shot to five-shot across both prompt languages, although the gains remain modest.
We evaluate each model at 1-, 3-, and 5-shot under English and Urdu prompts using validated demonstrations from a held-out pool. Few-shot prompting improves almost every setting: 23 of 24 configurations outperform their zero-shot baselines. Under the English prompt, mean gains reach +1.15, +2.20, and +2.67 points at 1-, 3-, and 5shot, while the Urdu prompt yields gains of +1.50, +2.35, and +2.28. Qwen3-8B under the Urdu prompt shows the largest improvement, increasing from 48.97% at zero-shot to 53.49% at fiveshot. Despite these gains, few-shot prompting does not change the overall ranking. Even at five-shot, all four models remain well below the ≥ 25B open-source tier and far behind proprietary models. Few-shot prompting also reduces prompt-language differences, with every English-Urdu gap staying within 0.71 points at five-shot. However, it does not compensate for missing Urdu-specific knowledge.
We introduced URDUMMLU, a broad-coverage, natively written MMLU-style benchmark for Urdu with 26,431 MCQs across 26 subjects and five domains, collected from Urdu MCQ banks and public SSC/HSSC examination PDFs. The benchmark combines standard academic subjects with Urduand region-specific content and uses dual human annotation with strict consensus filtering for examderived questions. Evaluating 30 open-source and closed-source LLMs under English and Urdu prompts reveals a clear gap in current model capability. Gemini-3.5-Flash performs best at 90.20% and 90.34% accuracy, while the strongest opensource model trails by 7.79 and 8.92 points. Models perform substantially better on STEM than on Urdu-centered Humanities, often losing 25 to 40 points on Urdu literature, Urdu language, and Islamic studies. Prompt language has limited effect for most models, and few-shot prompting yields only modest gains. Overall, URDUMMLU shows that strong English-centered benchmark performance does not ensure reliable Urdu educational and cultural knowledge and provides a stronger foundation for evaluating Urdu-capable LLMs. Future work can extend URDUMMLU beyond MCQ-based evaluation through open-ended generation, summarization, and translation tasks. Expanding the benchmark to include Indian Urdu curricula, undergraduate material, professional examinations, and dialectal content would further broaden its scope. Psychometrics also remains difficult for all evaluated models, motivating future Urdu reasoning benchmarks focused on analogies, logical patterns, and aptitude-style tasks. Finally, the weak performance of Urdu-targeted models highlights the need for stronger continued pretraining and instruction tuning on native Urdu educational and literary material.
Limitations Curriculum and source scope: URDUMMLU focuses on the Pakistani SSC/HSSC curriculum and a limited set of Urdu MCQ websites targeting the same educational setting. Strong performance therefore reflects competence on Pakistani secondary-school material rather than Urdu in its full linguistic diversity. The benchmark does not cover undergraduate content, Indian Urdu curricula, dialectal variation, or Urdu–English codeswitching. Although we reduce source skew through deduplication, annotation, and balancing, Ustad 360 still contributes 58.8% of the cleaned candidate pool. Format and ceiling effects: URDUMMLU uses a four-option multiple-choice format and therefore does not evaluate open-ended writing, summarization, translation quality, long-form reasoning, or conversational ability. Psychometrics partially offsets this limitation by introducing reasoning-heavy questions; however, no model exceeds 60% accuracy on this subdomain. Future work should extend evaluation toward more open-ended Urdu tasks. Prompt-language and few-shot effects are limited: English and Urdu instruction wrappers, together with 1-, 3-, and 5-shot prompting, change accuracy by only a few points and rarely alter model rankings. We also do not evaluate optionorder robustness, chain-of-thought prompting, or specialized reasoning modes. Our setup therefore prioritizes consistency and comparability over fully optimized prompting configurations.
Ethical Statement & Broad Impact We develop URDUMMLU to support more inclusive multilingual evaluation for Urdu, a widely spoken but underrepresented language in NLP research. The benchmark draws from publicly available educational and examination material and aims to improve evaluation coverage beyond English-centered benchmarks. Transparency and Reproducibility: We release the dataset, evaluation code, and prompting protocols to support reproducible research and transparent comparison across models. We also document the dataset construction pipeline, annotation procedure, and evaluation setup in detail.
Annotation and Data Quality: We use dual human annotation with strict consensus filtering for exam-derived questions and additionally verify answer labels for web-derived items. We further apply cleaning, deduplication, and normalization procedures to reduce OCR noise, malformed questions, and metadata inconsistencies. Bias and Scope Limitations: URDUMMLU primarily reflects the Pakistani SSC/HSSC curriculum and the educational content available through Urdu MCQ resources. As a result, it may not fully represent other Urdu-speaking communities, dialects, or educational systems. The benchmark also contains culturally and regionally grounded subjects such as Islamic studies and Pakistan studies that reflect the underlying curriculum sources. Broader Impact: We hope URDUMMLU supports the development of stronger Urdu-capable language models and more representative multilingual evaluation. At the same time, benchmark scores should not be interpreted as complete measures of reasoning ability, factual reliability, or cultural understanding beyond the educational scope represented in the dataset.
References Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Phi-3 technical report: and 110 others. 2024. A highly capable language model locally on your phone. Preprint, arXiv:2404.14219. Farah Adeeba, Brian Dillon, Hassan Sajjad, and Rajesh Bhatt. 2025. UrBLiMP: A benchmark for evaluating the linguistic competence of large language models in Urdu. Sarfraz Ahmad, Hasan Iqbal, Momina Ahsan, Numaan Naeem, Muhammad Ahsan Riaz Khan, Arham Riaz, Muhammad Arslan Manzoor, Yuxia Wang, and Preslav Nakov. 2025. UrduFactCheck: An agentic fact-checking framework for Urdu with evidence boosting and benchmarking. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22788–22802, Suzhou, China. Association for Computational Linguistics. Anthropic. 2025. Claude Haiku 4.5. https://www. anthropic.com/claude/haiku. Accessed 2026-0523.
Anthropic. 2026. Claude Opus 4.7 system card. https: //www.anthropic.com/news/claude-opus-4-7. Accessed: 2026-05-26. Anthropic. 2026. Claude Sonnet 4.6 System https://www-cdn.anthropic.com/ Card. 78073f739564e986ff3e28522761a7a0b4484f84. pdf. Accessed 2026-05-23. DeepSeek-AI. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The language model evaluation harness. Gemma Team. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Google. 2026a. Gemini 3.1 Flash-Lite. https://ai.google.dev/gemini-api/docs/models/ gemini-3.1-flash-lite. Accessed 2026-05-23. Google. 2026b. Gemma 4 model card. https://ai. google.dev/gemma/docs/core/model_card_4. Accessed 2026-05-23. Google DeepMind. 2026. Gemini 3.5 Flash Model Card. https://deepmind.google/models/ model-cards/gemini-3-5-flash/. Accessed 2026-05-23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427–5444, Online. Association for Computational Linguistics. Muhammad Taimoor Hassan, Jawad Ahmed, and Muhammad Awais. 2026. Qalb: Largest state-ofthe-art Urdu large language model for 230m speakers with systematic continued pre-training. Preprint, arXiv:2601.08141. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Samreen Kazi and Shakeel Khoja. 2021. UQuAD1.0: Development of an Urdu question answering training data for machine reading comprehension. arXiv preprint arXiv:2111.01543. Samreen Kazi and Shakeel Khoja. 2026. UQuAD+: Benchmark dataset for Urdu machine reading comprehension. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 25(2). Samreen Kazi, Maria Rahim, and Shakeel Ahmed Khoja. 2025. Crossing language boundaries: Evaluation of large language models on Urdu-English question answering. In Proceedings of the First Workshop on Natural Language Processing for IndoAryan and Dravidian Languages, pages 141–151, Abu Dhabi. Association for Computational Linguistics. Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, and Sreyoshi Bhaduri. 2025. IndicMMLU-Pro: Benchmarking Indic large language models on multi-task language understanding. Preprint, arXiv:2501.15747. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore. Association for Computational Linguistics. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, Bangkok, Thailand. Association for Computational Linguistics. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584.
Meta. 2024a. Llama 3.2 3b instruct model card. https://huggingface.co/meta-llama/Llama-3. 2-3B-Instruct. Accessed 2026-05-23. Meta. 2024b. Llama 3.3 70b instruct model card. https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct. Accessed 2026-05-23. Meta. 2025. Llama 4 model card. https: //github.com/meta-llama/llama-models/blob/ main/models/llama4/MODEL_CARD.md. Accessed 2026-05-23. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, and 57 others. 2025. Phi-4-Mini technical report: Compact yet powerful multimodal language models via Mixture-of-LoRAs. Preprint, arXiv:2503.01743. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Linguistics. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, and 38 others. 2025. INCLUDE: Evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations.
Muhammad Shafique, Areej Mehboob, Layba Fiaz, Muhammad Qadeer, and Hamza Farooq. 2026. UrduBench: An Urdu reasoning benchmark using contextually ensembled translations with human-in-theloop. Muhammad Ali Shafique, Kanwal Mehreen, Muhammad Arham, Maaz Amjad, Sabur Butt, and Hamza Farooq. 2025. Alif: Advancing Urdu large language models via multilingual synthetic data distillation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 271– 284, Suzhuo, China. Association for Computational Linguistics. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex BakerWhitcomb, Alex Beutel, Alex Karpenko, and 467 others. 2026. OpenAI GPT-5 System Card. Preprint, arXiv:2601.03267. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, WeiYin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4076–4104, Albuquerque, New Mexico. Association for Computational Linguistics. Munief Hassan Tahir, Sana Shams, Layba Fiaz, Farah Adeeba, and Sarmad Hussain. 2025. Benchmarking the performance of pre-trained LLMs across Urdu NLP tasks. In Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 17–34, Abu Dhabi, UAE. International Committee on Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhan, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekassyl Syzdykov, Nurkhan Laiyk, Alham Fikri Aji, Ekaterina Kochmar, Preslav Nakov, and Fajri Koto. 2025. KazMMLU: Evaluating language models on Kazakh, Russian, and regional knowledge of Kazakhstan. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14403–14416, Vienna, Austria. Association for Computational Linguistics. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2025. MILU: A multi-task Indic language understanding benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10076–10132, Albuquerque, New Mexico. Association for Computational Linguistics. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. MMLU-ProX: A multilingual benchmark for advanced large language model evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1513–1532, Suzhou, China. Association for Computational Linguistics.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
A
Candidate Pool Analysis
We construct URDUMMLU in two stages. First, an automatic preprocessing pipeline collects and cleans multiple-choice questions from Pakistani examination boards and Urdu MCQ websites to produce a candidate pool. Second, annotation, verification, deduplication, and balancing transform this pool into the final benchmark used in all evaluations. This appendix analyzes both stages and shows how the dataset composition changes throughout the construction process. Figure 5 shows the distribution of URDUMMLU items across the four Pakistani examination levels: SSC-I, SSC-II, HSSC-I, and HSSC-II. The left panel reports absolute item counts, while the right panel reports the within-level domain distribution. The Figure highlights two consistent trends, first, Humanities dominates the SSC levels, where language and literature subjects occupy a larger portion of the curriculum. Second, STEM and Social Sciences become more prominent at the HSSC levels, where students specialize into science, commerce, and humanities tracks. The level distribution therefore reflects the structure of the Pakistani curriculum rather than collection artifacts. We also analyze question length because stem length can influence model performance and varies across domains. Figure 6 summarizes the overall length distribution and the domain-wise split between short and long stems using a 9-word threshold. Most UrduMMLU stems are short, but the dataset retains a substantial long-question tier. STEM has the most balanced short/long distribution, while Humanities and Profession contain relatively more short stems. A.1
Source and Domain Distributions
Tables 6 and 7 compare the cleaned candidate pool (Raw) and released benchmark (Final), distinguishing the initial collection distribution from the curated evaluation benchmark. Source distribution: The cleaned candidate pool contains 40,427 items collected from nine Pakistani examination and MCQ-bank sources, of which 26,431 survive into the final benchmark. Table 6 shows that the raw pool is heavily concentrated in a few large sources. Ustad 360 alone contributes 23,788 raw items, while the four largest sources together account for more than 90% of the pool. The final benchmark is substantially less skewed.
Source
Raw
Final
% Share
Ustad 360 23,788 11,068 41.9% MCQTimes 6,099 5,918 22.4% TestPointPK 3,619 3,502 13.2% ETest 3,102 2,783 10.5% FBISE 2,406 1,459 5.5% ExamAunty 643 540 2.0% GoTest 566 515 1.9% PakMCQs 434 414 1.6% BISE Multan 2025 440 232 0.9% Total
40,427 26,431
100.0
Table 6: Source distribution of the cleaned candidate pool (Raw) and the released URDUMMLU benchmark (Final). Percentages and share bars correspond to the final benchmark distribution. Domain
Raw
Final
% Share
Humanities 11,539 11,010 41.7% Social Sciences 14,626 7,968 30.2% STEM 11,590 5,113 19.3% Other 2,030 1,365 5.2% Profession 642 975 3.7% Total
40,427 26,431
100.0
Table 7: Domain distribution before and after final benchmark selection. Raw denotes the cleaned candidate pool, while Final denotes the released URDUMMLU benchmark.
Annotation, deduplication, and balancing reduce the relative share of the largest sources, while smaller sources such as MCQTimes, TestPointPK, and ETest contribute proportionally more to the released benchmark. BISE Multan 2025 shows the largest reduction because of a high duplicate rate against other examination sources. Figure 7 expands the domain-level statistics from Table 7 to the subdomain level. Humanities is dominated by Urdu Literature and Urdu Language, whereas Social Sciences and STEM distribute more evenly across multiple medium-sized subdomains. Domain distribution: Table 7 compares the cleaned candidate pool and the final benchmark across domains. The candidate pool distributes relatively evenly across Humanities, Social Sciences, and STEM, while Other and Profession remain much smaller. The final benchmark shifts toward Humanities, which grows from 28.5% to 41.7%, while Social Sciences and STEM decrease to 30.2% and 19.3%, respectively. Profession is the only domain whose absolute count increases during balancing (642 → 975), which improves coverage for reliable domain-level evaluation.
100%
11,601
12,000
683
80%
10,000
25%
32%
8,000
6,838
1,266
6,000 2,155
4,125
3,867
4,000
Share of level (%)
# of MCQs
2,942
60%
11%
71%
40%
1,222
6,276
66% 18%
54% 2,549
2,000
43%
20%
2,931
26%
2,953
0
SSC-I
SSC-II
993
788
HSSC-I
HSSC-II
Humanities
0%
Social Sciences
SSC-I
STEM
SSC-II
Other
19%
HSSC-I
HSSC-II
Profession
Figure 5: Distribution of URDUMMLU items across Pakistani examination levels, grouped by domain. Left: absolute item counts per level. Right: within-level domain distribution. Humanities dominates SSC-I and SSC-II, while STEM and Social Sciences become more prominent at the HSSC levels, reflecting the structure of the Pakistani secondary-school curriculum. tier boundary (≤9 = short)
12000
short (≤9 words) long (>9 words)
11,010 (66% / 34%)
3000
3,748
# of MCQs
# of MCQs
10000
2000
3,191
6000 4000
1000
7,968 (60% / 40%)
8000
5,113 (51% / 49%) 2,517
7,262 4,777
2000
2,596
0
0
0
10
20
30
40
50
words in question
(a) Distribution of question lengths in words. The dashed vertical line at 9 words marks the short/long boundary.
s
tie
ni ma
Hu
ial
Soc
s
e enc
Sci
M STE
1,365 (52% / 48%) 654 711
Oth
er
975 (62% / 38%) 373 602
n
sio
fes
Pro
(b) Domain-wise counts of short and long questions, with within-domain percentages shown in parentheses.
Figure 6: Question-length analysis for UrduMMLU. Left: histogram of question lengths. Right: domain-wise counts of short (≤ 9 words) and long (> 9 words) questions. STEM is closest to a balanced split, while Humanities and Profession skew shorter.
These changes reflect a deliberate balancing step rather than artifacts of preprocessing or cleaning. We down-sample overrepresented STEM and Social Sciences items and preserve underrepresented Profession items to better align the benchmark with the structure of the Pakistani SSC and HSSC curriculum shown in Figure 5. This process improves coverage across domains while maintaining alignment with the underlying curriculum resulting in a benchmark that provides a balanced representation of subjects encountered in Pakistani education.
B Annotation Details The exam-derived portion of URDUMMLU came from Pakistani examination boards and MCQ sources that did not provide answer keys. To produce reliable gold labels, we recruited 17 Urdufluent annotators and ran a dual-annotator consensus process supported by a custom dashboard and written guidelines. This appendix documents the annotator pool, the annotation guidelines and dashboard, the inclusion and edit-resolution rules, and the resulting agreement statistics.
Urdu Literature Urdu Language Pakistan Studies Islamic Studies General Knowledge Chemistry Biology General Science Economics Mathematics Education Computer Science Sociology Urdu Grammar Health & Physical Ed. Civics Geography Physics Prof. Development Prof. Studies Psychology Current & Intl. Affairs Commerce Ethics Psychometrics Fine Arts
5,859 2,522 1,895 1,497 1,365 936 931 926 907 876 836 823 778 756 742 675 630 621 514 461 460 437 423 228 185 148 0
2,000
4,000
6,000
NUMBER OF QUESTIONS Humanities 11,010
STEM 5,113
Social Sciences 7,968
Other 1,365
Profession 975
Figure 7: Final URDUMMLU item counts by subdomain, grouped by domain. Urdu Literature and Urdu Language contribute the largest shares, while Social Sciences and STEM distribute across a larger number of medium-sized subdomains.
B.1
Annotator Demographics and Feedback
The annotation pool consisted of 17 annotators recruited for native Urdu fluency and familiarity with the Pakistani school curriculum. Table 8 summarizes the demographic profile of the pool. The annotators were approximately gender-balanced (52.9% female, 47.1% male), predominantly native Urdu speakers (94.1%), and concentrated in the 18–34 age range. Educationally, 88.3% held at least a bachelor’s degree and 41.2% held a master’s degree, which is important for a benchmark that targets SSC- and HSSC-level subject content. Most annotators also reported between one and six years of professional experience. After completing their assigned batches, all annotators filled out a post-task satisfaction survey. Table 9 summarizes the responses. Feedback was consistently positive, and no annotator selected Disagree or Strongly disagree for any statement. Instruction clarity, compensation fairness, guideline usefulness, and overall satisfaction received entirely positive responses. Task enjoyment received a smaller number of neutral responses (23.5%), suggesting that annotators found the process clear and manageable even if not inherently engaging.
Attribute
Count
%
Gender Female Male
9 8
52.9 47.1
Native Urdu speaker Yes No
16 1
94.1 5.9
Age range 18–24 25–34
7 10
41.2 58.8
Highest completed education High school diploma Some college / vocational Bachelor’s degree Master’s degree
1 1 8 7
5.9 5.9 47.1 41.2
Professional work experience Less than 1 year 1–3 years 4–6 years 7–9 years
5 6 4 2
29.4 35.3 23.5 11.8
Total annotators
17
100.0
Table 8: Demographic profile of the UrduMMLU annotator pool (n = 17; identities anonymised). Statement
SA
A
N
Pos.
Instructions clearly explained the task 82.4 17.6 0.0 100.0 Task was enjoyable and engaging 47.1 29.4 23.5 76.5 Paid fairly for the task 76.5 23.5 0.0 100.0 Examples / guidelines were helpful 76.5 23.5 0.0 100.0 Overall glad to have completed the task 76.5 23.5 0.0 100.0
Table 9: Post-task satisfaction survey results (n = 17, values in %). SA = Strongly agree, A = Agree, N = Neutral. No annotator selected Disagree or Strongly disagree on any item, so those columns are omitted. Pos. is the share of Agree plus Strongly agree.
B.2 Annotation Guidelines Before annotation began, we held a live online onboarding session in which we walked through the task end-to-end, demonstrated each flag and edit category on real items, and answered annotator questions. The full written guidelines were also embedded as an always-available help page inside the annotation dashboard so that annotators could re-check policies during their work, and admins remained reachable by email throughout the annotation period for cases not covered by the written guidelines. We also encouraged annotators to consult the guidelines whenever they encountered uncertain or ambiguous cases to ensure consistent decisions across annotation batches. These procedures helped ensure consistent annotation decisions.
Task overview: Annotators were asked to verify the answer to each MCQ by selecting the single best option from A/B/C/D. When multiple options looked plausible, annotators were instructed to select the most precise or directly relevant answer rather than guessing.
Check
Drop if
Dual annotation
Look-up and abstention policy: Annotators were encouraged to consult Google or Wikipedia for fact-based questions (dates, authors, capitals, scientific terms, historical events) rather than relying on memory, with a target pace of 15–30 seconds per question including verification. Annotators were asked to mark an item as unsure / skip rather than submit a confident guess in any of the following cases: (i) the answer could not be resolved within roughly a minute of search, (ii) two or more options remained equally plausible after verification, or (iii) the question required specialist context (e.g., niche fiqh details or obscure regional history) that they could not quickly acquire.
Consensus
Only one assigned annotator submitted an annotation. Either annotator flagged the item. Either annotator selected the unsure/skip option. Either annotator did not select one of A/B/C/D. The two annotators selected different answer keys. The annotation came from an annotator who was not assigned to that batch.
Flag vs. edit: Annotators were given a single rule of thumb to choose between the two actions: edit when the issue could be fixed in-place by changing text (a typo, missing space, duplicated word, wrong subdomain label), and flag when the issue required admin review and could not be repaired by text correction (no correct answer, multiple correct answers, ambiguity, missing visual, outof-scope content). Annotators were explicitly instructed not to rewrite question semantics, not to “fix” wrong distractors into correct ones, and not to flag a question solely because they had edited a typo in it. When to flag: Figure 8 illustrates the five flag categories covered in the guidelines. These are: (a) two or more options are simultaneously correct; (b) none of the listed options is the correct answer; (c) the question is ambiguous, vague, or under-specified; (d) the question references a diagram, image, or chart that is not included in the text; and (e) the question is out of scope for the benchmark (hyper-local trivia, sectarian content, opinion questions). For each case, annotators were asked to attach a short free-text note explaining the issue. This information helped reviewers verify and resolve flagged items during quality control. Flagging was independent of answer selection, and annotators could flag with or without picking an option.
Flag Unsure Valid pick
Assignment
Table 10: Inclusion rules for the final annotated pool. An item is dropped if any rule fires; only items that pass every check enter the gold-labelled set.
When to edit: Figure 9 illustrates the three most common edit categories: (a) spelling fixes and missing diacritics, where the intended word is clear from context; (b) duplicated words and other scraping artifacts; and (c) subdomain reassignment, where the original subdomain label is clearly inconsistent with the question content. Beyond these, the guidelines also permitted spacing corrections, removal of stray punctuation, stripping of redundant in-text option-letter prefixes (e.g., A., B., or their Urdu equivalents) already shown by the option badge, and translation of stray English option text when an unambiguous Urdu equivalent existed. Technical English terms, HTML/CSS tags, proper names, and brand names were left in English. Workflow: Annotators worked in batches of approximately 50 MCQs, with accuracy prioritized over speed. B.3 Inclusion Rules We applied a deterministic consensus filter (Table 10): an item was retained only when both assigned annotators independently submitted the same valid answer choice and neither flagged nor abstained. The agreed answer was stored as the final gold label. Two edge cases require clarification. First, when both annotators selected an option such as “none of these”, we treated it as a valid agreed answer because such options commonly appear in Pakistani MCQ examinations. Second, only the explicit unsure / skip action counted as abstention. Missing annotations triggered the incomplete-annotation rule instead, so abstentions always reflected deliberate annotator decisions.
(a) Two or more options are simultaneously correct.
(b) No option in the list is the correct answer.
(c) Question is ambiguous, vague, or under-specified.
(d) Question references a missing image, diagram, or chart.
(e) Hyper-local, sectarian, or opinion content that is out of scope for the benchmark.
Figure 8: Examples of the five flag categories used in the annotation guidelines. Annotators were asked to flag the item and attach a short free-text note for each case.
(a) Spelling fix: a missing letter is re- (b) Duplicated word from a scraping arti- (c) Wrong subdomain (art and drawstored from context. fact is removed. ing) is reassigned via the dropdown to pakistan studies.
Figure 9: Examples of the most common edit categories permitted by the annotation guidelines. Edits are restricted to OCR, scraping, formatting, and metadata corrections; annotators do not rewrite question semantics or modify answer correctness.
B.4
Edit Resolution
In addition to selecting answers, annotators could suggest edits to question text, answer options, or subdomain labels. These edits targeted minor extraction and metadata issues such as OCR errors, dropped diacritics, malformed option labels, and incorrect subdomain assignments rather than substantive question rewrites. We resolved all edits deterministically so that the final benchmark could be reconstructed directly from the raw annotations. Table 11 summarizes the resolution rules.
The resolution policy prioritizes agreed edits when both annotators propose the same correction and otherwise prefers the more conservative or informative revision. For subdomain edits, we recompute the corresponding domain label from the corrected subdomain to preserve consistency between the two fields in the released benchmark. This procedure ensures that metadata corrections remain internally consistent throughout the final dataset.
Case
Resolution
Only one annotator edited a field Both annotators made the same edit Both annotators edited the same field differently No edit was suggested Subdomain was edited
Use that edit. Use the agreed edit. Use the longer edit. Keep the original field. Recompute the domain from the corrected subdomain.
Table 11: Edit-resolution rules for annotated MCQs. The rules are applied per field, and the resolved values are written back to the item before the inclusion rules in Table 10 are evaluated.
B.5
Annotation Dashboard
Section B.2 described the annotation policies; here we illustrate the dashboard used to apply them. For each item, annotators could select an answer, mark it as unsure / skip, edit question or option text, or flag it for review with a free-text explanation. This design separated answer selection from quality-control feedback and text correction. Figures 10, 11, and 12 illustrate the workflow. Picking an answer: Figure 10 shows the standard annotation workflow. Annotators view the Urdu question stem, four labeled answer options, and metadata describing the subdomain, academic level, length tier, and item identifier (Figure 10a). Selecting an option highlights the choice but does not automatically advance to the next item (Figure 10b); annotators must explicitly confirm the selection before proceeding, which reduces accidental submissions. Keyboard shortcuts (1–5 for option selection, arrow keys for navigation) support efficient batch traversal. Editing an item: Figure 11 illustrates the inplace editing UI. The example contains OCR and formatting artifacts in the answer options (Figure 11a). After entering edit mode, annotators modify the question text and options through inline editable fields (Figure 11b). The interface records all changes and attaches revision tags to each edited field for later review, which can be reverted with a single click (Figure 11c). Flagging an item: Figure 12 shows the flagging UI. In the illustrated case, OCR corruption removes superscript formatting from a physics question, making all answer options invalid (Figure 12a).
The annotator marks the item as unsure / skip and submits a flag with a free-text explanation (Figure 12b). The dashboard visually highlights flagged items so admins can review them, and the inclusion rules in Section B.3 automatically remove flagged items from the consensus pool. B.6 Annotation Outcomes Outcome
Count
Input annotated MCQs Retained after consensus filtering
17,565 14,459
Dropped: annotator disagreement Dropped: flagged by annotator Dropped: unsure/skip selected Dropped: single annotated
1,611 1,247 243 5
Domain corrections
141
Table 12: Annotation outcomes for the exam-derived portion of URDUMMLU. Each excluded item appears under a single exclusion rule.
A total of 17,565 exam-derived MCQs entered annotation, of which 14,459 were retained after applying the edit-resolution and inclusion rules from Tables 11 and 10 (an overall yield of 82.3%). Table 12 breaks down the 3,106 exclusions. Answer disagreement is the dominant cause (51.9% of all drops), reflecting questions where two qualified Urdu annotators could not converge on a defensible answer and which are therefore unsuitable for evaluation under a strict consensus policy. Flagged items form the second-largest category and predominantly contain OCR corruption or malformed options similar to Figure 12a. Inter-annotator agreement was correspondingly high. Across all annotated items, observed agreement reached 89.98%, with a simplified Cohen’s κ of 0.8663. Figure 13 further breaks agreement down by annotator pair. Each cell reports the simplified Cohen’s κ together with the number of shared retained items, while blank cells indicate annotator pairs without overlap. Most populated cells exceed κ = 0.85, showing that agreement remains consistently strong across annotator pairs rather than depending on a small subset of annotators. This pattern indicates that annotation quality remained stable across the workforce. Loweragreement cells correspond mainly to pairs with relatively few shared items and therefore have limited influence on the aggregate statistic.
(a) Question view with answer options, metadata, and annotation controls.
(b) Answer selection interface before advancing to the next item.
Figure 10: Annotation dashboard workflow for answer selection.
(a) Original OCR-extracted question and (b) In-place editing of question and op- (c) Saved edits with editable revision options. tion text. markers.
Figure 11: Annotation dashboard workflow for text correction and normalization.
(a) Problematic OCR example marked as unsure / skip.
(b) Flagged item with an attached review reason.
Figure 12: Annotation dashboard workflow for flagging problematic items.
C
Dataset Format
Each URDUMMLU example is stored as a multiplechoice item containing a question, four answer options, a gold answer label, domain and subdomain labels, academic level information, and source metadata. The evaluation pipeline uses the question, options, and gold answer fields during inference, while the remaining metadata supports analysis, filtering, and reproducibility. Figure 14 reports two dataset-level sanity checks: item-length distribution and answer-key distribution. The median item length is 76 characters, and the answer labels remain close to uniformly distributed across A–D, reducing the risk of prompt-length or answerposition bias during evaluation.
Figure 15 shows the JSON schema used for each released benchmark item. C.1 Subject Acronyms and Education Levels Table 13 lists the full names, acronyms, and education levels for all 26 URDUMMLU subdomains, grouped by domain. We use these acronyms in the per-subdomain results tables (Tables 16 and 17).
D Evaluation Details This appendix provides the full model roster and prompt templates used in the URDUMMLU experiments.
Annotator-pair agreement (anonymized; final-included MCQs, n=14459) 1.00 (449)
0.95 (82)
0.78 (248)
0.83 (797)
0.90 (41)
A02
0.91 (487)
A03
1.00 (449)
0.78 (248)
A04
0.95 (82)
0.83 (797)
A05
0.90 (41)
0.91 (43)
A06
0.93 (91)
0.93 (38)
0.90 (813)
A07
0.86 (48)
0.83 (40)
0.85 (196)
A08
0.89 (84)
A09
0.83 (47)
0.92 (32)
A10
0.98 (78)
1.00 (7)
A11
0.76 (39)
0.87 (687)
A12
0.90 (92)
A13
0.93 (41)
A14
0.95 (88)
A15
0.87 (63)
A16
0.96 (76)
0.90 (98)
0.86 (48)
0.89 (84)
0.83 (47)
0.98 (78)
0.76 (39)
0.90 (92)
0.91 (43)
0.93 (38)
0.83 (40)
0.92 (32)
1.00 (7)
0.88 (775)
0.92 (32)
0.92 (98)
0.92 (96)
0.91 (144)
0.82 (751)
0.92 (96)
1.00 (41) 0.84 (933)
0.84 (933)
0.70 (94)
0.65 (95)
0.65 (95)
0.87 (99)
0.93 (39)
0.78 (6) 0.81 (48)
0.93 (39)
0.87 (811)
0.90 (729)
0.81 (48)
0.87 (811)
0.85 (139)
0.81 (49)
0.87 (99)
0.98 (368)
0.70 (45)
0.82 (408)
0.99 (89)
0.86 (37)
0.98 (368)
0.82 (408)
0.70 (45)
0.99 (89)
0.84 (34)
0.94 (48)
0.86 (37)
0.98 (77)
0.81 (765)
0.89 (74)
0.88 (796)
0.84 (34)
0.88 (92)
0.85 (139)
0.88 (796)
0.87 (135) 0.91 (144)
0.8
0.81 (49)
0.84 (786)
0.78 (6)
0.94 (48)
0.88 (92)
0.90 (729)
0.84 (49)
0.85 (99)
0.9
0.94 (48)
0.84 (49)
0.84 (786)
0.92 (98)
0.91 (93)
0.85 (99)
0.84 (692) 0.84 (692)
1.0
0.90 (90)
0.85 (196)
0.82 (751)
0.91 (93)
0.96 (76)
0.70 (94) 0.90 (813)
0.92 (32)
0.87 (63)
0.87 (687)
0.88 (775)
0.90 (39)
0.90 (90)
0.95 (88)
0.87 (135)
0.90 (39)
1.00 (41)
A17
0.93 (41)
0.90 (98)
0.93 (99) 0.93 (99)
0.93 (91)
0.81 (765)
0.84 (772) 0.84 (772)
0.94 (48)
0.98 (77)
0.89 (74)
0.95 (98)
0.7
Cohen's κ (simplified, vs 25% chance)
0.91 (487)
A01
0.6
0.95 (98) 0.81 (453)
0.81 (453)
7
6
A1
A1
5 A1
4
3
A1
2
A1
1
A1
0
A1
9
A1
8
A0
7
A0
6
A0
5
A0
A0
4 A0
3
2
A0
A0
A0
1
0.5
Figure 13: Pairwise annotator agreement on final-included MCQs. Each cell reports simplified Cohen’s κ, with the number of shared items shown in parentheses. Blank cells indicate annotator pairs with no shared final-included items.
D.1
Model Roster
Table 14 lists the 30 models evaluated in this work. We group models by family for readability, while the main paper discusses them using broader categories such as proprietary API models, open-weight multilingual models, compact models, mixture-of-experts models, reasoningoriented variants, and Urdu- or regionally specialized models. D.2
Prompt Templates
We use separate English and Urdu prompt templates for zero-shot evaluation. Both templates present the same Urdu question and answer options while changing only the instruction language and field labels. The output format remains identical in both settings to support automatic parsing and consistent evaluation.
English prompt: Figure 16 shows the English prompt template, which combines a fixed system prompt with a per-item user prompt. The system prompt instructs the model to answer in a strict twoline format consisting of an Answer key and Answer text, without additional explanation or formatting. This structure supports deterministic answer extraction and consistent measurement of invalid outputs across models. The user prompt fills the placeholders domain, subdomain, level, question, and A–D directly from the dataset schema in Appendix C, preserving the Urdu question content across both prompt-language settings. Urdu prompt: The Urdu prompt template mirrors the English template while translating the system instructions, user-field labels ( , , ), and surrounding instructional text into Urdu. The Urdu question stem and answer options remain unchanged across both settings.
8,000
Median: 76 chars
30%
7,000
25.3%
24.6%
24.8%
25.3%
B
C
D
SHARE OF ITEMS
# of MCQs
6,000 5,000 4,000 3,000
uniform (25%)
20%
10%
2,000 1,000 0
0
20
40
60
80
100
120
140
160
180
Characters per item
(a) Character-length distribution per item.
200
220
0%
A
CORRECT OPTION
(b) Gold answer-key distribution.
Figure 14: Dataset-level sanity checks for URDUMMLU. Most questions remain compact enough for standard MCQ prompting, while the gold answer keys remain close to uniformly distributed across A–D.
UrduMMLU JSON Schema
Zero-shot English Prompt
{
System Prompt: You are an expert multiple-choice question answering assistant. Read the question carefully and select the single best answer. Respond in EXACTLY this two-line format, with no extra text:
”id”: ”...”, ”question”: ”...”, ”options”: { ”A”: ”...”, ”B”: ”...”, ”C”: ”...”, ”D”: ”...” }, ”correct_key”: ”...”, ”correct_option”: ”...”, ”domain”: ”...”, ”subdomain”: ”...”, ”level”: ”...”, ”source”: [ { ”name”: ”...”, ”url”: ”...” } ] }
Figure 15: JSON schema for individual URDUMMLU question items.
• Answer key: <one of A, B, C, D> • Answer text: <verbatim text of the chosen option, copied character-for-character> Do not add explanations, preambles, markdown, or punctuation outside of the format. The Answer text must match the option text exactly so the response can be parsed programmatically. User Prompt: Subject: {domain} – {subdomain} Level: {level} Question: {question} A) {A} B) {B} C) {C} D) {D} Answer key: Answer text:
Figure 16: Prompt for multiple-choice question answering with strict answer formatting requirements.
We also preserve the same two-line response structure using the English fields Answer key: and Answer text:, which allows a single parser to process outputs under both prompt languages. This design ensures that prompt language is the only substantive difference between the two evaluation settings. Figure 17 shows the full Urdu prompt template. This minimal-difference setup makes the prompt-language comparison in Section 5 directly interpretable, since any performance change comes from the instruction language rather than changes in question content or evaluation logic.
D.3 Few-Shot Evaluation Setup We evaluate using the lm-evaluation-harness framework (Gao et al., 2024). Each item is formatted as a four-option multiple-choice question, with the answer choices labeled A through D and the gold label stored as an integer index in {0, 1, 2, 3}. The benchmark comprises 26,431 items spanning all subject-level splits. We report accuracy (acc) and length-normalized accuracy (acc_norm) under 0-shot, 1-shot, 3-shot, and 5-shot conditions.
Domain
Subdomain
Acronym
Levels
Humanities Humanities Humanities Humanities Humanities Humanities
Ethics Fine Arts Islamic Studies Urdu Grammar Urdu Language Urdu Literature
ETH FNA ISL UGR ULG ULT
SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II
Other
General Knowledge
GKN
SSC-I, SSC-II, HSSC-I, HSSC-II
Profession Profession
Professional Development Professional Studies
PRD PRS
SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II
STEM STEM STEM STEM STEM STEM
Biology Chemistry Computer Science General Science Mathematics Physics
BIO CHM CSC GSC MTH PHY
SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-I SSC-I, SSC-II, HSSC-II
Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences Social Sciences
Civics Commerce Current & International Affairs Economics Education Geography Health & Physical Education Pakistan Studies Psychology Psychometrics Sociology
CIV COM CIA ECO EDU GEO HPE PKS PSY PMT SOC
SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II HSSC-I, HSSC-II SSC-I, SSC-II, HSSC-I, HSSC-II HSSC-I, HSSC-II
Table 13: URDUMMLU domains, subdomains, acronyms, and corresponding education levels.
D.4
Implementation Details
All evaluations use a fixed random seed of 42. For locally loaded open-weight models, we use bfloat16 precision, greedy decoding, batch size 10, and automatic device placement. We evaluate instruction-tuned models with their chat templates. For API-based systems, including OpenAI, Anthropic, Google Gemini, and Hugging Face Inference API, we use the same prompt format and configuration whenever provider constraints permit. We retry failed API requests up to five times and terminate the pipeline after five consecutive failures to prevent silent evaluation errors.
E
Detailed Results
This section provides a more detailed view of the results summarized in Table 4. Table 15 reports the STEM–Humanities accuracy gap for each model under the Urdu prompt, sorted by STEM accuracy. Across nearly all model families, performance on STEM substantially exceeds performance on Humanities, and the gap generally widens as overall capability decreases.
The gap becomes small for the BLOOMZ family and the two Urdu-targeted models, but for different reasons. BLOOMZ checkpoints remain close to the random baseline on both domains, while the Urdu-targeted models show similarly low performance on STEM and Humanities because their STEM accuracy is already far below that of comparably sized general-purpose models. E.1 Per-Subdomain Results We expands the domain-level results from Table 4 to all 26 subdomains. Table 16 reports accuracy under the English prompt, while Table 17 reports accuracy under the Urdu prompt. Both tables follow the same ordering, with subdomains grouped by domain and sorted by dataset size. E.1.1 Subject-Wise Behavior The subdomain results sharpen the main finding from Section 5: STEM subjects transfer much more reliably than Urdu-centered Humanities subjects. The strongest models approach saturation on several STEM subdomains. In contrast, performance remains lower on Urdu-centered subjects.
Model
Size
Family
License
Ref.
large-traversaal/Alif-1.0-8B-Instruct
8B
Alif
Apache 2.0
(Shafique et al., 2025)
mistralai/Ministral-3-3B-Instruct-2512 mistralai/Ministral-3-8B-Instruct-2512
3B 8B
Ministral Ministral
Apache 2.0 Apache 2.0
(Liu et al., 2026) (Liu et al., 2026)
enstazao/Qalb-1.0-8B-Instruct
8B
Qalb
Apache 2.0
(Hassan et al., 2026)
Qwen/Qwen3-4B-Instruct-2507 Qwen/Qwen3-8B Qwen/Qwen3.6-27B Qwen/Qwen3.6-35B-A3B
4B 8B 27B 36B
Qwen 3 Qwen 3 Qwen 3.6 Qwen 3.6
Apache 2.0 Apache 2.0 Apache 2.0 Apache 2.0
(Yang et al., 2025) (Yang et al., 2025) (Yang et al., 2025) (Yang et al., 2025)
bigscience/bloomz-1b1 bigscience/bloomz-1b7 bigscience/bloomz-3b bigscience/bloomz-7b1-mt
1.1B 1.7B 3B 7B
BLOOMZ BLOOMZ BLOOMZ BLOOMZ
Bigscience Bloom Rail 1.0 Bigscience Bloom Rail 1.0 Bigscience Bloom Rail 1.0 Bigscience Bloom Rail 1.0
(Muennighoff et al., 2023) (Muennighoff et al., 2023) (Muennighoff et al., 2023) (Muennighoff et al., 2023)
deepseek-ai/DeepSeek-V4-Flash
158B
DeepSeek
DeepSeek License
(DeepSeek-AI, 2026)
google/gemma-3-4b-it google/gemma-2-9b-it google/gemma-4-26B-A4B-it google/gemma-4-31B-it
4B 9B 27B 31B
Gemma 3 Gemma Gemma Gemma
Gemma Gemma Gemma Gemma
(Gemma Team, 2025) (Team et al., 2024) (Google, 2026b) (Google, 2026b)
meta-llama/Llama-3.2-3B-Instruct meta-llama/Llama-3.1-8B-Instruct meta-llama/Llama-4-Scout-17B-16E-Instruct meta-llama/Llama-4-Maverick-17B-128E-Instruct meta-llama/Llama-3.3-70B-Instruct
3B 8B 109B 402B 70B
LLaMA 3.2 LLaMA 3.1 LLaMA 4 LLaMA 4 LLaMA 3.3
LLaMA License LLaMA License LLaMA License LLaMA License LLaMA License
(Meta, 2024a) (Grattafiori et al., 2024) (Meta, 2025) (Meta, 2025) (Meta, 2024b)
microsoft/Phi-4-mini-instruct microsoft/Phi-3.5-mini-instruct
3B 4B
Phi-4 Phi-3.5
MIT MIT
(Microsoft et al., 2025) (Abdin et al., 2024)
claude-haiku-4-5 claude-sonnet-4-6
N/D N/D
Claude Claude
Proprietary Proprietary
(Anthropic, 2025) (Anthropic, 2026)
gemini-3.1-flash-lite gemini-3.5-flash
N/D N/D
Gemini Gemini
Proprietary Proprietary
(Google, 2026a) (Google DeepMind, 2026)
gpt-5.4-mini gpt-5.4
N/D N/D
GPT GPT
Proprietary Proprietary
(Singh et al., 2026) (Singh et al., 2026)
Table 14: Language models evaluated in this study. Model sizes are reported when publicly disclosed; N/D denotes not disclosed.
Under the English prompt, Gemini-3.5-Flash reaches 97.86% on chemistry, 98.60% on biology, and 98.86% on mathematics, while DeepSeek-V4Flash reaches 98.71% on physics. These scores remain nearly unchanged under the Urdu prompt, and in some cases increase slightly. The consistency across prompt languages suggests that scientific and mathematical concepts transfer relatively cleanly once the model can process Urdu input. Humanities presents a much harder challenge. Islamic studies and Urdu grammar remain accessible for the strongest models, with Gemini-3.5Flash reaching 94.25% on Islamic studies and 88.34% on Urdu grammar under the English prompt. In contrast, Urdu literature remains difficult across the entire model suite. Even the strongest model reaches only 80.35% under the English prompt and 80.81% under the Urdu prompt. Most other proprietary and open-source models perform substantially worse, often trailing by another 10 to 20 points. Urdu language occupies an intermediate position, with top scores near 89%.
Across nearly all capable models, the same ordering persists: Islamic studies > Urdu grammar > Urdu language > Urdu literature. This consistency suggests that the differences reflect genuine variation in subject difficulty rather than isolated model behavior. Social Sciences contains both highly accessible and consistently difficult subdomains. Geography, civics, sociology, psychology, and commerce all exceed 93% accuracy for the strongest models. Pakistan studies also remains relatively strong despite its large size. In contrast, current and international affairs and psychometrics stand out as the two hardest Social Sciences subdomains. Current and international affairs peaks at roughly 78% under both prompts, likely because many questions depend on time-sensitive world knowledge beyond pretraining cutoffs. Psychometrics is even more difficult: no model in the evaluation exceeds 60% accuracy under either prompt language. This suggest that both subdomains are challenging even for the strongest models.
Zero-shot Urdu Prompt System Prompt: ۔
۔
ٓ
MCQ ٔ ، ٓ
<
ؕ :
D C, B, A > :Answer key ٓ
،
> : Answer text <
۔
، ۔
ٔ ؕ
، ٓ
،
ٔ
Answer text
User Prompt: {subdomain} – {domain} : {level} : {question} : {A} : (A {B} : (B {C} : (C {D} : (D Answer key: Answer text:
Figure 17: Urdu prompt for multiple-choice question answering with strict answer formatting requirements.
The smaller Profession and Other domains follow patterns similar to Social Sciences, with proprietary models reaching the low 90s and smaller open-source models trailing behind. These domains do not introduce additional failure modes. The subdomain results further clarify the behavior of smaller open-source models. Among models with fewer than 25B parameters, Gemma-29B-IT performs best on Humanities subjects, including Urdu language, Urdu grammar, ethics, and fine arts, while Qwen3-8B leads on STEM subjects such as chemistry, mathematics, computer science, and physics. This pattern mirrors the domain-level results: Qwen3-8B retains relatively strong scientific knowledge but struggles on Urducentered humanities content, whereas Gemma-29B-IT shows more balanced performance across subdomains. The Urdu-targeted models, Qalb-1.08B and Alif-1.0-8B, do not lead any subdomain and remain below similarly sized general-purpose models. BLOOMZ checkpoints remain close to the random baseline on most subdomains and should be interpreted alongside the high invalidoutput rates reported in Section 5.4.
Model
STEM
Hum.
Gap
Gemini-3.5-Flash DeepSeek-V4-Flash GPT-5.4 Gemini-3.1-Flash-Lite Qwen3.6-35B-A3B Claude-Sonnet-4.6 Gemma-4-31B-IT LLaMA-4-Maverick-17B Claude-Haiku-4.5 Qwen3.6-27B GPT-5.4-mini Gemma-4-26B-A4B-IT LLaMA-4-Scout-17B-16E LLaMA-3.3-70B Qwen3-8B Ministral-3-8B Gemma-2-9B-IT Qwen3-4B Ministral-3-3B Gemma-3-4B-IT LLaMA-3.1-8B LLaMA-3.2-3B Phi-4-mini Qalb-1.0-8B Phi-3.5-mini Alif-1.0-8B BLOOMZ-7B BLOOMZ-1.7B BLOOMZ-3B BLOOMZ-1.1B
97.81 97.57 97.40 97.09 96.32 96.26 93.86 92.38 91.96 91.12 88.25 87.21 85.59 78.39 74.37 71.37 69.02 68.75 57.25 51.79 46.49 37.24 37.08 36.26 33.83 33.27 29.24 28.74 26.56 24.53
85.31 67.32 74.82 74.38 58.12 72.69 63.25 63.25 59.31 55.71 62.35 57.73 56.55 56.10 30.87 45.74 48.08 43.00 43.07 38.27 37.61 29.32 28.70 32.72 27.25 29.00 30.88 28.76 27.70 25.83
12.50 30.25 22.58 22.71 38.20 23.57 30.61 29.13 32.65 35.41 25.90 29.48 29.04 22.29 43.50 25.63 20.94 25.75 14.18 13.52 8.88 7.92 8.38 3.54 6.58 4.27 –1.64 –0.02 –1.14 –1.30
Table 15: STEM–Humanities accuracy gap under the Urdu prompt. Models with gaps near zero either score at chance on both domains (BLOOMZ) or have an unusually low STEM score for their scale (Qalb-1.0-8B, Alif-1.0-8B). Values are taken directly from Table 4.
E.1.2 The Psychometrics Gap Psychometrics is the most difficult subdomain in our evaluation. The best English-prompt accuracy reaches only 57.30% (Gemini-3.5-Flash), while the best Urdu-prompt accuracy reaches 52.97% (Claude-Sonnet-4.6). No model exceeds 60% under either prompt setting, in contrast with the 90– 98% accuracies achieved on many STEM and Social Sciences subjects. The difficulty appears specific to the content rather than the prompt language. English- and Urdu-prompt results remain close, and model rankings on psychometrics largely mirror their overall rankings. Psychometrics questions in Urdu SSC/HSSC curricula frequently emphasize analogies, number series, logical patterns, and aptitudestyle reasoning tasks that require abstract structure recognition rather than factual recall. These results suggest that current LLMs still struggle on reasoning-heavy Urdu educational content even when they perform strongly on factual subjects.
STEM
Model CHM
BIO
GSC
MTH
Humanities CSC
PHY
ULT
ULG
ISL
UGR
Social Sciences ETH
Profession
Other
EDU
SOC
HPE
CIV
GEO
PSY
CIA
COM
PMT
PRD
PRS
GKN
85.27 74.64 82.06 77.39 72.85 77.87 76.32 75.12
95.12 88.82 90.75 86.76 86.25 88.43 88.82 88.05
92.17 71.16 81.13 67.65 65.63 80.19 69.54 68.73
94.06 78.37 87.26 82.96 77.93 87.41 77.93 80.00
94.29 79.52 87.62 82.86 75.08 84.76 81.90 80.63
94.78 81.09 90.00 81.52 77.83 86.96 83.70 83.70
71.56 42.56 54.92 47.60 48.51 52.40 39.13 43.02
95.04 77.54 86.05 77.07 72.81 79.43 78.25 78.96
40.22 44.32 40.54 29.19 39.46 34.59 47.57 41.08
89.30 71.60 79.57 71.79 68.09 78.79 65.18 70.04
90.46 79.83 88.29 81.78 80.91 84.82 85.47 81.56
86.44 70.55 79.49 73.77 69.52 80.66 69.60 69.82
24.26 32.45 35.94 32.85 58.99 46.42 36.38 49.61 49.72 61.08 39.36 32.52 52.04 56.45
21.65 38.32 34.05 39.79 65.07 51.08 43.42 55.02 55.74 62.68 43.95 34.69 56.70 62.68
24.42 42.29 39.43 40.78 75.71 64.78 48.33 65.17 62.47 74.55 54.88 38.82 67.99 71.08
24.53 32.25 31.89 34.58 51.35 41.64 35.85 43.53 46.63 53.91 31.85 31.27 49.33 52.43
24.00 34.82 39.19 31.66 64.00 58.07 48.89 56.89 51.26 67.26 42.67 33.48 57.19 59.85
23.65 29.73 29.39 30.99 62.86 53.33 42.70 52.70 55.17 65.40 37.62 31.75 56.83 57.46
25.33 27.67 33.48 33.68 61.30 46.09 32.17 47.61 50.87 63.70 37.39 30.22 55.65 59.35
25.93 29.10 28.60 27.21 40.96 31.58 21.74 26.77 37.07 37.30 21.74 18.76 36.84 29.98
26.48 31.26 32.59 32.25 63.12 49.65 36.88 47.75 49.88 59.57 39.01 34.75 50.35 57.92
24.86 36.07 24.55 28.89 30.81 23.78 31.89 30.81 28.65 42.70 29.19 20.00 29.73 30.27
25.10 29.26 27.42 26.40 49.81 41.25 34.82 41.63 42.50 51.75 34.63 26.65 43.19 45.91
27.11 45.62 38.79 38.24 71.37 55.10 37.31 56.18 56.40 68.33 46.85 38.39 63.12 68.55
25.63 31.43 28.39 28.11 54.58 45.57 35.60 44.54 47.91 54.43 32.67 28.28 47.84 50.18
71.59 88.28 91.98 94.61 78.05 87.07
80.26 91.40 94.16 94.27 81.15 88.20
75.72 83.85 88.04 88.54 76.32 82.30
87.66 92.80 93.70 93.70 89.85 91.26
74.90 83.96 90.70 93.40 67.52 86.93
81.78 92.74 94.37 94.96 83.11 91.56
82.06 92.54 94.60 95.87 83.02 91.43
85.43 94.35 95.43 95.00 84.78 91.74
48.24 65.45 67.05 78.49 56.75 70.02
79.67 91.49 94.56 94.80 79.43 91.25
40.76 52.43 38.92 57.30 35.68 42.70
69.84 85.60 87.55 92.22 73.74 83.66
82.65 88.94 93.28 91.97 85.03 88.29
74.14 86.01 86.08 91.43 75.24 84.10
40.10 36.57
40.84 36.38
42.86 36.12
52.52 47.69
36.07 28.44
47.21 47.11
47.19 43.49
40.39 35.22
26.44 30.89
40.28 38.06
25.41 24.32
33.27 37.74
47.24 43.17
41.04 39.56
FNA
PKS
ECO
Open-source Models: > 25B Parameters DeepSeek-V4-Flash Gemma-4-26B-A4B-IT Gemma-4-31B-IT LLaMA-3.3-70B LLaMA-4-Scout-17B-16E LLaMA-4-Maverick-17B-128E Qwen3.6-27B Qwen3.6-35B-A3B
97.65 85.47 91.88 79.06 84.19 91.99 91.88 88.14
98.06 83.57 93.45 78.41 83.24 90.23 86.57 86.25
96.54 85.64 92.33 86.50 86.29 91.79 89.31 89.42
98.40 85.05 93.49 77.40 78.08 91.89 89.61 86.64
95.86 88.46 93.44 85.54 85.78 92.70 92.95 92.47
98.71 88.41 96.46 81.32 88.57 94.04 93.72 90.66
56.71 45.75 51.63 46.25 47.79 52.47 45.49 46.13
77.11 66.10 71.93 62.49 62.77 70.26 61.70 62.37
91.24 78.02 84.64 75.55 75.02 83.37 74.68 78.49
79.58 71.16 77.25 64.68 65.21 74.07 69.71 67.46
88.60 71.93 84.21 79.82 73.25 83.77 77.19 77.63
BLOOMZ-1.1B BLOOMZ-1.7B BLOOMZ-3B BLOOMZ-7B Gemma-2-9B-IT Gemma-3-4B-IT LLaMA-3.2-3B LLaMA-3.1-8B Ministral-3-3B Ministral-3-8B Phi-4-mini Phi-3.5-mini Qwen3-4B-Instruct Qwen3-8B
24.49 26.76 25.27 26.46 65.28 46.37 32.48 44.98 49.79 61.43 35.47 33.44 66.88 66.88
24.06 25.98 30.66 27.94 64.02 46.29 34.05 42.53 48.98 62.41 28.46 28.57 57.79 60.47
19.03 32.58 29.68 29.66 71.06 55.83 40.28 51.94 59.40 70.52 36.43 29.70 68.68 70.63
24.11 27.88 28.16 27.02 64.16 43.49 38.47 44.86 54.68 65.41 45.21 43.72 71.12 75.57
25.64 34.90 26.72 28.97 71.57 60.27 42.41 52.86 60.39 77.52 41.88 34.51 77.40 77.52
24.32 24.79 26.18 26.04 66.67 47.67 35.10 44.28 58.94 71.98 35.48 33.01 72.14 76.01
27.90 27.54 30.08 27.15 40.98 33.96 21.42 28.21 41.25 39.38 25.57 19.54 38.11 30.77
27.45 27.86 29.19 27.04 51.90 40.64 31.72 44.13 44.09 47.22 32.87 28.15 43.54 46.07
25.10 29.93 31.88 29.15 55.58 43.22 32.53 46.43 48.73 57.31 32.46 26.25 49.43 50.50
25.96 23.44 30.65 30.45 54.37 41.14 35.98 46.30 46.30 52.51 30.29 21.43 51.59 51.59
Claude-Haiku-4.5 Claude-Sonnet-4.6 Gemini-3.1-Flash-Lite Gemini-3.5-Flash GPT-5.4-mini GPT-5.4
90.17 96.26 97.76 97.86 90.28 96.26
89.26 96.78 97.10 98.60 89.26 96.56
87.69 94.38 96.00 96.76 89.52 94.06
91.98 97.83 96.46 98.86 80.59 92.81
91.98 95.26 96.23 95.95 90.77 94.90
92.91 98.07 97.75 98.55 90.02 96.46
47.64 61.90 63.53 80.35 51.87 57.91
64.51 81.05 83.06 88.69 71.41 78.15
81.27 90.98 90.78 94.25 82.10 90.38
68.92 84.39 87.04 88.34 72.22 78.17
78.07 87.72 92.11 89.47 80.70 84.65
Alif-1.0-8B Qalb-1.0-8B
36.04 35.26
37.59 34.26
48.58 45.68
36.87 33.68
47.67 43.38
40.03 36.71
20.51 27.09
31.03 31.01
33.98 35.67
26.06 32.41
40.53 40.79
87.16 71.62 82.43 75.00 76.35 81.76 79.73 79.05
91.40 72.72 80.05 76.83 67.81 81.27 66.28 68.18
93.94 73.10 84.34 75.74 73.98 83.35 78.72 78.50
Open-source Models: ≤ 25B Parameters 28.07 32.00 31.56 32.70 65.35 50.44 41.67 53.07 50.44 64.47 41.67 35.09 48.68 60.09
21.38 23.57 31.43 29.46 64.86 54.73 27.03 49.32 58.78 56.76 26.35 20.27 60.14 46.62
24.91 31.46 30.54 31.48 52.30 42.53 33.83 47.81 42.27 52.19 33.61 27.02 43.91 45.01
Proprietary Models 77.55 89.19 85.14 87.16 79.05 86.49
Urdu Models 35.14 41.22
Table 16: Subdomain-level model performance on UrduMMLU under the English prompt. Accuracy (%) across all 26 subdomains grouped by domain. Subdomains are ordered within each domain by dataset size (descending); see Table 13 for acronym expansions. Boxed values mark the best overall score per column, while bold values indicate the best score within each model group. STEM
Model CHM
BIO
GSC
MTH
Humanities CSC
PHY
ULT
ULG
ISL
UGR
Social Sciences ETH
Profession
Other
EDU
SOC
HPE
CIV
GEO
PSY
CIA
COM
PMT
PRD
PRS
GKN
85.73 76.91 82.30 74.28 72.73 77.87 75.96 81.99
93.82 89.59 91.13 85.48 87.79 89.72 89.33 91.51
90.27 73.72 81.81 65.63 64.96 82.48 71.43 87.13
94.50 81.33 88.30 80.15 79.56 88.59 81.19 91.06
93.62 80.95 86.98 78.89 78.10 85.56 84.29 93.15
94.10 84.35 90.43 78.26 81.74 88.70 85.00 91.70
67.13 44.04 53.78 48.51 46.68 54.82 40.96 49.20
94.09 81.09 86.76 76.83 74.70 82.27 81.09 89.83
39.56 41.76 41.76 35.14 36.22 36.76 43.78 43.24
87.94 72.76 79.57 65.37 67.32 76.65 64.59 81.66
91.11 83.48 89.15 77.87 78.52 82.86 85.47 87.64
84.88 71.79 78.39 71.65 69.01 80.81 70.55 77.99
24.48 32.27 34.97 31.42 63.73 48.07 36.49 49.28 53.36 66.48 40.02 31.82 57.22 58.82
23.68 38.78 31.45 34.35 64.35 51.91 45.69 58.25 59.69 66.15 45.57 36.53 58.49 65.27
25.71 39.66 38.97 37.48 80.08 67.74 53.34 68.89 71.98 80.33 55.01 37.98 72.11 74.94
26.68 35.04 28.27 33.73 52.83 44.47 35.71 42.45 45.82 58.09 35.71 31.49 50.54 54.77
24.59 33.63 30.43 40.21 69.04 56.74 49.63 56.30 56.00 69.63 45.33 32.34 60.74 63.69
23.97 30.13 27.37 33.84 68.25 53.81 42.54 53.33 58.57 70.48 39.21 31.37 56.83 61.72
25.22 27.93 32.26 30.89 64.13 47.61 34.57 44.35 53.91 67.17 34.13 33.04 56.30 62.61
27.00 30.50 27.60 29.81 42.33 29.75 23.11 27.00 37.99 38.67 21.28 21.74 37.07 28.38
26.24 29.38 34.21 33.64 62.65 51.06 39.24 47.75 53.90 65.72 40.43 37.91 55.32 61.61
24.86 37.57 28.12 21.62 32.43 29.19 34.59 30.27 32.97 44.32 28.11 16.76 30.81 34.59
22.37 28.97 30.00 28.52 50.78 43.39 33.27 41.05 45.03 54.28 35.21 30.54 45.14 42.61
24.73 36.75 34.95 37.70 74.84 57.70 44.03 59.00 60.30 69.63 42.52 38.34 62.26 74.55
24.47 27.51 26.75 30.73 55.82 46.23 38.02 44.98 48.86 57.00 35.85 30.57 47.69 49.38
72.88 88.07 92.08 94.56 77.47 90.24
81.15 91.29 94.16 94.82 82.14 93.94
76.20 84.33 87.92 88.76 77.15 86.96
87.15 93.70 93.44 93.70 89.72 92.42
76.82 85.04 91.24 93.67 69.81 92.32
83.70 91.70 94.81 94.96 83.11 94.22
84.76 92.70 95.08 95.71 81.75 94.29
87.83 93.91 94.13 95.65 89.57 93.48
48.60 66.82 68.42 77.80 59.73 69.34
82.27 91.49 94.80 95.27 85.11 92.43
38.80 52.97 37.30 52.43 36.76 40.00
70.82 84.63 87.55 91.44 75.29 84.05
86.55 90.02 93.49 91.32 84.38 90.46
74.29 85.86 85.57 91.72 75.09 83.74
35.09 36.69
33.24 37.98
34.58 39.97
44.93 44.58
34.42 34.73
42.25 42.64
40.80 41.94
36.57 36.05
29.97 30.82
35.87 32.61
22.31 30.26
37.28 38.20
38.21 40.70
42.93 42.55
FNA
PKS
ECO
Open-source Models: > 25B Parameters DeepSeek-V4-Flash Gemma-4-26B-A4B-IT Gemma-4-31B-IT LLaMA-3.3-70B LLaMA-4-Scout-17B-16E LLaMA-4-Maverick-17B-128E Qwen3.6-27B Qwen3.6-35B-A3B
97.75 87.91 92.52 74.04 85.15 93.26 92.95 97.22
97.63 84.32 93.98 77.12 85.82 91.19 88.83 96.33
96.65 86.93 93.63 83.69 86.50 92.44 89.42 93.95
98.17 85.27 93.61 72.26 79.63 91.48 88.93 97.03
96.71 89.67 94.41 83.72 89.19 91.86 93.07 96.59
98.87 90.34 95.65 80.52 88.08 94.69 94.85 97.10
54.79 46.58 51.44 47.31 47.26 53.27 45.47 46.42
76.29 65.38 71.25 60.15 61.18 70.18 61.02 63.97
91.03 78.09 84.24 74.47 75.42 81.83 75.02 81.33
78.38 69.84 76.32 64.29 65.61 72.09 67.33 71.05
BLOOMZ-1.1B BLOOMZ-1.7B BLOOMZ-3B BLOOMZ-7B Gemma-2-9B-IT Gemma-3-4B-IT LLaMA-3.2-3B LLaMA-3.1-8B Ministral-3-3B Ministral-3-8B Phi-4-mini Phi-3.5-mini Qwen3-4B-Instruct Qwen3-8B
26.18 28.95 26.21 27.00 66.35 46.79 32.80 42.20 52.35 66.67 36.54 33.37 66.45 71.14
24.60 26.57 26.37 30.25 66.17 49.52 35.98 45.97 50.38 69.60 32.44 28.91 56.82 64.16
19.98 28.78 27.65 30.41 73.33 58.32 39.52 51.51 62.85 73.43 32.72 30.09 69.55 75.49
25.34 29.03 25.24 28.78 64.38 44.86 37.56 43.38 55.02 66.21 45.32 42.59 71.69 77.92
26.61 34.85 25.88 30.83 74.97 62.21 41.31 51.76 64.64 79.10 41.92 34.51 76.31 79.75
24.80 22.77 28.29 28.52 69.57 48.95 36.55 43.64 59.90 75.04 33.33 34.47 74.72 80.78
25.95 28.31 29.45 30.55 41.92 33.38 24.75 32.09 39.84 39.41 23.84 25.40 39.00 17.41
25.47 28.73 21.96 30.50 52.02 40.84 33.62 42.35 43.44 47.58 35.69 30.85 44.61 42.86
25.52 31.31 29.56 30.95 56.58 46.43 34.13 43.75 49.70 58.32 32.46 27.91 49.50 50.52
26.59 28.57 29.89 29.92 56.48 44.84 36.24 44.84 48.54 53.70 32.54 26.79 51.32 44.84
Claude-Haiku-4.5 Claude-Sonnet-4.6 Gemini-3.1-Flash-Lite Gemini-3.5-Flash GPT-5.4-mini GPT-5.4
91.77 96.69 97.65 97.86 89.64 97.44
92.05 96.56 98.07 98.60 91.19 97.85
89.42 94.38 96.76 97.08 91.04 95.79
92.91 97.26 96.35 98.97 78.20 99.20
92.95 95.50 96.23 96.10 90.16 95.87
93.23 97.58 97.42 98.23 89.21 98.55
47.84 62.13 64.00 80.81 51.03 67.09
66.10 80.65 83.26 88.89 70.82 78.98
82.10 90.92 90.98 94.19 82.63 90.71
71.16 85.05 85.32 88.76 72.35 80.95
77.19 87.72 90.35 89.04 82.02 89.47
Alif-1.0-8B Qalb-1.0-8B
30.10 33.33
29.75 36.80
34.54 37.76
33.29 34.49
40.41 44.32
32.94 31.82
27.89 31.85
29.94 32.27
31.48 34.99
28.30 32.90
31.52 38.12
85.53 74.56 82.46 77.09 77.63 83.33 80.26 85.90
85.81 75.34 85.62 75.68 75.68 76.35 78.38 75.68
89.74 71.19 80.53 73.72 69.71 81.64 66.91 82.88
93.04 78.50 84.67 75.08 76.07 84.79 81.48 89.85
Open-source Models: ≤ 25B Parameters 28.95 31.71 29.03 37.27 65.79 50.00 40.79 57.02 54.82 71.93 39.04 32.74 46.05 64.32
21.62 21.93 22.92 38.53 68.92 54.05 35.14 46.62 51.35 56.76 28.38 26.53 60.81 42.47
26.02 33.50 29.42 32.42 52.30 42.85 36.89 47.02 43.06 54.62 33.88 29.15 43.43 46.48
Proprietary Models 78.38 84.46 85.81 89.19 79.73 89.86
Urdu Models 33.80 42.86
Table 17: Subdomain-level model performance on UrduMMLU under the Urdu prompt. Accuracy (%) across all 26 subdomains grouped by domain. Subdomains are ordered within each domain by dataset size (descending); see Table 13 for acronym expansions. Boxed values mark the best overall score per column, while bold values indicate the best score within each model group.
E.1.3
Urdu Tuning Fails on Literature
Urdu literature is the largest subdomain in URDUMMLU, with 5,859 items, and contains content with limited overlap with English-dominated pretraining corpora, including classical poetry, prosody, and literary history. It therefore provides a useful test of Urdu-focused training on culturally grounded knowledge. Figure 18 compares two Urdu-targeted 8B models, Qalb-1.0-8B and Alif1.0-8B, with two general-purpose 8B instructiontuned models, Qwen3-8B and Ministral-3-8B.
The Urdu-targeted models do not outperform the general-purpose baselines on this subdomain. Ministral-3-8B achieves the highest accuracy under both prompts at 39.4%, while Qalb-1.0-8B and Alif-1.0-8B remain below 32%. Qwen3-8B performs competitively under the English prompt (30.8%) but drops to 17.4% under the Urdu prompt. In contrast, both Urdu-targeted models improve under the Urdu prompt, suggesting that Urdu-specific tuning improves instruction following more than literary knowledge. Overall, Urdu literature remains challenging even for Urdu-targeted LLMs.
Accuracy (%) on urdu literature
50
English prompt Urdu prompt
39.4
40 31.9
30
39.4
30.8 27.9
27.1
20.5
20
17.4
10
0
Qalb-1.0-8B
Alif-1.0-8B
Qwen3-8B
Ministral-3-8B
Figure 18: Urdu literature accuracy for four 8Bclass instruction-tuned models under English and Urdu prompts. Ministral-3-8B performs best under both settings, while Qwen3-8B shows the largest promptlanguage drop.
E.1.4 English-Prompt Subdomain Accuracy Table 16 reports per-subdomain accuracy for all 30 models under the English prompt. The table groups subdomains by domain and orders them by dataset size within each group, so earlier columns contribute more strongly to the corresponding domain-level scores in Table 4. Acronym expansions appear directly in the table header. The results provide a fine-grained view of model behavior across subjects: Gemini-3.5-Flash remains consistently strong across nearly all subdomains, DeepSeek-V4-Flash approaches proprietary-level performance on STEM subjects but drops on Urdu language and literature, and the BLOOMZ models remain close to the random baseline across most subjects. E.1.5 Urdu-Prompt Subdomain Accuracy Table 17 reports the same per-subdomain breakdown under the Urdu prompt. The table follows the same structure and ordering as Table 16, which allows direct comparison between the two prompt settings. Most differences remain small, reinforcing the main finding from Section 5 that the difficulty of URDUMMLU comes primarily from the question content rather than the instruction language. For most proprietary models and the Gemma family, English- and Urdu-prompt accuracies remain nearly identical across the majority of subdomains. A few model-specific shifts become clearer at the subdomain level. Qwen3.635B-A3B improves substantially under the Urdu prompt, driven mainly by STEM subjects, where several subdomain scores rise into the mid-90s under the Urdu prompt.
In contrast, Qwen3-8B loses accuracy primarily on Humanities subjects, especially Urdu language and Urdu literature, which explains its large drop in overall Humanities performance under the Urdu prompt. The Urdu-targeted models also show modest gains on several Humanities subdomains under the Urdu prompt, although these improvements do not substantially change their overall ranking. Together, these patterns further support the conclusion that prompt language plays a secondary role compared with the underlying educational and cultural knowledge required by the benchmark.
F Invalid-Output Examples Section 5.4 reports invalid-output rates across the model suite; this appendix provides representative examples of the corresponding failure modes. Each example is drawn from an actual model prediction under the Urdu prompt setting. We organize the examples by failure type in order to highlight recurring decoding behaviors and illustrate how invalid generations manifest in practice across different models. Repetition collapse: In some cases, the model enters a degenerate decoding loop and repeatedly emits the same token sequence without producing a meaningful or valid answer. Example F illustrates this behavior for BLOOMZ-7B, which repeatedly generates the token “Question:” dozens of times instead of producing a task-relevant response. Example: repetition collapse (BLOOMZ7B, Urdu literature, gold = B) Question: Options:
ٔ • A. • B. (gold) • C. • D. Model output: Question: Question: Question: Question: …
Prompt echo: The model copies part of the user prompt instead of answering the question. Example F shows BLOOMZ-1.7B reproducing the beginning of the question prompt and terminating before generating a valid answer.
Example: prompt echo (BLOOMZ-1.7B, Islamic studies, gold = B) ! ؕ
Question: ٔ۔
Example: empty output (BLOOMZ-3B, chemistry, gold = A)
Options: • A. ٔ • B.
(gold)
Question: ۔ Options:
• C. • D. Model output: :
Empty or placeholder output: The model emits a nearly empty response, often copied directly from a blank marker in the question. Example F shows BLOOMZ-3B returning only a placeholder token.
Humanities – islamic studies | : ؕ
• A. SSC-II
!
:
Refusal or clarification request: Instead of selecting an answer, the model returns a conversational clarification request. Example F shows Qalb1.0-8B treating the MCQ as an ambiguous user query. Example: refusal (Qalb-1.0-8B, mathematics, gold = A) Question: ۔
Options: • A. (gold) • B. • C. • D. Model output: ؟
ٓ
ٓ
۔
System-prompt echo: The model reproduces the system prompt instead of answering the question. Example F shows Alif-1.0-8B repeating the assistant role description without generating an answer. Example: system-prompt echo (Alif-1.08B, chemistry, gold = B) ؕ ٔ
Question:
۔
Options: • A. NaOH • B. H2 SO4 (gold) • C. HCl • D. KOH Model output: ۔ MCQ ٓ
ؕ
ٔٓ
ؕ
(gold)
• B. ٔ ٓ • C. ٔ ٓ • D. ٔ ٓ ؕ Model output: ___
Discussion: All five examples produce outputs that cannot be mapped to a valid answer option and therefore contribute to the invalid-output rate rather than to model accuracy. The failures arise from different causes: repetition collapse and empty outputs reflect decoding instability, prompt and system-prompt echoes reflect instruction-following failures, and clarification requests reflect conversational misalignment with the MCQ format. These behaviors are not unique to URDUMMLU, but their concentration under the Urdu prompt for weaker models motivates reporting invalid-output rates alongside accuracy.