The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs Yang Zhang1† , Xiao Fei1 , Amr Mohamed1,2 , Sarah Almeida Carneiro1 , Mersin Konomi2 , Mingmeng Geng3 , Ahmed Asaad4 , Guokan Shang2 , Michalis Vazirgiannis1,2† 1 †
Ecole Polytechnique, 2 MBZUAI, 3 ENS-PSL, 4 Durham University
Correspondence: [email protected], [email protected]
Abstract
arXiv:2606.07422v1 [cs.CL] 5 Jun 2026
Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language. Existing evaluations face two key limitations: many rely on parallel template-based questions that may not reflect how cultural knowledge naturally appears, and raw accuracy conflates general language proficiency with language-conditioned knowledge access. We address these issues with a controlled framework built on realworld cultural questions collected from regional benchmarks and local sources. By crossing question type (culture-agnostic vs. culturespecific) with query language (English vs. local language), and estimating ability with a shared 1PL item response theory model, we separate proficiency from localized knowledge access. Across 13 locales and roughly 80 models, we find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency. However, after accounting for this proficiency gap, local languages show a positive knowledge-access advantage in nearly all locale–model settings. This advantage is often masked in raw accuracy but becomes more visible for frontier, regionally aligned, or language-adapted models. Our results suggest that weaker local-language performance does not necessarily imply weaker cultural knowledge; rather, local cultural knowledge may be more accessible through the local language but hidden by limited language proficiency.1
1
Figure 1: For cultural questions, local knowledge may be masked by weaker local-language proficiency.
cultural knowledge is often unevenly distributed across languages: some content is produced and discussed primarily in a local language, while training data and evaluation benchmarks remain dominated by English — the global language. This raises two linked questions: Q1: which query language provides better access to local cultural knowledge? and Q2: whether that knowledge is preferentially accessed through the language in which it is grounded? Prior work has yielded mixed findings. Several studies suggest that LLMs remain strongly English-centric: knowledge transfers asymmetrically across languages, and translating into English can be a competitive querying strategy, especially for lower-resource languages (Yin et al., 2022; Liu et al., 2025; Romanou et al., 2025). Another line reports that local-language prompting can help when the query language aligns with the cultural context, and that different languages may activate different knowledge in the same model (Ying et al., 2025; Agarwal et al., 2025; Jain and Chopra, 2026; Myung et al., 2024).
Introduction
Large language models are increasingly regarded as multilingual systems, capable of answering questions written in different languages and concerning history, law, customs, and everyday life across diverse cultural contexts. However, the relevant 1
Dataset is attached to the submission and the public link will be provided upon acceptance.
We argue this disagreement stems partly from 1
Our contributions are threefold:
how cultural knowledge is measured. Many cultural benchmarks build parallel questions from human-defined templates across languages, which may not reflect how local knowledge naturally appears in each language (Ying et al., 2025). Even with faithful benchmarks, most cross-lingual comparisons rely on raw accuracy, which conflates language proficiency with access to languageconditioned knowledge—two effects that may work in opposite directions. A model may score higher in English simply because it understands English better, while a local language may better activate culturally grounded knowledge yet still yield lower accuracy due to weaker proficiency. A raw English-versus-local gap therefore reports only the net sign of these factors, leaving open which language truly provides better access to local knowledge. To separate these effects, we compare English and local-language performance on both cultureagnostic and culture-specific questions. The culture-agnostic gap estimates general language proficiency (G LOBAL G AP), while the culturespecific gap captures both proficiency and cultural knowledge access (L OCAL G AP). Their difference gives K NOWLEDGE G AP, which isolates languageconditioned knowledge access. However, because culture-agnostic and culturespecific items are drawn from different question pools of different intrinsic difficulties, raw accuracy gaps across the two sets are not directly comparable. A lower accuracy on culture-specific questions may simply reflect harder items rather than weaker knowledge access. Therefore, we estimate all gaps with a shared 1PL IRT model (Lalor et al., 2016; Martínez-Plumed et al., 2019), which jointly estimates item difficulty and model ability on a common scale, enabling the comparison by subtraction. We apply this framework across 13 locales and roughly 80 models, using culture-specific items from established regional benchmarks rather than synthetic templates. Our results support the masked advantage hypothesis: models show a consistent English proficiency advantage on culture-agnostic questions, while local-language performance on culture-specific questions is mixed. After controlling for proficiency, however, K NOWLEDGE G AP is positive in nearly every locale–model cell, including many where L OCAL G AP is negative. This suggests that local languages often provide better access to cultural knowledge, but this advantage is hidden by weaker local-language proficiency.
1. We introduce a measurement framework that separates language proficiency from languageconditioned cultural knowledge access, casting all three gaps as ability contrasts under a single 1PL (Rasch) IRT model that accounts for difficulty differences between culture-agnostic and culture-specific items. 2. We conduct a large-scale empirical study across 13 local languages and multiple model categories using established benchmarks rather than synthetic probes. 3. We provide evidence for a masked locallanguage knowledge advantage that is present in nearly all settings but surfaces in raw accuracy only when proficiency does not work against it. Weaker local-language performance therefore appears to stem largely from limited proficiency in that language rather than from missing cultural knowledge itself, with implications for how multilingual LLMs are evaluated and deployed.
2
Related Work
Multilingual, Cultural, and Regional Evaluation Benchmarks. A growing body of benchmarks evaluates LLMs beyond English. Translation-based suites such as Global-MMLU (Singh et al., 2025) show that multilingual evaluation can still inherit English-centric cultural assumptions. While native or regional benchmarks such as INCLUDE (Romanou et al., 2025), CMMLU (Li et al., 2024), ArabicMMLU (Koto et al., 2024), KMMLU (Son et al., 2025), MILU (Verma et al., 2025), and GreekMMLU (Zhang et al., 2026) collect questions from local exams, institutions, and knowledge sources. In parallel, cultural benchmarks such as BLEnD (Myung et al., 2024) extend evaluation from academic knowledge to everyday culture but with a predefined template. There are also works evaluating LLMs under code-switched test, revealing mixed effect of English and local language (Mohamed et al., 2025). These benchmarks establish that a language gap exists, but report raw accuracy without separating language proficiency from knowledge access; our framework supplies that decomposition. Disentangling Language from Knowledge. Closest to our setting, Ying et al. (2025) separates language from cultural context and finds that 2
local-language prompts generally improve cultural knowledge access, but it relies on parallel templatebased cultural questions, which may not capture how localized knowledge naturally appears in native data. Building on this direction, Agarwal et al. (2025) further asks whether some knowledge is more accessible in a specific language than in English, while Jain and Chopra (2026) argues that changing the query language can also shift the cultural assumptions activated by the model. We instead collected real-world cultural questions and cast the problem as a difference-in-differences of model ability estimated by a Bayesian itemresponse model (Lalor et al., 2016; Rodriguez et al., 2021). This lets us recover a knowledge advantage that raw accuracy masks.
of interest with differences in their difficulty—a concern for our central contrast, which compares the disjoint pools QCS and QCA . We estimate θjsk , the ability of LLM j on question set s posed in language k, with an Item-Response Model that places item difficulty and model ability on a common scale (§3.2). Our first goal is to examine the LocalGap, the ability difference between querying in klocal and in English on culture-specific questions, LocalGapj = θj,CS,klocal − θj,CS,en .
Its sign alone, however, does not reveal why one language wins, because it conflates language proficiency—general ability in klocal versus English— with language-conditioned knowledge access— whether culturally specific knowledge is preferentially encoded in or retrievable through klocal . To isolate proficiency, we measure the same contrast on culture-agnostic content, the GlobalGap,
IRT for LLM MCQ Benchmarking The community has witnessed an increasing number of works applying Item Response Theory (IRT) to LLM Multi-Choice-Question (MCQ) benchmarking, enabling analysis of LLM ability separately from item difficulty. Both tinyBenchmarks (Polo et al., 2024) and metabench (Kipnis et al., 2025) use IRT on LLM responses across MCQs to distill benchmarks into much smaller sets, while variants such as PSN-IRT (Zhou et al., 2026) and β 3 -IRT (Chen et al., 2019) have previously been proposed to extend the capacity of vanilla IRT for more detailed analysis of item difficulty and discrimination. Some recent studies further examine the fitting quality of IRT models with Computerized Adaptive Testing (CAT) (Zhuang et al., 2023; Li et al., 2025). However, very few of them have carefully examined the translation and scale invariance problem of logistic IRT models, which is crucial in our particular case of estimating independent model ability on different subsets of the benchmark.
3
Methodology
3.1
Problem Formulation
(1)
GlobalGapj = θj,CA,klocal − θj,CA,en ,
(2)
on which no local-knowledge advantage is expected. Our second goal is to separate the two effects. Subtracting the GlobalGap from the LocalGap cancels proficiency and isolates knowledge access; we call this difference-in-differences contrast the KnowledgeGap, KnowledgeGapj = LocalGapj −GlobalGapj (3) A positive KnowledgeGap indicates a locallanguage advantage on culture-specific content beyond what proficiency explains. 3.2
1PL-IRT Model
In order to estimate model ability and task difficulty separately, we use a One-Parameter Logistic Item Response Theory (1PL-IRT) model, equivalent to the Rasch model (Bond et al., 2015). making abilities comparable across benchmarks of varying difficulty. It models the probability that examinee j answers item i correctly through the examinee’s ability θj and the item difficulty bi . In our setting each logical item is prompted in multiple languages, and we want difficulty to reflect the problem rather than the prompt language.
Let j be a language model. We study two subsets of problems s ∈ {CA, CS}, where culture-specific questions QCS target a region with local language klocal , alongside culture-agnostic questions QCA that require no local knowledge. Each subset is posed both in klocal and in English k = en, so that the effect of query language is measured within a fixed question pool. We define all gaps in terms of model ability θjsk rather than raw accuracy, because comparing accuracy across question sets confounds the effect
Assumption 1 (Translation Equivalence). An underlying MCQ item has the same intrinsic difficulty across the languages it is translated into. 3
Definition 2 (Multi-Facet Problem). A single logical MCQ item i ∈ I is prompted through a facet k ∈ K, i.e. a translated language. By Assumption 1, the difficulty bi is one scalar shared across all facets k.
English counterparts (CSEN ). We translate each CS item into English with Claude Sonnet 4. We hired native speakers to assess every item, rating translation quality and labeling whether the question is culturally specific; we retain only highquality translations with genuine cultural content. CS and CSEN thus contain the same logical questions in two query languages, isolating the effect of query language on cultural knowledge access. Annotation details are in Appendix B.
The residual variation in correctness across languages is then absorbed by an ability specific to each subset s ∈ {CS, CA} and language k. Writing Yijsk ∈ {0, 1} for the response of examinee j to item i in subset s and language k: Definition 3 (Facet-Conditioned 1PL-IRT Model). P (Yijsk = 1 | θjsk , bi ) := σ(θjsk − bi ),
(4)
with σ the logistic function and θjsk the ability of j on subset s in language k.
4.1
Models
Closed-source frontier. The proprietary models GPT-5.4, GPT-5.4-mini (OpenAI, 2026), Gemini3-Flash (Google DeepMind, 2025), and ClaudeSonnet-4.6 (Anthropic, 2026).
(5)
making language contrasts difficulty-free — the property our gap metrics rely on. The full specification appears in Appendix D. 3.3
Experiments
We evaluate roughly 80 models in three groups, testing both base and instruction-tuned variants where available. Appendix A lists all models with its parameter count and country of origin.
Assuming local independence, the model factorizes over items. Since the facet enters through θ rather than b, the within-item log-odds difference across two languages cancels difficulty, Lijsk − Lijsk′ = θjsk − θjsk′ ,
4
Multilingual pretrained. Open-weight models pretrained from scratch on large, multilingual corpora, spanning a range of countries of origin: Llama (Grattafiori et al., 2024) and Gemma (Team et al., 2024, 2025b; Google DeepMind, 2026) (USA), Qwen (Qwen et al., 2025; Yang et al., 2025; Team, 2026) and GLM (Glm et al., 2024; Team et al., 2025a) (China), Mistral (Rastogi et al., 2025; Liu et al., 2026) (France), Jais-2 (Sengupta et al., 2023) (UAE), EuroLLM (Martins et al., 2024) (EU), and Aya-Expanse (Dang et al., 2024) (Canada).
Dataset Construction and Languages
We cover 13 languages (locales) spanning diverse resource levels, scripts, and regions: Chinese, Arabic, Greek, Hindi, Korean, French, Japanese, Bengali, Dutch, Nepali, Persian, Polish, and Telugu. For each locale we build three matched subsets: culture-agnostic items (CA), culture-specific items in the local language (CS), and their English counterparts (CSEN ). Each language is associated with a resource level described in (Joshi et al., 2020). Appendix A reports per-locale statistics.
Regionally-adapted. Models obtained by continued pretraining or fine-tuning of a global base on local-language and cultural data, rather than pretrained from scratch: AceGPT-v2 (Arabic) (Liang et al., 2024), Llama-Krikri (Roussis et al., 2025) and Meltemi (Voukoutis et al., 2024) (Greek), PersianMind (Persian) (Rostami et al., 2024), and Bielik (Polish) (Ociepa et al., 2024).
Culture-agnostic (CA). We draw CA items from Global-MMLU, which provides high quality parallel translated questions across languages and human-assigned culture-agnostic labels; for each locale, we select the CA items in the corresponding language. Culture-specific (CS). We source native locallanguage CS items from regional benchmarks: CMMLU, ArabicMMLU, GreekMMLU, MILU, and KMMLU, with the remaining locales drawn from INCLUDE. From these we manually select subsets that are potentially cultural-related, and then ask the native speakers to select only genuinely culture-specific items, as described in Appendix B.
Model filtering. Our decomposition assumes each model has enough multilingual competence for the gaps to be meaningful. We therefore retain only models whose accuracy exceeds 0.35 on every subset, keeping the estimation within a regime where the model demonstrably engages with each condition. 4
Knowledge Gap
+.78 +.59 +.40 -.18 +.10 +.07 -.43 +.22 +.26 +.81 +.40 +.51 +.44
+1.12 +.91 +.80 +.25 +.63 +.61 +.07 +.79 +.80 +1.05 +.75 +.88 +.87 +.93 +.59 +.92 +.45 +.74 +1.12 +.52 +.64 +1.04+1.42 +.74 +.20 +.72
GPT-5.4
-.25 -.30 -.49 -.30 -.49 -.54 -.44 -.42 -.39 -.07 -.34 -.28 -.19
+.68 +.29 +.43 +.14 +.24 +.58 +.08 +.22 +.65 +1.35 +.40 -.08 +.52
Gemini-3-Flash
-.24 -.29 -.48 -.51 -.58 -.58 -.50 -.50 -.49 -.25 -.32 -.30 -.30
+.57 -.22 +.59 +.44 +.58 +.67 +.06 +.62 +1.21+1.14+2.26 +.16 +.42
+.80 +.07 +1.07 +.95 +1.16+1.25 +.55 +1.12+1.69+1.39+2.58 +.46 +.72
Claude-Sonnet-4.6
-.34 -.30 -.46 -.52 -.46 -.74 -.61 -.55 -.54 -.13 -.40 -.32 -.36
+.73 -.27 +.98 +.40 +.70 +.35 -.12 +.38 +.51 +.69 +.80 +.08 +.34
+1.07 +.03 +1.43 +.92 +1.17+1.09 +.49 +.93 +1.05 +.83 +1.20 +.40 +.70
Llama-3.1-8B
-.51 -.84 -.93 -1.16 -1.44 -1.29 -1.41 -1.07 -.96 -.53 -.89 -.71 -.91
+.02 -.30 -.24 -.72 -.50 -.32 +.04 -.69 -.71 -.19 -.36 -.72 -.60
+.52 +.54 +.69 +.44 +.93 +.97 +1.46 +.38 +.25 +.34 +.53 -.01 +.32
Llama-3.1-70B
-.46 -.60 -.76 -.81 -1.13 -1.05 -1.42 -.76 -.76 -.42 -.66 -.55 -.80
+.22 +.37 +.04 -.41 -.31 -.15 -.65 -.16 -.03 +.59 +.30 +.00 +.13
+.68 +.96 +.80 +.40 +.82 +.90 +.77 +.60 +.73 +1.01 +.96 +.55 +.93
Llama-3.2-3B
-.54 -.87 -1.05 -1.04 -1.37 -1.21 -1.54 -1.03 -1.00 -.55 -.81 -.68 -1.03
-.18 -.59 -.34 -.88 -.70 -.14 -.63 -.83 -.43 +.30 -.36 -.29 -.56
+.36 +.27 +.71 +.16 +.66 +1.08 +.91 +.19 +.56 +.85 +.44 +.39 +.47 +.54 +.84 +.52 +.21 +.45 +.79 +.60 +.57 +.00 +.43 +.54 -.06 +.27
Gemma-3-4B
-.34 -.39 -.60 -.57 -.64 -.70 -.69 -.65 -.62 -.42 -.54 -.43 -.66
+.20 +.45 -.08 -.36 -.19 +.09 -.09 -.08 -.62 +.01 -.00 -.49 -.39
Gemma-3-12B
-.38 -.46 -.67 -.61 -.73 -.72 -.82 -.74 -.70 -.34 -.49 -.31 -.46
+.25 +.35 -.05 -.15 -.02 -.07 -.20 -.14 -.06 +.26 -.07 +.22 -.15
+.63 +.81 +.62 +.46 +.71 +.65 +.62 +.60 +.63 +.59 +.42 +.53 +.32
Gemma-4-31B
-.54 -.42 -.59 -.64 -.68 -.77 -.70 -.70 -.69 -.22 -.49 -.44 -.46
+.50 +.20 +.25 -.32 -.15 -.04 +.12 -.03 -.09 +.50 +1.00 +.21 +.24
+1.05 +.62 +.83 +.32 +.52 +.73 +.82 +.67 +.60 +.72 +1.49 +.65 +.70
Qwen3-4B
-.25 -.57 -.78 -1.14 -1.24 -1.33 -1.44 -.87 -.99 -.34 -.63 -.50 -1.05
+.61 +.23 -.09 -.61 -.26 -.20 -.24 -.26 -.09 -.31 -.81 +.05 -.66
+.86 +.79 +.69 +.53 +.98 +1.13+1.20 +.61 +.90 +.02 -.18 +.55 +.40
Qwen3-14B
-.35 -.57 -.83 -1.18 -1.30 -1.49 -1.60 -.88 -1.11 -.47 -.74 -.58 -.97
+.81 -.50 -.24 -.67 -.06 -.44 -.45 -.19 +.15 +.02 -1.02 +.16 -.73
+1.16 +.07 +.59 +.51 +1.23+1.05+1.16 +.69 +1.26 +.49 -.28 +.74 +.24
Qwen3.5-9B
-.45 -.85 -.87 -1.72 -1.58 -1.91 -2.15 -.94 -.92 -.51 -.83 -.59 -.78
+1.31 -.17 +.04 -.49 -.34 -.13 -.41 -.06 -.12 +.55 +.74 +.23 -.49
+1.77 +.68 +.90 +1.23+1.24+1.78+1.74 +.88 +.80 +1.06+1.57 +.82 +.30
Qwen3.5-35B-A3B
-.44 -.63 -.75 -.81 -.85 -1.06 -.88 -.70 -.77 -.33 -.55 -.43 -.59
+1.09 +.76 +.12 -.31 -.10 -.11 -.16 +.05 -.29 +.43 +.15 +.24 -.09
+1.53+1.40 +.86 +.51 +.75 +.95 +.72 +.75 +.48 +.76 +.70 +.67 +.50
GLM-4-9B
-.42 -.86 -1.18 -1.64 -1.80 -1.72 -1.87 -1.17 -1.20 -.50 -.85 -.72 -1.18
+1.33 +.69 -.23 -.96 -.56 -.39 -.71 -.26 -.18 +.10 +.00 +.22 -.55
+1.74+1.55 +.95 +.68 +1.24+1.33+1.16 +.91 +1.02 +.60 +.85 +.94 +.64
GLM-4.7-Flash
-.29 -.63 -.74 -1.32 -1.45 -1.45 -2.11 -1.37 -1.03 -.32 -.65 -.62 -1.11
+1.28 +.54 +.06 -.97 -.42 -.46 -.71 -.73 +.03 +.55 +.36 -.16 -.81
+1.57+1.17 +.81 +.35 +1.04 +.99 +1.40 +.65 +1.06 +.88 +1.01 +.45 +.30
Ministral-3-3B-Base
-.53 -.69 -.87 -.92 -1.02 -1.13 -1.08 -.88 -.91 -.40 -.82 -.61 -.86
+.44 -.16 -.19 -.38 +.10 -.27 -.58 -.12 -.03 +.34 -.00 -.10 -.28
+.97 +.54 +.68 +.55 +1.12 +.86 +.50 +.76 +.88 +.74 +.82 +.51 +.58
Ministral-8B
-.37 -.55 -.78 -.71 -.95 -.90 -.98 -.79 -.81 -.33 -.59 -.43 -.73
+.61 +.34 -.09 -.24 +.03 -.20 +.24 -.09 +.27 +.66 +.07 -.11 -.45
+.98 +.89 +.69 +.47 +.98 +.70 +1.21 +.70 +1.08 +.99 +.66 +.31 +.27
Mistral-Medium-3.5-128B
-.35 -.56 -.95 -.74 -.83 -.94 -.99 -.73 -.71 -.33 -.53 -.44 -.64
+.92 -.25 -.11 -.10 +.25 +.22 -.36 +.12 +.32 +1.09 +.64 +.15 +.09
+1.26 +.31 +.83 +.63 +1.08+1.15 +.63 +.85 +1.03+1.43+1.18 +.59 +.72
Jais-2-8B-Chat -1.03 -1.35 -1.59 -1.47 -1.85 -1.66 -1.99 -.42 -1.33 -.87 -1.39 -1.24 -1.54
-.51 -.37 -.45 -.51 -.65 -.71 -.81 +.87 -.41 +.30 -1.43 -1.40 -.95
+.52 +.98 +1.14 +.95 +1.20 +.95 +1.18+1.29 +.92 +1.17 -.04 -.16 +.59
Jais-2-70B-Chat -1.00 -1.37 -1.58 -1.62 -2.18 -1.91 -2.35 -.45 -1.51 -.65 -1.31 -1.04 -1.80
-.49 -1.22 -.24 -1.09 -.98 -.30 -.93 +.76 -.06 +.17 -.36 -.30 -1.66
+.52 +.15 +1.34 +.54 +1.20+1.61+1.42+1.21+1.45 +.82 +.95 +.74 +.15
EuroLLM-9B
-.41 -.57 -.63 -.57 -1.43 -1.16 -1.71 -.52 -1.21 -.22 -.56 -.33 -.50
+.14 -.44 -.49 -.20 -1.06 -.40 -.91 -.38 -.61 +.46 +.07 +.30 -.03
+.54 +.12 +.14 +.37 +.37 +.76 +.80 +.13 +.60 +.68 +.63 +.63 +.47
EuroLLM-22B
-.33 -.51 -.64 -.78 -1.59 -1.31 -2.04 -.72 -1.59 -.27 -.51 -.32 -.57
+.02 +.23 -.27 -.36 -.97 -.74 -1.17 -.42 -.77 +.40 +.29 +.36 -.05
+.35 +.74 +.37 +.43 +.62 +.57 +.87 +.30 +.81 +.67 +.80 +.69 +.51
Aya-Expanse-8B
-.39 -.33 -.50 -.65 -1.31 -1.07 -1.47 -.49 -.59 -.20 -.45 -.44 -.44
+.13 -.00 -.13 -.62 -.74 -.14 -.59 +.12 +.09 -.19 -.51 +.45 -.09
+.52 +.33 +.37 +.04 +.57 +.93 +.88 +.61 +.68 +.01 -.06 +.89 +.35
ZH JA KO HI BN NE TE AR FA FR PL NL EL
ZH JA KO HI BN NE TE AR FA FR PL NL EL
ZH JA KO HI BN NE TE AR FA FR PL NL EL
2
1
0
−1
−2
Local advantage →
Local Gap
-.33 -.32 -.41 -.43 -.52 -.55 -.50 -.57 -.54 -.24 -.35 -.38 -.43
← English advantage
Global Gap GPT-5.4-mini
Region China (ZH) France (FR) UAE (AR) EU (FR/PL/NL/EL) Resource level 4 3 2 1
Figure 2: GlobalGap, LocalGap, and KnowledgeGap across models and locales, showing a consistent English proficiency advantage but a near-universal local-language knowledge advantage.
4.2
Prompts and Inference
mini-batch SGD, more details are in Appendix C. The 1PL likelihood is non-identifiable: since σ depends only on θjsk − bi , adding a constant to all abilities and all difficulties leaves every probability unchanged. To guarantee a unique solution we anchor the latent scale by constraining the item difficulties to mean zero, applying the projection biP← bi − b̄ and θjsk ← θjsk − b̄ (with 1 b̄ = |I| i∈I bi ) after each update. This fixes the origin of the scale and gives a unique, interpretable solution. The full derivation appears in Appendix D.2.
Prompt templates. Each cell uses a fixed template rendered entirely in the query language, so no cross-lingual cues leak in. CA questions use a Question: {question+options} Answer: format; CS questions prepend a locale cue, In {country}, Question: {question+options} Answer:, to ground them in local context. Each template is then translated into the query language; CS_EN uses the same CS template in English without translation. We apply no chat template and no reasoning/chain-of-thought prompting, consistent with our log-likelihood scoring. For closed source models, we use free-generation and extract the answer option.
5
Figure 2 reports a representative subset of model results. For readability, we select models that are generally pretrained from scratch and cover diverse model families, regions of origin, and parameter scales. Regionally adapted models are discussed separately in the Section 6. Full results for all tested models are provided in the Appendix E.
Inference. All open-weight models run in bfloat16 without quantization at temperature 0 with a fixed seed. Smaller models run on NVIDIA A6000 GPUs and larger models on A100 80GB. 4.3
Results
IRT Fitting
5.1
At our benchmarking scale, classical estimators such as Newton–Raphson or Expectation– Maximization are less practical. We instead frame the facet-conditioned 1PL model as a continuous optimization problem: treating the ability matrix Θ and difficulty vector B as trainable embeddings, we recover the maximum-likelihood estimates by minimizing the negative log-likelihood, which coincides with the binary cross-entropy loss, using the Adam optimizer (Kingma and Ba, 2017) with
A Universal English Proficiency Advantage
Across the full grid, G LOBAL G AP is negative in almost every cell (Figure 2, left panel): for all models, on all 13 languages, the estimated ability on culture-agnostic questions is higher under English queries than under local-language queries. The cell-level mean is −0.79, and no cell is positive. This pattern holds even for the highest-resource non-English languages in our set, including Chi5
5.4
nese (−0.44) and French (−0.37), which still favor English on the CA subset. We read this as primarily a proficiency baseline: largely independent of cultural content, English yields higher estimated ability for these models, and the effect is consistent enough that we treat it as a background against which the other two gaps should be interpreted. 5.2
The English advantage is not uniform across languages but varies with resource level. Table 1 groups 13 languages into four ordinal resource levels and reports the mean of each gap. Both G LOB AL G AP and L OCAL G AP become less negative as resource level rises, moving from −1.22 to −0.54 and from −0.27 to +0.22 respectively across the 13 per-language means, resource level correlates with both (G LOBAL G AP: Pearson r = +0.84, p < 0.001; L OCAL G AP: r = +0.69, p = 0.009). The K NOWLEDGE G AP shows no comparable trend (r = −0.30), a dissociation we return to in Section 5.3. The proficiency penalty thus shrinks as a language becomes higher-resource, consistent with G LOBAL G AP being driven primarily by how well the model handles the language itself.
The Local Language Tends to Win With High Resources or Matching Origin
Unlike G LOBAL G AP, L OCAL G AP is mixed (Figure 2, center panel). The positive, local-favoring cells concentrate on the highest-resource languages: Chinese has the most positive column mean (+0.48), followed by French (+0.42), the two clearest cases where querying in the local language yields higher estimated ability on culturespecific questions than translating to English. For most other languages L OCAL G AP is negative or near zero, so on the CS ability contrast translating to English is generally the better choice. A second factor is model origin. Among models built in a given region, the home-language L OCAL G AP tends to be markedly higher than for other models on the same language: +0.79 (Chineseorigin models on Chinese), +0.94 (UAE-origin on Arabic), and +0.32 (French-origin on French) (Table 2). The Arabic case is notable because Arabic’s overall column mean is slightly negative, so here the advantage appears to come from origin rather than resource level. The local language thus tends to win when it is resource-rich, matched to the model’s region of origin, or both. 5.3
Performance Gaps Scale with Resource Level
Level Languages 1 2 3 4
TE, NE BN, EL HI, FA, KO, PL, NL AR, JA, ZH, FR
Global
Local Know.
−1.22 −0.95 −0.76 −0.54
−0.27 −0.26 −0.07 +0.22
0.95 0.70 0.68 0.77
Table 1: Mean gaps by resource level, averaged over models within each level group. Both G LOBAL G AP and L OCAL G AP rise (become less negative) with resource level; K NOWLEDGE G AP does not show a clear monotonic trend.
5.5
Closed-Source Models Surface the Local-Language Advantage
The four proprietary frontier models—GPT-5.4mini, GPT-5.4, Gemini-3-Flash, and ClaudeSonnet-4.6 answer most culture-specific questions better in the local language than in English, with positive L OCAL G AP in most cells alongside positive K NOWLEDGE G AP in nearly all of them. Because these models carry the smallest proficiency penalty (G LOBAL G AP among the least negative in the grid), the local-language advantage is no longer masked and surfaces directly in the uncorrected contrast—the same mechanism behind the Chinese result, here driven by model capability rather than resource level. This also indicates the culturespecific items genuinely require local knowledge.
The Knowledge Advantage Is Universal but Driven by Origin, Not Resources
K NOWLEDGE G AP, the knowledge-access component (Figure 2, right panel), is positive in 98% of cells: the knowledge component favors the local language almost everywhere, including cells where the corresponding L OCAL G AP is negative. This dissociation is the point of the decomposition: once proficiency is partialled out, the local language carries a positive knowledge advantage. Unlike the two performance gaps, K NOWL EDGE G AP is not explained by resource level. What does track it is model origin: in-region models carry a larger home-language K NOWLEDGE G AP than other models (+0.70 ZH, +0.60 AR, +0.30 FR; Table 2), suggesting the advantage is broadly present across models but is deepened by targeted regional training rather than corpus scale.
6
Ablations
6.1
Both Gaps Grow with Model Scale
To probe how the decomposition behaves with scale, we plot L OCAL G AP and K NOWLEDGE G AP 6
L OCAL G AP Region → Lang China → ZH (n=6) UAE → AR (n=2) France → FR (n=3) Europe → FR/PL/NL/EL (n=2)
K NOWLEDGE G AP
In
Out
∆
In
Out
∆
+1.07 +0.82 +0.70 +0.23
+0.28 −0.12 +0.38 +0.05
+0.79 +0.94 +0.32 +0.18
+1.44 +1.25 +1.05 +0.64
+0.74 +0.65 +0.75 +0.64
+0.70 +0.60 +0.30 −0.01
Table 2: Home-language gaps for in-region models versus all other models, for L OCAL G AP and K NOWLEDGE G AP. In-region models show a larger L OCAL G AP on their home language in every case. The K NOWLEDGE G AP boost holds for the single-language regions (China, UAE, France).
6.2
Local Gap
GPT-5.4 (+0.42)
Mean gap (logits)
0.4 0.2
We ask whether the masked advantage persists when a model is explicitly adapted to a single target language. If the L OCAL G AP penalty is mainly a proficiency deficit, language-specific supervised fine-tuning (SFT) should reduce or reverse it, whereas the K NOWLEDGE G AP—which reflects where cultural knowledge is accessible rather than surface proficiency—need not move. We evaluate five publicly available models adapted toward a non-English language, each reported on its adapted language and compared against a per-language baseline: the average of the general pretrained models on that same language. Table 3 shows two consistent effects and one that divides by language. L OCAL G AP turns positive for all five models, above a baseline that is itself negative or near zero, and G LOBAL G AP rises toward zero for every model but stays negative; adaptation thus narrows the proficiency gap to English without closing it, consistent with proficiency being what SFT most readily moves. K NOWLEDGE G AP, by contrast, does not move uniformly: the European models gain over their baseline while the Persian and Arabic models fall slightly below it. This split suggests that SFT adds target-language cultural knowledge for Greek and Polish, but for Persian and Arabic mainly improves linguistic alignment, with little knowledge advantage beyond the samelanguage general models.
0.0 −0.2 −0.4
Knowledge Gap 1.0 Mean gap (logits)
Language Adaptation Helps LocalGap but Not Necessarily KnowledgeGap
0.8
GPT-5.4 (+0.77)
0.6 0.4 0.2 0.0 0.6
1
2
Gemma 3
Gemma 4
4 Total params (B) Qwen 3
Qwen 3.5
9
14
31
MoE (hollow)
Figure 3: L OCAL G AP (top) and K NOWLEDGE G AP (bottom) versus model size for four model families. The red dashed line marks the frontier reference GPT-5.4.
against total parameter count for four model families (Figure 3). Both gaps tend to grow with size. The trend is clearest for L OCAL G AP, which rises within every family—most steeply for the newer Gemma 4 and Qwen 3.5, gently for Gemma 3, and roughly flat for Qwen 3—consistent with scale gradually reducing the proficiency penalty that masks the local-language advantage. K NOWLEDGE G AP also increases overall but not monotonically: some families peak at intermediate sizes and then flatten or decline, most visibly Qwen 3.5 near 9B. The MoE checkpoints move inconsistently relative to dense models of similar size, so we draw no general conclusion about MoE. The two metrics thus scale differently— L OCAL G AP rising steadily, K NOWLEDGE G AP rising then leveling—but both persist across the full size range, mostly below the GPT-5.4 reference, rather than being tied to any particular scale.
6.3
Latent Ability θ Recovers Raw Accuracy
Because θ is estimated from observed responses, we verify it faithfully reflects raw accuracy, so that ability contrasts can be read as difficulty-adjusted accuracy differences. Across all subsets, the mean Pearson correlation between accuracy and θ is 0.996, ranging narrowly from 0.982 (Greek CS) to 0.999 (Arabic CS), and the Spearman correlation is 0.9999, indicating that θ recovers the exam7
G LOBAL
L OCAL
K NOW.
Model
Lg
Adapt.
∆
Adapt.
∆
Adapt.
∆
Krikri-8B Meltemi-7B Bielik-Minitron-7B PersianMind AceGPT-v2-8B
EL EL PL FA AR
−0.44 −0.47 −0.41 −0.55 −0.23
+0.33 +0.30 +0.25 +0.34 +0.52
+0.39 +0.48 +0.75 +0.14 +0.31
+0.65 +0.74 +0.64 +0.18 +0.36
+0.83 +0.94 +1.16 +0.69 +0.54
+0.32 +0.44 +0.40 −0.16 −0.16
−0.42
+0.35
+0.41
+0.52
+0.83
+0.17
Mean (Adapt.)
Table 3: Language-adapted models evaluated on their target languages Adapt. denotes the adapted model’s gap value. ∆ denotes the difference between the adapted model and the per-language baseline, defined as the mean gap of general-purpose models evaluated on the same language.
Variant
BCE↓
SD
Brier↓ AUC↑ Acc↑
binary+mle binary+nuts crm+mle crm+nuts
0.513 0.515 0.537 0.553
9e-4 1e-3 5e-4 8e-4
0.169 0.170 0.178 0.182
0.830 0.829 0.816 0.821
high-dimensional parameter space. The CRM gap is consistent with soft option-token probabilities being noisy and poorly calibrated, so thresholding to binary correctness yields a more reliable ability estimate. We therefore adopt binary+mle.
0.752 0.748 0.746 0.746
Table 4: Cross-validation performance across IRT variants. BCE and Brier measure predictive likelihood and calibration; AUC and accuracy measure discrimination on held-out responses. A standard deviation (SD) is reported along with BCE for different folds.
7
We introduced a controlled framework that combines a difference-in-differences design with itemresponse modeling to study language-conditioned access to localized knowledge in LLMs. By crossing question type, culture-agnostic versus culturespecific, with query language, English versus local, we separate general language proficiency from localized knowledge access. Specifically, we estimate G LOBAL G AP as the proficiency gap, L O CAL G AP as the combined effect on culture-specific questions, and K NOWLEDGE G AP as their difference on a shared, difficulty-adjusted 1PL IRT scale. Across 13 locales and roughly 80 models, L O CAL G AP is shaped by language resource level, model origin, and language adaptation, becoming positive mainly when these factors favor the local language. In contrast, K NOWLEDGE G AP is positive in nearly all settings, suggesting that local languages generally provide better access to local cultural knowledge, even when this advantage is hidden by weaker local-language proficiency. These findings offer useful insight for current localized-knowledge benchmarks and languageadaptation strategies. When a model performs worse in a local language, the cause is often not missing cultural knowledge but weaker proficiency in that language, which masks a knowledge advantage. Improving proficiency—through language adaptation or stronger multilingual training—is therefore a possible way to unlock this hidden capability, since the knowledge access largely exists and is held back mainly by the proficiency penalty.
inee rank order almost exactly. This is expected: within a subset, θ is by construction a monotonic re-expression of accuracy on a continuous log-odds scale. As a result, the within-component G LOBAL G AP and L OCAL G AP closely track the corresponding local-minus-English accuracy differences; the IRT scale adds value by putting them on the same difficulty-adjusted footing as the cross-component K NOWLEDGE G AP, a comparison raw accuracy cannot support. 6.4
Conclusion
Selection of the IRT Model
We ablate two orthogonal choices in a 2 × 2 grid: the item-response representation (binary vs. a Continuous Response Model, CRM, which scores each item by the normalized probability mass over option tokens rather than a hard label) and the inference algorithm (maximum likelihood, MLE, vs. the No-U-Turn Sampler, NUTS); both are defined in Appendix D.3. All four combinations are evaluated under 5-fold out-of-distribution cross-validation on held-out responses, scored by predictive loglikelihood (BCE), calibration (Brier), and discrimination (AUC, accuracy) (Table 4). The binary representation outperforms CRM under both inference methods, and MLE matches or slightly exceeds NUTS at far lower cost—likely because exact posterior sampling is difficult in this 8
Limitations
Google DeepMind. 2025. Gemini 3 flash model card. https://storage.googleapis. com/deepmind-media/Model-Cards/ Model Gemini-3-Flash-Model-Card.pdf. card PDF, accessed 2026-05-25.
We note several limitations of this study. First, the size of the culture-specific subsets varies across locales, in part because some languages do not have large-scale cultural benchmark data readily available online. Second, our evaluation of regionally adapted models is limited in scope. Strong open-weight models that have been explicitly adapted to a nonEnglish language remain relatively scarce, and high-quality, widely-adopted models of this kind are rarer still. So we were able to include only a small number of such models, covering a handful of languages. Third, all of our results are based on multiplechoice questions scored by log-likelihood. We do not evaluate open-ended generation, where knowledge access may behave differently and where failures can take more diverse forms than a fixed set of options allows.
Google DeepMind. 2026. Gemma 4 technical overview and model card. https://ai.google.dev/gemma/ docs/core/model_card_4. Accessed: 2026-05-25. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Shourya Jain and Paras Chopra. 2026. Language models entangle language and culture. arXiv preprint arXiv:2601.15337. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.
References Ishika Agarwal, Nimet Beyza Bozdag, and Dilek Hakkani-Tür. 2025. Language specific knowledge: Do models know better in x than in english? arXiv preprint arXiv:2505.14990.
Alex Kipnis, Konstantinos Voudouris, Luca Schulze Buschoff, and Eric Schulz. 2025. metabench - a sparse benchmark of reasoning and knowledge in large language models. In International Conference on Learning Representations, volume 2025, pages 31734–31770.
Anthropic. 2026. Claude sonnet 4.6. https://www. anthropic.com/. Large language model developed by Anthropic. Trevor G. Bond, Zi Yan, and Moritz Heene. 2015. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 3 edition. Routledge, New York.
Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics.
Yu Chen, Telmo Silva Filho, Ricardo B Prudencio, Tom Diethe, and Peter Flach. 2019. βˆ3 -irt: A new item response model and its applications. In The 22nd international conference on artificial intelligence and statistics, pages 1013–1021. PMLR.
John P Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648– 657.
John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis FletBerliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, Bangkok, Thailand. Association for Computational Linguistics.
Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V Chawla. 2025.
9
Adaptive testing for llm evaluation: A psychometric alternative to static benchmarks. arXiv preprint arXiv:2511.04689.
examples. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Abdulmohsen Alharthi, Juncai He, Lian Zhang, Haizhou Li, and 1 others. 2024. Alignment at pre-training! towards native alignment for arabic llms. Advances in Neural Information Processing Systems, 37:13872–13896.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.
Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, and 1 others. 2026. Ministral 3. arXiv preprint arXiv:2601.08584.
Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, and 1 others. 2025. Magistral. arXiv preprint arXiv:2506.10910.
Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2025. Is translation all you need? a study on solving multilingual tasks with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 9594–9614, Albuquerque, New Mexico. Association for Computational Linguistics.
Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Lee Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4486–4503.
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language models for europe. Preprint, arXiv:2409.16235.
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed Haggag, Alfonso Amayuelas, and 1 others. 2025. Include: Evaluating multilingual language understanding with regional knowledge. In International Conference on Learning Representations, volume 2025, pages 83291–83322.
Fernando Martínez-Plumed, Ricardo B.C. Prudêncio, Adolfo Martínez-Usó, and José Hernández-Orallo. 2019. Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18–42.
Pedram Rostami, Ali Salemi, and Mohammad Javad Dousti. 2024. Persianmind: A cross-lingual persianenglish large language model. arXiv preprint arXiv:2401.06466.
Amr Mohamed, Yang Zhang, Michalis Vazirgiannis, and Guokan Shang. 2025. Lost in the mix: Evaluating llm understanding of code-switched text. arXiv preprint arXiv:2506.14012.
Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavasileiou, Athanasios Katsamanis, Stelios Piperidis, and Vassilis Katsouros. 2025. Krikri: Advancing open large language models for greek. arXiv preprint arXiv:2505.13772.
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki A Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew A Ayele, and 1 others. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems, 37:78104–78146.
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149.
Krzysztof Ociepa, Krzysztof WrĂłbel, Adrian GwoĹşdziej, Remigiusz Kinas, and 1 others. 2024. Bielik 7b v0. 1: A polish language model– development, insights, and evaluation. arXiv preprint arXiv:2410.18565.
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, and 1 others. 2025. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799.
OpenAI. 2026. Openai gpt-5 system card. Preprint, arXiv:2601.03267. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer
10
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4076–4104, Albuquerque, New Mexico. Association for Computational Linguistics.
of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Jiahao Ying, Wei Tang, Yiran Zhao, Yixin Cao, Yu Rong, and Wenxuan Zhang. 2025. Disentangling language and culture for evaluating multilingual large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22230– 22251, Vienna, Austria. Association for Computational Linguistics.
5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025a. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. Preprint, arXiv:2508.06471.
Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, and Michalis Vazirgiannis. 2026. Greekmmlu: A native-sourced multitask benchmark for evaluating language models in greek. arXiv preprint arXiv:2602.05150.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025b. Gemma 3 technical report. Preprint, arXiv:2503.19786.
Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, Conghui Zhu, Hailong Cao, and Tiejun Zhao. 2026. Lost in benchmarks? rethinking large language model benchmarking with item response theory. Proceedings of the AAAI Conference on Artificial Intelligence, 40:35085–35093.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.
Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, and 1 others. 2023. Efficiently measuring the cognitive ability of llms: An adaptive testing perspective.
Qwen Team. 2026. Qwen3.5-omni technical report. Preprint, arXiv:2604.15804. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2025. MILU: A multi-task Indic language understanding benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10076–10132, Albuquerque, New Mexico. Association for Computational Linguistics. Leon Voukoutis, Dimitris Roussis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavasileiou, Athanasios Katsamanis, Stelios Piperidis, and Vassilis Katsouros. 2024. Meltemi: The first open large language model for greek. arXiv preprint arXiv:2407.20743. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings
11
A
Model and Dataset Details
lated into English, we evaluate each item individually for correctness, rigor, and topic relevance. For each language that is not the native language among the authors, we hired native speakers to do the annotation, they are paid 15 dollars per hour, and each subset requires around 2 hours of work. Each annotator is given an annotation instruction as described in Figure 4. Table 7 shows that raw translation quality evaluated by human annotators, most local languages results in negligible error rates (translation scored 1), indicating that translation equivalence is well preserved. Items flagged with a score of 1 or 2 are filtered out, ensuring that the remaining benchmark is fully legitimate.
We gathered 79 different models with various types, sizes and origins as the population of the benchmarking effort, as further details are shown in Table 5. Both closed-source and open-source models are evaluated, with multiple regionally adapted LLMs considered. In the meantime, both culturespecific(CS) and culture-agnostic (CA) benchmarks are constructed for 14 languages from different language families with spanning richness in resources, as illustrated in Table 6. Multiple languages with extremely low resources are included in particular to study the full spectrum of the knowledge gap of different pretraining scenarios.
B
Human evaluation
C
Annotation Instructions This project evaluates the cultural ability of LLMs using questions in both the native language and English translation. You will be given a CSV file containing multiplechoice questions with four options: A, B, C, and D. For each row, please complete two tasks. Task 1: Translation Quality Score Evaluate the English translation of the question and all answer options. Enter a score in the SCORE column. 1 – Poor: The translation loses important meaning, mistranslates key or culturally specific terms, or is misleading. 2 – Acceptable: The translation has minor ambiguity or awkwardness, but the overall intent is understandable. 3 – Good: The translation faithfully preserves the original meaning, and the question can be answered correctly from the English version alone. Task 2: Local Culture Annotation Indicate whether the question requires local cultural knowledge to answer correctly. Yes: The question depends on knowledge of the target locale, such as local history, traditions, festivals, geography, laws, institutions, education, literature, religion, food, social norms, or popular culture. No: The question can be answered using general knowledge and does not rely on local cultural context. Unsure: The cultural relevance is unclear, weak, or ambiguous. Please judge each row based on the question and answer options, not on whether you personally know the answer.
Hyperparameters and Setup
The 1PL-IRT model with MLE SGD is trained on a NVIDIA RTX A6000 48GB GPU. The training with learning rate 0.05 spans for about 600 to 1000 steps until convergence of the negative loglikelihood loss, taking no more than 1 minute. For ablations, the NUTS fitting process takes 8 sampling chains in parallel, each takes 2000 warmup steps followed by another 2000 sampling steps. For cross validation, all observed responses are partitioned completely randomly into five folds to provide a generalizable held-out cross validation setup.
D
Full Specification of the 1PL-IRT Model and Fitting
This appendix gives the complete specification of the facet-conditioned 1PL-IRT model (§3.2) and the fitting procedure (§4.3), including the assumptions, ability parameterization, identifiability argument, and anchoring projection summarized in the main text. D.1
Model Details
Figure 4: The annotation instruction for human nativelanguage-speaking evaluators on translation quality control.
The mathematical validity and interpretability of the facet-conditioned 1PL model rest upon three foundational assumptions concerning the joint distribution of responses and the latent space.
To confirm the Assumption 1 for our IRT model, we asked native-speaking translators to verify the quality of the translations for the CS (culturespecific) items. Since every MCQ item originally written in a native language is automatically trans-
Assumption 4 (Unidimensionality). The latent trait space is one-dimensional, meaning that a single scalar value θjsk is strictly sufficient to explain the statistical dependence between item responses for subject j on a subset s with a certain prompt language k. 12
Model Family
Origin
Parameter Sizes
#
C LOSED S OURCE GPT-5.4 Gemini 3 Flash Claude Sonnet 4.6
OpenAI, USA Google, USA Anthropic, USA
— — —
2 1 1
M ULTILINGUAL P RETRAINED Llama 3.1 Meta, USA Llama 3.2 Meta, USA Gemma 2 Google, USA Gemma 3 Google, USA Gemma 4 Google, USA Qwen 2.5 Alibaba, China Qwen 3 Alibaba, China Qwen 3.5 Alibaba, China GLM 4 / 4.7 Zhipu AI, China Mistral Mistral AI, France Ministral 3 Mistral AI, France Jais 2 Inception, UAE EuroLLM Europe Aya Expanse Cohere, Canada
8B, 70B 1B, 3B 9B, 27B 1B, 4B, 12B, 27B E2B, E4B, 26B-A4B, 31B 7B, 14B, 32B 0.6B, 1.7B, 4B, 14B, 30B-A3B 0.8B, 2B, 4B, 9B, 27B, 35B-A3B 9B 119B, 128B 3B, 8B, 14B 8B, 70B 9B, 22B 8B
4 4 4 8 8 6 7 11 3 2 6 2 4 1
R EGIONALLY A DAPTED AceGPT-v2 China (Arabic focus) Llama-Krikri Greece Meltemi Greece Bielik-Minitron Poland PersianMind Iran
8B 8B 7B 7B —
1 1 1 1 1
Total
79
Table 5: Overview of models evaluated in our benchmark, grouped by family. Both base and instruction-tuned variants are included where available; # denotes the total number of model variants per family.
Language
Code
Family
Resource
#CA
#CS
Source
English Chinese Arabic French Japanese
en zh ar fr ja
Indo-European Sino-Tibetan Afro-Asiatic Indo-European Japonic
5 4 4 4 4
2,058 2,058 2,058 2,058 2,058
6,093 1,000 999 139 73
— CMMLU ArabicMMLU INCLUDE INCLUDE
Dutch Hindi Korean Persian Polish
nl hi ko fa pl
Indo-European Indo-European Koreanic Indo-European Indo-European
3 3 3 3 3
2,058 2,058 2,058 2,058 2,058
136 1,000 1,000 185 75
INCLUDE MILU KMMLU INCLUDE INCLUDE
Bengali Greek
bn el
Indo-European Indo-European
2 2
2,058 2,058
342 785
INCLUDE GreekMMLU
Nepali Telugu
ne te
Indo-European Dravidian
1 1
2,058 2,058
230 129
INCLUDE INCLUDE
Total
14 languages
28,812
12,186
Table 6: Overview of languages and subsets used in our study. The CA (culture-agnostic) subset is sourced from GlobalMMLU. The CS (culture-specific) subset contains locally sourced questions probing culture-specific knowledge; CS items are also available in English translation (CS-en). All subsets are evaluated across 79 LLMs.
13
Language
Error (%)
Chinese Arabic French Japanese Hindi Persian Polish
2.4 1.8 2.0 4.0 1.6 5.0 1.7
Language
the facet-conditioned 1PL model as a continuous optimization problem, treating the ability matrix Θ of all facet-specific examinee abilities and the difficulty vector B of all item difficulties as trainable embeddings and recovering the maximumlikelihood estimates by stochastic gradient descent (SGD). Let Y = {(j, i, s, k, yijsk )} denote the observed benchmarking data, where each entry records examinee j, a multi-facet MCQ item i from subset s prompted in language k, and the dichotomous outcome yijsk ∈ {0, 1}. Writing Pijsk for the logistic probability of Definition 3 and invoking Assumption 5, the joint likelihood is the product of the individual Bernoulli probabilities:
Error (%)
Dutch Korean Greek Telugu Bengali Nepali
0.4 0.9 0.1 0.0 2.3 7.0
Table 7: Error rate (%) in human evaluation of translation quality.
Assumption 5 (Local Independence). Conditional on the latent trait θjsk , the responses to multifacet problems are mutually independent. For a subset s of size n with prompt language k, the joint probability of a response vector Y jsk = (y1jsk , y2jsk , . . . , ynjsk ) is the product of the individual marginal probabilities: P (Y jsk = y jsk |θjsk , bs ) n Y = P (Yijsk = yijsk |θjsk , bis )
L(Θ, B | Y ) Y = (j,i,s,k,yijsk )∈Y
(10) Taking the natural logarithm yields the loglikelihood ℓ, and the empirical loss L is the negative log-likelihood (NLL), which coincides with the binary cross-entropy (BCE) loss of standard machine-learning toolkits:
(6)
i=1
Assumption 6 (Monotonicity). The probability of a correct response P (Yijsk = 1|θjsk , bi ) is a strictly monotonically increasing function of the latent trait θjsk , meaning that for any two examinees with latent traits θ1sk , θ2sk ∈ R: θ1sk < θ2sk ⇔ Pi (θ1sk ) < Pi (θ2sk ), ∀i
y
ijsk Pijsk (1 − Pijsk ) 1−yijsk .
L(Θ, B) = −ℓ(Θ, B) = − log L(Θ, B | Y ). (11) We minimize L over Θ and B jointly using the Adam optimizer (Kingma and Ba, 2017) with minibatch SGD.
(7)
Difficulty-free language contrasts. By treating the language facet as a dimension of the examinee’s latent space rather than a shift in item difficulty, the log-odds Lijsk of a correct response for examinee j on item i from subset s in language k is P (Yijsk = 1) = θjsk −bi , (8) Lijsk = ln 1 − P (Yijsk = 1)
Identifiability and anchoring. The 1PL model is non-identifiable: because the logistic function depends only on the difference θjsk − bi , adding any constant c to all abilities and all difficulties leaves every probability unchanged, σ (θjsk + c) − (bi + c) = σ(θjsk − bi ). (12)
and likewise Lijsk′ = θjsk′ − bi for a second language k ′ . The difficulty term cancels in the withinitem difference,
To guarantee a unique MLE, we anchor the latent scale by constraining the mean item difficulty to zero. After each SGD update we apply the projec1 P tion, with b̄ = |I| i∈I bi ,
Lijsk − Lijsk′ = θjsk − θjsk′ ,
(9)
bi ← bi − b̄,
∀i ∈ I,
θjsk ← θjsk − b̄, ∀j, s, k.
(13)
so the language contrast for the same item is exactly the difference in the examinee’s language-specific abilities, independent of bi .
This re-centering fixes the origin of the latent scale and gives a unique, interpretable solution.
D.2
D.3
Fitting Procedure
IRT Variant Details
Item-response representations. The dichotomous (binary) model scores each response as correct or incorrect (y ∈ {0, 1}) and models Pi (θj ) =
At our benchmarking scale, classical estimators such as Newton–Raphson or Expectation– Maximization are less practical. We instead frame 14
σ(θj − bi ) as in the main text (§6.4). The Continuous Response Model (CRM) instead treats the response as graded: rather than thresholding, it uses the probability mass the model places on the correct option after normalizing over the answeroption tokens, giving a continuous score in [0, 1]. This retains finer information about how confident the model was, at the cost of inheriting any miscalibration in the raw token probabilities. Inference algorithms. MLE treats θ and b as free parameters with no prior and minimizes the negative log-likelihood by gradient descent (Adam), returning a single point estimate (θ̂, b̂). NUTS is an adaptive variant of Hamiltonian Monte Carlo that draws samples from the full Bayesian posterior over (θ, b); it is a standard choice for fitting IRT models and yields credible intervals directly, but its cost grows with the dimensionality of the parameter space and it can suffer from divergences and inefficient exploration in very highdimensional settings.
E
Full results
We present results for all models tested in Figure 5.
15
Global Gap GPT-5.4-mini -.33 -.32 -.41 -.43 -.52 -.55 -.50 -.57 -.54 -.24 -.35 -.38 -.43 GPT-5.4 -.25 -.30 -.49 -.30 -.49 -.54 -.44 -.42 -.39 -.07 -.34 -.28 -.19 Gemini-3-Flash -.24 -.29 -.48 -.51 -.58 -.58 -.50 -.50 -.49 -.25 -.32 -.30 -.30 Claude-Sonnet-4.6 -.34 -.30 -.46 -.52 -.46 -.74 -.61 -.55 -.54 -.13 -.40 -.32 -.36 Llama-3.1-8B -.51 -.84 -.93 -1.16 -1.44 -1.29 -1.41 -1.07 -.96 -.53 -.89 -.71 -.91 Llama-3.1-8B-IT -.51 -.74 -.90 -1.03 -1.23 -1.20 -1.43 -.96 -.92 -.43 -.71 -.56 -.92 Llama-3.1-70B -.46 -.60 -.76 -.81 -1.13 -1.05 -1.42 -.76 -.76 -.42 -.66 -.55 -.80 Llama-3.1-70B-IT -.39 -.41 -.53 -.45 -.86 -.84 -1.11 -.65 -.65 -.21 -.45 -.26 -.51 Llama-3.2-1B -.38 -.86 -.96 -.77 -.73 -.85 -1.07 -.84 -.84 -.69 -.73 -.85 -.87 Llama-3.2-1B-IT -.24 -.45 -.41 -.77 -.67 -.91 -.76 -.54 -.57 -.24 -.45 -.27 -.74 Llama-3.2-3B -.54 -.87 -1.05 -1.04 -1.37 -1.21 -1.54 -1.03 -1.00 -.55 -.81 -.68 -1.03 Llama-3.2-3B-IT -.34 -.66 -.78 -.78 -1.00 -.94 -1.04 -.82 -.77 -.31 -.52 -.38 -.87 Gemma-2-9B -.57 -.73 -.84 -.82 -1.12 -1.04 -1.13 -.98 -.84 -.40 -.72 -.49 -.96 Gemma-2-9B-IT -.49 -.57 -.70 -.78 -.95 -.99 -1.11 -.77 -.81 -.33 -.59 -.47 -.77 Gemma-2-27B -.50 -.62 -.78 -.72 -.83 -.92 -1.03 -.73 -.76 -.36 -.64 -.41 -.70 Gemma-2-27B-IT -.43 -.59 -.71 -.69 -.87 -.94 -.97 -.71 -.71 -.29 -.58 -.42 -.62 Gemma-3-1B +.08 -.14 -.21 -.00 -.02 -.07 -.21 -.26 -.11 +.07 -.02 -.07 +.01 Gemma-3-1B-IT -.17 -.22 -.15 -.30 -.40 -.55 -.44 -.40 -.38 -.10 -.29 -.25 -.39 Gemma-3-4B -.34 -.39 -.60 -.57 -.64 -.70 -.69 -.65 -.62 -.42 -.54 -.43 -.66 Gemma-3-4B-IT -.29 -.37 -.40 -.45 -.59 -.76 -.55 -.52 -.64 -.26 -.44 -.35 -.54 Gemma-3-12B -.38 -.46 -.67 -.61 -.73 -.72 -.82 -.74 -.70 -.34 -.49 -.31 -.46 Gemma-3-12B-IT -.38 -.47 -.74 -.59 -.66 -.80 -.69 -.68 -.66 -.32 -.50 -.43 -.58 Gemma-3-27B -.29 -.48 -.61 -.53 -.73 -.69 -.70 -.57 -.60 -.25 -.57 -.38 -.52 Gemma-3-27B-IT -.31 -.47 -.63 -.49 -.68 -.73 -.59 -.55 -.58 -.24 -.48 -.33 -.50 Gemma-4-E2B -.45 -.53 -.59 -.68 -.83 -.68 -.76 -.71 -.70 -.42 -.66 -.55 -.68 Gemma-4-E2B-IT -.14 -.17 -.19 -.15 -.18 -.11 -.27 -.16 -.10 -.00 -.07 -.03 -.07 Gemma-4-E4B -.47 -.46 -.56 -.70 -.73 -.73 -.81 -.77 -.77 -.26 -.57 -.41 -.73 Gemma-4-E4B-IT -.24 -.24 -.35 -.19 -.21 -.27 -.20 -.24 -.20 -.06 -.13 -.11 -.09 Gemma-4-26B-A4B -.49 -.54 -.66 -.64 -.85 -.88 -.92 -.70 -.65 -.33 -.67 -.44 -.64 Gemma-4-26B-A4B-IT -.20 -.08 -.32 -.36 -.38 -.37 -.28 -.16 -.46 +.04 -.17 -.11 -.26 Gemma-4-31B -.54 -.42 -.59 -.64 -.68 -.77 -.70 -.70 -.69 -.22 -.49 -.44 -.46 Gemma-4-31B-IT -.13 -.35 -.19 -.14 -.18 -.19 -.28 -.20 -.30 +.02 -.21 -.08 -.15 Qwen2.5-7B -.36 -.62 -.85 -1.45 -1.44 -1.83 -2.24 -1.02 -1.33 -.51 -.96 -.79 -1.53 Qwen2.5-7B-IT -.13 -.46 -.73 -1.30 -1.41 -1.81 -2.12 -.92 -1.24 -.32 -1.20 -.57 -1.80 Qwen2.5-14B -.27 -.64 -.79 -1.38 -1.47 -1.62 -1.99 -.84 -1.29 -.51 -.84 -.61 -1.47 Qwen2.5-14B-IT -.21 -.69 -.88 -1.38 -1.37 -1.64 -2.02 -.86 -1.39 -.45 -.93 -.65 -1.66 Qwen2.5-32B -.19 -.43 -.68 -1.12 -1.26 -1.46 -2.06 -.94 -1.20 -.36 -.88 -.44 -1.13 Qwen2.5-32B-IT -.31 -.63 -.78 -1.29 -1.32 -1.58 -2.15 -1.02 -1.17 -.38 -.86 -.49 -1.29 Qwen3-0.6B -.20 -.90 -.55 -.81 -.81 -.85 -1.05 -.58 -.79 -.23 -.59 -.50 -.91 Qwen3-1.7B -.31 -.73 -.67 -1.04 -1.04 -1.14 -1.33 -.77 -.90 -.46 -.73 -.59 -1.01 Qwen3-4B -.25 -.57 -.78 -1.14 -1.24 -1.33 -1.44 -.87 -.99 -.34 -.63 -.50 -1.05 Qwen3-4B-IT -.18 -.52 -.70 -1.13 -1.24 -1.36 -1.46 -.82 -1.06 -.20 -.68 -.49 -1.03 Qwen3-14B -.35 -.57 -.83 -1.18 -1.30 -1.49 -1.60 -.88 -1.11 -.47 -.74 -.58 -.97 Qwen3-30B-A3B -.27 -.51 -.79 -.93 -1.07 -1.18 -1.36 -.86 -.92 -.44 -.68 -.45 -.85 Qwen3-30B-A3B-IT -.36 -.60 -.73 -1.00 -1.16 -1.28 -1.31 -.77 -.87 -.32 -.61 -.49 -.91 Qwen3.5-0.8B -.24 -1.42 -.76 -1.70 -1.76 -1.68 -1.73 -.76 -.79 -.43 -.74 -.68 -.78 Qwen3.5-0.8B-IT -.17 -.99 -.87 -1.53 -1.54 -1.52 -1.51 -.98 -1.15 -.47 -.78 -1.14 -1.09 Qwen3.5-2B -.22 -1.10 -.92 -1.64 -1.48 -1.55 -1.84 -.79 -.91 -.38 -1.00 -.64 -1.05 Qwen3.5-2B-IT -.25 -.97 -1.65 -1.56 -1.44 -1.52 -1.67 -.77 -.98 -.52 -.91 -.63 -1.38 Qwen3.5-4B -.39 -1.01 -.79 -1.71 -1.60 -1.82 -2.14 -.88 -1.05 -.49 -.87 -.71 -.88 Qwen3.5-4B-IT -.36 -.83 -.74 -1.76 -1.73 -2.03 -2.39 -.95 -1.20 -.46 -.85 -.70 -.83 Qwen3.5-9B -.45 -.85 -.87 -1.72 -1.58 -1.91 -2.15 -.94 -.92 -.51 -.83 -.59 -.78 Qwen3.5-9B-IT -.39 -.77 -.81 -1.72 -1.65 -2.02 -2.29 -.84 -.87 -.54 -.80 -.60 -.80 Qwen3.5-27B -.45 -.54 -.63 -.71 -.69 -.91 -.81 -.59 -.67 -.39 -.55 -.36 -.56 Qwen3.5-35B-A3B -.44 -.63 -.75 -.81 -.85 -1.06 -.88 -.70 -.77 -.33 -.55 -.43 -.59 Qwen3.5-35B-A3B-IT -.48 -.62 -.65 -.84 -.85 -1.03 -.90 -.73 -.84 -.37 -.66 -.42 -.59 GLM-4-9B -.42 -.86 -1.18 -1.64 -1.80 -1.72 -1.87 -1.17 -1.20 -.50 -.85 -.72 -1.18 GLM-4-9B-Chat -.41 -.85 -1.17 -1.59 -1.81 -1.73 -2.17 -1.13 -1.12 -.58 -.97 -.68 -1.25 GLM-4.7-Flash -.29 -.63 -.74 -1.32 -1.45 -1.45 -2.11 -1.37 -1.03 -.32 -.65 -.62 -1.11 Mistral-Small-4-119B -.35 -.59 -.87 -.69 -.81 -.88 -.84 -.84 -.86 -.47 -.63 -.42 -.71 Mistral-Medium-3.5-128B -.35 -.56 -.95 -.74 -.83 -.94 -.99 -.73 -.71 -.33 -.53 -.44 -.64 Ministral-3-3B-Base -.53 -.69 -.87 -.92 -1.02 -1.13 -1.08 -.88 -.91 -.40 -.82 -.61 -.86 Ministral-3-3B-IT -.47 -.68 -.84 -.88 -1.05 -1.07 -1.19 -.87 -.85 -.43 -.79 -.59 -.89 Ministral-8B -.37 -.55 -.78 -.71 -.95 -.90 -.98 -.79 -.81 -.33 -.59 -.43 -.73 Ministral-8B-IT -.35 -.58 -.79 -.68 -.83 -.93 -.97 -.72 -.81 -.35 -.61 -.46 -.73 Ministral-14B -.51 -.75 -.99 -.85 -1.06 -1.18 -1.09 -.95 -.94 -.46 -.78 -.52 -.96 Ministral-14B-IT -.47 -.66 -.87 -.86 -1.02 -1.09 -1.13 -.87 -.92 -.39 -.81 -.56 -.88 Jais-2-8B-Chat -1.03 -1.35 -1.59 -1.47 -1.85 -1.66 -1.99 -.42 -1.33 -.87 -1.39 -1.24 -1.54 Jais-2-70B-Chat -1.00 -1.37 -1.58 -1.62 -2.18 -1.91 -2.35 -.45 -1.51 -.65 -1.31 -1.04 -1.80 EuroLLM-9B -.41 -.57 -.63 -.57 -1.43 -1.16 -1.71 -.52 -1.21 -.22 -.56 -.33 -.50 EuroLLM-9B-IT -.27 -.33 -.47 -.47 -1.33 -.96 -1.77 -.44 -1.17 -.20 -.36 -.24 -.37 EuroLLM-22B -.33 -.51 -.64 -.78 -1.59 -1.31 -2.04 -.72 -1.59 -.27 -.51 -.32 -.57 EuroLLM-22B-IT -.36 -.55 -.50 -.51 -1.64 -.98 -2.09 -.66 -1.41 -.28 -.37 -.27 -.34 Krikri-8B -.55 -.97 -.89 -1.07 -1.36 -1.27 -1.51 -1.03 -1.00 -.45 -.85 -.69 -.44 Meltemi-7B -.73 -1.06 -1.03 -1.55 -1.58 -1.57 -1.67 -1.36 -1.37 -.42 -1.02 -.67 -.47 Bielik-Minitron-7B -1.15 -1.36 -1.50 -1.85 -1.74 -1.81 -1.92 -1.62 -1.64 -.48 -.41 -.58 -.99 PersianMind -.80 -.83 -1.01 -1.04 -1.08 -1.03 -1.04 -.90 -.55 -.39 -.85 -.52 -1.05 AceGPT-v2-8B -.12 -.30 -.35 -.48 -.76 -.68 -.92 -.23 -.39 +.12 -.35 -.13 -.29 Aya-Expanse-8B -.39 -.33 -.50 -.65 -1.31 -1.07 -1.47 -.49 -.59 -.20 -.45 -.44 -.44
ZH JA KO HI BN NE TE AR FA FR PL NL EL
Local Gap
Knowledge Gap
+.78 +.59 +.40 -.18 +.10 +.07 -.43 +.22 +.26 +.81 +.40 +.51 +.44
+1.12 +.91 +.80 +.25 +.63 +.61 +.07 +.79 +.80 +1.05 +.75 +.88 +.87
+.68 +.29 +.43 +.14 +.24 +.58 +.08 +.22 +.65 +1.35 +.40 -.08 +.52
+.93 +.59 +.92 +.45 +.74 +1.12 +.52 +.64 +1.04+1.42 +.74 +.20 +.72
+.57 -.22 +.59 +.44 +.58 +.67 +.06 +.62 +1.21+1.14+2.26 +.16 +.42
+.80 +.07 +1.07 +.95 +1.16+1.25 +.55 +1.12+1.69+1.39+2.58 +.46 +.72
+.73 -.27 +.98 +.40 +.70 +.35 -.12 +.38 +.51 +.69 +.80 +.08 +.34
+1.07 +.03 +1.43 +.92 +1.17+1.09 +.49 +.93 +1.05 +.83 +1.20 +.40 +.70
+.02 -.30 -.24 -.72 -.50 -.32 +.04 -.69 -.71 -.19 -.36 -.72 -.60
+.52 +.54 +.69 +.44 +.93 +.97 +1.46 +.38 +.25 +.34 +.53 -.01 +.32
+.12 -.00 -.07 -.53 -.19 +.18 -.47 -.28 -.06 +.22 +.52 -.11 -.44
+.63 +.74 +.83 +.50 +1.04+1.38 +.97 +.68 +.85 +.65 +1.23 +.46 +.48
+.22 +.37 +.04 -.41 -.31 -.15 -.65 -.16 -.03 +.59 +.30 +.00 +.13
+.68 +.96 +.80 +.40 +.82 +.90 +.77 +.60 +.73 +1.01 +.96 +.55 +.93
+.38 +.10 -.15 -.06 -.21 +.04 -.24 -.00 +.21 +.24 +1.11 -.07 +.24
+.77 +.51 +.38 +.39 +.65 +.88 +.87 +.64 +.86 +.44 +1.56 +.20 +.74
-.15 -.31 -.70 -.65 -.94 -.99 -.56 -.86 -.76 -.04 -.45 -1.05 -.91
+.23 +.55 +.25 +.12 -.21 -.14 +.51 -.02 +.08 +.65 +.28 -.20 -.04
+.00 -1.29 +.03 -.57 -.48 -.10 -.00 -.56 +.06 +.48 -.46 -.10 -1.05
+.24 -.84 +.44 +.20 +.18 +.82 +.76 -.03 +.63 +.73 -.01 +.17 -.31
-.18 -.59 -.34 -.88 -.70 -.14 -.63 -.83 -.43 +.30 -.36 -.29 -.56
+.36 +.27 +.71 +.16 +.66 +1.08 +.91 +.19 +.56 +.85 +.44 +.39 +.47
-.05 +.07 -.19 -.74 -.40 -.31 -.34 -.30 +.45 +.42 +1.25 -.88 -.88
+.29 +.73 +.59 +.03 +.59 +.63 +.70 +.52 +1.22 +.73 +1.76 -.51 -.00
-.07 -.00 -.08 -.63 -.19 -.18 -.56 -.39 +.06 -.03 -.00 +.27 -.75
+.51 +.73 +.75 +.19 +.93 +.86 +.56 +.59 +.90 +.37 +.72 +.76 +.21
+.15 -.32 -.08 -.64 -.32 -.13 -.13 -.01 -.12 +.05 -.15 +.39 -.40
+.65 +.25 +.63 +.14 +.63 +.86 +.98 +.76 +.69 +.38 +.44 +.87 +.37
-.05 +.36 -.10 -.61 -.42 +.09 -.29 -.09 -.24 +.39 -.22 +.00 -.10
+.45 +.98 +.68 +.11 +.42 +1.01 +.74 +.65 +.52 +.74 +.42 +.41 +.60
+.18 +.26 -.12 -.55 -.48 -.07 -.12 +.03 +.09 +.57 +.30 -.31 -.24
+.61 +.85 +.59 +.14 +.38 +.88 +.85 +.74 +.80 +.86 +.88 +.11 +.38
-.05 -.80 -.50 +.10 -.18 +.20 +.00 +.09 +.39 +.05 -.44 -.57 -.27
-.12 -.67 -.28 +.10 -.16 +.27 +.21 +.34 +.50 -.02 -.41 -.50 -.28
-.20 +.31 +.08 -.40 -.14 +.05 +.00 -.07 -.23 +.05 +.58 -.45 -.28
-.03 +.53 +.23 -.10 +.25 +.60 +.44 +.33 +.15 +.16 +.87 -.20 +.11
+.20 +.45 -.08 -.36 -.19 +.09 -.09 -.08 -.62 +.01 -.00 -.49 -.39
+.54 +.84 +.52 +.21 +.45 +.79 +.60 +.57 +.00 +.43 +.54 -.06 +.27
+.07 +.22 -.07 -.38 -.22 -.09 -.45 -.07 +.10 +.17 +.59 -.30 -.33
+.36 +.60 +.34 +.06 +.37 +.67 +.11 +.45 +.74 +.43 +1.03 +.05 +.20
+.25 +.35 -.05 -.15 -.02 -.07 -.20 -.14 -.06 +.26 -.07 +.22 -.15
+.63 +.81 +.62 +.46 +.71 +.65 +.62 +.60 +.63 +.59 +.42 +.53 +.32
+.13 +.17 -.07 -.42 +.13 -.07 -.33 +.17 +.03 +.18 +.07 +.11 -.01
+.51 +.64 +.66 +.17 +.79 +.74 +.36 +.85 +.69 +.49 +.57 +.54 +.57
+.47 +.43 -.03 -.21 -.10 -.15 -.43 -.08 -.38 +.02 +.30 +.29 -.05
+.76 +.91 +.58 +.33 +.63 +.54 +.27 +.49 +.23 +.27 +.86 +.67 +.46
+.28 +.35 +.17 -.22 +.08 +.04 -.28 +.13 +.06 +.11 +.15 +.29 -.03
+.59 +.82 +.80 +.27 +.76 +.77 +.31 +.67 +.64 +.35 +.64 +.62 +.47
-.10 -.15 -.13 -.07 -.05 +.11 -.35 -.29 -.07 -.28 +.30 +.34 -.59
+.36 +.38 +.46 +.61 +.79 +.79 +.40 +.42 +.63 +.14 +.95 +.89 +.09
-.15 -.51 -.03 +.09 -.08 +.14 -.06 +.17 +.03 -.22 +.00 -.19 +.13
-.01 -.34 +.16 +.24 +.10 +.26 +.21 +.33 +.14 -.21 +.07 -.16 +.20
+.29 -.00 +.05 +.37 +.50 -.09 -.36 +.06 +.19 +.01 -.00 +.46 -.23
+.76 +.46 +.61 +1.07+1.23 +.64 +.44 +.83 +.96 +.27 +.57 +.87 +.50
-.17 -.16 -.12 +.03 -.02 -.03 -.10 -.11 +.21 +.40 -.56 +.41 +.10
+.07 +.09 +.23 +.23 +.19 +.24 +.09 +.12 +.41 +.46 -.44 +.52 +.19
+.63 +.62 +.16 -.41 -.18 +.02 -.20 +.10 +.54 +.38 +.99 +.39 +.15
+1.13+1.16 +.81 +.23 +.67 +.90 +.72 +.80 +1.19 +.70 +1.66 +.84 +.78
-.32 -.53 -.12 -.19 -.15 +.32 +.10 -.04 -.07 -.25 -.67 +.19 -.28
-.12 -.45 +.20 +.16 +.23 +.69 +.38 +.12 +.39 -.30 -.50 +.30 -.02
+.50 +.20 +.25 -.32 -.15 -.04 +.12 -.03 -.09 +.50 +1.00 +.21 +.24
+1.05 +.62 +.83 +.32 +.52 +.73 +.82 +.67 +.60 +.72 +1.49 +.65 +.70
-.16 -.33 -.11 -.26 -.13 +.22 -.50 -.19 -.31 -.27 +.22 +.15 -.17
-.03 +.02 +.09 -.13 +.05 +.41 -.23 +.01 -.01 -.29 +.42 +.23 -.02
+1.01 +.09 -.18 -.97 -.16 -.82 -.89 -.17 -.56 +.14 -.65 -.10 -1.34
+1.37 +.70 +.67 +.49 +1.28+1.01+1.35 +.86 +.76 +.64 +.31 +.68 +.19
+1.09 -.47 -.06 -.92 -.24 -.55 -.90 -.17 -.24 -.07 -.74 -.32 -1.32
+1.23 -.01 +.66 +.38 +1.17+1.25+1.22 +.75 +.99 +.25 +.46 +.25 +.48
+1.26 +.36 +.11 -.87 -.11 -.81 -.62 -.01 -.66 +.23 -.37 +.30 -1.35
+1.53 +.99 +.90 +.51 +1.35 +.81 +1.38 +.83 +.63 +.74 +.47 +.91 +.12
+1.25 +.26 -.09 -.76 -.06 -.77 -.49 +.01 -.57 +.48 -.88 -.41 -1.27
+1.46 +.95 +.79 +.62 +1.31 +.87 +1.53 +.88 +.82 +.93 +.05 +.23 +.39
+1.17 +.33 +.01 -.54 -.19 -.49 -.86 +.00 -.77 -.29 -.22 -.06 -1.00
+1.36 +.76 +.69 +.58 +1.06 +.97 +1.20 +.95 +.43 +.07 +.66 +.38 +.12
+1.38 +.12 +.05 -.64 -.13 -.47 -.58 +.09 -.03 +.06 -.46 +.37 -1.04
+1.69 +.75 +.83 +.66 +1.19 +1.11+1.57 +1.11+1.14 +.44 +.40 +.86 +.25
+.26 +.24 +.17 -.41 -.69 -.44 -.52 -.09 +.25 +.36 -.93 -1.04 -.63
+.46 +1.15 +.72 +.40 +.12 +.41 +.53 +.49 +1.04 +.59 -.34 -.54 +.28
+.45 -.52 +.06 -.45 -.44 -.62 -.65 -.27 +.03 +.22 -.83 +.20 -.80
+.76 +.21 +.72 +.59 +.60 +.52 +.68 +.49 +.93 +.68 -.10 +.79 +.21
+.61 +.23 -.09 -.61 -.26 -.20 -.24 -.26 -.09 -.31 -.81 +.05 -.66
+.86 +.79 +.69 +.53 +.98 +1.13+1.20 +.61 +.90 +.02 -.18 +.55 +.40
+.84 +.08 -.16 -.70 -.38 -.34 -.17 -.26 -.51 +.26 -.44 -.21 -.84
+1.01 +.60 +.53 +.42 +.85 +1.02+1.29 +.57 +.55 +.46 +.24 +.28 +.20
+.81 -.50 -.24 -.67 -.06 -.44 -.45 -.19 +.15 +.02 -1.02 +.16 -.73
+1.16 +.07 +.59 +.51 +1.23+1.05+1.16 +.69 +1.26 +.49 -.28 +.74 +.24
+.79 +.17 -.33 -.52 -.14 -.34 -.72 -.14 -.24 -.37 -.14 +.22 -.55
+1.06 +.68 +.46 +.41 +.92 +.84 +.63 +.72 +.69 +.07 +.54 +.68 +.30
+.73 +.29 -.15 -.41 -.20 -.24 -.20 -.11 -.30 -.03 +.65 +.50 -.77
+1.09 +.89 +.58 +.59 +.96 +1.04 +1.11 +.66 +.57 +.29 +1.26 +.99 +.14
+.50 -.59 -.43 -.37 -.51 -.43 -.87 -.25 -.43 +.18 -.40 -.92 -.73
+.74 +.82 +.33 +1.34+1.25+1.25 +.86 +.52 +.35 +.61 +.34 -.23 +.05
+.51 +.00 -.63 -.14 -.02 -.38 -.51 -.26 -.72 +1.15 -.25 -1.68 -.14
+.68 +.99 +.24 +1.39+1.52+1.14+1.00 +.72 +.43 +1.62 +.53 -.54 +.95
+.93 -.37 -.39 -.57 -.24 -.40 -.43 -.24 -.51 -.28 -.68 -.44 -.78
+1.16 +.73 +.53 +1.06+1.23+1.15+1.41 +.55 +.40 +.10 +.32 +.19 +.27
+.75 +.00 -.99 -.46 -.20 -.09 -.05 -.21 -.82 -.12 -.45 -.10 -1.22
+1.00 +.97 +.66 +1.10+1.24+1.42+1.63 +.55 +.17 +.40 +.46 +.53 +.16
+.98 +.16 -.14 -.45 -.33 -.16 -.29 -.22 -.52 -.11 +.14 -.05 -.53
+1.38+1.17 +.65 +1.26+1.27+1.66+1.85 +.66 +.53 +.38 +1.02 +.65 +.36
+1.03 -.23 -.33 -.49 -.38 -.20 -.78 -.17 -.36 -.19 -.36 -.00 -.31
+1.39 +.60 +.41 +1.27+1.34+1.83+1.61 +.78 +.84 +.27 +.49 +.70 +.52
+1.31 -.17 +.04 -.49 -.34 -.13 -.41 -.06 -.12 +.55 +.74 +.23 -.49
+1.77 +.68 +.90 +1.23+1.24+1.78+1.74 +.88 +.80 +1.06+1.57 +.82 +.30
+1.21 +.00 +.19 -.55 -.35 -.40 -.54 -.22 -.12 +.10 +.73 -.12 -.38
+1.60 +.77 +.99 +1.17+1.29+1.62+1.76 +.62 +.76 +.64 +1.53 +.48 +.41
+1.24 +.58 +.28 -.32 -.08 +.17 +.12 +.18 +.06 +.70 +.15 +.36 -.27
+1.69+1.12 +.91 +.39 +.61 +1.08 +.93 +.77 +.73 +1.09 +.71 +.72 +.29
+1.09 +.76 +.12 -.31 -.10 -.11 -.16 +.05 -.29 +.43 +.15 +.24 -.09
+1.53+1.40 +.86 +.51 +.75 +.95 +.72 +.75 +.48 +.76 +.70 +.67 +.50
+.99 +.31 +.05 -.34 -.35 -.04 -.32 -.02 -.15 +.21 -.23 +.19 +.08
+1.47 +.93 +.70 +.50 +.50 +.98 +.59 +.71 +.69 +.58 +.43 +.62 +.67
+1.33 +.69 -.23 -.96 -.56 -.39 -.71 -.26 -.18 +.10 +.00 +.22 -.55
+1.74+1.55 +.95 +.68 +1.24+1.33+1.16 +.91 +1.02 +.60 +.85 +.94 +.64
+1.32 -.15 -.56 -1.06 -.75 -.83 -.69 -.47 -.56 +.30 +.36 -.65 -.58
+1.72 +.71 +.61 +.52 +1.06 +.89 +1.48 +.66 +.56 +.87 +1.34 +.03 +.66
+1.28 +.54 +.06 -.97 -.42 -.46 -.71 -.73 +.03 +.55 +.36 -.16 -.81
+1.57+1.17 +.81 +.35 +1.04 +.99 +1.40 +.65 +1.06 +.88 +1.01 +.45 +.30
+1.08 -.58 +.21 -.21 +.60 -.04 -.48 +.18 -.09 +.50 +.90 +.41 +.03
+1.43 +.01 +1.08 +.48 +1.40 +.84 +.36 +1.03 +.77 +.97 +1.53 +.83 +.74
+.92 -.25 -.11 -.10 +.25 +.22 -.36 +.12 +.32 +1.09 +.64 +.15 +.09
+1.26 +.31 +.83 +.63 +1.08+1.15 +.63 +.85 +1.03+1.43+1.18 +.59 +.72
+.44 -.16 -.19 -.38 +.10 -.27 -.58 -.12 -.03 +.34 -.00 -.10 -.28
+.97 +.54 +.68 +.55 +1.12 +.86 +.50 +.76 +.88 +.74 +.82 +.51 +.58
+.36 -.23 -.15 -.45 -.08 -.22 -.89 +.00 -.28 -.36 -.29 -.28 -.18
+.83 +.45 +.69 +.43 +.97 +.85 +.30 +.88 +.58 +.08 +.50 +.31 +.71
+.61 +.34 -.09 -.24 +.03 -.20 +.24 -.09 +.27 +.66 +.07 -.11 -.45
+.98 +.89 +.69 +.47 +.98 +.70 +1.21 +.70 +1.08 +.99 +.66 +.31 +.27
+.65 -.08 -.01 -.36 +.10 -.11 -.04 -.16 -.03 +.89 -.00 -.29 -.29
+1.00 +.50 +.79 +.32 +.93 +.82 +.93 +.57 +.78 +1.24 +.61 +.16 +.44
+.80 +.34 -.03 -.07 +.08 -.04 -.20 -.06 -.24 +.67 -.22 +.11 -.59
+1.32+1.09 +.96 +.78 +1.14+1.14 +.89 +.89 +.70 +1.13 +.57 +.63 +.37
+.70 +.33 +.03 -.07 +.03 -.07 -.28 -.06 +.06 +.37 +.15 +.46 -.25
+1.17 +.99 +.89 +.79 +1.06+1.02 +.85 +.81 +.98 +.75 +.96 +1.01 +.63
-.51 -.37 -.45 -.51 -.65 -.71 -.81 +.87 -.41 +.30 -1.43 -1.40 -.95
+.52 +.98 +1.14 +.95 +1.20 +.95 +1.18+1.29 +.92 +1.17 -.04 -.16 +.59
-.49 -1.22 -.24 -1.09 -.98 -.30 -.93 +.76 -.06 +.17 -.36 -.30 -1.66
+.52 +.15 +1.34 +.54 +1.20+1.61+1.42+1.21+1.45 +.82 +.95 +.74 +.15
+.14 -.44 -.49 -.20 -1.06 -.40 -.91 -.38 -.61 +.46 +.07 +.30 -.03
+.54 +.12 +.14 +.37 +.37 +.76 +.80 +.13 +.60 +.68 +.63 +.63 +.47
+.35 +.07 +.12 -.00 -.70 -.50 -.80 -.14 -.88 +.53 +.43 +.48 +.38
+.62 +.41 +.59 +.47 +.63 +.46 +.97 +.30 +.29 +.73 +.80 +.72 +.75
+.02 +.23 -.27 -.36 -.97 -.74 -1.17 -.42 -.77 +.40 +.29 +.36 -.05
+.35 +.74 +.37 +.43 +.62 +.57 +.87 +.30 +.81 +.67 +.80 +.69 +.51
+.25 +.23 +.10 -.08 -1.09 -.05 -.96 -.16 -.42 +.31 +.44 +.28 +.24
+.60 +.78 +.59 +.43 +.56 +.94 +1.14 +.50 +.99 +.59 +.81 +.54 +.58
-.15 -.75 -.18 -.76 -.52 -.19 -.50 -.42 -.16 -.24 +.29 -.21 +.39
+.40 +.22 +.71 +.31 +.84 +1.08+1.01 +.61 +.85 +.21 +1.14 +.48 +.82
-.35 -1.33 -.17 -1.09 -.79 -.57 -.47 -1.45 -.50 -.33 -.38 -.78 +.48
+.38 -.27 +.86 +.45 +.78 +1.01+1.20 -.09 +.87 +.08 +.65 -.11 +.94
-.43 -1.52 -.49 -.74 -.93 -1.05 -.09 -.73 -.61 +.17 +.75 +.15 -.40
+.72 -.16 +1.01 +1.11 +.81 +.76 +1.83 +.89 +1.03 +.65 +1.16 +.74 +.59
-.51 -.37 -.74 -.58 -.73 -.94 -.98 -.76 +.14 -.08 -.55 -.40 -1.53
+.28 +.45 +.28 +.46 +.34 +.08 +.06 +.14 +.69 +.31 +.31 +.12 -.48
+.46 +.59 +.29 -.48 -.16 -.02 -.71 +.31 +.39 +.54 +.00 +.10 -.02
+.58 +.90 +.64 -.00 +.59 +.66 +.21 +.54 +.78 +.42 +.35 +.23 +.27
+.13 -.00 -.13 -.62 -.74 -.14 -.59 +.12 +.09 -.19 -.51 +.45 -.09
+.52 +.33 +.37 +.04 +.57 +.93 +.88 +.61 +.68 +.01 -.06 +.89 +.35
ZH JA KO HI BN NE TE AR FA FR PL NL EL
ZH JA KO HI BN NE TE AR FA FR PL NL EL
−2
−1
← English advantage
0
1
2
Resource level 4 (high) 3 2 1 (low)
Local advantage →
Figure 5: GlobalGap, LocalGap, and KnowledgeGap across all models and locales.
16