arXiv CS open access

Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

2026 · arxiv_cs

arXiv CS · Papers · 2026

artificial-intelligence knowledge-representation reasoning

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

C ULTIVAR: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness Pinzhen Chen1

Koel Dutta Chowdhury2

Xiaoya Xu1

David Tan3

Doreen Osmelak3,4 Ona de Gibert5 Ariun-Erdene Tumurchuluun Ashok Urlana6,7 Fedor Sizov3 Hale Sirin8 Jesujoba O. Alabi3 Karrar Talib Abed9 Mateusz Klimaszewski10 Nikolay Bogoychev11 Niyati Bafna8 Patrícia Schmidtová12 Preksha Manjunath Shanbhag1 Sherrie Shen13 Vilém Zouhar14 Vivek Iyer13 Yasser Hamidullah15 Yusser Al Ghussin3 Zheng Zhao13 1

Queen’s University Belfast, 2 University of Technology Nuremberg, 3 Saarland University 4 University of Melbourne, 5 University of Helsinki, 6 IIIT Hyderabad, 7 TCS Research 8 Johns Hopkins University, 9 Imam Ja’afar Al-Sadiq University, 10 Cohere, 11 Last Token 12 Charles University, 13 University of Edinburgh, 14 ETH Zurich, 15 University of Zurich

arXiv:2608.09766v1 [cs.CL] 10 Aug 2026

p.chen@qub.ac.uk Abstract

Yet, massively multilingual test suites share a common limitation due to construction: regardless of language, each instance is anchored to a single information origin, which is typically English, and then translated. While this facilitates multiway parallelism, it introduces two vulnerabilities. First, because the original English content is publicly available, test sets become susceptible to data contamination over time (Magar and Schwartz, 2022; Sainz et al., 2023; Yao et al., 2024; Tan et al., 2026). Second, using such test sets to evaluate translation into English creates a language-content mismatch (Chen et al., 2024), which diverges from real-world use cases, where the input content is natively produced in the input language. BOUQuET (Andrews et al., 2025) alleviates this problem by starting with original texts in eight distinct languages and then translating into more. A separate line of effort trades away multiway parallelism for naturally written, culturally grounded text, such as WMT since 2019 (Barrault et al., 2019) and multilingual benchmarks like TyDi QA (Clark et al., 2020), which improves the authenticity of evaluation from a multicultural perspective but loses comparability. Nonetheless, we argue that source-origin content alone is insufficient. Existing test sets are fundamentally language-oriented, assuming that each language corresponds to a single cultural context. In practice, however, a language can often be used in multiple regions, with diaspora communities adding an extra layer of variation. Consequently, evaluating a model on a single test for each language either fails to capture other locales

Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation—a design that is prone to contamination over time and overlooks locale and cultural considerations. We therefore advocate for source-contrastive evaluation and instantiate it with Cultivar, a localised subset of FLORES, which enables locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancy allows the probing of data contamination and localisation robustness. We benchmark 32 open-weight models and find that MT-specialised models are less robust, a few models potentially overfit FLORES, and models tend to translate US content better than that of other locales, regardless of language.

Introduction

Benchmarks are an essential element for developing translation systems. As machine translation technology reaches maturity and is deployed for more languages, recent years have seen a shift from English-centric evaluation, like the early WMT (Bojar et al., 2014) and IWSLT (Cettolo et al., 2014) evaluation campaigns, towards multilingual test suites such as FLORES (Goyal et al., 2022; Team et al., 2024), NTREX (Federmann et al., 2022), and WMT24++ (Deutsch et al., 2025), enabling evaluation for hundreds of languages in arbitrary directions. Cultivar is potted at https://hf.co/datasets/pinzhen chen/Cultivar-flores, harvestable under CC BY-SA 4.0.

or receives an arbitrary, mixed-origin result. Thus, we argue that translation and multilingual evaluation should move beyond language-oriented tests towards locale-oriented tests, where multiple localisations of the same language are separately assessed to better reflect real-world use. We therefore advocate a source-contrastive paradigm that grounds an input in distinct contexts (Bawden et al., 2018; Futeral et al., 2023; Alam et al., 2024). Instead of testing a single static sentence, an evaluation suite should feature paired instances that preserve sentence structure while systematically varying locale-specific content based on desired locations. Such a controlled design isolates a model’s sensitivity to varied content from general translation capability. Crucially, it enables analyses that are impossible with conventional single-version tests: diagnosing benchmark contamination and quantifying true localisation robustness through paired performance discrepancies. To instantiate this paradigm, we present the Cultivar benchmark, grounded in 27 locales, created through an LLM-human co-creation pipeline starting from FLORES. We benchmark 32 language and translation models and share our findings. To summarise, our contributions are:

ing; 3) human post-editing to ensure quality. Although we create Cultivar by localising FLORES, technically, the pipeline can be applied to any existing benchmark. Unlike efforts that extend FLORES to new languages by translating from scratch (Dale et al., 2025), we formulate localisation as creative paraphrasing. Our design preserves the sentence structure of original instances while adapting cultural content to a specific location, thereby producing sentence pairs that remain directly comparable. An LLM-assisted annotation workflow is particularly suitable for this setting because localisation requires both creativity and diversity beyond a translation task, making creation from scratch difficult for humans. Our use of human post-editing ensures quality and cultural appropriateness. In this study, we only localise any-to-English translation directions. The resulting English target sentences across locales are no longer the same, but they remain comparable. Namely, it is still meaningful to compare error counts or even wordlevel metrics like BLEU between into-English directions.

• Resource: Cultivar, a multilingual translation test set grounded in locales, produced via a human-LLM co-creation pipeline. We preserve the comparability of massively multilingual test sets while incorporating native source content.

Localisation primarily affects named entities such as people, organisations, locations, events, etc. We thus first identify named entities in each instance of the English FLORES dev set using spaCy (Honnibal et al., 2020) and manually correct missing or erroneous annotations. This process finds 658 sentences (approximately 65% of the dev set) containing at least one named entity. From these, we manually select 200 sentences having at least 2 named entities. We exclude instances dominated by numerals or topics such as medicine and science, where named entities are globally shared. The resulting named entity-rich subset forms the input to the LLM localisation step.

2.1

• Evaluation: We evaluate 32 models on Cultivar using metrics like BLEU, chrF, and word recall. We report standalone locale-level scores and conduct a contrastive study with FLORES, quantifying localisation robustness and examining overfitting. • Findings: Performance of MT-specialised and smaller models tends to drop on localised content. Significant gaps between original and localised tests reveal potential FLORES overfitting. Two smaller-scale studies also show that models perform better on content grounded in the US than on localised content in the official language of a particular region.

2.2

Named entity-rich instance identification

LLM paraphrasing

Given a source language of interest, for each selected English instance, an LLM generates a localised source-English pair from the original FLORES source-English pair and a target location. The model is instructed to preserve the sentence structure whenever possible while replacing content with appropriate alternatives for the new location. Our full prompt is provided in Figure 4 in Section A. We use GPT-5.5 (gpt-5.5-2026-0423).

Cultivar

We construct Cultivar through a three-stage pipeline that localises FLORES: 1) named entityrich instance selection; 2) LLM-assisted paraphras2

Download JSON

Load JSON

1 / 200 reviewed 0 revised pairs

Reset draft

Loaded saved draft from this browser. Download the annotation JSON when the review is done.

deu_Latn_Germany_0

confirmed

ANNOTATION

ORIGINAL GERMAN

REPHRASED GERMAN

Am Montag haben die Wisenschaftler der Stanford University School of Medicine die Erfindung eines neuen Diagnosetools bekanntgegeben, mit dem Zellen nach ihrem Typ sortiert werden können: ein winziger, ausdruckbarer Chip, der für jeweils etwa einen US-Cent mit StandardTintenstrahldruckern hergestellt werden kann.

Am Montag haben die Wissenschaftler der Charité – Universitätsmedizin Berlin die Erfindung eines neuen Diagnosetools bekanntgegeben, mit dem Zellen nach ihrem Typ sortiert werden können: ein winziger, ausdruckbarer Chip, der für jeweils etwa einen Euro-Cent mit StandardTintenstrahldruckern hergestellt werden kann.

ORIGINAL ENGLISH

REPHRASED ENGLISH

On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.

On Monday, scientists from Charité – Universitätsmedizin Berlin announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one euro cent each.

Confirm final version

Der Eurofighter Typhoon stürzte gegen 9:30 Uhr Ortszeit (02:30 UTC) auf eine Startbahn und explodierte, sodass der Flughafen für kommerzielle Flüge geschlossen werden musste.

ORIGINAL ENGLISH

REPHRASED ENGLISH

The JAS 39C Gripen crashed onto a runway at around 9:30 am local time (0230 UTC) and exploded, closing the airport to commercial flights.

The Eurofighter Typhoon crashed onto a runway at around 9:30 am local time (0230 UTC) and exploded, closing the airport to commercial flights.



ANNOTATION

REPHRASED GERMAN



Revised rephrased English sentence

pending confirmation

Der JAS 39C Gripen stürzte gegen 9:30 Uhr Ortszeit (02:30 UTC) auf eine Startbahn und explodierte, sodass der Flughafen für kommerzielle Flüge geschlossen werden musste.

English diff: 17

Revised rephrased German sentence

deu_Latn_Germany_2 ORIGINAL GERMAN

German diff: 14

German diff: 6

English diff: 6

Confirm final version

Revised rephrased German sentence

Der Eurofighter Typhoon stürzte gegen 9:30 Uhr Ortszeit (02:30 UTC) auf eine Startbahn und explodierte, sodass der Flughafen für kommerzielle Flüge geschlossen werden musste. Revised rephrased English sentence

The Eurofighter Typhoon crashed onto a runway at around 9:30 am local time (0230 UTC) and exploded, closing the airport to commercial flights.

deu_Latn_Germany_3

confirmation German diff: 4 English diff: 4 Figure 1: An illustration of the annotation interface and the content before andpending after localising German written ORIGINALscript GERMAN(deu_Latn) to Germany, REPHRASED GERMAN in Latin paired with English.ANNOTATION The difference between the original and replaced Confirm final version Der Pilot wurde als Staffelführer Dilokrit Der Pilot wurde als Staffelführer Werner content is highlighted. The annotation panel on the right is, by default, populated with the LLM-rephrased content, Pattavee identifiziert. Mölders identifiziert. Revised rephrased German sentence which the annotator can edit; as they edit, the highlights update accordingly. Once done, annotators check “Confirm Der Pilot wurde als Staffelführer Werner Mölders identifiziert. ORIGINAL ENGLISH REPHRASED ENGLISH this sentence pair. final version” to freeze the edit box to finalise

The pilot was identified as Squadron Leader Dilokrit Pattavee.

2.3

The pilot was identified as Squadron Leader Werner Mölders.

Revised rephrased English sentence

Human annotation

3. revert to the original FLORES entry. The pilot was identified as Squadron Leader Werner Mölders.

Annotator profile We use three criteria to select Similar to the LLM prompting stage, annotators are an annotator: 1) proficient in the source language; instructed to preserve the original sentence struc2) proficient in the target language (English); and ture whenever possible to maintain comparability 3)deu_Latn_Germany_5 they live(d) in the location. These requirements between the original localised pendingand confirmation German diff:benchmarks. 6 English diff: 6 ensure both language competence and familiarity The first two options produce approved localised ORIGINAL GERMAN REPHRASED GERMAN ANNOTATION Confirm final version with cultural Der local 28-jährige Vidal war vorconventions. drei Der 28-jährige Goretzka war vor drei instances. Annotators were also allowed to use opSpielzeiten von Sevilla zu Barça Spielzeiten von Schalke zu Bayern Revised rephrased German sentence The majority of annotatorsgekommen. are NLP researchers tion 3 sparingly, when localisation is unnecessary, gekommen. Der 28-jährige Goretzka war vor drei Spielzeiten von Schalke zu with some understanding of NLP evaluation. All culturally inappropriate, or difficult. Bayern gekommen. ORIGINAL ENGLISH REPHRASED ENGLISH annotators arehadincluded theirhadcontri28-year-old Vidal joined Barça as authors, 28-year-oldas Goretzka joined Bayern three seasons ago, from Sevilla. three seasons ago, from Schalke. Before annotation starts, annotators are inbutions directly shape the benchmark and analyses. Revised rephrased English sentence structed to read through a brief motivation and Annotation process We ask annotators to inspect description of the project, annotation guidelines, each instance pair before and after LLM localisa- and a few examples of what they do and do not tion and perform one of the three actions below: need to edit. Detailed annotation instructions are 1. accept the LLM paraphrase; presented in Figure 5 in Section B. Annotation time 2. post-edit the paraphrase; ranges from 4 hours to 12 hours. 3

Lang

Script

Location

acm apc ben bul cat ces cmn

Arab Arab Beng Cyrl Latn Latn Hans

deu

Latn

fil gom hin

Latn Deva Deva

jpn khk plt pol rus slk spa tel tur yor

Jpan Cyrl Latn Latn Cyrl Latn Latn Telu Latn Latn

Iraq Syria India Bulgaria Spain Czechia China Singapore UK† US† Germany Switzerland Philippines India Germany† India US† Japan Mongolia Madagascar Poland Russia Slovakia Spain India Turkey Nigeria

Human Edited

Average Edits

Kept LLM Paraphrase

Reverted to FLORES

Source Only

Target Only

Both

Source

English

176 155 146 161 136 109 168 153 170 180 117 154 193 174 1 112 113 89 68 187 176 136 144 117 173 147 127

0 0 0 0 8 0 1 1 7 4 0 1 1 8 1 7 34 2 0 0 3 8 1 6 0 6 0

14 14 41 2 1 15 2 7 10 0 3 6 4 0 0 0 8 7 53 5 0 26 3 2 5 13 56

1 2 0 0 1 12 7 5 0 1 2 1 0 0 0 7 0 32 14 0 19 0 1 3 2 1 1

9 29 13 37 54 64 22 34 13 15 78 38 2 18 198 74 45 70 65 8 2 30 51 72 20 33 16

4.80 4.78 7.67 4.41 4.70 4.61 4.01 4.47 4.37 4.34 4.48 4.66 4.24 6.14 6.90 7.21 7.44 3.63 5.25 4.12 4.28 4.90 4.68 4.91 7.30 4.21 5.28

4.28 4.38 4.19 4.14 4.43 4.44 4.27 4.72 4.22 4.28 4.46 4.50 4.15 3.99 4.32 4.53 4.47 4.47 4.46 3.92 4.12 4.33 4.29 4.44 4.36 4.19 4.15

140.07

3.67

11.00

4.15

41.11

5.10

4.31

Average

Table 1: Summary of human edits to LLM-localised FLORES instances: LLM paraphrases are kept as is, reverted to FLORES, or edited by the annotators. Localisation targets regions where a language holds official or de facto status, except for the four entries marked with a †.

Annotation interface Annotation is performed through a custom web interface shown as Figure 1. It displays the original and localised sentence pairs side by side while highlighting changed spans. The edit box is initialised with the LLM paraphrase, allowing annotators to efficiently post-edit the text. Differences and highlights are updated as edits are made. After completing each entry, annotators finalise it by checking a confirmation box. This deliberate step acts as a check on attention and responsibility, as it encourages careful review and reduces the likelihood of inadvertent skipping. 2.4

contributed by a single annotator. A breakdown of annotation statistics for each locale is in Table 1. On average, annotators accept 70% of LLM-generated localisations without modification, post-edit roughly 28%, and revert fewer than 2% to the original FLORES entries. Consequently, over 98% of Cultivar is newly localised content. A few outliers exist, for example: all but one of the LLM paraphrases are human-edited when Hindi is localised to Germany; only 7 and 13 are changed when Filipino and Malagasy are localised, respectively. At the source-target pair level, we count a modified continuous string surrounded by original strings as an edit. We only count edits to an instance if the LLM paraphrase is revised—either reverted to FLORES or edited by the annotator. On average, there are 4 edits per English instance and 5 edits per source instance. However, the latter is likely inflated by the Indian languages in our dataset (ben,gom,hin,tel), all of which average around 7 edits per source.

Annotated data statistics

The final localised Cultivar benchmark consists of 27 language-script-location combinations— referred to as locales. We represented them with the same language and writing script codes as FLORES, together with the country name where the data is grounded. There are 21 unique languages or dialects, 8 unique writing scripts, and 21 unique countries or regions. Each locale has 200 instances 4

Experimental Setup

3.1

Models

between the localised and original tests as: ∆BLEU = BLEUlocalised − BLEUoriginal ∆chrF = chrFlocalised − chrForiginal

We evaluate 32 open-weight models that can be deployed on no more than two NVIDIA A100 80GB GPUs. Our model selection considers four dimensions: 1) translation-specific or general-purpose; 2) 0.8–122B in size; 3) English-focused or multilingual; 4) made in or targeting diverse locations. We list them under three groups: • Pre-trained machine translation models (MT): nllb-200-distilled-{600M,1.3B} (Team et al., 2024) and madlad400-{3b,7b}-mt (Kudugunta et al., 2023).

As ∆ becomes increasingly negative, the model exhibits greater performance degradation when translating localised source texts. Importantly, ∆ is largely invariant to translation quality—a model that performs poorly or strongly on both test sets will receive a ∆ → 0 for its consistent behaviour. ∆BLEU and ∆chrF To aggregate over multiple locales after localisation, we pair all locales with their language, forming pairs i = 1, 2, · · · , n and compute a mean pairwise ∆metric applicable to both BLEU and chrF:

• LLM-based translation models (LLM+MT): Seed-X-PPO-7B (Cheng et al., 2025) and HyMT2-{1.8B,7B,30B-A3B} (Zheng et al., 2026).

∆metric =

• LLMs: tiny-aya-global (Salamanca et al., 2026), aya-expanse-8b (Dang et al., 2024), SmolLM-1.7B-Instruct (Allal et al., 2024), emma-500-llama3.1-8b-{mono,bi} (Ji et al., 2026), Qwen3.5-{0.8B,2B,4B,9B,27B,35BA3B,122B-A10B-FP8} (Qwen Team, 2026), Olmo-3-7B-Instruct (Olmo Team et al., 2025), gemma-3-{1b,4b}-it (Gemma Team, 2025), Llama-3.1-8B-instruct, Llama-3.2-{1B,3B}-instruct, Llama-3.370B-Instruct (Grattafiori et al., 2024), Phi4-mini-instruct (Abouelenin et al., 2025), Ministral-3-{3B,8B}-Instruct-2512 (Liu et al., 2026), and EuroLLM-{1.7B-Instruct, 9B-instruct-2512} (Martins et al., 2025). 3.2

1X (scorelocalised,i − scoreoriginal,i ) n i=1

Likewise, the mean ∆metric can be averaged across models but without requiring pairing. Locale-specific word recall Standard metrics reveal overall quality (difference) but provide little focus on locale-specific content. Since both LLMs and human annotators were instructed to avoid changing the sentence structure, most word edits would be directly related to the effort of languagelocation grounding. We therefore introduce a simple lexical analysis that looks at the translation of a localised input, its original FLORES reference, and the localised Cultivar reference: • False positives (FP): unique terms in the original FLORES reference that appear in the translation of a localised instance. These should no longer appear after localisation.

Evaluation

BLEU and chrF We compute BLEU (Papineni et al., 2002) and chrF (Popović, 2015), two standard automatic metrics for machine translation evaluation. Specifically, we use spBLEU with flores200 tokenizer and chrF with 0 word order, as implemented in sacrebleu (Post, 2018). This way, the metrics operate on subwords and characters, respectively, without complications due to tokenization for certain languages.

• False negatives (FN): unique terms in the localised Cultivar reference that do not appear in the translation. Since the localisation process introduced such words, they should appear. A high false positive indicates that a model continues to generate terms in the original FLORES despite their removal during localisation, suggesting memorization. Conversely, a high false negative means that models fail to generate newly introduced locale-specific content. Nonetheless, as multiple valid translations may exist for a localised concept, false negatives should be interpreted as an upper bound on localisation failures rather than an exact error rate.

∆BLEU and ∆chrF as measures of robustness Our primary research objective is not to discover the best translation models, but to quantify their robustness via results before and after localisation. Therefore, we define a performance discrepancy 5

BLEU Localised

τb

acm_Arab apc_Arab ben_Beng bul_Cyrl cat_Latn ces_Latn cmn_Hans cmn_Hans cmn_Hans cmn_Hans deu_Latn deu_Latn fil_Latn gom_Deva hin_Deva hin_Deva hin_Deva jpn_Jpan khk_Cyrl plt_Latn pol_Latn rus_Cyrl slk_Latn spa_Latn tel_Telu tur_Latn yor_Latn

Iraq Syria India Bulgaria Spain Czechia China Singapore UK US Germany Switzerland Philippines India Germany India US Japan Mongolia Madagascar Poland Russia Slovakia Spain India Turkey Nigeria

0.97 0.97 0.98 0.93 0.97 0.98 0.95 0.94 0.95 0.98 0.98 0.97 0.97 0.95 0.96 0.95 0.98 0.94 0.97 0.98 0.93 0.96 0.97 0.97 0.96 0.97 0.99

0.89 0.88 0.92 0.80 0.88 0.89 0.84 0.80 0.85 0.91 0.90 0.88 0.92 0.85 0.87 0.85 0.92 0.82 0.86 0.92 0.82 0.87 0.89 0.88 0.90 0.87 0.92

0.97 0.98 0.98 0.94 0.98 0.98 0.96 0.95 0.98 0.99 0.99 0.98 0.98 0.97 0.97 0.96 0.97 0.95 0.98 0.99 0.96 0.97 0.97 0.98 0.97 0.98 0.99

0.90 0.91 0.92 0.83 0.91 0.90 0.85 0.84 0.90 0.92 0.93 0.90 0.93 0.89 0.89 0.88 0.90 0.83 0.90 0.94 0.85 0.88 0.89 0.91 0.90 0.90 0.96

0.98

0.91

0.98

0.89

Average

tion, since for each locale, all source instances are localised to relevant locations. To study the models’ behaviour on the localised source, we report previously introduced metric scores in Table 3: 1) mean pairwise ∆BLEU and ∆chrF ; 2) BLEU and chrF on FLORES; 3) BLEU and chrF on Cultivar; and 4) false positive and false negative rates for unique word recall. It is worth noting that a mean pairwise ∆ score does not equal the difference between the averaged scores for FLORES and Cultivar, because one language can be paired with multiple locales. As Table 3 shows, for most models, ∆BLEU ranges between −2 and +2, and ∆chrF ranges between −2 and +0.7, with averages of −0.49 and −1.1, respectively. Specifically, SmolLM-1.7BInstruct acts as a “random” baseline—as the model is not good at translation, reflected by its BLEU and chrF—with ∆ scores close to 0. However, two models at the bottom see significantly lower ∆ scores when the input is localised, with a > 8 BLEU drop and a > 5 chrF drop. Most models’ false negatives fall within 10–30%, with a few higher numbers associated with weaker translation models showing low BLEU and chrF scores, which is largely expected. If we treat these weaker models with FN >44% as random baselines, we can conclude that most models can adapt to the localised source to a certain extent.

chrF

Original

Table 2: Spearman’s rs and Kendall’s τb for BLEU and chrF, measuring the alignment between rankings of 32 models based on the original FLORES and the localised Cultivar when translating into English.

Empirical Results and Analysis

4.1

Do Cultivar and FLORES yield consistent model rankings?

Model type Considering how models have been trained, MT-optimised ones rank lower by ∆ scores, including both pre-trained translation models (NLLB,MADLAD) and LLMs further optimised for translation (Hy-MT2,Seed-X-PPO-7B). Even though some of these have a small magnitude, they are consistently clustered in the negative region.

We use the 32 models to individually translate FLORES and Cultivar test sets into English. We then compute BLEU and chrF scores for the translations using their respective references. Finally, we compute Spearman’s and Kendall’s rank correlation coefficients, rs and τb , between model rankings for each language or locale, as determined by BLEU and chrF respectively. Table 2 shows rs ≥ 0.93, τb ≥ 0.80 for all translation directions and rs ≥ 0.98, τb ≥ 0.89 on average. These indicate that Cultivar and FLORES counterparts induce highly consistent outcomes in ranking models. 4.2

Model size We then plot ∆BLEU and average ∆chrf against model size in Figure 2 using different marks and colours to represent model families. In general, we see ∆BLEU and ∆chrf increase as the model gets larger for 6 out of 8 families. Two families do not follow this pattern: Hy-MT2 slightly drops as it gets larger, and Llama 3.X peaks at 8B followed by a sharp decline at 70B—interpreted with reservations, because Llama spans version 3.1– 3.3. Since ∆ measures the performance difference between FLORES and our Cultivar, an in-family scaling of ∆ means that the BLEU scores on the two test sets diverge as models get larger. This could imply that FLORES may underestimate the true multilingual translation capabilities of stronger models.

Do models perform better or worse with localised source content?

Our localised Cultivar demonstrates a more natural and aligned use case of source-to-English transla6

Pairwise Type

Model

LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM LLM MT LLM LLM MT MT MT LLM+MT LLM+MT LLM+MT LLM LLM+MT

FLORES

Cultivar

∆BLEU ↓

∆chrF

BLEU

chrF

BLEU

chrF

F Pos

F Neg

Qwen3.5-35B-A3B Llama-3.1-8B-instruct Ministral-3-8B-Instruct-2512 EuroLLM-9B-instruct-2512 Phi-4-mini-instruct gemma-3-4b-it Qwen3.5-122B-A10B-FP8 Llama-3.2-3B-instruct Qwen3.5-27B Olmo-3-7B-Instruct Qwen3.5-9B Qwen3.5-2B Ministral-3-3B-Instruct-2512 EuroLLM-1.7B-Instruct Qwen3.5-4B Llama-3.2-1B-instruct gemma-3-1b-it tiny-aya-global SmolLM-1.7B-Instruct emma-500-llama3.1-8b-mono emma-500-llama3.1-8b-bi madlad400-7b-mt Llama-3.3-70B-Instruct Qwen3.5-0.8B nllb-200-distilled-1.3B madlad400-3b-mt nllb-200-distilled-600M Hy-MT2-1.8B Hy-MT2-7B Hy-MT2-30B-A3B aya-expanse-8b Seed-X-PPO-7B

1.70 1.42 1.40 1.36 1.07 0.96 0.92 0.89 0.80 0.61 0.61 0.56 0.52 0.48 0.42 0.30 0.27 0.03 -0.02 -0.03 -0.10 -0.53 -0.58 -0.70 -0.79 -0.90 -1.21 -2.06 -2.16 -2.28 -8.87 -9.68

0.20 0.31 -0.04 -0.03 -0.53 -0.24 -0.17 -0.35 -0.25 -0.48 -0.37 -0.40 -0.38 -0.43 -0.64 -0.93 -0.79 -0.88 0.07 -0.96 -0.75 -1.34 -0.98 -1.41 -1.43 -1.50 -2.05 -2.09 -2.06 -2.21 -5.84 -6.32

41.46 35.36 35.25 36.98 29.25 35.95 43.26 32.22 42.70 27.05 39.29 27.46 29.72 27.83 35.33 21.82 25.35 32.90 3.92 36.58 43.20 42.38 44.16 19.70 41.44 40.88 37.40 38.54 44.29 47.46 50.51 53.18

64.31 59.39 60.10 59.38 54.96 60.47 65.71 56.00 65.14 53.42 62.51 52.46 55.71 49.99 59.82 44.89 50.84 56.88 20.15 59.64 65.18 64.31 65.84 43.80 63.76 63.25 60.95 61.45 65.54 68.17 67.37 71.99

43.34 37.58 37.40 39.61 31.57 37.63 44.38 34.33 43.79 28.75 40.38 28.58 31.07 30.45 36.32 23.21 26.63 33.37 4.38 37.32 43.57 42.67 44.40 19.66 40.71 40.84 36.60 37.77 42.98 45.77 44.50 45.64

64.86 60.30 60.75 60.63 55.86 61.01 65.80 56.85 65.25 54.11 62.68 52.65 56.20 51.99 59.78 45.32 51.19 56.52 21.83 59.25 64.83 63.78 65.49 42.83 62.75 62.52 59.51 60.68 64.38 66.58 63.81 67.19

0.0069 0.0076 0.0087 0.0077 0.0092 0.0080 0.0067 0.0094 0.0069 0.0114 0.0077 0.0098 0.0100 0.0082 0.0082 0.0098 0.0097 0.0106 0.0085 0.0092 0.0074 0.0089 0.0073 0.0106 0.0094 0.0092 0.0098 0.0098 0.0092 0.0066 0.0097 0.0122

0.1186 0.1717 0.1690 0.1753 0.2579 0.1908 0.1039 0.2394 0.1149 0.3205 0.1520 0.2928 0.2291 0.2978 0.1916 0.4400 0.3675 0.2780 0.6930 0.2155 0.1569 0.2001 0.1214 0.4403 0.2382 0.2266 0.2924 0.2243 0.1661 0.1217 0.2115 0.1665

Average

-0.49

-1.10

36.66

59.43

36.10

58.79

0.0089

0.2370

Table 3: Aggregated results for each model: mean pairwise ∆ scores; FLORES scores averaged across languages; Cultivar scores averaged across locales; and locale-specific word recalls averaged across locales.

4.3

Are models FLORES contaminated?

not record significantly higher false positives compared to models at the top. Hence, there is no direct evidence that these models have memorised the exact FLORES text. Nonetheless, disproportionate drops in BLEU and chrF could hint at a lack of robustness to localisation or some degree of overfitting to FLORES, since localisation still retains the original sentence structure. This is corroborated by an error analysis in Section 5.2 later.

Looking at BLEU and chrF scores, aya-expanse8b and Seed-X-PPO-7B are strong on both FLORES and Cultivar tests, beating much larger general-purpose LLMs like Llama-3.3-70BInstruct and Qwen3.5-122B-A10B-FP8. However, they exhibit a much larger performance decline (∆BLEU < −8) than other models (−2 < ∆BLEU < +2) when moving from FLORES to Cultivar. A further score breakdown in Section C Figure 6 shows that, for the two models, a substantial BLEU or chrF drop occurs in most source-toEnglish directions, implying a model artifact. We further test for text memorization—whether verbatim FLORES English words are undesirably recalled when translating counterpart Cultivar instances. We probe this via false positives as defined in Section 3.2 and reported in Table 3. Most models fall below 1%. Looking at the models at the bottom of the table with negative ∆ scores, they do

4.4

Which locales are harder?

Table 4 lists ∆BLEU and ∆chrF scores again, but this time averaged across models for each locale. Most ∆BLEU range from −2 to +2, and most chrF range from −2 to +1. According to BLEU differences, models perform better on localised plt_Latn_Madagascar and gom_Deva_India; according to chrF differences, models perform better on hin_Deva_US, cmn_Hans_US, and plt_Latn_Madagascar. At the bottom of the table, cmn_Hans_Sin7

0.5

2 EuroLLM

1 0

Gemma 3

Ministral MADLAD

−1

EuroLLM

−0.5 ∆chrF

∆BLEU

Qwen 3.5

Llama 3.X

−2 1

10 Total Parameters (B)

Llama 3.X

Qwen 3.5

Llama 3.X MADLAD

−2

Hy-MT2

Hy-MT2 NLLB

100

Gemma 3

Qwen 3.5

−1

−1.5

NLLB

Ministral

Gemma 3

10 Total Parameters (B)

EuroLLM

Ministral

MADLAD

100 NLLB

Hy-MT2

Figure 2: Plots of ∆BLEU (left) and ∆chrf (right) averaged across locales against model size (logarithmic scale). ∆chrF by Location

∆BLEU by Location 1 0 0 −2

−1 Hindi Chinese

−4 US

−2 DE

Figure 3: Plots of ∆BLEU and ∆chrF averaged across models for translating localised Hindi and Chinese to English.

gapore receives significantly low ∆, and another five localised test sets also have ∆BLEU and ∆chrF around −2. These results demonstrate that the difficulty of a test set could change after being localised—some appear easier while others become harder. Nonetheless, it is noteworthy that localisations were produced by an LLM and edited by individual annotators, who may bring in their personal biases, so the difficulty of individual locales could be attributed to a combination of model artifact, e.g. training data, as well as data artifact relative to the original FLORES instances. 4.5

separated by location. We see negative ∆ for Germany, China, and Singapore, mixed results for India, and positive ∆s for the US. Moreover, since the ∆ scores are relative to FLORES, we can compare the performance between two localisations directly by comparing ∆s. For example, when translating from Chinese to English, by definition: BLEUcmn_US − BLEUcmn_SG = ∆BLEU cmn_US − ∆BLEU cmn_SG . A positive ∆BLEU cmn_US and a negative ∆BLEU cmn_SG in Figure 3 suggests that the difference between BLEUcmn_US and BLEUcmn_SG is significant. The same applies to chrF scores too. Overall, the results indicate that, relative to the original FLORES, models translate US content better in both Hindi and Chinese, but translate content about China and Singapore worse in Chinese. The positive ∆ for the US implies that, compared to FLORES, Cultivar’s US localisation is easier for models, even though the source languages, Hindi and Chinese, are not de facto in the US. On the other hand, the negative ∆s are not ideal, because they indicate that models are less capable of translating source-localised content into English, which

Same language, different localisations

In our test set, Hindi and Chinese have been localised to locations where they are not the official or de facto language, allowing us to conduct a smallscale ablation study on how models respond to localisation while controlling the language. In detail, Hindi has been localised to the US and Germany in addition to India; Chinese has been localised to the US and UK in addition to China and Singapore. In Figure 3, we visualise the ∆BLEU and ∆chrF for both languages, averaged across models but 8

Lang

Script

Location

∆BLEU ↓

∆chrF

plt gom yor hin tel hin ben cmn pol spa deu bul hin slk fil tur cmn cat ces cmn rus deu acm khk jpn apc cmn

Latn Deva Latn Deva Telu Deva Beng Hans Latn Latn Latn Cyrl Deva Latn Latn Latn Hans Latn Latn Hans Cyrl Latn Arab Cyrl Jpan Arab Hans

Madagascar India Nigeria US India India India US Poland Spain Germany Bulgaria Germany Slovakia Philippines Turkey UK Spain Czechia China Russia Switzerland Iraq Mongolia Japan Syria Singapore

2.53 2.05 1.48 1.38 1.14 1.00 0.96 0.92 0.38 0.05 0.02 -0.21 -0.28 -0.30 -0.68 -0.96 -1.03 -1.24 -1.30 -1.51 -1.70 -1.97 -2.06 -2.13 -2.65 -2.77 -4.29

0.86 -0.06 0.09 1.36 -0.27 -0.76 -0.38 1.11 -1.89 -0.76 -1.25 -0.65 -1.31 -1.55 -0.57 -1.91 -0.52 -1.26 -2.16 -1.06 -1.80 -2.31 -2.32 -2.88 -2.81 -2.49 -2.22

-0.27

-1.05

Average

wanted reasoning or formatting tokens. We classify errors based on their manifestation rather than underlying cause, with errors directly concerning localised content categorised as localespecific. An observed error may nevertheless arise from either locale-specific or general difficulties. Repeated errors of the same type within an instance are counted only once, while different types are counted separately. 5.2

Table 5 presents the results of our human error analysis, and we also include BLEU for reference. Across the three regional variants, Chinese content grounded in the US yields the fewest errors, with this difference becoming more pronounced among smaller models with higher error rates. Across models, we observe a clear scaling trend: locale-specific (and general) errors decrease as the model’s total parameter count increases, while the number of active parameters plays a minor role. This pattern is evident in the Qwen3.5 mixture-ofexperts models: both 35B-A3B and 122B-A10B exhibit substantially fewer errors than the 2B,4B,9B dense models, despite activating a smaller or comparable number of parameters. We observe two interesting patterns for SeedX-PPO-7B: 1) Its US split BLEU is much higher than the UK and China splits, whereas BLEU differences across locations are smaller for the other models. 2) We find a moderate-to-strong negative correlation between BLEU and total error count for Chinese, with Seed-X-PPO-7B being an outlier— Pearson’s rincl = −0.64 and rexcl = −0.86 when including and excluding it. Despite having an error count comparable to Qwen3.5-9B and higher than larger models, it obtains a substantially higher BLEU than all other models in Table 5. This mismatch between error counts and BLEU hints at the possibility of FLORES-like data contamination, whereby overfitting to domain, style, or sentence structure may inflate automatic scores despite the presence of errors. Broadly, our observed error patterns in model size and locale difficulty, despite a small scale, are consistent with, and provide qualitative support for, our earlier findings based on BLEU, chrF, and ∆ scores.

Table 4: Average ∆BLEU and ∆chrF for each locale.

is the main use case of source-to-English translation. From a model building perspective, this contrastive evaluation reflects a US-centric bias in training data, and Cultivar has provided a systematic way to probe such locale-centric bias.

Manual Error Analysis

5.1

Setup

Observations corroborate earlier findings

In addition to analysis based on automatic evaluation results, we conduct a manual inspection of translations of regionally localised content. In particular, a Chinese native speaker who is fluent in English inspected the Chinese-to-English outputs from selected models together with sources and references. Nine error types are pre-defined and grouped into two categories: • Four locale-specific errors: named entities, idioms and culturally specific expressions, number or unit, as well as Pinyin, a deterministic phonetic conversion system for transcribing Mandarin Chinese names into the Latin alphabet.

5.3

• Five general errors: semantic mistranslation, omitted information, hallucinated or added information, original language retained, and un-

Locale-specific error patterns

Locale-specific errors substantially outnumber general translation errors. In aggregate, named entities 9

66 277

0 0 0 0 0 0 0 0 0 0

0 1 0 0 0 0 0 0 0 0

1 5 3 2 5 2 1 2 1 0

171 0 1 22

85 279

13 16 18 13 23 17 9 5 8 9

0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 0 0 0 0 0

1 1 1 0 2 0 0 0 0 0

131 0 1 5

ral To tal

21 44.94 41 34.00 41 30.92 30 37.22 59 31.89 36 35.41 20 37.23 13 38.01 11 37.29 7 37.59

8 12 14 12 15 10 5 2 4 3

BL EU

12 23 24 16 39 24 14 9 6 4

NE PY IC NU

Ge ne

133 54 10 14

21 41.87 37 33.45 50 32.32 35 37.81 60 32.79 26 35.59 21 37.32 8 38.30 10 38.00 9 38.48

To tal

Total

4 12 11 5 13 7 7 2 2 3

ner a

1 2 3 2 4 1 0 0 0 1

To tal

8 7 1 15 7 1 27 7 2 16 10 2 33 7 3 12 6 0 10 3 1 4 2 0 5 3 0 3 2 0

ner

Seed-X-PPO-7B Llama-3.1-8B-Instruct Ministral-3-8B-Instruct-2512 EuroLLM-9B-Instruct-2512 Qwen3.5-2B Qwen3.5-4B Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B-A3B Qwen3.5-122B-A10B-FP8

BL EU

U BL E

NE PY IC NU

Chinese_US→English Locale

Chinese_UK→English Locale

Model

Chinese_CN→English Locale

4 7 7 5 20 6 4 4 3 2

19 47.31 24 35.36 26 33.63 18 38.82 45 34.02 23 35.74 13 38.10 9 39.44 11 38.28 11 38.63

62 199

Table 5: Error counts and BLEU by model and country. We list locale-specific, general, and total errors separately. Locale-specific errors include named entities (NE), Pinyin (PY), idiom and culture (IC), and number and unit (NU).

(NE) errors are the most frequent, accounting for > 80% of locale-specific errors and > 50% of all recorded errors. This pattern is consistent across models and locations, indicating that the accurate translation of localised entities remains a primary source of errors even when the general translation is successful.

Overall, the error analysis indicates that localisation affects the difficulty of translation, reflected in the nature of the errors that models make, which vary for different locales. This underscores the value of a locale-oriented test set with a unique source of locale-specific difficulties.

The remaining locale-specific error types exhibit more pronounced regional variation. Errors involving idioms and culturally specific expressions (IC) are more frequent in the China split. As expected, Pinyin (PY) transcription errors are exclusive to China, making up 20% of all errors. These errors arise when Chinese names require romanisation rather than translation. The asymmetry with the UK and US splits is consequential: names in those splits originate from English and are represented in Chinese before being translated back into English, whereas the China split requires the hard-togeneralise Chinese-to-Latin script mapping ability. Thus, the China split presents a unique difficulty of script- and language-specific entity transformation that is absent from other localisations.

Conclusion

This work introduces Cultivar, a source-contrastive, locale-oriented test set that complements the current massive multilingual translation evaluation. Cultivar isolates a model’s robustness to cultural content from general translation capability and serves as a diagnostic tool for data contamination. Evaluating 32 models reveals that smaller and MTspecialised systems perform worse on localised content. Furthermore, our contrastive design exposes potential FLORES overfitting and a pervasive US-centric bias, where models translate USgrounded content better than content from native regions. Ultimately, Cultivar demonstrates that true multilingual proficiency requires locale-aware evaluation to accurately reflect real-world use cases.

Number and unit errors (NU) also show substantial regional variation. The UK and China splits contain 22 and 14 such errors, respectively, compared with only 5 in the US split. These errors involve region-specific measurements and formatting, such as jin (weight), British pounds (weight and currency), and chronological conversions (century and decade to year). The concentration of such errors in the China and UK splits suggests that models are more likely to struggle when the source text contains numbers or units that differ from those in general-purpose translation data.

Acknowledgements AI assistants have been used for coding, visualisations, and paper editing. Conceptualisation, methodology, experiments and analysis, and writing were done by the authors. We thank Teresa Sy Ortin and Bouazza Laracha for data contribution. We are grateful for the computing resources from the Northern Ireland High Performance Computing (NI-HPC) service funded by EPSRC (EP/T022175).

References

Pinzhen Chen, Simon Yu, Zhicheng Guo, and Barry Haddow. 2024. Is it good data for multilingual instruction tuning or just bad multilingual evaluation for large language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, and others. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743.

Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jingwen Chen, Zhichao Huang, and others. 2025. SeedX: Building strong multilingual translation LLM with 7B parameters. arXiv preprint arXiv:2507.13618.

Md Mahfuz Ibn Alam, Sina Ahmadi, and Antonios Anastasopoulos. 2024. CODET: A benchmark for contrastive dialectal evaluation of machine translation. In Findings of the Association for Computational Linguistics: EACL 2024.

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8.

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. SmolLM - blazingly fast and remarkably powerful.

David Dale, Laurie Burchell, Jean Maillard, Idris Abdulmumin, Antonios Anastasopoulos, Isaac Caswell, and Philipp Koehn. 2025. Findings of the WMT 2025 shared task of the open language data initiative. In Proceedings of the Tenth Conference on Machine Translation.

Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa-jussà, Joe Chuang, David Dale, Mark Duppenthaler, Nathanial Paul Ekberg, Cynthia Gao, Daniel Edward Licht, Jean Maillard, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Eduardo Sánchez, Ioannis Tsiamas, Arina Turkatenko, Albert Ventayol-Boada, and Shireen Yates. 2025. BOUQuET : dataset, benchmark and open initiative for universal quality evaluation in translation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, and others. 2024. Aya Expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261.

Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1).

Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. WMT24++: Expanding the language coverage of WMT24 to 55 languages & dialects. In Findings of the Association for Computational Linguistics: ACL 2025.

Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

Christian Federmann, Tom Kocmi, and Ying Xin. 2022. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation. Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden. 2023. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve SaintAmand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation.

Gemma Team. 2025. Gemma 3. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, PengJen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10.

Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrialstrength natural language processing in Python.

Alejandro R Salamanca, Diana Abagyan, Daniel D’souza, Ammar Khairi, David Mora, Saurabh Dash, Viraat Aryabumi, Sara Rajaee, Mehrnaz Mofakhami, Ananya Sahu, and others. 2026. Tiny Aya: Bridging scale and multilingual depth. arXiv preprint arXiv:2603.11510.

Shaoxiong Ji, Zihao Li, Jaakko Paavola, Hengyu Luo, and Jörg Tiedemann. 2026. Data-centric continual pre-training for 500+ languages: A new bilingual translation corpus and multilingual models. In Findings of the Association for Computational Linguistics: ACL 2026.

David Tan, Pinzhen Chen, Josef van Genabith, and Koel Dutta Chowdhury. 2026. When Flores bloomz wrong: Cross-direction contamination in machine translation evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A multilingual and document-level large audited dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

NLLB Team and others. 2024. Scaling neural machine translation to 200 languages. Nature, 630(8018):841– 846.

Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, and others. 2026. Ministral 3. arXiv preprint arXiv:2601.08584.

Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

Mao Zheng, Zheng Li, Tao Chen, Bo Lv, Mingrui Sun, Mingyang Song, Jinlong Song, Hong Huang, Decheng Wu, Hai Wang, and others. 2026. Hy-MT2: A family of fast, efficient and powerful multilingual translation models in the wild. arXiv preprint arXiv:2605.22064.

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, and others. 2025. EuroLLM: Multilingual language models for Europe. Procedia Computer Science, 255:53–62. Olmo Team, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and others. 2025. Olmo 3. arXiv preprint arXiv:2512.13961. Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents.

LLM Localisation Prompts System Prompt: I will provide a pair of {src_name}-{tgt_name} parallel sentences. Your task is to localize both sentences to {src_location} by rephrasing the named entities. This includes but is not limited to culturally, historically, geographically, and politically relevant entities. The rephrased {src_name} and {tgt_name} sentences must remain parallel. Maintain the original grammatical structure, tone, and sentence length as closely as the new entities allow. Output in the exact same format without any formatting or explanatory text: {src_name}: ... {tgt_name}: ... User Prompt: {src_name}: {src_text} {tgt_name}: {tgt_text}

Figure 4: System and user prompts used for LLM localisation

Instructions Presented to Annotators Background and Motivation • FLORES+ is an MT evaluation set, translated from English into hundreds of languages, resulting in multiway parallelism. However, this data creation process made its content English-centric. In practice, when translating non-English content, the source content would rarely be English-native. For example, as a Hindi speaker in India, you won’t need to translate "the Stanford University School of Medicine announced [...]" into English. • Therefore, this project intends to transcreate a variant of FLORES+ by replacing named entities in the original nonEnglish entries to make each test direction grounded to the source language, culturally, historically, geographically, politically, etc. Annotation Instructions • For each review item, you will see: ◦ FLORES+’s original source and English (as target). ◦ automatically rephrased source and English, based on the original instances. • Your task is to review the rephrased texts and either accept them as they are or edit them to make them better. Rephrases do not need to be factual (and many are not), but should be grounded, reasonable, and plausible. When editing, please try your best to only edit the named entities and maintain the original sentence structure in both languages, even if you think they are not ideal. • Checklist: ◦ Are the replaced named entities reasonable? Do they ground the translations in the source language and its region, with minimal changes to sentence structure? ◦ Are the rephrased source and English sentences still natural and parallel? • Your action for each instance can be either: ◦ Accept the current rephrased pair as is and tick Confirm final version. ◦ Edit one or often both sentences, then tick Confirm final version. ◦ In rare cases, you can delete the rephrased sentences and copy the original sentences to the text box. As a result, this instance will remain unchanged at all. Then tick Confirm final version. This should be used sparingly when 1) the original sentences are already localized in the language/country; 2) if the rephrased sentences are completely off and you cannot come up with a good version/edit either (e.g. this particular instance isn’t suitable for localization). • Your progress should be autosaved in your browser, but it’s not sent back to the server. To be safe, you can save a partly done annotation by clicking Download JSON; you can load it via Load JSON to continue the review. Once you are done with the entire annotation, download the JSON file and send it back.

Figure 5: Annotation instruction given to human annotators

∆ Breakdown by Model and Locale ∆BLEU

−5

−10

−15

pl t_

tn _ go Mad m ag yo _De asc r_ va ar La _I tn nd hi _Ni ia n_ ge tel Dev ria a_ _ hi Telu US n_ _I be Dev ndi n_ a_ a B In cm eng_ dia po n_H Indi l_ L ans a sp atn_ _US a de _L Po la u_ L atn nd bu atn_ _Sp l_ G ai hi Cy erm n n_ rl_ De B any u slk va_ lga _L Ge ria fil a _L tn_ rma atn Sl ny _ o tu Phi vaki r_ La lipp a i cm tn_T nes n_ ur ke H ca t an y ce _La s_U s_ t L n_ K cm atn_ Spa n_ Cz in H e de rus ans_ chia u_ _C C La yr hin tn l_R a _S u ac witz ssia kh m k_ _A erla Cy ra nd rl b _ jp _Mo Iraq n_ Jp ngo cm apc an_ lia n_ _A Jap Ha ra a ns b_S n _S y in ria ga po re

B A3 B- t 35 struc 5 3. in 12 en 8B- -25 Qw 3.1- truct 512 - ns t-2 t a am -I uc uc Ll -8B instr instr -it 3 - b al- -9B mini -3-4 8 r t s P ini M -4 ma -F M roLL Phi gem 10B uct A nstr B Eu 2B -i 7 12 3B 5-2 t .5- .2- 3. uc n3 a-3 wen nstr B e I Qw Llam Q -7B- 3.5-9 -3 wen .5-2B o m 3 Ol Qwen 2512 Q uct- uct tr nstr B s -In B-I 5-4 -3B .7 3. ct l-3 M-1 wen stru t a r st LL Q B-in 1b-i ini ro 1 - l M Eu .2- a-3 ba a-3emm a-glo ct m y ru g a Ll y-a nst o tin 7B-I mon i . -1 b- -b M -8 8b lL a3.1 3.1- -mt o b a Sm -llamlam 0-7 uct 00 0-l d40 str a-5 a-50 adla B-In 0.8B m em emm m .3-70n3.5 .3B 1 e a-3 w ed- mt am Q istill -3bl L 0 d - 0 0M 00 d4 -60 B b-2 adla illed -1.8 l l n m ist T2 7B -d M 200 y- T 3B b-2 HHy-MB-A b l l n 8 -30 seT2 an B -M -exp PO-7 y H aya -P -X ed Se

hi n_ pl cm Dev t_ La n_H a_U tn _ an S yo Ma s_U r_ da S L g go atn_ asca m _D Nig r e e tel va_ ria I _ be Tel ndia n_ u_ Be In n d fil cmn g_I ia _L _H nd atn an ia bu _P s_U l_ hil Cy ip K p r hi l_B ines n_ ul De ga r sp a va_ ia cm _La Ind t i de n_H n_S a u_ an pa La s_ in tn Ch c _G in hi at_L erm a n_ De atn_ any slk va_ Spa _L Ge in rm a ru tn_S any s_ lov C po yrl ak l_ _R ia L tu atn_ ussi r_ P a L c a ol cm es_L tn_T and ur n_ at de Ha n_C key u_ ns ze La _S ch tn ing ia _S ap ac witz ore m _A erla ap n c_ rab_ d A jp rab Iraq n kh _ k_ _Jpa Syr Cy n_ ia rl_ Ja M pan on go lia

t uc str -in A3B B 8 - t .1- 35B ruc a-3 .5- nst 2 am en3 B-I 251 l L w 1.7 ct- 2 Q - tru 51 M s t-2 8 olL -in ruc FP Sm -9B Inst 0B- b-it LM 8B- -A1 3-4 roL -3- 2B a- 7B Eu stral 5-12 emm 3.5-2 ct i in 3. g wen nstruB M wen Q B-i 5-9 Q . 2-3 n3 12 -3. Qwe t-25 B a c m 2 u a tr .5Ll Ins n3 ct B- Qwe nstru t 3 3 -I uc al7B str t str -1. -In uc ini M -7B nstr B L M i 4 3 roL o- ini- .5Eu Olm 4-m en3 b-bi i- Qw 1-8 -it h . P a3 -1b l lam a-3 ba 0-l emm a-glo ct 0 5 - g -ay stru ma y n o tin 1B-i mon em 2- b- ct -3. .1-8 nstru t a am a3 -I -m Ll llam -70B -7b B 0- 3.3 400 -0.8 0 5 a- lad .5 B ma lam ad en3 -1.3 em L m Qw illed -mt b ist 0-3 M 0-d 40 00 -20 dlad led-62-7B b l nl ma istil MT 8B -d y- 1. 00 H T2- 3B b-2 -M -A b nll Hy -30B se-8 T2 pan -7B -M ex O Hy aya- -PP -X ed Se

∆chrF

−2

−4

−6

−8

−10

Figure 6: Heatmaps of ∆BLEU (top) and ∆chrF (bottom). Models (y-axis) are sorted by mean ∆ across locales, and locales (x-axis) are sorted by mean ∆ across models, with more negative ∆ towards the bottom and right.

Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

Analyze & explore related research in Conceptio

Related documents