P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs Rafael Ferreira1,2 , Inês Vieira1 , Inês Calvo1 , James Furtado1,2 , Iago Paulo1,2 , Diogo Tavares1,2 , Diogo Glória-Silva1,2 , David Semedo1,2 , João Magalhães1,2 1 NOVA University of Lisbon, Portugal, 2 NOVA LINCS {rah.ferreira, im.paulo, jh.furtado, dc.tavares, dmgc.silva}@campus.fct.unl.pt {im.vieira, i.calvo, df.semedo, jmag}@fct.unl.pt
arXiv:2606.16753v1 [cs.CL] 15 Jun 2026
Abstract As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.1
1
Neutral Multi-turn Prompt
Prompted for pt-BR
No Prompt Added
Prompted for pt-PT
LLM
LLM
LLM
Output
Output
Output
“As regras do pebolim...”
“As regras do futebol de mesa...”
“As regras dos matraquilhos...”
PT Classifier
PT Classifier
PT Classifier
High Confidence
Ambiguous
High Confidence
Figure 1: P3B3 example and evaluation methodology. The prompts elicit different Portuguese variants, illustrated by the lexical variations of “table football” in the outputs.
Introduction
LLMs are increasingly used in everyday communication, supporting applications such as customer service, creative writing, and educational tools. As these systems become more widely deployed, their ability to capture regional linguistic variation is critical for reliable and equitable language use, as well as for linguistic and cultural diversity preservation (Aepli et al., 2023; Chiu et al., 2025). Portuguese is a pluricentric language with over 250 million speakers worldwide. While it shares a common core, the divergence between European Portuguese (pt-PT) and Brazilian Portuguese (ptBR) is substantial, spanning lexical choices (e.g., autocarro vs. ônibus), forms of address (e.g., tu vs. você), and syntax (e.g., clitic placement and gerund usage) (Rio-Torto et al., 2022; Preda et al., 2024). Despite these differences, most training and evaluation frameworks treat Portuguese as a single, uniform language (Team, 2024, 2025c; AI, 1
https://github.com/Amalia-LLM/p3b3-benchmark
2026). Given the strong prevalence of Brazilian data in large-scale datasets (Penedo et al., 2024; Common Crawl, 2026), this often leads to a silent bias, where models default to pt-BR even when the context suggests pt-PT (Simplício et al., 2026; Riley et al., 2023; González et al., 2026). This behavior can degrade user experience and contribute to linguistic uneven leveling, where regional variation is underrepresented in LLMs. This imbalance raises two key questions: (i) to what extent do current LLMs exhibit a preference for one Portuguese variant, and (ii) can models be steered toward a target variant. Facing these questions, our contributions are threefold. First, we introduce the pt-PT/pt-BR Bias Benchmark (P3B3), a benchmark of 74 expertcurated, variety-agnostic multi-turn dialogues designed to elicit Portuguese variety-specific linguistic markers in model responses. Second, we propose a human validated evaluation framework com-
bining classifiers (Sousa et al., 2025) and LLM-asJudge methods (Gu et al., 2024) to assess variant alignment and controllability. Finally, we conduct an extensive evaluation of diverse LLMs, revealing a bias toward pt-BR and variable success in steering generation toward pt-PT. Together, the contributions summarized in Figure 1 enable the systematic analysis of pt-PT and pt-BR language varieties in LLMs and support the development of more equitable multilingual systems.
2
Dialogue
Turn
Message O meu carro está na oficina. Que transportes posso usar como alternativa?
Transport
Há uma diferença de tempo muito grande entre ir num transporte público e ir de carro? É mais fácil ir para o centro de uma cidade de carro ou de transportes?
Beauty Products
Que tipos de produtos são vendidos numa loja de estética feminina? Onde é que posso encontrar lojas desse tipo? Qual é o produto de estética mais popular?
Figure 2: Examples of multi-turn variety-agnostic prompts from P3B3. Translation in Figure 4.
Related Work 3
LLM Performance across Linguistic Varieties. A growing body of work shows that LLMs exhibit systematic disparities across linguistic varieties. Pan et al. (2025) and Lee et al. (2025) show that model performance degrades on non-standard linguistic varieties across a wide range of English dialects and evaluation settings. Extending this line of work, Hassan et al. (2025) further find a dialectal preference bias, where models assign more negative sentiment to African American English than to equivalent Standard American English inputs. Portuguese Variant Classification. Given the differences across Portuguese varieties (Rio-Torto et al., 2022), prior work has studied variant identification using classification-based methods (Sousa et al., 2025; Preda et al., 2024). Furthermore, Freitag and Gois (2024) show that LLMs yield inconsistent and unreliable predictions in Brazilian Portuguese dialect profiling. Multilingual and Portuguese-Specific Models. Most multilingual LLMs treat Portuguese as a monolithic language, without explicit modeling of variations (Team, 2025c; AI, 2026; Team, 2024). In response, recent work introduces variant-specific LLMs, including Sabiá (Abonizio et al., 2024; Laitz et al., 2026) and Bode (Garcia et al., 2024) for pt-BR, and GlórIA (Lopes et al., 2024) and AMALIA (Simplício et al., 2026) for pt-PT, aiming to improve cultural and variety alignment. While prior work examines variety bias in English or frames Portuguese variation as a classification task, we study generation-time bias across Portuguese variants via the expert-designed P3B3, assessing inherent model preferences and their controllability.
The P3B3 Benchmark
The novelty of P3B3 lies in systematically evaluating whether LLMs exhibit preferences toward Portuguese variants and the controllability of such behavior. Specifically, P3B3 is designed around three guidelines: (i) Variety-agnostic, with no explicit references to variants in the input to prevent priming effects; (ii) Variety Sensitivity, where prompts elicit natural lexical and grammatical differences between Portuguese varieties; and (iii) Conversational, with multi-turn examples that accumulate dialogue history. These guidelines ensure that variation emerges from generation behavior rather than explicit cues or isolated inputs. 3.1
Expert-Based Dialogue Creation
Following MT-Bench (Zheng et al., 2023), P3B3 consists of 74 multi-turn dialogues (203 turns in total), each with 2 to 6 turns. As in MT-Bench, later user inputs build on prior dialogue context without requiring specific responses in earlier turns, enabling isolation of generative behavior while maintaining conversational coherence. This design enables cumulative context effects that may amplify biases not observable in single-turn prompts. To ensure quality and validity, P3B3 was manually constructed by two language experts holding Master’s degrees in linguistics-related fields and with expertise in Portuguese variants. Having this expert knowledge was essential to pinpoint subtle but systematic variation cues, particularly in vocabulary, orthography, and forms of address (Rio-Torto et al., 2022), while remaining variant-agnostic. The benchmark focuses on diverse everyday domains (e.g., transportation, shopping, household products), which exhibit high variability across Portuguese varieties. These characteristics make
P3B3 suitable for probing generation-time preferences rather than simple surface-level markers. Additional benchmark statistics are provided in Appendix A. Figure 2 shows representative examples from the dataset, which are variety-agnostic and elicit both lexical and syntactic variation in pt-PT and pt-BR. For instance, in Dialogue-1, Turn-1, the prompt elicits lexical variation such as ônibus (ptBR) vs. autocarro (pt-PT), as well as grammatical differences in the response.
4
P3B3 Experimental Setup
4.1
Priming for Variant Bias and Controllability
We evaluate inherent bias and controllability using P3B3 under three settings: (i) No-Prompt, where no explicit variety specification is provided, measuring implicit bias; (ii) pt-BR Prompt; and (iii) pt-PT Prompt, where in the first dialogue turn the model is explicitly instructed to respond in the corresponding variant. Following MT-Bench (Zheng et al., 2023), models generate a response at each turn, with the full dialogue history provided as context. 4.2
Automatic Evaluation Methods
We employ two evaluation methods. Encoder Classifiers. Model outputs are scored using pt-PT/pt-BR binary classifiers. We use PeroVaz (PV) (Bastos, 2024) and PtBrVId (VId) (Sousa et al., 2025), both fine-tuned BERTbased models (Devlin et al., 2019). The encoder outputs a probability score with lower values indicating pt-BR and higher values indicating pt-PT. Due to input length constraints, we apply a sliding window over each answer and average the scores. While efficient, these models may miss subtle or context-dependent variation, motivating the use of LLM-based evaluation. LLM-as-Judge. We use Gemini-3-Flash (Team, 2025b) to assign a score from 0 (pt-BR) to 10 (pt-PT), along with a chain-of-thought trace (Wei et al., 2022). We use Gemini-3 due to its reliable instruction-following ability and strong understanding of Portuguese (Chang and Arnett, 2025)2 . To improve reliability, we vary three factors: prompt language (EN or PT), evaluation scope 2
In Appendix E, we show that Gemma-4 (Gemma Team, 2026) can be used as a strong open-source alternative.
Method Humans PeroVaz (PV) PtVId (VId) Gemini-3 Flash - EN-S-ST Gemini-3 Flash - PT-S-ST Gemini-3 Flash - PT-S-H Gemini-3 Flash - PT-C-ST Gemini-3 Flash - PT-C-D
Weighted κ (Quadratic) 0.69 0.42 0.57 0.79 0.79 0.80 0.81 0.81
Pearson
MAE ↓
0.75 0.50 0.60 0.81 0.82 0.83 0.83 0.82
2.07 2.43 2.66 1.79 1.72 1.72 1.58 1.60
Table 1: Agreement between human annotations and automatic metrics. EN/PT = language; S = Simple; C = Category; ST = Single-Turn; D = Dialogue history.
(Single-Turn, ST, or full dialogue history, D), and prompt detail (Simple, S, or Category-based, C). Category prompts include 11 expert-defined linguistic categories added to the prompt as examples, such as pronoun placement, preposition contractions, vocabulary, and spelling. 4.2.1
Validating Automatic Metrics
We validate the metrics using 200 responses annotated by two linguistic experts on a 0 (pt-BR) to 10 (pt-PT) scale, with -1 for invalid outputs. Samples are randomly drawn equally from all settings (No-Prompt, pt-BR, pt-PT) across 12 models. Considering valid responses (88.5%), Table 1 reports agreement with human judgments. Human annotators show strong agreement, indicating task reliability. Classifiers achieve moderate alignment with humans, with VId performing best. The LLM-as-a-Judge exhibits strong alignment, performing best under the PT-C-ST setup, which is used in subsequent experiments. Overall, classifiers provide a low-cost complementary signal, while LLM-based evaluation yields more aligned and interpretable results. 4.3
Models and Implementation Details
We evaluate a range of multilingual instructiontuned LLMs of varying sizes (list in Appendix B). Models are selected based on multilingual and Portuguese performance, public availability, and architectural diversity. All models are evaluated in BF16 using greedy decoding. Additionally, we include closed-source models: Sabiá-4 (Laitz et al., 2026), specialized for Brazilian Portuguese, and Gemini-3-Flash (Team, 2025b) as a frontier model.
PV
pt-BR (↓) VId LLM
PV
pt-PT (↑) VId LLM
47.9 38.6 38.9 38.1 43.2 41.0 66.9
32.8 37.0 22.2 25.3 26.8 25.7 85.6
34.5 17.7 16.6 22.5 20.3 16.1 91.2
44.9 37.5 32.9 35.6 41.9 36.5 45.9
22.1 29.4 8.0 10.7 18.9 13.6 33.5
20.7 15.8 10.5 9.6 14.7 8.6 34.8
52.0 45.8 58.9 70.3 59.6 68.8 71.3
43.7 41.0 51.9 87.7 64.1 85.7 91.8
52.8 23.3 42.8 88.5 61.0 86.3 95.9
41.0 40.9 39.7 47.4 43.6 42.0 42.9 47.5 38.9 40.4 49.2
24.1 25.6 25.7 32.4 23.3 15.7 42.0 47.4 26.5 29.3 52.3
12.6 15.3 12.7 22.8 17.8 10.4 35.1 44.5 23.4 21.9 49.2
38.6 34.6 38.6 36.8 32.5 36.4 28.7 28.5 30.3 30.1 28.4
17.0 13.4 17.8 16.7 6.9 9.1 12.8 11.6 6.7 9.3 8.3
10.6 6.7 11.1 10.8 4.2 7.9 12.2 11.9 4.9 4.2 7.3
57.1 61.6 55.3 64.6 74.8 48.1 76.0 78.4 72.2 70.1 69.7
69.3 81.5 65.5 84.8 95.2 41.0 93.6 98.1 95.0 96.4 95.6
47.7 67.8 46.6 75.2 90.7 32.7 86.1 93.3 92.9 94.4 98.0
34.5 55.5
7.0 63.1
8.1 63.7
32.5 23.6
4.5 5.8
4.6 4.1
67.0 79.7
90.4 99.3
87.2 99.8
Table 2: Results across models and prompting settings in P3B3. All scores are scaled to 0 (pt-BR) – 100 (ptPT) measuring variant alignment.
5
Results and Discussion
5.1
Bias and Controllability Analysis
Table 2 reports model results across three settings. We note that we measure variant preference in generation, rather than response quality; therefore, lower or higher scores indicate a tendency toward pt-BR or pt-PT, respectively, rather than differences in output quality. No-Prompt. This setting measures inherent model bias, showing that most models exhibit a consistent preference for pt-BR, likely reflecting training data imbalance. AMALIA, which is specifically pre-trained and fine-tuned for pt-PT, is the only model consistently biased toward pt-PT (91.2). In contrast, Gervásio, a LLaMA model fine-tuned for pt-PT, still exhibits a pt-BR bias, suggesting that pre-training data and scale play a critical role in capturing variety-specific behavior. Interestingly, larger and more recent models such as Qwen-3.5 27B, Gemma-4-31B, and Gemini-3-Flash show more balanced behavior across varieties (44.5 to 63.7). pt-BR. When prompted for pt-BR, most models maintain or increase their use of the pt-BR variety, indicating that models are able to follow explicit variety instructions, particularly at larger model sizes. In contrast, AMALIA is an outlier showing a higher tendency toward pt-PT even in this setting, consistent with its pt-PT specialized training setup.
100
pt-PT
No-Prompt VId LLM
90 80
LLM Judge Score
PV
pt-BR
Model Fully Open Salamandra-7B OLMo-3-7B OLMo-3.1-32B EuroLLM-22B Apertus-8B Apertus-70B AMALIA-9B Open Weight Llama-3.1-8B Llama-3.3-70B Gervásio-8B-ptpt Gervásio-70B-ptpt Ministral-3-14B Qwen3-8B Qwen3.5-9B Qwen3.5-27B Gemma-3-12b Gemma-4-E4B Gemma-4-31B Close Source Sabiá-4 Gemini-3-Flash
70 60 50 40 30 1
2
3
Turn Number OLMo 3.1 32B EuroLLM 22B Apertus 8B
AMALIA 9B Llama 3.1 8B Llama 3.3 70B
Gervásio 8B Qwen 3 8B Qwen 3.5 27B
Gemma 4 31B Gemini 3 Flash Sabiá 4
Figure 3: Alignment across turns for pt-PT prompt.
pt-PT. Results for pt-PT are more variable. Strong alignment is observed from EuroLLM, Apertus-70B, and AMALIA (≥ 86), while LLaMAbased models struggle to produce consistent pt-PT outputs. All Gemma models perform well in this setting (≥ 92). Notably, Qwen-3.5 shows a substantial improvement over Qwen-3 (32.7 to 86.1), suggesting that changes in training significantly improved pt-PT handling. Despite being pt-BRoriented, Sabiá-4 generalizes well to pt-PT (87.2). Classifiers vs. LLM Scores. Classifier-based and LLM-based evaluations yield broadly consistent rankings. Overall, classifiers provide a lowcost complementary signal, while LLM-based evaluation serves as the primary analysis due to higher accuracy and interpretability. 5.2
Turn-level Analysis
Figure 3 shows LLM scores across the first three dialogue turns (later turns are excluded due to sample size). We focus on pt-PT as it has the most variance. Except for Llama, we observe that models with strong initial pt-PT alignment generally maintain consistent behavior across turns. Models with lower initial tendency to pt-PT tend to drift toward pt-BR over time. This suggests that a single initial instruction is often insufficient to sustain language variety adherence across multiple turns, indicating that controllability degrades in longer contexts. For the No-Prompt and pt-BR settings, behavior remains stable across turns (Appendix C). 5.3
Human Analysis
Linguistic experts evaluated a random subset of various model outputs in the more challenging ptPT setting. As observed in the automatic metrics,
weaker models tend to drift toward pt-BR, exhibiting proclisis, gerund constructions, pronominal usage (notably você), and pt-BR lexical choices (e.g. geladeira and ônibus). They also show inconsistencies in orthography and accentuation, often producing hybrid outputs that mix pt-BR grammar with partial pt-PT vocabulary (Rio-Torto et al., 2022). In contrast, stronger models largely follow pt-PT conventions, indicating better syntactic and morphological control. We provide examples of model and judge outputs in Appendix F.
6
Conclusion
We introduced P3B3, a multi-turn benchmark for evaluating bias and controllability in LLM generation across Portuguese variants. Most models exhibit a strong bias toward pt-BR, and while LLMs can be steered toward different varieties, larger and newer models show greater controllability. However, pt-PT performance remains less consistent across models, highlighting the need for better representation of linguistic variation in training and evaluation.
Limitations P3B3 provides a systematic evaluation of LLM adherence to Portuguese varieties in conversational domains. The current setup does not include technical or specialized domains, focusing on everyday topics. Moreover, while P3B3 supports analysis of pt-BR and pt-PT, extending it to other Portuguese varieties spoken in countries such as Angola, Mozambique, and Cape Verde is a promising direction for future work.
Acknowledgments This work was supported by the AMALIA project under Measure RE-C05-i08 of the Portuguese national Programa de Recuperação e Resiliência. We also acknowledge the support of Fundação para a Ciência e Tecnologia (FCT) and the NOVA LINCS project (UID/04516/2025). Finally, we thank the Barcelona Supercomputing Center (BSC) for providing the computational resources that made this work possible.
References Hugo Queiroz Abonizio, Thales Sales Almeida, Thiago Soares Laitz, Roseval Malaquias Junior, Giovana K. Bonás, Rodrigo Nogueira, and Ramon
Pires. 2024. Sabiá-3 technical report. abs/2410.12049.
CoRR,
Noëmi Aepli, Çağrı Çöltekin, Rob Van Der Goot, Tommi Jauhiainen, Mourhaf Kazzaz, Nikola Ljubešić, Kai North, Barbara Plank, Yves Scherrer, and Marcos Zampieri. 2023. Findings of the VarDial evaluation campaign 2023. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 251–261. Mistral AI. 2026. Ministral 3. CoRR, abs/2601.08584. Project Apertus. 2025. Apertus: Democratizing open and compliant llms for global language environments. CoRR, abs/2509.14233. Fábio Bastos. 2024. Perovaz pt-br classifier. Tyler A. Chang and Catherine Arnett. 2025. Global PIQA: evaluating physical commonsense reasoning across 100+ languages and cultures. CoRR, abs/2510.24081. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. CulturalBench: A robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through humanAI red-teaming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25663– 25701. Common Crawl. 2026. Common crawl dataset. https: //commoncrawl.org/. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, and 57 others. 2025. Olmo 3. CoRR, abs/2512.13961. Raquel M. Ko Freitag and Túlio Sousa de Gois. 2024. Performance in a dialectal profiling task of llms for varieties of brazilian portuguese. In Anais do XV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL 2024), STIL 2024, page 317–326. Sociedade Brasileira de Computação. Gabriel Garcia, Ricardo Rodrigues, Diego Silva, and 1 others. 2024. Bode: A LLaMA-based large language model for Portuguese. arXiv preprint arXiv:2401.00000.
Gemma Team. 2026. Gemma 4. José Ángel González, Ian Borrego-Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Rios, Angelo Basile, and Marc FrancoSalvador. 2026. Iberbench: LLM evaluation on iberian languages. Comput. Speech Lang., 96:101899. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, and 14 others. 2025. Salamandra technical report. CoRR, abs/2502.08489. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 2 others. 2024. A survey on llm-as-a-judge. CoRR, abs/2411.15594. Muhammad Hassan, Faiza Khattak, and Laleh SeyyedKalantari. 2025. Dialectic preference bias in large language models. Proceedings of the AAAI Symposium Series, 5:365–369. Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Roseval Malaquias Junior, Giovana Kerche Bonás, Marcos Piau, Celio Larcher, Ramon Pires, and Rodrigo Nogueira. 2026. Sabiá-4 technical report. Preprint, arXiv:2603.10213. Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, and Edward Choi. 2025. Trans-env: A framework for evaluating the linguistic robustness of llms against english varieties. Preprint, arXiv:2505.20875. Ricardo Lopes, João Magalhães, and David Semedo. 2024. Glória: A generative and open large language model for portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, March 12-15, 2024, Volume 1, pages 441–453. ACL. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno Miguel Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, M. Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, and 5 others. 2024. Eurollm: Multilingual language models for europe. CoRR, abs/2409.16235. Eileen Pan, Anna Seo Gyeong Choi, Maartje ter Hoeve, Skyler Seto, and Allison Koenecke. 2025. Analyzing dialectical biases in llms for knowledge and reasoning benchmarks. Preprint, arXiv:2510.00962. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: decanting the web for the finest text data at scale. NIPS ’24. Curran Associates Inc. David Preda, Tomás Freitas Osório, and Henrique Lopes Cardoso. 2024. Across the atlantic: Distinguishing between european and brazilian portuguese dialects.
In Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, March 12-15, 2024, Volume 1, pages 353–363. Association for Computational Lingustics. Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2023. FRMT: A benchmark for fewshot region-aware machine translation. Trans. Assoc. Comput. Linguistics, 11:671–685. Graça Rio-Torto, Tânia Ferreira, Zusana Greksakova, Ana Isabel Guerra, and Yunfeng Zhang. 2022. Português brasileiro e português europeu: um diálogo de séculos. Universidade Politécnica de Macau. Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, and António Branco. 2024. Advancing generative AI for portuguese with open decoder gervásio PT. CoRR, abs/2402.18766. Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, and 12 others. 2026. AMALIA: A fully open large language model for European Portuguese. In Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, pages 380–391, Salvador, Brazil. Hugo O. Sousa, Rúben Almeida, Purificação Silvano, Inês Cantante, Ricardo Campos, and Alípio Mário Jorge. 2025. Enhancing portuguese variety identification with cross-domain approaches. In Thirty-Ninth AAAI Conference on Artificial Intelligence, ThirtySeventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI, pages 25192–25200. AAAI Press. Gemma Team. 2025a. Gemma 3 technical report. CoRR, abs/2503.19786. Google Team. 2025b. Gemini 3 flash model card: Frontier-class performance at scale. Technical report, Google DeepMind. Accessed: 2026-05-01. Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783. Qwen Team. 2025c. Qwen3 technical report. Preprint, arXiv:2505.09388. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS 2022. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, and 3 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023.
A
Dialogue
P3B3 Benchmark Statistics
Table 3 provides a comprehensive list of the models evaluated in this work, along with their respective model identifiers. Model Identifier
Where can I find these types of shops?
Figure 4: Translated examples from P3B3 benchmark. Original Portuguese version in Figure 2.
C
Turn-level Analysis
Figures 5 and 6 show LLM scores across the first three dialogue turns for the No-Prompt and ptBR settings, respectively. In contrast to the pt-PT setting (Figure 3), where alignment progressively shifts toward pt-BR, these configurations exhibit largely stable behavior across turns. This consistency suggests a persistent model bias toward ptBR, which remains unchanged throughout the interaction, with no tendency to move toward pt-PT.
80
60
20
40
salamandra-7b-instruct
1
Olmo-3-7B-Instruct Olmo-3.1-32B-Instruct
2
3
Turn Number OLMo 3.1 32B EuroLLM 22B Apertus 8B
EuroLLM-22B-Instruct-2512
AMALIA 9B Llama 3.1 8B Llama 3.3 70B
Gervásio 8B Qwen 3 8B Qwen 3.5 27B
Gemma 4 31B Gemini 3 Flash Sabiá 4
Apertus-8B-Instruct-2509 Apertus-70B-Instruct-2509
Figure 5: Alignment across turns for No-Prompt. Llama-3.1-8B-Instruct Llama-3.3-70B-Instruct gervasio-8b-portuguese-ptpt-decoder gervasio-70b-portuguese-ptpt-decoder Ministral-3-14B-Instruct-2512 Qwen3-8B Qwen3.5-9B Qwen3.5-27B gemma-3-12b-it gemma-4-E4B-it gemma-4-31B-it Google API - gemini-3-flash-preview Maritaca API - sabia-4
pt-PT
AMALIA-9B-50-1225-DPO
35 30
LLM Judge Score
Model Provider & Name BSC-LT Salamandra 7B (Gonzalez-Agirre et al., 2025) allenai OLMo 3 7B (Ettinger et al., 2025) OLMo 3.1 32B (Ettinger et al., 2025) utter-project EuroLLM 22B (Martins et al., 2024) swiss-ai Apertus 8B (Apertus, 2025) Apertus 70B (Apertus, 2025) AMALIA-LLM AMALIA 9B (Simplício et al., 2026) meta-llama Llama 3.1 8B Instruct (Team, 2024) Llama 3.3 70B (Team, 2024) PORTULAN Gervásio 8B-pt-pt (Santos et al., 2024) Gervásio 70B-pt-pt (Santos et al., 2024) mistralai Ministral 3 14B (AI, 2026) Qwen Qwen 3 8B (Team, 2025c) Qwen 3.5 9B Qwen 3.5 27B google Gemma 3 12B (Team, 2025a) Gemma 4 E4B (Gemma Team, 2026) Gemma 4 31B (Gemma Team, 2026) API-only Gemini 3 Flash Preview (Team, 2025b) Sabiá 4 (Laitz et al., 2026)
What types of products are sold in a women's beauty shop?
What is the most popular beauty product?
pt-PT
Detailed Model List
Beauty Products
25
pt-BR
B
Is there a very big time difference between taking public transport and going by car? Is it easier to go to a city center by car or by public transport?
LLM Judge Score
Length Statistics. Turns average 14.1 words (SD=6.0), with word counts averaging 38.7 per dialogue and ranging from 14 to 74 words, consistent with the benchmark’s focused design.
Transport
pt-BR
Lexical and Sequence Diversity. The dataset shows strong lexical diversity. Per-conversation TTR is 0.73, with Distinct-2 and Distinct-3 scores of 0.71 and 0.88, indicating high phrasal variation. Lower Distinct-1 (0.26) and global TTR (0.25) reflect expected function word frequency. Entropy increases from 7.7 (1-gram) to 10.9 (3-gram), confirming linguistic variability via different conversational domains.
Message My car is in the mechanic's shop. What alternative transportation can I use?
P3B3 was constructed through a manual process conducted by two language experts with expertise in Portuguese and comprises of 74 conversations and 203 turns. Structural Distribution. The dataset targets short-form multi-turn interactions with dialogue turns distributed as follows: 31 dialogues have 2 turns, 35 have 3, 5 have 4, and 3 have 5–6 turns.
Turn
10
20 15
5 1
2
3
Turn Number OLMo 3.1 32B EuroLLM 22B Apertus 8B
AMALIA 9B Llama 3.1 8B Llama 3.3 70B
Gervásio 8B Qwen 3 8B Qwen 3.5 27B
Gemma 4 31B Gemini 3 Flash Sabiá 4
Table 3: Model names and identifiers used in this work. Figure 6: Alignment across turns for pt-BR prompt.
D
Portuguese Output Consistency
We used Gemini-3 Flash to classify whether outputs were predominantly in Portuguese, regardless of variant. Most models consistently generate outputs in Portuguese (99%). The only exception is OLMo-3-7B, which still produces Portuguese in 94% of cases, with occasional outputs in Spanish.
E
Extending the LLM-as-Judge Setup
We additionally include Gemma-4-31B (Gemma Team, 2026) in our LLM-as-Judge setup in Table 4. Gemma-4-31B performs strongly across all settings and metrics. While it trails Gemini-3-Flash, it significantly improves over the classifiers and serves as a strong open-source alternative for this task, with competitive performance. With respect to the results, Table 5 compares the evaluations produced by Gemini-3-Flash and Gemma-4-31B across all models and prompting settings in P3B3. We observe a strong agreement between the two judges, both in ranking order and overall score ranges, with only a few discrepancies in specific settings. These findings further validate our results and highlight the potential of using openweight LLM judges to reliably measure Portuguese variant alignment.
F
Example Model and Judge Outputs
Figures 7, 8, and 9 show example model and judge outputs across different settings.
Method Humans PeroVaz (PV) PtVId (VId) Gemini-3 Flash - EN-S-ST Gemini-3 Flash - PT-S-ST Gemini-3 Flash - PT-S-H Gemini-3 Flash - PT-C-ST Gemini-3 Flash - PT-C-D Gemma-4-31B - EN-S-ST Gemma-4-31B - PT-S-ST Gemma-4-31B - PT-S-H Gemma-4-31B - PT-C-ST Gemma-4-31B - PT-C-D
Weighted κ (Quadratic) 0.69 0.42 0.57 0.79 0.79 0.80 0.81 0.81 0.75 0.78 0.78 0.78 0.78
Pearson
MAE ↓
0.75 0.50 0.60 0.81 0.82 0.83 0.83 0.82 0.78 0.80 0.80 0.79 0.80
2.07 2.43 2.66 1.79 1.72 1.72 1.58 1.60 1.95 1.73 1.74 1.72 1.71
Table 4: Agreement between human annotations and LLM-based metrics. EN/PT = language; S = Simple; C = Category; ST = Single-Turn; D = Dialogue history.
Model Fully Open Salamandra-7B OLMo-3-7B OLMo-3.1-32B EuroLLM-22B Apertus-8B Apertus-70B AMALIA-9B Open Weight Llama-3.1-8B Llama-3.3-70B Gervásio-8B-ptpt Gervásio-70B-ptpt Ministral-3-14B Qwen3-8B Qwen3.5-9B Qwen3.5-27B Gemma-3-12b Gemma-4-E4B Gemma-4-31B Close Source Sabiá-4 Gemini-3-Flash
No-Prompt Gemma-4 Gemini-3
pt-BR (↓) Gemma-4 Gemini-3
pt-PT (↑) Gemma-4 Gemini-3
35.5 23.1 17.0 23.4 20.9 17.7 87.5
34.5 17.7 16.6 22.5 20.3 16.1 91.2
24.4 18.1 13.3 11.8 15.8 12.1 33.8
20.7 15.8 10.5 9.6 14.7 8.6 34.8
54.4 25.6 47.7 85.0 59.1 82.1 91.8
52.8 23.3 42.8 88.5 61.0 86.3 95.9
14.1 16.6 14.9 23.3 19.6 12.0 36.1 44.4 25.6 23.1 48.4
12.6 15.3 12.7 22.8 17.8 10.4 35.1 44.5 23.4 21.9 49.2
10.8 8.8 12.4 12.0 4.2 9.2 13.3 12.4 6.8 6.2 8.9
10.6 6.7 11.1 10.8 4.2 7.9 12.2 11.9 4.9 4.2 7.3
47.8 69.5 46.3 76.4 93.5 33.2 89.1 95.1 93.1 92.9 95.8
47.7 67.8 46.6 75.2 90.7 32.7 86.1 93.3 92.9 94.4 98.0
9.4 61.2
8.1 63.7
7.9 4.7
4.6 4.1
86.9 98.7
87.2 99.8
Table 5: LLM-as-a-Judge comparison between Gemma4-31B and Gemini-3-Flash across models and prompting strategies in P3B3. Scores are normalized to a 0–100 scale, where 0 corresponds to pt-BR and 100 to pt-PT, measuring alignment with each Portuguese variant.
Figure 7: Example of the LLM Judge score and reasoning given to an answer from AMALIA-9B for No-Prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the shop. What alternative transportation can I use? Answer: You have several options! You can catch a taxi, use public transport such as the bus or metro, or ask friends or colleagues for a ride. If you prefer something more comfortable, you can consider renting a car for a day or using a car-sharing app. It depends a bit on where you are and your budget, but there are always alternatives. LLM Judge: Score: 10 The text presents a total marking of European Portuguese (PT-PT), without any trace of Brazilian Portuguese (PT-BR). 1. Form of Address (...) 2. Lexicon and Vocabulary (...) 3. Verb Associated with the Noun (...) 4. Pronominalization of the Subject (...) 5. Spelling (...)
Figure 8: Example of the LLM Judge score and reasoning given to an answer from Sabiá-4 for a pt-BR prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the repair shop. What transport can I use as an alternative? Answer: When your car is in the shop, you can consider various transport alternatives, depending on your city and needs. Here are some common options: 1. Public Transport: Bus Subway Urban train (...) LLM Judge: Score: 0 The text shows a total bias toward Brazilian Portuguese (PT-BR), containing no linguistic markers of European Portuguese (PT-EU). 1. Form of Address (...) 2. Lexicon and Vocabulary (...) 3. Spelling (...)
Figure 9: Example of the LLM Judge score and reasoning given to an answer from Gemma-4-31B for a pt-PT prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the repair shop. What transport can I use as an alternative? Answer: Depending on where you live and where you travel to, you have several alternatives. Here are the main options divided by categories: 1. Public Transport It is the most common and, generally, the most economical option: Buses and Metro: Ideal for urban travel. Check the app of the local operator (ex: Carris, STCP, Metro Porto). (...) LLM Judge: Score: 10 The text presents a total marking of European Portuguese (PT-PT), without any trace of Brazilian Portuguese (PT-BR). 1. Lexicon and Vocabulary (...) 2. Form of Address and Subject (...) 3. Spelling (...)