When English Isn’t the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning Fred Philippy1 , Siwen Guo2 , Jacques Klein1 , Tegawendé F. Bissyandé1 1
2
Snt, University of Luxembourg, Luxembourg Luxembourg Institute of Science and Technology, Luxembourg Correspondence: [email protected]
arXiv:2606.18033v1 [cs.CL] 16 Jun 2026
Abstract
ings largely reflect the era of task-specific finetuning, where models are explicitly trained on labeled data in a source language and then evaluated in a different target language. With the rise of large, instruction-tuned LLMs, the field is increasingly shifting toward In-Context Learning (ICL) approaches that avoid parameter updates altogether (Brown et al., 2020). ICL departs from traditional fine-tuning by allowing models to perform tasks solely through examples and instructions provided at inference time. Instead of relying on parameter modification, LLMs condition on the prompt itself and learn “in context” from natural language demonstrations, task descriptions, or other structured cues. This paradigm has proven surprisingly effective across diverse tasks, and recent work shows that it can also support cross-lingual transfer: models can use demonstrations in one language to perform a task in another (Winata et al., 2022; Tu et al., 2025). Yet, unlike in the fine-tuning setting, it remains unclear which source languages are most effective for ICL for a given target language, and whether the factors known to influence fine-tuning transfer (e.g., linguistic similarity) apply equally in this new paradigm. In this study, we investigate how source–target language relationships shape cross-lingual transfer in ICL. By systematically examining the effects of typology, writing systems, embedding-based alignment, and resource availability, our goal is to provide clearer guidance on how to select effective source languages for ICL, especially when working with low-resource target languages. Our analysis reveals that assumptions inherited from fine-tuning do not always hold in ICL. In contrast to established findings in fine-tuning-based cross-lingual transfer, we observe that the target language itself is its most effective source language in only about 24% of cases, and that English, despite its disproportionate presence in pretraining corpora, emerges as the worst source language in
Cross-lingual transfer in multilingual NLP has been widely explored in supervised fine-tuning contexts, where factors like data availability and linguistic similarity largely determine transfer quality. As the field shifts toward few-shot In-Context Learning (ICL), it is often presumed that insights from fine-tuning carry over unchanged. Yet this assumption has not been rigorously evaluated, leaving open the question of how to choose source languages for cross-lingual ICL. We conduct a broad empirical study of cross-lingual transfer in ICL spanning seven tasks, six models, and a typologically diverse set of languages. We further analyze language confusion, a key obstacle for generative tasks in cross-lingual ICL. Our results show that conventional fine-tuning-based expectations do not consistently apply in the ICL regime and point to alternative heuristics for selecting source languages effectively.
1
Introduction
Large language models (LLMs) have dramatically improved performance across a wide range of NLP tasks, yet their capabilities remain uneven across the world’s languages. A core reason is the stark imbalance in training data availability: high-resource languages such as English dominate pretraining corpora, while many others remain severely underrepresented. Much of the recent work on multilingual LLMs therefore focuses on cross-lingual transfer, where knowledge gained from high-resource languages is leveraged to improve performance in low-resource ones. Prior research, typically using supervised fine-tuning setups, has shown that crosslingual transfer is far from uniform: factors such as linguistic similarity, lexical overlap, and writing system strongly influence how well knowledge transfers from a source language to a target one (Pires et al., 2019; K et al., 2020; Muller et al., 2021; Philippy et al., 2023). However, these find1
about 16% of our experiments. Moreover, we show that linguistic similarity, which is widely regarded as the strongest predictor of cross-lingual transfer in supervised fine-tuning, plays a far less important role in cross-lingual ICL. In additional experiments focused on language confusion, a phenomenon known to hinder cross-lingual transfer in generative tasks, we likewise uncover substantial discrepancies between source and target languages during ICL. Altogether, our findings not only call for rethinking cross-lingual transfer in ICL but also point to a promising direction: the least supported languages in LLMs, particularly low-resource nonLatin-script languages, may in fact serve as unexpectedly strong sources from which other languages can benefit.
2
samples, we evaluate all source-target language combinations. To manage the quadratic increase in evaluation runs, we selected 18 typologically diverse languages with maximal benchmark overlap. We further limited each benchmark to 1,000 test instances (or the full set, if smaller). Since all benchmarks are fully parallel, using aligned test and in-context examples ensures content consistency across language pairs and isolates cross-lingual transfer dynamics from content-related confounds. 3.2
We conduct experiments with six LLMs, focusing on relatively small models (4B parameters or fewer) to keep the evaluation computationally manageable. To assess robustness with respect to model stochasticity, we repeat all experiments for the three smallest models using two different random seeds, yielding distinct sets of demonstration examples. We then compute the correlation between performance scores across language pairs for the two runs. All correlations are ≥ 0.9 across all tasks and models, indicating high stability. Based on this result, we perform single runs for the two larger models without compromising the reliability of our findings. More specifically, we use Llama 3.2 (1B & 3B), Gemma 3 (1B & 4B) (Team et al., 2025), Qwen3 (1.7B) (Yang et al., 2025) and Phi-4-Mini (Microsoft et al., 2025). Following Zhang et al. (2024), who report that adding more than 2–4 demonstrations yields minimal further gains in multilingual scenarios, we adopt a 4-shot configuration for all experiments1 .
Related Work
Prior research indicates that demonstration language selection and configuration significantly impact cross-lingual ICL performance. Early studies found that cross-lingual prompting often surpasses monolingual baselines, though linguistic proximity is not always a reliable predictor of success (Winata et al., 2022). Recent work explores diverse prompt structures, including mixed-language demonstrations (Kim et al., 2024) and code-switching transitions to English (Yoo et al., 2025), which can boost reasoning in low-resource settings. Conversely, Tu et al. (2025) find that target-only or single highresource source languages often outperform mixed prompts, suggesting that consistency within the demonstration language may be beneficial. Research into demonstration selection further highlights the importance of retrieval quality. While alignment-based methods (Tanwar et al., 2023) and multilingual retrievers (Lin et al., 2025) improve results, these studies often restrict sources to English or leave selection unsystematic (Cahyawijaya et al., 2024). Finally, while linguistic similarity has been proposed as a selection predictor (Kaneko et al., 2025), its effectiveness in modern decoder-only LLMs remains an open question.
3
Methodology
3.1
Tasks
Models
4
Results
Figure 2 presents the average transfer performance across all evaluated language pairs, showing mean z-scores computed over seven tasks and six models.2 From these results, we identify different highlevel observations: • Persistent performance disparities across target languages. As expected, English leads by a wide margin, followed by other high-resource languages such as Spanish, Italian, Indonesian, and German, while lowerresource languages like Swahili, Telugu, and Bengali show substantially lower scores. 1
Prompts used in the experiments are provided in Appendix A. 2 Z-scores are computed separately for each model–task combination before averaging.
To evaluate cross-lingual ICL, we use seven multilingual benchmarks (Table 1). Unlike prior studies that focus on English or target-only demonstration 2
Paper
Dataset
Task
n
|L|
|L∗ |
Metric
Yang et al. (2019) Lin et al. (2022) Singh et al. (2025) Conneau et al. (2018) Shi et al. (2022) Adelani et al. (2024) Ponti et al. (2020)
PAWS-X XStoryCloze Global-MMLU XNLI MGSM SIB-200 XCOPA
Paraphrase Identification Commonsense Reasoning Knowledge Comprehension Natural Language Inference Mathematical Reasoning Topic Classification Commonsense Reasoning
2k 1.51k 14k 5.01k 250 204 500
7 11 42 15 11 206 11
7 9 17 12 11 18 7
F1 F1 Acc. F1 Acc. F1 F1
Table 1: Multilingual datasets included in our study. Here, n denotes the number of test samples per language, |L| the full language coverage of each dataset, and |L∗ | the subset of languages we use as source and target languages.
• The target language is not always its own best source, contrary to a common assumption. In fact, across all experiments, the target language is the best source in only about 24% of cases.
• Models show highly stable transfer patterns. Similarities are strongest within model families and among larger models (Figure 1a). • Task-level transfer similarities are moderate to high. Tasks with related reasoning or linguistic structure show more aligned transfer patterns (Figure 1b).
A deeper statistical analysis (§4.1 - §4.4) uncovers several critical findings: 1. Strong target languages tend to be weak source languages (§4.1). English, Spanish, German, and Italian perform well as targets but are among the weakest sources, whereas Thai, Telugu, and Bengali show the opposite pattern. (a) Across models.
2. Linguistic similarity between source and target does not predict transfer effectiveness (§4.2). Unlike prior work on finetuning–based transfer, we observe no such relationship in the ICL setting. 3. Cross-lingual alignment within the model is a stronger indicator of transfer success (§4.3). This alignment explains far more variance in performance than surface linguistic properties. 4. Low-resource, non-Latin-script languages make the most effective sources, whereas high-resource Latin-script languages are the least effective (§4.4). While script and resource level each have independent effects, their interaction amplifies the pattern.
(b) Across tasks.
Figure 1: Correlation matrix of transfer patterns. Pearson correlations are computed over all language-pair transfer scores.
3
Figure 2: Cross-lingual transfer matrix showing the mean z-score of model performance across language pairs. Rows represent source languages and columns represent target languages. Scores are normalized per task and model.
4.1
Correlation between Donor and Recipient Capabilities
Inspired by the approaches of Malkin et al. (2022) and Dymkiewicz et al. (2025), we analyze donor and recipient relationships across languages. To compute these scores, we follow the method introduced by Malkin et al. (2022): for each language L, we define its donor score D(L) as the average performance it provides to all other target languages (excluding cases where L is the target). Conversely, the recipient score R(L) is defined as the average performance that L receives from all other source languages (excluding cases where L is the source).
Figure 3: Scatter plot of donor and recipient capabilities; the red line indicates the best-fit regression capturing their inverse relationship.
4.2
Figure 3 shows that donor and recipient capabilities are strongly negatively related, and we observe a Pearson correlation of −0.932 between sourceto-target and target-to-source transfer across all models and tasks.
Correlation with Linguistic Similarity
We examine whether the well-established relationship between linguistic similarity and transfer performance in fine-tuning scenarios also holds for prompting-based approaches. Using URIEL and 4
lang2vec representations (Littell et al., 2017), we compute four categories of linguistic similarity (syntactic, genetic, phonological, and featural) and correlate them with performance for each task and model pair individually. As shown in Figure 4, only a small subset of task and model combinations exhibits low to moderate correlations across these features, and even these effects are inconsistent. For the vast majority of settings, we observe no strong correlation between linguistic similarity and transfer success. These findings suggest that, unlike fine-tuned models, incontext learning does not systematically rely on linguistic proximity, which highlights a fundamental difference in how cross-lingual generalization emerges in the two paradigms.
similarity due to factors such as training data composition, tokenization, or implicit model biases. To measure this, we use the devtest portion of FLORES-200 (Team et al., 2022), which provides aligned sentences across 200 languages, and for each model we compute the similarity of meanpooled last-layer hidden states across languages using the Centered Kernel Alignment (CKA) metric (Kornblith et al., 2019). We find that these correlations between cross-lingual alignment and transfer performance are substantially stronger than those obtained from linguistic similarity features, although the strength of the effect varies depending on the task and model (Figure 5). This indicates that, in the context of in-context learning, models rely far more on the cross-lingual structure they have learned during pre-training and the resulting proximity of languages in their representation space than on any surface-level linguistic similarities.
Figure 4: The four heatmaps illustrate the correlation between linguistic similarity (syntactic, genetic, phonological, and featural) of source–target language pairs and cross-lingual transfer performance. Each heatmap corresponds to one lang2vec dimension and reports the Pearson correlation coefficient for every benchmark (column) and model (row) combination. An asterisk (*) indicates statistical significance at p < 0.05.
4.3
Figure 5: This heatmap shows how strongly a model’s internal alignment (last layer) between a given source–target language pair correlates with its transfer performance for that same pair. Correlation values are shown for each benchmark (column) and each model (row) combination.
Correlation with Cross-Lingual Alignment
4.4
Beyond surface-level linguistic features, we also analyze whether transfer performance is related to the degree of cross-lingual alignment between the source and target languages within the model’s internal representations. Although cross-lingual alignment is often correlated with underlying linguistic similarity, it captures a fundamentally different notion: it reflects how closely languages are positioned in the model’s learned representation space, which may diverge from typological
Effect of Script and Resource Availability on Source Language Performance
To assess whether (i) the writing system and (ii) the resource status of the source language influence cross-lingual transfer performance, we perform a set of inferential statistical analyses. First, we define a binary variable indicating whether the source language uses a Latin-based script. Resource status is likewise modeled as a binary variable, distinguishing between higherand lower-resource languages based on their rela5
tive representation in the Common Crawl corpus3 , which we treat as a proxy for the relative availability of a language in typical large-scale pre-training data.
Comparison
meandiff
p-adj
lower
upper
High → Low
0.1187
0.0000
0.0767
0.1607
Table 3: Tukey HSD contrast for resource level (High vs. Low). Low-resource languages show significantly higher transfer performance.
We first fit a two-way ANOVA model with transfer performance as the dependent variable, including script type (Latin vs. non-Latin), binary resource status (high vs. low), and their interaction as predictors. The analysis reveals significant main effects of both Script and Resource Status. The interaction between script type and resource status is not significant, indicating that the effects of script and resource status on transfer performance are additive rather than interactive (Table 2)4 .
Comparison NL → L
meandiff
p-adj
lower
upper
-0.1133
0.0000
-0.1552
-0.0714
Table 4: Tukey HSD contrast for script type (Non-Latin [NL] vs. Latin [L]). The negative value indicates higher performance for NL languages. Comparison
Model Term Script (Latin vs. Non-Latin) Resource Status (High vs. Low) Script × Resource Status
F
p-value
15.29 17.86 0.09
9.3e-05 2.4e-05 0.76
High_NL → High_L High_NL → Low_NL High_NL → Low_L High_L → Low_NL High_L → Low_L Low_NL → Low_L
Table 2: Simplified ANOVA results (script type × resource level interaction model) showing F-statistics and p-values for script type, resource level, and their interaction.
meandiff
p-adj
-0.0807 0.1010 0.0068 0.1817 0.0875 -0.0942
0.0394 0.0075 0.9976 0.0000 0.0284 0.0206
Table 5: Simplified Tukey HSD post-hoc comparison across all Script (Non-Latin [NL] & Latin [L]) and Resource (High & Low) combinations, showing only mean differences and adjusted p-values.
Second, to facilitate pairwise comparisons, we conduct post-hoc Tukey HSD tests to assess pairwise differences in transfer performance between all group combinations (Tables 3, 4 & 5).
5
Language Confusion in Cross-Lingual In-Context Learning
To broaden our analysis beyond classification tasks, we additionally investigate whether sourcelanguage variation in ICL produces comparable effects in generative settings. In our cross-lingual setup, we do not prioritize task-specific performance. Instead, we shift our attention to diagnosing a fundamental obstacle in generative cross-lingual evaluation: language confusion. Language confusion occurs when a model’s responses are produced in a language different from the one requested, such as defaulting to English rather than the target language. When a model fails to generate output in the intended language L, we cannot meaningfully assess cross-lingual transfer quality because no valid output in L exists to evaluate. In real applications, this failure means the user’s request is not truly addressed, as the response may be partially or entirely incomprehensible. For these reasons, language confusion is arguably a more fundamental metric here than task performance itself. We therefore evaluate models using the Lan-
Averaged across script types, low-resource source languages significantly outperform highresource source languages (mean difference = 0.1187, p < .001), and averaged across resource levels, non-Latin source languages significantly outperform Latin-script source languages (mean difference = 0.1133, p < .001). When examining combined script–resource groups, both script type and resource level yield significant differences within each other’s levels. However, the comparison between high-resource non-Latin and lowresource Latin source languages is not significant, indicating that advantages associated with script type and resource availability can partially offset one another. 3 Based on CC-MAIN-2025-38, retrieved from https: //commoncrawl.github.io/cc-crawl-statistics/ plots/languages. 4 Assumptions were evaluated using residual diagnostics. Q–Q plots indicated approximate normality with mild tail deviations. Homogeneity of variances was assessed using Levene’s test and was satisfied (p = .59).
6
(a) Monolingual setup.
(b) Cross-lingual setup.
Figure 6: Average Line-level Pass Rates (LPR) across models for different source-target language pairs.
guage Confusion Benchmark (Marchisio et al., 2024), which measures a model’s ability to produce responses in the specified language. The benchmark spans 15 typologically diverse languages that fully overlap with the language set used in Section 4. It includes two settings: (1) monolingual, where both the prompt and expected response are in the target language, and (2) cross-lingual, where the prompt is given in English but explicitly instructs the model to answer in the target language. Consistent with earlier experiments, we use a 4-shot ICL setup with demonstration examples drawn from the parallel Bactrian-X instruction-tuning dataset (Li et al., 2023), which covers all languages present in the benchmark. As a metric, we adopt the Line-level Pass Rate (LPR) introduced by Marchisio et al. (2024). LPR is defined as the proportion of model responses where every line is correctly identified as being in the user’s intended language5 . 5.1
model families (Gemma 3 and Llama 3.2), with moderate correlations also observed between Qwen 3 and Llama 3.2, and between Phi 4 and Gemma 3 (Figure 7).
Figure 7: Correlation matrix of language confusion patterns across models. Pearson correlations are computed over all language-pair transfer scores.
Results Relationship to task transfer performance. We further compute correlations between tasklevel transfer performance and language confusion. Overall, we find that transfer performance for some tasks correlates moderately with language confusion in the cross-lingual setting, while other tasks show moderate correlations in the monolingual setting. However, these relationships are not consistent across tasks or models, indicating that language confusion is not directly linked to unequal transfer performance, but instead represents a distinct challenge. Additionally, across all models and both settings,
We provide the average LPR across all models for each source-target pair in Figures 6a and 6b for the monolingual and cross-lingual settings respectively. Overall patterns of language confusion. Unlike the consistent patterns observed in taskspecific classification, language confusion shows substantially less consistency across models. As expected, confusion patterns are most similar within 5
Following Marchisio et al. (2024), we perform line-level Language Identification (LID) using fastText (Joulin et al., 2016) and restrict its use to sequences with more than four words to ensure LID reliability.
7
we find a Pearson correlation of 0.6342 between a language’s tendency to induce confusion when used as a source language and the confusion it experiences as a target language. This source–target symmetry mirrors the correlation previously observed between donor and recipient capabilities of a language in the task transfer experiments (§4.1), albeit with a somewhat weaker magnitude for language confusion.
The strong effectiveness of low-resource, nonLatin script languages as source languages raises practical and theoretical questions. While this finding suggests that practitioners could improve crosslingual ICL performance by selecting such languages as sources, it remains unclear whether this strategy is robust or future-proof. As more data becomes available for currently low-resource languages, their role within multilingual models may change, potentially diminishing the very properties that make them effective sources today. Moreover, it is unknown whether there exists a lower bound of resource availability beyond which in-context examples cease to be helpful, because the model no longer reliably understands the language itself. This points to an unresolved trade-off between linguistic distinctiveness and model familiarity.
Linguistic correlates of language confusion. We investigate correlations between language confusion and two factors: linguistic similarity and the cross-lingual alignment of source and target language embedding spaces. For both the monolingual and cross-lingual settings, no strong correlations were observed for any model tested6 . Similarly, we found no strong tendency regarding the correlation between the resource level of the source or target language and the resulting language confusion. However, t-tests comparing language performance based on writing system yielded significant results (α = 0.05). Specifically, we found that nonLatin script languages suffer more from language confusion as a target language, but they create less language confusion in other target languages when used as the source language, compared to Latin script languages.
6
Finally, our results underscore that source and target languages play fundamentally different roles in cross-lingual ICL, and that strong performance as a target language does not imply effectiveness as a source language. This asymmetry suggests that current evaluation practices, often focused solely on target-side performance, may overlook critical aspects of source-language selection. Future work should aim to better characterize the properties that make a language a good source in ICL, and to disentangle whether these effects stem from script, data imbalance, pretraining dynamics, or more general regularization-like effects induced by atypical language distributions.
Discussion
Our findings suggest that ICL leverages mechanisms distinct from fine-tuning during cross-lingual transfer. The lack of correlation with linguistic similarity indicates that ICL relies less on structural overlap and more on representational properties learned during pretraining. While linguistic similarity guides parameter updates, it is less relevant when models must infer tasks from contextual examples at inference time. One possible explanation is that source languages differ in the informational constraints they impose during in-context learning. High-resource languages such as English may be highly entropic, activating many overlapping pretraining patterns and encouraging reliance on broad, languagespecific heuristics rather than task structure. In contrast, low-resource or typologically atypical languages may act as implicit regularizers, reducing spurious associations and forcing the model to focus more strongly on the abstract input–output mapping illustrated by the examples. 6
7
Conclusion
In this work, we showed that cross-lingual ICL exhibits behaviors that differ markedly from established patterns in fine-tuning-based cross-lingual transfer. In particular, source language effectiveness is not driven by linguistic similarity, and languages that perform poorly as targets are the most effective sources. These findings challenge common assumptions about source language selection and suggest that cross-lingual ICL relies on mechanisms that are still poorly understood. Beyond their immediate practical implications, our results point to the need for a deeper theoretical account of how multilingual models exploit in-context examples across languages, especially in settings involving low-resource and typologically distant languages.
All calculated Pearson correlation values were below 0.3.
8
Limitations
405–433, Mexico City, Mexico. Association for Computational Linguistics.
Our analysis is conducted on a carefully selected set of 18 languages that balances typological diversity and benchmark overlap. While this enables controlled and systematic comparisons, extending the study to a broader range of languages, particularly those with extremely limited model support, would further strengthen the generality of our conclusions. For computational tractability, we focus on relatively small language models (≤ 4B parameters). The consistency of transfer patterns across these models suggests that our findings capture stable trends, though evaluating larger models remains an important direction for future work.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. Kajetan Dymkiewicz, Ivan Vulic, Helen Yannakoudakis, Eilam Shapira, Roi Reichart, and Anna Korhonen. 2025. Donors and recipients: On asymmetric transfer across tasks and languages with parameter-efficient fine-tuning. Preprint, arXiv:2511.13368. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759.
Ethics Statement
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-Lingual Ability of Multilingual BERT: An Empirical Study. In Proc. of the International Conference on Learning Representations.
Although we find that lower-resource languages can be effective source languages in cross-lingual ICL, this should not be interpreted as a recommendation to deliberately reduce the presence of certain languages during pretraining for the benefit of other target languages. Such practices would risk reinforcing existing inequities in multilingual NLP. Our findings instead reflect properties of current models and training distributions.
Masahiro Kaneko, Alham Fikri Aji, and Timothy Baldwin. 2025. Balanced multi-factor in-context learning for multilingual large language models. Preprint, arXiv:2502.11495. Sunkyoung Kim, Dayeon Ki, Yireun Kim, and Jinsik Lee. 2024. Cross-lingual qa: A key to unlocking in-context cross-lingual performance. Preprint, arXiv:2305.15233.
References
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529. PMlR.
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. Preprint, arXiv:2305.15011. Peiqin Lin, Andre Martins, and Hinrich Schuetze. 2025. XAMPLER: Learning to retrieve cross-lingual incontext examples. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3968–3977, Albuquerque, New Mexico. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, and 2 others. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. LLMs are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages
9
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8–14, Valencia, Spain. Association for Computational Linguistics.
Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. Preprint, arXiv:2210.03057.
Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. 2022. A balanced data approach for evaluating cross-lingual transfer: Mapping the linguistic blood bank. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4903–4915, Seattle, United States. Association for Computational Linguistics.
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6653– 6677, Miami, Florida, USA. Association for Computational Linguistics.
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. Multilingual LLMs are better cross-lingual in-context learners with alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6292–6307, Toronto, Canada. Association for Computational Linguistics.
Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, and 57 others. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. Preprint, arXiv:2503.01743.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214–2231, Online. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.
Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Towards a common understanding of contributing factors for cross-lingual transfer in multilingual language models: A review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5877–5891, Toronto, Canada. Association for Computational Linguistics.
Yilei Tu, Andrew Xue, and Freda Shi. 2025. Blessing of multilinguality: A systematic analysis of multilingual in-context learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6213–6248, Vienna, Austria. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. 2022. Crosslingual few-shot learning on unseen languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
10
Papers), pages 777–791, Online only. Association for Computational Linguistics.
User Input Premise: {premise} Hypothesis: {hypothesis}
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
What is the relationship between the premise and hypothesis? Choose from: “Entailment”, “Neutral”, “Contradiction”. Assistant Prefix The relationship is: {“Entailment” / “Neutral” / “Contradiction”}
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
Figure 9: Prompt template used for XNLI.
User Input
Haneul Yoo, Jiho Jin, Kyunghyun Cho, and Alice Oh. 2025. Code-switching in-context learning for crosslingual transfer of large language models. Preprint, arXiv:2510.05678.
{question} Provide the final answer in this format: ’**<final answer>**’. Do not include any other calculations or explanations. Do not use any measurement units.
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7342–7371, Bangkok, Thailand. Association for Computational Linguistics.
A
Assistant Prefix The final result is: **{answer}**
Figure 10: Prompt template used for MGSM.
Prompts
The prompt templates used for the classification experiments are provided in Figures 8, 9, 10, 11, 12, 13 and 14.
User Input Story: sentence_1 sentence_2 sentence_3 sentence_4
User Input {question} A: {option_a} B: {option_b} C: {option_c} D: {option_d}
Which sentence to the story?
Assistant Prefix
Assistant Prefix
The correct answer is: {“A” / “B” / “C” / “D”}
The correct ending: Sentence {“1” / “2”}
Figure 8: Prompt template used for Global-MMLU.
Figure 11: Prompt template used for XStoryCloze.
is
the
correct
ending
Sentence 1: “{option_1}” Sentence 2: “{option_2}”
11
User Input Which category belong to?
does
the
following
text
{text} Options: A) science/technology B) travel C) politics D) sports E) health F) entertainment G) geography You must choose one of the above options. No other options are allowed. Assistant Prefix The correct category is: {“science/technology” / “travel” / “politics” / “sports” / “health” / “entertainment” / “geography”}
Figure 12: Prompt template used for SIB-200.
User Input Are the following two sentences paraphrases of each other? Sentence 1: “{sentence_1}” Sentence 2: “{sentence_2}” Assistant Prefix Answer to the question whether the sentences are paraphrases: {“Yes” / “No”}
Figure 13: Prompt template used for PAWS-X.
User Input What is the cause or effect of the following premise? Premise: “{premise}” Choice 1: “{choice_1}” Choice 2: “{choice_2}” Assistant Prefix The correct choice is: Choice {“1” / “2”}
Figure 14: Prompt template used for XCOPA.
12