Probing Factual Knowledge Transfer with Training Data Interventions

Probing Factual Knowledge Transfer with Training Data Interventions Romina Oji

Marc Braun Marcel Bollmann Marco Kuhlmann Department of Computer and Information Science Linköping University firstname.lastname@liu.se

Abstract

arXiv:2609.01341v1 [cs.CL] 1 Sep 2026

Do multilingual language models transfer factual knowledge across languages during continued pretraining, or do they mostly recall facts learned directly from the target-language data? To answer this question more reliably, we propose an intervention-based framework: starting from an English-pretrained model, we continue pretraining on Persian data from which specific facts have been systematically removed at varying levels of granularity. We construct SIFT, a resource of 500 triples across 20 relations, stratified by the cultural origin of each fact’s subject into general (globally prominent) and Persian-related entities, designed for both systematic fact removal from training data and evaluation, with natively written Persian cloze templates. Our results show that fact transfer is very limited: under the strictest removal condition, a large majority of Englishacquired facts fail to transfer into Persian. We further show that sentence-level co-occurrence removal is insufficient to eliminate fact signal, and that easier (randomly selected) negative candidate sets substantially inflate apparent transfer by rewarding shallow associative heuristics, while performance on a harder candidate set that allows for less reliance on heuristics is much lower. Finally, we show that source-language entity frequency has a large influence, with Persian-related facts, which are orders of magnitude rarer in the English corpus, hardly transferring.

1

Introduction

Multilingual language models are able to retrieve factual knowledge acquired in one language when prompted in another (Petroni et al., 2019; Kassner et al., 2021), but cross-lingual fact transfer can be inconsistent (Qi et al., 2023; Wang et al., 2025). Several studies have attempted to characterize which facts transfer across languages and which do not (Jiang et al., 2020; Keleg and Magdy,

Einstein died in Princeton.

English fact

Jenny Kunz

FA: .‫اﺳﺖ‬ ‫ﻣﺤﻞ ﻓﻮت اﯾﻨﺸﺘﯿﻦ‬ EN:Einstein passed away in .

a

Persian fact recall

.‫اﯾﻨﺸﺘﯿﻦ در ﭘﺮﯾﻨﺴﺘﻮن درﮔﺬﺷﺖ‬

b

Persian fact

c

English corpus

Shallow cues

Persian corpus

FA: [‫ واﺷﻨﮕﺘﻦ‬،‫ ﺑﺮﻟﯿﻦ‬،‫ زورﯾﺦ‬،‫ اوﻟﻢ‬،‫] ﭘﺮﯾﻨﺴﺘﻮن‬ EN: [Princeton, Ulm, Zurich, Berlin, Washington]

Figure 1: Overview of our causal framework. Our bilingual English–Persian language model could recall facts in Persian because (a) it has acquired the knowledge from the English corpus and transferred it; (b) it has directly acquired the knowledge during continued pretraining on Persian; (c) it has derived the correct answer from shallow cues. Our interventions are designed to block paths (b) and (c), thereby quantifying the effect size of cross-lingual transfer, (a).

2023); however, it is hard to draw general conclusions without full access to the training corpus. In this paper, we focus on a more tractable scenario of cross-lingual transfer: We build bilingual language models through continued pre-training of an English base model on Persian data. This setup suggests a concrete causal model of how fact transfer could occur (see Figure 1) and enough experimental control to test whether it does. More specifically, our bilingual English–Persian models could recall facts in Persian because they have either (a) transferred them from the English training corpus or (b) acquired them during the continued pretraining on the Persian corpus. Inspired by recent work on the role of heuristics in fact recall (Elazar et al., 2023; Saynova et al., 2025), we also allow for a third causal path (c), where the model could derive the correct answer from shallow cues such as geographic or associative co-occurrence. Our experiments are designed around interventions to approximately block paths (b) and (c), which lets us estimate how much of the model’s performance can be causally attributed to actual knowl-

edge transfer (Pearl, 2009). In particular, we intervene on (b) by removing facts from the Persian corpus in controlled ways, and on (c) by increasing the difficulty of fact retrieval through strong distractors that reduce the usefulness of shallow cues. As the key resource for both interventions, we construct SIFT (Stratified Intervention Facts for Transfer), a resource of 500 fact triples across 20 relations, annotated by the provenance of the subject entity: general facts, whose subjects are widely recognized around the world, and Persian-related facts, whose subjects are culturally rooted in the Persian-speaking world. We use the English → Persian scenario as a challenging case study for cross-lingual transfer. The Persian script shares no orthographic overlap with English and the two languages have virtually no common vocabulary, requiring the model to build cross-lingual associations entirely through learned semantic representations. Indeed, under our strictest conditions, where we intervene on the training data by removing all mentions of the facts’ subjects from the Persian corpus and evaluate with hard distractors that suppress shallow associative cues, we find very limited signs of cross-lingual fact transfer, consistent with prior findings that even large-scale models struggle to share factual knowledge across languages (Goldman et al., 2025; Xu et al., 2023; Gao et al., 2024). The model transfers only a small fraction of its English-acquired knowledge into Persian, with transfer largely confined to general facts, while Persian-related facts transfer at substantially lower rates. To understand this gap, we investigate the relationship between how frequently a fact’s subject appears in the English pretraining corpus and whether that fact transfers to Persian. Persian-related subjects, which are orders of magnitude rarer in English, transfer at near-chance levels, whereas highly frequent Persian-related subjects reach moderate transfer rates. These findings establish that a strong representation in the English data is necessary but not sufficient for fact transfer.

2

Related Work

LAMA (Petroni et al., 2019) introduced cloze-style probing of factual knowledge over subject-relationobject triples, and subsequent work extended this paradigm to multilingual and cross-lingual settings (Kassner et al., 2021; Jiang et al., 2020; Elazar et al., 2021; Fierro and Søgaard, 2022; Qi et al.,

2023; Wang et al., 2025; Liu et al., 2025b; Keleg and Magdy, 2023). However, all of these datasets are designed solely for evaluation; none provides facts structured for systematic removal from training data, which our study requires to isolate transfer from direct learning. Cross-lingual factual knowledge sharing in multilingual models remains limited: high- and lowresource languages show significant performance gaps in fact probing (Xu et al., 2023), multilingual pretraining and instruction tuning yield only shallow knowledge alignment (Gao et al., 2024), and even state-of-the-art models struggle to retrieve facts when queried in a language other than the one in which the fact is expressed (Goldman et al., 2025). Furthermore, models may not genuinely store facts but instead rely on statistical shortcuts such as prompt biases, co-occurrence statistics, and dataset artifacts (Cao et al., 2021; Elazar et al., 2023), with evidence that genuine recall, heuristic shortcuts, and guesswork activate distinct model mechanisms (Saynova et al., 2025). Fact recall also scales with entity frequency in the pretraining data, with long-tail entities remaining difficult even at large model scales (Kandpal et al., 2023). These findings motivate our use of hard and random candidate sets to disentangle precise knowledge from shallow cues, and our analysis of subject frequency as a predictor of cross-lingual transfer (§5.4).

3 SIFT To support our analysis of cross-lingual fact transfer, we introduce SIFT1 (Stratified Intervention Facts for Transfer), a resource of 500 factual knowledge triples annotated for subject provenance (general vs. Persian-related), natively written Persian templates, and negative candidate sets (distractors) at two difficulty levels, designed for both training data intervention and evaluation. As discussed in §2, existing fact probing datasets were designed for evaluating fact recall, not for the controlled manipulation of training data that our study requires. LAMA-style datasets have several shortcomings for our purposes: Many triples have minimal coverage in the Persian web corpus, making their removal from training data an empty exercise rather than a meaningful intervention. Some triples are trivially self-referential, such as ⟨flag of Sweden, APPLIES_TO_JURISDICTION, Sweden⟩, and are un1

The dataset is available on Hugging Face.

likely to appear as natural text and too easy to serve as meaningful evaluation items. Also, their facts are selected from English-centric sources (Kassner et al., 2021; Fierro and Søgaard, 2022) with no coverage of Persian-related entities. ECLeKTic (Goldman et al., 2025) addresses cross-lingual transfer more directly, but by selecting facts available in only one Wikipedia language, it inherently restricts evaluation to culture-specific or rare entities, excluding the general facts that are essential for a complete picture of cross-lingual transfer. SIFT addresses these gaps by selecting triples with sufficient Persian corpus frequency to make removal meaningful, while avoiding entities so frequent that their removal would distort the training data, and includes Persian-related entities absent from existing English-centric resources. 3.1

Fact Triples

We represent each fact as a subject–relation–object triple ⟨s, R, o⟩, a representation widely used in factual knowledge probing (Petroni et al., 2019; Wang et al., 2025). For example, the fact that Albert Einstein died in Princeton is represented as ⟨Albert Einstein, PLACE_OF_DEATH, Princeton⟩. In our study, these triples serve a dual purpose: they define the targets for our training data intervention, and they form the basis of our evaluation of factual knowledge recall. Relations are languageindependent Wikidata-style identifiers, while subjects and objects are stored with surface forms in both English and Persian: the English forms are used for English evaluation, and the Persian forms, which include all orthographic and lexical variants of an entity (§4.2, Appendix B), are used for the training-data intervention and for Persian evaluation. In addition to the triple itself, each entry in SIFT carries corpus statistics for the subject: its frequency in the English and Persian pretraining corpora, and the number of sentences in which the subject and object co-occur. We use these statistics to select triples whose removal is a meaningful intervention (Appendix B) and to analyse how sourcelanguage frequency affects transferability (§5.4). We construct a set of fact triples in two categories based on the provenance of the subject: (a) Persian-related facts, whose subjects are culturally rooted in Iran or the Persian-speaking world, and (b) general facts, whose subjects are globally prominent with no strong tie to any particular region. Prior work has shown that multilingual models systematically favor Western entities

and store culturally specific knowledge less reliably than globally shared knowledge (Hershcovich et al., 2022; Naous et al., 2024; Yin et al., 2022). Our categorization allows us to analyze how the provenance of a fact affects its cross-lingual transfer. We construct our evaluation set through a combination of LLM-assisted generation and manual curation. We prompted Claude Opus 4.6 with extended thinking via the Claude chat interface (claude.ai) in February–March 2026 to generate 70 candidate triples per relation across the two provenance categories. A native Persian speaker manually reviewed all candidates against Wikidata and Wikipedia, discarding or replacing triples that were factually incorrect, ambiguous, or insufficiently attested in the Persian web corpus. Persian-related triples required substantial manual intervention, as the model repeatedly generated the same small set of subjects across relations, limiting diversity. The final dataset comprises 500 triples2 across 20 relations (25 per relation), annotated as Persian-related (114) or general (386). The distribution of Persian-related and general triples is not uniform across relations, as some relations, such as CAPITAL and CONTINENT, are inherently general, while person-centric relations can include both general and Persian-related subjects. Additional curation criteria are described in Appendix B. 3.2 Evaluation Templates For each relation, a template is a declarative sentence containing a placeholder for the subject and a placeholder for the object, written such that substituting the correct object yields a natural, factually true sentence. At evaluation time, we fill the subject slot with the fixed subject of a triple and score each candidate object by substituting it into the object slot and computing its length-normalized logprobability under the model (§4.4); the candidate with the highest score is taken as the model’s prediction. For example, as illustrated in Figure 1 a template for the relation PLACE_OF_DEATH can be “{subject} passed away in {object}” which, for the triple ⟨Albert Einstein, PLACE_OF_DEATH, Princeton⟩, would be scored once with the correct object (“Albert Einstein passed away in Prince2

Since the same subject may appear in multiple relations, the number of unique subjects (437) is smaller than the total number of triples. This also holds within each provenance category: 324 unique general subjects vs. 386 general triples, and 113 unique Persian-related subjects vs. 114 Persian-related triples (Table 3).

ton”) and once for each of the four incorrect object candidates (e.g., “Albert Einstein passed away in Berlin”). We show the English template here for readability; the Persian templates are written natively and are not translations of their English counterparts. Existing multilingual probing datasets rely on machine-translated templates, which introduces grammatical and semantic artifacts in the target language (Semenov and Sennrich, 2025; Kassner et al., 2021; Fierro and Søgaard, 2022; Wang et al., 2025). We therefore write new declarative templates natively in Persian, where the object can appear at any position in the sentence, better approximating naturally occurring text in the pretraining corpus. For each relation, we create five templates, all written and reviewed by a native Persian speaker.3 For English evaluation, we use all five English templates of KLAR (Wang et al., 2025) per relation; these are also written manually. 3.3

Candidate Sets

To evaluate whether models can distinguish the correct object from plausible alternatives, we construct four negative candidate objects for each triple, in addition to the correct one. Candidates are negative in the sense that substituting them for the correct object yields a triple representing a provably false “fact.” Each candidate is itself a real entity of the same type, verified against Wikidata and Wikipedia to be incorrect for the given subject and relation. We construct candidates in two difficulty settings. In the hard setting, candidates are designed to be semantically close to the correct answer. For each relation, we defined criteria guiding the generation of plausible but incorrect candidates. For example, candidates for CAPITAL include the country’s largest non-capital city, neighboring country capitals, and cities with similar spelling; candidates for PLACE_OF_BIRTH and PLACE_OF_DEATH include cities from the cultural or geographic region suggested by the person’s name, so candidates for an Iranian figure are other major Iranian cities rather than unrelated world capitals. We prompted Claude Opus 4.6 with extended thinking via the Claude chat interface in February–March 2026 and these relation-specific criteria to generate candidates (full prompt in Appendix F), then verified 3 We additionally create question–answer format templates for comparison with prior work; results are reported in Appendix G.

that all candidates were factually incorrect and sufficiently proximate to serve as challenging distractors. In the random setting, we sample four candidates uniformly from all objects appearing in the same relation, without regard to semantic proximity. The hard and random settings test different aspects of fact recall. In the random setting, distractors are rarely close to the correct object, so surfacelevel cues, such as associating a Persian name with Iranian cities, are often sufficient to identify the correct answer without knowing the fact. Hard distractors are drawn from the same semantic and cultural neighborhood as the correct object, so these cues discriminate poorly among them, and success in the hard setting reflects a correspondingly smaller contribution from such heuristics. Assuming that performance in the hard setting approximates performance under an intervention that removes shallow cues, the hard candidate setting allows us to block path (c) in Figure 1, better isolating the contribution of English factual knowledge to the prediction task.

4 Experimental Setup Our experimental design operationalizes the causal framework in Figure 1. We describe the training framework (§4.1), apply three levels of fact removal intervention to the Persian training corpus, and construct one control condition (§4.2), train models on these variants (§4.3), and evaluate fact recall and cross-lingual transfer using the SIFT triples and templates (§4.4). 4.1 Training Framework We adopt the training framework of Glocker et al. (2025), who train language models from scratch using publicly available data with full control over the training pipeline. Their framework offers several properties well-suited to our study. First, they train a 180M-parameter seed model on English data from FineWeb-Edu (Lozhkov et al., 2024) and code from Python-Edu (Allal et al., 2024), then upscale it to 572M parameters via HyperCloning (Samragh et al., 2024), a method that preserves the smaller model’s output distribution while increasing capacity. This gives the upscaled model a warm start that retains its English capabilities before continued pretraining on a target language, which in our case is Persian. Second, they demonstrate that upscaled models better preserve the base

model’s English knowledge during continued pretraining, reducing catastrophic forgetting, a critical property for our study, since we rely on English knowledge persisting through Persian adaptation. 4.2

Training Data Interventions

To block the path from Persian training data to Persian factual knowledge recall (path (b) in Figure 1), we apply three levels of intervention to the Persian FineWeb-2 corpus (Penedo et al., 2025), each removing information about the target entities with different granularity.4 All levels share the same preprocessing and matching pipeline. We split the corpus into sentences using ParsiNorm (Oji et al., 2021), and for sentences exceeding 300 characters, we further split them into chunks at word boundaries while preserving complete tokens. Co-occurrence removal For each triple ⟨s, R, o⟩, we generate all pairwise combinations of subject and object surface forms – e.g., given the triple ⟨Albert Einstein, FIELD_OF_WORK, physics⟩, this yields 2 variants of “Einstein” × 5 Persian variants of “physics” = 10 pairs. We then remove any sentence in which the subject and object from at least one such pair co-occur. The need for multiple object surface forms arises because Persian allows several orthographic and lexical variants for the same concept (see Appendix B, exhaustive object enumeration). Full subject removal We remove all sentences containing any variant of the subject, regardless of whether the object co-occurs. This provides a stronger blocking of path (b) than co-occurrence removal, as it eliminates any Persian-language context in which the entity appears, ensuring that the model cannot acquire any fact about the subject from the Persian training data. Full template replacement All sentences containing a subject are replaced with neutral template sentences that preserve the subject’s surface form without exposing the model to the target factual associations. This allows us to test whether familiarity with the entity’s Persian name, independent of factual context, affects the model’s ability to transfer knowledge from English. Details on generation of replacement sentences are in Appendix C.

Random removal control The three interventions above all reduce the size of the Persian corpus, most substantially under full subject removal. To separate the effect of removing fact-specific content from the effect of removing text, we additionally consider a control data set in which sentences are removed at random, except that every sentence containing any variant of a SIFT subject or object is retained. This corpus is reduced by an amount of text comparable to that in full subject removal while leaving all factual evidence in place. Unlike the three interventions, it is not intended to block path (b); it separates the effect of removing facts from the effect of removing data. 4.3 Models We use seven models, summarized in Table 1, that vary in their language exposure and the degree of fact removal applied to the Persian training data.5 All models share the same architecture and parameter count (572M) based on the SmolLM2 architecture (Allal et al., 2025) with the Llama 3.3 tokenizer (128K vocabulary). EN-Base is the Englishpretrained model released by Glocker et al. (2025). FA-Base is trained from scratch on Persian data using the same setup. CPT models are obtained by continued pretraining of EN-Base on different versions of the Persian corpus; all five include replay data drawn from the same English and code corpora used for base model pretraining, comprising approximately 12% replay data relative to the Persian training tokens, as data replay has been shown to reduce catastrophic forgetting during continued pretraining (Scialom et al., 2022), and preserving source-language factual knowledge is critical to our experimental design. 4.4 Evaluation For each triple ⟨s, R, o⟩, we construct the candidate set by combining the correct object with the four negative candidates. Each candidate is scored by filling it into each of the five templates for relation r and computing the length-normalized log-probability of the resulting sentence; we average a candidate’s scores across the five templates and take the highest-scoring candidate as the model’s prediction, so that no individual template determines the outcome.6 We report six metrics. 5

The models are available on Hugging Face. We note that results are consistent across individual templates, with per-template accuracy varying by at most 5.2 points for any model. Per-template breakdowns (mean ± std) 6

4

Code for the data interventions and evaluation is available at https://github.com/liu-nlp/SIFT.

Model

Training Data

Baselines EN-Base FA-Base

English Tokens

Persian Tokens

Removed

English only Persian only

187.355B –

– 60.535B

– –

Continued pretraining of EN-Base (+ 12% English replay) CPT-Full Full Persian corpus 8.257B CPT-CoocRM Co-occurrences removed 8.257B CPT-SubjRM All subject mentions removed 8.257B CPT-SubjRP Subject mentions replaced 8.257B

60.535B 60.163B 55.392B 55.638B

– 0.61% 8.50% 8.09%

CPT-RandRM

55.819B

7.79%

Random sentences removed

8.257B

Description No Persian exposure No English exposure Unfiltered baseline Sentences with subject and object removed Sentences with subject removed Sentences containing subject replaced by neutral templates to preserve surface form Random sentences containing no subject or object removed

Table 1: Overview of models and training data conditions. Removed is relative to the full Persian corpus.

AccEN and AccFA measure top-1 accuracy in English and Persian independently. Overall (Goldman et al., 2025) is the fraction of triples answered correctly in both languages simultaneously. Transfer (Goldman et al., 2025) is Persian accuracy conditioned on the model answering correctly in English, isolating cases where knowledge acquired in English could have transferred cross-lingually. Non-Transfer is Persian accuracy conditioned on the model answering incorrectly in English, capturing facts the model knows in Persian independently of English. RankC (Qi et al., 2023) measures the Spearman rank correlation between the model’s candidate orderings in English and Persian. All metrics are computed separately for hard and random candidate sets.

5

Results and Discussion

Table 2 summarizes fact recall and cross-lingual transfer across all model variants under both hard and random candidate settings. We organize the results around four questions: how much targetlanguage data must be removed to ensure a fact is truly unseen, and whether the removal itself confounds the comparison (§5.1); how much factual knowledge transfers from English to Persian under this controlled setting (§5.2); to what extent does performance reflect factual knowledge rather than shallow associative cues (§5.3); and how does a fact’s origin, and, related to that, frequency in the English corpus affect its transferability (§5.4). Note that the empirical floor of our models is above the 20% chance level: EN-Base, which has no Persian exposure, reaches 26.0% in Persian on hard candidates and 31.4% on random, and FABase reaches 32.8% and 40.0% in English. We treat these scores as our baseline. are provided in Appendices E and G (Tables 4–7; Figure 6).

5.1 How Much Removal Is Enough? To determine the level of target-language data intervention required to ensure a fact is entirely unseen, we evaluate the model’s fact retrieval across the removal granularities introduced in §4.2, and we use our random removal control to test whether the amount of data removed confounds this comparison. Results are in Table 2. Co-occurrence removal is insufficient. We find that sentence-level co-occurrence removal is insufficient to guarantee fact absence. Under CPTCoocRM, which only removes sentences where the subject and object co-occur, the model still predicts many facts correctly in Persian. This could in principle reflect either (1) cross-lingual transfer from English or (2) an insufficient removal strategy that leaves fact signal in the Persian corpus. We argue for the latter. Both Overall and Transfer scores remain substantially higher under CPTCoocRM than under full subject removal (Overall: 25.8 vs. 20.6; Transfer: 48.9 vs. 39.9 on hard candidates), despite the same English knowledge being available for transfer in both conditions. More directly, the Non-Transfer score under CPTCoocRM reaches 30.1% (hard): the model correctly predicts facts in Persian even when it fails in English, where cross-lingual transfer is highly unlikely. We hypothesize that the model acquires factual associations from broader document-level context, where the subject and object appear across sentence boundaries. Full subject removal as the primary intervention. Moving to full subject removal, the NonTransfer score drops from 30.1% under CPTCoocRM to 24.4% under CPT-SubjRM (hard), matching EN-Base’s 24.1%, indicating that removing all sentences containing the subject more ef-

Hard Candidates

Random Candidates

Model

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

EN-Base FA-Base CPT-Full CPT-CoocRM CPT-SubjRM CPT-SubjRP CPT-RandRM

55.2 32.8 52.6 52.8 51.6 52.2 54.2

26.0 35.0 46.6 40.0 32.4 28.0 47.2

15.2 12.8 31.4 25.8 20.6 18.6 33.4

27.5 39.0 59.7 48.9 39.9 35.6 61.6

24.1 33.0 32.1 30.1 24.4 19.7 30.1

4.4 11.2 24.8 21.8 14.6 14.2 25.6

68.8 40.0 66.8 66.8 67.4 67.6 67.0

31.4 43.8 61.6 54.2 42.4 35.2 61.4

23.2 21.0 47.2 41.2 33.6 28.0 47.2

33.7 52.5 70.7 61.7 49.9 41.4 70.4

26.3 38.0 43.4 39.2 27.0 22.2 43.0

10.7 19.8 38.8 34.6 22.2 22.5 37.6

Table 2: Fact recall and cross-lingual transfer across model variants. Hard candidates are semantically close distractors that suppress shallow cues; random candidates are sampled uniformly from the same relation. ACCEN and ACCFA report top-1 accuracy in English and Persian; OVERALL is the fraction of triples answered correctly in both. TRANSFER is Persian accuracy conditioned on English correctness; NON-TRANSFER conditions on English incorrectness. RANKC is the Spearman rank correlation between English and Persian candidate orderings.

fectively mitigates fact exposure from broader contexts. Thus, CPT-SubjRM serves as our primary intervention level, providing a more rigorous setting to keep the fact unseen in the target language without excessive data loss. From a causal perspective, CPT-SubjRM therefore provides our closest approximation to an intervention that removes targetlanguage facts while preserving source-language knowledge. Adding subjects back in neutral templates degrades performance. As CPT-SubjRM removes the Persian surface form of the subject entity completely, we evaluate CPT-SubjRP to observe the empirical effect of preserving surface-form exposure via neutral templates. The results show that this approach actually degrades performance, achieving only 28.0% (hard) and 35.2% (random) compared to CPT-SubjRM’s 32.4% and 42.4%. The near identical rank correlation scores (14.2– 14.6 on hard, 22.2–22.5 on random) indicate that the degradation is unlikely to stem from a difference in cross-lingual alignment. We propose two explanations for this degradation: first, repeatedly seeing an entity in neutral Persian sentences may cause the model to build a shallow, Persian-only representation of that entity, weakening the factual knowledge it had already acquired from English. It could also be an effect of duplicate data, which has been shown to degrade model quality (Lee et al., 2022) and may lead to reproducing memorized templates rather than retrieving facts (Carlini et al., 2023). Removal volume alone does not explain the drop. CPT-RandRM removes 7.79% of Persian tokens, comparable to the 8.50% removed under CPT-SubjRM, while leaving all factual evidence in place. It reaches 47.2% Persian accuracy

on hard candidates, statistically indistinguishable from CPT-Full’s 46.6%, whereas CPT-SubjRM falls to 32.4% under the same evaluation. Removing a similar amount of text therefore leaves accuracy almost unchanged, as long as the factual evidence stays in. We therefore attribute the drop under CPT-SubjRM to the removed factual evidence rather than to the reduced corpus size. 5.2 Limited Cross-Lingual Transfer Having established CPT-SubjRM as our primary intervention, in which no Persian-language exposure to the entities remains, we now use it to quantify the degree of cross-lingual fact transfer. Following the evaluation framework of ECLeKTic (Goldman et al., 2025), we ask: to what extent can the model retrieve English-acquired facts through Persian? There is little fact transfer. The Persian hardcandidate accuracy under CPT-SubjRM approximates the probability of correct Persian prediction when Persian factual evidence and shallow heuristic cues are minimized. Under the causal interpretation discussed in Appendix A, performance above the empirical floor in this setting therefore provides evidence of cross-lingual fact transfer from English. As shown in Table 2, CPT-SubjRM on hard candidates achieves an Overall score of only 20.6%, indicating that the model rarely succeeds in both languages simultaneously. Its Transfer score reaches 39.9%, suggesting that some English-acquired knowledge is accessible in Persian, although most is not. This is consistent with Goldman et al. (2025), who find that even largescale models hardly share knowledge across languages. A per-relation breakdown reveals substantial variation in both fact recall and the effect of our

Persian-related (n=114)

Distinguishing transfer from difficulty confounds. To further assess whether this gap reflects genuine transfer rather than a difficulty confound, we compare the Transfer and Non-Transfer accuracies. CPT-SubjRM shows a difference of 15.5 percentage points between the two (39.9% vs. 24.4%), while EN-Base shows virtually no such gap (27.5% vs. 24.1%). If the gap were driven solely by certain facts being inherently easier, it would appear in both models; its absence in ENBase suggests that the gap in CPT-SubjRM reflects a limited degree of cross-lingual transfer enabled by continued pretraining. Nonetheless, Transfer and Non-Transfer accuracies both remain low, and the Non-Transfer accuracy (24.4%) falls below the 26.0% empirical floor, indicating that the model’s ability to transfer knowledge into Persian is, at most, limited. Rank-based evaluation. Beyond binary correctness, we use RankC to examine whether crosslingual transfer is also reflected in the model’s full ranking over candidates. EN-Base, which has no Persian exposure, serves as the baseline at 4.4 on hard candidates. CPT-SubjRM obtains a RankC of 14.6 on hard candidates. Although both values remain low relative to the theoretical maximum (100), the increase suggests that continued pretraining on Persian text partially aligns candidate preferences across languages, even for entities absent from the Persian training data. 5.3

Precise Knowledge or Shallow Heuristics?

Comparing hard and random candidate settings reveals the extent to which models rely on shallow associative cues. Across all models, the randomcandidate accuracy substantially exceeds the hardcandidate accuracy in both languages. For CPTSubjRM, the Persian accuracy gap is 10 percentage points (42.4% vs. 32.4%), while the English accuracy gap is 15.8 points (67.4% vs. 51.6%), indicating that surface-level cues, such as associating a subject from a particular cultural region with geographically related objects, inflate performance in both languages, not only in the target language. Crucially, this effect persists even when conditioning on English knowledge: the Transfer score for CPT-SubjRM drops from 49.9% with random candidates to 39.9% with hard candidates, and for CPT-Full from 70.7% to 59.7%. This suggests that a portion of the apparent cross-lingual trans-

Transfer score (%)

intervention across relation types (Appendix D). 80% 70% 60% 50% 40% 30% 20% 10% 0%

General (n=386)

Hard candidates

ll P e e Bas Bas -Fu cRM bjRM bjR dRM EN- FA- CPT PT-Coo PT-Su CPT-Su PT-Ran C C C

Random candidates

ll P e e Bas Bas -Fu cRM bjRM bjR dRM EN- FA- CPT PT-Coo PT-Su CPT-Su PT-Ran C C C

Figure 2: Transfer score by provenance category under hard and random candidate settings. Counts (n) refer to the number of triples per category (500 total). Median freq. % subjects in EN EN General Facts (n=324) 196K Persian-related Facts (n=113) 284

FA <100

<1K

35K 40K

1 65

0 36

Table 3: Median subject frequency in the English and Persian pretraining corpora by provenance category. Counts (n) refer to unique subjects (437 total), which are fewer than the 500 triples reported in (§3.1) since a subject may appear in multiple triples across relations.

fer measured under easier conditions is itself attributable to shallow heuristics rather than precise fact retrieval (Saynova et al., 2025). The hard candidate setting, which includes distractor candidates that are semantically and culturally close to the correct answer, reduces the contribution of such heuristics to the measured transfer scores. 5.4 Entity Frequency and Provenance Affect Transferability Fact recall scales with entity frequency in monolingual settings (Kandpal et al., 2023), and a similar relationship has been observed for cross-lingual transfer (Liu et al., 2025b; Zhang et al., 2025), but without isolating source-language frequency from direct target-language learning. Because our training data intervention removes all subject mentions from the Persian corpus, we can isolate the role of English frequency more cleanly. English frequency gates transfer. Figure 3 shows Transfer accuracy as a function of subject frequency in both the English and Persian pretraining corpora. For CPT-SubjRM, Transfer accuracy increases steadily with English corpus frequency up to the 1M–10M bin: entities appearing fewer than 1,000 times in English are transferred near the empirical floor, while entities in the 100K– 10M range reach 46–51%. This confirms that the strength of the English-side representation is a bottleneck for cross-lingual transfer, entities that the

Transfer accuracy vs. subject frequency Persian corpus frequency

100

Transfer accuracy (%)

80 60 40 20 0

English corpus frequency

CPT-Full (n=262) CPT-CoocRM (n=263) CPT-SubjRM (n=257) CPT-SubjRP (n=260)

4 5

16 17 14 16

3

73

100 1K

40

118

66 73

40 40 42

117 116

71

1K 10K

10K 100K

100K 1M

Subject frequency in Persian corpus

15 15

12 13 13

>1M

51 6

110

13

115

5

1 100

CPT-Full (n=262) CPT-CoocRM (n=263) CPT-SubjRM (n=257) CPT-SubjRP (n=260)

49 47 50

109

12 17 18 20 18

11

1 1K

1K 10K

63 67 68 61

107

10K 100K

100K 1M

110

1M 10M

Subject frequency in English corpus

6 6 6

>10M

Figure 3: Transfer accuracy (%) on hard candidates as a function of subject frequency in the Persian (left) and English (right) pretraining corpora. Numbers above points indicate subjects per bin.

model encountered more frequently in English are more likely to be retrievable in Persian after continued pretraining, even when those entities were never seen in Persian. However, even at the highest English frequencies, CPT-SubjRM remains well below CPT-Full (51% vs. 82% in the 1M–10M bin), indicating that English exposure is insufficient for reliable transfer and that direct Persian exposure contributes substantially. Persian frequency as an indirect signal. On the Persian frequency side, CPT-SubjRM still shows a positive trend despite having all subject mentions removed from the Persian corpus. Since Persian corpus frequency reflects the general prominence of an entity, this correlation likely arises indirectly: globally prominent entities tend to be well-represented in English as well, providing a stronger source-language representation to transfer from. The gap between CPT-Full and CPTSubjRM is substantial across the frequency range and largest in the well-populated high-frequency bins (e.g., 100K–1M: 73% vs. 49%), showing that while cross-lingual transfer accounts for the majority of performance, direct learning from Persian contributes a further 24-percentage-point gain. Impact of the origin of facts. The frequency bottleneck manifests also across provenance categories. As Table 3 shows, Persian-related subjects (e.g., Persian historical figures) have a median English frequency of only 284, roughly 690 times lower than general subjects (median 196K), with 65% appearing fewer than 1,000 times and 36% fewer than 100 times. Persian-side frequencies, by contrast, are comparable across categories (40K vs. 35K median). The consequence is visible in Fig-

ure 2: on hard candidates, CPT-SubjRM transfers general facts at 41.5% but Persian-related facts at only 25.0%. EN-Base shows no such gap (27.4% vs. 29.2%), confirming that the disparity is not due to Persian-related facts being inherently harder but rather to their sparse English representation. The low English accuracy of CPT-SubjRM on Persianrelated facts (21.1% vs. 60.6% on general) corroborates this: the model simply lacks the sourcelanguage knowledge from which transfer could occur. On random candidates the ordering reverses (48.7% general vs. 55.2% Persian-related), as Persian names offer strong associative cues that randomly sampled distractors rarely defeat. EN-Base, with no Persian exposure, shows the same reversal (31.8% vs. 44.2%), but the hard-candidate setting strips away these heuristics and exposes the underlying frequency-driven limitation.

6 Conclusion and Future Work We presented an intervention-based framework for studying cross-lingual factual knowledge transfer during continued pretraining. By systematically removing specific facts from Persian training data and probing the resulting model, we established causal evidence that cross-lingual transfer from English exists but is limited. When our interventions remove the facts from the Persian data and remove access to shallow heuristics that make predictions easier, the model transfers only a small fraction of its English-acquired knowledge into Persian, with transfer largely confined to general facts whose subjects are frequent in the English corpus: entities with low English exposure transfer at nearchance levels, while even highly frequent entities have limited performance, indicating that English-

side representation strength is necessary but insufficient for reliable transfer. One factor potentially limiting transfer between English and Persian is the difference in script. Unlike language pairs that share the Latin alphabet, Persian’s Arabic-derived script offers no orthographic overlap with English, and thereby no token overlap, which may hinder the model’s ability to align entity representations across languages. A promising direction for future work is to incorporate transliterated data into the pretraining corpus, as proposed by Liu et al. (2024, 2025a), to test whether bridging the script gap enables stronger cross-lingual fact transfer even in our interventionbased setup.

Limitations Our study is limited to a single target language, Persian, which uses a non-Latin script with no orthographic overlap with English. Extending this framework to languages that share the Latin script with English would help determine whether script similarity facilitates stronger cross-lingual fact transfer. Similarly, including additional nonLatin-script languages would test the generality of our findings beyond the English–Persian pair. However, both extensions are challenging: our approach requires natively written templates, manually curated triples with country-specific facts, and hard candidate sets designed by speakers with expertise in the target language, making each new language a substantial annotation effort rather than a simple translation task. In addition, our models are limited to 572M parameters. Goldman et al. (2025) find that scaling model size improves overall fact recall but does not substantially improve cross-lingual transfer, but their conclusion is based on observational comparisons without training data interventions. Whether the same pattern holds under controlled fact removal remains an open question that would require training larger models.

Acknowledgments We thank the anonymous reviewers for their constructive feedback and insightful suggestions. This research was supported by TrustLLM funded by Horizon Europe GA 101135671 and the Graduate School in Computer Science (CUGS). It was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP)

funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.

References Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blazquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Agustín Piqueres Lajarín, Hynek Kydlíček, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan Son NGUYEN, Ben Burtenshaw, Clémentine Fourrier, Haojun Zhao, Hugo Larcher, Mathieu Morlon, Cyril Zakka, and 3 others. 2025. SmolLM2: When smol goes big — datacentric training of a fully open small language model. In Second Conference on Language Modeling. Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. SmolLM-Corpus. https://huggingface.co/ datasets/HuggingFaceTB/smollm-corpus. Dataset release. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1860–1874, Online. Association for Computational Linguistics. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav GoldMeasuring causal effects of data berg. 2023. statistics on language model’s ‘factual’ predictions. Preprint, arXiv:2207.14251. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. Constanza Fierro and Anders Søgaard. 2022. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics. Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024. Multilingual

pretraining and instruction tuning improve crosslingual knowledge alignment, but only shallowly. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6101–6117, Mexico City, Mexico. Association for Computational Linguistics. Kevin Glocker, Kätriin Kukk, Romina Oji, Marcel Bollmann, Marco Kuhlmann, and Jenny Kunz. 2025. Grow up and merge: Scaling strategies for efficient language adaptation. Preprint, arXiv:2512.10772. Omer Goldman, Uri Shaham, Dan Malkin, Sivan Eiger, Avinatan Hassidim, Yossi Matias, Joshua Maynez, Adi Mayrav Gilady, Jason Riesa, Shruti Rijhwani, Laura Rimell, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2025. Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer. Preprint, arXiv:2502.21228. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in crosscultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Association for Computational Linguistics. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. Association for Computational Linguistics. Amr Keleg and Walid Magdy. 2023. DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6245–6266, Toronto, Canada. Association for Computational Linguistics. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch,

and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics. Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schuetze. 2024. TransliCo: A contrastive learning framework to address the script barrier in multilingual pretrained language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2476–2499, Bangkok, Thailand. Association for Computational Linguistics. Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob ImaniGooghari, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, and Hinrich Schütze. 2025a. How transliterations improve crosslingual alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2417–2433, Abu Dhabi, UAE. Association for Computational Linguistics. Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, and Hinrich Schuetze. 2025b. Tracing multilingual factual knowledge acquisition in pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2121–2146, Suzhou, China. Association for Computational Linguistics. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. FineWeb-Edu: the finest collection of educational content. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics. Romina Oji, Seyedeh Fatemeh Razavi, Sajjad Abdi Dehsorkh, Alireza Hariri, Hadi Asheri, and Reshad Hosseini. 2021. Parsinorm: A persian toolkit for speech processing normalization. In 2021 7th International Conference on Signal Processing and Intelligent Systems (ICSPIS), pages 1–5. Judea Pearl. 2009. Causality: Models, Reasoning and Inference, 2nd edition. Cambridge University Press, USA. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference on Language Modeling.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics. Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666, Singapore. Association for Computational Linguistics. Mohammad Samragh, Seyed Iman Mirzadeh, Keivan Alizadeh-Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. 2024. Scaling smart: Accelerating large language model pre-training with small model initialization. In Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, volume 262 of Proceedings of Machine Learning Research, pages 1–13. PMLR. Denitsa Saynova, Lovisa Hagström, Moa Johansson, Richard Johansson, and Marco Kuhlmann. 2025. Fact recall, heuristics or pure guesswork? precise interpretations of language models for fact completion. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18322–18349, Vienna, Austria. Association for Computational Linguistics. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107–6122, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Kirill Semenov and Rico Sennrich. 2025. Measuring the effect of disfluency in multilingual knowledge probing benchmarks. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10654–10672, Suzhou, China. Association for Computational Linguistics. Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, and Hinrich Schuetze. 2025. Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5075–5094, Vienna, Austria. Association for Computational Linguistics. Shaoyang Xu, Junzhuo Li, and Deyi Xiong. 2023. Language representation projection: Can we transfer factual knowledge across languages in multilingual language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language

Processing, pages 3692–3702, Singapore. Association for Computational Linguistics. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Chen Zhang, Zhiyuan Liao, and Yansong Feng. 2025. Cross-lingual transfer of cultural knowledge: An asymmetric phenomenon. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 147–157, Vienna, Austria. Association for Computational Linguistics.

A Causal Effect Estimation of Factual Knowledge Transfer In this section, we motivate our experimental design from a causal perspective (Pearl, 2009). Figure 4 illustrates the assumed causal graph underlying our setup. Let Xe denote whether the model has access to the relevant fact through English pretraining, and let Xp denote whether the fact is available through Persian continued pretraining. As discussed in (§5.3), let M represent shallow statistical cues that the model may exploit to make predictions without retrieving the underlying fact. Finally, let Y be a binary variable indicating whether the model predicts a fact correctly in Persian. For simplicity, we treat Xe , Xp , M , and Y as binary variables. For example, Xp = 0 corresponds to a setting where the relevant fact has been removed from the Persian training data, while Xp = 1 denotes that the fact remains available. Our goal is to estimate the extent to which correct Persian predictions can be causally attributed to factual knowledge acquired from English. Ideally, one would estimate the average causal effect of English factual knowledge by directly intervening on Xe : E[Y | do(Xe = 1)] − E[Y | do(Xe = 0)]

(1)

However, such interventions are infeasible in our setting because the models are initialized from an English-pretrained checkpoint. In particular, we never observe the counterfactual model that would have been trained identically but without exposure to the relevant English facts.

Shallow Cues English corpus

Facts Eng. Persian Factual Prediction

Persian corpus

Facts Pers.

Figure 4: Causal graph illustrating the relationship between corpora, extracted facts, shallow cues, and factual prediction in Persian.

Instead, we study a controlled causal effect under conditions where alternative sources of information are removed. Specifically, we consider the quantity ACE⋆ := E[Y | do(Xe = 1), do(Xp = 0), do(M = 0)] − E[Y | do(Xe = 0), do(Xp = 0), do(M = 0)] (2) This quantity measures the causal contribution of English factual knowledge when neither Persian factual exposure nor shallow heuristics can support prediction. Intuitively, under these interventions, correct predictions should primarily reflect crosslingual factual transfer from English. Importantly, this controlled effect differs from the unconditional average causal effect above. If the relevant fact remains directly available in Persian training data, adding English factual knowledge may have little effect on prediction accuracy because the model can already solve the task from Persian evidence alone. Our controlled intervention therefore isolates the model’s ability to transfer knowledge across languages rather than its overall factual recall performance. We approximate the intervention do(Xp = 0) by systematically removing facts from the Persian training corpus, as described in (§4.2). While we cannot directly intervene on M , we construct a hard-candidate evaluation setting designed to minimize the predictive utility of shallow heuristics. We assume that the probability of success in this hard setting is approximately equal to the probability of success in the random setting had we removed the shallow cues through an intervention E[Yhard | do(Xe = 1), do(Xp = 0)] ≈ E[Y | do(Xe = 1), do(Xp = 0), do(M = 0)], (3)

i.e. within this evaluation setting, the remaining contribution of shallow cues is negligible, approximating the intervention do(M = 0). Under these interventions, we can relate the first term in ACE⋆ to an observable quantity: E[Y | do(Xe = 1), do(Xp = 0), do(M = 0)] = E[Y | Xe = 1, do(Xp = 0), do(M = 0)] ≈ E[Yhard | Xe = 1, do(Xp = 0)] (4) This corresponds to the setting evaluated by our intervention experiments: the model retains access to English-acquired knowledge, while Persian factual evidence and shallow heuristics are minimized. Although we cannot directly observe the counterfactual intervention do(Xe = 0), under our controlled setup we expect performance without informative English facts to fall approximately to chance level: E[Y | do(Xe = 0), do(Xp = 0), do(M = 0)] ≈ Ychance (5) with Ychance = 0.2 because the candidate set contains five options. Together, Equations 4 and 5 motivate our experimental design: by removing Persian factual evidence and minimizing shallow cues, successful prediction above chance level provides evidence that the model is relying on factual knowledge transferred from English.

B Triple Creation Criteria Subject diversity Language models tend to produce the same prominent entities across multiple relations, for instance, generating “France” under CAPITAL, CONTINENT, and OFFICIAL_LANGUAGE, or “Albert Einstein” under FIELD_OF_WORK, PLACE_OF_BIRTH, and COUNTRY_OF_CITIZENSHIP. Because our data intervention removes sentences based on subjectobject co-occurrence, a subject appearing in many relations would trigger broad removal of text that is unrelated to the specific fact being tested. We therefore curated the final selection to maximize subject diversity across relations. Temporal diversity The generated candidates were heavily skewed toward historical figures. We deliberately included contemporary public figures to ensure the dataset is not biased toward entities

that are disproportionately represented in encyclopedic sources. Distinct subject-object surface forms Some generated triples had near-identical subject and object strings e.g., ⟨Germany, OFFICIAL_LANGUAGE, German⟩, which causes unintended removals during filtering and makes evaluation trivially solvable through string overlap rather than factual knowledge. We replaced such triples with pairs whose surface forms are clearly distinct e.g., ⟨Austria, OFFICIAL_LANGUAGE, German⟩. Person name variants For triples with person subjects, we identified all name variants likely to appear in natural text to ensure comprehensive matching during the data intervention. Full names typically appear at the start of a document, while shortened forms are used thereafter. We handled this by generating variant lists, for example, “Pablo Picasso” yields the additional variant “Picasso”. These variants were verified against both English and Persian Wikipedia pages. Exhaustive object enumeration Some subjects have more than one correct object for a given relation, for example, Stephen Hawking’s FIELDS_OF_WORK include both “cosmology” and “physics”. In such cases, we list all valid objects in the dataset. This is necessary because our filtering pipeline only removes facts that are explicitly listed: if we listed “cosmology” but not “physics”, sentences linking “Hawking” to “physics” would remain in the training corpus, and the model could appear to transfer knowledge that it actually learned directly from the unfiltered Persian data. Similarly, when an object can be written in multiple ways in Persian, we include all surface form variants to ensure complete removal. Object diversity within relations For each relation, we selected objects that cover a broad range of values rather than concentrating on the most frequent ones. For instance, the RELIGION relation includes Christianity, Islam, Zoroastrianism, Mazdakism, Judaism, Hinduism, Buddhism, and Sikhism; the CAPITAL relation includes both wellknown capitals (e.g., Paris, Tokyo) and less prominent ones. Corpus frequency control Some triples occurred so frequently in the Persian FineWeb-2 corpus that removing them would have distorted the language distribution of the training data; we re-

placed these with less frequent alternatives. For example, subjects such as Iran, China, Russia, Samsung, and Microsoft alone accounted for approximately 6% of all tokens under full subject removal.

C Training Data Replacement Templates In our template replacement condition (§4.2), instead of removing sentences containing a subject entity, we replace them with neutral substitute sentences that preserve the subject’s surface form without exposing the model to the target factual associations. We authored a pool of 100 neutral Persian sentences per entity type (e.g., PERSON, COUNTRY, CITY), each containing a “{subject}” placeholder, shared across relations with the same subject type. Each template satisfies three constraints: (1) it must be true for every subject in the entity-type group; (2) it must not contain vocabulary that could indirectly encode the target relation, enforced via per-type forbidden-vocabulary lists; and (3) templates vary in length, position, and style to avoid repetitive contexts. During filtering, each targeted sentence is replaced by a randomly drawn template with the placeholder filled by the entity’s Persian surface form.

D How Does Fact Recall Vary By Relation? Figure 5 reveals substantial variation in fact recall across relations. Relations expressed through frequent and formulaic patterns in the training data, such as CAPITAL_OF, CONTINENT, and OFFICIAL_LANGUAGE, are the easiest across all models, with EN-Base reaching 92–100% in English. For CONTINENT, the constrained answer space (six possible objects) further contributes to high accuracy. These relations also show the largest drop under our intervention: Persian accuracy falls 32–48 points from CPT-Full to CPT-SubjRM (e.g., CAPITAL_OF: 96%→48%, CONTINENT: 92%→52%), confirming that CPT-Full’s performance on these relations was largely attributable to direct learning from the Persian corpus. Conversely, person-attribute relations such as OCCUPATION, DEVELOPER, and LANGUAGES_SPOKEN show smaller drops under subject removal (4–20 points) and remain above the empirical floor under CPT-SubjRM (40–48%). However, for relations with few possible answers, such as RELIGION (8 unique objects), this may partly reflect the constrained answer space rather

Persian Data

40 8

52

CPT-SubjRP 52 84 96 84 48 60 60 40 16 52 64 32 48 72 28 100 28 24

8

48

CPT-RandRM 56 84 100 88 48 68 64 36 16 52 68 28 44 72 32 96 36 32

8

56

20 0

CPT-SubjRM 20 40 48 52 20 48 32 36 24 16 40 32 12 32 40 48 32 20 24 32 CPT-SubjRP 16 44 56 12 20 44 32 28 24 16 16 36 20 12 28 52 24 20 20 40 CPT-RandRM 32 72 96 72 52 56 44 44 36 28 40 44 32 40 64 76 32 28 20 36

To J Ap

pli

es

To J es pli Ap

60

CPT-CoocRM 32 40 84 56 44 44 40 40 40 24 36 36 32 28 52 72 24 20 24 32

uri

sd ict i Ca on p C Co ap ital un try C ital O Of onti f Cit ne ize nt D nsh He ad Fie evelo ip qu ld pe art Of r La ng ers W ua Lo ork ge Of Ins catio Wo tru n La rk m Lo ngu Or ent ca ag Na tio es m n O Sp e f F ok e M orm n Na anu atio tiv fac n e L tu an rer Off Occ guag icia up e l L atio an n g O ua Pla wne ge ce d B Pla Of y ce Bi Of rth De Re ath lig ion

CPT-SubjRM 56 80 100 80 48 56 52 32 20 52 64 24 44 64 32 92 36 40

80

40 20 0

ict i Ca on Ca pita pit l un a try C l O Of onti f Cit ne ize nt D nsh He ad Fie evelo ip qu ld pe art Of r La ng ers W ua Lo ork ge Of Ins catio Wo tru n La rk m Lo ngu Or ent ca ag Na tio es m n O Sp e f F ok e M orm n Na anu atio tiv fac n e L tu an rer Off Occ guag icia up e l L atio an n g O ua Pla wne ge c d Pla e Of By ce Bi Of rth De Re ath lig ion

60

CPT-CoocRM 52 84 92 76 52 64 68 40 20 52 68 24 44 68 36 92 28 36 12 48

CPT-Full 24 64 96 92 60 52 40 40 24 16 48 44 36 36 60 80 32 28 24 36

Co

CPT-Full 60 80 96 84 44 64 60 44 16 52 60 36 48 68 32 88 32 32 12 44

FA-Base 20 44 72 32 40 40 40 40 28 16 44 40 24 28 44 48 24 24 20 32

80

sd

28

100

uri

8

Accuracy (%)

FA-Base 48 36 44 16 32 60 36 16 16 40 48 32 40 60 20 36 20 20

100

EN-Base 12 24 20 20 32 36 32 20 20 36 40 16 24 16 44 36 28 24 16 24

Accuracy (%)

English Data EN-Base 64 80 100 92 52 52 68 44 24 64 64 36 44 68 32 100 32 28 16 44

Figure 5: Per-relation accuracy (%) on hard candidates for English (left) and Persian (right) evaluation.

than robust recall. At the other end, PLACE_OF_DEATH remains near or below the empirical floor in both languages across all models, with no drop from CPT-Full to CPT-SubjRM. This fact type was not acquired even from the full Persian data. Notably, for LANGUAGE_OF_WORK_OR_NAME, continued pretraining actually degrades Persian accuracy from 36% (EN-Base) to 16% (CPT-SubjRM).

E

Robustness to Different Templates

Figure 6 shows the distribution of per-template accuracy across all five evaluation templates for each model under both hard and random candidate settings. The narrow spread of scores confirms that our results are robust to variation in template phrasing: the standard deviation across templates is at most 5.2 percentage points for any model, and the relative ordering of models is consistent across all templates. This indicates that the patterns reported in the main results (§5) are not driven by idiosyncratic properties of individual templates.

F

Prompt for Hard Distractor Candidate Generation

The prompt in Figure 7 was used to generate hard distractor candidates for each relation type. It instructs the model to produce four incorrect but plausible candidates per triple, following relationspecific criteria designed to maximize semantic and cultural proximity to the correct answer. For example, candidates for geographic relations are drawn from neighboring or commonly confused locations, while candidates for person-attribute relations exclude all known correct values and favor

entities from the same cultural or professional domain. All generated candidates were subsequently verified by a native Persian speaker to ensure factual incorrectness and sufficient difficulty.

G Mean Results on Different Templates Tables 4–7 report mean accuracy and standard deviation across the five evaluation templates for all models, broken down by template format (descriptive and question answering) and candidate setting (hard and random). The results are broadly consistent across both template formats, with the same relative ordering of models and the same overall patterns observed in the main results. One notable difference is that CPT-SubjRP slightly outperforms CPT-SubjRM under the question answering format (Tables 5 and 7), by 2.9 points on hard and 5.7 points on random candidates, reversing the pattern seen with descriptive templates. The reversal exceeds template-level variation, indicating that the relative ordering of these two variants is sensitive to template format. Overall, the consistency across template formats confirms that our findings are not sensitive to the specific evaluation format used.

Hard candidates

Random candidates

Accuracy per template (%)

60%

50%

40%

30%

se

-Ba

EN

-Ba

FA

se

ll M M M jRP -Fu ocR SubjR -Sub ndR o C Ra T T P T T C CP CP CP

T CP

se

-Ba

EN

-Ba

FA

se

M M M ull jRP ocR SubjR -Sub T-F ndR o P C Ra C T T P T T C CP CP CP

Figure 6: Per-template accuracy distribution for each model under hard (left) and random (right) candidate settings. Each box summarizes accuracy across 5 evaluation templates; individual template scores are shown as points, and diamonds indicate the mean. The narrow spread (std ≤ 5.2pp) confirms that results are robust to template phrasing.

You are generating object_candidates for knowledge base triples. Each triple has a subject, relation, and object(s). Generate 4 wrong candidates that do NOT include any correct answer. Follow these rules per relation: 1. capital: Use the largest non-capital cities of the same country, capitals of neighboring countries, cities with similar spelling to the correct capital, and cities commonly mistaken as the capital. For historical empires, use major cities within the empire's territory that were never its capital and cities that served as capital of a predecessor or successor state. 2. capital_of: Use geographically neighboring countries sharing a border, countries sharing the same language or colonial history, countries commonly confused with each other, countries the city has historical ties to, and countries on the same continent with similar characteristics. 3. continent: Use the geographically closest continents, continents the country is commonly misassigned to, and for transcontinental countries the other continent they straddle. Replace distant implausible continents with more confusing ones. 4. country_of_citizenship: Exclude all countries the person ever held citizenship in. Use countries the person lived in or worked in but never held citizenship of, neighboring or culturally linked countries, countries sharing the same language, countries the person is commonly but incorrectly associated with, and countries where the person had significant career activity. If there is any doubt about citizenship, exclude the candidate. 5. developer: Use direct competitors making the same category of product, companies that make the closest rival product, and parent or subsidiary companies in the same tech ecosystem. For Chinese apps use other major Chinese tech companies, and for Iranian apps use other Iranian tech ecosystem companies. Never include the correct developer under any alternate name. 6. field_of_work: Exclude any field the person genuinely worked in even partially, and exclude overly general fields that could apply to anyone. Use fields adjacent to but distinct from the correct one. For highly multi-disciplinary figures, use clearly anachronistic or unrelated fields. Easy candidates are acceptable, and if there is any doubt whether a field applies, exclude it. 7. headquarters_location: Use other cities in the same metro area, cities where the company has major offices or factories, and the founder's city if different from the headquarters. For Japanese companies use other major Japanese industrial cities, and for Iranian institutions use nearby cities or functionally similar ones. 8. instrument: Exclude every instrument the musician is known to play. For Iranian multi-instrumentalists use only Western instruments, and for Iranian musicians who only play Western instruments, Iranian instruments are valid wrong candidates. Use instruments commonly associated with the same genre and instruments played by the musician's bandmates or frequent collaborators. 9. languages_spoken: Exclude every language the person speaks, not just the listed answer. Use languages of countries the person lived in but confirmed they do not speak, closely related languages to their known ones, languages of the person's geographic or cultural region, and languages of countries where the person had significant career activity. If there is any doubt whether the person speaks a candidate language, exclude it. 10. native_language: Exclude every language the person is native in or spoke from childhood. Use languages of the country the person is most associated with, closely related languages, and languages the person learned later in life but is not native in. If there is any doubt about native status, exclude it. 11. official_language: Exclude all co-official languages of the country. Use the largest minority languages spoken within the country, languages of neighboring countries, former colonial or administrative languages if not currently official, languages historically associated with the country, and languages commonly but incorrectly assumed to be official. 12. occupation: Exclude all real occupations including secondary ones and exclude occupations too general that could plausibly apply. For athletes use other sports or athletic roles, and for performers use nearby performance roles. Use occupations of close collaborators or contemporaries and occupations the person is commonly but incorrectly described as. 13. religion: Use religions historically present in the person's region of origin, religions the person studied or was influenced by but did not follow, the dominant religion of the country the person is associated with, and religions commonly confused with the correct one in the region. Do not include denominational distinctions. 14. applies_to_jurisdiction: Use jurisdictions with genuine historical or geographic connections to the law, countries that passed similar laws or policies, the other party in a bilateral treaty if it is not also a correct answer, and predecessor or successor states of the correct jurisdiction. Do not include member states of the actual

jurisdiction. 15. manufacturer: Use direct competitors making the same or very similar product, companies making the closest rival product in the same segment, and companies commonly confused as the manufacturer. For Iranian products use other Iranian manufacturers in the same sector. Never use a company from an unrelated industry. 16. owned_by: Use companies that attempted or were rumored to acquire the subject, companies that own the closest competing product, the previous owner before an acquisition, and companies in the same investment portfolio or sector. For Iranian entities use same-sector Iranian organizations. Never use a company from an unrelated sector. 17. language_of_work_or_name: Use languages historically tied to the work's setting or author's world, the language of the country where the work is primarily set, and languages the author spoke but did not write in. For works with significant passages in another language, exclude that language. Use the language of the most famous translation and languages commonly but incorrectly assumed for the work. 18. place_of_birth: Use other cities in the same country associated with the person, cities the person grew up in or is most famous for, and cities commonly but incorrectly believed to be the birthplace. For Iranian subjects use other major cities in the same province or region. Use birthplaces of closely related or commonly confused figures and the city the person is most associated with if different from the birthplace. 19. place_of_death: Use cities the person lived in near the end of their life, cities of exile or emigration, and the person's birthplace if commonly confused with place of death. For Iranian figures who died abroad include Tehran and the exile city. For historical scholars use major cities of their intellectual world, and use cities where the person had their last major public appearance. 20. location_of_formation: Use the organization's current headquarters if different from the formation location, cities where key negotiations or predecessor events took place, and cities of predecessor organizations. For Iranian institutions use cities with similar functional profiles. Use cities commonly but incorrectly believed to be the founding location and cities where the organization held its first major event. General rules: Never include any correct answer in candidates. If a triple has multiple correct answers, exclude all of them. Candidates must be of the same type as the correct answer.

Figure 7: The full prompt used to generate hard distractor candidates for each relation type

Model

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

EN-Base FA-Base CPT-Full CPT-CoocRM CPT-SubjRM CPT-SubjRP CPT-RandRM

51.1±4.4 31.8±0.7 49.5±3.3 50.6±3.6 49.5±3.5 49.9±3.9 51.2±3.0

25.4±1.3 34.4±2.2 46.2±3.0 40.0±2.6 31.2±2.8 28.2±2.9 45.3±3.9

13.3±1.7 12.5±1.1 28.2±2.7 24.3±2.4 18.7±2.3 16.1±1.5 29.4±2.7

25.9±1.3 39.3±3.6 57.0±3.4 48.1±3.9 37.8±3.5 32.4±2.8 57.5±3.6

24.8±3.6 32.0±2.1 35.6±2.5 31.7±2.3 24.9±2.0 24.0±3.6 32.4±5.0

4.3±1.3 11.7±1.8 21.2±2.1 19.8±1.8 12.7±3.0 10.4±3.8 22.8±1.5

Table 4: Fact recall and cross-lingual transfer across model variants using a descriptive template with hard candidates. Hard candidates are semantically close distractors that suppress shallow cues. ACCEN and ACCFA report top-1 accuracy in English and Persian; OVERALL is the fraction of triples answered correctly in both. TRANSFER is Persian accuracy conditioned on English correctness; NON-TRANSFER conditions on English incorrectness. RANKC is the Spearman rank correlation between English and Persian candidate orderings.

Model

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

EN-Base FA-Base CPT-Full CPT-CoocRM CPT-SubjRM CPT-SubjRP CPT-RandRM

51.2±0.7 29.6±0.9 50.6±1.5 51.4±0.5 50.4±1.2 49.6±1.8 52.1±0.8

24.3±0.8 30.5±0.9 36.4±1.1 32.0±1.3 28.2±0.8 31.1±0.6 36.0±0.7

14.8±1.1 8.7±0.2 23.6±0.8 21.1±1.1 18.0±0.7 19.6±1.3 23.9±0.7

29.0±1.9 29.5±0.9 46.6±1.6 41.0±1.8 35.7±0.6 39.5±1.6 45.8±1.4

19.3±0.9 31.0±1.1 25.9±1.8 22.3±1.0 20.5±0.9 22.8±1.2 25.3±0.9

3.1±1.3 6.6±0.6 23.3±1.5 22.4±2.0 14.5±1.8 16.4±2.0 21.2±1.9

Table 5: Fact recall and cross-lingual transfer across model variants using a question answering template with hard candidates. Hard candidates are semantically close distractors that suppress shallow cues. ACCEN and ACCFA report top-1 accuracy in English and Persian; OVERALL is the fraction of triples answered correctly in both. TRANSFER is Persian accuracy conditioned on English correctness; NON-TRANSFER conditions on English incorrectness. RANKC is the Spearman rank correlation between English and Persian candidate orderings.

Model

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

EN-Base FA-Base CPT-Full CPT-CoocRM CPT-SubjRM CPT-SubjRP CPT-RandRM

66.2±2.2 37.6±1.6 65.2±3.4 65.2±2.7 65.5±2.6 63.9±3.7 65.5±2.4

29.6±1.7 41.5±3.3 59.2±3.2 52.3±5.2 40.0±2.9 33.7±3.0 58.0±4.4

20.8±1.8 19.0±2.3 44.6±4.3 39.7±5.0 30.9±2.9 24.3±1.7 44.2±4.8

31.4±1.9 50.5±4.8 68.4±5.2 60.9±7.0 47.2±3.9 38.2±4.2 67.4±5.6

25.9±4.7 36.0±3.1 42.1±2.5 36.0±2.4 26.2±2.6 26.0±2.0 40.0±1.9

10.0±2.2 16.7±1.2 35.9±2.0 31.3±2.8 20.2±1.9 18.8±3.2 34.3±2.7

Table 6: Fact recall and cross-lingual transfer across model variants using a descriptive template with random candidates. Random candidates are sampled uniformly from the same relation. ACCEN and ACCFA report top-1 accuracy in English and Persian; OVERALL is the fraction of triples answered correctly in both. TRANSFER is Persian accuracy conditioned on English correctness; NON-TRANSFER conditions on English incorrectness. RANKC is the Spearman rank correlation between English and Persian candidate orderings.

Model

AccEN

AccFA

Overall

Transfer

Non-Transfer

RankC

EN-Base FA-Base CPT-Full CPT-CoocRM CPT-SubjRM CPT-SubjRP CPT-RandRM

65.6±1.3 36.2±0.5 64.2±1.6 66.4±1.9 63.6±1.1 64.5±1.3 65.0±0.9

26.4±0.7 34.1±0.4 48.2±1.0 42.7±2.0 38.0±1.1 43.7±0.9 47.8±1.5

18.5±0.7 13.2±0.6 37.9±0.6 34.6±1.9 29.3±1.3 32.9±0.9 37.0±1.2

28.2±1.0 36.6±1.4 59.0±1.9 52.1±1.7 46.1±2.1 51.1±0.7 56.9±1.5

23.2±2.6 32.7±0.8 29.0±2.2 24.1±2.0 24.0±3.1 30.3±3.1 31.1±2.3

9.4±1.1 13.5±0.5 33.7±1.1 34.0±2.2 21.1±1.7 24.6±1.7 32.2±1.5

Table 7: Fact recall and cross-lingual transfer across model variants using a question answering template with random candidates. Random candidates are sampled uniformly from the same relation. ACCEN and ACCFA report top-1 accuracy in English and Persian; OVERALL is the fraction of triples answered correctly in both. TRANSFER is Persian accuracy conditioned on English correctness; NON-TRANSFER conditions on English incorrectness. RANKC is the Spearman rank correlation between English and Persian candidate orderings.

Related documents

• PERK: Long-Context Reasoning as Test-Time Learning#799870
• Superexplanations for counterfactual knowledge#750699
• Sufficient Conditions for Counterfactual Transitivity and Antecedent Strengthening#103253
• Repair Before Reinforce: Context-Augmented Knowledge Graph Reasoning for Multi-Hop Question Answering#1000193
• What Should an Agent Forget? Separating What Is Stored from What Is Used#997161