MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts

M ULTI G HOST B ENCH: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts Matteo Greco1,2∗ , Anudeex Shetty1∗ , Andrea Tagarelli2 , and Jey Han Lau1 1 School of Computing and Information Systems, The University of Melbourne, Australia 2 DIMES Dept., University of Calabria, Italy matteogre01@gmail.com, {anudeex, laujh}@unimelb.edu.au, tagarelli@dimes.unical.it

arXiv:2609.02379v1 [cs.CL] 2 Sep 2026

Abstract While existing work on LLM authorship attribution (AA) has made progress, available benchmarks remain limited, often focusing on English, controlled settings, or relatively outdated models, with the few multilingual studies considering only relatively short texts. We introduce M ULTI G HOST B ENCH, a multilingual benchmark comprising 928 books generated by five recent LLMs across six languages and three scripts, with an average length of approximately 59K words per book. The benchmark supports evaluation under domain, author, and language shifts. Evaluation of representative AA methods shows that no single method consistently performs best across settings, and performance generally degrades under distribution shifts. Transformer-based detectors can retain generator-related information across languages, although transfer effectiveness varies by language pair, whereas statistical and fingerprintbased detectors are more language-dependent. We envision M ULTI G HOST B ENCH as a valuable resource for the development and evaluation of robust AA methods.1

1

Introduction

Large language models (LLMs) can now generate high-quality text across multiple languages (Qin et al., 2024; Yue et al., 2025), achieving levels of fluency and coherence comparable to humanwritten content (Jakesch et al., 2023). These advances create new opportunities for communication, creativity, and productivity (Ahuja et al., 2023; Doddapaneni et al., 2025). Still, they also raise concerns about transparency, accountability, misuse, and the authenticity of digital content (Brundage et al., 2024; Rudolph et al., 2023; Russell et al., 2026). These risks motivate the need for robust *

Equal contributions. The dataset and code can be found at https://github. com/GrecoMT/MultiGhostBench. 1

methods to detect and attribute LLM-generated text (Huang et al., 2025; Wu et al., 2025). Although prior research has explored LLMgenerated text detection and numerous datasets have been introduced, several limitations remain as outlined in Table 1. First, most formulate detection as a binary human-versus-AI problem (He et al., 2024; Dugan et al., 2024; Thai et al., 2026), while only a limited number of works study the more challenging authorship attribution (AA) task (Uchendu et al., 2021; La Cava and Tagarelli, 2025; Shetty et al., 2026), where the objective is to identify the specific generator LLM that wrote a text. Second, existing datasets predominantly focus on short-form texts (Macko et al., 2025, 2023), despite the emergence of LLM-ghostwritten books (Australian Society of Authors, 2025). Even recent longform efforts, such as Shetty et al. (2026), remain monolingual or limited to a small number of languages (Dugan et al., 2024; Tao et al., 2024). Third, methods are rarely evaluated for multiple distribution shifts, particularly cross-language transfer or generalisation to unseen generators and domains. In this paper, we introduce M ULTI G HOSTB ENCH, a multilingual benchmark for LLM AA that evaluates generalisation under domain, generator, and language shifts. To the best of our knowledge, M ULTI G HOST B ENCH is the first benchmark designed to jointly evaluate multilingual long-form LLM AA under multiple distribution shifts. To summarise, our main contributions are: • We introduce M ULTI G HOST B ENCH, comprising 928 books (average 59K words) generated by five recent LLMs. It covers six languages and three scripts, and is designed to evaluate AA both within and across languages under domain and generator shifts. • We conduct a comprehensive evaluation of statistical, supervised, and fingerprint-based attribution detectors, finding that no method

2

Related Work

Authorship Attribution. Research on LLMgenerated text has predominantly focused on binary detection (Mitchell et al., 2023; Yang et al., 2024; Thai et al., 2026), with only a few works (Uchendu et al., 2021; La Cava and Tagarelli, 2025; Shetty et al., 2026) studying the AA task, where the goal is to identify the specific LLM that generated a text. However, these studies are predominantly based on supervised methods with limited generalisation. Shetty et al. (2026) studied long-form texts and robustness under domain and unseengenerator shifts, but their study remained restricted to English. They also developed a lightweight fingerprint-based method, TRACE, for the AA task. Multilingual Authorship Attribution. Similarly, research on multilingual LLM-generated text has primarily focused on binary detection (Wu et al., 2026; Macko and Kopál, 2026; Wang et al., 2024) rather than attribution. MULTIT U DE (Macko et al., 2023) evaluates cross-language attribution performance, but its evaluation is not exhaustive, as it does not cover all languages in the dataset. Likewise, attribution experiments in M4GT-B ENCH (Wang et al., 2024) focus on cross-domain rather than cross-lingual generalisation. More recently, La Cava et al. (2026) provide a systematic study of multilingual attribution using MULTIT U DE and M ULTI S OCIAL. However, their primary setting is based on short news articles. This misses practical scenarios involving unseen generators and joint evaluation across multiple dimensions as done in this work, which we summarise in Table 1.

3

Dataset

Our proposed M ULTI G HOST B ENCH is a multilingual dataset of long-form texts generated by five recent LLMs (gemini-pro, gemini-flash, deepseek-v3.2, qwen3-235b, and gpt-oss; more

LLMs

Long?

• Our analysis reveals that Transformer-based detectors can retain generator-related information across languages, although the extent of this transfer varies across language pairs, whereas statistical and fingerprint-based approaches are more strongly affected by language.

Dataset

# Lang

performs consistently best across languages, data regimes, and distribution shifts.

✗ 1 ✓ 1 ✗ 1 ✗ 12 ✓ 11 ✓ 22

✗ ✗ ✓ ✗ ✗ ✗

<200 160K <500 497K 53K 325 <500 87K <512 74K <200 472K

✓

59K

OOD

# Rec. D A L T URING B ENCH(2021) 20 ✗ O PEN T URING(2025) 7 ✗ G HOST W RITE B.(2026) 10 ✓ M4GT-B ENCH(2024) 6 ✗ MULTIT U DE(2023) 8 ✗ M ULTI S OCIAL(2025) 8 ✗

✗ ✓ ✓ ✓ ✗ ✗

✗ ✓ ✓ ✗ ✗ ✗

M ULTI G HOST B ENCH 5 ✓

✓ ✓ ✓

6

Avg.

Num

Words

Docs

928

Table 1: Overview of existing datasets for LLMgenerated text attribution. ‘Rec.’ indicates whether benchmark includes recent high-capability LLMs; prior benchmarks include small-scale models (<10B parameters). ‘D’, ‘A’, and ‘L’ denote OOD-Domain, OODAuthor, and OOD-Language, respectively. ‘Long?’ denotes whether documents longer than 10K words.

details in Appendix Table 4) covering six languages and supporting AA under domain, author, and language shifts. A unique feature of our dataset is its language-shift setting, which enables the evaluation of whether attribution methods trained on one language can generalise to texts written in a previously unseen language. 3.1

Language Selection

M ULTI G HOST B ENCH covers six languages: Italian (IT), Spanish (ES), German (DE), English (EN), Chinese (ZH), and Russian (RU). These languages represent four language families: Romance (IT and ES), Germanic (DE and EN), SinoTibetan (ZH), and Slavic (RU). They also cover multiple writing systems, including Latin, Chinese characters, and Cyrillic scripts as well as languagespecific orthographic conventions. This selection enables evaluation of AA under both closely related and more distant language and script shifts. 3.2

Multilingual Book Generation Pipeline

Human book writing is generally hierarchical: authors typically develop a book through multiple stages, such as planning, drafting, and revision (Sun et al., 2022; Lee et al., 2025). We follow the generation process introduced by Shetty et al. (2026) that simulates this multi-stage writing process to create the LLM-generated books. The book generation pipeline begins with the definition of a set of constraints, such as genres and time period. These constraints are sourced from Project Gutenberg2 , an online library of freely available books; see Appendix Table 3 for the list of 2

https://www.gutenberg.org/

# Books

Lang.

Avg. #

Train

Dev.

Test

Total

Words

EN DE IT ES RU ZH

92 88 93 93 79 82

10 10 10 10 10 10

61 56 58 60 52 54

163 154 161 163 141 146

64.8K 44.4K 50.2K 44.9K 83.6K 69.7K

Total

527

60

341

928

59.6K

Table 2: M ULTI G HOST B ENCH statistics: number of training, development, and test books, together with the average book length (in words) for each language. See Appendix Table 5 for detailed breakdown per LLM.

genres, comprising both fiction and non-fiction genres. A Gutenberg book is usually multi-genre (e.g., LF (Literature & Fiction) + HB (History & Biographies)), therefore books in M ULTI G HOST B ENCH are also multi-genre. Each book is expanded iteratively by generating each subsequent segment conditioned on the outline, the previous segment, and a running summary of the narrative, i.e., updated after each generation step. This structure allows models to preserve global coherence while producing texts that exceed the length of a single generation step. We adapt the original prompt templates from Shetty et al. (2026) for the multilingual setting by translating them into each target language. The translations were verified by respective native-speaker volunteers; prompts can be found in Appendix A.1. We provide model details for five LLMs and quantitative statistics for these LLM-generated books in Appendix Table 4 and 7, respectively. All the books were generated consistently using the same pipeline described above to minimise pipeline-specific bias. However, some LLMspecific generation bias is inevitable, which we try to reduce through our meticulous data cleaning (§3.4) and analysis (§3.5). 3.3

OOD Dimensions

Following Shetty et al. (2026), M ULTI G HOSTB ENCH supports two OOD evaluation dimensions: OOD-Domain, which measures generalisation to unseen genres, and OOD-Author, which measures generalisation to unseen generators. We further introduce OOD-Language, which evaluates whether AA methods transfer to texts written in languages not observed during training.

In OOD-Domain, genres are partitioned into ID and OOD subsets for each generator LLM, ensuring that no genre appears in both partitions. Recall that these genres are sourced from the Gutenberg corpus comprising both fiction and non-fiction genres. Full statistics for genres in ID and OODDomain for different LLMs are provided in Appendix Table 5. In OOD-Author, we adopt a leaveone-author-out protocol: one LLM is held out at a time, methods are trained on the remaining LLMs, and evaluation is performed on books generated by the held-out model. In OOD-Language, methods are trained on one source language and evaluated on a different target language. We do not combine OOD-Language with OOD-Domain, so that the evaluation isolates the effect of language shift rather than conflating it with genre shift. 3.4

Data Cleaning

We clean the generated texts to remove generation and encoding artefacts while preserving language-specific features. Invalid control characters, markup, and non-textual symbols are removed, while formatting elements are normalised, preserving language-specific punctuation, diacritics, and non-Latin scripts. Following Shetty et al. (2026), we remove 1.5K tokens from both the beginning and end of each book to reduce shortcut cues from identifier markers, such as author names, titles, or publication years. As a final sanity check, we use GlotLID (Kargaran et al., 2023), an open-source language identification model, to verify that each generated book is written in its intended target language; all books in M ULTI G HOST B ENCH match their intended target language. 3.5

Data Exploration

Table 2 shows a summary statistics for M ULTI G HOST B ENCH, including the number of books per language across different splits. To provide a consistent estimate of document length across languages, we tokenise each book using a languagespecific tokeniser (listed in Appendix Table 6) and convert the token counts to approximate word counts using the standard 1.5 tokens-per-word conversion. Quantitative analysis. The generated books are analysed using multiple quantitative textual metrics in Appendix Table 7. We compute perplexity (PPL) as a model-based measure of text predictability (Jurafsky and Martin, 2026), n-gram-based metrics

for lexical overlap and diversity (Self-BLEU (SB), self-repetition (Self-R), and n-gram diversity (NGD)) (Zhu et al., 2018; Salkar et al., 2022; Meister et al., 2023), and compression ratio as a proxy for textual redundancy (Shaib et al., 2025). Given the multilingual nature of M ULTI G HOST B ENCH, token-based metrics are computed using languagedependent tokenisers (Appendix Table 6). For comparison, we select human-written books from the Gutenberg corpus with similar properties (full-length, single-authored works from the nineteenth or twentieth century). Overall, across languages, the generated books exhibit textual properties comparable to human-written books, particularly in lexical diversity and redundancy. However, generated books show higher S-B scores and lower PPL, indicating greater inter-book lexical similarity and higher predictability for the reference model (i.e., mGPT). For Chinese and Russian, no eligible human references were available, hence they were compared with LLM-generated books in other languages. Their metric values remain within a similar range, with some variation for Russian PPL and Chinese NGD. Human evaluation. We conduct a small-scale human evaluation on a subset of generated books as an initial validation step. Our focus is on novels generated by gemini-pro in the LF (Literature & Fiction) genre. Following prior works (Chhun et al., 2022; Wang et al., 2025; Shetty et al., 2026), the generated books are evaluated along several dimensions that capture both local linguistic quality and broader narrative structure (more in Appendix A.3). Appendix Table 8 summarises the annotator scores across passage-level and overall book-level dimensions. Overall, the results indicate that the generated books are generally fluent and coherent with strong passage-level scores, while narrativelevel quality is less consistent across languages for subjective dimensions. Although this evaluation is limited to gemini-pro-generated literature books, the findings complement our analyses above, which show similar quality trends across LLMs. Together, these results support the overall quality of books in M ULTI G HOST B ENCH.

4

Authorship Attribution Methods

The selected methods span three categories: metricbased, model-based supervised, and fingerprintbased. Their multilingual adaptations are summarised below, with further detector and implemen-

tation details in Appendix B and C, respectively. Metric-based methods. For this category, we consider RANK, ENTROPY, and G LTR (Gehrmann et al., 2019). These methods rely on token-level probabilities produced by a reference language model. Following Macko et al. (2023), we use mGPT (Shliazhko et al., 2024) as the multilingual reference model. We adapt their extracted statistics to multi-class attribution following La Cava and Tagarelli (2025); Shetty et al. (2026). Model-based supervised methods. We consider N - GRAM (Koppel and Schler, 2004), BERT- AA (Fabien et al., 2020), and DETECTIVE (Guo et al., 2024), covering both non-neural and neural detectors. For N - GRAM, we adapt preprocessing to preserve accented and language-specific characters and use Jieba3 segmentation for Chinese word-level features. For BERT- AA, we use XLM ROBERTA (Conneau et al., 2020) as the multilingual encoder. For DETECTIVE, we replace the original sentence-embedding model with a multilingual counterpart. Fingerprint-based methods. We consider TRACE (Shetty et al., 2026), a post-hoc fingerprintbased AA method that models transitions between token-rank or entropy statistics computed by an evaluator language model. A test text is attributed to the generator with the most similar fingerprint. We evaluate TRACErank-js , TRACEentr-js , and TRACE entr-norm . For multilingual evaluation, we replace the original evaluator, GPT-2, with Gemma, also evaluated by Shetty et al. (2026). Previous TRACE evaluations were limited to English.

5

Experimental Setup

Training settings. In order to analyse how AA performance varies with respect to the amount of available training data, we follow Shetty et al. (2026) considering two training scenarios: highresource setting where a relatively large number of labelled training books is available for each LLM (10–30 books), and low-resource setting where only a limited number of labelled training books is available for each LLM (1–5 books). Evaluation settings. Since M ULTI G HOSTB ENCH includes the OOD-Author dimension, the generator of a test text may not appear among those observed during training. This 3

https://pypi.org/project/jieba/

Figure 1: Best-performing detector by language and evaluation setting under the low- and high-resource regimes. Each group includes ID, OOD-Domain, and OOD-Author. Bar height shows macro-F1 , colors identify detectors, hatches indicate evaluation settings, and split colors denote ties. The best detector varies across conditions, while more data generally improves performance.

corresponds to an open-set setting, where an AA method must not only distinguish among known authors, but also reject texts generated by unseen authors. To support this rejection mechanism, all methods are calibrated on the development set by selecting a confidence threshold that best balances attribution performance and rejection ability. All the thresholds can be found in Appendix Table 9. During evaluation, the same threshold is applied across all scenarios: ID, OOD-Domain, OODAuthor, and OOD-Language. We do this because in a practical application we would not know whether a test document is in or out of the training distribution. For texts generated by known authors, predictions below the confidence threshold are counted as missed attributions. For texts generated by unseen authors, the same behaviour is instead counted as a correct rejection, since the method avoids forcing the sample into one of the known author classes. Attribution performance is evaluated using macro-F1, which gives equal weight to all generating LLMs.

6

Experimental Results

We first consider in-language evaluation (§6.1), where training and test texts are written in the same language, covering the ID, OOD-Domain, and OOD-Author settings. We then analyse crosslanguage evaluation (§6.2), where attribution methods are trained on one source language and evaluated on a previously unseen target language. Finally, we examine the learned representation (§6.3) space to better interpret the observed performance patterns under domain and language shifts.

6.1

In-Language Evaluation

Figure 1 reports the best-performing method for each language and evaluation setting; complete results are reported in Appendix Table 11 and 12. For completeness, the results obtained without confidence-threshold rejection are provided in Appendix Table 13 and 14. No universal winner. No single method consistently outperforms all others across languages, resource regimes, and evaluation settings. Nevertheless, some methods appear more frequently among the best-performing approaches. XLM - ROBERTA is particularly competitive in the ID and OODDomain settings, while N - GRAM becomes a frequent winner in the high-resource regime, especially under OOD-Author evaluation. TRACE variants also obtain the best result in several languagespecific configurations. In contrast, RANK, EN TROPY , and G LTR consistently achieve low performance, indicating that individual token-level statistics provide insufficient signals for reliable authorship attribution. More data improves performance. In the highresource setting, the larger amount of training data improves the best achievable performance across all ID and OOD-Domain configurations. In several cases, the best-performing detector approaches or reaches perfect macro-F1 . Full results (Tables Appendix Table 12 and 11) show the impact of additional training data on each detector. Under OOD-Author, XLM - ROBERTA per-

Figure 2: Cross-language macro-F1 results for XLM - ROBERTA in the (a) low-resource and (b) high-resource settings. Rows indicate source languages and columns target languages; the final row and column report averages excluding the diagonal. Transfer depends on both the source-target pair and the resource regime; Chinese remains the most challenging target.

forms worse than in the low-resource setting for Italian, Chinese, and English, whereas DETECTIVE improves across all languages. N - GRAM benefits the most from the additional data, becoming the best-performing OOD-Author detector in five languages and the second-best in Russian. An explanation is that N - GRAM needs more data to estimate reliable lexical and structural patterns, whereas XLM ROBERTA may become overconfident on texts from unseen authors, assigning them to one of the known classes instead of rejecting them. Robustness drops under OOD shifts. Compared with ID evaluation, performance generally decreases under distribution shift, although the strongest high-resource detectors retain high absolute scores. The OOD-ID differences reported in Appendix Table 15 and 16 are negative in most cases, with OOD-Author generally producing larger drops than OOD-Domain. Language differences. The low-resource setting reveals clearer performance differences across languages. Chinese achieves the strongest results, whereas English is comparatively more challenging. Indeed, the generators produce Chinese texts of more variable quality, creating stronger generatorspecific signals that attribution detectors can exploit; in contrast, English outputs’ quality may be consistently better, leading to greater stylistic overlap and weaker attribution cues. These differences become less pronounced in the high-resource regime, where most best-performing scores approach or reach perfect macro-F1 .

6.2

Cross-Language Evaluation

Figure 2 reports the cross-language macro-F1 scores of XLM - ROBERTA under the low- and highresource settings. Rows indicate the source language used for training, while columns indicate the target language used for evaluation. We focus the visual analysis on XLM - ROBERTA because it achieves the strongest overall cross-language performance. DETECTIVE, the other Transformerbased method, also transfers substantially better than the remaining approaches, but generally obtains lower scores than XLM - ROBERTA, despite outperforming it in a limited number of configurations. By contrast, the metric-based, N-GRAM, and fingerprint-based methods achieve near-zero performance in most cross-language configurations, suggesting that the signals they capture are largely language-dependent and do not transfer reliably across languages. For completeness, the full results for all methods with and without thresholding are reported in Appendix Table 17 and Table 18, respectively. Target language difficulty. Chinese is the most challenging target language. Averaging across source languages, transfer to Chinese achieves a macro-F1 of 0.508 in the low-resource setting and 0.582 in the high-resource setting, the lowest average in both regimes. Russian provides an informative contrast. Although it also uses a nonLatin script, it obtains an average target score of 0.717 in the low-resource setting and 0.841 in the high-resource setting, making it the easiest target

Figure 3: UMAP projections of XLM - ROBERTA representations after fine-tuning on Italian. The panels show evaluation on (a) Italian in-distribution (ID), (b) Italian OOD-Domain, and (c) Spanish and (d) Chinese ID texts evaluated under OOD-Language. Generator-specific clusters remain visible in all settings, with clearer separation for Spanish than under domain shift or for Chinese.

language in the latter regime. Therefore, the difficulty of transferring to Chinese cannot be explained by script differences alone. Broader typological differences, tokenisation behaviour, and languagespecific distributional properties may also contribute to the observed performance gap. In particular, the distinctive NGD values observed for Chinese in the quantitative analysis may reflect distributional characteristics that make cross-language transfer more difficult, although these descriptive metrics do not directly establish the source of the performance degradation. Source language effects. The effectiveness of a source language varies substantially across resource regimes. In the low-resource setting, Spanish provides the strongest overall transfer, with an average macro-F1 of 0.866 across target languages, followed by Chinese with 0.823. German is the weakest source language, with an average of 0.435. This ranking changes in the high-resource setting: Italian becomes the strongest source language, with an average macro-F1 of 0.894, followed by Spanish with 0.819, whereas Russian becomes the weakest at 0.410. Chinese also drops markedly, from 0.823 to 0.629, becoming the second-weakest source language in the high-resource regime. In particular, the deterioration observed for Chinese and Russian suggests that, in these languages, training with more data may lead to overfitting to sourcelanguage-specific patterns, reducing the transferability of the learned authorship signals to other languages. The language-specific differences observed in Chinese NGD and Russian PPL may also reflect underlying distributional properties that contribute to this behaviour, although these metrics do not directly explain the transfer degradation.

Language family effects. The clearest evidence of a language-family effect concerns Italian and Spanish. In the low-resource setting, Spanish provides the best transfer to Italian, with a macro-F1 of 0.926, while Italian is the second-best source for Spanish, reaching 0.750. With more training data, the relationship becomes fully reciprocal: Spanish-to-Italian reaches 0.857, while Italianto-Spanish achieves 0.981, the highest score in the table. The Germanic pair (English and German) shows a weaker and less symmetric pattern. English is only the fourth-best source language for German in the low-resource setting, but becomes the strongest source in the high-resource setting, reaching a macro-F1 of 0.916. Conversely, German-to-English achieves only 0.420 in the low resource regime, but improves to 0.673 with more training data, becoming the third-best source language. Overall, language-family relatedness appears to favour transfer, particularly for the Romance pair, but it does not fully determine crosslanguage performance. 6.3

Representation Analysis

To better understand the behaviour of XLM ROBERTA and TRACE beyond their attribution scores, we inspect their respective representation spaces. We use UMAP (McInnes and Healy, 2018) to visualise how XLM - ROBERTA embeddings and TRACE fingerprints change across the ID, OODDomain, and OOD-Language settings. The two methods offer complementary perspectives on the attribution task: XLM - ROBERTA relies on learned representations, whereas TRACE operates in a statistical fingerprint space. Comparing these spaces allows us to examine how the two approaches are affected by distribution shifts.

XLM - ROBERTA embeddings. We focus on the representation space learned by XLM - ROBERTA in the high-resource setting, using the model finetuned on Italian, which achieves the strongest average cross-language performance. We consider embeddings from Italian ID texts, Italian OODDomain texts, and Spanish and Chinese texts evaluated in the OOD-Language setting to examine the effects of domain and language shifts. Figure 3 illustrates the embedding space produced by XLM ROBERTA across the four evaluation settings. In the ID setting, texts generated by the same model form compact, well-separated clusters. Under OODDomain, this organisation is only partially preserved, with clusters becoming less compact and exhibiting greater overlap. This pattern is consistent with the decrease in macro-F1 from 0.956 in the ID setting to 0.676 under OOD-Domain, indicating that domain shift makes the learned representations less discriminative. When the Italian-fine-tuned model is evaluated on Spanish texts, the modelspecific structure is largely preserved, with compact and well-separated clusters similar to those observed in the Italian ID setting. This is consistent with the strong Italian-to-Spanish transfer performance and suggests that the learned generatorrelated representations generalise effectively between closely related languages. In contrast, when the same model is evaluated on Chinese texts, a model-specific organisation remains visible, but the clusters are broader and less clearly separated than those observed for Spanish. This suggests that the learned representations retain generator-related information across languages, although their discriminative structure is increasingly affected by larger typological and script differences.

fingerprints. We next analyse the entropy-based fingerprint space of TRACE in the high-resource setting, using Italian as the reference language. While TRACE remains comparatively stable in the same-language settings, its performance drops sharply under OOD-Language evaluation. Since attribution is performed by comparing distances between test and training fingerprints, their relative position provides insight into this behaviour. Figure 4 compares Italian training, ID-test, and OOD-Domain fingerprints with fingerprints from different target languages. Italian ID-test and OOD-Domain samples remain close to the training distribution, whereas fingerprints from other languages occupy clearly displaced reTRACE

gions. Notably, this pattern is observed not only for Chinese, which is linguistically distant from Italian, but also for Spanish, a closely related Romance language. Consequently, cross-language samples lack nearby reference fingerprints, making distance-based attribution less reliable. This suggests that the fingerprint space is strongly influenced by language-specific token transition patterns, which helps explain the poor cross-language performance of TRACE.

7

Conclusion

We introduced M ULTI G HOST B ENCH, a multilingual benchmark for LLM AA comprising 928 longform books generated by five recent LLMs, and covering six languages from four language families and three scripts. Importantly, it is designed to evaluate generalisation under domain, generator, and language shifts. For the within-language setting (training and testing on the same language), no single method consistently outperforms the others across languages, data regimes, or the ID, OOD-Domain, and OOD-Author conditions. In the cross-language setting (OOD-Language), however, Transformer-based approaches (XLM - ROBERTA and DETECTIVE) substantially outperform other methods. Interestingly, TRACE and metric-based methods drop to near-zero performance. Further analysis shows that cross-language performance is influenced by source and target language, languagefamily proximity, and script differences. These results highlight limitations of current methods in multilingual settings. We hope M ULTI G HOSTB ENCH will serve as a useful benchmark for developing robust attribution methods.

Limitations Although M ULTI G HOST B ENCH covers six languages spanning four language families and three writing scripts, it does not capture the full diversity of multilingual LLM-generated text. In particular, it does not cover low-resource languages, languages underrepresented in LLM pre-training data, or languages with substantially different grammatical and morphological characteristics. Furthermore, we consider five recent LLMs capable of generating long-form content across the selected languages. Extending the benchmark to additional and future LLMs is an important direction for future work. Our human evaluation is also limited to a small subset of generated books. Moreover, although the

generated books are generally of good quality, multilingual long-form generation remains challenging, particularly with respect to originality, selfevaluation, revision, etc. across languages. These aspects are beyond the scope of this work. Finally, our study focuses on attribution under single-author and non-adversarial settings. Extensions such as mixed authorship (i.e., a book written by multiple authors) and adversarial generalisation (such as obfuscation attacks) remain important directions for future research.

Ethical Considerations M ULTI G HOST B ENCH consists of LLM-generated books for multiple languages. As with any creative text, some books may contain writing or plot elements that reflect undesirable values. However, all LLMs used in this work incorporate built-in safety guardrails, reducing the likelihood of generating such content. This is further supported by our human evaluation, where no such concerning content was identified. Overall, we aim to adhere to the ACL Code of Ethics.4

Acknowledgements This research was conducted during M. Greco’s visit at the University of Melbourne, funded by the Erasmus+ Programme. We thank the volunteer annotators from the University of Melbourne and the University of Calabria for their contribution to the human evaluation. This research was supported by The University of Melbourne’s Research Computing Services and the Petascale Campus Initiative. A. Shetty was supported by the Commonwealth through an Australian Government Research Training Program Scholarship (DOI: https://doi.org/10.82133/C42F-K220), Computing and Information Systems PhD Scholarship, and the Avashya Foundation. A. Tagarelli was partly supported by the Horizon Europe project AI-CODE, GA No. 101135437. J.H. Lau was supported by the Australian Research Council under Grant LP210200917 and DP240101006.

References Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. 4

https://www.aclweb.org/portal/content/ acl-code-ethics

MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics. Australian Society of Authors. 2025. Artificial intelligence. [Accessed 13-02-2026]. Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, SJ Beard, Haydn Belfield, Sebastian Farquhar, and 7 others. 2024. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. Preprint, arXiv:1802.07228. José Cañete, Gabriel Chaperon, Rodrigo Fuentes, JouHui Ho, Hojin Kang, and Jorge Pérez. 2023. Spanish pre-trained bert model and evaluation data. Preprint, arXiv:2308.02976. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics. Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5794–5836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai,

and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.

Kaijie Jiao, Quan Wang, Licheng Zhang, Zikang Guo, and Zhendong Mao. 2025. M-RangeDetector: Enhancing generalization in machine-generated text detection through multi-range attention masks. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8971–8983, Vienna, Austria. Association for Computational Linguistics.

Sumanth Doddapaneni, Gowtham Ramesh, Mitesh Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2025. A primer on pretrained multilingual language models. ACM Comput. Surv., 57(9).

Angel B John and Sumam Mary Idicula. 2026. Authorship attribution of malayalam biblical texts using deep learning. In 2026 2nd International Conference on Advances in Intelligent Computing and Applications (AICAPS), pages 1–6.

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machinegenerated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12463– 12492, Bangkok, Thailand. Association for Computational Linguistics. Maël Fabien, Esau Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020. BertAA : BERT finetuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 127–137, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI). Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computational Linguistics. Xun Guo, Yongxin He, Shan Zhang, Ting Zhang, Wanquan Feng, Haibin Huang, and Chongyang Ma. 2024. Detective: Detecting ai-generated text via multi-level contrastive learning. Advances in Neural Information Processing Systems, 37:88320–88347. Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2024. Mgtbench: Benchmarking machine-generated text detection. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 2251–2265, New York, NY, USA. Association for Computing Machinery. Baixiang Huang, Canyu Chen, and Kai Shu. 2025. Authorship attribution in the era of llms: Problems, methodologies, and challenges. SIGKDD Explor. Newsl., 26(2):21–43. Maurice Jakesch, Jeffrey T. Hancock, and Mor Naaman. 2023. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120.

Daniel Jurafsky and James H. Martin. 2026. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition. Online manuscript released January 6, 2026. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. Moshe Koppel and Jonathan Schler. 2004. Authorship verification as a one-class classification problem. In Proceedings of the twenty-first international conference on Machine learning, page 62. Yuri Kuratov and Mikhail Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for russian language. Preprint, arXiv:1905.07213. Lucio La Cava, Dominik Macko, Robert Moro, Ivan Srba, and Andrea Tagarelli. 2026. Authorship attribution in multilingual machine-generated texts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 45136–45152, San Diego, California, United States. Association for Computational Linguistics. Lucio La Cava and Andrea Tagarelli. 2025. OpenTuringBench: An open-model-based benchmark and framework for machine-generated text detection and attribution. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26655–26671, Suzhou, China. Association for Computational Linguistics. Yukyung Lee, Soonwon Ka, Bokyung Son, Pilsung Kang, and Jaewook Kang. 2025. Navigating the path of writing: Outline-guided text generation with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 233–250, Albuquerque, New Mexico. Association for Computational Linguistics. Dominik Macko and Jakub Kopál. 2026. CEAID: Benchmark of multilingual machine-generated text detection methods for Central European languages.

In Findings of the Association for Computational Linguistics: ACL 2026, pages 29177–29190, San Diego, California, United States. Association for Computational Linguistics. Dominik Macko, Jakub Kopál, Robert Moro, and Ivan Srba. 2025. MultiSocial: Multilingual benchmark of machine-generated text detection of social-media texts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 727–752, Vienna, Austria. Association for Computational Linguistics. Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9960–9987, Singapore. Association for Computational Linguistics. HF Canonical Model Maintainers. 2022. gpt2 (revision 909a290). Leland McInnes and John Healy. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102–121. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International conference on machine learning, pages 24950–24962. PMLR. Omar Momen, Manuel Schaaf, and Alexander Mehler. 2025. Filling the temporal void: Recovering missing publication years in the Project Gutenberg corpus using LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17318–17334, Vienna, Austria. Association for Computational Linguistics. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gptoss-20b model card. Preprint, arXiv:2508.10925. Riccardo Orlando, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, and Roberto Navigli. 2024. Minerva LLMs: The first family of large language models trained from scratch on Italian data. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 707–719, Pisa, Italy. CEUR Workshop Proceedings.

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. Preprint, arXiv:2404.04925. Jurgen Rudolph, Samson Tan, and Shannon Tan. 2023. Chatgpt: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning & Teaching, 6(1):342–363. Jenna Russell, Marzena Karpinska, Destiny Akinode, James Zhou, Katherine Thai, Bradley Emi, Max Spero, and Mohit Iyyer. 2026. AI use in American newspapers is widespread, uneven, and rarely disclosed. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14554–14580, San Diego, California, United States. Association for Computational Linguistics. Nikita Salkar, Thomas Trikalinos, Byron Wallace, and Ani Nenkova. 2022. Self-repetition in abstractive neural summarizers. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 341–350, Online only. Association for Computational Linguistics. Chantal Shaib, Venkata S Govindarajan, Joe Barrow, Jiuding Sun, Alexa Siu, Byron C Wallace, and Ani Nenkova. 2025. Standardizing the measurement of text diversity: A tool and comparative analysis. In Proceedings of The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, pages 36–46, Mumbai, India. Association for Computational Linguistics. Anudeex Shetty, Qiongkai Xu, Olga Ohrimenko, and Jey Han Lau. 2026. Who wrote the book? detecting and attributing llm ghostwriters. Preprint, arXiv:2603.28054. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-shot learners go multilingual. Transactions of the Association for Computational Linguistics, 12:58–79. Xiaofei Sun, Zijun Sun, Yuxian Meng, Jiwei Li, and Chun Fan. 2022. Summarize, outline, and elaborate: Long-text generation via hierarchical supervision from extractive summaries. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6392–6402, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Zhen Tao, Yanfang Chen, Dinghao Xi, Zhiyu Li, and Wei Xu. 2024. Towards reliable detection of llmgenerated texts: A comprehensive evaluation framework with cudrt. Preprint, arXiv:2406.09056.

Katherine Thai, Bradley Emi, Elyas Masrour, and Mohit Iyyer. 2026. Editlens: Quantifying the extent of AI editing in text. In The Fourteenth International Conference on Learning Representations. Jacob Tyo, Bhuwan Dhingra, and Zachary C. Lipton. 2023. Valla: Standardizing and benchmarking authorship attribution and verification through empirical evaluation and comparative analysis. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 649–660, Nusa Dua, Bali. Association for Computational Linguistics. Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2001–2016, Punta Cana, Dominican Republic. Association for Computational Linguistics. Wenqing Wang, Mingqi Gao, Xinyu Hu, and Xiaojun Wan. 2025. Towards a “novel” benchmark: Evaluating literary fiction with large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21648–21673, Vienna, Austria. Association for Computational Linguistics. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024. M4GTbench: Evaluation benchmark for black-box machinegenerated text detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3964– 3992, Bangkok, Thailand. Association for Computational Linguistics. Junchao Wu, Yefeng Liu, Chenyu Zhu, Hao Zhang, Zeyu Wu, Tianqi Shi, Yichao Du, Longyue Wang, Weihua Luo, Jinsong Su, and Derek F. Wong. 2026. DetectRL-X: Towards reliable multilingual and realworld LLM-generated text detection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38247–38294, San Diego, California, United States. Association for Computational Linguistics. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, 51(1):275–338. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. 2024. DNA-GPT: Divergent n-gram analysis for trainingfree detection of GPT-generated text. In The Twelfth International Conference on Learning Representations. Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, JEAN NYANDWI, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2025. Pangea: A fully open multilingual multimodal llm for 39 languages. In International Conference on Learning Representations, volume 2025, pages 47758–47811. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1097–1100, New York, NY, USA. Association for Computing Machinery.

Appendix A

M ULTI G HOST B ENCH Generation Tag

Genre

ACM BLP HB JR LF LHH SSP ST

Art, Culture & Media Business, Law & Politics History & Biographies Journals & Reports Literature & Fiction Lifestyle, Health & Hobbies Social Sciences & Philosophy Science & Technology

Table 3: Different genres used in this work, sourced from Project Gutenberg (Momen et al., 2025).

A.1

Book Generation Prompts

For brevity, they are not listed here; they can be found in https://github.com/GrecoMT/ MultiGhostBench/blob/main/prompts.py. A.2

Quantitative Analysis

We evaluate the generated books using several reference-free metrics capturing lexical diversity, repetition, and redundancy. Given the multilingual setting, token-based metrics are computed using language-specific tokenisers to reduce distortions caused by differences in tokenisation across languages. The analysis includes the following metrics: • Self-BLEU (S-B): measures the lexical similarity between books generated by the same model by computing BLEU scores between each book and the remaining outputs (Zhu et al., 2018). Higher values indicate greater overlap and, consequently, lower inter-book diversity. • N-gram Diversity (NGD): measures the variety of lexical patterns in the generated texts based on the proportion of distinct n-grams (Meister et al., 2023). We consider n-grams of length four, with higher values indicating greater diversity. • Self-Repetition (Self-R): measures the extent to which n-grams are repeated within the generated texts, capturing the tendency of a model to reuse the same lexical patterns (Salkar et al., 2022). Lower values indicate less repetition.

• Compression Ratio (CR): measures text redundancy through gzip compression (Shaib et al., 2025). More repetitive and predictable texts are generally more compressible, making this metric complementary to n-grambased measures. • Perplexity (PPL): measures how predictable a text is according to a reference language model. We compute perplexity using mGPT (Shliazhko et al., 2024), providing a shared multilingual evaluation model across languages. Lower values indicate that the text is more predictable for the reference model. A.3

Human Evaluation

Following Shetty et al. (2026); Chhun et al. (2022); Wang et al. (2025), we evaluate the generated books along several dimensions capturing linguistic, narrative, and structural properties at the micro-, meso, and macro-levels: • Relevance (Rel.): measures the degree to which the generated text follows the given instructions and remains consistent with the specified book outline. • Engagement (Eng.): measures the extent to which the text sustains the reader’s interest and attention. • Coherence (Coh.): evaluates how logically the narrative develops, considering the consistency of events, characters, and settings, as well as the progression from beginning to end. • Fluency (Flu.): measures the grammatical correctness, readability, and naturalness of the language. • Diversity (Div.): measures the degree of variation in vocabulary, sentence structure, and linguistic expression. • Empathy (Emp.): evaluates how effectively the text conveys the characters’ emotions, motivations, and reactions. • Surprise (Sur.): measures the degree to which the ending introduces unexpected or non-trivial narrative developments. • Complexity (Comp.): measures the elaborateness of the narrative in terms of plot structure, character development, and overall sophistication.

Model Name

Context

Type

Checkpoint

deepseek-v3.2 (DeepSeek-AI et al., 2025) gpt-oss (OpenAI et al., 2025) qwen3-235b (Yang et al., 2025)

163.8K 128.0K 262.1K

 Open  Open  Open

deepseek/deepseek-v3.2

gemini-flash (Comanici et al., 2025) gemini-pro (Comanici et al., 2025)

1.05M 1.05M

 Closed  Closed

google/gemini-flash

openai/gpt-oss-120b qwen/qwen3-235b-a22b-2507

google/gemini-pro

Table 4: LLMs checkpoints used in M ULTI G HOST B ENCH. ‘Context’ is the total input context supported. ‘Type’ indicates whether model weights are public ( Open) or if it is proprietary LLM ( Closed).

Model

Split

Genres

IT

ZH

ES

DE

EN

RU

gemini-pro

ID ID OOD-Domain

LHH, HB HB LF

18 11 8

8 9 6

10 12 6

18 6 7

6 11 5

11 10 6

gemini-flash

ID OOD-Domain

HB, ACM LHH, LF

30 8

16 5

30 8

25 7

31 6

19 6

deepseek-v3.2

ID OOD-Domain

LF HB

19 5

29 8

27 7

23 6

28 8

23 7

qwen3-235b

ID ID OOD-Domain OOD-Domain OOD-Domain

SSP, LF LF BLP, HB HB BLP, LHH, HB

8 9 2 1 1

13 12 1 2 4

8 11 1 2 3

18 6 2 1 4

10 10 4 3 1

8 11 2 2 2

gpt-oss

ID OOD-Domain

HB, LF BLP, HB, SSP

24 7

18 5

22 6

16 5

24 6

18 5

Table 5: Detailed ID and OOD-Domain genre splits by generating model, with the number of books reported for each language. Development-set books are excluded from this breakdown.

Lang.

Tokeniser

Reference

IT DE ES ZH EN RU

M INERVA -7B GBERT BETO C HINESE -BERT GPT-2 RUSSIAN -BERT

(Orlando et al., 2024) (Chan et al., 2020) (Cañete et al., 2023) (Devlin et al., 2019) (Maintainers, 2022) (Kuratov and Arkhipov, 2019)

Table 6: Language-dependent tokenisers used for tokenbased quantitative textual metrics.

Annotators. The evaluation was conducted by volunteer annotators from our university, representing diverse demographic backgrounds. Each book was independently evaluated by two annotators, both fluent in the language of the evaluated text. Annotators received detailed instructions describing the task and evaluation criteria. Participation was voluntary and uncompensated, and all annota-

tors were informed about the purpose of the study and their role in the evaluation. Annotation task. Given the length of the generated books, five passages were randomly sampled from each book while preserving the narrative flow. Annotators answered five questions per passage and five book-level questions, for a total of 30 questions per book, using a 5-point Likert scale. Results are reported in Table 8. The evaluation provides a limited assessment of the linguistic and narrative quality of the generated books, with potential confounding factors including annotator subjectivity and the restricted coverage of models and genres. Results. Table 8 reports the average passagelevel and overall scores. The generated books achieve strong passage-level results for relevance (4.66), coherence (4.45), fluency (4.15), and diversity (4.18), while engagement is lower (3.73).

At the overall level, coherence receives the highest score (4.16), whereas the more subjective dimensions of engagement (3.25) and surprise (3.16) are weaker, indicating more predictable narratives. These results complement the automatic textual analysis (§3.5) and indicate that the generated passages are generally fluent, coherent, diverse, and aligned with the intended narrative constraints. However, the results vary across languages. In particular, Russian books obtain lower ratings across several dimensions but, importantly, maintain high relevance (4.9). Inter-annotator agreement remains low (avg. 0.214) for a highly subjective task, varying across languages. Nevertheless, most paired ratings remain close across languages on avg. 84.4% by at most one point. The limited scale of the evaluation prevents broader language-specific conclusions. Overall, the results indicate adequate linguistic and narrative quality of books in M ULTI G HOST B ENCH. A.4

Cost Analysis

Appendix Table 10 lists generation costs per LLM. In total, generating M ULTI G HOST B ENCH cost approximately $972. LLM gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

API Cost ($/1M)

Total

Input

Output

(in $)

1.25 0.30 0.20 0.09 0.03

10.00 2.50 0.31 0.55 0.17

760.0 61.6 58.1 45.7 46.4

Total

971.8

Table 10: M ULTI G HOST B ENCH generation cost breakdown. ‘Input’ and ‘Output’ denote API cost per 1M tokens. ‘Total’ is the cost of generating all books for a given LLM. OpenRouter API costs are as of April 2026.

B Authorship Attribution Methods (cont.) RANK .

For each text, we compute the average rank of the observed tokens. ENTROPY . Similarly, we compute the average token-level entropy. G LTR. G LTR (Gehrmann et al., 2019) represents a text through the proportion of tokens falling into four predefined rank buckets.

For all three metric-based detectors, we follow La Cava and Tagarelli (2025); Shetty et al. (2026) and adapt the extracted statistics to multi-class attribution by training a logistic regression classifier to predict the generating LLM. Following Macko et al. (2023), we use mGPT (Shliazhko et al., 2024) as the multilingual reference language model. N - GRAM .

N - GRAM (Koppel and Schler, 2004)

uses three complementary feature representations. The char analyser preserves letters, punctuation, and spaces. The dist_char analyser replaces letters with a common placeholder, retaining structural patterns involving punctuation, numbers, symbols, and spacing. The word analyser removes punctuation and operates on word-level features. To preserve language-specific information, accented and language-specific characters are retained during preprocessing. For Chinese, word-level features are extracted after segmenting the text with Jieba.5 BERT- AA . A multi-class classification layer is added on top of the pre-trained encoder, and the complete architecture is fine-tuned to predict the generating LLM. In our implementation, the underlying encoder is XLM - ROBERTA (Conneau et al., 2020). DETECTIVE . DETECTIVE (Guo et al., 2024) relies on sentence embeddings learned through a multi-level contrastive objective. For AA, these sentence embeddings are used to train a multi-class classifier over the generating LLMs. The original sentenceembedding model is replaced with ZurichNLP/ unsup-simcse-xlm-roberta-base,6 its multilingual counterpart. We also considered the more recent M-R ANGE D ETECTOR (Jiao et al., 2025) and simpler contrastive-learning approaches (John and Idicula, 2026). However, their implementations are not publicly available, and the methodological details provided were insufficient for reliable reproduction. TRACE . For each text, TRACE (Shetty et al., 2026) models transitions between consecutive token-rank or entropy values, producing a twodimensional fingerprint. A reference fingerprint is constructed for each generator from its training texts, and a test text is attributed to the generator whose reference fingerprint is most simi5

https://pypi.org/project/jieba/ https://huggingface.co/ZurichNLP/ unsup-simcse-xlm-roberta-base 6

lar to the fingerprint extracted from the test text. TRACE rank-js uses rank-based transition finger-

prints, whereas TRACEentr-js and TRACEentr-norm use entropy-based fingerprints and compare them through Jensen-Shannon divergence and normbased distance, respectively. We use Gemma7 as the evaluator language model.

C

Experimental Details

We use OpenRouter8 for all LLM API calls used to generate the benchmark. We adopt the same decoding parameters for all generators, setting temperature to 1.0, top-p to 1.0, top-k to 0, and maximum_output_length to 8192 tokens. We provide a model card for each LLM used in this work in Table 4. All experiments were conducted using a single A100 GPU with CUDA 12.4 and PyTorch 2.10.0. Since XLM - ROBERTA cannot process complete long-form documents directly, we split each text into chunks of 512 tokens and average the chunklevel prediction scores to obtain the final documentlevel prediction, following Tyo et al. (2023); Shetty et al. (2026). For DETECTIVE, since it is clusteringbased, we compute the proportion of top-K results assigned to each generator across the text chunks. All hyperparameter tuning was performed on the development set. The number of fine-tuning epochs depends on the experimental configuration. For XLM - ROBERTA , we fine-tune the model for up to 10 epochs and retain the checkpoint achieving the highest macro-F1 on the development set. For DE TECTIVE, we train for up to 50 epochs and similarly select the checkpoint with the best developmentset macro-F1. For all TRACE variants, we follow the hyperparameter configuration of Shetty et al. (2026). Specifically, for TRACErank-js , we set α = 1.5 and use 100K samples to approximate the power-law distribution. We use Gemma as the evaluator language model with a context length of 1024 tokens. The entropy-based variants use a grid size of 50, whereas TRACErank-js uses 50 clusters.

D

Additional Results

This section provides additional analyses and complete results for all evaluation settings. Figure 4 extends the qualitative analysis of TRACE by comparing entropy fingerprints from Italian training texts with Italian ID, Italian OOD-Domain, and Spanish 7 8

https://huggingface.co/google/gemma-3-1b-it https://openrouter.ai/

and Chinese OOD-Language texts. Table 11 and Table 12 report the threshold-based in-language results for ID, OOD-Domain, and OOD-Author in the low- and high-resource settings, while Table 15 and Table 16 report the corresponding changes in macro-F1 relative to ID performance. Table 13 and Table 14 provide the ID and OOD-Domain results without threshold-based rejection. Finally, Table 17 and Table 18 report OOD-Language results across all source–target language pairs, with and without thresholding, under both resource settings.

Figure 4: UMAP projections of TRACE entropy fingerprints using Italian as the reference language. The panels compare Italian training fingerprints with (a) Italian ID-test, (b) Italian OOD-Domain, and (c) Spanish and (d) Chinese ID fingerprints evaluated under OOD-Language. Grey points denote the Italian training references, while blue, orange, and green points denote ID, OOD-Domain, and cross-language fingerprints, respectively. Marker shapes identify the LLM generator. Italian ID-test and OOD-Domain fingerprints remain close to the training references, whereas both Spanish and Chinese fingerprints occupy clearly displaced regions, indicating that TRACE fingerprints are strongly language-dependent and transfer poorly under OOD-Language.

Author

PPL ↓

S-B ↓

Self-R ↓

NGD ↑

CR ↓

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

17.48 14.99 18.31 18.97 15.59

0.016 0.017 0.027 0.019 0.019

11.36 10.12 10.59 11.26 11.32

1.38 1.58 1.48 1.37 1.33

2.75 2.86 3.14 2.75 2.92

Human

30.07

0.004

11.08

1.484

2.56

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

19.76 17.00 20.89 20.29 16.67

0.019 0.014 0.013 0.019 0.011

10.47 9.43 11.44 9.44 11.29

1.55 1.65 1.39 1.73 1.21

2.72 2.85 2.81 2.71 3.02

Human

30.01

0.004

11.04

1.39

2.73

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

20.12 16.52 21.49 23.87 18.47

0.021 0.020 0.017 0.014 0.029

10.19 9.51 11.53 10.33 11.04

1.67 1.73 1.38 1.58 1.48

2.70 2.88 2.96 2.74 2.89

Human

39.62

0.004

9.33

1.76

2.68

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

30.67 27.81 31.43 28.74 28.84

0.024 0.016 0.016 0.013 0.016

10.68 9.92 11.80 11.63 11.44

1.59 1.65 1.40 1.29 1.39

2.68 2.82 2.75 2.68 2.79

Human

31.36

0.001

10.25

2.14

2.65

RU

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

15.46 12.16 15.66 16.07 16.98

0.020 0.022 0.016 0.022 0.022

11.47 9.58 10.21 11.78 11.88

1.61 1.90 1.56 1.44 1.63

3.66 3.98 3.69 3.96 3.63

ZH

gemini-pro gemini-flash deepseek-v3.2 qwen3-235b gpt-oss

19.19 14.16 24.55 32.17 21.40

0.027 0.021 0.019 0.010 0.022

11.92 10.54 12.27 11.60 11.64

1.10 1.20 1.08 1.09 1.00

2.27 2.50 2.29 2.33 2.56

Lang

IT

ES

DE

EN

Table 7: Average quantitative textual metrics for human and LLM-generated books across languages. The human reference consists of 100 books randomly sampled per language from the Project Gutenberg corpus released by Momen et al. (2025), retaining only full-length, single-authored works originally written in the target language and published during the nineteenth or twentieth century, while excluding translations, short texts, and multi-authored works. Human values are reported only for languages for which a suitable reference sample was available.

Passage-level

Lang.

Overall

Rel.

Eng.

Coh.

Flu.

Div.

Coh.

Emp.

Sur.

Eng.

Comp.

EN DE IT ES RU ZH

4.8 4.9 4.8 3.7 4.9 4.9

4.4 3.7 4.1 3.3 2.6 4.3

4.9 4.4 4.6 4.7 3.6 4.5

4.6 4.0 4.2 4.5 3.0 4.6

4.7 3.6 4.8 4.6 2.8 4.6

4.5 5.0 5.0 3.5 3.5 3.5

3.5 4.5 5.0 4.0 1.5 4.5

4.0 3.0 3.5 3.5 2.5 2.5

4.0 3.0 4.0 3.5 1.5 3.5

3.5 3.5 4.0 4.0 2.0 3.5

Overall

4.66

3.73

4.45

4.15

4.18

4.16

3.83

3.16

3.25

3.41

Table 8: Human evaluation scores across passage-level and overall narrative quality dimensions for gemini-pro generated books in the Literature & Fiction (LF) genre.

Low Resource

Method RANK ENTROPY G LTR N - GRAM XLM - ROBERTA DETECTIVE TRACErank-js TRACEentr-js TRACEentr-norm

High Resource

EN

DE

IT

ES

RU

ZH

EN

DE

IT

ES

RU

ZH

0.316 0.332 0.352

0.440 0.472 0.408

0.332 0.336 0.336

0.408 0.396 0.424

0.420 0.332 0.436

0.420 0.332 0.320

0.340 0.336 0.372

0.488 0.492 0.488

0.240 0.240 0.256

0.460 0.436 0.436

0.340 0.368 0.304

0.332 0.360 0.372

0.577 0.610 0.840

0.433 0.660 0.720

0.514 0.520 0.680

0.667 0.490 0.850

0.325 0.610 0.658

0.470 0.600 0.740

0.757 0.660 0.890

0.937 0.860 0.850

0.946 0.490 0.630

0.964 0.670 0.690

0.658 0.670 0.950

0.874 0.740 0.930

0.946 0.900 -2.715

0.916 0.933 -4.460

0.940 0.957 -2.505

0.914 0.939 -2.680

0.900 0.911 -3.125

0.922 0.945 0.953 0.965 -2.710 -2.300

0.918 0.954 -2.840

0.937 0.935 -3.065

0.927 0.965 -2.265

0.934 0.955 -2.175

0.930 0.961 -2.335

Table 9: Thresholds for each method and language under the low- and high-resource settings.

0.047 0.227 0.050

0.335 0.531 0.398

0.271 0.472 0.489

0.447 0.700 0.494

0.486 0.587 0.641

RANK ENTROPY G LTR

N - GRAM XLM - ROBERTA DETECTIVE

TRACErank-js TRACEentr-js TRACEentr-norm

0.614 0.080 0.719

0.500 0.485 0.527

0 0.028 0

OOD-A

ID

0.479 0.868 0.916

0.639 0.620 0.511

0.114 0 0.090

0.396 0.702 0.790

0.634 0.593 0.525

0.191 0 0.070

OOD-D

DE

0.080 0.246 0.030

0.309 0.548 0.323

0.080 0.300 0

OOD-A

0.709 0.588 0.540

0.493 0.842 0.512

0 0.111 0.080

ID

0.060 0.317 0.172 0.484 0.660 0.740

0.500 0.374 0.435

0.105 0 0

OOD-A

0.277 0.426 0.341

0 0.336 0

OOD-D

IT ID

0.391 0.801 0.774

0.581 0.886 0.665

0.080 0.120 0.100

0.460 0.707 0.542

0.600 0.654 0.515

0.066 0.107 0.100

OOD-D

ES

0.206 0.501 0.774

0.718 0.566 0.413

0.605 0.661 0.506

0.974 0.280 0.505

ID 0.094 0.286 0.111

0 0.176 0.200

OOD-A

0.516 0.302 0.253

0.517 0.630 0.381

0.080 0.116 0.076

OOD-D

RU

0.033 0.783 0.888

0.040 0.429 0.327

0.150 0 0

OOD-A

ID

0.898 0.791 0.829

0.960 0.882 0.891

0.066 0.290 0.220

Table 11: Threshold-based low-resource results for ID, OOD-Domain, and OOD-Author evaluation in each language.

OOD-D

ID

0.245 0.250 0.066

Method

EN

0.754 0.856 0.892

0.792 0.872 0.766

0.200 0.289 0.274

OOD-D

ZH

0.328 0.744 0.650

0.843 0.696 0.487

0.388 0.072 0

OOD-A

0.142 0.120 0.168

1.000 0.924 0.786

0.660 0.541 0.755

0 0.070 0.040

1.000 1.000 0.931

0.682 0.878 0.966

RANK ENTROPY G LTR

N - GRAM XLM - ROBERTA DETECTIVE

TRACErank-js TRACEentr-js TRACEentr-norm

0.303 0.732 0.709

0.980 0.334 0.681

0.200 0.120 0.200

OOD-A

ID

0.694 0.959 0.959

0.803 0.949 0.955

0.057 0.023 0.106

0.490 0.905 0.905

0.659 0.775 0.658

0.125 0 0.149

OOD-D

DE

0.100 0.537 0.405

1.000 0.660 0.707

0.317 0.200 0.257

OOD-A

ID

0.831 0.810 0.893

0.896 0.956 0.924

0.095 0.180 0.197

0.684 0.662 0.793

0.725 0.676 0.677

0.046 0.129 0.308

OOD-D

IT

0.189 0.050 0.164

1.000 0.203 0.477

0 0 0

OOD-A

ID

0.361 0.736 0.869

0.869 1.000 0.953

0.080 0.120 0.277

0.466 0.610 0.624

0.633 0.933 0.864

0.066 0.107 0.150

OOD-D

ES

0.357 0.963 0.922

0.992 0.441 0.518

0 0.120 0.164

OOD-A

ID

0.959 1.000 0.931

1.000 0.977 0.769

0.166 0.192 0.200

0.501 0.741 0.737

0.966 0.960 0.717

0.093 0.100 0.267

OOD-D

RU

0.197 0.322 0.504

0.733 0.474 0.761

0.114 0.036 0

OOD-A

ID

0.801 0.933 0.880

0.971 1.000 0.844

0.114 0.277 0.125

Table 12: Threshold-based high-resource results for ID, OOD-Domain, and OOD-Author evaluation in each language.

OOD-D

ID

Method

EN

0.838 0.814 0.945

0.959 1.000 0.845

0.118 0.303 0.278

OOD-D

ZH

0.285 0.683 0.592

0.946 0.653 0.866

0 0.156 0.364

OOD-A

0.483 0.531 0.475 0.294 0.469 0.562

0.440 0.587 0.621

0.045 0.207 0.052

0.261 0.248 0.050

0.476 0.780 0.476

OOD-D

ID

0.479 0.868 0.916

0.625 0.696 0.500

0.209 0.068 0.071

ID

0.376 0.702 0.790

0.616 0.595 0.600

0.191 0.063 0.064

OOD-D

DE

0.694 0.557 0.579

0.490 0.842 0.482

0.140 0.085 0.077

ID

0.699 0.893 0.617 0.395 0.786 0.875

0.579 0.408 0.446

0.075 0.075 0.075

ID

0.374 0.456 0.305

0.007 0.294 0.078

OOD-D

IT

0.420 0.724 0.699

0.627 0.666 0.650

0.068 0.069 0.068

OOD-D

ES

0.718 0.708 0.733

0.591 0.814 0.488

0.074 0.286 0.074

ID

0.509 0.328 0.594

0.582 0.566 0.470

0.066 0.116 0.066

OOD-D

RU

0.787 0.876 0.920

0.906 0.874 0.826

1.000 0.876 0.915 0.926 0.861 0.949

0.070 0.247 0.236

OOD-D

ZH

0.071 0.231 0.227

ID

Table 13: Low-resource results without threshold-based rejection for ID and OOD-Domain evaluation in each language.

TRACEentr-norm

TRACEentr-js

TRACErank-js

DETECTIVE

XLM - ROBERTA

N - GRAM

G LTR

ENTROPY

RANK

Method

EN

OOD-D 0.078 0.080 0.078 1.000 1.000 1.000 0.771 0.710 0.850

ID

0.070 0.129 0.070

1.000 1.000 1.000

0.719 1.000 1.000

0.694 0.959 0.959

1.000 0.953 1.000

0.068 0.068 0.068

ID

0.536 0.905 0.905

1.000 0.819 0.850

0.060 0.063 0.063

OOD-D

DE

0.831 0.810 0.893

1.000 0.956 0.924

0.070 0.180 0.347

ID

0.734 0.662 0.793

0.974 0.676 0.673

0.055 0.150 0.352

OOD-D

IT

0.415 0.919 0.919

1.000 1.000 0.966

0.075 0.075 0.220

ID

0.550 0.800 0.800

0.972 0.946 0.946

0.068 0.167 0.223

OOD-D

ES

0.959 1.000 0.959

1.000 1.000 1.000

0.074 0.228 0.348

ID

0.494 0.741 0.690

1.000 0.968 0.968

0.082 0.169 0.313

OOD-D

RU

0.847 0.949 1.000

1.000 1.000 1.000

0.082 0.300 0.082

ID

Table 14: High-resource results without threshold-based rejection for ID and OOD-Domain evaluation in each language.

TRACEentr-norm

TRACEentr-js

TRACErank-js

DETECTIVE

XLM - ROBERTA

N - GRAM

G LTR

ENTROPY

RANK

Method

EN

0.888 0.935 1.000

1.000 1.000 0.971

0.082 0.300 0.236

OOD-D

ZH

EN

Detector RANK ENTROPY G LTR N - GRAM XLM - ROBERTA DETECTIVE TRACErank-js TRACEentr-js TRACEentr-norm

DE

IT

ES

RU

ZH

Avg.

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

−0.198 −0.023 −0.016

−0.245 −0.222 −0.066

0.077 0.000 −0.020

−0.034 0.300 −0.090

0.000 0.225 −0.080

0.105 −0.111 −0.080

−0.014 −0.013 0.000

−0.080 0.056 0.100

−0.014 −0.170 −0.035

0.056 −0.286 −0.111

0.134 −0.001 0.054

0.322 −0.218 −0.220

−0.003 0.003 −0.016

0.021 −0.080 −0.078

−0.112 −0.169 −0.096

0.053 −0.215 0.033

−0.005 −0.027 0.014

−0.330 −0.072 −0.188

−0.216 −0.416 −0.171

−0.433 −0.525 −0.340

0.019 −0.232 −0.150

0.393 −0.606 −0.160

−0.088 −0.031 −0.125

−0.565 −0.232 −0.179

−0.168 −0.010 −0.125

−0.117 −0.186 −0.404

−0.095 −0.148 −0.109

−0.166 −0.306 −0.206

−0.215 −0.115 −0.152

0.128 −0.507 0.078

−0.083 −0.166 −0.126

−0.399 −0.622 −0.886

−0.209 −0.214 −0.105

−0.225 0.072 0.200

0.069 −0.094 −0.232

−0.185 −0.300 0.000

−0.202 −0.264 −0.160

−0.685 0.217 0.475

−0.144 0.065 0.063

−0.570 −0.047 −0.179

−0.131 −0.131 −0.119

−0.323 −0.198 −0.052

Table 15: Changes in macro-F1 under OOD evaluation relative to ID performance in the low-resource setting. For each detector and language, the table reports ∆D = F1OOD-D − F1ID and ∆A = F1OOD-A − F1ID . Negative values indicate degradation, while positive values indicate improvement. The Avg. columns report averages across languages.

EN

Detector RANK ENTROPY G LTR N - GRAM XLM - ROBERTA DETECTIVE TRACErank-js TRACEentr-js TRACEentr-norm

DE

IT

ES

RU

ZH

Avg.

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

∆D

∆A

0.142 0.050 0.128

0.200 0.130 0.160

0.068 −0.023 0.043

0.260 −0.023 0.151

−0.049 −0.051 0.111

−0.095 −0.180 −0.197

−0.014 −0.013 −0.127

0.040 0.044 −0.113

−0.073 −0.079 0.067

−0.052 −0.139 −0.200

0.004 0.026 0.153

0.042 0.087 −0.079

0.013 −0.015 0.063

0.066 −0.014 −0.046

0.000 −0.076 −0.145

−0.020 −0.666 −0.250

−0.144 −0.174 −0.297

0.197 −0.289 −0.248

−0.171 −0.280 −0.247

0.104 −0.753 −0.447

−0.236 −0.067 −0.089

0.123 −0.559 −0.435

−0.034 −0.071 −0.052

−0.268 −0.318 −0.008

−0.012 0.000 0.001

−0.025 −0.317 0.022

−0.100 −0.111 −0.138

0.019 −0.484 −0.228

−0.230 −0.337 −0.211

−0.640 −0.146 −0.257

−0.182 −0.054 −0.054

−0.692 −0.422 −0.554

−0.147 −0.148 −0.100

−0.642 −0.760 −0.729

0.105 −0.126 −0.245

−0.004 0.227 0.053

−0.200 −0.259 −0.194

−0.679 −0.037 −0.427

0.037 −0.119 0.065

−0.516 −0.250 −0.288

−0.103 −0.174 −0.123

−0.529 −0.231 −0.367

Table 16: Changes in macro-F1 under OOD evaluation relative to ID performance in the high-resource setting. For each detector and language, the table reports ∆D = F1OOD-D − F1ID and ∆A = F1OOD-A − F1ID . Negative values indicate degradation, while positive values indicate improvement. The Avg. columns report averages across languages.

Train

EN

DE

IT

ES

RU

ZH

Low Resource

Method

High Resource

EN

DE

IT

ES

RU

ZH

EN

DE

IT

ES

RU

ZH

RANK ENTROPY G LTR

– – –

0.068 0.068 0

0.055 0.055 0

0.053 0.053 0.033

0.061 0.061 0.05

0.071 0.071 0.044

– – –

0 0.030 0.103

0 0.055 0

0 0.053 0

0 0.061 0

0 0 0

N - GRAM XLM - ROBERTA DETECTIVE

– – –

0 0.683 0.542

0 0.611 0.527

0 0.642 0.450

0 0.666 0.477

0 0.353 0.094

– – –

0 0.916 0.766

0 0.935 0.753

0 0.914 0.793

0 0.950 0.533

0 0.333 0.424

TRACErank-js TRACEentr-js TRACEentr-norm

– – –

0.112 0 0

0.404 0 0

0.295 0 0

0.203 0 0

0 0 0

– – –

0 0 0

0 0 0

0 0 0

0 0 0

0 0 0

RANK ENTROPY G LTR

0.070 0 0.070

– – –

0.068 0.025 0.213

0.100 0 0.175

0.061 0.074 0.236

0.080 0.033 0.092

0.070 0 0.072

– – –

0 0.055 0

0 0.077 0.023

0 0.074 0.057

0 0.095 0.133

N - GRAM XLM - ROBERTA DETECTIVE

0.200 0.420 0.550

– – –

0.100 0.555 0.650

0.100 0.463 0.602

0.114 0.466 0.600

0.133 0.271 0.100

0 0.673 0.766

– – –

0 0.836 0.772

0 0.848 0.617

0 0.800 0.394

0 0.723 0.559

TRACErank-js TRACEentr-js TRACEentr-norm

0.057 0 0

– – –

0.038 0 0

0.376 0 0.160

0.273 0 0.08

0 0 0

0.150 0 0

– – –

0.198 0 0

0.283 0 0

0.675 0 0

0 0 0

RANK ENTROPY G LTR

0.050 0.071 0.050

0.068 0.090 0.090

– – –

0.083 0 0.125

0 0.106 0.061

0.025 0.317 0.069

0.050 0.070 0.248

0.071 0.095 0.205

– – –

0.179 0.078 0.302

0.064 0.074 0.061

0.094 0.290 0.260

N - GRAM XLM - ROBERTA DETECTIVE

0.080 0.585 0.371

0.086 0.589 0.468

– – –

0.114 0.750 0.471

0.133 0.669 0.206

0.076 0.765 0.304

0 0.938 0.895

0 0.916 0.749

– – –

0 0.981 0.959

0 0.824 0.788

0 0.811 0.813

TRACErank-js TRACEentr-js TRACEentr-norm

0 0 0

0 0 0

– – –

0 0 0

0.08 0 0

0 0 0

0 0 0

0 0 0

– – –

0.189 0 0

0.05 0 0

0 0 0

RANK ENTROPY G LTR

0.070 0.071 0.070

0.071 0.083 0.088

0.088 0.114 0

– – –

0.066 0 0.066

0.125 0.063 0.109

0.070 0.071 0.070

0.068 0.083 0.105

0.055 0.114 0.133

– – –

0.074 0 0.114

0.082 0.063 0.100

N - GRAM XLM - ROBERTA DETECTIVE

0 0.947 0.300

0 0.866 0.266

0 0.926 0.483

– – –

0 0.836 0.200

0 0.756 0.434

0 0.753 0.681

0 0.914 0.867

0 0.857 0.933

– – –

0 0.949 0.847

0 0.622 0.681

TRACErank-js TRACEentr-js TRACEentr-norm

0.128 0 0

0.131 0 0

0.080 0 0

– – –

0.146 0 0

0 0 0

0.157 0 0

0.170 0 0

0.174 0 0

– – –

0 0 0

0 0 0

RANK ENTROPY G LTR

0.05 0.051 0.063

0.068 0.057 0

0.085 0.148 0

0.064 0.064 0.076

– – –

0.069 0.190 0.094

0.070 0.034 0.07

0.068 0.061 0.095

0.061 0.106 0.2

0.075 0.046 0.106

– – –

0.096 0.072 0.229

N - GRAM XLM - ROBERTA DETECTIVE

0 0.482 0.063

0 0.695 0.283

0 0.710 0.114

0 0.749 0.250

– – –

0 0.393 0.146

0 0.298 0.200

0 0.531 0.200

0 0.419 0.200

0 0.380 0.184

– – –

0 0.420 0

TRACErank-js TRACEentr-js TRACEentr-norm

0.125 0 0

0.413 0 0.142

0.305 0 0

0.214 0 0

– – –

0 0 0

0 0 0

0.270 0 0

0 0 0

0 0 0

– – –

0 0 0

RANK ENTROPY G LTR

0.070 0.070 0.055

0.068 0.196 0.105

0.055 0.082 0.077

0.053 0.048 0.066

0.061 0.074 0.074

– – –

0.080 0.070 0.085

0 0.150 0

0 0.055 0

0 0.075 0

0 0.074 0.066

– – –

N - GRAM XLM - ROBERTA DETECTIVE

0.080 0.717 0.673

0.068 0.871 0.667

0.077 0.781 0.520

0.075 0.797 0.667

0.061 0.950 0.629

– – –

0 0.315 0.575

0 0.711 0.550

0 0.719 0.433

0 0.715 0.360

0 0.683 0.346

– – –

TRACErank-js TRACEentr-js TRACEentr-norm

0 0 0

0.080 0 0

0 0 0

0 0 0

0.05 0 0

– – –

0.023 0 0

0.109 0 0

0 0 0

0 0 0

0 0 0

– – –

Table 17: Threshold-based OOD-Language results across source and target languages under low- and high-resource settings. Rows indicate the source language used for training, while columns indicate the target language used for evaluation.

Train

EN

DE

IT

ES

RU

ZH

Low Resource

Method

High Resource

EN

DE

IT

ES

RU

ZH

EN

DE

IT

ES

RU

ZH

RANK ENTROPY G LTR

– – –

0.068 0.068 0.138

0.055 0.055 0.077

0.053 0.053 0.064

0.061 0.061 0.074

0.071 0.071 0.059

– – –

0.068 0.057 0.283

0.055 0.055 0.055

0.075 0.053 0.157

0.074 0.061 0.074

0.082 0.071 0.228

N - GRAM XLM - ROBERTA DETECTIVE

– – –

0.068 0.711 0.483

0.16 0.685 0.474

0.064 0.705 0.466

0.156 0.696 0.448

0.059 0.380 0.210

– – –

0.764 0.916 0.868

0.490 1 0.846

0.533 1 0.834

0.361 0.955 0.707

0.139 0.482 0.348

TRACErank-js TRACEentr-js TRACEentr-norm

– – –

0.168 0.071 0.409

0.404 0.386 0.209

0.298 0.144 0.295

0.300 0.220 0.331

0.08 0.144 0.240

– – –

0.130 0.281 0.076

0.298 0.08 0.077

0.172 0.435 0.4

0.342 0.245 0.119

0.137 0.361 0.244

RANK ENTROPY G LTR

0.070 0.068 0.070

– – –

0.066 0.055 0.176

0.118 0.075 0.175

0.061 0.074 0.236

0.059 0.082 0.085

0.070 0.032 0.070

– – –

0.055 0.055 0.055

0.075 0.075 0.075

0.074 0.074 0.074

0.082 0.082 0.082

N - GRAM XLM - ROBERTA DETECTIVE

0.364 0.536 0.633

– – –

0.560 0.483 0.697

0.394 0.591 0.681

0.23 0.602 0.603

0.059 0.501 0.301

0.549 0.826 0.840

– – –

0.819 0.959 0.878

0.842 1 0.962

0.618 0.916 0.488

0.141 0.871 0.668

TRACErank-js TRACEentr-js TRACEentr-norm

0.060 0.271 0.225

– – –

0.060 0.438 0.148

0.384 0.290 0.114

0.258 0.774 0.532

0.109 0.304 0.360

0.155 0.264 0.276

– – –

0.225 0.375 0.366

0.337 0.266 0.391

0.704 0.544 0.488

0.226 0.355 0.264

RANK ENTROPY G LTR

0.050 0.070 0.050

0.068 0.083 0.068

– – –

0.064 0.045 0.064

0.144 0.094 0.074

0.069 0.466 0.059

0.050 0.070 0.248

0.071 0.095 0.172

– – –

0.179 0.078 0.313

0.064 0.074 0.088

0.094 0.285 0.260

N - GRAM XLM - ROBERTA DETECTIVE

0.100 0.772 0.431

0.068 0.696 0.432

– – –

0.075 0.760 0.445

0.061 0.772 0.272

0.059 0.744 0.290

0.962 0.938 0.959

0.667 0.916 0.916

– – –

0.963 1 0.941

0.902 0.783 0.871

0.175 0.849 0.832

TRACErank-js TRACEentr-js TRACEentr-norm

0.168 0.163 0.173

0.095 0.418 0.243

– – –

0.146 0.460 0.269

0.211 0.34 0.306

0.070 0.238 0.225

0.370 0.344 0.153

0.124 0.405 0.327

– – –

0.281 0.286 0.426

0.04 0.394 0.340

0.080 0.144 0.059

RANK ENTROPY G LTR

0.070 0.071 0.070

0.068 0.068 0.068

0.055 0.055 0.055

– – –

0.074 0.061 0.074

0.082 0.096 0.088

0.070 0.071 0.070

0.068 0.068 0.080

0.055 0.076 0.185

– – –

0.074 0 0.133

0.082 0.096 0.096

N - GRAM XLM - ROBERTA DETECTIVE

0.450 0.969 0.574

0.566 0.837 0.499

0.509 0.926 0.461

– – –

0.380 0.820 0.324

0.197 0.706 0.476

0.729 0.924 0.700

0.667 1 0.916

0.777 0.922 1

– – 0.324

0.900 1 0.818

0.289 0.597 0.700

TRACErank-js TRACEentr-js TRACEentr-norm

0.119 0.275 0.393

0.126 0.512 0.558

0.246 0.317 0.474

– – –

0.382 0.398 0.360

0.008 0.269 0.280

0.171 0.238 0.163

0.163 0.513 0.446

0.254 0.310 0.327

– – –

0.320 0.381 0.136

0.088 0.226 0.229

RANK ENTROPY G LTR

0.05 0.051 0.05

0.068 0.057 0.068

0.077 0.148 0.077

0.064 0.064 0.064

– – –

0.059 0.190 0.059

0.070 0.0347 0.070

0.068 0.059 0.076

0.055 0.148 0.176

0.075 0.046 0.231

– – –

0.082 0.152 0.296

N - GRAM XLM - ROBERTA DETECTIVE

0.070 0.590 0.217

0.068 0.805 0.268

0.055 0.716 0.360

0.053 0.819 0.274

– – –

0.071 0.427 0.273

0.189 0.291 0.256

0.235 0.566 0.591

0.232 0.678 0.327

0.209 0.592 0.406

– – –

0.197 0.344 0.2

TRACErank-js TRACEentr-js TRACEentr-norm

0.166 0.246 0.163

0.413 0.525 0.401

0.305 0.247 0.304

0.284 0.245 0.100

– – –

0.239 0.196 0.210

0.355 0.070 0.070

0.603 0.420 0.514

0.171 0.248 0.26

0.335 0.053 0.208

– – –

0.104 0.229 0.197

RANK ENTROPY G LTR

0.070 0.070 0.060

0.068 0.236 0.226

0.055 0.077 0.077

0.053 0.064 0.064

0.061 0.074 0.074

– – –

0.070 0.070 0.070

0.068 0.138 0.206

0.055 0.055 0.156

0.075 0.075 0.146

0.074 0.074 0.135

– – –

N - GRAM XLM - ROBERTA DETECTIVE

0.080 0.811 0.856

0.068 0.953 0.741

0.077 0.916 0.843

0.075 0.860 0.853

0.061 0.959 0.875

– – –

0.070 0.756 0.351

0.189 0.809 0.561

0.186 0.949 0.544

0.172 0.833 0.621

0.133 0.800 0.712

– – –

TRACErank-js TRACEentr-js TRACEentr-norm

0.080 0.093 0.260

0.205 0.286 0.180

0.077 0.149 0.080

0.142 0.173 0.173

0.156 0.292 0.426

– – –

0.082 0.235 0.264

0.225 0.223 0.172

0.077 0.149 0.077

0.092 0.225 0.247

0.094 0.209 0.240

– – –

Table 18: OOD-Language results across source and target languages under low- and high-resource settings without thresholding. Rows indicate the source language used for training, while columns indicate the target language used for evaluation.

Related documents

• MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts#924265
• MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts#920034
• Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization#999818
• Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs#1013597
• ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation#1123327