SV-Detect: AI-generated Text Detection with Steering Vectors Mikhail Vishnyakov Independent Researcher [email protected]
Tatiana Gaintseva Queen Mary University of London [email protected]
arXiv:2606.07313v1 [cs.CL] 5 Jun 2026
Abstract Detecting machine-generated text is especially difficult under distribution shift, such as transfer across domains, source models, and editing attacks. We propose a fake-text detector based on steering vectors extracted from the hidden representations of a frozen language model. At each layer, we construct a direction that separates human-written from machinegenerated text, and represent each input by its layer-wise alignment with these directions. A lightweight classifier trained on these projection features yields the final detection score. Our method achieves strong performance both in-distribution and under distribution shift, including across domains, source models, and machine-editing transformations such as polishing and rewriting. Interpretation analyses show that the learned directions align with recognizable stylistic cues while capturing substantial additional signal beyond surface features. These results position fake-text detection as a representation-space probing problem and show that steering vectors provide a simple and effective solution. Code is available at https://github.com/Atmyre/sv-detect/
1
Figure 1: Token-level steering-vector projections distinguish LLM-generated text (top) from human writing (bottom). Word saturation reflects each token’s signed contribution to the classifier.
A large body of prior work approaches this problem through token-level statistics, likelihood-based scores, perturbation tests (Bao et al., 2023; Yang et al., 2023; Hans et al., 2024), or supervised classifiers trained directly on text (Jawahar et al., 2020; Chen et al., 2024). While these methods can be effective in-distribution, they often degrade under transfer, suggesting that many detectors rely on artifacts tied to a particular generator, dataset, or attack style (Li et al., 2024; Wu et al., 2024). This motivates a different perspective: instead of using solely surface form or output probabilities, can we identify a more stable signal in the intermediate representations of a language model?
Introduction
The rapid deployment of large language models has made machine-generated text increasingly fluent, diverse, and difficult to distinguish from human writing. (Wu et al., 2024, 2025). This creates practical challenges for content moderation, authorship verification, and benchmark integrity, and has led to growing interest in methods for fake-text detection (Wu et al., 2025; Weber-Wulff et al., 2023). At the same time, the problem has become harder: modern detectors must operate not only on direct generations, but also under distribution shift, including transfer across domains, source models, and editing-based attacks such as paraphrasing, polishing, or rewriting (Kehkashan et al., 2025).
In this paper, we propose SV-Detect, a faketext detector based on steering vectors extracted from the hidden states of a frozen language model. Our central hypothesis is that human-written and machine-generated texts induce systematically different directions in representation space, and these directions can be used as robust detection features. Concretely, for each transformer layer, we construct a steering vector that separates human and machine-generated texts, represent a new input 1
by its layer-wise alignment with these directions, and train a lightweight classifier on the resulting projection features. This yields a simple detector that does not require fine-tuning the underlying language model and naturally supports interpretation at the level of layers and directions. We evaluate SV-Detect on two complementary benchmarks: DetectRL (Wu et al., 2024) and MIRAGE (Fu et al., 2025). DetectRL emphasizes robustness across domains, source models, and attack families, and MIRAGE focuses on direct generation and machine-assisted editing scenarios such as polishing and rewriting. SV-Detect performs strongly in both settings, achieving near-perfect in-distribution performance, robust transfer across settings, and effective cross-benchmark transfer from MIRAGE to DetectRL. Beyond accuracy, we analyze what SV-Detect is actually using. The learned steering directions align with interpretable lexical and stylistic cues, but they also capture substantial signal beyond hand-picked surface features. This suggests that SV-Detect is not only effective, but also a useful tool for studying how machine-generated text differs from human writing at the representation level.
model-generated corpora. These methods typically use pretrained encoders such as BERT or RoBERTa and often achieve strong in-domain performance when train and test distributions are matched (Jawahar et al., 2020; Chen et al., 2024). Variants include boundary-based and topology-aware detectors (Kushnareva et al., 2024, 2021), as well as methods that improve transfer by removing brittle components of encoder representations (Kuznetsov et al., 2024). Unlike these approaches, we construct the detector directly from layer-wise steering directions in a frozen language model. Zero-shot and score-based detectors. Another line of work aims to detect machine-generated text without training a dedicated classifier. Early methods rely on token-level statistics such as likelihood, entropy, rank, log-rank, and likelihood-ratio scores (Su et al., 2023; Wu et al., 2024). More recent methods derive zero-shot criteria from a reference LM, including DetectGPT (Mitchell et al., 2023), Fast-DetectGPT (Bao et al., 2023), DNAGPT (Yang et al., 2023), and Binoculars (Hans et al., 2024). In contrast, SV-Detect is not zero-shot: it learns a lightweight detector from representationlevel features rather than directly from text. Robustness and evaluation under distribution shift. Recent work has emphasized that strong indomain results are not sufficient for realistic faketext detection (Tufts et al., 2025; Li et al., 2024; Kushnareva et al., 2024). DetectRL (Wu et al., 2024) benchmarks transfer across domains, source LLMs, and attack families, while MIRAGE (Fu et al., 2025) focuses on generation, polishing, and rewriting. Following this robustness-centered perspective, we evaluate SV-Detect on both benchmarks and additionally test cross-benchmark transfer from MIRAGE to DetectRL.
Overall, our contributions are as follows: • We introduce SV-Detect, a simple and effective fake-text detector based on steering directions extracted from the hidden representations of a frozen language model. • We demonstrate strong generalization on challenging benchmarks, including crosssetting and cross-benchmark transfer on DetectRL and MIRAGE. • We provide interpretation analyses showing that the learned directions align with meaningful lexical and stylistic cues while also encoding additional representation-level signal.
Methodology
3.1
Overview
The key idea of our method is based on hypothesis that human-written and machine-generated texts induce systematically different activation patterns inside a pretrained model. We capture these differences as directions in representation space and use them to construct features for a downstream detection model. Our pipeline is illustrated in Fig. 2 and consists of four stages: (i) extracting layer-wise activations from a frozen language model, (ii) extracting directions (steering vectors) that distinguish human-
Overall, our results suggest that fake-text detection can be viewed as a representation-space probing problem, where machine-generated text is identified by the directions along which it differs from human writing in hidden-state space.
2
3
Related Work
Supervised detection of machine-generated text. A common approach to machine-generated text detection is to train a classifier on human-written and 2
machine-generated texts Dfake (f ) N
(r)
f r Dreal = {xi }N i=1 , Dfake = {xj }j=1
For each layer l, we use the pooled activations (r)
(f )
N
f (f ) r a(r) = {al (xi )}N = {al (xj )}j=1 i=1 , a
to construct a steering vector vl ∈ Rd that captures the direction separating fake from real text representations. We study three methods for constructing steering vectors common in the literature (Zou et al., 2023). Mean-difference. The simplest choice is the normalized difference between class means: Nf Nr 1 X 1 X (r) (f ) (f ) (r) al (xi ), µl = al (xj ), µl = Nf Nr i=1 j=1
Figure 2: Overview of SV-Detect. A frozen LLM is used to extract mean-pooled hidden activations from each layer. These activations are projected onto layer-wise steering vectors, and the resulting cosinesimilarity scores are standardized and passed to a logistic-regression classifier for fake-text detection.
vlmean =
(f )
− µl
(f )
− µl ∥2
∥µl
(r)
(r)
.
This vector points from the average humanwritten representation toward the average machinegenerated representation.
written and machine-generated texts, (iii) projecting text representations onto these directions to obtain low-dimensional features, and (iv) training a lightweight classifier on top of these features. This design transforms high-dimensional hidden states into interpretable scores that quantify how strongly a text aligns with fake-text directions across the network. 3.2
µl
Logistic-regression. One alternative is to fit a linear classifier directly in the activation space of each layer and use normal vector of the separating hyperplane as steering vector. For each layer l, we train a logistic regression model to separate fake and real activations. Let wl denote the learned weight vector for layer l. We define the steering vector for this layer as its normalized version:
Layer-wise text representations
Let f be a frozen transformer language model with L layers. Given an input text x tokenized into T tokens, let Hl (x) ∈ RT ×d denote the hidden states at layer l ∈ {1, . . . , L}, where d is the hidden dimension. We construct the text representation at layer l by mean-pooling hidden states across the token dimension:
vllogreg =
wl . ∥wl ∥2
PCA. As a third variant, we compute paired differences between fake and real activations and define the steering vector as the leading principal component of these differences: (f ) (r) vlpca = PC1 {al (xj ) − al (xi )} .
T
1X al (x) = Hl (x)t ∈ Rd . T t=1
The full representation of x is therefore the collection of its representations at each layer: a(x) = a1 (x), . . . , aL (x) .
This construction aims to identify the dominant axis associated with the real-to-fake shift in representation space.
3.3
Once steering vectors are obtained, we construct feature representation of a text by using its alignment with these directions across layers. For each
3.4
Constructing steering vectors
Suppose we are given a training data that consists of a set of human-written texts Dreal and a set of 3
Projection features
DetectRL evaluation details. We evaluate on the three settings provided by DetectRL: (i) Multi-Domain, spanning ArXiv, XSum, Writing, and Review; (ii) Multi-LLM, spanning GPT-3.5, Claude, PaLM-2, and Llama-2; and (iii) MultiAttack, spanning prompt-based, paraphrase, perturbation, and data-mixing attacks. For Multi-Domain and Multi-LLM, fake examples are balanced across latent factors by subsampling each (llm_type, data_type) combination, using 50 fake samples per combination for MultiDomain and 200 for Multi-LLM. For Multi-Attack, both real and fake training sets are subsampled to 10,192 examples. We always evaluate on the full benchmark test split. Within each setting, we report both same-source and cross-source evaluation: same-source trains and tests on the same domain, source model, or attack family, while cross-source trains on one source and evaluates on another.
layer, we compute the layer-wise score as a cosine similarity between the pooled activation of this layer and its steering vector sl (x) =
⟨al (x), vl ⟩ . ∥al (x)∥2
The final feature vector is obtained by concatenating scores from all layers: s(x) = s1 (x), . . . , sL (x) ∈ RL . Thus, instead of classifying directly from the full hidden states, we classify from a compact representation that summarizes how strongly the text aligns with fake-text directions throughout the model. 3.5
Detection head
To actually decide if the text is machine-generated, its projection features are fed into a lightweight downstream classifier. Our default detector consists of feature standardization followed by logistic regression. Given a feature vector s(x), the detector outputs p(y = 1 | x) = σ w⊤ s̃(x) + b ,
MIRAGE evaluation details. We evaluate on MIRAGE under two settings, DIG (Disjoint-Input Generation) and SIG (Shared-Input Generation), and three transformations: generate, polish, and rewrite. The test splits are large-scale, e.g. 16,411 DIG-generate, 14,776 DIG-polish, 15,735 DIGrewrite, and 16,388 SIG-generate examples. Following (Fu et al., 2025), we use the data provided by (Fu et al., 2025) and (Chen et al., 2024) to construct steering vectors. The original MIRAGE training setup contains 500 human-written/machine-generated pairs, where the machine-generated side is GPT-3.5-Turbo-polished text. We augment this with two task-specific subsets from (Chen et al., 2024): 150 GENERATE pairs, where the machine side is directly generated GPT3.5-Turbo XSum text, and 150 REWRITE pairs, where the machine side is a GPT-3.5-Turbo rewrite of the corresponding human text. We then compute one steering vector per task (GENERATE, POLISH, REWRITE) at every layer, forming a layer-wise system of three directions. These are combined into an orthonormal basis via QR decomposition, and the downstream logistic regression is trained on the resulting projection features. We also report results using only the original 500 POLISH pairs; this already performs strongly on GENERATE, but is noticeably weaker on POLISH and REWRITE, motivating our task-specific multivector construction.
where s̃(x) is the standardized version of s(x), σ(·) is the sigmoid function, and y = 1 denotes machine-generated text. This choice isolates the effect of the steeringbased representation from the complexity of the classifier. We ablate our choice of classifier considering alternative models in Sec. 4.3. At test time, we extract activations for the new text, compute its projection features, and apply the trained classifier. Importantly, the reference LLM remains frozen throughout the entire pipeline.
4
Experiments
4.1
Experimental setup
Datasets. We evaluate on two recent and complementary benchmarks: DetectRL (Wu et al., 2024) and MIRAGE (Fu et al., 2025). We also report additional results on the COLING dataset (Wang et al., 2025) in Appendix Section E. We choose DetectRL and MIRAGE because they stress different but equally important aspects of fake-text detection: DetectRL emphasizes robustness under distribution shift across domains, source models, and attack families, while MIRAGE focuses on more realistic machine-writing scenarios, including generation, polishing, and rewriting.
Evaluation protocol. Following recent work (Fu 4
(a) In-domain performance across the Multi-Domain, Multi-LLM, and Multi-Attack (b) Cross-setting generalization: mean F1 settings, shown in terms of AUROC (left) and F1 (right) over train-test pairs with train ̸= test.
Figure 3: Performance on DetectRL.
et al., 2025; Chen et al., 2024; Bao et al., 2023; Mitchell et al., 2023), for fair comparison with baseline methods, in all our experiments, the reference model used for activation extraction is a frozen GPT-Neo-2.7B (Black et al., 2021). Texts are tokenized with truncation to a maximum length of 2048 tokens. For each text, we extract the meanpooled hidden representation from every layer and compute cosine-similarity features with the corresponding steering directions, as described in Section 3. We use mean pooling because it is simple, stable, and computationally cheap; richer summary statistics such as max pooling, attention-weighted pooling, or sentence-level aggregation are promising extensions but are left to future work. The downstream detector is trained on the same train data split used to construct the steering vectors. Unless stated otherwise, the final detector is a pipeline consisting of StandardScaler followed by LogisticRegression with liblinear solver, ℓ2 -regularization parameter C = 1.0, and random seed 42. For evaluation, we report AUROC, AUPR, TPR@FPR=5%, Balanced Accuracy, MCC, and F1 . Full numerical tables corresponding to all result figures in this section are provided in Appendix Sec. F. 4.2
Rank (Gehrmann et al., 2019), LRR, NPR (Su et al., 2023), DetectGPT (Mitchell et al., 2023), DNA-GPT (Yang et al., 2023), Revise-Detect (Zhu et al., 2023), Binoculars (Hans et al., 2024), and Fast-DetectGPT (Bao et al., 2023), while the supervised baselines are RoBERTa-Base (Park et al., 2021), RoBERTa-Large, XLM-RoBERTa-Base, and XLM-RoBERTa-Large (Conneau et al., 2019). Fig 3a shows that the proposed detector achieves near-perfect performance across all three DetectRL settings. On Multi-Domain, AUROC ranges from 99.87 to 100.0, with F1 between 99.10 and 100.0. On Multi-LLM, AUROC ranges from 99.88 to 99.99, with F1 between 98.47 and 99.65. On Multi-Attack, AUROC is between 99.96 and 99.99 across all attack families, with F1 above 99.55 throughout. These results are competitive with, and in several cases slightly exceed, strong supervised baselines such as RoBERTa-Base and XLM-RoBERTa. They also substantially outperform zero-shot score-based detectors, especially under more challenging attack settings where methods such as DetectGPT, NPR, or likelihood-based scores degrade markedly. DetectRL cross-setting generalization. In the cross-source setting, again following (Wu et al., 2024), we compare against the baselines reported there: LRR and Fast-DetectGPT (zero-shot), and RoBERTa-Base (supervised). Fig 3b summarizes cross-source generalization on DetectRL, reporting mean F1 over evaluations where the training and test sources differ. Our method achieves the strongest and most consistent cross-source performance among the com-
Experimental results
DetectRL in-distribution performance. We follow (Wu et al., 2024) and compare SV-Detect against both zero-shot and supervised baselines. In the same-source setting, the zero-shot baselines are Log-Likelihood (Solaiman et al., 2019), Entropy (Lavergne et al., 2008), Rank, Log5
pared detectors. This suggests that the steeringbased representation captures a more stable real-vsfake signal that persists across different generation conditions, rather than relying on source-specific artifacts. Overall, the cross-source results support the main claim of our approach: steering vectors extracted from hidden representations encode a transferable signature of AI-generated text, enabling strong generalization beyond the setting on which the detector was trained. We additionally report full tables with scores per model and setting in the Appendix.
most cases, it substantially outperforms the best zero-shot baseline, Binoculars, and remains reasonably close to the strongest supervised baseline, RoBERTa-Base trained directly on DetectRL. These results suggest that steering vectors learned from generation- and editing-oriented supervision capture a benchmark-independent signal of machine-generated text, rather than merely exploiting artifacts specific to a single dataset. 4.3
Ablation studies
Ablation summary. Figure 5 summarizes two design choices in SV-Detect on DetectRL MultiDomain transfer: the downstream classifier and the steering vector construction method. For each choice, we report three complementary quantities: mean in-distribution AUROC, mean transfer AUROC (averaged over off-diagonal pairs), and worst-case transfer AUROC (minimum over transfer pairs). This compact view captures not only average performance, but also robustness under the hardest distribution shifts.
MIRAGE results. On MIRAGE, following (Fu et al., 2025), we compare SV-Detect against LogLikelihood, LogRank, Entropy, RoBERTa-Base, RoBERTa-Large, LRR, DNA-GPT, NPR, DetectGPT, Fast-DetectGPT, ImBD (Chen et al., 2024), and DetectAnyLLM (Fu et al., 2025). Fig. 4a illustrates the results. SV-Detect with steering vectors built using only original 500 POL ISH pairs is already strong on GENERATE, reaching AUROC 0.9777 on DIG-G ENERATE and 0.9779 on SIG-G ENERATE, both above the previous best results. However, it underperforms the strongest MIRAGE baselines on POLISH and REWRITE, showing that the original MIRAGE training setup is insufficient for capturing all three transformation types with a single direction. The SV-Detect version that uses two additional task-specific subsets for training steering vectors resolves this gap. As shown in Figure 4a, it achieves the best performance on all six MIRAGE settings: AUROC 0.9912/0.9509/0.9503 on DIG-G ENERATE/P OLISH/R EWRITE, and 0.9909/0.9499/0.9421 on the corresponding SIG tasks. These results support our multi-direction formulation: separate task-specific steering directions provide a more robust representation than training on the original POLISH-only data alone.
Choice of downstream classifier. Among the classifiers we consider, logistic regression is the most reliable overall and is therefore used as the default detector throughout the paper. While CatBoost remains competitive in-distribution, it is consistently weaker under transfer, especially in the worst case. KNN is substantially less stable and often collapses to near-chance performance on off-diagonal evaluations. This suggests that the main strength of SV-Detect lies in the steering-based representation itself rather than in a highly expressive nonlinear classification head. Choice of steering vector construction. The choice of steering vector construction has an even stronger effect. Logistic-regression-based steering vectors are clearly the most robust, achieving both the strongest average transfer and the strongest worst-case transfer. In contrast, meandifference and PCA-based directions degrade much more severely, with worst-case transfer often approaching chance level. This shows that explicitly learning a discriminative direction in activation space is substantially more effective than relying on unsupervised variance directions or raw class-mean differences, and justifies our use of LogReg-based steering in the main experiments.
Transfer from MIRAGE-style steering vectors to DetectRL. To further test generalizability, we evaluate on DetectRL steering vectors learned from the training setup used for MIRAGE-style tasks. As shown in Figure 4b, this cross-benchmark transfer remains strong across all three DetectRL settings: multi-domain, multi-LLM, and multi-attack. The transferred detector achieves AUROC above 93% on all domains, above 86% on all source LLMs, and above 90% on all attack families. In
Choice of frozen LM backbone. We use GPT-Neo as the default backbone in the main experiments for fair comparison with prior work. In Appendix Sec6
(a) In-distribution performance on MIRAGE across the G ENERATE, P OLISH, and (b) Transfer of MIRAGE-based steering R EWRITE tasks under the DIG (top) and SIG (bottom) settings. vectors to DetectRL (AUROC)
Figure 4: Results on MIRAGE and cross-benchmark transfer.
feature per layer–direction pair on MIRAGE. To identify the most important layers, we inspect the magnitude-weighted coefficients |wl | · σl . On MIRAGE, the signal concentrates in the final layers (L29–31), and on DetectRL it is more distributed, with peaks at L0–2, L14–18, and L21. A more detailed layer-wise analysis is given in supplementary Section D. Logit-lens interpretation. For the topcontributing layers, we interpret the steering vector vl by projecting it through the final layer norm and LM head:
Figure 5: Ablation summary on DetectRL MultiDomain transfer. Bars show mean in-distribution AUROC, mean transfer AUROC, and worst-case transfer AUROC. Logistic regression performs best both as the downstream classifier and as the steering vector construction method, while alternatives often degrade toward chance under transfer.
logitsl = LM_head(LNf (vl )). Top-ranked tokens correspond to the +vl direction (fake-text side), and bottom-ranked tokens to −vl (human-text side). This is a standard logit-lens probe applied to a learned direction. Lexical signatures. Figure 7a shows that the learned directions are lexically interpretable, though differently on MIRAGE and DetectRL. On MIRAGE, the +vl direction at L29-31 surfaces content-bearing, often technical or formal fragments, while the −vl direction is dominated by punctuation-heavy fragments together with a few proper nouns and specific lexical items. On DetectRL, the +vl direction at L14-18 reflects a more polished LLM-like register, while the −vl direction surfaces more colloquial and topical fragments. Thus, the steering vectors align with meaningful lexical and stylistic differences between humanwritten and machine-generated text. Beyond surface cues. To quantify how much of
tion C, we compare several alternative backbones of similar scale. The results show that SV-Detect is not tied to GPT-Neo: all tested backbones perform strongly, while Qwen backbones are consistently the most robust under transfer, and Gemma backbones are also competitive, though slightly less consistent on the hardest cross-domain pairs. Thus, backbone choice affects robustness, but the overall steering vector approach generalizes beyond the particular LM used in the main paper.
5
What do the steering vectors detect?
We study both where SV-Detect extracts signal from and what lexical or stylistic patterns are associated with the learned directions. Our detector is a logistic regression on layerwise projection scores, s(x) = (s1 (x), . . . , sL (x)), with one feature per layer on DetectRL and one 7
Figure 6: Consensus tokens across the four LMs. A colored dot indicates that the token appears in that LM’s pooled top-token set. (a) Top tokens at the most contributing layers.
the signal is explained by simple stylistic features, we train a logistic regression on interpretable regexbased counts derived from the logit-lens analysis. The full setup in given in the Appendix Sec. D. These features achieve 76-82 AUROC on MIRAGE and 88-91 AUROC on DetectRL, but the full steering vector pipeline improves over them by 13-24 AUROC points across settings (Figure 7b). This suggests that the learned directions capture not only recognizable surface markers, but also additional signal in the hidden representations.
(b) In-domain AUROC of a regex detector vs. the full pipeline.
Figure 7: Steering vector interpretation.
Cross-LM logit-lens analysis. We further test whether the lexical signal identified above is specific to GPT-Neo or shared across reference LMs. We retrain the steering vector pipeline on four LMs (GPT-Neo-2.7B, Qwen3-1.7B (Yang et al., 2025), Gemma-3-1B-pt, and Gemma-3-1B-it (Team et al., 2025)) and decode the resulting dense steering vectors as in Sec. D.2. Figure 6 summarizes the result. Despite tokenizer- and corpus-specific differences, the same broad positive-side register recurs across models: all four surface endeavors, utilization, and utilizing, while the negative side is consistently more casual and discourse-like. This supports the view that SV-Detect captures a broader LLM-associated register rather than a GPT-Neospecific artifact.
6
Conclusion
We introduced SV-Detect, a fake-text detector based on steering vectors extracted from the hidden representations of a frozen language model. By representing each text through its alignment with layer-wise real-vs-fake directions, SV-Detect provides a simple, interpretable alternative to text-level score-based and fully supervised detectors. Experiments show that SV-Detect performs strongly both in-distribution and under challenging transfer settings, including shifts across domains, source models, attack families, and machineediting transformations. It also transfers effectively across benchmarks, suggesting that it captures a benchmark-independent signal of machinegenerated text.
Per-token contribution. SV-Detect also admits a token-level decomposition of the final score. For a fixed layer l, we project each token activation at that layer onto the corresponding steering vector and use the resulting signed value as that token’s contribution to the detector output. Figure 1 visualizes this signal. LLM-generated text tends to exhibit localized spans with consistently strong positive evidence, whereas human-written text shows weaker and more heterogeneous contributions. This qualitative contrast is consistent with the analyses above.
Our interpretation analyses further show that the learned directions align with meaningful lexical and stylistic cues while also encoding substantial additional representation-level signal. Overall, these results support a representation-space view of fake-text detection: AI-generated text can be identified not only from its surface form, but also from stable directions in hidden-state space. 8
7
Limitations and Future Work
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. https://github.com/EleutherAI/gpt-neo.
The proposed SV-Detect has several limitations. First, although it generalizes well across domains, source models, attack families, and machineediting settings, it still depends on a probe LLM whose representation geometry affects performance. Our backbone ablations in Section C show broadly consistent behavior across LM backbones, though some variation in transfer robustness remains. Second, while the method is much cheaper than perturbation-based detectors, it still requires a full LLM forward pass and is therefore less efficient than small encoder-based supervised baselines such as RoBERTa. Third, our experiments focus on English benchmarks and on a fixed set of contemporary generation and editing scenarios. Broader multilingual evaluation and more adaptive human-LLM collaboration settings remain to be studied. Finally, although our interpretation analyses show that the learned directions align with meaningful lexical and stylistic cues, a full causal account of why these directions generalize remains open. Further, SV-Detect remains a supervised detector: although it generalizes better than standard text classifiers in our experiments, it still relies on labeled human/machine examples and may require periodic updating as generators and attacks evolve. These limitations point to several directions for future work. One natural extension is to study SVDetect in multilingual and cross-lingual settings, where both the stylistic signal and the hidden-state geometry may differ substantially. It would also be valuable to explore whether steering vector representations can be combined with lighter-weight encoders to improve efficiency without sacrificing robustness. More broadly, our results suggest that representation space detection is a promising interface between robustness, interpretability, and model internals, and future work could study whether similar ideas apply to other modalities or to broader forms of synthetic content detection. Future work could also compare LogReg-based/PCA/mean-diff steering to other supervised linear projections such as shrinkage LDA or contrastive objectives.
Jiaqi Chen, Xiaoye Zhu, Tianyang Liu, Ying Chen, Xinhui Chen, Yiwen Yuan, Chak Tou Leong, Zuchao Li, Tang Long, Lei Zhang, Chenyu Yan, Guanghao Mei, Jie Zhang, and Lefei Zhang. 2024. Imitate before detect: Aligning machine stylistic preference for machine-revised text detection. CoRR, abs/2412.10432. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. Jiachen Fu, Chun-Le Guo, and Chongyi Li. 2025. Detectanyllm: Towards generalizable and robust detection of machine-generated text across domains and models. CoRR, abs/2509.14268. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. 2019. GLTR: statistical detection and visualization of generated text. CoRR, abs/1906.04043. Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: Zero-shot detection of machine-generated text. CoRR, abs/2401.12070. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks V. S. Lakshmanan. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 2296– 2309. International Committee on Computational Linguistics. Tanzila Kehkashan, Raja Adil Riaz, Ahmad Sami AlShamayleh, Adnan Akhunzada, Noman Ali, Muhammad Hamza, and Faheem Akbar. 2025. Ai-generated text detection: A comprehensive review of methods, datasets, and applications. Comput. Sci. Rev., 58:100793. Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov, Ekaterina Artemova, Serguei Barannikov, Alexander Bernstein, Irina Piontkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. 2021. Artificial text detection via examining the topology of attention maps. CoRR, abs/2109.04825.
References
Laida Kushnareva, Tatiana Gaintseva, German Magai, Serguei Barannikov, Dmitry Abulkhanov, Kristian Kuznetsov, Eduard Tulchinskii, Irina Piontkovskaya, and Sergey Nikolenko. 2024. Aigenerated text boundary detection with roft. Preprint, arXiv:2311.08349.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. CoRR, abs/2310.05130.
9
Kristian Kuznetsov, Eduard Tulchinskii, Laida Kushnareva, German Magai, Serguei Barannikov, Sergey I. Nikolenko, and Irina Piontkovskaya. 2024. Robust ai-generated text detection by restricted embeddings. CoRR, abs/2410.08113.
large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, Findings of ACL, pages 4824–4841. Association for Computational Linguistics.
Thomas Lavergne, Tanguy Urvoy, and François Yvon. 2008. Detecting fake content with relative entropy scoring. In Proceedings of the ECAI’08 Workshop on Uncovering Plagiarism, Authorship and Social Software Misuse, Patras, Greece, July 22, 2008, CEUR Workshop Proceedings. CEUR-WS.org.
Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, Jinyan Su, Minh Ngoc Ta, Mervat Abassy, Kareem Ashraf Elozeiri, Saad El Dine Ahmed El Etter, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Nurkhan Laiyk, and 7 others. 2025. Genai content detection task 1: English and multilingual machine-generated text detection: AI vs. human. CoRR, abs/2501.11012.
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGE: machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 36– 53. Association for Computational Linguistics.
Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomás Foltýnek, Jean Guerrero-Dib, Olumide Popoola, Petr Sigut, and Lorna Waddington. 2023. Testing of detection tools for ai-generated text. CoRR, abs/2306.15666.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. CoRR, abs/2301.11305.
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia S. Chao, and Derek Fai Wong. 2025. A survey on llm-generated text detection: Necessity, methods, and future directions. Comput. Linguistics, 51(1):275–338.
nostalgebraist. 2020. interpreting gpt: the logit lens. LessWrong post, https://www. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. Accessed 2026-05-25.
Junchao Wu, Runzhe Zhan, Derek F. Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S. Chao. 2024. Detectrl: Benchmarking llm-generated text detection in real-world scenarios. CoRR, abs/2410.23746. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won-Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Tae Hwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, and 12 others. 2021. KLUE: korean language understanding evaluation. CoRR, abs/2105.09680.
Xianjun Yang, Wei Cheng, Linda R. Petzold, William Yang Wang, and Haifeng Chen. 2023. DNAGPT: divergent n-gram analysis for training-free detection of gpt-generated text. CoRR, abs/2305.17359.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. CoRR, abs/1908.09203.
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. 2023. Beat llms at their own game: Zero-shot llm-generated text detection via querying chatgpt. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 7470–7483. Association for Computational Linguistics.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. 2023. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. CoRR, abs/2306.05540. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
Andy Zou, Long Phan, Sarah Li Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. Representation engineering: A top-down approach to AI transparency. CoRR, abs/2310.01405.
Brian Tufts, Xuandong Zhao, and Lei Li. 2025. A practical examination of ai-generated text detectors for
10
A
Contents
Potential Risks
B Inference Latency and Compute Overhead 11
Like other fake-text detectors, SV-Detect could be misused in high-stakes settings such as academic misconduct accusations, moderation, or authorship disputes if its predictions are treated as definitive evidence rather than probabilistic signals. False positives may unfairly penalize human authors, while false negatives may allow machine-generated text to evade detection. We therefore view SVDetect as a decision-support tool rather than a standalone adjudicator. A second risk is adversarial adaptation: public knowledge of the detector may encourage authors or model providers to modify generation style or post-process outputs in ways that reduce detectability. Although our experiments show robustness across several shifts and editing settings, we do not claim robustness to all future laundering or obfuscation strategies. Finally, broad deployment of fake-text detectors may have downstream social effects, including over-policing legitimate writing assistance or disadvantaging users whose writing style differs from the training distribution. For these reasons, such systems should be deployed with clear uncertainty communication, human oversight, and regular reevaluation on new domains and attack settings.
C Ablation on the choice of LLM backbone 13
B
1
Introduction
1
2
Related Work
2
3
Methodology 3.1 Overview . . . . . . . . . . . . . 3.2 Layer-wise text representations . . 3.3 Constructing steering vectors . . . 3.4 Projection features . . . . . . . . 3.5 Detection head . . . . . . . . . .
2 2 3 3 3 4
4
Experiments 4.1 Experimental setup . . . . . . . . 4.2 Experimental results . . . . . . . 4.3 Ablation studies . . . . . . . . . .
4 4 5 6
5 What do the steering vectors detect?
7
6
Conclusion
8
7
Limitations and Future Work
9
A Potential Risks
11
D Interpretability Analysis 14 D.1 Per-layer contribution to the detector 14 D.2 Logit-lens decoding of the steering vectors . . . . . . . . . . . . . . . 14 D.3 Stylistic-feature baseline . . . . . 15 D.4 Cross-LM Logit-lens decoding of steering vectors . . . . . . . . . . 15 D.5 Per-token visualization . . . . . . 16
All measurements were obtained on a single NVIDIA A100-PCIe GPU (40 GB) in float16. We benchmark inference on 64 texts from MIRAGE-DIG/G ENERATE (mean length: 980 characters), truncated to 512 tokens. Each detector is warmed up for three runs before timing. We report per-text latency at batch size 1, throughput at batch size 16, total parameter count (frozen backbone plus detector head), and peak GPU memory. We will release code to reproduce all experiments upon acceptance.
E Results on the COLING-2025 MGT benchmark 16 E.1 Setup . . . . . . . . . . . . . . . 17 E.2 Results . . . . . . . . . . . . . . . 17 E.3 Ablation on steering vector construction . . . . . . . . . . . . . . 17 F Results Tables F.1 DetectRL . . . . . . . . . . . . . F.2 MIRAGE . . . . . . . . . . . . . F.3 MIRAGE → DetectRL . . . . . . F.4 Ablation . . . . . . . . . . . . . .
Inference Latency and Compute Overhead
SV-Detect adds almost no overhead beyond a single LM forward pass. SV-Detect runs in 25.7 ms per text, essentially matching a bare LogLikelihood scoring pass through the same GPTNeo-2.7B backbone (27.6 ms). The small advantage in favor of SV-Detect is expected: LogLikelihood requires a full 50,257-way log-softmax at every token, whereas SV-Detect only uses hidden states, followed by lightweight mean pooling,
17 18 18 18 18 11
Detector
Params
Latency (ms/text, b=1)
Throughput (texts/s, b=16)
Peak GPU (MB)
SV-Detect (ours) Log-Likelihood Fast-DetectGPT
2.65 B 2.65 B 2.65 B
25.71 27.63 49.79
74.3 75.1 37.2
8,951 7,746 10,703
RoBERTa-Base RoBERTa-Large
0.12 B 0.36 B
7.21 11.40
1,295 714
327 783
Table 1: Inference cost comparison on an A100 40 GB (fp16, 512-token cap, MIRAGE-DIG texts). Parameter counts include the frozen backbone and the detector head. For SV-Detect, the detector-specific overhead is only a small logistic-regression head on top of the LM representations.
L or L × 3 features and fits in seconds on a CPU. By contrast, RoBERTa-based supervised baselines require full encoder fine-tuning for each training setup, which is the main difference in training cost rather than inference cost.
cosine projections, and a tiny logistic-regression head. In practice, the detector-specific computation is negligible compared to the backbone forward pass. SV-Detect is much cheaper than perturbationbased LLM detectors. Fast-DetectGPT requires two forward passes through a 2.7B-class model and is therefore about 1.9× slower than SV-Detect in our setup (49.8 ms vs. 25.7 ms per text). The gap would be even larger for DetectGPT, whose cost scales linearly with the number of perturbations. Using the standard K=100 setting, a simple extrapolation from the Log-Likelihood baseline yields roughly 2.8 s per text, i.e. around 100× the cost of SV-Detect. We therefore do not benchmark DetectGPT separately, since its overhead is determined directly by repeated LM forward passes. RoBERTa-based supervised detectors are cheaper, but less robust. RoBERTa-Base and RoBERTa-Large are substantially faster than SV-Detect, with latencies of 7.2 ms and 11.4 ms per text, respectively. The throughput difference is even larger at batch size 16, reflecting both smaller parameter counts and lower memory pressure. This is the main compute–robustness trade-off of our approach: SV-Detect inherits the cost of an LLM forward pass, whereas small encoder-based detectors are far cheaper. However, as shown in the main experiments, these lighter supervised baselines degrade substantially under source-model and attack-family shifts. In contrast, SV-Detect remains close to the cost of a single zero-shot LM forward while providing much stronger generalization. Training overhead is modest. The training cost of SV-Detect is dominated by a single offline pass through the reference LM to extract layer-wise activations, which can then be cached once per benchmark. The downstream classifier operates on only 12
C
Ablation on the choice of LLM backbone
In the main experiments, SV-Detect uses GPTNeo as the probe LLM in order to match the backbone used by prior work and enable a fair comparison. Here, we study how the method changes when the probe backbone is replaced with a different small open-weight LLM. We consider four alternatives with parameter counts broadly comparable to GPT-Neo: Qwen/Qwen3-1.7B, Qwen/Qwen3-1.7B-Base, google/gemma-3-1b-pt, and google/gemma-3-1b-it. This also lets us compare pretrained/base and instruction-tuned variants within the same model family. Table 2 reports AUROC on DetectRL MultiDomain transfer. Overall, all tested backbones perform strongly, with near-perfect diagonal performance and robust off-diagonal transfer. The strongest results are obtained with the Qwen backbones. Both Qwen3-1.7B and Qwen3-1.7B-Base improve over GPT-Neo on most train–test pairs, and Qwen3-1.7B-Base is the strongest overall. In particular, it gives the best off-diagonal transfer when trained on A R X IV, XS UM, W RITING, or R E VIEW , suggesting that its representation geometry is especially well suited to learning stable real-vsfake directions. GPT-Neo remains competitive, but is generally weaker than the Qwen variants on the harder crossdomain pairs, especially those involving transfer into or out of W RITING. For example, the A R X IV → W RITING and XS UM → W RITING pairs improve substantially under both Qwen backbones. The Gemma backbones are also competitive. Gemma3-1B-pt performs strongly across most transfer pairs and often approaches GPT-Neo, while Gemma3-1B-it achieves similarly high diagonal performance and remains robust under transfer. Overall, Gemma backbones remain slightly less consistent than Qwen on the hardest off-diagonal pairs, but still yield strong AUROC throughout. Taken together, these results indicate that SVDetect is not tied to GPT-Neo and remains effective across several backbone families. Backbone choice still matters, however: Qwen-style representations provide the strongest and most stable transfer, while GPT-Neo and Gemma remain competitive but slightly less robust on the hardest crossdomain pairs.
GPT-Neo-2.7B Train
ArX.
XSum
Writ.
Rev.
ArXiv XSum Writing Review
100.00 97.80 86.74 94.17
97.34 99.94 81.82 93.96
86.82 83.02 99.98 99.38
94.05 95.24 99.40 99.96
Train
ArX.
XSum
Writ.
Rev.
ArXiv XSum Writing Review
100.00 99.40 91.23 97.24
99.34 99.98 91.69 98.43
87.49 96.29 99.98 99.48
96.28 98.84 99.60 99.93
Train
ArX.
XSum
Writ.
Rev.
ArXiv XSum Writing Review
100.00 99.50 95.29 97.87
99.24 99.98 94.02 97.94
94.21 97.39 99.99 99.77
98.19 99.41 99.76 99.96
Qwen3-1.7B
Qwen3-1.7B-Base
Gemma3-1B-pt Train
ArX.
XSum
Writ.
Rev.
ArXiv XSum Writing Review
100.00 98.77 86.19 93.72
98.26 99.98 93.72 96.41
89.97 94.96 99.98 99.32
96.51 98.48 99.67 99.97
Gemma3-1B-it Train
ArX.
XSum
Writ.
Rev.
ArXiv XSum Writing Review
100.00 98.43 85.76 95.44
98.83 99.94 95.58 96.60
86.38 95.13 99.98 99.03
96.05 98.16 99.30 99.94
Table 2: Ablation on the probe LLM backbone for DetectRL Multi-Domain transfer. Each block shows AUROC for one backbone; rows denote the training domain and columns denote the evaluation domain.
13
D
Interpretability Analysis
This section provides the full methodology underlying the interpretability analyses summarized in Section 5. Specifically, we study: (i) per-layer attribution of the detector, (ii) logit-lens decoding of steering vectors, (iii) a hand-crafted stylisticfeature baseline, and (iv) token-level visualization of where the detection signal concentrates within a text. Unless stated otherwise, all analyses use the same frozen reference model (GPT-Neo-2.7B) and the same trained detectors as in the main experiments. D.1
Figure 8
which we associate with surface-level lexical cues, a broader ridge at L14-18, and a secondary peak around L21. This profile motivates our choice of L25–31 for MIRAGE and L14-18 for DetectRL as the main targets of the logit-lens analysis below.
Per-layer contribution to the detector
D.2 Logit-lens decoding of the steering vectors
Setup. Our detector is a logistic regression applied to layer-wise projection scores. For DetectRL, we use one feature per transformer layer:
Method. We apply a standard logit-lens probe (nostalgebraist, 2020) to each layer-ℓ steering vector vℓ by projecting it through the model’s final layer norm and unembedding head:
s(x) = (s1 (x), . . . , sL (x)) ∈ RL , sℓ (x) =
⟨aℓ (x), vℓ ⟩ , ∥aℓ (x)∥2
logitsℓ = WU⊤ LNf (vℓ ) ∈ R|V| , where |V| = 50,257 is the GPT-Neo-2.7B vocabulary size. The interpretation is directional: the top-ranked indices correspond to tokens aligned with the +vℓ direction, which we interpret as LLMleaning, while the bottom-ranked indices correspond to tokens aligned with −vℓ , i.e. the humanleaning direction. We report raw logits rather than softmax probabilities, since the goal is to inspect relative alignment with the direction rather than recover a calibrated next-token distribution.
where aℓ (x) is the mean-pooled residual representation at layer ℓ, and vℓ is the corresponding steering vector. For MIRAGE, we retain three orthonormalized directions per layer (one per task), so s(x) ∈ RL×3 is flattened before being passed to the classifier. In all cases, we train a standardized logistic regression (StandardScaler followed by ℓ2 -regularized LogisticRegression with the default liblinear solver). Attribution. After fitting the classifier, we attribute importance to each scalar feature using the magnitude-weighted coefficient
Results. Fig. 7a shows the top six tokens on each side. The resulting patterns are consistent across the selected layers:
cℓ = |wℓ | · σℓ ,
• MIRAGE, −vℓ (human-leaning). Stylized closing punctuation (!", ...", ’).’), formal sentence-ending subwords (authorised, itially), and decorative or exaggerated tokens (£, ?????).
where wℓ is the learned coefficient and σℓ is the empirical standard deviation of feature ℓ on the training set. This quantity is invariant to feature rescaling and corresponds to the effective contribution of a standardized feature to the classifier logit. For MIRAGE, we sum cℓ across the three task-specific directions to obtain a single per-layer attribution score. Fig. 8 shows that, on MIRAGE, the detector’s signal is concentrated in the final block of layers (L29-31), consistent with prior observations that higher-level stylistic features tend to emerge closer to the model output. On DetectRL, the contribution is more distributed: there is a sharp peak at L0-2,
• MIRAGE, +vℓ (LLM-leaning). Surnames and journal abbreviations from the arXivderived split together with fragments more characteristic of academic prose (baugh, foreseen, paralleled, flavorful). • DetectRL, +vℓ (LLM-leaning). Subword fragments of formal or emphatic vocabulary (astical, TRY, ermanent, ebted) and atypical punctuation or Unicode glyphs. 14
• DetectRL, −vℓ (human-leaning). More colloquial or topical fragments, including organising, Playstation, traged, adolesc, and bigot.
We standardize the resulting feature vectors and fit an ℓ2 -regularized logistic regression, tuning the regularization strength C by 5-fold cross-validation on a held-out split.
We stress that token-level logit-lens readings are noisy at any single layer. The patterns above should therefore be read as the consensus across the top contributing layers, rather than as literal interpretations of isolated tokens. Tokens that appear only at one layer are likely artifacts and should not be over-interpreted.
Results. Table 3 reports the in-distribution AUROC of this stylistic-feature baseline across all evaluated settings, together with the full steering vector pipeline for reference; the corresponding dumbbell visualization is shown in Fig. 7b. Two observations stand out:
D.3
1. Stylistic features alone form a non-trivial baseline, reaching 76–82 AUROC on MIRAGE and 88–91 AUROC on DetectRL. This is consistent with prior work showing that surface-level statistical and stylistic cues can already carry substantial signal for distinguishing machine-generated from human-written text.
Stylistic-feature baseline
To quantify how much of the detector’s signal can be explained by hand-recognizable surface cues, we train a baseline logistic regression on a fixed bank of regex-derived features and compare its in-distribution AUROC to the full steering vector pipeline.
2. The full steering vector pipeline still improves on this baseline by 13–24 AUROC points on MIRAGE and 9–12 points on DetectRL. We interpret this remaining gap as the part of the signal that is present in hidden representations but not recoverable from the regex bank. At the same time, the highest-weight regex features overlap substantially with the tokens surfaced by the logit-lens analysis (Fig. 7), including em-dashes, formal connectives, and prompt-leakage templates.
Feature bank. For each text, we compute counts and per-1,000-character rates for the following classes of patterns: • Typography: em-dash usage (—), curly quotes, Unicode ellipsis (. . . ) versus threedot ellipsis (...), and space-before-period. • Markdown / formatting: bold, italics, inline code, fenced code blocks, headers, and Oxford-comma lists.
D.4 Cross-LM Logit-lens decoding of steering vectors
• Discourse: formal connectives (Moreover, Furthermore, Additionally, In conclusion, Overall), summary prefixes, casual hedges (kinda, sorta, tbh), and inclusive pronouns (we, us, our).
The main interpretability analyses in Sec D.1 decode steering vectors through the same reference LM that produced them, namely GPT-Neo-2.7B. A natural question is whether the real-vs-fake directions learned by SV-Detect are specific to GPTNeo’s residual stream, or whether similar directions emerge in other modern LMs. To address this, we retrain the steering vector pipeline independently on several additional reference LMs and then decode the resulting dense steering vectors through each model’s own unembedding.
• Lexical: “polished paraphrase” phrasing (e.g. utilizing, endeavor, showcase), consonant-cluster words, and log character length. • Prompt leakage and template artifacts: [Assistant]: prefixes, templated review phrasing, generic summary formulas, and refusal-style stubs.
Reference LMs and datasets. We use four reference LMs spanning three families: EleutherAI/gpt-neo-2.7B, Qwen/Qwen3-1.7B, google/gemma-3-1b-pt, and google/gemma-3-1b-it. For each LM, we extract per-layer mean-pooled residual activations on the corresponding in-domain training corpora
• Mid-word case flips: regex patterns that detect uppercase letters inside otherwise lowercase words, a known artifact under aggressive perturbation attacks. 15
Three tokens ( endeavors, utilization, and utilizing) appear in the top pool of all four LMs, and a broader cluster including captivating, cybersecurity, leveraging, optimization, favorable, and unfavorable appears in three of the four. On the negative side, the consensus is weaker but still interpretable: the most stable tokens are casual discourse markers such as anyway, basically, etc, maybe, really, and very. Here, the three smaller modern LMs agree more strongly with one another, while GPT-Neo more often surfaces named-entity fragments and topical content. Overall, this analysis suggests that the lexical contrast captured by SV-Detect is not specific to GPT-Neo. When the steering vector pipeline is retrained independently on different LM families and their dense directions are decoded through each model’s own unembedding, the same broad semantic register reappears: LLM-polished, formal, and abstract language on the positive side, and more casual, discourse-like language on the negative side.
Figure 9: Top-pooled tokens per reference LM, separately for the +vℓ (LLM-leaning, left) and −vℓ (humanleaning, right) sides. For each LM, we pool the top-K tokens across the last third of layers and across all analyzed DetectRL / MIRAGE settings, then list the most frequent normalized tokens. Despite differences in family and tokenizer, the positive side consistently surfaces a polished LLM-style register, while the negative side is more casual and discourse-like.
and fit the standard logreg steering vector construction described in Sec. 3. This yields one dense steering vector per layer for each (LM, dataset) pair. GPT-Neo is analyzed on four datasets (MIRAGE G ENERATE/P OLISH/R EWRITE and DetectRL direct_prompt), while the other three LMs are analyzed on these four settings plus four additional DetectRL per-domain subsets (arxiv, xsum, writing_prompt, yelp_review), for a total of 28 (LM, dataset) pairs.
D.5
Per-token visualization
The token-level figure in the main paper (Fig. 1) is constructed as follows. Given a text x and the bestAUROC layer ℓ⋆ identified by an AUROC-probe on the held-out set, we forward x through the frozen reference model and record the per-token residual vector
Method. For each (LM, dataset, layer), we decode the steering vector through the LM’s own final layer norm and unembedding using logit-lens probe as described in Sec. D.2. Because single-layer readings are noisy and strongly affected by tokenizer idiosyncrasies, we aggregate within each LM before comparing across LMs. Specifically, for each LM we pool the topKpool = 8 tokens at every layer in the last third of the network depth, separately by sign, then take the union across the LM’s datasets. Layer ranges are L17–25 for Gemma-3, L18–27 for Qwen3, and L21–31 for GPT-Neo. Before computing cross-LM overlap, we normalize tokens by stripping whitespace, lowercasing, applying NFKC normalization, dropping control/format characters, and filtering out pure punctuation as well as tokens with less than 50% ASCII letters. This removes tokenizerspecific artifacts and makes the consensus comparison more meaningful.
hℓ⋆ (xt ) ∈ Rd
for t ∈ {1, . . . , T }.
Each token is then assigned the dot product ρt = ⟨hℓ⋆ (xt ), vℓ⋆ ⟩ where vℓ⋆ is the steering vector at layer ℓ⋆ . Tokens are colored according to ρt , with the scale normalized to the 70th percentile of |ρ| in the corpus so that most informative tokens reach full saturation. Red denotes positive projection (+v, LLMleaning) and blue denotes negative projection (−v, human-leaning). The classification banner above each text corresponds to the full text-level prediction produced by the trained detector used in the main experiments.
E
Results. Fig. 9 shows the top pooled tokens for each LM separately, and Fig. 6 summarizes the cross-LM consensus. On the positive side, the four LMs converge on a clear LLM-polished register.
Results on the COLING-2025 MGT benchmark
For completeness, we additionally evaluate SVDetect on the COLING-2025 Multilingual MGT benchmark, English split (Wang et al., 2025). Unlike DetectRL and MIRAGE, this benchmark pools 16
Setting
Subset
DIG DIG DIG
Generate Polish Rewrite
SIG SIG SIG
Generate Polish Rewrite
Gap
standardized activations, our default construction).
99.12 95.09 95.03
+22.1 +14.3 +19.1
99.09 94.99 94.21
+23.5 +13.3 +17.8
2. Generator-pool filter. The training split contains a long tail of weak open-source generators whose outputs are often much less realistic than those of stronger LLMs. To test whether these generators help or hurt, we compare:
Stylistic SV-Detect AUROC AUROC MIRAGE-DIG 76.98 80.74 75.94
MIRAGE-SIG 75.59 81.71 76.36
DetectRL (Multi-Attack) DetectRL DetectRL DetectRL DetectRL
Direct Prompt Paraph. Perturb.
90.73 90.11 88.69 87.81
99.99 99.96 99.98 99.99
+9.3 +9.9 +11.3 +12.2
• all: all 27 generators • trimmed: remove 20 weak generators (Dolly, BLOOMZ, OPT-*, Flan-T5-*, T0-*), dropping 133,193 samples and retaining 248,652 fake examples. Generators are only removed from the training and dev splits, the whole test split is still used.
Table 3: In-distribution AUROC of the stylistic-feature baseline (Section D.3) compared to the full steering vector pipeline. The Gap column shows the additional discriminative signal captured by hidden representations beyond the regex feature bank.
We use the same logistic regression detector as described in Sec. 3. The downstream classifier is used elsewhere in the paper. We sweep C ∈ {10−3 , 10−2 , 10−1 , 1, 10, 100} and select the best value on dev AUROC. All other hyperparameters match the main experiments as described in Sec. 4.1.
many generators into a single binary detection task and provides an official held-out test set with labels. It therefore offers a complementary large-scale setting for evaluating the same steering vector pipeline and testing its sensitivity to the composition of the generator pool. E.1
Setup
E.2
Data. The COLING-2025 MGT English split contains three partitions: (i) train: 610,767 examples (228,922 human, 381,845 machine-generated from 27 generators), (ii) dev: 261,758 examples (98,328 human, 163,430 machine-generated), and (iii) test: 73,941 examples, released with ground-truth labels for leaderboard evaluation. The task is binary classification (human vs. machinegenerated).
Results
Llama-based SV-Detect outperforms all baselines by a substantial margin. Removing weak models from the training data helps Llama-based SVDetect further, gaining about 2 points in Accuracy and 1.7 in F1 . However, for GPT-Neo-based SVDetect, removing weak models hurts. We attribute this to the size and capability of the LM backbone: larger and more capable backbones, such as Llama, benefit from a cleaner and more homogeneous generator pool, whereas smaller backbones like GPTNeo appear to benefit from the additional diversity provided by weaker generators.
Backbones. We evaluate SV-Detect with two reference LMs: the EleutherAI/gpt-neo-2.7B used in the rest of the paper, and the larger meta-llama/Llama-2-7b-hf. For each backbone, we extract per-layer mean-pooled residual activations and use the same downstream logisticregression detector as in the main paper.
E.3
Ablation on steering vector construction
Tab. 5 and Tab. 6 report results for all six combinations of steering vector construction and generatorpool filter on both backbones. As in DetectRL and MIRAGE, the LogReg construction consistently outperforms Mean and PCA. The larger Llama-27B backbone further improves substantially over GPT-Neo-2.7B.
Ablation axes. We vary two components of the pipeline: 1. Steering vector construction. We compare the three constructions used elsewhere in the paper: Mean (normalized class-mean difference), PCA (leading principal component of fake-minus-real activation differences) and LogReg (ℓ2 -regularized logistic regression on
F
Results Tables
In this section, we provide comprehensive tables with results for all experiments in Sec. 4 17
Rank Team / Method 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 – 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
SV-Detect (Llama-2-7b, trimmed) SV-Detect (Llama-2-7b, all) Advacheck Unibuc-NLP Fraunhofer SIT Grape TechExperts(IPN) TurQUaz SzegedAI AAIG DCBU SV-Detect (GPT-Neo, all) Alfa L3i++ LuxVeri azlearning SV-Detect (GPT-Neo, trimmed) honghanhh Baseline VX1291 cuettransform rockstart batirsdu IPN-CIC Ai-Monitors semanticcuet hmcgovern abhirak0603 cnlpnitspp mail6djj bennben saehyunma yuwert777 seven fangsifan yaoxy jojoc dominikmacko tropaleum starlight1 nitstejasrikar
Macro-F1 84.8 83.7 83.07 83.01 82.80 81.88 81.53 80.68 79.10 78.74 77.13 75.6 75.37 74.63 74.58 74.14 74.0 73.94 73.42 72.93 72.32 72.24 71.01 70.68 70.57 70.05 68.48 68.02 65.02 64.66 63.32 62.80 62.14 59.09 58.48 57.28 54.16 49.94 49.57 47.57 44.89
GPT-Neo-2.7B
Acc. 85.00 84.00 83.11 83.33 82.89 82.23 81.81 80.74 79.29 79.34 78.01 76.50 76.42 75.54 75.68 75.17 75.50 75.14 74.89 74.83 73.16 73.89 71.42 72.42 72.65 71.96 69.51 70.50 68.76 68.46 67.48 67.25 66.69 63.20 62.68 64.20 60.37 50.78 50.60 56.65 57.24
Test AUROC
Acc.
F1
Mean Mean PCA PCA LogReg LogReg
all trimmed all trimmed all trimmed
0.865 0.855 0.740 0.737 0.975 0.952
0.860 0.856 0.751 0.753 0.866 0.873
0.718 0.737 0.559 0.557 0.765 0.755
0.696 0.721 0.439 0.433 0.756 0.740
SV construction
Pool
Dev AUROC
Test AUROC
Acc.
F1
Mean Mean PCA PCA LogReg LogReg
all trimmed all trimmed all trimmed
0.922 0.908 0.864 0.858 0.984 0.974
0.912 0.887 0.796 0.798 0.938 0.937
0.716 0.694 0.650 0.647 0.840 0.850
0.690 0.666 0.615 0.610 0.837 0.848
Table 6: SV-Detect on the COLING-2025 English benchmark with a Llama-2-7B backbone.
F.4
Ablation
Tables 11 and 12 provide results of ablating choice of the downstream classifier and steering vector construction method. Sec. 4.3 provides a summary of these results.
DetectRL
MIRAGE
Table 9 reports performance of SV-Detect and the baselines on the MIRAGE benchmark. Fig. 4a provides a visual summary of these results. F.3
Dev AUROC
Llama-2-7B
Tables 7 and 8 report the in-domain and crosssource performance of SV-Detect and the baselines, respectively. Fig. 3a and Fig. 3b provide a visual summary of these results. F.2
Pool
Table 5: SV-Detect on the COLING-2025 English benchmark with a GPT-Neo-2.7B backbone.
Table 4: Adapted English leaderboard from Table 4 of Wang et al. (2025), with our four LogReg-based SVDetect variants inserted. Ranking is by Macro-F1.
F.1
SV construction
MIRAGE → DetectRL
Table 10 provides results of using steering vectors learned from the training setup used for MIRAGEstyle tasks on DetectRL. Fig. 4b provides a visual summary of these results. 18
ArXiv Method
AUROC
XSum F1
Writing F1
AUROC
Review
AUROC
F1
AUROC
F1
68.00 39.06 56.26 67.58 64.65 54.96 58.95 73.24 94.38 74.21 99.99 99.54 99.84 99.85 99.98
59.38 20.55 50.90 57.57 53.09 52.30 50.83 64.29 79.73 67.84 99.65 97.73 98.76 98.31 99.60
75.84 28.82 55.08 76.40 76.61 50.09 44.43 75.01 90.00 77.02 99.97 99.76 99.88 96.40 99.96
67.22 2.14 51.90 69.88 68.99 45.39 35.25 68.71 84.32 71.62 99.50 98.90 99.05 92.66 99.30
Multi-Domain Log-Likelihood Entropy Rank Log-Rank LRR NPR DetectGPT Revise-Detect. Binoculars Fast-DetectGPT RoBERTa-Base RoBERTa-Large XLM-RoBERTa-Base XLM-RoBERTa-Large SV-Detect
65.35 48.39 57.17 67.01 70.54 53.85 22.15 70.40 84.03 43.69 100.0 99.99 100.0 99.98 100.0
57.55 29.71 54.62 60.09 61.34 49.65 0.00 37.51 76.77 24.46 100.0 99.90 100.0 99.85 100.0
45.68 67.84 36.87 46.74 50.09 34.59 12.21 50.34 77.39 39.19 99.99 99.85 99.97 99.84 99.94
GPT-3.5 Method
AUROC
41.32 57.23 22.47 42.60 38.38 18.31 0.00 46.07 72.18 28.39 99.85 98.95 99.55 98.95 99.45
Claude
F1
PaLM-2
Llama-2
F1
AUROC
F1
AUROC
F1
28.10 30.42 22.78 30.12 18.91 32.91 12.56 25.67 27.28 39.35 0.00 99.80 92.48 97.48 84.24 98.91
70.03 45.34 50.40 70.89 71.17 44.64 26.72 71.48 69.84 93.30 65.99 99.94 97.93 98.77 98.73 99.88
60.73 16.56 41.74 63.09 65.51 34.77 0.00 60.77 59.03 88.20 57.58 99.40 92.64 97.19 94.43 99.36
75.65 43.48 57.05 77.97 83.65 52.53 36.71 75.22 75.65 96.64 76.79 99.84 86.72 99.69 99.66 99.99
65.90 66.75 54.40 66.66 75.51 48.68 20.40 62.89 65.87 92.30 69.08 99.45 76.17 98.57 97.67 99.60
AUROC Multi-LLM
Log-Likelihood Entropy Rank Log-Rank LRR NPR DetectGPT DNA-GPT Revise-Detect. Binoculars Fast-DetectGPT RoBERTa-Base RoBERTa-Large XLM-RoBERTa-Base XLM-RoBERTa-Large SV-Detect
62.89 46.84 52.19 62.84 61.61 50.29 43.46 61.87 70.10 88.14 65.56 99.97 99.77 99.88 99.55 99.99
Direct Method
AUROC
57.80 23.29 49.32 56.87 52.12 43.81 26.27 55.04 62.72 82.50 59.55 99.70 98.86 99.45 97.56 99.60
43.32 52.25 41.68 43.32 43.30 41.64 32.86 48.88 49.87 55.15 30.01 99.98 96.23 98.26 91.67 99.91
Prompt F1
AUROC
Paraph. F1
AUROC
Perturb.
Mixing
F1
AUROC
F1
AUROC
F1
57.59 26.01 52.60 59.17 55.20 46.25 16.89 54.94 62.03 81.56 62.89 99.12 96.78 98.34 97.67 99.55
35.51 68.62 8.04 37.71 45.91 6.78 18.21 40.45 43.98 76.89 49.56 98.32 96.12 98.56 97.89 99.99
0.78 68.95 0.00 0.78 29.27 0.00 0.00 2.73 7.56 69.34 41.23 97.45 94.67 97.89 96.34 99.95
63.70 49.37 52.05 64.63 66.12 48.63 26.28 62.14 65.27 89.12 67.23 99.12 97.56 99.01 98.67 99.83
53.31 28.52 42.46 56.86 53.81 37.65 0.00 50.89 54.39 83.67 59.78 98.76 96.34 98.56 97.89 98.75
Multi-Attack Log-Likelihood Entropy Rank Log-Rank LRR NPR DetectGPT DNA-GPT Revise-Detect. Binoculars Fast-DetectGPT RoBERTa-Base RoBERTa-Large XLM-RoBERTa-Base XLM-RoBERTa-Large SV-Detect
89.25 26.47 83.50 89.25 85.83 77.98 52.84 88.01 86.88 94.87 79.56 99.87 98.73 99.56 99.45 99.99
82.09 0.00 76.27 81.45 77.40 71.61 40.90 80.78 79.61 89.73 72.45 99.60 97.83 99.12 98.67 99.70
86.87 26.18 81.21 86.35 80.80 77.15 51.83 85.62 84.89 93.45 78.43 99.78 98.45 99.23 98.89 99.96
78.16 0.00 72.86 77.51 74.30 70.63 37.98 77.47 76.21 88.12 70.34 99.47 97.56 99.01 97.98 99.56
64.55 48.12 60.60 64.69 63.99 56.94 31.79 65.61 67.26 88.34 70.12 99.67 97.89 98.89 98.23 99.98
Table 7: Performance of detectors on DetectRL in the Multi-Domain, Multi-LLM, and Multi-Attack settings.
19
SV-Detect Train
Fast-DetectGPT
RoBERTa-Base
ArXiv XSum Writing Review ArXiv XSum Writing Review ArXiv XSum Writing Review Multi-Domain (F1 )
ArXiv 100.00 XSum 93.09 Writing 80.57 Review 87.93
93.32 99.45 75.01 87.60
80.78 75.72 99.60 97.09
88.89 88.93 96.64 99.30
24.46 28.43 34.81 40.70
23.71 28.39 33.60 37.66
SV-Detect Train
59.67 62.99 67.84 68.25
60.17 63.08 68.30 71.62
100.0 68.43 78.58 82.64
75.90 99.85 72.72 84.15
Fast-DetectGPT
77.68 71.79 99.65 85.10
70.69 67.17 94.24 99.50
RoBERTa-Base
GPT-3.5 PaLM-2 Claude Llama-2 GPT-3.5 PaLM-2 Claude Llama-2 GPT-3.5 PaLM-2 Claude Llama-2 Multi-LLM (F1 )
GPT-3.5 PaLM-2 Claude Llama-2
99.60 99.60 99.45 99.55
86.58 98.91 94.87 96.41
90.80 94.83 99.36 92.13
97.36 99.26 98.21 99.60
59.55 55.77 0.19 56.28
59.56 57.58 0.00 57.74
SV-Detect Train
12.96 8.20 0.00 8.65
69.93 68.43 1.18 69.08
99.97 99.25 96.83 99.45
Fast-DetectGPT
70.34 99.40 83.92 93.02
62.90 93.43 99.80 87.56
94.68 99.25 89.77 99.45
RoBERTa-Base
Prompt Paraph. Perturb. Mixing Prompt Paraph. Perturb. Mixing Prompt Paraph. Perturb. Mixing Multi-Attack (F1 )
Direct Prompt Paraphrase Perturb Mixing
98.93 99.56 96.85 97.50 99.02
96.54 96.00 99.55 98.86 98.46
95.46 91.66 95.89 99.95 97.38
96.91 96.45 97.91 99.10 98.75
64.01 64.00 61.54 64.01 65.89
40.45 39.94 38.32 40.45 46.38
41.02 40.40 36.86 41.14 45.78
31.81 31.25 27.90 31.93 40.93
95.73 97.18 93.66 87.01 93.46
94.91 94.98 98.26 91.46 91.93
64.32 86.18 78.81 98.66 95.26
89.07 92.92 78.81 91.38 93.64
Table 8: Cross-source generalization on DetectRL, reported as F1 . Rows denote the training source and columns denote the evaluation source. We compare SV-Detect with Fast-DetectGPT as a zero-shot baseline and RoBERTaBase as a supervised baseline across the Multi-Domain, Multi-LLM, and Multi-Attack settings. MIRAGE-DIG (Disjoint-Input Generation) Methods Likelihood LogRank Entropy RoBERTa-Base RoBERTa-Large LRR DNA-GPT NPR DetectGPT Fast-DetectGPT ImBD DetectAnyLLM SV-Detect (polish-only) SV-Detect (3-task)
AUROC
Generate Accuracy MCC
TPR@5%
AUROC
Polish Accuracy MCC
TPR@5%
AUROC
Rewrite Accuracy MCC
TPR@5%
0.4936 0.4992 0.6522 0.5523 0.4716 0.5215 0.5733 0.6120 0.6402 0.7768 0.8597 0.9525 0.9777 0.9912
0.5091 0.5128 0.6150 0.5397 0.5217 0.5341 0.5595 0.6140 0.6258 0.7234 0.7738 0.8988 0.9241 0.9547
0.0147 0.0220 0.1099 0.1250 0.0871 0.0701 0.0776 0.0191 0.0275 0.4310 0.4065 0.7770 0.8862 0.9579
0.4653 0.4512 0.5543 0.4859 0.5171 0.4081 0.4771 0.5071 0.5469 0.5720 0.7888 0.9297 0.9113 0.9509
0.5000 0.5000 0.5417 0.5010 0.5151 0.5000 0.5004 0.5370 0.5531 0.5570 0.7148 0.8732 0.8364 0.8851
0.0214 0.0195 0.0954 0.0460 0.0633 0.0200 0.0309 0.0318 0.0355 0.1189 0.2730 0.7756 0.6414 0.7990
0.4337 0.4225 0.5805 0.5020 0.5570 0.3930 0.4453 0.4710 0.5061 0.5455 0.7825 0.9234 0.9105 0.9503
0.5000 0.5000 0.5566 0.5049 0.5385 0.5000 0.5001 0.5201 0.5266 0.5432 0.7068 0.8705 0.8332 0.8856
0.0000 0.0000 0.1650 0.0293 0.0864 0.0000 0.0080 0.0663 0.0826 0.1015 0.4139 0.7447 0.6672 0.7713
0.0148 0.0132 0.1189 0.0569 0.0895 0.0188 0.0251 0.0226 0.0283 0.1025 0.2933 0.7778 0.6437 0.7956
AUROC
Generate Accuracy MCC
TPR@5%
AUROC
Polish Accuracy MCC
TPR@5%
AUROC
Rewrite Accuracy MCC
TPR@5%
0.4968 0.5008 0.6442 0.5368 0.4703 0.5214 0.5759 0.6088 0.6353 0.7706 0.8612 0.9526 0.9779 0.9909
0.5207 0.5183 0.6123 0.5392 0.5236 0.5311 0.5647 0.6170 0.6241 0.7193 0.7791 0.9059 0.9241 0.9516
0.0145 0.0186 0.1074 0.1101 0.0910 0.0657 0.0813 0.0185 0.0193 0.4200 0.4183 0.7722 0.8839 0.9516
0.4599 0.4468 0.5640 0.4741 0.5150 0.4076 0.4788 0.5074 0.5434 0.5727 0.7951 0.9316 0.9089 0.9499
0.5002 0.5000 0.5439 0.5011 0.5157 0.5000 0.5001 0.5277 0.5515 0.5619 0.7199 0.8740 0.8290 0.8837
0.0233 0.0211 0.0946 0.0395 0.0702 0.0238 0.0340 0.0293 0.0309 0.1238 0.3036 0.7779 0.6291 0.7811
0.4319 0.4221 0.5858 0.5099 0.5576 0.3978 0.4457 0.4738 0.5079 0.5480 0.7694 0.9158 0.9039 0.9421
0.5000 0.5000 0.5645 0.5122 0.5426 0.5000 0.5002 0.5204 0.5260 0.5495 0.6920 0.8643 0.8265 0.8749
0.0111 0.0118 0.1198 0.0668 0.0762 0.0174 0.0258 0.0177 0.0239 0.1097 0.2868 0.7574 0.6169 0.7706
0.0183 0.0260 0.2543 0.1434 0.0842 0.0777 0.1196 0.2604 0.2758 0.4628 0.5497 0.7975 0.8483 0.9095
0.0000 0.0000 0.1247 0.0088 0.0340 0.0000 0.0110 0.1071 0.1328 0.1293 0.4300 0.7487 0.6730 0.7706
MIRAGE-SIG (Shared-Input Generation) Methods Likelihood LogRank Entropy RoBERTa-Base RoBERTa-Large LRR DNA-GPT NPR DetectGPT Fast-DetectGPT ImBD DetectAnyLLM SV-Detect (polish-only) SV-Detect (3-task)
0.0196 0.0182 0.1592 0.0529 0.0417 0.0314 0.0603 0.1571 0.1719 0.2078 0.5599 0.8119 0.8488 0.9032
0.0030 0.0000 0.0516 0.0048 0.0283 0.0000 0.0036 0.0612 0.0668 0.0607 0.4451 0.7483 0.6583 0.7703
0.0000 0.0000 0.0918 0.0221 0.0405 0.0000 0.0048 0.0340 0.0431 0.0525 0.3936 0.7320 0.6532 0.7507
Table 9: Results across three tasks (G ENERATE, P OLISH, R EWRITE) under two evaluation settings (MIRAGE-DIG and MIRAGE-SIG) on MIRAGE. Metrics reported are AUROC, Accuracy, MCC, and TPR@5%.
20
Setting
AUROC
AUPR
TPR@5%
Bal. Acc.
MCC
F1
0.9114 0.8650 0.8931 0.8682
0.8231 0.7301 0.7908 0.7413
0.9096 0.8669 0.8905 0.8616
0.9631 0.7854 0.7956 0.9188
0.9253 0.5711 0.5975 0.8366
0.9632 0.7958 0.7973 0.9195
0.9072 0.8848 0.8645 0.9082 0.8720 0.7941 0.8908 0.9444 0.8313 0.7991 0.8447 0.8452 0.8869 0.8601 0.8952
0.8138 0.7696 0.7296 0.8171 0.7435 0.5903 0.7843 0.8879 0.6644 0.5982 0.6898 0.6914 0.7744 0.7197 0.7913
0.9061 0.8861 0.8652 0.9054 0.8719 0.7959 0.8919 0.9447 0.8309 0.8018 0.8493 0.8444 0.8848 0.8623 0.8946
Multi-Domain ArXiv Writing XSum Review
0.9647 0.9370 0.9517 0.9298
0.9712 0.9472 0.9636 0.9459
0.8670 0.7490 0.8333 0.7748
Multi-LLM GPT-3.5 Claude PaLM-2 Llama-2
0.9935 0.8661 0.8740 0.9729
0.9942 0.8610 0.8904 0.9721
0.9692 0.4414 0.5734 0.8759
Multi-Attack Direct prompt (no attack) Prompt attacks (all) Few-shot prompting ICO prompting Paraphrase attacks (all) Back translation DIPPER paraphrase Polish using LLMs Perturbation attacks (all) Character-level perturb. Sentence-level perturb. Word-level perturb. Data mixing (all) LLM-centered mixing Multi-LLM mixing
0.9674 0.9546 0.9375 0.9657 0.9409 0.8695 0.9530 0.9842 0.9070 0.8784 0.9251 0.9205 0.9465 0.9366 0.9607
0.9695 0.9591 0.9424 0.9690 0.9467 0.8677 0.9582 0.9855 0.9175 0.8914 0.9299 0.9296 0.9575 0.9408 0.9656
0.8501 0.8066 0.7450 0.8621 0.7658 0.5238 0.8194 0.9355 0.6607 0.5912 0.6825 0.6984 0.8055 0.7271 0.8280
Table 10: Results on DetectRL with the three-direction version of SV-Detect across the Multi-Domain, Multi-LLM, and Multi-Attack settings. Metrics reported are AUROC, AUPR, TPR@FPR=5%, balanced accuracy, MCC, and F1 .
Classifier →
Logistic regression (base method) KNN (n_neighbors = 5) CatBoost (default parameters) Multi-Domain
Train ↓ / Eval → ArXiv XSum Writing Review
ArXiv XSum Writing Review ArXiv XSum Writing Review
ArXiv XSum Writing Review
100.00 50.25 87.13 99.65 77.89 63.40 53.90 64.96
100.00 97.34 97.80 99.94 86.74 81.82 94.17 93.96
86.82 83.02 99.98 99.38
94.05 95.24 99.40 99.96
50.00 72.37 99.85 80.85
50.50 100.00 96.61 84.70 97.52 99.93 97.97 85.53 78.78 99.74 93.85 91.24
92.39 83.21 99.99 99.10
96.65 94.42 99.35 99.96
Table 11: Ablation on the downstream classifier for generalization evaluation on DetectRL in the Multi-Domain setting. Metric: AUROC.
Steering vector construction → Logistic regression (base method)
Mean difference
PCA
Multi-Domain Train ↓ / Eval →
ArXiv XSum Writing Review
ArXiv XSum Writing Review ArXiv XSum Writing Review
ArXiv XSum Writing Review
100.00 97.34 97.80 99.94 86.74 81.82 94.17 93.96
100.00 77.44 91.02 99.54 85.27 90.26 89.63 87.84
86.82 83.02 99.98 99.38
94.05 95.24 99.40 99.96
71.93 79.25 99.06 98.14
79.29 88.26 97.16 99.40
91.53 83.37 69.20 96.20 76.18 82.37 83.72 84.66
84.32 56.26 93.82 89.67
83.56 50.83 92.69 96.24
Table 12: Ablation on the steering vector construction method for generalization evaluation on DetectRL in the Multi-Domain setting. Metric: AUROC.
21