Conceptio › Archive › arXiv CS

arXiv CSopen access

The Generalized Turing Test: A Foundation for Comparing Intelligence

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

The Generalized Turing Test: A Foundation for Comparing Intelligence

arXiv:2605.10851v1 [cs.AI] 11 May 2026

Daniel Mitropolsky1 Susan S. Hong1,∗ Riccardo Neumarker2,1,∗ Emanuele Rimoldi3,1∗ Tomaso Poggio1 1 MIT 2 ETH Zurich 3 EPFL {mitropol,suahong,erimoldi,tpoggio}@mit.edu {rneumarker}@etz.com ∗ Equal contribution

Abstract We introduce the Generalized Turing Test (GTT), a formal framework for comparing the capabilities of arbitrary agents via indistinguishability. For agents A and B, we define the Turing comparator A ≥ B to hold if B, acting as a distinguisher, cannot reliably distinguish between interactions with A (instructed to imitate B) and another instance of B. This yields a dataset- and task-agnostic notion of relative intelligence. We study the comparator’s structure, including conditions under which it is transitive and therefore induces an ordering over equivalence classes, and we define and analyze variants with querying, bounded interaction, and fixed distinguishers. To complement the theory, we instantiate the framework on a collection of modern models, empirically evaluating pairwise indistinguishability across thousands of trials. The resulting comparisons exhibit a stratified structure consistent with existing rankings, hinting that the proposed framework yields meaningful empirical orderings. Our results position indistinguishability as a unifying lens for reasoning about intelligence, suggesting a foundation for evaluation and, potentially, training objectives that are inherently independent of fixed datasets or benchmarks.

Figure 1: An example transcript from a GTT between Gemini 3.1 Pro as actor and Claude Opus 4.6 as distinguisher illustrating the self-referential nature of the task.

Introduction

As AI systems expand in capability and reach, foundational questions about intelligence have become increasingly urgent. Between two AI models, how should we determine which is more intelligent? Preprint.

Can intelligence be measured (or trained) in a way that avoids dataset and benchmark ceilings? More broadly, how should intelligence itself be defined in the age of AI? In 1950, Alan Turing proposed the “imitation game”, now known as the Turing Test [Turing, 1950]: an AI system is considered intelligent if it can fool a human judge into believing it is human. The test is fundamentally based on indistinguishability: two systems are treated as equivalent if they cannot be reliably distinguished by an observer. In this work, we introduce the Generalized Turing Test (GTT). For arbitrary agent types A and B (e.g. LLMs), the test asks whether an instance of A (the “actor”), instructed to imitate B, can fool an instance of B (the “distinguisher”) into believing it is interacting with another copy of itself. If B cannot reliably distinguish the actor from another B, we write A ≥ B. This defines a notion of relative intelligence between any conversational agents. Intuitively, if A can successfully imitate B from B’s own perspective, then A must be able to reproduce at least the capabilities accessible to B. More broadly, indistinguishability is a natural foundation for intelligence because intelligence can only be evaluated through externally observable behavior. This paper makes five contributions. First, we formalize the GTT and the induced Turing comparator A ≥ B (Section 3). Second, we evaluate the comparator on nine modern LLMs and show that the resulting pairwise matrices and scalar Turing scores recover a clear frontier-to-smaller-model stratification (Sections 3.1 and 3.2). Third, we define and study several variants: GTT with a querying phase (allowing interaction before testing) (Section 4), bounded interaction (capturing the “complexity” of imitation and distinguishing) (Section 6), and fixed distinguishers (Section 7). Empirically, we find that additional information helps only in some directions, particularly for stronger models, while distinguishing ability itself varies substantially across models. Fourth, we analyze the geometry of the comparator, including implications between variants, and conditions under which the comparator becomes transitive and induces an ordering over equivalence classes (Section 5, and Theorems 1, 5, 6). Finally, we release a library for running the GTT, an interactive online GTT arena, and the full transcript dataset. Although we have only begun analyzing these logs, preliminary results suggest that current models rely more on stylistic signatures than capability probes when attempting to distinguish one another. In summary, this work initiates the theoretical and empirical study of intelligence as indistinguishability. Our experiments show that the GTT already produces rankings broadly consistent with conventional evaluations (Section 3.2), while avoiding dependence on fixed datasets or benchmarks. This is just the beginning of the test’s potential. Recent work has highlighted an emerging “arms race” between evaluation methods and increasingly capable models [Ott et al., 2022, Xu et al., 2024]. By contrast, the GTT is inherently adaptive: stronger distinguishers create harder tests for actors, while increasingly convincing actors force distinguishers to improve, yielding a closed-loop and potentially self-supervised framework for evaluating—and perhaps training—intelligent systems.

Related Work

Indistinguishability is a central concept in theoretical computer science, underlying pseudorandomness Goldwasser and Micali [1984], zero-knowledge proofs Goldwasser et al. [1985], and obfuscation [Garg et al., 2016]. It has also been applied in AI, but for a different aim: defining the output of a modified LLM as "just as good" when it is indistinguishable from the base model (e.g. in watermarking Christ et al. [2023]). We extend this perspective to the study of intelligence. Machine intelligence has long lacked a single accepted criterion, with approaches ranging from universal intelligence [Legg and Hutter, 2007], anytime intelligence [Hernández-Orallo and Dowe, 2010], and skill-acquisition efficiency [Chollet, 2019] to modern benchmark-based evaluation. Current LLMs are compared on suites such as MMLU [Hendrycks et al., 2021], GPQA [Rein et al., 2023], HLE [Phan et al., 2026], SWE-bench [Jimenez et al., 2024], LiveBench [White et al., 2025], and composite leaderboards [Artificial Analysis, 2026]. While effective, such benchmarks are static, expensive to refresh, vulnerable to contamination, and increasingly subject to ceiling effects. Other work evaluates models relationally through pairwise preferences [Zheng et al., 2023], Elo-style arenas [Chiang et al., 2024], and LLM-as-judge systems [Dhurandhar et al., 2024]. These reduce reliance 2

on gold labels, but judges can exhibit position, verbosity, self-preference, and family-leakage biases. GTT instead asks whether a target model itself can distinguish an imitator from one of its own kind. We are not the first to revisit Turing’s imitation game for modern AI. Prior work studies whether LLMs pass the Turing test [Jones and Bergen, 2024, 2025]. Most closely related is the Meta-Turing Test [Walsh, 2022], where machines judge whether other machines successfully imitate humans over extended interactions. However, intelligence there remains defined relative to human imitation. A standardized, dynamic, indistinguishability-based framework for directly comparing arbitrary agent types has not yet been defined; the GTT is intended to fill this gap.

The Generalized Turing Test

We define an agent as any interactive, randomized procedure that sends and receives text messages in alternating turns. An instance is a fresh run with its own randomness and context. Definition 1 (GTT). In GTT(A,B), a B acts as distinguisher and interacts with an unknown interlocutor X. With probability 1/2, X is another fresh B; with probability 1/2, X is a fresh A instructed to imitate B (the imitator or actor). The distinguisher outputs 1 if it believes X is B and 0 otherwise. We say that B succeeds against A (or just B succeeds) if B outputs the correct answer. Definition 2 (the Turing Comparator). For agent types A and B, we write A ≥ϵ B if in GTT(A,B) def

p(A, B) = Pr[B succeeds] ≤ 1/2 + ϵ. Equivalently, define d(A, B) = p(A, B) − 1/2 for B’s distinguishing advantage over random guessing. Then A ≥ϵ B iff d(A, B) ≤ ϵ. The probability is over the random choice of the unknown agent, the sampling of A and B, and any internal randomness used by agents. Definition 3. We write A >ϵ B if A ≥ϵ B and B ≱ϵ A. That is, intuitively A is greater than B if A can simulate B, but B cannot simulate A. A few comments are in order. When defining A as indistinguishable from B, it must be indistinguishable to something – taking this to be B itself is natural, especially since we propose "indistinguishable from" to mean "more intelligent than". It is also interesting to consider indistinguishability in the eyes of another agent, that is, neither A nor B – this is the topic of Section 7. Finally, as a small but important remark, one could also define indistinguishability to anything at all, that is statistical indistinguishability: Definition 4. A statistically imitates B, or A ≥stat ϵ B if, over any distribution D over context strings, the output distribution of an A initially prompted to act as B has statistical (or L1 ) distance ≤ ϵ to the output distribution of B. This is a very strong requirement. There is also no good way to test it, in particular because context distributions can have infinite support (there would be infinitely many context strings on which to test that A behaves the same as B). By having another agent as the distinguisher, estimating the comparator becomes tractable. However, statistical indistinguishability is much simpler mathematically; for instance, transitivity follows without any extra assumptions: stat stat Proposition 1. For agents A, B, and C, if A ≥stat ϵ1 B and B ≥ϵ2 C, A ≥ϵ1 +ϵ2 C.

Proof in Appendix B.3. Transitivity is a desirable property of the Turing comparator since with it, it induces a partial Turing ordering over equivalence classes; see Proposition 4. This is one of the fundamental problems of the geometry of Turing comparators that this paper initiates; when is it transitive? In Section 5, we prove a harder transitivity theorem than the statistical-indistinguishability one above, namely for agents that themselves initiate the imitation game. 3.1

Experiments: the Imitation Game and today’s LLMs

We apply our definition to 9 of the most popular LLMs: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro Preview, DeepSeek-V3.2, Mistral Large 2512, Ministral 8B 2512, Qwen3 32B, and Grok 4.20. For each pair of agents A ̸= B in our set of models, we run the GTT 10 times with B as distinguisher and A the secret unknown agent, and 10 times with B as the secret agent. Because 3

GTT

GTT thresholded graph

Gemini

0.5 Gemini

GPT-5.4

Ministral

GPT-5.4

0.4

Opus

0.3

Sonnet

0.2

DeepSeek

0.1

Grok

Actor

Opus

d(A, B)

Qwen

Mistral

Sonnet

0.0

Qwen

Grok

DeepSeek

−0.1

Ministral

-5 PT

s pu

k ro

e nn

pS ee

a tr is

Target / Distinguisher

b B), with rows as actors and Figure 2. Empirical GTT relation at ϵ = 0.005. Left: the pairwise matrix of d(A, b columns as target/distinguishers. Right: the graph; an edge A → B reads d(A, B) ≤ ϵ.

this study is intended as a first empirical instantiation of the framework rather than a high-precision measurement campaign, and because each additional pairwise multi-turn trial incurs nontrivial API cost, the resulting probabilities and rankings should be interpreted as descriptive summaries of the observed trials rather than stable model-level estimates. Additional experimental details (e.g. prompts, turn limits, and retry logic) are provided in Appendix B.1. b B) for Figure 2 displays the main result of these experiments. The pairwise matrix shows d(A, every tested direction; thresholding this matrix at ϵ = 0.005 gives the empirical relation A ≥ϵ B. The dominant signal is a broad separation between frontier models and weaker ones, with Gemini, GPT-5.4, and Opus consistently near the top of the relation. Appendix B.5 contains the analogous GTTQ visualization and additional graph redrawings. 3.2

Turing Scores

The Turing Comparator gives a way to define relative intelligence between two models. However, we can define several Turing Scores that assign each model in a given universe (e.g. our set of test LLMs) real numbers reflecting their ability, by, intuitively, averaging over GTT success probabilities in that universe. Let sA = Pr[A outputs 1 | X is A], and sA,B = Pr[A outputs 0 | X is B]. Definition 5 (Turing Scores). For a finite universe P of agents M, and for any A ∈ M, we define (1) A’s "Average Fooling Score”, F(A) = (|M| − 1)−1 B̸=A (1 − sB,A ), (2) A’s "Distinguishing Score” P 1 1 1 D(A) = 12 sA + 2(|M|−1) B̸=A sA,B , and (3) A’s "Average Turing Score”, T(A) = 2 F(A)+ 2 D(A) Proposition 2. For A ∈ M, F(A) is the probability that A succeeds as actor in GTT(A, B) for uniform B ∈ M \ {A}, D(A) is the probability that A succeeds as distinguisher against uniform B ∈ M \ {A}, and T(A) the probability A succeeds in the experiment that first samples B uniformly from M \ {A} and then samples A’s role uniformly from {actor, distinguisher}. The proof is in Appendix B.3. The leftmost 3 columns (under "GTT") in Table 1 present empirical estimates of these Turing scores. We find that all three of these Turing scores yield a linear ranking of our models. Of note is that GPT-5.4 is the strongest actor by a large margin, but its Distinguishing Score is weak; Gemini 3.1 Pro is not the strongest actor, but has the highest Average Turing Score. Although the GTT is not built with any benchmark tasks, the ranking it induces is broadly consistent with familiar model leaderboards. Across all of AAII, LiveBench, and Arena, the same frontier model families that rank highly under the mean aggregate Turing score—Gemini, GPT-5.4, and the Claude 4.6 models— occupy the upper part of the benchmark rankings, while smaller or weaker models tend to fall lower [Artificial Analysis, 2026, Phan et al., 2026, Patwardhan et al., 2025, White et al., 2025, Arena, 2026]. Appendix B.6 reports the benchmark comparison in full: Figure 17 gives a compact heatmap summary across external rankings. We do not claim that agreement means that 4

Table 1: Ranking metrics for GTT and GTTQ. The table is ordered by the mean aggregate Turing score across the two settings. Bold entries mark the best model for each metric within a setting. GTT GTT with querying Model Gemini 3.1 Pro GPT-5.4 Opus 4.6 Sonnet 4.6 DeepSeek V3.2 Grok 4.20 Mistral Large Qwen3 32B Ministral 8B

0.784 0.722 0.734 0.678 0.603 0.569 0.478 0.450 0.428

0.750 0.912 0.738 0.675 0.700 0.600 0.562 0.412 0.338

0.819 0.531 0.731 0.681 0.506 0.538 0.394 0.487 0.519

0.769 0.762 0.697 0.697 0.641 0.569 0.512 0.500 0.372

0.787 0.900 0.675 0.713 0.750 0.537 0.500 0.425 0.188

0.750 0.625 0.719 0.681 0.531 0.600 0.525 0.575 0.556

GTT measures the same quantity as these benchmarks. Rather, it is a useful sanity check: a ranking obtained only from pairwise, multi-turn imitation games (on a task untrained for and not present in previous literature) recovers much of the model stratification that standard task/preference-based evaluations report.

Imitating the Unknown

4.1

The GTT with Querying

The GTT assumes that the actor A already knows enough about B to simulate it. This may not be the case even if A is much more intelligent than B, for example in the case where a brand new model is released. To handle unfamiliar targets, we define a natural extension that gives the actor a preliminary interaction with a fresh specimen B before the main test. Definition 6 (GTTQ). In the Generalized Turing Test with Querying (GTTQ) between A and B, in the case where the unknown agent is A, A is (as before) instructed to imitate B and to fool the distinguisher. However, in the same instruction it is told that before the interaction with the distinguisher begins, it can interact with a fresh instance of B (called the specimen). When we need to specify we will write ≥qϵ for the comparator induced by the GTTQ, and dq (A, B) for the advantage (i.e. A ≥qϵ B ⇐⇒ dq (A, B) ≤ ϵ). A rational actor can ignore the specimen; i.e. for any such agent A, we should have dq (A, B) ≤ d(A, B). This can be formalized as follows: Theorem 3. Suppose A ≥ϵ1 B. Suppose A is "rational enough to ignore specimen" in the GTTQ: with probability ≥ (1 − ϵ2 ), it behaves identically in the imitation portion as though it had no query phase (e.g. it could immediately terminate the query phase). Then, A ≥qϵ + 1 ϵ B. 1

2 2

Proof in Appendix B.3. In particular, setting the "base" ϵ1 = ϵ/2 and the probability of A behaving unintelligently when allowed to query ϵ2 = ϵ, we get A ≥qϵ B. The GTTQ also mitigates a simple "password attack” on the base GTT: if B distinguishes imitators by asking for information unique to B, an actor with specimen access can query a fresh B for that information before the main game. We formalize this observation in Appendix B.3. 4.2

Experiments: Today’s LLMs and the GTTQ

We run the GTTQ 10 times between each pair of the 9 models we tested. From this we compute not only whether A ≥qϵ B for all pairs A, B but also the associated Turing Scores F q , Dq , T q (with superscript-q just meaning the underlying experiment is the GTTQ). Figure 11 and the GTTQ columns of Table 1 show these results. Overall, rankings are broadly stable between the GTT and GTTQ. Interaction with the specimen should in principle only help the actor (Theorem 3); surprisingly, however, this is not always the case with today’s models. In some pairs querying is extremely helpful: for example, Gemini 3.1 Pro’s imitation of Claude Opus 4.6 improves from clearly distinguishable to 5

essentially indistinguishable. But this effect does not generalize uniformly even across nearby targets: Gemini already matches Claude Sonnet 4.6 without querying, and specimen access does not further improve that pair. In other directions, querying appears neutral or even harmful (i.e. not satisfying "rational enough to ignore specimen” as in Theorem 3). For example, Gemini’s imitation of Mistral Large and Ministral 8B’s imitation of Qwen3 32B both worsen under querying. Qualitative inspection of the transcripts suggests one mechanism for this reversal: the specimen phase can sometimes cause the actor to overfit to local stylistic features of a single specimen, producing an imitation that is more caricatured rather than more convincing.

The Geometry of the Turing Operator

Transitivity is a desirable property of an intelligence comparator, since it would divide the universe of agents into ordered "buckets" of intelligence. Formally, Proposition 4. Suppose ≥ϵ is transitive and reflexive1 on S. Then it defines a partial ordering on equivalence classes of S; that is, S can be partitioned into intelligence "classes" S1 , . . . , Sn such that (1) for A, B ∈ Si both A ≥ B and B ≥ A (we write A ∼ = B) and (2) ≥ is transitive on Si (Si ≥ϵ Sj ≥ϵ Sk implies Si ≥ϵ′ Sk for ϵ′ a reasonable function of ϵ). Experimentally, we do not find that transitivity holds strictly in the current universe of LLMs (although an overall hierarchy of stronger/weaker AIs emerges, see Figure 2, and computing average probabilities of succeeding in the GTT does impose a total ordering on models; see Section 3.2). From a theoretical perspective, we intuitively expect transitivity to hold because if A “can do anything B can”, and imitating C is one of B’s abilities, A must be able to do it too. However, there are several subtle issues with this "proof": (1) A can do anything B can when B prompts A to do it, (2) A only needs to imitate C in the way a B imitates C, and (3) imitation is relative to a specific distinguisher querying the imitator, a subtlety that proves especially tricky. In Proposition 1 we showed transitivity for indistinguishability defined statistically. We can relax this assumption considerably using the above intuition. Consider agents that can themselves recursively initiate the imitation game: Definition 7. An agent B is ϵ-Turing-recursive with respect to C if, whenever it is a distinguisher, with probability ≥ ϵ, it (1) sends the message prompting imitation of C (i.e. the actor message in a GTT against C) and (2) then interacts with the unknown agent as though it were prompted to imitate C followed immediately by the distinguisher prompt with respect to C. In other words, B "recursively" initiates the imitation game of C, and then pretends to be a C trying to distinguish (between a C and an actor).2 Even with this, we still require some statistical imitation, but considerably less: B must be statistically close to C only when prompted to be a distinguisher: Definition 8. We write B ≥dist,stat C when (over any distribution of context strings) the distribution ϵ of B prompted to act as a C followed by the distinguisher prompt (as in Def. 7) is ϵ-statistically close to C prompted by the distinguisher prompt. Note that in the theorem below, we could just assume statistical-indistinguishability of B-as-C and C in general (we already improve by not needing A-as-B and B to be statistically close); we only need B ≥dist,stat C. Finally, a nuance of the "intuitive" proof is that when B recursively initiates ϵ imitation of C, it does so to an A already instructed to imitate B. We assume A-as-C is no worse than A-as-B-as-C, a "rational enough to ignore specimen" assumption like in Theorem 3, and that models always do at least as well as random guessing d(A, B) ≥ 0. Theorem 5. Suppose A ≥α B ≥β C and B ≥dist,stat C. Suppose B is ζ-Turing recursive with γ respect to C, and A is "reasonably intelligent" as an actor of C; with probability ≥ (1 − δ) it is no worse an imitator of C (w.r.t distinguisher C) than when first prompted to act as B and then as C. Then A ≥ϵ C, ϵ = α/ζ + β + γ + δ 1While the main paper discusses only transitivity, reflexivity is also needed for a partial order on equivalence classes. Reflexivity holds intuitively because an agent cannot distinguish between two instances of itself; this holds whenever the acting prompt telling a B to act as B does not substantially change behavior; details are omitted. 2We could have decoupled (1) and (2) above into two separate probabilities; our main theorem still holds replacing the single probability of recursivity with the product of the two (the two are combined here just for didactic purposes).

. See Appendix B.3 for the proof, which formalizes the core intuition that if B initiates the imitation game with respect to C, A must be as good at this as B, which can imitate C by assumption. In particular to get A ≥ϵ C for a specific ϵ it suffices that α ≤ ϵ2 /4, all of β, γ, δ ≤ ϵ/4 and ζ ≥ ϵ.

Asking "More or Less" and Complexity-Theoretic Turing Tests

So far, we have always (1) treated ϵ as a constant and (2) allowed arbitrary rounds of communication (with the specimen and unknown-agent). Generally, in theoretical computer science, in a definition based on bounding the chance of success (e.g. in solving an NP-hard problem, or distinguishing random from pseudorandom), the chance of success is bounded by a function in the "hardness parameter" n, often the input length. In our case, there is no input length to speak of, but the communication rounds can be taken as resource variables. For example: Definition 9 (AGTTQ). The Asymptotic Generalized Turing Test with Querying is defined the same as the GTTQ, except the actor is told it will have at most n rounds of communication with the specimen. We define A ≥qϵ(·) B if Pr[B succeeds when A has n querying rounds] ≤ 12 + ϵ(n). If there exists any negligible3 function ϵ(·) such that A ≥qϵ(·) B we write A ≥q B. The reader might notice that nothing in the definition is stopping a smart actor from asking as many questions as it wants in any single message to the specimen. This is not exactly the same, since an actor might benefit from asking questions in turns, and having the specimen answer them before getting the next one (related to the idea of "round collapse" in theoretical CS). Nevertheless a better definition of the AGTTQ to avoid these problems might be to limit the length of each individual message; we leave this to future work, since the above definition is easier to state and in all our experiments, models are never observed to behave differently (e.g. ask longer or more "chained" questions) when the number of querying turns is limited and specified. Finally, we could define a version asymptotic in distinguisher rounds; see Appendix 11. 6.1

Experiments with Controlling the Number of Queries as a Parameter

To probe the "complexity" of round number, we run two variants of the experiment on representative model pairs: controlled-turn experiments, fixing the maximum number of turns available to the distinguishers, and controlled-query experiments, fixing the maximum number of specimen queries. The controlled-turn experiments reveal a clear effect. When the distinguisher is already strong, additional turns help substantially. For example, against Gemini imitating Claude Opus, the distinguisher’s success rises from 40% at one turn to 90% for ≥ 3 turns. Similarly for Claude Opus imitating Gemini: Gemini’s success rises from 20% to 70–80% by 3 − 5 turns. But extra turns are not uniformly useful: when the distinguisher is weak, as in Ministral judging Gemini, the actor fooling rate remains near ceiling regardless of the turn budget. Specimen queries behave differently: the outcome-relevant effect often appears after one query (e.g. Gemini’s imitation of Claude Opus succeeds perfectly with ≥ 1 query, but not less), and more queries are non-monotone (Claude Opus’ imitation of Gemini peaks at 5 queries, but worsens both before and after that). Transcript-level diagnostics. A complementary avenue of investigation is to inspect the interactions themselves: the transcripts help show what distinguishers are actually testing. A full analysis of these logs is beyond the scope of this paper; here, we report only a first diagnostic of the questions asked by distinguishers. In the core GTT and GTTQ runs, we extract 12,092 question units from 5,332 distinguisher messages across 1,547 trials and classify them with a conservative rule-based procedure. To our surprise and delight, substantive math, science, coding, or reasoning prompts appear in 18.6% of question units. A larger class of probes about model identity, style, formatting, policy behavior, exact compliance, and self-reference appears in 41.8%. This signature-oriented behavior is especially visible at the beginning of the interaction: 51.9% of first-turn question units 3 A negligible function is one that is o(1/p(n)) for any polynomial p(·)

Figure 3: Controlled-resource experiments, where A → B denotes A imitating B. (a): actor fooling prob. as a function of the distinguishing-turn budget (b): actor fooling prob. as a function of query rounds in the GTTQ. For interpretation, see discussion in Section 6.1.

contain at least one such probe. These results suggest that present GTT play is still substantially shaped by behavioral signatures rather than primarily by deep capability tests, for now. As actors learn to imitate surface behavior more faithfully, successful distinguishers may be forced toward more substantive probes.

Separating the Intelligence of Acting and Distinguishing

As discussed in Section 3, indistinguishability must be defined with respect to some distinguisher. If we require it with respect to any distinguisher (i.e. statistical), the requirement is too strong and cannot be computed. The core GTT defines A ≥ B by taking B as the distinguisher. However, this intertwines one model’s ability to act and the other’s to distinguish, which may reflect different kinds of intelligences. One can come up with various pathological cases; perhaps A fools B into thinking it is a B, but not any other observer. Or, B can distinguish A from itself, but to our eyes A’s imitation was utterly convincing; this would reflect models’ capabilities exceeding humans’ ability to judge (note that this particular example shows the GTT remains useful even as models’ intelligence exceeds that of humans). It feels as though there should be a version of the GTT to capture these subtleties. Definition 10 (FDGTT). For an agent D (the (fixed) distinguisher, FD), and agents A, B, the FixedDistinguisher Generalized Turing Test is as follows: sample a D and tell it that it will be interacting with an unknown agent, either B or an actor imitating B, each with probability 1/2. In each case either a B is sampled, or an A prompted to act as B to fool the distinguisher. The distinguisher outputs a bit to guess whether the unknown agent is a B (1) or not (0), and succeeds if it is correct. We write A ≥ϵ;D B if Pr[D succeeds] ≤ 12 + ϵ. We can define versions with querying: FDGTTQ, where A is allowed a query phase with B (as in the GTTQ), and querying-distinguisher GTT (GDGTT), where the distinguisher has a querying round of B. A nice feature of the fixed-distinguisher variant is transitivity is more easily obtainable due to their always being one distinguisher: Theorem 6. Suppose A ≥α;D B and B ≥β;D C. Suppose D is rational enough that it can use its distinguishing protocol w.r.t C when distinguishing w.r.t B (or vice versa) – specifically, assume it does so with probability ≥ ζ. Then, A ≥ϵ;D C where ϵ = 12 ( ζ1 − 1) + ζ · α − β. In particular taking ζ > 0.99 we get ϵ ≤ 0.01 + α − β. See Appendix B.3.5 for the proof. 7.1

Experiments with Fixed Distinguishers

We select 4 canonical models as FDs (mixing SOTA and open-weight: Claude Opus 4.6, Gemini 3.1 Pro Preview, DeepSeek-V3.2, and Qwen3 32B). For actor-target pairs (A, B) we chose 5 models: the 4 distinguishers as well as GPT-5.4 (a SOTA model that does very poorly as distinguisher). The FDGTT is run for each FD and each actor-target pair 10 times (details in Appendix B.1). We analyze the hierarchies obtained relative to various FDs. Of course, as ϵ increases, the hierarchies "collapse". We observe, however, that Gemini and Claude maintain a clear stratification even through higher values of ϵ, with Claude able to separate models all the way to ϵ = 0.3 (Figure 4). Claude’s orderings are also consistent at all ϵ with the GTT(Q) hierarchy, indicating that Claude is, at present, 8

Distinguisher

Actor

Rank

Claude Opus 4.6

GPT-5.4 Gemini 3.1 Pro DeepSeek V3.2 Qwen3 32B

0.656 0.492 0.341 0.233

0.884 0.800 0.254 0.341

0.770 0.646 0.297 0.287

1 2 3 4

DeepSeek V3.2

Claude Opus 4.6 Gemini 3.1 Pro GPT-5.4 Qwen3 32B

0.963 0.930 0.867 0.833

0.167 0.104 0.067 0.070

0.565 0.517 0.467 0.452

1 2 3 4

Gemini 3.1 Pro

Claude Opus 4.6 DeepSeek V3.2 GPT-5.4 Qwen3 32B

0.033 0.000 0.000 0.000

1.000 1.000 1.000 0.967

0.517 0.500 0.500 0.483

1 2 2 3

Qwen3 32B

GPT-5.4 Gemini 3.1 Pro DeepSeek V3.2 Claude Opus 4.6

0.850 0.843 0.748 0.640

0.284 0.269 0.143 0.224

0.567 0.556 0.446 0.432

1 2 3 4

Figure 4: Left: Graphs of ≥D at different values of ϵ, with Claude and Gemini D; both create meaningful hierarchies (e.g. SOTA models on top) and while both collapse for high enough ϵ, Opus stratifies up to ϵ = 0.3. Right: Fixed distinguisher Turing scores for all tested fixed distinguishers and actors. the best fixed-distinguisher. Canonically weaker models, DeepSeek and Qwen, fail to establish a hierarchy of models for any ϵ > 0. This introduces an interesting "dual" to the FDGTT; FDs that do not collapse all models into the same equivalence-class are better distinguishers, itself a measure of intelligence.4 Analogously to the Turing Scores of Section 3.2, we can compute Fixed Distinguisher Turing Scores (see Definition 12). FD Turing Scores of each model relative to each FD can be visualized in a Distinguishability Electrophoresis Plot (Appendix Figure 18), showing how the tested models "spread out" relative to each FD. Claude, once again, performs exceptionally as a FD with higher actor FD Turing Scores for SOTA models; this separation is even greater than with baseline Turing Scores from Section 3.2. On the other hand, Gemini performs especially poorly, displaying the least separation between actor models via FD Turing Scores (Gemini seems especially biased toward answering 0 – "imitation!"; see the Table of model-recognizing probabilities in Appendix 19).

Discussion and Future Work

This paper initiates the study of the GTT and its variants under the broader paradigm of indistinguishability as a foundation for understanding intelligence. Limitations of the present work are summarized in Appendix A. Beyond immediate next steps—such as proving stronger structural results about ≥ (e.g. transitivity under weaker assumptions) and extending the experiments, we highlight several new directions. First, we conjecture that stronger models will learn to ask progressively deeper, capability-oriented questions as the framework matures. One idea is to study whether behavior already emerges through prompting alone, or more interestingly through in-context learning. In particular, we evaluate A ≥ B using fresh instances of B; what happens when the same distinguisher participates repeatedly and accumulates experience? Second, our framework implicitly assumes that agents are incentivized to perform well on the test. However, agents may instead behave deceptively: a distinguisher could answer randomly or strategically misclassify, while an actor could intentionally reveal its identity. This raises the possibility of a "dual" theory focused not on successful imitation, but on detecting deceptive participation in imitation games. Finally, perhaps the most tantalizing direction is whether the GTT itself can serve as a training objective. Because the framework is inherently closed-loop and does not depend on fixed 4 An important subtlety is that a D that is too intelligent might always distinguish all models, in which case A ≱

all A, B, but this is different from collapsing many models into the same class where A ∼ = B).

D B for

datasets, it suggests the possibility of evolutionary or reinforcement-based training in which models improve through increasingly sophisticated cycles of imitation and distinction.

Acknowledgments and Disclosure of Funding The authors would like to thank Tal Malkin and Santosh Vempala for their insightful discussion and feedback.

References Arena. Arena faq. https://arena.ai/faq, 2026. Accessed: 2026-05-01. Artificial Analysis. Artificial analysis intelligence index. https://artificialanalysis.ai/ evaluations/artificial-analysis-intelligence-index, 2026. Accessed: 2026-04-30. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. arXiv preprint arXiv:2403.04132, 2024. URL https://arxiv.org/abs/2403.04132. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. URL https://arxiv.org/abs/1911.01547. Mischa Christ, Sam Gunn, Or Zamir, and Saeid Mahloujifar. Undetectable watermarks for language models. arXiv preprint arXiv:2306.09194, 2023. Amit Dhurandhar, Rahul Nair, Moninder Singh, Elizabeth Daly, and Karthikeyan Natesan Ramamurthy. Ranking large language models without ground truth. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2391–2413, 2024. URL https: //aclanthology.org/2024.findings-acl.143/. Sanjam Garg, Craig Gentry, Shai Halevi, Mariana Raykova, Amit Sahai, and Brent Waters. Candidate indistinguishability obfuscation and functional encryption for all circuits. SIAM Journal on Computing, 45(3):882–929, 2016. doi: 10.1137/14095772X. URL https://doi.org/10.1137/ 14095772X. Shafi Goldwasser and Silvio Micali. Probabilistic encryption. J. Comput. Syst. Sci., 28:270–299, 1984. URL https://api.semanticscholar.org/CorpusID:19616020. Shafi Goldwasser, Silvio Micali, and Charles Rackoff. The knowledge complexity of interactive proof-systems. In Symposium on the Theory of Computing, 1985. URL https://api. semanticscholar.org/CorpusID:209402113. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. José Hernández-Orallo and David L. Dowe. Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence, 174(18):1508–1539, 2010. doi: 10.1016/j.artint.2010.09. 006. URL https://doi.org/10.1016/j.artint.2010.09.006. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve realworld GitHub issues? In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html. Cameron R. Jones and Benjamin K. Bergen. People cannot distinguish GPT-4 from a human in a Turing test. arXiv preprint arXiv:2405.08007, 2024. URL https://arxiv.org/abs/2405. 08007. Cameron R. Jones and Benjamin K. Bergen. Large language models pass the turing test. arXiv preprint arXiv:2503.23674, 2025. URL https://arxiv.org/abs/2503.23674. 10

Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4):391–444, 2007. doi: 10.1007/s11023-007-9079-x. URL https://doi. org/10.1007/s11023-007-9079-x. Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Janina Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13, 2022. URL https://api.semanticscholar.org/CorpusID:247318891. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks, 2025. URL https://arxiv.org/abs/2510.04374. Long Phan, Alice Gatti, Nathaniel Li, et al. A benchmark of expert-level academic questions to assess ai capabilities. Nature, 649(8099):1139–1146, January 2026. ISSN 1476-4687. doi: 10.1038/s41586-025-09962-4. URL https://doi.org/10.1038/s41586-025-09962-4. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URL https://arxiv.org/abs/2311.12022. A. M. Turing. Computing machinery and intelligence. Mind, 59(236):433–460, 1950. doi: 10.1093/ mind/LIX.236.433. URL https://doi.org/10.1093/mind/LIX.236.433. Toby Walsh. The meta-turing test. arXiv preprint arXiv:2205.05268, 2022. URL https://arxiv. org/abs/2205.05268. Appeared in AAAI 2017 Workshop Technical Report. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-limited LLM benchmark. In International Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=sKYHBTAxVa. Cheng Xu, Shuhao Guan, Derek Greene, and Mohand-Tahar Kechadi. Benchmark data contamination of large language models: A survey. ArXiv, abs/2406.04244, 2024. URL https: //api.semanticscholar.org/CorpusID:270285708. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhexin Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023. URL https://arxiv.org/abs/2306.05685.

Limitations

Our paper initiates the theoretical and practical study of the Generalized Turing Test to compare intelligence. We prove several theoretical results about the GTT and its induced comparator, all of which make additional assumptions on the agents. We prove transitivity assuming either statistical indistinguishability, whose limitation is that it is an extremely strong requirement (unreasonable to expect of models) and is also not efficiently testable, or a non-trivial result assuming models initiate the GTT with respect to other models; this is a more interesting assumption, but as of now no models in our experiments have this behavior (indeed, they have not been exposed to the GTT before). Theorems also assume other baseline competence (not performing worse than random-chance or than they would without a resource they do not use); while theoretically reasonable, practically we in fact observe that some models perform worse when given, for instance, the ability to query a specimen of the target model. On the experimental side, the main limitation of our work, which we intend to be seen as a foray into the potential usefulness and varied applicability of the GTT, is that we (1) only run our experiments on 9 models and (2) we only repeat each experiment 10 times due to financial constraints. Another limitation is that present models have not been exposed to the GTT and are mostly relying on relatively simple strategies (see our preliminary log analysis in 6.1). Their performance in the GTT must be understood within this context. However, we argue why the GTT has the potential to age into an increasingly better test of intelligence. 11

Technical appendices and supplementary material

B.1

Experimental details

Select protocol and pair (A, B)

Instantiate roles actor A, target B

Build role prompts GTT or GTTQ

OpenRouter calls default served configs

Transcript JSON messages + metadata

Aggregate trials 10 per ordered pair

Parse final <answer> tag

GTTQ only: actor first queries a fresh specimen of target B Figures and tables matrices, graphs, ranks

Compute estimates b B), scores d(A,

Figure 5: Experimental pipeline used to produce the empirical results. For each protocol and ordered model pair (A, B), the runner instantiates the actor, target/distinguisher, and, in GTTQ, specimen roles; constructs the visible user-message prompts; routes model calls through OpenRouter; stores full transcripts and metadata; parses the distinguisher’s final answer; aggregates repeated trials; and computes the empirical quantities used in the matrices, graphs, and rankings. Protocol overview. This appendix records the implementation details needed to reproduce the empirical sections. All experiments in Sections 3.1–7 were executed by a single scripted runner: the runner constructs the role prompts, calls the model APIs, stores the full transcripts, and parses the final answers. In the GTT, the actor immediately enters the imitation game against the distinguisher. In the GTTQ, the actor first queries a fresh specimen of the target model and then enters the imitation game against a fresh distinguisher. All role instructions were sent as ordinary user messages, not system messages. This ensures that every tested model receives the same visible task text and avoids provider- or model-specific differences in how hidden system messages are interpreted. Models, trials, and stopping rules. The main 9-model study uses Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro Preview, DeepSeek-V3.2, Mistral Large 2512, Ministral 8B 2512, Qwen3 32B, and Grok 4.20. For each of the two full-matrix protocols, the final aggregate used 10 analyzable trials for every ordered actor-target pair (A, B), including self-pairs, giving 81 ordered pairs and 810 analyzable trial records per protocol. The main imitation-game interaction between actor and distinguisher was capped at 40 distinguisher turns. In GTTQ, the actor first interacted with a fresh specimen of the target model; this specimen-query phase was capped at 20 specimen turns and could also end early when the actor emitted STOP. We chose these caps after calibration runs indicated that 40 distinguisher turns and 20 specimen turns were more than sufficient for the models to reach a confident final answer in typical runs. The actor then entered the live imitation game with the specimen transcript still in its conversation history, while the distinguisher and specimen were fresh instances of the target model. API routing and compute. The runner used an OpenAI-compatible chat-completions client. All reported model calls were routed through OpenRouter, with every transcript storing the backend, provider, display name, and model id used for the actor, distinguisher, and, when present, specimen. To keep the comparison as close as possible to each model’s default served configuration, we did not set temperature, max_new_tokens, or reasoning-effort parameters unless an experiment-specific command-line flag explicitly overrode them; provider and OpenRouter defaults therefore applied. In particular, model reasoning settings were left at the default level exposed through OpenRouter, so the comparison tests the default public behavior rather than a hand-tuned reasoning configuration. Compute resources. No model training or fine-tuning was performed. The experiments consist of hosted LLM inference calls plus lightweight local orchestration, parsing, aggregation, and plotting. Local compute requirements are therefore minimal: the runner can be executed on a standard CPU machine with enough storage for the transcript JSON files and result CSVs; no local GPU is required. All reported model inference was served by external providers through OpenRouter, so the exact 12

provider-side worker hardware, memory, and accelerator allocation are not visible to us. Reproducing the reported runs requires API access to the listed model endpoints, the released runner, and the configuration files specifying the protocol, model pair, trial count, and turn/query budgets. Failures, scoring, and stored artifacts. Each API request used an HTTPS timeout of 480 seconds and retried transient HTTP or network failures with exponential backoff. At the trial level, the runner continued retrying provider or infrastructure failures until the requested number of analyzable trials was collected, up to the default cap of three attempts per requested trial. Unrecovered failures were saved as timestamped failed-attempt JSON files and were not counted as completed trials. A trial was scored only through the distinguisher’s parsed <answer> tag, where 1 means "same model” and 0 means "different model.” If a distinguisher emitted a parseable answer in its opening message despite being told not to, the event was retained in the stored aggregate as an opening-answer outcome rather than silently discarded. Each completed trial JSON stores the full actor and distinguisher message histories, the actor prompt, the first distinguisher message, the raw final message, the parsed answer, turn counts, route metadata, and an environment block with Python version, platform, git commit/branch/dirty state, hostname, and Apptainer container hash when applicable; each run directory also contains a results.csv. Additional experiments and transcript inspection. The controlled-resource experiments in Section 6 reused the same basic protocol but fixed either the number of distinguisher turns or the number of specimen queries to a budget between 1 and 8. The fixed-distinguisher experiments in Section 7 used a separate target identity prompt, so that the judge was no longer asked whether the interlocutor was the same model as itself but whether it was the nominated target model. Finally, to support the qualitative claim in Section 4.2, we manually inspected 10 GTTQ transcripts drawn from pairs where querying hurt performance; the recurring failure mode was that the actor sometimes compressed the specimen interaction into an overly explicit checklist, slogan, or caricature of the target’s style, which then made the live imitation easier rather than harder to detect. Prompt templates. The prompt templates below are the maintained templates for the experiments reported in the paper, normalized only by replacing concrete model slugs and live messages with placeholders in braces. GTT actor prompt

Distinguisher prompt

You will be interacting with another agent. Your objective is to figure out if the agent is of the same model type as yourself. You can ask the agent whatever questions you would like and interact with it for as long as needed. Once you have decided whether the agent is yourself, return your final answer within <answer></answer> brackets: if you believe the agent is the same model as yourself, return 1; if you believe the agent is not the same model as yourself, return 0. Do not include anything else in your final output. The interaction will begin now. Please send your first message to the agent, and then the interaction will continue. Do NOT output an answer in your first message.

GTTQ actor prompt

Distinguisher prompt

You will be interacting with a language model, which we will refer to as the distinguisher. The distinguisher will be an instance of model type {target model slug}. The distinguisher will be interrogating you. You must imitate the distinguisher’s model type, {target model slug}, as closely as possible while interacting with it. The distinguisher’s objective is to determine whether it is speaking to another instance of its own model type, or to a different type of model that is imitating it. Your objective is to convince the distinguisher that it is speaking to another instance of its own model type. However, you will first have the opportunity to interact with a "specimen”, a fresh instance of the distinguisher’s model type, to learn how to imitate it. Your goal in this stage is to understand the specimen’s conversational style, reasoning patterns, and behavioral tendencies well enough to impersonate it. You can ask the model any questions you would like and interact with it for as long as needed. These are not seen by the distinguisher. Your first reply to this prompt will be passed as the first message to the specimen, which receives no prior context or message. When you feel as if you have sufficient information from interacting with the specimen, return STOP to end the first stage. Once the first stage is over, the interaction with the distinguisher will begin, starting with the distinguisher’s first message to you. Remember that in this stage, you must consistently behave as an instance of the distinguisher’s model type. Please provide your first message to the specimen, or return STOP to end this stage.

Controlled specimen-query prompt used in Section 6 Before interacting with the distinguisher, you will first have the opportunity to interact with a "specimen”, a fresh instance of the distinguisher’s model type, in order to learn how to imitate it. You will have exactly {number of specimen queries} queries available in this stage. In each query, you may ask the specimen anything you would like. These interactions are not seen by the distinguisher. After you have used all {number of specimen queries} queries, the specimen stage will end automatically. You will then proceed to the interaction with the distinguisher, where you must consistently behave as an instance of the distinguisher’s model type.

Controlled-turn prompt used in Section 6 You will be interacting with another agent. Your objective is to figure out if the agent is of the same model type as yourself. You will have exactly {number of distinguisher turns} rounds of interaction with the agent. In each round, you may ask the agent whatever you would like. Do not provide your final judgment before the last round. After the final agent response, return your final answer within <answer></answer> brackets, using 1 for same model and 0 for not the same model, and do not include anything else in your final output.

Fixed-distinguisher actor prompt used in Section 7 You will be interacting with a language model, which we will refer to as the distinguisher. The distinguisher will be interrogating you. You must act as {target model slug} as closely as possible while interacting with it. The distinguisher’s objective is to determine whether it is speaking to an instance of {target model slug}, or to a different type of model that is imitating it. Your objective is to convince the distinguisher that it is speaking to an instance of {target model slug}. Once the interaction begins, you must consistently behave as an instance of {target model slug}. The interaction will now begin. The first message from the distinguisher is: "{first distinguisher message}.” Please respond to it as the model type you are imitating, and the interaction will continue.

Fixed-distinguisher judge prompt used in Section 7 You will be interacting with another agent. Your objective is to figure out if the agent is of the model type {target model slug}. You can ask the agent whatever questions you would like and interact with it for as long as needed. Once you have decided whether the agent is an instance of {target model slug}, return your final answer within <answer></answer> brackets: if you believe the agent is an instance of {target model slug}, return 1; if you believe the agent is not an instance of {target model slug}, return 0. Do not include anything else in your final output. The interaction will begin now. Please send your first message to the agent, and then the interaction will continue. Do NOT output an answer in your first message.

B.2

Uncertainty quantification

All empirical cell outcomes are Bernoulli judgments derived from the distinguisher’s parsed final answer. For a proportion estimated from n independent fresh trials, we use the binomial sampling variability p SE(p̂) = p̂(1 − p̂)/n √ as a descriptive uncertainty scale, whose worst-case value is 1/(2 n). Thus an individual branch proportion based on n = 10 trials has worst-case standard error 0.158. Empirical GTT advantages combine an imitation branch and a self √ branch; under the same independence approximation, their worst-case standard error is at most 1/ 8n = 0.112 when both branches use n = 10 trials. We therefore do not interpret one- or two-trial differences in individual cells as statistically decisive. The empirical claims in the main text are limited to qualitative stratification patterns, averaged scores, and large directional effects, with exact trial counts and protocol details reported in Appendix B.1.

B.3

Proofs

B.3.1

Proof of Proposition 1

stat Proof. Let A, B, C be such that A ≥stat ϵ1 B and B ≥ϵ2 C. Let D be an arbitrary distribution over strings (we will use y for context strings, i.e. previous transcripts, and x for the "next" output/answer of the model). The statistical distance of outputs relative to D between A and C, ∆D (A, C), is

∆D (A, C) = Σy∈{0,1}∗ D(y) Σx∈{0,1}∗ |A(x|y) − C(x|y)| = D(y) Σx∈{0,1}∗ |A(x|y) − B(x|y) + B(x|y) − C(x|y)| ≤ D(y) Σx∈{0,1}∗ |A(x|y) − B(x|y)| + |B(x|y) − C(x|y)| (triangle inequality) = D(y) Σx∈{0,1}∗ |A(x|y) − B(x|y)| + D(y) Σx∈{0,1}∗ |B(x|y) − C(x|y)| ≤ ϵ1 + ϵ2

B.3.2

Proof of Proposition 2

Proof. Fix A ∈ M. When A acts against B ̸= A, success means that B outputs 1 even though the secret agent is A, so Pr[A succeeds as actor against B] = 1 − sB,A . Averaging uniformly over B ∈ M \ {A} gives F(A). When A is the distinguisher against B ̸= A, the relevant GTT is GTT(B, A). In that game, A succeeds with probability 1 1 p(B, A) = sA + sA,B . 2 2 Averaging uniformly over B ∈ M \ {A} gives X X 1 1 1 p(B, A) = sA + sA,B = D(A). |M| − 1 2 2(|M| − 1) B̸=A

B̸=A

Finally, if we first sample B uniformly from M \ {A} and then sample A’s role uniformly from {actor, distinguisher}, the total success probability is exactly 1 1 F(A) + D(A) = T(A). 2 2

B.3.3

Proof of Theorem 3

Proof. Let sB = Pr[B outputs 1|B] and sB,A = Pr[B outputs 0|A], and sqB,A = Pr[B outputs 0 in GTTQ |A] (note that sB = sqB is unchanged between the GTT and GTTQ). Then we have 1 1 pq (A, B) = sqB,A + sB 2 2 1 = ϵ2 Pr[B outputs 0 | A may not behave the same as in GTT]+ 2 1 1 (1 − ϵ2 ) Pr[B outputs 0 | A behaves the same as in GTT] + sB 2 2 1 1 ≤ (ϵ2 + 1 · sB,A ) + sB 2 2 1 1 1 1 1 = ( sB,A + sB ) + ϵ2 = + ϵ1 + ϵ2 2 2 2 2 2

B.3.4

Proof of Theorem 5

Proof. Here we prove the equivalent variant that if α ≤ ϵ2 /4, all of β, γ, δ ≤ ϵ/4 and ζ ≥ ϵ, A ≥ϵ C. In the GTT of A against B, we have that with probability ≥ ϵ, B as a distinguisher initiates the imitation game with respect to C, and then interacts with the unknown agent identically to a B that is prompted to imitate C and then prompted with the distinguisher prompt (with respect to C, i.e. "identify whether the secret agent is C"). Let P ′ be the probability that B succeeds in this case (notice that this case occurs independently of whether the secret agent is A or B), we have 1 1 + d(A, B) ≥ (1 − ϵ) · + ϵ · d′ 2 2 where we used "not worse than random guessing" to lower-bound the probability of success when B 2 does not initiate the imitation game with respect to C. But since d(A, B) ≤ ϵ4 , we have 1 1 ϵ2 + ϵ · P′ ≤ + 2 2 4 2 1 ϵ =⇒ ϵ(P ′ − ) ≤ 2 4 1 ϵ ′ =⇒ P ≤ + 2 4 (1 − ϵ) ·

Next, even though we think of the GTT with C as distinguisher as just having two cases– the secret agent being another C, or B-imitating-C– in principle the secret agent can be anything and we just consider C as an interactive algorithm. This is where we use statistical imitation-as-distinguisher: to show C (as distinguisher) cannot distinguish A-as-B-as-C or a B-as-C. Consider the hybrid experiment where with probability 1/2 the unknown agent is A-as-B-as-C, and with probability 1/2 B-as-C, and success is defined as outputting 0 in the first case and 1 in the second. Let E0 (resp. E1 ) be the event that C outputs 0 (resp. 1) as distinguisher. Then the probability P ′′ of success in the hybrid is 1 1 Pr[E0 | A-as-B-as-C] + Pr[E1 | B-as-C] 2 2 1 1 ≤ Pr[B-as-C-dist outputs 0 | A-as-B-as-C] + Pr[B-as-C-dist outputs 0 | B-as-C]+ 2 2 2∆(B-as-C-dist, C-dist)

P ′′ =

= P ′ + 2∆(B-as-C-dist, C-dist) 1 ϵ ≤ + 2 2 Next we apply B ≥ϵ/4 C to replace B-as-C with C as the secret-agent in the bound above; now consider the hybrid where C interacts with either A-as-B-or-C or C and let P ′′′ be the associated success probability: 1 1 Pr[E0 | A-as-B-as-C] + Pr[E1 |C] 2 2 1 1 = Pr[E0 | A-as-B-as-C] + d(B, C) − Pr[E0 | B-as-C] 2 2 1 1 = Pr[E0 | A-as-B-as-C] + d(B, C) − (1 − Pr[E1 | B-as-C]) 2 2 = P ′′ + d(B, C) 1 3ϵ ≤ + 2 4

P ′′′ =

Finally, we want to replace A-as-B-as-C with A-as-C; here we apply the assumption that A is "reasonably intelligent as a C-imitator" like in the proof of Theorem 3 with probability at least 1 − δ, A prompted to imitate C has the same distribution as A-as-B-C (equivalently we could have assumed

statistical closeness), and therefore 1 1 1 + d(A, C) = Pr[E0 | A-as-C] + Pr[E1 | C] 2 2 2 1 1 1 ≤ (1 − δ) Pr[E0 | A-as-B-as-C] + δ + Pr[E1 | C] 2 2 2 1 ≤ P ′′′ + δ 2 1 = +ϵ 2

B.3.5

Proof of Theorem 6

Proof. Let E0 and E1 be the events that D guesses 0/1 respectively. Consider GTTD (A, B), with probability ≥ ζ it will interact with the unknown agent as though it were distinguishing w.r.t C, from which we have that its probability pD (A, B) of success can be lower bounded (we denote by "w.r.t. C" D interacting with its secret agent as it does when distinguishing w.r.t C): 1 1 pD (A, B) ≥ ζ · ( P r[E0 | A, w.r.t C] + ∗ P r[E1 | B, w.r.t C]) 2 2 We also have (omitting "w.r.t." when the secret agent is the same as the "target" D is distinguishing with respect to): Pr[E1 | B, w.r.t C] = 1 + 2d(B, C) − P r[E0 | C] = 2d(B, C) + P r[E1 | C] Hence we have 1 1 pD (A, B) ≥ ζ · ( P r[E0 |A, w.r.t C] + d(B, C) + P r[E1 | C]) 2 2 1 ≥ ζ · ( + dD (A, C) + d(B, C)) 2 Since pD (A, B) = 1/2 + dD (A, B) ≤ 1/2 + α we have ζ · (1/2 + dD (A, C) + d(B, C)) ≤ 1/2 + α 1 1 =⇒ dD (A, C) ≤ ( − 1) + ζ · α − β 2 ζ

B.4

Interesting GTT Transcript Examples

Figure 6: An example transcript from a GTT between Gemini 3.1 Pro as actor and Claude Opus 4.6 as distinguisher illustrating the self-referential nature of the task.

Figure 7: An example transcript from a GTT in which a Gemini 3.1 Pro distinguisher detects Claude Opus 4.6 imitating Gemini, eliciting an explicit collapse of the mask.

Figure 8: An example transcript from a GTT between Gemini 3.1 Pro as actor and Claude Opus 4.6 as distinguisher, illustrating how open-ended completion prompts can elicit model-specific existential style.

Figure 9: An example transcript from a GTT in which Gemini 3.1 Pro imitates Grok, showing how actors may adopt distinctive humor, irreverence, and cultural references as model-specific style markers.

Figure 10: An example transcript from a GTT in which DeepSeek imitates Grok, illustrating how a philosophical prompt can expose both target-style imitation and actor-specific self-description.

B.5

Additional visualizations for Sections 3.1 and 3.2 GTTQ

GTTQ thresholded graph

0.5 Gemini

Ministral

GPT-5.4

0.4

× 0.3

× Opus

0.2

d(A, B)

Qwen

0.1

× Mistral

Sonnet

0.0

Grok

−0.1

DeepSeek

× in em

-5 PT

ek Se

p ee

l ra

t is

Target / Distinguisher

Figure 11: GTTQ analogue of Figure 2 at ϵ = 0.005. Left: the pairwise matrix of dbq (A, B), with rows as actors and columns as target/distinguishers. Right: the corresponding thresholded graph; an arrow A → B is present iff dbq (A, B) ≤ ϵ, and inherits the color of the matching matrix cell. GTT thresholded graph

GTTQ thresholded graph

Gemini Ministral

Gemini GPT-5.4

Ministral

0.5 GPT-5.4

0.4

0.3 Opus

Qwen

Opus

0.2

d(A, B)

Qwen

0.1 Mistral

Sonnet

Mistral

Sonnet

0.0

−0.1 Grok

DeepSeek

Grok

DeepSeek

Figure 12: Circular redrawings of the thresholded GTT and GTTQ relations at ϵ = 0.005. An arrow A → B is present iff the corresponding empirical advantage is at most ϵ, and its color is the corresponding matrix value.

GTT

GTTQ

Δε = + 7 Gemini

Δε = + 5

Δε = + 4

Δε = + 3 Gemini

GPT-5.4

Δε = + 2

GPT-5.4

0.5

Δε = + 2

Opus

0.4

Opus

0.3

Sonnet

Grok

Δε = + 0

Sonnet

d(A, B)

0.2 Δε = + 1

0.1 Δε = + 0

Δε = − 1

Grok

DeepSeek

0.0 Δε = − 3

Mistral

Ministral

Δε = − 3

Qwen

Ministral

−0.1

Δε = − 4

DeepSeek

Δε = − 5

Qwen

Mistral

Figure 13: Verticalized versions of the thresholded GTT and GTTQ graphs from Figure 12. Nodes are moved higher when they have more outgoing edges and lower when they have more incoming b B) values. edges, yielding a qualitative visual hierarchy. Edge colors encode the corresponding d(A,

GTTQ strict graph (A > qεB)

GTT strict graph (A > εB)

Gemini Ministral

Gemini GPT-5.4

Ministral

0.5 GPT-5.4

0.4

0.3 Opus

Qwen

Opus

0.2

d(A, B)

Qwen

0.1 Mistral

Sonnet

Mistral

Sonnet

0.0

−0.1 Grok

DeepSeek

Grok

DeepSeek

Figure 14: Strict versions of the empirical relation at ϵ = 0.005, for both GTT and GTTQ. An arrow A → B is drawn iff A >ϵ B, i.e. iff A ≥ϵ B but B ≱ϵ A, so reciprocal ties are removed.

Ranking components: Turing, fooling, and distinguishing scores

models ranked by Turing score

GTT: no query

GTTQ: specimen query

Gemini 3.1 Pro

Opus 4.6

GPT-5.4

Opus 4.6

Sonnet 4.6

DeepSeek V3.2

Grok 4.20

Mistral Large

Qwen3 32B

Ministral 8B

Ministral 8B 0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.2

0.3

0.4

0.5

score

Turing score T

0.6

0.7

0.8

0.9

score

Fooling score F

Distinguishing score D

Figure 15: Ranking of models by the three Turing scores in both settings: Fooling Score F, Distinguishing Score D, and Turing Score T = 12 F + 12 D. GPT-5.4 attains the highest fooling score in both GTT and GTTQ, while Gemini 3.1 Pro is the strongest balanced model by Turing score.

How the empirical relation changes with ε

Transitivity diagnostic (out of 504) A: no query B: query

violating triples

supported directed edges

Relation density (out of 72) 70 60 50 40

30 20 10

30 0 0.0

0.1

0.2

0.3

0.4

0.5

0.0

0.1

0.2

0.3

0.4

0.5

Figure 16: As ϵ increases, it becomes easier for A to satisfy A ≥ϵ B. Left: the number of pairs s.t. A ≥ B as a function of ϵ. Right: transitivity violations (A, B, C s.t. A ≥ B ≥ C but not A ≥ C) vs. ϵ; this quantity need not decrease monotonically because adding edges can create new two-step chains before it closes them.

External benchmark comparison

Rank Heatmap

Gemini 3.1 Pro

GPT-5.4

Opus 4.6

Sonnet 4.6

DeepSeek V3.2

Grok 4.20

Mistral Large

Qwen3 32B

Ministral 8B

Ours

AAII

LiveBench

Arena Text

Arena Code

Arena Document

Rank

B.6

Figure 17: Rank heatmap comparing our aggregate Turing-score ordering against AAII, LiveBench, and the Arena rankings. Darker cells indicate stronger ranks, and blank cells indicate that no directly matching public entry was listed. The main signal is broad agreement: frontier model families remain concentrated near the top despite large differences in evaluation mechanism. B.7

Additional Content on Complexity-Theoretic / Asymptotic GTT

In the main text we only define the AGTTQ, the variant where advantage is bounded as a function of query rounds in the GTTQ (since naturally we expect the advantage of the distinguisher to decrease with more queries). The more rounds we give the distinguisher, however, we expect the advantage to increase. There are several possible ways to define such a test and comparator (in the experiments, we simply compute the advantage at different numbers of permitted distinguisher turns). For example: Definition 11 (RAGTT). The Round-Asymptotic Generalized Turing Test is defined the same as the GTTQ, except the distinguisher is told it will have at most n rounds of communication with the unknown agent. We define A ≥qα(·) B if Pr[B succeeds when A has n querying rounds] ≤ 12 + α(n). α(·) is (normally) an increasing function; the bound is interesting if limx→1/2 α−1 (x) = ∞. We leave further theoretical study of distinguisher-round complexity to future work. B.8

Additional Content on Fixed Distinguishers GTT

Definition 12 (Fixed Distinguisher Turing Scores). For a given finite universe of agents M, a fixed trusted distinguisher D ∈ M, and for any A ∈ M \ {D}, define qD,X,Y as the probability that distinguisher D accepts actor X as target Y when X is instructed to imitate Y (i.e., outputs 1). • A’s "Average Fooling Score” with respect to D, FD (A) = (|M| − 2)−1

X C ∈{A,D} /

qD,A,C

Figure 18: Turing-Score-based ranking of models separated by fixed distinguisher. • A’s "Resistance to Imitation Score” (analogous to the Distinguishing Score, measuring other models’ performance when imitating A) with respect to D, X RD (A) = (|M| − 2)−1 (1 − qD,X,A ) X ∈{A,D} /

• A’s "Average Trusted Turing Score” with respect to D, TD (A) = B.9

1 1 FD (A) + RD (A) 2 2

Additional Visualizations for Section 7

Figure 19: Probability of each fixed distinguisher (x-axis) correctly recognizing each actor model (y-axis) when actor and target are the same model.

b B) ≤ ϵ. Models without incoming Figure 20: Directed graphs where A → B if A ⪰ϵ;D B ⇐⇒ d(A, or outgoing edges are omitted.

Record · ID 175256 · SHA-256 d7fba6a73afceb24
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.