arXiv CSopen access

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data

arXiv:2606.30372v1 [cs.AI] 29 Jun 2026

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data Haobo Yang Department of Computer Science and Engineering, SUSTech University [email protected]* June 30, 2026

Abstract Quantitative research across the social and behavioral sciences depends on human subject experiments that are expensive, slow, and subject to sampling bias. Here we show that pretrained large language models induce risk-equivalent estimators of conditional expectations under squared loss, establishing restricted functional risk equivalence: under squared loss, the LLM induces an estimator whose risk matches the Bayes optimal risk for squared-loss prediction of conditional expectations for any inference that depends on the data only through the conditional mean. We formalize the LLM as a misspecified functional estimator T (P̂n ) trained on i.i.d. data, decompose the estimation error into representation bias ϵrep and optimization error, and prove that under mild regularity conditions the LLM’s expected error converges to the irreducible population variance plus the squared representation bias, with the representation bias bounded by the Pinsker inequality. The identifiability error δ propagates into the effective bias, inflating the asymptotic risk floor. We establish restricted functional risk equivalence via a bidirectional Le Cam deficiency analysis: the forward deficiency vanishes asymptotically while the reverse deficiency is exactly zero. We provide finite-sample concentration bounds and a calibration protocol with explicit decision rules. An open-source Python implementation is available at https://github.com/yhbcode000/llmstat_tool. The result is a precise, provable statement: a well-calibrated LLM achieves the Bayes-optimal risk for conditional-mean-dependent inference, bounded by explicit scope conditions. In practical applications, this means that under satisfied conditions and well-calibrated models, large language models can be used in many prediction and decision-making tasks that originally relied on human experiments, approximating near-optimal statistical inference at lower cost.

Area of research: Artificial Intelligence, Statistical Learning Theory, Formal Methods, and Decision Theory. * This preprint has not undergone professional peer review. If you find any errors or have concerns, please contact

the corresponding author.

1

1

Introduction

Human subject experiments are the cornerstone of quantitative research in the social, behavioural, and health sciences. A typical between-subjects study requires recruiting dozens to hundreds of participants per condition to detect meaningful effect sizes with adequate statistical power 1 . Recruitment is expensive, time-consuming, and bounded by participant availability, self-selection bias, and demographic skew 1 . Underpowered studies produce noisy estimates, and the cost of adequate sample sizes limits the scope and ambition of experimental research programs. The emergence of large language models trained on vast corpora of human-generated text raises a fundamental question: if an LLM has learned statistical regularities of human behaviour from its training data, can its conditional-mean estimator achieve the same squared-loss risk as estimators built from human-subject data? This question is not merely speculative. Researchers across disciplines have begun using LLMs to simulate survey respondents, annotate behavioural data, and generate synthetic experimental outcomes across psychology, political science, marketing, and health research. Existing approaches to validating LLM-based risk agreement rely exclusively on empirical benchmarking: comparing LLM outputs to human baselines on specific prompts and tasks. While these comparisons are informative, they provide no formal guarantee that the risk agreement is mathematically justified. A favourable comparison on one set of prompts does not guarantee validity on another; a high correlation does not guarantee unbiased effect size estimation; a small observed bias does not guarantee that the LLM estimator converges to the relevant population functional. We address this gap by constructing an axiomatic framework for LLM–human functional risk equivalence under squared loss, and by constructing rigorous proofs from first principles. Our approach has three distinguishing features. First, all theorems are rigorous, relying only on stated axioms. We assume nothing about the LLM’s architecture, training algorithm, or parameter count. The core requirements are that training data are representative enough for the learned conditional distribution to converge to the KL projection in the model class, that the conditional-mean functional is Lipschitz under the stated bounded-response assumptions, and that optimization error is op (1). We prove that these assumptions suffice for asymptotic risk closure under squared loss. Second, we make no architectural hypotheses about the LLM. The framework does not require the LLM to be a transformer, to use a specific attention mechanism, or to have any particular depth or width. The model is treated as a black-box estimator whose statistical properties follow from its training objective and data distribution alone. This agnosticism ensures that the framework applies to any sequence-modeling architecture trained with a proper scoring rule on populationrepresentative data. Third, we provide explicit scope boundaries that guide practitioners in determining applicability and prevent over-claiming. We identify the precise conditions under which functional risk equivalence holds (quantitative continuous response, discrete conditions, i.i.d. training data, calibration validation) and those under which it does not (qualitative research, novel paradigms without training-data analogs, safety-critical applications, behavioural mechanism research). These boundaries are formal consequences of the assumptions rather than ad hoc caveats. The remainder of this paper is organized as follows. Section 2 formalizes the study population, the LLM as a statistical estimator, and the connection between cross-entropy pretraining and conditional mean estimation. 2

Section 3 presents the five core theorems with full derivations. Section 4 extends the asymptotic results to finite-sample guarantees via concentration inequalities. Section 5 addresses cross-domain transfer when the training population differs from the target. Section 6 establishes restricted functional risk equivalence under squared loss, and Section 7 provides the calibration protocol with detailed worked examples. The final sections discuss scope and limitations, then give an applied workflow, statistical-method comparison, and costed example before concluding. In practical applications, this means that under satisfied conditions and well-calibrated models, large language models can be used in many prediction and decision-making tasks that originally relied on human experiments, approximating near-optimal statistical inference at lower cost.

2

Mathematical Framework

2.1

Population Model and the LLM Estimator

We formalize a quantitative human subject study as a population P characterized by a finite set of experimental conditions and their associated response distributions. K K Definition 2.1 (Study Population). A study population is a tuple P = (K, {pk }K k=1 , {µk }k=1 , {vk }k=1 ) where:

• K ∈ N is the number of experimental conditions; P • pk ≥ 0 are allocation proportions with K k=1 pk = 1; • µk = E[Y | X = k] ∈ R is the population mean response for condition k; • vk = Var(Y | X = k) ≥ 0 is the population variance for condition k. Definition 2.2 (LLM as Misspecified Functional Estimator). Let P̂n (· | k) denote the conditional distribution learned by the LLM after training on n samples, and let T : P → R be the conditionalmean functional T (Q) = EY ∼Q [Y | X = k]. For a model class F, define the KL projection of the true conditional distribution onto F by PF∗ (· | k) = arg min DKL (P (· | k) ∥ Q) ,

(1)

Q∈F

assuming the minimizer exists and is unique (Assumption A8). The LLM estimator is LLMn (k) = T (P̂n (· | k)).

(2)

We decompose the estimation error into three terms: LLMn (k) − µk = T (P̂n (· | k)) − T (P (· | k)) = T (PF∗ (· | k)) − T (P (· | k)) | {z } ϵrep (k)

+ T (P̂nstat (· | k)) − T (PF∗ (· | k)) |

{z

ηn (k)

}

+ T (P̂n (· | k)) − T (P̂nstat (· | k)) . | {z } ξn (k)

3

(3)

Here P̂nstat denotes the exact empirical risk minimizer or misspecified MLE in the model class F, while P̂n denotes the distribution actually reached by the training algorithm. The terms have different sources of randomness. The representation bias ϵrep (k) is deterministic once P , F, and T are fixed. The statistical estimation term ηn (k) is random through the training sample and converges to zero in probability under the misspecified M-estimation limit. The optimization term ξn (k) is random through algorithmic choices (initialization, minibatch order, numerical optimization) and is assumed to satisfy ξn (k) = op (1) under standard stochastic-approximation conditions for convergent empirical-risk optimization. This is the sole optimization assumption; it is independent of the KL-projection argument, which concerns the deterministic model-class limit. We write ϵopt (k, n) = ηn (k) + ξn (k) for their sum. In the fixed-class regime, ϵrep (k) is constant in n. In the growing-class regime Fn ↑ F∞ , the representation bias is ϵrep,n (k) = T (PF∗n (· | k)) − T (P (· | k)) and may shrink with model-class richness. Special case: well-specified Gaussian model. When the model class P contains the true conditional −1 distribution and the output layer is Gaussian, T (P̂n (· | k)) = nk Xi =k Yi , the empirical mean. This is the idealized case analyzed in the formal proofs. For misspecified models, ϵrep (k) may be non-zero; Section 5.4 quantifies its impact. The allocation proportions {pk } represent the relative frequency with which each condition appears in the population. In a balanced experimental design, pk = 1/K for all k; in observational settings, the pk reflect natural prevalence. The mean parameters µk capture the central tendency of responses, and the variance parameters vk capture within-condition variability due to irreducible individual differences. Equation (2) models the LLM as a functional of the learned conditional distribution. This is not an arbitrary simplification: it follows from the structure of modern LLM training. We establish this formally in Section 2.4. For now, we note that under a Gaussian output layer, the conditional mean functional reduces to the empirical mean, which is the maximum likelihood estimator of µk ; more generally, any consistent estimator of the conditional distribution whose mean functional is continuous converges to µk + ϵrep (k) as n → ∞. Theorem 2.3 (Representation Limit Identification). Fix a condition k and suppose P̂n (· | k) converges to the KL projection PF∗ (· | k) in Hellinger distance: H(P̂n (· | k), PF∗ (· | k)) → 0 in probability. Under A7–A8, the LLM estimator has the unique limit p

LLMn (k) = T (P̂n (· | k)) → − T (PF∗ (· | k)) = µk + ϵrep (k). Equivalently, ϵrep (k) is the unique limiting offset between the LLM estimator and the true conditional mean. Proof. By A8, T is Lipschitz in Hellinger distance on the mean-regular model class: |T (Q1 ) − T (Q2 )| ≤ LH(Q1 , Q2 ). Therefore |T (P̂n (· | k)) − T (PF∗ (· | k))| ≤ LH(P̂n (· | k), PF∗ (· | k)) → 0. The final equality follows from the definition ϵrep (k) = T (PF∗ (· | k)) − T (P (· | k)) and T (P (· | k)) = µk .

4

Theorem 2.4 (Final Risk Closure under Misspecification). Under A7–A10, if H(P̂n (· | k), PF∗ (· | k)) → 0 in probability for each k, then h
2
2 i E T (P̂n (· | k)) − T (P (· | k)) = T (PF∗ (· | k)) − T (P (· | k)) + o(1) = ϵrep (k)2 + o(1). Equivalently, since LLMn (k) = T (P̂n (· | k)) and µk = T (P (· | k)),   E (LLMn (k) − µk )2 = ϵrep (k)2 + o(1). Consequently, K X   E E(LLMn ) = Emin + pk ϵrep (k)2 + o(1). k=1

Proof. Theorem 2.3 gives T (P̂n (· | k)) − T (P (· | k)) → ϵrep (k) in probability. By A7, both T (P̂n (· | k)) and T (P (· | k)) lie in [−B, B], hence the squared difference is bounded by (2B)2 and uniformly integrable. Convergence in probability plus uniform integrability implies convergence in L1 of the squared difference, yielding the first display. Summing the per-condition limits with weights pk and adding the irreducible variance term from Theorem 2.8 yields the aggregate risk closure. Theoretical separation. The framework rests on three logically independent layers: Notation. Throughout, µk = E[Y | X = k] denotes the population (theoretical) conditional mean; LLMn (k) denotes the LLM estimator (a random variable); ϵrep (k) is the non-random asymptotic representation bias; ηn (k) is sampling/statistical estimation error over the training-data probability space; ξn (k) is algorithmic optimization error over the training procedure’s randomness; bbk and δb are empirical estimates from calibration data (random). Hats (b·) mark quantities estimated from finite samples; tildes (˜·) mark quantities adjusted for identifiability error. (1) Statistical identity (Section 3): Under squared loss, the conditional expectation µk = E[Y | X = k] is the unique admissible predictor. This is a mathematical fact about the loss function; it does not reference the LLM. (2) Learning theory (Section 2.4): Under cross-entropy training with i.i.d. data, the LLM’s learned distribution converges to the KL-projection of the true distribution onto the model class. The conditional mean functional T is Lipschitz-continuous, so the LLM estimator LLMn (k) = T (P̂n ) converges to T (PF∗ ). (3) Decision theory (Section 6): Because the risk depends on θ only through µk , the convergence of LLMn (k) to µk + ϵrep (k) implies convergence of decision risk. The representation bias ϵrep (k) sets a lower bound on achievable risk. No layer assumes the conclusion of another: Layer 1 is purely analytic, Layer 2 is empirical (depends on training data and model class), and Layer 3 is decision-theoretic (depends on the loss class). Dependency ordering. The three layers are strictly ordered: Layer 1 (statistical identity) is a mathematical fact about squared loss that holds independently of any estimator or model class. 5

Layer 2 (learning theory) depends on the model class F, training data, and the functional delta stability assumption (A9). Layer 3 (decision theory) depends on the convergence established in Layer 2 and the risk decomposition of Layer 1. No layer assumes the conclusion of any later layer; the argument is a strict forward chain 1 → 2 → 3. Definition 2.5 (Stochastic Identifiability of Conditions). There exists a measurable mapping ϕ : S → [K] from the space of prompt strings S to the set of experimental conditions [K], such that for a prompt s encoding condition k, P(ϕ(s) = k | s encodes k) ≥ 1 − δ,

(4)

where δ ∈ [0, 1) is the misclassification rate. The mapping ϕ and the bound δ are estimated during calibration (Section 7, Step 1b). For finite-sample deployment, δ is a calibrated upper bound on prompt-to-condition ambiguity. For asymptotic statements, we write δ = δn and assume −1/2 δn = Op (Mv ) from the held-out validation sample of size Mv ; when Mv grows proportionally to the training/evaluation budget, this gives the stated δn = Op (n−1/2 ) regime. When δ = 0, we recover the deterministic case; when δ > 0, the non-identifiability failure mode (Theorem 5.14, case ii) applies. Effective estimator under misclassification. When δ > 0, the LLM effectively estimates a mixture of condition means rather than the target condition mean. For condition k, the expected output is X E[LLMn (k) | misclassification] = (1 − δ) · LLMn (k) + δ · wkj LLMn (j), (5) j̸=k

P

where wkj are the misclassification weights ( j̸=k wkj = 1). The effective representation bias is therefore ϵ̃rep (k; δ) = ϵrep (k) + δ · ∆k , (6) P where ∆k = µk − j̸=k wkj µj is the misclassification contrast. For δ → 0, we recover the original bias; for δ > 0, the effective bias is shifted by at most δ · (maxj µj − minj µj ) in magnitude, and all downstream risk limits use ϵ̃rep (k; δ) in place of ϵrep (k).

2.2

Risk Functional Setup

All equivalence and validity statements in this paper are statements about the squared-loss risk functional, not about equality of response distributions. For a deterministic predictor f : [K] → R, define the population risk K   X   RP (f ) = E (Y − f (X))2 = pk E (Y − f (k))2 | X = k .

(7)

k=1

This is the same object denoted E(f ) below. The Bayes risk is RP∗ =

inf f :[K]→R

RP (f ) = Emin =

K X k=1

6

pk vk ,

(8)

attained by the Bayes act f ∗ (k) = µk . For a random estimator fˆn learned from training data and algorithmic randomness, its estimator risk is the outer expectation h i Rn (fˆn ) = Etrain,opt RP (fˆn ) . (9) The LLM estimator risk is obtained by setting fˆn (k) = LLMn (k). In the fixed-class misspecified limit this risk converges to K X mis RLLM,∞ = Emin + pk ϵrep (k)2 , (10) k=1

or, under stochastic identifiability error, with ϵrep (k) replaced by ϵ̃rep (k; δ) from Eq. 6. For two information objects or estimators E and F with induced risk functionals RE and RF , define E ∼ L2 F

⇐⇒

inf RE (f ) = inf RF (f ), f

f

(11)

where the infimum is over the same conditional-mean-dependent decision class. This is the formal restricted risk-equivalence relation used throughout the paper. The squared-loss restriction is load-bearing because the conditional mean is the Bayes act; analogous extensions to Bregman divergences replace the conditional mean by the corresponding Bayes act, but those extensions are outside the present paper’s formal claims. For backward compatibility with the theorem statements below, we write E(f ) = RP (f ).

2.3

(12)

The Bias-Variance Decomposition

The central structural result of our framework is the decomposition of expected error into an irreducible component and an excess component attributable to the predictor’s deviation from the conditional mean. Lemma 2.6 (Conditional Bias-Variance Identity). For any condition k and any predictor value e = f (k) ∈ R,   E (Y − e)2 | X = k = vk + (µk − e)2 . (13) Proof. Expand the square and use the definitions of µk and vk : E[(Y − e)2 | X = k] = E[Y 2 − 2eY + e2 | X = k] = E[Y 2 | X = k] − 2eE[Y | X = k] + e2 = (Var(Y | X = k) + E[Y | X = k]2 ) − 2eµk + e2 = vk + µ2k − 2eµk + e2 = vk + (µk − e)2 . Line 3 uses the identity E[Y 2 ] = Var(Y ) + E[Y ]2 applied conditionally. A more general form of this identity holds for arbitrary probability weights. This will be useful when we consider weighted error decompositions across conditions. 7

P Lemma 2.7 (Weighted Bias-Variance Identity). Let {wk }K with k=1 be nonnegative weights k wk = P K 1, and let {yk }k=1 be arbitrary real numbers. Define the weighted mean m = k wk yk and the P weighted variance v = k wk (yk − m)2 . Then for any e ∈ R, K X

wk (yk − e)2 = v + (m − e)2 .

(14)

k=1

Proof. Expand each term around m: X X  2 wk (yk − e)2 = wk (yk − m) + (m − e) k

k

=

X

wk (yk − m)2 + 2(m − e)

k

X

wk (yk − m) + (m − e)2

k

X

wk .

k

P P P The cross term vanishes because k wk (yk − m) = k wk yk − m k wk = m − m = 0. The first P term is v, and the third term is (m − e)2 since k wk = 1. Lemma 2.7 is the algebraic core of the entire framework. It cleanly separates the variance intrinsic to the weighted distribution from the squared deviation of the estimator e from the mean m. Applying Lemma 2.6 condition by condition and weighting by pk yields the fundamental decomposition. Theorem 2.8 (Error Decomposition). For any predictor f : [K] → R, E(f ) =

K X

pk vk +

K X

k=1

k=1

| {z }

|

Emin

pk (µk − f (k))2 , {z

Excess(f )

(15)

}

where Emin depends only on the population and Excess(f ) ≥ 0 quantifies the squared bias of f . Proof. From (12) and Lemma 2.6: X X  X  X E(f ) = pk E[(Y −f (k))2 | X = k] = pk vk +(µk −f (k))2 = p k vk + pk (µk −f (k))2 . k

k

k

k

The second term is nonnegative since each pk ≥ 0 and squares are nonnegative. P We denote Emin = k pk vk as the irreducible error. This quantity represents the minimum achievable expected squared error for any predictor of Y given X under squared loss: it is the Bayes risk. No predictor, human or machine, can achieve expected error below Emin , because within-condition variance reflects genuine irreducible heterogeneity in the population.

2.4

Connection to Maximum Likelihood Estimation

Modern LLMs are trained via next-token prediction with cross-entropy loss. For a vocabulary V and a sequence of tokens, the training objective at each position is to minimize − log Pθ (v | context) over model parameters θ, where Pθ is the model’s predicted distribution over V. The theory 8

in this paper analyzes the limiting functional induced by cross-entropy training under idealized infinite-data, vanishing-optimization-error conditions. Finite-step, non-convex training dynamics are represented only through the ξn = op (1) optimization error term in Eq. 3. Layer 1: Distribution learning. Under cross-entropy training, the LLM minimizes a strictly proper scoring rule. Let F be the model class. Under standard regularity conditions (compact parameter space, smooth log-likelihood), the learned distribution converges to the best approximation in the model class: p P̂n (· | k) → − PF∗ (· | k) as n → ∞, (16) where PF∗ = arg minQ∈F DKL (P ∥Q) is the KL-projection of the true distribution onto F. When the model is correctly specified (P ∈ F), PF∗ = P . Layer 2: Functional extraction. The conditional mean functional T (Q) = EY ∼Q [Y | X = k] is Lipschitz-continuous with respect to Hellinger distance under the bounded-response assumption (|Y | ≤ B). Specifically, |T (P ) − T (Q)| ≤ 2B · H(P, Q). By the continuous mapping theorem and Hellinger convergence (Layer 1; the functional delta stability of T , Assumption A9, guarantees that Hellinger convergence of distributions implies convergence of conditional means): p

LLMn (k) = T (P̂n (· | k)) → − T (PF∗ (· | k)).

(17)

Layer 3: Bias-variance decomposition. The limit T (PF∗ (· | k)) equals µk + ϵrep (k) where ϵrep (k) = T (PF∗ (· | k)) − µk is the asymptotic representation bias. Under correct specification, p ϵrep (k) = 0 and LLMn (k) → − µk . R √ √ Convergence metric. Let H 2 (P, Q) = 21 ( p − q)2 dµ denote the squared Hellinger distance. Under the regularity conditions of MLE theory 2 , the learned distribution converges in Hellinger distance to the KL-projection: p

H 2 (P̂n (· | k), PF∗ (· | k)) → − 0

as n → ∞.

(18)

When the model is correctly specified (P ∈ F), the limit is the true distribution P . Otherwise, it is the best approximation within the model class. Lipschitz continuity of the mean functional. For responses bounded in [−B, B], the conditional mean functional T (Q) = EY ∼Q [Y | X = k] satisfies the Lipschitz bound R|T (P ) − T (Q)| ≤ 2B · H(P, Q). This R follows from the dual representation EP [Y ] − EQ [Y ] = y(p(y) − q(y))dy and the bound | f (p − q)| ≤ 2B · H(P, Q) for bounded f (see 2 , Lemma 5.34). Consequently, Hellinger convergence of distributions implies convergence of conditional means, even under misspecification: p

− 0. |LLMn (k) − T (PF∗ (· | k))| ≤ 2B · H(P̂n (· | k), PF∗ (· | k)) → This bound ensures that not only the mean but any Lipschitz functional of the conditional disb used in calibration) converges to its model-class optimum. tribution (including the variance ratio λ Correct specification is sufficient but not necessary for mean convergence; bounded responses and Hellinger consistency suffice. Cross-entropy is not merely a convenient loss function; it is a strictly proper scoring rule 5 . Under correct specification, the MLE recovers the true conditional distribution; under misspecification, it recovers the KL-projection PF∗ . In either case, the LLM learns the best achievable approximation to the full conditional distribution within its model class. This is the fundamental 9

bridge from estimator equivalence to distributional approximation: the LLM learns the conditional distribution up to model-class limitations, and the learned distribution converges to the best approximation in a strong metric. The remaining gap—that the LLM’s sampling process is not identical to the human response process—is addressed by the decision-theoretic framework of Section 6, which shows that for squared-loss inference, distributional identity is not required; statistical sufficiency of the conditional mean suffices.

2.5

Assumptions and their Justification

We now enumerate and justify the assumptions that underpin the entire framework. Assumption summary. The headline theorem requires the following compact set of conditions: finite conditions; representative i.i.d. or stationary ergodic training data; finite conditional variance and bounded responses; a mean-regular model class with unique KL projection PF∗ ; Lipschitz continuity of the conditional-mean functional T in Hellinger distance; convergence of the learned distribution to the KL projection; optimization error ξn = op (1) under standard stochastic-approximation −1/2 conditions; and calibrated identifiability error δn = Op (Mv ) (or Op (n−1/2 ) when Mv ≍ n). The enumerated assumptions below spell out these conditions and their roles. A1. Finite conditions. K < ∞. This holds for any experiment with a finite set of treatments, conditions, or groups. The framework does not apply to continuous treatments (dose-response with continuous dosage) without discretization. A2. Independent and identically distributed training data. {(Xi , Yi )}ni=1 ∼i.i.d. PX,Y . This is the standard assumption in statistical learning theory. It is satisfied when the LLM’s training corpus is a representative random sample from the target population. Violations (e.g., temporal drift, selection bias in web-scraped data) require domain adaptation techniques that we address partially in Section 5. A3. Finite variance. vk < ∞ for all k. Required for the Strong Law of Large Numbers to apply. For bounded response scales (Likert scales, proportions, test scores), this is automatically satisfied. For unbounded responses (reaction times, monetary amounts), this is a mild regularity condition verified in most behavioural data. A4. Conditional mean functional estimation. LLMn (k) = T (P̂n (· | k)), where PT (Q) = EQ [Y | −1 X = k]. In the well-specified Gaussian special case this reduces to nk Xi =k Yi , but the general theory treats the LLM as a functional of the learned conditional distribution. A5. No architectural assumptions. The framework does not assume any specific neural network architecture, depth, width, attention mechanism, or activation function. The LLM is treated as a black-box statistical estimator whose asymptotic properties depend only on the training objective and data distribution. A6. Within-distribution evaluation. The target experimental conditions are represented in the training distribution. That is, for each condition k of interest, pk > 0. Conditions with zero training prevalence cannot be estimated.

10

A7. Bounded responses. P(|Y | ≤ B) = 1 for some known constant B < ∞. This holds for all common response scales (Likert scales, proportions in [0, 1], standardized test scores) and ensures the conditional mean functional is Lipschitz-continuous with respect to Hellinger distance (Section 2.4). A8. Mean-regular model class. The model class F is mean-regular: the functional T (Q) = EQ [Y | X = k] is uniquely defined at the KL-projection PF∗ , and T is Lipschitz-continuous on F with respect to Hellinger distance: |T (Q1 ) − T (Q2 )| ≤ L · H(Q1 , Q2 ) for all Q1 , Q2 ∈ F, with Lipschitz constant L = 2B under A7 (bounded responses). This ensures ϵrep (k) = T (PF∗ (· | k)) − µk is a well-defined, finite quantity. All standard parametric families (exponential families, location-scale families, mixture models with bounded support) satisfy this condition. A9. Functional delta stability. The conditional mean functional T (Q) = EQ [Y | X = k] satisfies the functional delta method bound: |T (PF∗ ) − T (P )| ≤ C · H(P, PF∗ ), where C = 2B under A7. This ensures that KL-projection preserves the conditional mean up to Hellinger-controlled error, even under misspecification. Equivalently, the model class is mean-closed in the limit: limn→∞ EP̂n [Y | X = k] = T (PF∗ (· | k)). A10. Stationary ergodic training distribution. The joint process {(Xi , Yi )}ni=1 is stationary and ergodic conditional on X. This is the standard assumption in misspecified learning theory (cf. White, 1982). For i.i.d. data, stationarity and ergodicity are automatically satisfied. These ten assumptions are substantially weaker than those required by existing LLM evaluation frameworks.

2.6

Joint Asymptotic Regime

All asymptotic statements distinguish two regimes. (i) Data limit: n → ∞ with fixed model class F. The learned distribution converges to the fixed KL projection PF∗ , estimation/optimization error vanishes, and the representation bias ϵrep (k) = T (PF∗ (· | k)) − T (P (· | k)) is invariant in n. −1/2

(ii) Identifiability limit: δ = δn → 0 as n → ∞, with δn = Op (Mv ) from the validation sample and δn = Op (n−1/2 ) when Mv ≍ n. Finite-n risk uses the effective bias ϵ̃rep (k; δ) = ϵrep (k) + δ∆k from Eq. 6; this extra term vanishes under the joint identifiability limit. (iii) Growing-class limit: Fn ↑ F∞ , with KL projections PF∗n and KL-diameter DFn = sup inf DKL (P ∥Q) P

Q∈Fn

satisfying DFn → 0. In this regime ϵrep,n (k) = T (PF∗n (· | k)) − T (P (· | k)) tends to zero by Theorem 5.10. P The fixed-class regime proves the misspecified limit Emin + k pk ϵrep (k)2 (Theorem 2.4 and Theorem 5.6). The growing-class regime is required only for the stronger Bayes-risk claim E(LLMn ) → Emin . 11

Theorem 2.9 (Unified Functional Risk Closure). Let P (· | k) be the true conditional response law, let F be a model class with KL projection PF∗ (· | k), and let T (Q) = EQ [Y | X = k] be Lipschitz in Hellinger distance. Let LLMn (k) = T (P̂n (· | k)) be the estimator induced by cross-entropy training, with statistical error ηn (k) → 0 in probability and optimization error ξn (k) = op (1). Under Assumptions A1–A10 and stochastic identifiability error δn , the fixed-class misspecified regime satisfies, for each k,  
2
2 E (LLMn (k) − µk )2 = ϵrep (k)+δn ∆k +o(1) = T (PF∗ (· | k))−T (P (· | k)) +o(1)+O(δn ), and hence

K X  
2 E E(LLMn ) = Emin + pk ϵrep (k) + δn ∆k + o(1). k=1

If additionally δn → 0 and either the model is well specified or Fn ↑ F∞ with DFn → 0, then E[E(LLMn )] → Emin , and the LLM estimator is L2 -risk-equivalent to the human-data estimator in the sense of Eq. 11. Proof. The representation identity ϵrep (k) = T (PF∗ (· | k)) − T (P (· | k)) is Definition 2.2. The convergence of T (P̂n ) to T (PF∗ ) is Theorem 2.3, using the Lipschitz property of T and Hellinger convergence. The per-condition squared-risk limit is Theorem 2.4; the δn ∆k term is Corollary 5.9; and the aggregate identity follows from the risk decomposition Theorem 2.8. If DFn → 0, the representation bias vanishes by Theorem 5.10; if δn → 0, the identifiability contribution vanishes by Definition 2.5. The final equivalence statement is exactly the relation ∼L2 in Eq. 11. Remark 2.10 (Interpretation of the main theorem). Theorem 2.9 is the paper’s central claim. It does not say that an LLM reproduces human response distributions or simulates individual human behaviour. It says that, for conditional-mean-dependent decisions under squared loss, the LLM estimator has a fully characterized asymptotic risk: Bayes risk plus a KL-projection representationbias floor, with calibrated identifiability and optimization errors separated explicitly.

3

Core Theorems

3.1

Optimality of the Conditional Mean

The error decomposition (15) immediately yields the optimality of the population conditional mean. Theorem 3.1 (Irreducible Error is the Global Minimum). For any predictor f : [K] → R, Emin ≤ E(f ).

(19)

Equality is attained if and only if f (k) = µk for every k with pk > 0. In particular, the optimal predictor is f ∗ (k) = µk . Proof. From Theorem 2.8, E(f ) = Emin + Excess(f ). SinceP each term pk (µk − f (k))2 ≥ 0, we have Excess(f ) ≥ 0, establishing (19). For equality, we need k pk (µk − f (k))2 = 0. Since each term in the sum is nonnegative, this requires pk (µk − f (k))2 = 0 for all k. When pk > 0, this forces f (k) = µk . When pk = 0, f (k) can be arbitrary without affecting the error. 12

Corollary 3.2 (Uniqueness of the Optimal Predictor). The optimal predictor f ∗ is unique on the support of the allocation distribution: if f and g both achieve E(f ) = E(g) = Emin , then f (k) = g(k) = µk for all k with pk > 0. Theorem 3.1 establishes that the population conditional mean is the uniformly best predictor under squared loss. Any deviation from the conditional mean incurs a positive excess error proportional to the squared deviation weighted by condition prevalence. This is the fundamental justification for why the LLM, which estimates the conditional mean, is the correct target: if the LLM converges to µk , then its expected error converges to the theoretical minimum. Remark 3.3 (Prediction bound, not causal bound). The irreducible error Emin is the minimum achievable expected squared prediction error for the conditional mean E[Y | X = k]. It is a prediction-theoretic lower bound under squared loss, not a causal or mechanistic limit. It does not bound the error of estimating treatment effects under confounding, mediation, or structural equation models. Throughout this paper, “optimal” and “minimal” refer to prediction risk, not causal identification. Remark 3.4. The result is independent of the response distribution. No normality, symmetry, or unimodality assumptions are required. The squared error loss is the only structural element: it is a proper scoring rule whose expected minimizer is the conditional mean 2 .

3.2

Effect Size Consistency

In between-subjects experimental designs, the quantity of primary interest is the difference in means between conditions: δ(i, j) = µi − µj . The LLM-based estimator of this effect size is δbn (i, j) = LLMn (i) − LLMn (j). We now establish that this estimator is strongly consistent. Theorem 3.5 (Strong Consistency of Effect Size Estimates). Let training samples {(Xi , Yi )}ni=1 be i.i.d. from P with E[Y | X = k] = µk and Var(Y | X = k) = vk < ∞ for all k. Assume pk > 0 for all k ∈ [K]. Then for any i, j ∈ [K], a.s. δbn (i, j) = LLMn (i) − LLMn (j) −→ µi − µj

as n → ∞.

(20)

Proof. We proceed in three steps. Step 1: Per-condition convergence. Fix a condition k. The subsequence {Yi : Xi = k} consists of nk i.i.d. draws from the conditional distribution of Y | X = k, with mean µk and finite variance vk . By Etemadi’s Strong Law of Large Numbers 3 , which requires only pairwise independence (weaker than mutual independence), we have: 1 X a.s. Yi −→ µk as nk → ∞. LLMn (k) = nk i:X =k i

a.s.

Since pk > 0, we have nk −→ ∞ as n → ∞ (the number of visits to each condition grows without bound almost surely), so the condition nk → ∞ is satisfied almost surely. Step 2: Finite intersection of almost-sure events. For each k, define the event Ak = {ω : LLMn (k)(ω) → µk }. Step 1 establishes P(Ak ) = 1 for each k. The intersection of finitely many almost-sure events is almost-sure: ! ! K K K \ [ X P Ak = 1 − P Ack ≥ 1 − P(Ack ) = 1. k=1

k=1

13

k=1

2 Step 3: Continuous mapping. The function g(x, y) T = x − y is continuous on R . By the continuous mapping theorem, on the almost-sure event k Ak , we have:

δbn (i, j) = LLMn (i) − LLMn (j) = g(LLMn (i), LLMn (j)) → g(µi , µj ) = µi − µj . Convergence holds almost surely. Corollary 3.6 (Joint Consistency). The vector of LLM estimates (LLMn (1), . . . , LLMn (K)) converges almost surely to (µ1 , . . . , µK ). Consequently, any continuous functional of the mean vector, including all pairwise contrasts, ANOVA F-statistics, and linear combinations, is strongly consistent. The practical implication is that with sufficient training data drawn from the target population, the LLM recovers the true effect sizes with probability approaching one. The convergence rate is −1/2 Op (nk ) by the Central Limit Theorem; we quantify this in Section 4.

3.3

Power Amplification

A key practical advantage of LLM-based studies is the ability to increase sample size at near-zero marginal cost. Human studies are bounded by recruitment budgets; LLM studies can generate additional responses at the cost of compute. We formalize this advantage through a monotonicity result for statistical power. Theorem 3.7 (Power Monotonicity). Consider a two-sided z-test for the null hypothesis H0 : δ = 0 against H1 : δ = ∆ > 0, with known variance σ 2 > 0 and significance level α ∈ (0, 1). For sample sizes N ≥ M ≥ 1, √ √ N∆ M∆ − zα/2 ≥ − zα/2 . (21) σ σ Consequently, if Φ denotes the standard normal CDF, the power satisfies Power(N ) ≥ Power(M ). √ √ √ Proof. Since · is strictly increasing on [0, ∞), we have N ≥ M . Multiplying by the positive constant ∆/σ > 0 preserves the inequality, and subtracting the constant zα/2 yields (21). Since Φ is strictly increasing, applying it to both sides yields: ! ! √ √ N∆ M∆ Φ − zα/2 ≥ Φ − zα/2 . σ σ The left side is the power at sample size N (the probability of rejecting H0 when H1 is true), and the right side is the power at M . Corollary 3.8 (Sample Size Amplification Factor). For a target power level β, the required sample size is n∗ = ⌈(zα/2 + zβ )2 σ 2 /∆2 ⌉. If the LLM enables √ a c-fold increase in sample √ size (NLLM = cNhuman ) at lower cost, the power increases from Φ( Nhuman ∆/σ − zα/2 ) to Φ( cNhuman ∆/σ − zα/2 ). Example 3.9. For a small effect (∆/σ = 0.2) with Nhuman = 100 and α = 0.05, the human-study power is Φ(0.2 × 10 − 1.96) √ = Φ(0.04) ≈ 0.516. With c = 10 (LLM enables NLLM = 1000), the power becomes Φ(0.2 × 1000 − 1.96) = Φ(4.36) > 0.999. This transforms an underpowered study into a virtually certain detection of the true effect. 14

The amplification √is most dramatic for small effect sizes, where human studies are perennially underpowered. The n scaling of the test statistic means that a c-fold increase in sample size yields √ only a c-fold increase in the noncentrality parameter, but for modest c and small baseline power, the gain in detection probability can be transformative.

3.4

Main Result: Asymptotic Validity

We now combine the preceding results to establish the main theorem: the LLM predictor achieves the Bayes-optimal risk under squared loss, matching or exceeding the risk performance of any human predictor. Theorem 3.10 (Risk Convergence under Squared Loss). Under Assumptions A1–A10 and the joint limit (i)–(ii) of Section 2.6, as n → ∞, a.s.

E(LLMn ) −→ Emin , Emin ≤ E(h) for every predictor h : [K] → R.

(22) (23)

Remark 3.11 (Misspecification extension). Theorem 3.10 addresses the well-specifiedP case (ϵrep (k) = 0 for all k). Under misspecification (ϵrep (k) ̸= 0), the asymptotic risk limit is Emin + k pk ϵrep (k)2 (Theorem 5.6), with the representation bias bounded by Theorem 5.10 (Pinsker). The L1 convergence is established in Lemma 5.7. Proof. Proof of (22). Using the error decomposition (15) applied to LLMn : E(LLMn ) = Emin +

K X

pk (µk − LLMn (k))2 .

k=1 (k)

For each k with pk > 0, define the sequence Zn = pk (µk − LLMn (k))2 . By Theorem 3.5, a.s. LLMn (k) −→ µk . Since the function z 7→ pk (µk − z)2 is continuous, the continuous mapping (k) a.s. theorem yields Zn −→ pk (µk − µk )2 = 0. A finite sum of almost-surely vanishing sequences vanishes almost surely: K X

a.s.

Zn(k) −→ 0.

k=1

P

(k) a.s. → Emin . k Zn −

Therefore E(LLMn ) = Emin + Proof of (23). This is Theorem 3.1 applied to an arbitrary predictor h, which yields Emin ≤ E(h) for all h : [K] → R. In particular, for any individual human subject viewed as a predictor (whose response on condition Pk is a random draw from the conditional distribution), the expected error is E(human) = Emin + k pk vk = 2Emin (since a single human draw has variance 2vk around its own mean), while the LLM achieves Emin asymptotically. Remark 3.12 (Interpretation). Equation (22) states that the LLM’s expected prediction error converges to the Bayes risk — the theoretical minimum achievable by any predictor with knowledge of the true conditional distribution. Equation (23) states that this Bayes risk is a universal lower bound: no predictor, whether an individual human, an ensemble of humans, or any other statistical procedure, can achieve lower expected squared error. Together, they establish that a well-trained LLM is, in the asy... 15

4

Finite-Sample Analysis

The asymptotic results of Section 3 guarantee correctness in the limit, but practitioners require guarantees at finite sample sizes. We provide concentration bounds that quantify the rate of convergence and the probability of large deviations.

4.1

Concentration of the LLM Estimator

Theorem 4.1 (Hoeffding Bound for Per-Condition Estimates). Assume that for each condition k, the response Y | X = k is bounded in [ak , bk ] with bk − ak ≤ Mk . Then for any ε > 0 and any k with nk > 0,  
2nk ε2 . (24) P |LLMn (k) − µk | ≥ ε nk ≤ 2 exp − Mk2 Proof. Conditional on nk , the nk observations in condition k are i.i.d. and bounded. Hoeffding’s inequality for bounded random variables 4 yields the stated bound directly. The unconditional probability integrates over the random variable nk ∼ Binomial(n, pk ). For large n, nk ≈ npk with high probability. Corollary 4.2 (Uniform Concentration over Conditions). With probability at least 1 − δ, s log(2K/δ) max |LLMn (k) − µk | ≤ M , k:nk >0 2 mink nk where M = maxk Mk . Proof. Apply Theorem 4.1 with a union bound over the K conditions. For each k, set the right side of (24) to δ/K and solve for ε. The maximum bound follows from taking the worst-case nk . Corollary 4.3 (Concentration of Effect Size Estimates). For any pair of conditions i, j, with probability at least 1 − δ, s s log(4/δ) log(4/δ) + Mj . |δbn (i, j) − δ(i, j)| ≤ Mi 2ni 2nj This follows from the triangle inequality and applying Theorem 4.1 to each condition with δ/2.

4.2

Sample Size Determination

For the LLM estimator to achieve a desired precision ε with confidence 1−δ in estimating a specific effect size δ(i, j), the required per-condition sample size is: 2Mk2 log(4K/δ) . (25) ε2 This formula provides a direct bridge between the theoretical framework and practical study design. For a balanced design with K = 4 conditions, Mk = 1 (e.g., proportion responses bounded in [0, 1]), ε = 0.05, and δ = 0.05, we need nk ≥ 2 · log(320)/0.0025 ≈ 4, 616 samples per condition, or approximately 18, 500 total LLM responses. While this exceeds typical human-study sample sizes, LLM inference costs are negligible compared to human recruitment, and the sample size is achievable with modern inference infrastructure. nk ≥

16

4.3

Berry-Esseen Refinement

For response distributions that are not severely non-normal, the Central Limit Theorem provides a 3/2 more precise characterization. Let ρk = E[|Y − µk |3 | X = k]/vk denote the skewness of the conditional distribution. By the Berry-Esseen theorem, √  nk (LLMn (k) − µk ) Cρk sup P ≤ t nk − Φ(t) ≤ √ , √ vk nk t∈R where C ≤ 0.4748 is the universal Berry-Esseen constant. For nk ≥ 100 and ρk ≤ 2 (moderate skewness), the normal approximation error is below 0.1, justifying standard z-test and t-test procedures for LLM-generated data.

5

Cross-Domain Transfer

Practical deployment often involves an LLM pretrained on a source population Psrc that differs from the target population Ptgt of scientific interest. The training corpus for large-scale LLMs is typically a web-scale dataset whose demographic composition may not match any specific study population. We quantify the resulting performance degradation.

5.1

Problem Setup

Let the source and target populations share the same condition set [K] but differ in their allocation proportions and conditional moments: Psrc = (K, {pk }, {µk }, {vk }), Ptgt = (K, {p′k }, {µ′k }, {vk′ }). The LLM is trained on data from Psrc and produces estimates LLMn (k) converging to µk (the source conditional means). When evaluated on Ptgt , the expected error is: Etgt (LLMn ) = Emintgt +

K X

p′k (µ′k − LLMn (k))2 .

k=1

5.2

Transfer Gap and its Bound

Definition 5.1 (Mean Squared Gap). The mean squared gap between source and target populations is K X G= p′k (µ′k − µk )2 . k=1

The maximum mean discrepancy is ρ = maxk∈[K] |µ′k − µk |. Maximum Mean Discrepancy. For two distributions P, Q over responses conditioned on k, define MMDk (P, Q) =

sup

|EY ∼P [f (Y ) | X = k] − EY ∼Q [f (Y ) | X = k]|,

f ∈F :∥f ∥∞ ≤1

17

(26)

where F is the unit ball in a reproducing kernel Hilbert space. For squared loss, taking f (Y ) = Y yields the conditional mean difference as a lower bound: |µk − µ′k | ≤ MMDk . The cross-domain bound then upgrades to: X X G= p′k (µ′k − µk )2 ≤ p′k · MMD2k ≤ max MMD2k . k

k

k

Lemma 5.2 (Gap Bound). G ≤ ρ2 . ′ 2 2 Proof. P ′ For each k, (µk − µk ) ≤ ρ by definition of the maximum. Weighted summation with k pk = 1 yields: X X X p′k ρ2 = ρ2 p′k = ρ2 . G= p′k (µ′k − µk )2 ≤ k

k

k

The bound is tight when all the discrepancy is concentrated in a single condition with p′k = 1, or when the discrepancy is uniform across conditions. In general, G will be substantially smaller than ρ2 when discrepancies vary in sign and magnitude across conditions, as the weighting by p′k and the squaring penalize large deviations in prevalent conditions. a.s.

Theorem 5.3 (Cross-Domain Transfer Bound). Let LLMn be an estimator satisfying LLMn (k) −→ µk (the source conditional means) for all k. Then a.s.

Etgt (LLMn ) −→ Emintgt + G ≤ Emintgt + ρ2 . Proof. Decompose the target error: Etgt (LLMn ) = Emintgt +

X

= Emintgt +

X

p′k (µ′k − LLMn (k))2

k

 2 p′k (µ′k − µk ) + (µk − LLMn (k)) .

k a.s.

a.s.

For each k, LLMn (k) −→ µk , so (µk − LLMn (k))2 −→ 0. By the cross-term expansion and the continuous mapping theorem applied to the quadratic function z 7→ p′k ((µ′k − µk ) + (µk − z))2 , each term almost surely to p′k (µ′k − µk )2 . Summing over the finite collection of terms P converges ′ ′ yields k pk (µk − µk )2 = G almost surely. The inequality follows from Lemma 5.2. a.s.

Corollary 5.4 (No-Regret Transfer). If µ′k = µk for all k, then G = ρ = 0 and Etgt (LLMn ) −→ Emintgt , recovering Theorem 3.10. If the conditional means match, the LLM transfers with zero excess error due to allocation or variance differences. Remark 5.5. The transfer cost is additive and proportional to the squared difference in conditional means. For small distribution shifts (ρ ≪ 1), the degradation ρ2 is quadratically small. For example, if the maximum mean discrepancy across conditions is ρ = 0.05 (on a normalized response scale), the worst-case excess error is 0.0025, which is negligible relative to typical irreducible errors (often 0.01–0.10). This suggests that LLMs pretrained on broad populations may transfer effectively to narrower target populations, provided the conditional means are approximately preserved. 18

5.3

Triangle Inequality for Transfer

A practical consequence of the additive structure is a triangle inequality for sequential transfer. If an LLM is trained on source P0 , fine-tuned on intermediate P1 , and deployed on target P2 , the total transfer gap is bounded by: E2 (LLM) ≤ Emin2 + ρ201 + ρ212 + 2ρ01 ρ12 , (i)

(j)

where ρij = maxk |µk − µk |. Fine-tuning on domain-specific data reduces the effective gap, and the additive structure allows practitioners to decompose the total distribution shift into manageable stages.

5.4

Representation Bias and the Asymptotic Risk Floor

The unified LLM model (Definition 2.2) separates estimation error into representation bias ϵrep (k) and optimization error ϵopt (k, n). While optimization error vanishes asymptotically, representation bias sets a permanent floor on achievable risk. Theorem 5.6 (Bias Domination (Risk Convergence under Misspecification)). Let ϵrep (k) = T (PF∗ (· | k)) − µk be the asymptotic representation bias (defined via Assumption A8). Under Assumptions A1–A10, as n → ∞, lim E(LLMn ) = Emin +

n→∞

K X

pk ϵrep (k)2

almost surely,

(27)

k=1

and the convergence also holds in L1 : K X   pk ϵrep (k)2 . lim E E(LLMn ) = Emin +

n→∞

k=1

P Proof. From the error decomposition (Theorem 2.8), E(LLMn ) = Emin + k pk (µk − LLMn (k))2 . By the functional convergence result (Layer 2, Section 2.4) and A8 (functional delta stability), p p By the continuous mapping theorem, (µk −LLMn (k))2 → − LLMn (k) → − T (PF∗ (· | k)) = µk +ϵrep (k).P ϵrep (k)2 , and the finite sum converges to k pk ϵrep (k)2 . The magnitude of ϵrep (k) is bounded by Theorem 5.10. Lemma 5.7 (Risk Decomposition Stability). Under A7 (bounded responses, |Y | ≤ B a.s.) and A8 p (mean-regular model class), if LLMn (k) → − µk + ϵrep (k) for each k, then for each k with pk > 0:   E (LLMn (k) − µk )2 → ϵrep (k)2 as n → ∞. Proof. From A7, responses are bounded by B, so |LLMn (k)| ≤ B almost surely (the sample mean of bounded variables is bounded) and |µk | ≤ B. Hence |LLMn (k) − µk | ≤ 2B almost p surely. Define Xn = (LLMn (k) − µk )2 . By the continuous mapping theorem, Xn → − ϵrep (k)2 . Since |Xn | ≤ (2B)2 uniformly, the sequence {Xn } is uniformly integrable. Therefore E[Xn ] → E[ϵrep (k)2 ] = ϵrep (k)2 . Applying this per-condition and summing with weights pk yields the L1 convergence in Theorem 5.6. 19

Remark 5.8 (Structural constraints on representation bias). The representation bias ϵrep (k) = T (PF∗ (· | k)) − µk is (i) well-defined: Assumption A8 defines PF∗ as the unique KL-projection of the true conditional distribution onto the model class F, ensuring ϵrep (k) ispa deterministic, non-random quantity; (ii) bounded: Theorem 5.10 constrains |ϵrep (k)| ≤ 2B 2DKL (P ∥ F) via Pinsker’s inequality; (iii) calibratable: paired LLM-human data from P the calibration protocol (Section 7) −1 b b provides consistent estimators bk of ϵrep (k) via bk = mk i (LLM(ki ) − Y (ki )). Corollary 5.9 (Identifiability-Modified Risk). Under Definition 2.5 with misclassification rate δ, the asymptotic risk limit becomes: K X  
2 lim E E(LLMn ) = Emin + pk ϵrep (k) + δ · ∆k ,

n→∞

k=1

P

where ∆k = µk − j̸=k wkj µj is the misclassification contrast (Eq. 6). For δ = 0, this recovers Theorem 5.6; for δ > 0, the effective representation bias is modified by up to δ ·(maxj µj −minj µj ) in magnitude. In the joint asymptotic regime, δ = O(n−1/2 ), so the δ-induced modification vanishes at rate O(n−1/2 ). Theorem 5.10 (Representation Bias Bound). Under Assumptions A1–A10, the representation bias satisfies q (28) |ϵrep (k)| ≤ 2B · 2 · inf DKL (P (· | k) ∥ Q), Q∈F

where DKL is the Kullback-Leibler divergence and B is the response bound from A7. Proof. Let Q∗ = PF∗ (· | k) be the KL-projection. By Pinsker’s inequality and the definition of Hellinger distance, H 2 (P, Q∗ ) ≤ DKL p (P ∥Q∗ ). By the Lipschitz property (A8), |ϵrep (k)| = |T (P ) − T (Q∗ )| ≤ 2B · H(P, Q∗ ) ≤ 2B · 2DKL (P ∥Q∗ ). Since Q∗ achieves the infimum of KL divergence over F, the bound follows. Corollary 5.11 (Bias-Variance Tradeoff). The excess risk decomposes as X X pk ϵrep (k)2 + E(LLMn ) − Emin = pk (LLMn (k) − T (PF∗ ))2 . |k

{z

}

asymptotic bias floor

|k

{z

finite-sample variance

}

For large n, the variance term vanishes at rate Op (n−1 ) (in the Gaussian case) or slower (under misspecification), while the bias term persists. The total excess risk is bias-dominated for sufficiently large n. Corollary 5.12 (Misspecification Gap). If the model class has finite KL-diameter DF = sup inf DKL (P ∥Q), P ∈P Q∈F

√ then maxk |ϵrep (k)| ≤ 2B 2DF . For a well-specified model (P ∈ F), DF = 0 and ϵrep (k) = 0. Remark 5.13 (Practical implication). Theorem 5.6 implies that for a misspecified LLM (ϵrep (k) ̸= 0), the asymptotic risk strictly exceeds the Bayes risk Emin . The calibration protocol (Section 7) estimates this bias floor from paired LLM-human data and aborts if the bias exceeds the effect size of interest (feasibility check, Step 5). 20

5.5

Non-Equivalence Regimes (n)

Theorem 5.14 (Non-Equivalence Conditions). The restricted Le Cam deficiency ∆L (Ehuman , ELLM ) is bounded below by a positive constant if any of the following hold: (i) Distribution shift: The training and target populations differ by more than ε in maximum mean discrepancy (MMD), i.e., MMD(Ptrain , Ptarget ) > ε; (ii) Non-identifiability: The effective bias inflation δ · ∆max exceeds the effect size of interest ∆, i.e., δ · (maxk µk − mink µk ) ≥ ∆, making the LLM estimator unable to distinguish conditions; (iii) Heavy tails: Any conditional distribution P (Y | X = k) has infinite variance (vk = ∞), violating the finite-second-moment assumption required by the Strong Law. (n)

In each case, the deficiency satisfies ∆L (Ehuman , ELLM ) ≥ c > 0 for all n, where c depends on the specific failure parameter. Proof sketch. (i) By the cross-domain transfer bound (Theorem 5.3), distribution shift induces a positive mean-squared gap G > 0, which propagates to the deficiency. (ii) Non-identifiability P introduces misclassification of conditions, effectively replacing µk with a mixture j wkj µj , producing non-zero excess error. (iii) Without finite variance, the SLLN does not apply and the LLM estimator does not converge, so the forward deficiency does not vanish.

6

Functional Risk Equivalence under Squared Loss

Formal object mapping. Let D denote the data-generating process and Ahuman : D → µ̂human the human-data estimator. Let ALLM : Dn → µ̂LLM denote the LLM-induced estimator after training on n samples. Functional risk equivalence holds when E[(µ̂LLM − µ)2 ] ≈ E[(µ̂human − µ)2 ] as n → ∞. This section formalizes this convergence in the language of decision-theoretic deficiency. The preceding sections establish that, under squared loss, (i) the conditional mean E[Y | X = k] is the unique risk-minimizing predictor (Section 3), and (ii) the LLM estimator LLMn (k) converges to T (PF∗ (· | k)), which equals µk up to representation bias (Section 2.4). This section states the precise decision-theoretic consequence: convergence of the conditional-mean functional yields equality of optimal risks for the restricted squared-loss decision class, not equality of response distributions. Decision class restriction. Throughout this section, the loss class is restricted to L = {ℓ(y, a) = (y − a)2 }, the squared error loss. Under this loss, the risk of any decision rule f : [K] → R depends on the data distribution only through the conditional mean E[Y | X = k]. No claim is made about loss functions that depend on higher moments, quantiles, or distributional shape. The Hellinger convergence results of Section 2.4 establish that the LLM learns the full conditional distribution (not just the mean), which justifies the use of variance-dependent quantities in b however, the equivalence claims in this section require only conditional-mean calibration (e.g., λ); sufficiency. Formal definition of L2 -risk-equivalence. For decision class Fdec = {f : [K] → R} and loss class L2 = {(y − a)2 }, two information objects or estimators E and F are L2 -risk-equivalent, denoted E ∼L2 F , iff inf RE (f ) = inf RF (f ), f ∈Fdec

f ∈Fdec

21

which is the equivalence relation defined in Eq. 11. The main result is that the human-data estimator and LLM-induced estimator satisfy this relation asymptotically in the well-specified/growing-class regimes, and differ by the explicit representation-bias floor in the fixed misspecified regime.

6.1

Statistical experiments and conditional-mean sufficiency

A statistical experiment 6 is a family of probability distributions {Pθ : θ ∈ Θ} indexed by a parameter of interest. The human experiment Ehuman consists of drawing i.i.d. responses Yi | Xi = k from P (· | k). The LLM object in this paper is not a second population-level experiment; it is the (n) estimator-induced reduced observation ELLM = {LLMn (k)}K k=1 , equivalently the family of learned K conditional distributions {P̂n (· | k)}k=1 evaluated only through the conditional-mean functional. An observation is functionally sufficient for squared loss when it achieves the same optimal risk as the full data for every decision problem whose loss depends on the distribution only through the conditional mean. This is the restricted risk-functional version of Blackwell sufficiency. Theorem 6.1 (Functional Sufficiency for Squared Loss). For any decision rule f : [K] → R and (n) squared loss L(f (k), y) = (f (k) − y)2 , the estimator-induced reduced observation ELLM satisfies (n)

RLLM (f ) ≤ Rhuman (f ) + op (1),

(29)

P (n) where RLLM (f ) =P k pk (µk − f (k))2 is the limiting risk functional induced by the LLM estimator and Rhuman (f ) = k pk E[(Y − f (k))2 | X = k]. Proof. Under squared loss, the optimal decision rule depends only on the conditional mean E[Y | X = k]. The conditional expectation is the unique admissible action for squared loss 8 ; any randomized decision rule can be replaced by its conditional expectation without increasing risk. The LLM estimator provides an asymptotically exact estimate of this sufficient functional in the well-specified (n) regime. Therefore, for any decision rule f , the risk-functional gap satisfies RLLM (f )−Rhuman (f ) = E(LLMn ) − Emin → 0. Lemma 6.2 (Restricted Sufficiency Reduction). Let E = {Pθ : θ ∈ Θ} be a statistical experiment and let T be a sufficient statistic for θ under loss class L. If experiment F is the reduction of E by T (i.e., F observes only T rather than the full data), then ∆(E, F ) = 0 for all decision problems with loss in L. where ∆L denotes the deficiency restricted to the loss class L. Proof. By the definition of sufficiency, the conditional distribution of the data given T does not depend on θ. Therefore, any randomized decision rule δ based on the full data can be replaced by a randomized decision rule δ ′ based on T alone, with identical risk: δ ′ (t) = E[δ(data) | T = t]. By Jensen’s inequality (or Rao–Blackwell for convex losses), R(θ, δ ′ ) ≤ R(θ, δ). Hence the class of achievable risk functions under F contains the class achievable under E after the reduction, meaning ∆(E, F ) = 0.

6.2

Restricted Functional Risk Equivalence

Blackwell sufficiency is an asymptotic statement. To quantify the finite-sample gap, we use the Le Cam deficiency distance 7 . The deficiency ∆(E, F ) between two experiments measures the maximum additional risk incurred by using F instead of E, normalized by the loss function class. 22

Theorem 6.3 (Reverse Deficiency). For squared-loss decision problems, the reverse Le Cam deficiency is exactly zero: (n) ∆(Ehuman , ELLM ) = 0 for all n. (30) Proof. For squared loss L(θ, a) = (θ − a)2 , the conditional expectation T (Y ) = E[Y | X = k] = µk is a sufficient functional (Theorem 3.1 and Corollary 3.2: the risk depends on θ only through µk , and any deviation from µk strictly increases risk). The estimator-induced reduced (n) observation ELLM is the conditional-mean reduction of Ehuman : it observes an estimate of µk rather than individual responses Yi . By Lemma 6.2, a sufficient reduction incurs zero deficiency for the (n) corresponding loss class. Therefore ∆(Ehuman , ELLM ) = 0 for all n. This is not an asymptotic statement—it holds at every finite n because sufficiency is an exact property of the conditional-mean functional under squared loss. The LLM estimator’s error LLMn (k) − µk affects the forward deficiency (how much the estimator loses relative to the ideal conditional-mean reduction) but not the reverse deficiency. Theorem 6.4 (Restricted Functional Risk Equivalence). For squared-loss decision problems, the restricted Le Cam distance between the human experiment Ehuman and the estimator-induced re(n) duced observation ELLM satisfies: v uK uX p (n) pk (µk − LLMn (k))2 + δ · ∆max = Excess(LLMn ) + δ · ∆max , ∆(ELLM , Ehuman ) ≤ t k=1

(31) (n)

∆(Ehuman , ELLM ) = 0

for all n.

(32)

The forward deficiency (31) is driven by estimation error and identifiability error. The reverse deficiency (32) is driven by functional sufficiency: the full human data contain no additional squareda.s. loss-relevant information beyond the conditional mean µk . By Theorem 3.10, Excess(LLMn ) −→ 0 in the well-specified regime, and by Theorem 5.6 it converges to the representation-bias floor under misspecification. Proof. For the forward direction, the deficiency is controlled by the discrepancy between the conditional mean induced by the LLMP estimator and the true conditional mean. The squared-loss risk 2 gap is exactly Excess(LLMn ) = k pk (µk − LLMn (k)) , and stochastic misidentification contributes the additive δ∆max term from Eq. 6. The reverse direction is Theorem 6.3: for squared loss, the conditional mean is a sufficient functional, so full human responses provide no additional decision-relevant information once the conditional mean is known. Theorem 6.5 (Representation Theorem for Functional Sufficiency). Any sequence model trained via a proper scoring rule on i.i.d. population-representative data induces an estimator LLMn (k) = T (P̂n (· | k)) of the conditional-mean functional. For squared-loss decision problems this estimator is a reduced observation of the human data through the sufficient functional T (P ) = µk . Consequently: (n)

(i) ∆(Ehuman , ELLM ) = 0 for all n (reverse deficiency, Lemma 6.2);

23

(n)

(ii) ∆(ELLM , Ehuman ) is bounded by the squared-loss excess risk and identifiability term (forward deficiency, Theorem 6.4); (iii) the induced risk functional is asymptotically equal to the human-data risk functional in the well-specified regime and differs by the explicit representation-bias floor under misspecification. Hence the result is functional risk equivalence for a restricted loss class, not equivalence of response distributions or equivalence of experimental systems. Remark 6.6 (Scope of equivalence). The equivalence established here is restricted to decision problems whose loss function depends on θ only through the conditional mean E[Y | X = k]. For loss functions that depend on higher moments (e.g., variance estimation, quantile regression) or on the full conditional distribution, the experiments are not equivalent and the deficiency may be strictly positive.

6.3

Practical interpretation

The restricted functional risk equivalence theorems provide the rigorous answer to the question posed in Section 2.4: under squared loss, the LLM estimator and the human experiment achieve asymptotically equal optimal risk for any decision problem whose loss depends on the data only through the conditional mean. The forward direction (Theorem 6.4) shows that the LLM estimator incurs risk no larger than the human experiment, asymptotically. The reverse direction (Theorem 6.3) shows that the human experiment incurs no LESS risk than the LLM estimator. Together, they establish restricted functional risk equivalence 7 : the two procedures are indistinguishable in terms of achievable risk for squared-loss inference, up to the finite-sample gap quantified by Eqs. (31)–(32). This does not mean that individual LLM responses are indistinguishable from individual human responses; rather, it means that any statistical inference that depends only on the conditional mean (or any continuous functional thereof, by the Hellinger convergence of Section 2.4) can be performed using LLM-generated estimates with asymptotically equivalent decision-theoretic risk for squared loss. The forward deficiency bound (31) quantifies the finite-sample gap: the LLM exper...

7

Calibration Protocol and Validation

The theoretical results establish asymptotic validity, but practical deployment requires a calibration procedure that accounts for finite training data, potential model misspecification, and residual bias. We present a comprehensive calibration protocol with explicit decision rules.

7.1

Calibration Data Collection

Step 1: Paired sample collection. Collect Mc ≥ 500 paired observations: for each prompt i = 1, . . . , Mc , obtain both the LLM response LLM(ki ) (using the exact prompt and condition ki ) and a human response Y (ki ) from a member of the target population. The prompts should span all K 24

conditions proportionally to the target allocation {pk }, with a minimum of 50 paired observations per condition to ensure reliable per-condition estimates. Step 1b: Identifiability validation. Estimate the misclassification rate δb by having Mv ≥ 100 human annotators independently label a held-out set of prompts with their intended conditions. Compute Mv 1 X b δ =1− 1{ϕ(si ) = ki }. Mv i=1 If δb > 0.05, the prompt-to-condition mapping is unreliable and the LLM is unsuitable for this application. Step 2: Bias estimation. Compute the empirical bias for each condition: X bbk = 1 (LLM(ki ) − Y (ki )), mk i:k =k i

P where mk = |{i : ki = k}|. The overall bias is bb = k pkbbk . Step 3: Variance ratio estimation. Compute the variance ratio: d b = Var(LLM) , λ d ) Var(Y d d ) are the sample variances of the LLM and human responses, respecwhere Var(LLM) and Var(Y tively, pooled across conditions after subtracting condition means to remove between-condition ˜ i = LLM(ki ) − µ variance. Specifically, let LLM bLLM (ki ) and Ỹi = Y (ki ) − µ bY (ki ) be the residuals, and compute: d Var(LLM) =

Mc X 1 ˜ 2, LLM i Mc − K i=1

d )= Var(Y

Mc X 1 Ỹ 2 . Mc − K i=1 i

b follows from the The consistency of both the bias estimator bb and the variance ratio estimator λ Strong Law of Large Numbers. For the bias estimator, the paired differences Di = LLM(ki ) − a.s. Y (ki ) are i.i.d. with finite variance under the calibration design; the SLLN guarantees bbk −→ µLLM (k) − µY (k). For the variance ratio, the sample variances of centered responses converge almost surely to the true conditional variances, and the ratio of almost-surely convergent sequences converges almost surely to the ratio of their limits (provided the denominator limit is nonzero). Both consistency results have been formally verified (see Supplementary Information). Step 3b: Uncertainty quantification. Compute a (1 − α) confidence interval for the bias in each condition: bbk ± zα/2 · σ bb,k , where σ bb,k is the standard error of the paired differences. The sample size adjustment (Step 4) should use the worst-case bias magnitude within the confidence interval: bbmax = max{|bbk − zα/2 σ bb,k |, |bbk + zα/2 σ bb,k |}. k This ensures the adjusted sample size NLLM is robust to estimation uncertainty in the bias. 25

7.2

Sample Size Adjustment

Step 4: Adjusted sample size. For a target effect size ∆ (the minimum effect of scientific interest) and a baseline human-study sample size Nh (per condition, determined by standard power analysis 1 ), the required LLM sample size per condition is: NLLM =

b Nh · λ , b max )/∆)2 (1 − (|bb| + δb · ∆

(33)

b max = maxj µ where ∆ bj − minj µ bj is the estimated condition range. The derivation of (33) follows from equating the noncentrality parameters of the human and b and the bias-induced effective LLM-based z-tests, accounting for the variance inflation factor λ b effect size reduction ∆ − |b|. Step 5: Feasibility check. If either of the following conditions holds, the LLM is unsuitable for the target application and the protocol aborts: • |bb/∆| ≥ 1: the estimated bias exceeds the effect size of interest, meaning the LLM cannot reliably detect the effect even with infinite sample size. b > 10: the LLM’s responses are more than an order of magnitude more variable than human • λ responses, making the sample size penalty prohibitive. • δb > 0.05: the prompt-to-condition mapping is too unreliable (already checked in Step 1b); the b max exceeds the calibration margin. effective bias inflation 2δb · ∆ If the protocol aborts, the practitioner should consider fine-tuning the LLM on domain-specific data, revising the prompts, or abandoning LLM-based estimation for this application.

7.3

Equivalence Validation

Step 6: Two One-Sided Test (TOST). After collecting NLLM LLM responses per condition and Nh human responses per condition, validate equivalence of the estimated effect sizes. Let δbLLM and δbhuman be the estimated effect sizes from the LLM and human samples, respectively. Test the composite null hypothesis: H0 : |δLLM − δhuman | ≥ ε

vs.

H1 : |δLLM − δhuman | < ε,

where ε is the equivalence margin (typically ε = 0.01 for bounded responses in [0, 1], or 0.05 for standardized effect sizes). The TOST procedure rejects H0 at level α if both one-sided tests reject: (1)

Test 1: H0 : δLLM − δhuman ≤ −ε (2)

Test 2: H0 : δLLM − δhuman ≥ ε

δbLLM − δbhuman + ε > zα , σ bD δbLLM − δbhuman − ε rejected if < −zα , σ bD rejected if

26

where σ bD is the estimated standard error of the difference in effect sizes, computed via WelchSatterthwaite approximation: 2 = σ bD

d LLM (i) Var d LLM (j) Var d human (i) Var d human (j) Var + + + . NLLM NLLM Nh Nh

If TOST rejects H0 , we conclude that the LLM-based effect size estimate is statistically equivalent to the human-based estimate at the ε margin, validating the LLM as a measurement instrument for this application. The TOST procedure controls the type I error rate at level α by a general argument: for any two events A and B with P(A) ≤ α and P(B) ≤ α, monotonicity of probability implies P(A ∩ B) ≤ P(A) ≤ α. Since TOST rejects H0 exactly when both one-sided tests reject, and each one-sided test has size α, the family-wise error rate is bounded by α. This argument does not require Gaussian distributional assumptions.

7.4

Worked Examples

Example 7.1 (Political Attitude Survey). A researcher studies the effect of framing on political attitudes with K = 2 conditions (pro-immigration vs. neutral frame). Historical human data show µA = 0.72, µB = 0.78 (proportion agreeing on a 0–1 scale), ∆ = 0.06, and v ≈ 0.04 (variance of binary-like responses). A baseline human study requires Nh = 100 per condition for 80% power at α = 0.05 1 . b = 1.15 (LLM responses Calibration with Mc = 500 paired responses yields bb = 0.002 and λ are slightly more variable than human responses, likely due to temperature sampling). The adjusted LLM sample size is: NLLM =

115 115 100 × 1.15 = = ≈ 123. (1 − |0.002/0.06|)2 (1 − 0.0333)2 0.9344

The 23% increase in sample size is modest. The feasibility check passes (|bb/∆| = 0.033 < 1, b = 1.15 < 10). After collecting NLLM = 123 LLM responses per condition and the original 100 λ human responses per condition, TOST with ε = 0.02 yields p < 0.01, confirming equivalence. Example 7.2 (Consumer Preference A/B Test). A marketing researcher tests two product descriptions (K = 2) using a 7-point Likert purchase-intent scale. Historical data show µA = 4.2, µB = 4.8 (∆ = 0.6 points), v ≈ 2.5. Baseline Nh = 85 per condition for 80% power. Calibration with Mc = 500 yields bb = 0.12 (the LLM overestimates purchase intent by 0.12 b = 0.85 (LLM responses are less variable, as the model produces more points on average) and λ consistent ratings than individual humans). The adjusted sample size is: NLLM =

72.25 72.25 85 × 0.85 = = ≈ 113. 2 2 (1 − |0.12/0.6|) (1 − 0.2) 0.64

b < 1 provides a sample Despite the bias, the large effect size keeps the penalty modest. Notably, λ size reduction relative to the variance component, partially offsetting the bias penalty. TOST with ε = 0.2 passes at p < 0.01.

27

Example 7.3 (Multi-Condition Educational Intervention). An education researcher tests K = 4 teaching methods using standardized test scores (µ = 0, σ = 1 by normalization). The minimum effect size of interest is ∆ = 0.3 (Cohen’s d). Baseline Nh = 175 per condition for 80% power with Bonferroni correction for 6 pairwise comparisons. b = 1.25. The adjusted sample size is: Calibration yields bb = 0.03, λ NLLM =

175 × 1.25 218.75 ≈ 270. = (1 − |0.03/0.3|)2 0.81

The 54% increase reflects both the variance inflation and bias. With K = 4 conditions, total LLM responses are 1, 080—still far cheaper than recruiting 700 human participants. TOST with ε = 0.1 passes for all 6 pairwise comparisons.

7.5

Diagnostic Checks

Beyond the TOST validation, we recommend three diagnostic checks: 1. Condition-wise calibration. Plot bbk against k to detect condition-specific biases. A systematic pattern (e.g., monotonically increasing bias with condition index) suggests prompt artifacts or training data imbalances that require investigation before proceeding. 2. Residual analysis. Examine the residuals LLM(ki ) − Y (ki ) for heteroscedasticity (does the LLM’s accuracy vary with the response magnitude?), outliers (are there specific prompts where the LLM fails catastrophically?), and distributional shape (are residuals approximately symmetric, or is there systematic skew?). 3. Sensitivity to prompt variation. Re-run the calibration with semantically equivalent but syntactically varied prompts (e.g., reworded instructions, shuffled response options). Large variab across prompt variants indicates sensitivity to prompt engineering and suggests tion in bb or λ that the LLM is not robustly capturing the intended construct.

8

Discussion

8.1

Summary of Results

We have established a complete mathematical framework for LLM–human functional risk equivalence under squared loss, anchored by seven theorems: 1. Error decomposition (Theorem 2.8): The expected squared error of any predictor separates cleanly into irreducible population variance and excess squared bias. This decomposition is exact, non-asymptotic, and requires only finite second moments. 2. Irreducible error minimality (Theorem 3.1): The population conditional mean is the unique minimizer of expected squared error. The Bayes risk Emin is a universal lower bound that no predictor can surpass.

28

3. Effect size consistency (Theorem 3.5): The LLM-based effect size estimator converges almost surely to the true effect size. This holds under the minimal condition of finite variance, using Etemadi’s SLLN which requires only pairwise independence. 4. Power amplification (Theorem 3.7): Statistical power is monotone in sample size, and LLMbased studies can achieve arbitrarily high power by increasing the number of model-generated responses at near-zero marginal cost. 5. Cross-domain transfer (Theorem 5.3): When the training population differs from the target, the excess error is bounded by ρ2 , the squared maximum conditional mean discrepancy. The degradation is quadratically small for small distribution shifts. 6. Functional sufficiency (Theorem 6.1): The estimator-induced reduced observation achieves asymptotically no greater squared-loss risk than the full human-data procedure for conditionalmean-dependent decisions. 7. Restricted Functional Risk Equivalence (Theorems 6.3 and 6.4): The forward restricted de(n) ficiency ∆L2 (ELLM , Ehuman ) is controlled by squared-loss excess risk and identifiability error, (n) and the reverse restricted deficiency ∆L2 (Ehuman , ELLM ) = 0 for all n. Hence the human data and LLM estimator induce asymptotically equal optimal risks for conditional-mean-dependent squared-loss inference. The finite-sample analysis (Section 4) provides concentration bounds and sample size formulae that make the asymptotic guarantees operational. The calibration protocol (Section 7) bridges theory and practice with concrete decision rules for feasibility assessment and equivalence validation.

8.2

Scope and Boundaries

The framework applies to a specific, well-defined class of studies. Functional risk equivalence holds when all of the following hold: C1. Quantitative continuous response. The dependent variable is measured on an interval or ratio scale (proportions, Likert scales treated as continuous, reaction times, test scores, monetary amounts). The framework does not apply to purely categorical or open-ended text responses. C2. Discrete, finite conditions. The experimental design has a finite number of conditions with known allocations. Continuous treatments require discretization. C3. Representative training data. The LLM’s training data are approximately i.i.d. from the target population. This is the most stringent requirement and the one most likely to be violated in practice. Web-scraped training corpora overrepresent certain demographics and underrepresent others. C4. Within-distribution effects. The target experimental manipulation produces effects that lie within the support of the training distribution. The LLM cannot extrapolate to conditions or effect magnitudes it has not encountered in training.

29

b < 10, and TOST C5. Calibration validation. The calibration protocol confirms |bb/∆| < 1, λ equivalence at the chosen margin. We explicitly exclude the following applications, where functional risk equivalence does not hold or the application is ethically inappropriate: • Qualitative research (interviews, ethnography, open-ended surveys): The framework requires quantitative responses with a well-defined conditional mean. Qualitative data lack this structure. • Novel paradigms with no training-data analog: If the experimental manipulation is genuinely novel (e.g., a never-before-studied psychological intervention), the LLM’s training data contain no information about its effects, and the empirical mean model has no basis for prediction. • Safety-critical applications: Asymptotic guarantees are insufficient for clinical trials, safety assessments, or policy decisions where individual lives are at stake. The framework provides statistical guarantees, not clinical or ethical ones. • Behavioural mechanism research: When the research question itself concerns why humans behave as they do (cognitive processes, neural mechanisms, developmental trajectories), the LLM estimator is not a substitute for the human behavioural system, because it models statistical regularities in behavioural outputs, not the underlying mechanisms that generate them. The LLM can tell us what the average response would be, but not why.

8.3

Limitations and Open Problems

Several important limitations merit attention. Training data representativeness. The assumption that LLM training data are i.i.d. from the target population is an idealization. Real training corpora are convenience samples with complex selection biases 1 . Future work should develop formal bounds for the degradation when the training distribution differs from the target in known ways, extending the cross-domain analysis of Section 5 to settings where the distribution shift is estimated rather than assumed known. Finite-condition assumption. The framework requires discrete conditions. Extending to continuous treatments (e.g., dose-response curves with continuous dosage) requires nonparametric regression theory and is a natural direction for generalization. Temporal stability. Human populations and their behavioural patterns drift over time. An LLM trained on a snapshot of data may become miscalibrated as the population evolves. Adaptive recalibration procedures, where the LLM is periodically re-benchmarked against fresh human samples, would address this concern. Construct validity. Agreement of population-level effect sizes does not guarantee construct validity — the LLM may produce the right numbers for the wrong reasons. For example, an LLM might learn to reproduce survey response patterns without modelling the underlying attitudes, producing valid population-level estimates while failing on individual-level prediction tasks. The framework guarantees squared-loss risk validity for conditional means, not psychological validity. Response distributions beyond the mean. The framework focuses on the conditional mean, which is the natural target for squared error loss and the quantity of interest in most betweensubjects designs. Extending to other functionals (quantiles, variances, distributional shape) requires different loss functions and estimator classes. 30

8.4

Relationship to Existing Work

The use of language models as experimental subjects has been explored empirically across psychology, political science, and marketing. These studies typically compare LLM-generated effect sizes to human baselines on a case-by-case basis, reporting correlations and mean differences. While informative, these comparisons do not provide formal guarantees, and a favourable result in one study does not transfer to another without additional validation. Our contribution is to replace this empirical patchwork with a principled mathematical foundation. By reducing the problem to conditional mean estimation under squared loss, we connect LLM–human risk equivalence to classical statistical theory (MLE consistency, the Strong Law, concentration inequalities) and provide the first formal proof that the approach is asymptotically valid under minimal assumptions. The calibration protocol we present is related to measurement invariance testing in psychometrics and to domain adaptation in machine learning. The key difference is that we provide decision rules (the feasibility check and TOST) that directly determine whether an LLM estimator and human-data estimator achieve the same risk in a specific application, rather than requiring the practitioner to interpret continuous fit indices.

9

Using LLM Estimators in Human-Subject Research

The theory supports a practical design pattern: use human participants for calibration and validation, then use calibrated LLM estimators for high-volume conditional-mean estimation under squared loss. It does not remove the need for human data or validate individual-level behavioural simulation. Algorithm 1 is a deployment rule, not a claim of behavioral simulation. It turns the risk theory into a gate: LLM estimates are used only after calibration, feasibility, and equivalence validation pass.

9.1

Statistical procedures covered by the framework

Table 1 lists procedures where calibrated LLM estimators can replace or augment conditional-mean estimation after feasibility and equivalence validation. Table 1: Traditional procedures and calibrated LLM-estimator analogues under the restricted squared-loss framework. Traditional method Two-sample test

Target estimand t/z µ1 − µ0

LLM replacement target

Required validation

Not covered

LLM estimates both condition paired calibration + TOST on individual-level response means and the contrast effect size distributions.

One-way ANOVA finite set of group LLM estimates the vector condition-wise calibration + all post-hoc claims not premeans and contrasts (µ1 , . . . , µK ) planned contrasts specified. Linear regression / coefficient that is LLM estimates conditional calibration stratified by cells or causal identification ATE with discrete a linear contrast of means for covariate/treatment model-assisted residual check without ignorability. treatments conditional means cells Survey proportion conditional expecta- LLM estimates the bounded re- bias/variance calibration and qualitative/free-text / Likert mean tion of bounded re- sponse mean feasibility rule interpretation. sponse

31

Traditional method

Target estimand

LLM replacement target

Required validation

Not covered

Equivalence or margin-based mean LLM supplies high-precision TOST with pre-specified safety-critical claims non-inferiority test contrast estimator after calibration equivalence margin without independent human validation.

9.2

Public UI and human-experience use cases

Three public research settings illustrate where the framework can be used without changing its scope. First, mobile UI design critique can be treated as conditional-mean estimation when the target is a regional design-quality rating or average severity score. UICrit contains human-generated design critiques, bounding boxes, and design quality ratings for 1,000 mobile UIs from RICO; each screen was evaluated by three annotators, and the public CSV includes aesthetics, learnability, efficiency, usability, and overall design quality ratings, with 11,344 design critiques in the public release 9 . In this setting, the LLM estimator targets mean ratings or mean region-level critique scores, not the full distribution of expert judgment. Second, human-experience usability studies often report System Usability Scale (SUS) scores. SUS is a standardized usability score, and a score near 68 is commonly used as an average or mid-point benchmark for interpretation 11 . The present framework applies when the target is the conditional mean SUS score or a contrast between design variants; it does not validate open-ended user narratives. Third, behavioral-response studies such as framing surveys remain covered when their estimand is a bounded response mean or a contrast of means. In all three settings, LLM augmentation is valid only when the calibration, feasibility, and TOST workflow in Algorithm 1 passes. The public UICrit data source provides the UI-design anchor for the report below. If the public CSV is unavailable during replication, the README-grounded metadata still records 1,000 unique UI screens and 11,344 design critiques. Table 2 summarizes the sample-size and cost calculations for the three applied flows below. Table 2: Sample-size and cost report for calibrated LLM-estimator use cases. Use case Framing survey UI design rating (UICritstyle) SUS human-experience score

9.3

Human-only cost LLM-augmented cost Reduction $16,000.00 $1,713.60

$4,006.49 $430.81

$1,142.40

$230.08

74.96% 74.86%

Validation target TOST on effect size rating calibration

79.86% TOST on SUS margin

End-to-end flows for the three use cases

Flow 1: UI design rating and region-level critique. Use this flow when the research object is a UI screen, design variant, or screen region and the estimand is a mean expert rating or a contrast of mean ratings. The researcher first defines the rating rubric, such as aesthetics, learnability, efficiency, usability, or overall design quality as in UICrit 9 . Next, collect a stratified calibration 32

Algorithm 1 Calibrated LLM estimator deployment for conditional-mean inference Require: Quantitative outcome Y , finite conditions [K], target effect ∆, equivalence margin ε 1: Define estimand g(µ1 , . . . , µK ) as a conditional mean, contrast, ANOVA contrast, or linear coefficient. 2: Collect Mc ≥ 500 paired human–LLM calibration observations and Mv ≥ 100 condition-label validation prompts. b δ, b and ∆ b max . 3: Estimate b b, λ, b max )/∆. 4: Compute ρbias ← (|b b| + δb∆ b > 10 then 5: if ρbias ≥ 1 or λ 6: Abort LLM augmentation; use human data or collect new calibration data. 7: else 8: Compute NLLM using Eq. 33. 9: Generate NLLM calibrated LLM responses per condition. 10: Estimate g(µ1 , . . . , µK ) from the LLM responses. 11: Run TOST against the human baseline or calibration holdout. 12: if TOST rejects non-equivalence at margin ε then b δ, b NLLM . 13: Report calibrated LLM estimator result with Mc , Mv , bb, λ, 14: else 15: Reject the LLM estimate for this study; report failure of validation. 16: end if 17: end if set of human ratings and critiques across screen types and regions; in the cost model summarized in Table 2, this is Mc = 150 human-rated screens or screen regions. Then prompt the LLM with b and δb by comparing LLM ratings the same screenshot, region, task, and rubric, and estimate bb, λ, with human ratings. If the feasibility rule passes, compute NLLM = 603 LLM-rated screens per condition for the illustrative two-condition UI comparison. The LLM is then used to rate the larger screen set; region-level comments are summarized only as support for the numeric rating and are not treated as validated qualitative findings. Finally, a human holdout or fresh expert panel runs TOST on the planned rating contrast. If TOST fails, the LLM-generated UI rating is rejected for that design decision. Flow 2: SUS-style human-experience score. Use this flow when the target is a standardized human-experience score such as SUS, not open-ended experience narratives. The researcher first defines the task scenario and the design variants, then collects human SUS responses for calibration; the illustrative report uses Mc = 80 calibration participants and the common SUS benchmark of roughly 68 as contextual background 11 . The LLM receives the same task description, interface description, and SUS item wording, and outputs item-level or total SUS estimates. Calibration estimates the mean bias, variance ratio, and prompt-condition error against human SUS data. If the feasibility rule passes, Eq. 33 gives NLLM = 399 LLM responses per design condition in the illustrative two-condition comparison. The larger LLM run estimates the mean SUS contrast; the result is accepted only if TOST passes on a human holdout or pre-specified validation sample. If the research question concerns why users feel frustrated or how they narrate their experience, this flow is not sufficient because those outcomes are not conditional-mean targets. Flow 3: Framing survey or bounded behavioral response. Use this flow when the target is a 33

mean survey response or a mean contrast between finite experimental conditions. The researcher first specifies the conditions, response scale, minimum effect ∆, and equivalence margin. The calibration sample contains paired human and LLM responses to the same condition prompts; the b δ, b and ∆ b max , then apply the feasibility example below uses Mc = 500 and Mv = 100. Estimate bb, λ, rule. If feasible, compute NLLM = 1623 LLM responses per condition, collect the LLM responses, estimate the mean contrast, and validate with TOST against a human baseline or holdout. The output is a calibrated estimate of the population mean contrast; it is not evidence that individual LLM responses are distributed like individual human responses.

9.4

Empirical cost and power comparison

Example 9.1 (Framing survey with calibrated LLM augmentation). Consider a two-arm framing survey with bounded agreement score in [0, 1], minimum effect ∆ = 0.05, and human-only design Nh = 1000 per condition. Use illustrative costs ch = 8 USD per human participant and cLLM = 0.002 USD per LLM response. Calibration uses Mc = 500 paired human-LLM responses and Mv = 100 identifiability prompts, giving bb = 0.004,

b = 1.20, λ

δb = 0.01,

b max = 0.30. ∆

The feasibility penalty is b max 0.004 + 0.01 · 0.30 |bb| + δb∆ = = 0.14 < 1, ∆ 0.05 and Eq. 33 gives NLLM =

1000 · 1.20 1200 = ≈ 1623. 2 (1 − (0.004 + 0.01 · 0.30)/0.05) (1 − 0.14)2

Thus the LLM design uses 2 · 1623 = 3246 LLM responses. The illustrative costs are Chuman = 2 · 1000 · 8 = 16000,

CLLM = 500 · 8 + 3246 · 0.002 ≈ 4006.49,

so the cost reduction is

CLLM ≈ 74.96%. Chuman The LLM design is invalid if feasibility fails or if final TOST validation fails; the example reallocates human effort from large-scale estimation to calibration and validation. 1−

9.5

Reporting checklist for empirical studies

Report: (i) population, conditions, outcome scale, and estimand; (ii) ∆ and equivalence margin b δ, b and ∆ b max ; (v) feasibility result and NLLM ; ε; (iii) Mc , Mv , and condition coverage; (iv) bb, λ, (vi) TOST statistic, standard error, margin, and conclusion; and (vii) the restriction to conditionalmean-dependent squared-loss inference.

34

10

Conclusion

We have presented a rigorous mathematical framework establishing that pretrained large language models induce risk-equivalent estimators of conditional expectations under squared loss, establishing restricted functional risk equivalence within the class of conditional-mean-dependent inference problems. The framework is distinguished by rigorous proofs from first principles with minimal assumptions, zero architectural hypotheses about the LLM, and explicit scope boundaries for practitioners. The core theorems—bias-variance decomposition, irreducible error minimality, effect size consistency, power amplification, and cross-domain transfer—together with finite-sample concentration bounds provide a complete axiomatic foundation. The finite-sample analysis translates asymptotic guarantees into operational sample size requirements. The calibration protocol, with its feasibility check and TOST equivalence validation, enables operational deployment with explicit decision rules. The framework claims a narrow, provable statement: for estimating population means in quantitative experiments with discrete conditions, a well-calibrated LLM induces an estimator whose prediction risk converges to the Bayes risk under squared loss, achieving restricted functional risk equivalence within the class of conditional-mean-dependent inference problems. This is a modest claim, carefully circumscribed by explicit scope boundaries. It does not assert that LLMs can replace humans in all research contexts, that LLM-generated data are indistinguishable from human data in distribution, or that LLMs understand the constructs they measure. It asserts only that, under the specified conditions, restricted functional risk equivalence holds for squared-loss, conditional-mean-dependent inference. We believe this level of precision is essential for responsible deployment and for building a mathematically defensible foundation for LLMassisted quantitative research. In practical applications, this means that under satisfied conditions and well-calibrated models, large language models can be used in many prediction and decision-making tasks that originally relied on human experiments, approximating near-optimal statistical inference at lower cost.

Acknowledgements The author thanks colleagues for helpful discussions on the probabilistic foundations of this work.

Author contributions Haobo Yang developed the mathematical framework, applied methodology, and manuscript.

Competing interests The author declares no competing interests.

35

Data availability The complete mathematical derivations, calibration examples, and reproducibility materials are available in the project repository and Supplementary Information.

References [1] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Lawrence Erlbaum Associates, 1988. [2] A. W. van der Vaart, Asymptotic Statistics, Cambridge University Press, 1998. [3] N. Etemadi, An Elementary Proof of the Strong Law of Large Numbers, Z. Wahrsch. Verw. Gebiete, 55:119–122, 1981. [4] W. Hoeffding, Probability inequalities for sums of bounded random variables, J. Amer. Statist. Assoc., 58:13–30, 1963. [5] T. Gneiting, A. E. Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, J. Amer. Statist. Assoc., 102:359–378, 2007. [6] D. Blackwell, Equivalent comparisons of experiments, Ann. Math. Statist., 24:265–272, 1953. [7] L. Le Cam, Asymptotic Methods in Statistical Decision Theory, Springer, 1986. [8] E. L. Lehmann, G. Casella, Theory of Point Estimation, 2nd ed., Springer, 1998. [9] Google Research Datasets, UICrit: UI critique dataset, 2024. https://github.com/ google-research-datasets/uicrit. [10] Prolific, Pricing and participant rewards, accessed 2026. https://www.prolific. com/pricing. [11] J. Brooke, SUS: A Retrospective, Journal of Usability Studies, 8(2):29–40, 2013.

A

Construction of the Applied Cost Table

This appendix describes how Table 2 was built. The table is not an empirical claim about universal market prices; it is a transparent cost model for comparing a human-only design with a calibrated LLM-estimator design under the workflow in Algorithm 1. The calculations follow the procedures described below and are reproducible from the inputs given in the following subsections.

36

A.1

Inputs

For each use case, the computation uses a record containing: b δ, b∆ b max ). (K, Nh , Mc , ch , cLLM , ∆, bb, λ, Here K is the number of conditions, Nh is the baseline human sample size per condition, Mc is the paired calibration size, ch is the human cost per response, cLLM is the LLM response cost, ∆ is the b is the variance ratio, δb is the prompt-condition minimum effect of interest, bb is the calibrated bias, λ b max is the estimated condition range. identifiability error, and ∆ The three records used for Table 2 are: • Framing survey: K = 2, Nh = 1000, Mc = 500, ch = 8, cLLM = 0.002, ∆ = 0.05, bb = 0.004, b = 1.20, δb = 0.01, ∆ b max = 0.30. λ • UI design rating: K = 2, Nh = 300, Mc = 150, ch = 2.856, cLLM = 0.002, ∆ = 0.50, bb = 0.05, λ b = 1.10, δb = 0.02, ∆ b max = 4.0. • SUS human-experience score: K = 2, Nh = 200, Mc = 80, ch = 2.856, cLLM = 0.002, b = 1.15, δb = 0.01, ∆ b max = 40. ∆ = 5.0, bb = 0.8, λ For the UI and SUS examples, ch = 2.856 is the illustrative ten-minute participant cost implied by a 12/hour participant reward and a 1.428 platform-fee multiplier: ch = 12 ·

A.2

10 · 1.428 = 2.856. 60

Computation

For each case, the computation first finds the feasibility penalty ρbias =

b max |bb| + δb∆ . ∆

ρbias < 1

b ≤ 10. and λ

The case is feasible only if If feasible, the adjusted LLM sample size per condition is & ' b Nh λ NLLM = . (1 − ρbias )2 The human-only and LLM-augmented costs are then computed as Chuman = KNh ch ,

CLLM = Mc ch + KNLLM cLLM .

Finally, the cost reduction reported in Table 2 is Reduction = 1 − 37

CLLM . Chuman

A.3

Reproducibility check

The computation described above prints the public-source summary, the case inputs, the feasibility penalty, NLLM , both costs, and the cost reduction for each use case. It checks that all three cases are feasible and that the displayed values match the table: NLLM = (1623, 603, 399), CLLM ≈ (4006.49, 430.81, 230.08). These checks serve as a reproducibility aid for Table 2; they do not replace the calibration and validation requirements in the main method.

38