Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
arXiv:2609.18878v1 [cs.LG] 16 Sep 2026
Matteo Marchi1 , João Pedro Silvestre1 , Bahman Gharesifard2 , and Paulo Tabuada1 Abstract— Large Language Models (LLMs) are now routinely trained using synthetic data, since high-quality human data has been exhausted by the ever increasing needs of larger and larger models. However, recursive training on synthetic data frequently induces model collapse, a degenerative feedback loop where models progressively forget the true underlying data distribution. Training on a mixture of synthetic and fresh human data is a logical countermeasure and can prevent model collapse. However, it is an open question as to what is the exact minimum required ratio of human-to-synthetic data to maintain training stability. In this paper, we establish rigorous theoretical guarantees on the minimum rate of human data required to prevent model collapse. Although previous work established a formal lower bound for this ratio, such bound can be vacuous for very high dimensions, as the analysis relies on the usual Euclidean metric in Rn and is not adapted to the space of categorical probability distributions. Instead, in this paper we explicitly leverage the information-geometric structure of the probability simplex by analyzing the dynamics of the process under the Fisher-Rao metric. We derive quantitative contraction and invariance bounds that are stable and do not become trivial as the dimensions increase. Thus, we show that the effective required data ratio to prevent model collapse is different than previously implied.
I. I NTRODUCTION In recent years, generative AI, and in particular Large Language Models (LLMs) have become deeply ingrained in our society, primarily driven by a remarkable leap in performance and generation capabilities of recent models [1]. Today, modern LLMs can produce text that is virtually indistinguishable from human writing; in fact, recent studies show that human evaluators are often misguided by flawed heuristics when trying to identify AI-generated language [2]. However, this sudden leap in quality is not without drawbacks, including steep economic costs [3], heavy computational demands [4], and a reliance on increasingly vast training corpora [5]. Driven by the empirical scaling laws required to push state-of-the-art performance, developers are now training foundational models on trillions of tokens [6]. Unfortunately, sustaining this path requires such a large amount of data that some studies project the supply of 1 Matteo Marchi, João Pedro Silvestre, and Paulo Tabuada are with the Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA (e-mail: {matmarchi,joaosilvestre,tabuada}@ucla.edu). 2 Bahman Gharesifard is with the Department of Mathematics and Statistics at Queen’s University, Kingston, ON, Canada (e-mail: [email protected]). This research was supported in part by the US ARL Cooperative Agreement W911NF-17-2-0196 and by the NSF award 2502536. The work of João Pedro Silvestre was partially supported by the PhD fellowship 2023.01843.BD from the Fundação para a Ciência e a Tecnologia (FCT), Portugal.
fresh, high-quality human text will soon be completely exhausted [7]. The high caliber of AI-generated text presents a tempting solution to this impending data scarcity by leveraging the models’ own synthetic outputs for future training. However, recursive training on machine-generated data leads to a critical failure mode commonly referred to as model collapse, as demonstrated both empirically [8], [9] and theoretically [10]. Instead of learning effectively, models caught in this degenerative feedback loop deviate from the true data distribution, amplifying their own errors and producing repetitive, homogenized outputs [8], [11]. Integrating fresh human data into the iterative training cycle appears to be a logical countermeasure, but current evidence suggests that simplistic strategies, such as injecting small, fixed proportions of real data, are insufficient to stop model collapse [12]. Alternative mitigation strategies are being actively investigated, such as employing data verification and curation pipelines to filter out degraded synthetic outputs [13]. Although curating synthetic data can delay the onset of degeneration, it introduces substantial computational overhead and relies heavily on the quality and robustness of the verifier itself. Recent empirical and theoretical works have established that incorporating sufficient human data can prevent model collapse [14], [15], yet characterizing the precise dynamics of this mixed-data regime remains a significant challenge. This naturally raises a fundamental question: can we rigorously bound the amount of human data required to preclude model collapse? We tackle this problem by building upon the framework introduced in [15], modeling the iterative training of generative models as a closed-loop stochastic process. This previous work has primarily focused on analyzing asymptotic equilibrium states [10], [15] and worked with the traditional Euclidean metric, which progressively distorts distances between probability distributions as the dimensions increase. Our approach fundamentally departs from this prior analysis by explicitly leveraging the information-geometric structure of the probability simplex. Specifically, by working with the Fisher-Rao metric, we derive quantitative contraction and invariance bounds that remain stable and meaningful as the underlying dimensions of the model increase. This geometric perspective shows that the effective amount of human data required to prevent collapse is greater than previously implied. II. N OTATION AND P RELIMINARIES A. Notation We denote by Rn the n-dimensional Euclidean space, R+ 0 as the set of nonnegative real numbers, N as the set of natural
Pn n numbers with zero, ∆n ≜ x ∈ (R+ 0) | i=1 xi = 1 as the n-dimensional probability simplex, ∥ · ∥1 as the 1-norm, ∥ · ∥2 as the 2-norm, and ⟨·, ·⟩ as the inner product. We use standard asymptotic notations O(·) and Θ(·) to describe the limiting behavior of sequences. In particular, for sequences of functions an : N → R and bn : N → R>0 , we write an = O(bn ) whenever there exist c ∈ R+ and n0 ∈ N such that: |an | ⩽ c bn ,
for all n ⩾ n0 .
We also write an = Θ(bn ) whenever there exist c1 , c2 ∈ R+ 0 and n0 ∈ N such that: c1 bn ⩽ |an | ⩽ c2 bn ,
for all n ⩾ n0 .
Given vectors v ∈ [0, 1]n and w ∈]0, 1]n , we define: ∥v∥2diag(w) = v ⊤ diag(w)v, where diag(w) ∈ Rn×n denotes the square matrix whose diagonal consists of the entries of w, and whose off-diagonal elements are 0. Given a scalar function f : R → R and a vector argument x ∈ Rn , we denote f (x) = (f (x1 ), f (x2 ), . . . , f (xn )) ∈ Rn as its element-wise application to x. Consider the interior of the probability n simplex ∆nint = ∆n \∂∆n . We equip with the FisherPn∆int Rao Riemannian metric gθ (u, v) =P i=1 uθi ivi , for u, v in the n tangent space Tθ ∆n = {u ∈ Rn : i=1 ui = 0}. This metric induces the geodesic Hellinger distance on the probability simplex [16]: n p X dFR (θ, ϑ) = arccos θ i ϑi , (1) i=1
which is naturally √ related √ to the squared √ √Hellinger distance H 2 (θ, ϑ) := 21 ∥ θ − ϑ∥22 = 1 − ⟨ θ, ϑ⟩. Let θ, ϑ ∈ ∆nint .
Definition 1 (Kullback-Leibler divergence). The Kullback-Leibler (KL) divergence of θ from ϑ is: DKL (θ∥ϑ) :=
n X i=1
θi log
θi . ϑi
DKL is non-negative and equals zero if and only if θ = ϑ. B. Generative Models In this section we describe the mathematical model used to analyze a generative model and its iterative training process. This is largely based on the model first presented in [10] that we extend in this work. We define a generative model to be a function ϕ : Rp → n ∆ that maps a parameter vector w ∈ Rp to an output distribution Θ = ϕ(w) ∈ ∆n . The i-th entry of ϕ(w) is the nominal probability of producing the i-th element from a list of outcomes Y = {Y1 , . . . Yn } when the model is queried. With no loss of generality, we assume that the i-th element of Y is the n-dimensional vector containing 1 in its i-th entry and 0 in all others1 . 1Within machine learning literature, this is referred to as “one-hot encoding” of a categorical variable.
In practice, when a model generates data, the actual output probability distribution is modulated via a temperature function τ : ∆n → ∆n , defined for i = 1, 2, . . . , n as: 1/T
Θ τi (Θ) = Pn i
1/T j=1 Θj
,
T > 0.
(2)
The temperature function describes the common practice of converting a T1 -scaled vector of raw logits, produced by a generative model, into a vector of probabilities. This specific form of τ , induced by the standard softmax function, is ubiquitous across nearly all modern generative models used in practice. For a more detailed discussion and a broader class of temperature functions, we refer the reader to [10]. C. Iterative Training We now consider a sequence of generative models, Θ(k) = ϕ(w(k)), indexed by k ∈ N, each trained on a dataset Dk with cardinality ℓk = |Dk |. The dataset at any time step k is a multiset2 Dk = {Y1 , . . . , Yℓk } where each element belongs to the set of possible outcomes Y. To any nonempty dataset, we can associate a corresponding “empirical” probability vector: ℓ
Θ′ (k) =
k 1 X Yi , ℓk i=1
(3)
whose entries are the relative frequencies of each possible outcome within the dataset. Dk evolves by accumulating some amount of “fresh” human generated data3 and some amount of synthetic data generated by the model trained at the current time step. Specifically, at any time step k, we assume that αk ∈ N outcomes of Y are sampled according to τ (Θ(k)) to form: Dksyn = {Yℓk +1 , . . . , Yℓk +αk }. Additionally, βk ∈ N outcomes of Y are sampled according to a fixed external distribution H ∈ ∆n to form: Dkhuman = {Yℓk +αk +1 , . . . , Yℓk +αk +βk }. The training dataset available at step k + 1 is thus: Dk+1 = Dk ∪ Dksyn ∪ Dkhuman , of cardinality ℓk+1 = ℓk + αk + βk . The training of a generative model at time step k +1 occurs by using the newly available dataset Dk+1 (and possibly the previously trained w(k)) to compute a new parameter vector w(k + 1) = f (w(k), Dk+1 ), where f abstracts away the details of the training and optimization process. Then, the evolution of Θ is described by the stochastic process: Θ(k + 1) = ϕ f (w(k), Dk+1 ) . (4) 2 Multisets generalize sets by allowing them to contain multiple instances of the same element. We cannot use a simple set to describe the training data, as we need to track the relative frequency of each element in Dk . 3While we use the expression “human data” to evoke the notion of humanproduced content on the Internet, this refers to any kind of data coming from a fixed external probability distribution.
In [10], the authors analyze the asymptotic behavior of (4) in the absence of fresh data (βk = 0). They show that in the limit of k → ∞ the distribution learned by the generative model exhibits a high degree of degeneration or model collapse with high probability. Specifically, Θ(k) becomes arbitrarily close to the boundary of the simplex (in fact, to the corners of the simplex) or to its center (uniform probability distribution). The question of whether this degeneration can be mitigated by injecting human data in the loop was investigated by the authors of [15]. Assuming βk = µαk for a constant ratio µ > 0, the limiting behavior of (4) is determined almost surely by the behavior of the continuous-time dynamical system: θ̇(t) = τ (θ(t)) − θ(t) + µ θ0 − θ(t) + ε(t), (5)
Assumption 1. There exists η ⩾ 0 such the term ε(t) ∈ Rn in (5) satisfies ∥ε(t)∥∞ ⩽ η for all t ⩾ 0, and all entries of θ0 , the human distribution, are strictly positive:
where θ0 := H is the human distribution, ε(t) is a bounded perturbation, and wePassume the flow of (5) preserves the n simplex, implying i=1 θ̇i = 0. The magnitude of the perturbation ε is a measure of training accuracy, and a model that learns a dataset distribution with low error has a correspondingly small perturbation ε. Under these assumptions, the results in [15] state that for a sufficiently high ratio of human data µ, the trajectories of (5) converge to an Euclidean ball B(θe , ϵ) around an equilibrium θe satisfying τ (θe ) − θe + µ(θ0 − θe ) = 0 and provide an expression bounding the size of this ball as a function of µ and the other problem parameters. However, analyzing these dynamics under the Euclidean metric yields bounds that become increasingly uninformative as the dimension n of the probability simplex grows. For n → ∞, the Euclidean distance between almost any two probability distributions approaches zero [17]. Thus, requiring trajectories to converge to an Euclidean ball of a fixed radius becomes a progressively weaker condition, as such a ball eventually encompasses the majority of the simplex regardless of the choice of ratio µ. For an example illustrating this, take n to be even and consider the family of “disjoint” probability vectors of form p = 2 2 2 2 , . . . , , 0, . . . , 0 and q = 0, . . . , 0, , . . . , . Despite n n n n representing completely distinct categorical outcomes, their √ Euclidean distance scales as O(1/ n) and vanishes as n → ∞. By contrast, the Fisher-Rao metric captures the underlying information-geometric structure of ∆n (see [18]) and assigns a constant positive distance dFR (p, q) = arccos(0) = π2 between p and q regardless of n. Because the Euclidean metric progressively under-penalizes the distance between distributions, it yields an overly optimistic assessment of the required human data scaling. The main contribution of this paper is to substantially refine this analysis by working directly on the Fisher-Rao manifold of ∆n .
We can now introduce the main contribution of this work in the following theorem. This result identifies a ball in the Fisher-Rao metric that θ will converge to, the convergence rate to this ball, and a minimum threshold for the humanto-synthetic data ratio that guarantees this behavior. Here, we merely state the theorem and prove it in the following section.
III. M AIN R ESULT We first need to make the following assumptions. We require that the magnitude of the perturbation ε is bounded, the human data distribution does not lie exactly on the boundary of the probability simplex, and that there exists an equilibrium of (5) when ε = 0.
δ := min θ0,i > 0. i
Assumption 2. There exists θe ∈ ∆n satisfying: τ (θe ) − θe + µ(θ0 − θe ) = 0.
(6)
Further, it is convenient to define the following quantity: Definition 2. Let θemin be the smallest element of θe , then we define the normalized inverse temperature ηFR as: ηFR :=
1 . T θemin
Theorem 1. Suppose that Assumptions 1-2 hold and that µδ > η, where µ is the ratio between human and synthetic data, i.e., µ = βk /αk . Fix κ ∈ [0, 1) and consider t ⩾ tκ 1 1 where tκ = 1+µ ln 1−κ . If the following inequality holds: ) ( 2η 1 + T ηFR κ η + κFR , , (7) µ ⩾ max T δκ δ there exists λ > 0 such that every solution of (5) satisfies: q π dFR (θ(t), θe ) ⩽ e−λ(t−tκ ) DKL (θ(tκ )∥θe ) + ϵFR , (8) 2 where: p π η n maxi θe,i . (9) ϵFR = √ 2ηFR maxi θe,i 2 κ (µδ − η) 1 − κ(µδ − η) Note that (8) implies that θ converges to a Fisher-Rao ball of size ϵFR for t → ∞. We now compare the bound provided by (9) in Theorem 1 with the bound obtained in [15] that establishes convergence to an Euclidean ball of size: ϵ=
ηκ(µδ − η)T . (µ + 1)(κ(µδ − η)T − 1)
(10)
To more easily exhibit the relative scaling of the bounds, we assign scaling laws to δ and µ as functions of n, the dimension of the probability simplex ∆n . Proposition 1. Suppose that the assumptions of Theorem 1, and the ones in [15, Theorem 1], hold and that: δn ∼ n−β0 ,
µn ∼ c np ,
∥ε∥∞ = η,
for some β0 > 1, c > 0, and p > 2β0 . Then: 1) In the Euclidean case (10): ϵ = Θ n−p .
2) In the Fisher-Rao case (9): 1 ϵFR = Θ n 2 +β0 −p . Proof. We first prove (1). Consider (10) given by: ϵ=
ηκ(µn δn − η)T , (µn + 1) κ(µn δn − η)T − 1
with δn ∼ n−β0 and µn ∼ cnp . Since µn δn ∼ cnp−β0 , with p ⩾ β0 , and µn + 1 ∼ µn , we have that: η ϵ∼ = Θ(n−p ), µn which proves the claim. To prove (2), we first have to determine how θe scales with n under the assumptions. Manipulating (6), we obtain: 1 θe − θ 0 = τ (θe ) − θ0 . 1 + µn Taking ℓ∞ norms and noting τ (θe ), θ0 ∈ ∆n , we get: ∥θe − θ0 ∥∞ ⩽
∥θe − θ0 ∥∞ = O(n−p ). and
max θe,i = max θ0,i + O(n−p ) −p
⩽ 1 − (n − 1) min θ0,i + O(n i
= 1 − Θ(n
1−β0
i
= Θ(n
κ(µδ − η) =: θ > 0, 1+µ
−p
Proof. For each i, we have that:
)
) − O(n
)
−p
θ̇i = −θi + µ(θ0,i − θi ) + τi (θ) + εi ,
θ̇i ⩾ −θi + µ(θ0,i − θi ) − η = −(1 + µ)θi + µθ0,i − η. −β0
) = Θ(n
).
We are now in a position to prove (2), by considering (9). Recall that µn ∼ cnp and δn ∼ n−β0 , so: µn δn ∼ cnp−β0 .
Therefore: θi (t) ⩾ e−(1+µ)t θi,0 + 1 − e−(1+µ)t
µθ0,i − η . 1+µ
The first term decays exponentially, while the second term µθ0,i −η approaches the steady-state value 1+µ . To ensure a uniform lower bound after some finite time, we select t ⩾ tκ such that: 1 − e−(1+µ)t ⩾ κ,
Using maxi θe,i ⩽ 1 − Θ(n1−β0 ) and mini θe,i = Θ(n−β0 ), we obtain: maxi θe,i 1−Θ(n1−β0 ) 2 −β 0 T mini θe,i 2ηFR maxi θe,i Θ(n ) = ⩽ κ(µn δn − η) κ(µn δn − η) Θ(np−β0 ) β0 n = Θ n2β0 −p , =Θ p−β 0 n
where κ ∈ [0, 1), which is equivalent to enforcing that: 1 1 t⩾ ln . 1+µ 1−κ
which tends to 0 for every p > 2β0 . Hence:
As a result, for all t ⩾ tκ , we then obtain:
1−
(11)
where by assumption τi (θ) ⩾ 0 and εi ⩾ −η. Hence:
min θe,i = min θ0,i − O(n −β0
) + O(n
)
), −p
i
θi (t) ⩾ provided that µδ > η.
i
= 1 − (n − 1)Θ(n
We proceed in steps, proving some intermediate lemmas before the main result. First, we show that solutions of (5) cannot approach the boundary asymptotically.
such that, for all t ⩾ tκ and all i ∈ {1, 2, . . . , n}:
Noting that maxi θ0,i ⩽ 1 − (n − 1) mini θ0,i mini θ0,i = δn ∼ n−β0 , we can establish that:
−β0
IV. P ROOF OF THE MAIN RESULT
Lemma 1. Suppose that Assumptions 1-2 hold, and let θ(·) be the solution to (5). For any κ ∈ [0, 1), there exists a time: 1 1 ln tκ = , 1+µ 1−κ
2 . 1 + µn
Hence, if µn ∼ cnp with p ⩾ 1 we have:
i
Without structural information on θe beyond the simplex constraints, a natural choice for the amount of human data required to obtain (for example) a O(1/n) decay is µn = 5 n 2 +γ , with γ > 0. This keeps the Fisher-Rao error uniformly controlled as the dimension grows. By contrast, the Euclidean estimate does not account for the geometric cost of placing probability mass across n coordinates: with µn ∼ n it predicts an error of order 1/n. The Fisher-Rao geometry analysis above reveals that this can be misleading, as µn ∼ n does not prevent the Fisher-Rao error from growing, and in this sense, a stronger growth of µn is required. We devote the next section to proving Theorem 1.
2ηFR maxi θe,i = Θ(1). κ(µn δn − η)
Moreover, µn δn − η ∼ µn δn , so the denominator of (9) is asymptotically proportional to µn δn , and ϵFR scales like: p p 1 π η n maxi θe,i n (1 − Θ(n1−β0 )) 2 +β0 −p √ ⩽ . = Θ n Θ(np−β0 ) 2 κ µ n δn
θi (t) ⩾ κ
µθ0,i − η . 1+µ
Finally, since by Assumption 1 each θ0,i ⩾ δ, we have: θi (t) ⩾
κ(µδ − η) =: θ, 1+µ
which is positive whenever µδ > η, establishing (11).
The uniform floor θ comes directly from the dynamics and is independent of metric; it is essential for our Fisher-Rao analysis, as the metric is not well-defined on the boundary set ∂∆n . We now adopt the KL-divergence between θ(t) and the equilibrium θe as a Lyapunov function, and establish its Lie derivative along the vector field (5). Lemma 2. Let:
where a1 = c11 , and therefore: D E θ log , τ (θ) − τ (θe ) ⩽ ηFR a1 V (θ). θe For (B), we have that: D E X X θ θi θe,i log , θ − θe = θi log + θe,i log θe θ θi e,i i i = DKL (θ∥θe ) + DKL (θe ∥θ) ⩾ V (θ),
V (θ) = DKL (θ∥θe ). and therefore: Then, along any solution of (5) we have: D E θ(t) V̇ (θ(t)) = log , τ (θ(t)) − τ (θe ) θe D E D E (12) θ(t) θ(t) −(1 + µ) log , θ(t) − θe + log , ε(t) . θe θe Proof. First note that ∇V (θ) = log(θ/θe ) + 1, where 1 is the vector whose all entries are 1. For brevity, we now drop the dependency of the trajectoriesPon t. Since the trajectory n d stays on the simplex, ⟨1, θ̇⟩ = dt i=1 θi = 0, we have that: D θ E V̇ (θ) = ⟨∇V (θ), θ̇⟩ = log , θ̇ . θe Using now (5), adding and subtracting τ (θe ), we have that: D E θ V̇ (θ) = log , τ (θ) − τ (θe ) θe E D D θ E θ + log , τ (θe ) − θ + µ(θ0 − θ) + log , ε . θe θe
D E θ −(1 + µ) log , θ − θe ⩽ −(1 + µ)V (θ). (13) θe For (C), by Hölder’s inequality, we have that: D θ E θ log , ε ⩽ log ∥ε∥∞ . θe θe 1 Letting x = log θ − log θe and using Cauchy-Schwarz: X θ = ∥x∥1 = |xi | log θe 1 i X 1 1/2 X 1/2 ⩽ θe,i x2i . θe,i i i P Note that by definition, i θe,i x2i = ∥ log θ − log θe ∥2diag(θe ) . Hence, using Lemma 8 again, we have that: X 1 θe,i x2i ⩽ V (θ). c1 i
which gives (12).
Whenever θi ⩾ θ, we obtain: p θ log ⩽ ã2 V (θ), θe 1 q P where ã2 = c11 i 1/θe,i . Note that since 1/θe,i ⩽ 1/θ, P we have that i 1/θe,i ⩽ n/θ and hence: q ã2 ⩽ a2 := cn1 θ .
We are now finally fully equipped to prove the results stated in Theorem 1.
Substituting these bounds into the expression for V̇ yields: p V̇ (θ) ⩽ (ηFR a1 − (1 + µ))V (θ) + a2 η V (θ), η := ∥ε∥∞ .
We use the equilibrium condition τ (θe ) − θe + µ(θ0 − θe ) = 0 to rewrite: τ (θe ) − θ + µ(θ0 − θ) = − (θ − θe ) + µ(θ − θe ) = −(1 + µ)(θ − θe ),
Proof of Main Theorem. For t ⩾ tκ , Lemma 1 ensures θi (t) ⩾ θ. From Lemma 2, along any trajectory we have: θ V̇ (θ) = log , τ (θ) − τ (θe ) θe | {z }
Next we verify that the lower bound on µ in our assumption guarantees that (1+µ)−ηFR a1 > 0. Recall that θ = κ(µδ−η) 1+µ , and that c1 = 2 maxθi θe,i by Lemma 8. Therefore: a1 =
(A)
1 2 maxi θe,i 2 maxi θe,i (1 + µ) 2(1 + µ) = = ⩽ , c1 θ κ(µδ − η) κ(µδ − η)
E D θ θ E −(1 + µ) log , θ − θe + log , ε . θe θ | {z } | {ze }
where we have used the fact that maxi θe,i ⩽ 1. Thus: 2ηFR (1 + µ) − ηFR a1 ⩾ (1 + µ) 1 − , κ(µδ − η)
For (A), by Lemma 7, τ is ηFR –Lipschitz in the Fisher-Rao metric, so: D E θ log , τ (θ) − τ (θe ) ⩽ ηFR ∥ log θ − log θe ∥2diag(θe ) . θe
and to ensure the right-hand side is positive, we require: 2ηFR 2ηFR 1− > 0 ⇐⇒ µδ − η > , κ(µδ − η) κ
Since θi (t) ⩾ θ, the conditions of Lemma 8 apply:
µ⩾
D
(B)
(C)
∥ log θ − log θe ∥2diag(θe ) ⩽ a1 V (θ),
but this is exactly enforced by the assumption that η+
2ηFR κ . Given this, we let: δ
0 < λ < (1 + µ) − ηFR a1
2
b and apply Young’s inequality ab ⩽ λa2 + 4λ with a = and b = a2 η to obtain:
p V (θ)
a2 η 2 V̇ (θ) ⩽ − (1 + µ) − ηFR a1 − λ V (θ) + 2 . 4λ We can indeed choose: ηFR , λ = 12 (1 + µ) − c1
R EFERENCES
so that (1 + µ) − ηFR a1 − λ = λ, and therefore: a22 η 2 . 4λ Consequently, for all t ⩾ tκ we have that: V̇ (θ) ⩽ −λV (θ) +
V (θ(t)) ⩽ e−λ(t−tκ ) V (θ(tκ )) + ϵ2FR 1 − e−λ(t−tκ ) , a2 η 2
2 where: ϵ2FR := 4λ 2 . By substituting a2 , λ, and θ, we have:
ϵ2FR =
nη 2 (1 + µ) c1 κ(µδ − η) (1 + µ) − ηcFR 1
2 .
(14)
Using: c1 =
θ κ(µδ − η) = , 2 maxi θe,i 2(1 + µ) maxi θe,i
we obtain one of the terms in the denominator of (14): c1 κ(µδ − η) =
κ2 (µδ − η)2 . 2(1 + µ) maxi θe,i
To compute the other term, we write: (1 + µ) −
the information-geometric structure of the probability simplex naturally accounts for the distortions introduced by working on the simplex, yielding bounds that scale properly with the dimension of the underlying categorical distributions.
ηFR 2(1 + µ)ηFR maxi θe,i = (1 + µ) − c1 κ(µδ − η) 2ηFR maxi θe,i = (1 + µ) 1 − , κ(µδ − η)
and substituting both identities yields: 2n η 2 maxi θe,i 2 . 2ηFR maxi θe,i 2 2 κ (µδ − η) 1 − κ(µδ − η) p Finally, by Lemma 4, dFR (θ, θe ) ⩽ π2 V (θ), and therefore: q π dFR (θ(t), θe ) ⩽ e−λ(t−tκ ) DKL (θ(tκ )∥θe ) + ϵ2FR 2q π ⩽ e−λ(t−tκ ) DKL (θ(tκ )∥θe ) + ϵFR , 2 with ϵFR matching (9), which establishes (8). ϵ2FR =
V. C ONCLUSION In this work, we tackled the fundamental challenge of bounding the amount of human data required to prevent model collapse. By modeling the iterative training of generative models as a closed-loop stochastic process, we demonstrated that trajectories converge to a stable ball in the Fisher-Rao metric when the human-to-synthetic data ratio exceeds a specific threshold. We fundamentally departed from prior analysis that relies on the standard Euclidean metric. Ultimately, leveraging
[1] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023. [2] M. Jakesch, J. T. Hancock, and M. Naaman, “Human heuristics for AIgenerated language are flawed,” Proceedings of the National Academy of Sciences, vol. 120, no. 11, p. e2208839120, 2023. [3] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?,” in Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623, 2021. [4] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 3645–3650, 2019. [5] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., “Training compute-optimal large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016–30030, 2022. [6] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [7] P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho, “Will we run out of data? an analysis of the limits of scaling datasets in machine learning,” arXiv preprint arXiv:2211.04325, 2022. [8] I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, “AI models collapse when trained on recursively generated data,” Nature, vol. 631, no. 8022, pp. 755–759, 2024. [9] S. Alemohammad, J. Casco-Rodriguez, L. Luzi, A. I. Humayun, H. Babaei, D. LeJeune, A. Siahkoohi, and R. Baraniuk, “Self-consuming generative models go MAD,” in The Twelfth International Conference on Learning Representations, 2024. [10] M. Marchi, S. Soatto, P. Chaudhari, and P. Tabuada, “Heat death of generative models in closed-loop learning,” in 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 1524–1530, 2024. See also arXiv:2404.02325. [11] D. Herel and T. Mikolov, “Collapse of self-trained language models,” in The Second Tiny Papers Track at ICLR, 2024. [12] M. Briesch, D. Sobania, and F. Rothlauf, “Large language models suffer from their own output: An analysis of the self-consuming training loop,” arXiv preprint arXiv:2311.16822, 2023. [13] Y. Feng, E. Dohmatob, P. Yang, F. Charton, and J. Kempe, “Beyond model collapse: Scaling up with synthesized data requires verification,” in The Thirteenth International Conference on Learning Representations, 2025. [14] M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, T. Korbak, H. Sleight, R. Agrawal, J. Hughes, D. B. Pai, A. Gromov, D. Roberts, D. Yang, D. L. Donoho, and S. Koyejo, “Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data,” in First Conference on Language Modeling, 2024. [15] B. Gharesifard and P. Tabuada, “Preventing model collapse when training LLMs with synthetic data,” in 2025 IEEE 64th Conference on Decision and Control (CDC), pp. 1124–1129, IEEE, 2025. [16] H. K. Miyamoto, F. C. Meneghetti, J. Pinele, and S. I. Costa, “On closed-form expressions for the Fisher–Rao distance,” Information Geometry, vol. 7, no. 2, pp. 311–354, 2024. [17] C. C. Aggarwal, A. Hinneburg, and D. A. Keim, “On the surprising behavior of distance metrics in high dimensional space,” in International conference on database theory, pp. 420–434, Springer, 2001. [18] S.-i. Amari, Information Geometry and Its Applications, vol. 194. Springer, 2016. [19] A. B. Tsybakov, Introduction to Nonparametric Estimation. Springer Series in Statistics, 2009.
A PPENDIX Following we state and prove a number of technical Lemmas that the main result relies on. Lemma 3. For any θ, ϑ ∈ ∆n , the following inequalities hold: √ 2H(θ, ϑ) ⩽ dFR (θ, ϑ) ⩽ √π2 H(θ, ϑ). √ √ Proof. Let c := θ, ϑ ∈ [0, 1]. By definition, dFR (θ, ϑ) = arccos(c) and H 2 (θ, ϑ) = 1 − c. Using the identity 1 − cos x = 2 sin2 (x/2), we have H(θ, ϑ) = √ 1 2 sin 2 dFR (θ, ϑ) . Since π2 y ⩽ sin y ⩽ y for y ∈ [0, π2 ], substituting y = dFR /2 yields the stated bounds. Lemma 4. For any θ, ϑ ∈ ∆nint , we have: πp DKL (θ∥ϑ). dFR (θ, ϑ) ⩽ 2
Hence ψ is increasing on (0, ∞); in particular, ψ(r) ⩾ ψ(1) = 0 for r ⩾ 1. Therefore 1 r(log r)2 , r ⩾ 1. 2 This establishes the bounds for the case where r ⩾ 1. Case 0 < r ⩽ 1: For the lower bound, we naturally use ψ this time. As above, ψ ′ (r) = (log r)2 ⩾ 0 for all r > 0, so ψ is increasing on (0, ∞) and: g(r) ⩽
ψ(r) ⩽ ψ(1) = 0,
0 < r ⩽ 1.
Thus:
Proof. By [19, Lemma 2.4], the KL divergence bounds the squared Hellinger distance as DKL (θ∥ϑ) ⩾ 2H 2 (θ, ϑ). Combining this with the upper bound in Lemma 3 immediately establishes the result. Lemma 5. For all u > 0 we have: √
For the upper bound, we use ψ: d ψ ′ (r) = r(log r)2 − 2g ′ (r) dr = (log r)2 + 2 log r − 2 log r = (log r)2 ⩾ 0.
2
u log u − (u − 1) ⩾ (1 − u) . √ Proof. Set s = u and define g(s) := 2s2 log s−s2 +2s−2. We write g as g(a) = 2s (s log s − s + 1). Since 2s > 0 for s > 0, it suffices to show that h(s) = s log s − s + 1 > 0 for s > 0. By Taylor’s theorem h(s) = h(1) + h′ (1)(s − 1) + 1 ′′ ′ 2 ′ 2 h (s )(s − 1) for some s > 0. Computing the derivatives we obtain h(s) = 1 + 0(s − 1) + 12 s1′ (s − 1)2 . By inspection we see that h(s) > 0.
1 r(log r)2 , 2 For the upper bound, we use ϕ again. From: 1 , ϕ′ (r) = 2 log r 1 − r
r(log r)2 −2g(r) ⩽ 0
=⇒
g(r) ⩾
0 < r ⩽ 1.
we see that for 0 < r ⩽ 1 we have log r ⩽ 0 and 1 − 1r ⩽ 0, so ϕ′ (r) ⩾ 0 on (0, 1]. Hence ϕ is increasing on (0, 1] and: ϕ(r) ⩽ ϕ(1) = 0,
0 < r ⩽ 1.
Therefore: 1 (log r)2 , 0 < r ⩽ 1. 2 This establishes the bound for the case where 0 < r ⩽ 1. Combining the two cases yields (15) for all r > 0. 2g(r)−(log r)2 ⩽ 0
=⇒
g(r) ⩽
Lemma 7. Let θe ∈ ∆n , T > 0, θemin := mini θe,i > 0, and:
Lemma 6. Let g : R>0 → R be defined as: g(r) := r log r − (r − 1).
ηFR :=
Then, for all r > 0:
1 . T θemin
Then, for all θ ∈ ∆nint the following inequality holds:
1 1 min{r, 1}(log r)2 ⩽ g(r) ⩽ max{r, 1}(log r)2 . (15) 2 2 Proof. We begin by computing the derivative of g: g ′ (r) = log r,
r > 0.
(16)
We define the functions: 2
ϕ(r) := 2g(r) − (log r) ,
log θ−log θe , τ (θ)−τ (θe ) ⩽ ηFR ∥ log θ−log θe ∥2diag(θe ) . (17) Proof. Let φ = log θ and φe = log θe and consider: n X f (φ) := T log eφi /T . i=1
2
ψ(r) := r(log r) − 2g(r).
Since g(1) = 0, we have ϕ(1) = ψ(1) = 0. We now prove the result by considering two cases: Case r ⩾ 1: For the lower bound, using (16): 2 log r 1 ϕ′ (r) = 2g ′ (r) − = 2 log r 1 − . r r If r ⩾ 1, then log r ⩾ 0 and 1 − 1r ⩾ 0, so ϕ′ (r) ⩾ 0 on [1, ∞). Thus ϕ(r) ⩾ ϕ(1) = 0 for all r ⩾ 1, which implies: 1 g(r) ⩾ (log r)2 , 2
r ⩾ 1.
Clearly, ∇f (φ) = τ (θ). For i, j ∈ {1, . . . , n}, we have that: ∂(∇f )i ∂τi (θ) 1 = = τi (θ) δij − τi (θ)τj (θ) , ∂φj ∂φj T where δij = 1 when i = j and zero otherwise, which follows from taking the derivative of: eφi /T τi (θ) = τi (φ) = Pn . φk /T k=1 e Therefore: ∇2 f (φ) =
1 diag(τ (θ)) − τ (θ)τ (θ)⊤ . T
For simplicity of calculations to follow, we let p := τ (θ). We also let v ∈ Rn be arbitrary, with ∥v∥2 = 1. Then: X 2 X v ⊤ diag(p) − pp⊤ v = pi vi2 − pi vi i
⩽
X
i
pi vi2 ⩽ ∥v∥22 ,
P as we previously established that i θe,i (ri − 1) = 0. By Lemma 6 in the Appendix, for every r > 0: 1 1 min{r, 1}(log r)2 ⩽ g(r) ⩽ max{r, 1}(log r)2 . 2 2 Applying it with r = ri and noting xi = log ri , we obtain: n
i
P
where we have used the fact that i pi = 1. Hence, taking the supremum over unit vectors: 1 1 sup v ⊤ diag(p) − pp⊤ v ⩽ . (18) ∇2 f (φ) 2 = T ∥v∥2 =1 T Therefore, for all x, y ∈ Rn we have that:
1X θe,i min{ri , 1}x2i ⩽ DKL (θ∥θe ) 2 i=1 n
1X θe,i max{ri , 1}x2i . ⩽ 2 i=1
We now produce uniform bounds on the factors min{ri , 1} and max{ri , 1}. Because θ ∈ I and θ ∈ ∆n :
1 ∥x − y∥22 , T where we used (18) as an upper bound for the Lipschitz constant of ∇f . Applying this with x = φ and y = φe gives:
θ ⩽ θi ⩽ 1,
0 < θe,i ⩽ θ̄e := max θe,j , j h θ θi 1 i ri = ∈ , . θe,i θe,i θe,i
⟨x − y, ∇f (x) − ∇f (y)⟩ ⩽
⟨φ − φe , τ (θ) − τ (θe )⟩ ⩽
Hence:
1 ∥φ − φe ∥22 . T
Since diag(θe ) ⪰ θemin I, we have ∥φ − φe ∥22 ⩽ (θemin )−1 ∥φ − φe ∥2diag(θe ) , and therefore, a substitution, yields (17). Lemma 8. Fix θ ∈ (0, 1) and let: n o I := θ ∈ ∆n | θi ⩾ θ for all i .
⩽ c2 ∥ log θ − log θe ∥2diag(θe ) , with: c1 =
θ , 2 maxi θe,i
c2 =
(19)
1 . 2 mini θe,i
i=1
θi ∈ (0, ∞), xi := log ri = log θi − log θe,i . θe,i P P P Since i θi = i θe,i = 1, we have i θe,i (ri − 1) = 0. To proceed, observe that the KL divergence can be written as a Bregman divergence. Recall that for a convex function f : R+ → R, the Bregman divergence between u, v > 0 is: ri :=
Df (u∥v) = f (u) − f (v) − f ′ (v)(u − v). Consider the convex function: f ′ (u) = log u + 1,
f ′′ (u) =
For each i: Df (ri ∥1) = f (ri ) − f (1) − f ′ (1)(ri − 1) = ri log ri − (ri − 1) =: g(ri ), since f (1) = 0 and f ′ (1) = 1. Therefore: DKL (θ∥θe ) =
n X i=1
θe,i g(ri ),
n θ
Finally, since θe,i ⩽ θ̄e and θe,i ⩾ θe := mini θe,i , we have: n X
Proof. Let us define:
f (u) = u log u,
o nθ o ,1 , , 1 ⩾ min θe,i θ̄e n 1 o 1 max{ri , 1} ⩽ max ,1 = , θe,i θe,i
min{ri , 1} ⩾ min
where for the last equality we use the fact that θe,i ⩽ 1. Substituting these bounds into (20) gives: n n n θ oX 1 1X 2 min ,1 θe,i x2i ⩽ DKL (θ∥θe ) ⩽ x . (21) 2 2 i=1 i θ̄e i=1
Then, for all θ ∈ I: c1 ∥ log θ − log θe ∥2diag(θe ) ⩽ DKL (θ∥θe )
(20)
1 . u
n
x2i ⩽
1 X θe,i x2i . θe i=1
(22)
Combining (21) and (22) yields: n n n θ oX 1 1 X θe,i x2i ⩽ DKL (θ∥θe ) ⩽ θe,i x2i , min ,1 2 2θ θ̄e e i=1 i=1 which is exactly: nθ o 1 min , 1 ∥ log θ − log θe ∥2diag(θe ) ⩽ DKL (θ∥θe ) 2 θ̄e 1 ⩽ ∥ log θ − log θe ∥2diag(θe ) . 2θe Note that θ̄e = max n i θe,io⩾ 1/n while θ ⩽ 1/n, so θ/θ̄e ⩽ 1 and therefore min θ̄θe , 1 = θ̄θe , obtaining the lower constant c1 = θ/(2θ̄e ). This establishes (19) with c1 = θ/(2θ̄e ) and c2 = 1/(2θe ).