Generative Criticality in Large Language Model Temperature Scaling
Huajian Ruan1,2,3
Jinyang Li4,5,6
Xingyu Guo1,2,3,∗
Lingxiao Wang5,7,∗
1
arXiv:2606.06238v1 [cs.LG] 4 Jun 2026
State Key Laboratory of Nuclear Physics and Technology, Institute of Quantum Matter, South China Normal University, Guangzhou 510006, China 2 Key Laboratory of Atomic and Subatomic Structure and Quantum Control (MOE), Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Guangzhou 510006, China 3 Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nuclear Science, Guangzhou 510006, China 4 KEK Theory Center, Institute of Particle and Nuclear Studies, Tsukuba, Japan 5 RIKEN Center for Interdisciplinary Theoretical and Mathematical Sciences (iTHEMS), Wako, Saitama 351-0198, Japan 6 Graduate University for Advanced Studies (SOKENDAI), Oho 1-1, Tsukuba, Ibaraki 305-0801, Japan 7 Institute for Physics of Intelligence, The University of Tokyo, Hongo, Tokyo 113-0033, Japan [email protected], [email protected] *Corresponding authors
Abstract We propose a statistical-field framework for text generated by large language models (LLMs), treating token embeddings as continuous spin variables on a one-dimensional chain. Defining a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field, we vary the softmax temperature T and observe a sharp susceptibility peak near a characteristic Tc with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below Tc . The intrinsic dimension estimated by the two nearest neighbor (TwoNN) method independently corroborates these findings, reaching a minimum near Tc . Results are robust across model scales (Qwen3: 0.6B–32B) and prompt categories. While the phenomenology closely resembles a continuous phase transition, the nonequilibrium nature of autoregressive generation warrants further investigation. Our framework provides quantitative tools for probing the collective statistical structure of LLM outputs and suggests connections between decoding strategies and critical phenomena.
1
Introduction
Understanding the structure of language remains a central challenge in linguistics [1, 2] and artificial intelligence. While large language models (LLMs) have enabled text modeling at unprecedented scale [3], most analyses rely on information-theoretic quantities such as entropy and mutual information [4, 5, 6], which do not capture microstructural or emergent macroscopic behavior. Recent work has begun mapping LLM-generated texts to statistical field theories [7, 8], but rigorous definitions of the relevant physical quantities are still lacking. We address this gap by constructing a statistical-mechanics framework for LLM-generated text [9]. Within the token embedding space, we define susceptibility and an order parameter [10], and study Preprint.
their behavior as a function of the softmax temperature T [11, 12]. We find that these quantities exhibit critical behavior [13, 14]: near a critical temperature Tc , the susceptibility diverges and the order parameter undergoes a rapid change. To corroborate this from a geometric perspective, we apply the two nearest neighbor (TwoNN) intrinsic-dimension estimator [15, 16], which independently identifies the same critical region through non-monotonic features in Id (T ). Our main contributions are: (i) a statistical-field framework for LLM outputs with well-defined physical observables; (ii) evidence of critical behavior driven by the temperature parameter; and (iii) independent geometric validation via the TwoNN-estimated intrinsic dimension.
2
LLM Building Blocks and the O(N) Model
An LLM tokenizes input text into subword units via a fixed vocabulary [17, 18, 19], then maps each token to an N -dimensional vector through a trainable embedding matrix [18, 20]. Since token embeddings are high-dimensional vectors whose norms concentrate around a characteristic scale, this motivates a heuristic physical picture, treating the text sequence as a one-dimensional lattice chain of N -dimensional vectors [21, 10, 22], for constructing an effective Hamiltonian over the high-dimensional semantic space, H=
X
Jστ tσ tτ +
σ,τ
X
Hσ tσ ,
(1)
σ
where J is the coupling matrix (encoding non-local interactions between all pairs of lattice sites), H is an external field set by the prompt, and Greek indices label all different lattice sites. To simplify the problem, we omit the higher-order interaction terms, e.g., O(t2 ), but LLMs can still determine the effective couplings and thus governs how each site responds to H. Inter-token interactions are mediated by the self-attention mechanism [11]. Given query, key, and value projections Qi =FQ (ti ), Ki =FK (ti ), Vi =FV (ti ): p Attention(Q, K, V) = softmax QK T / dk V , (2) where dk is the dimension of the key vectors. Stacking M layers of FF ◦ Attention with residual connections yields the full Transformer [11]. AtP generation time, the next-token distribution is controlled by temperature T [23]: pi = exp(zi /T )/ j exp(zj /T ), interpolating between deterministic (T →0) and uniform (T →∞) sampling. Although T is not a thermodynamic temperature in the strict physical sense, its structural role in the softmax mirrors that of temperature in the Boltzmann distribution, and we treat this parallel as a formal correspondence.
3
Generative Criticality
We apply tools from statistical field theory and intrinsic dimension estimation to probe the critical structure of LLM outputs. Critical Behavior.
We define the susceptibility of generated text as, 1 Xh 1 X (i) (i) 1 X (i) X (j) i χ= tσ · t τ − 2 t · tτ , L σ,τ Ns i Ns i σ j
(3)
where L is the sequence length, Greek indices σ, τ label lattice sites (token positions), Latin indices i, j label members of an ensemble of size Ns , and the dot products are taken in the N -dimensional embedding space. Although this expression is formally analogous to a variance, the non-trivial content lies in the diverging fluctuations near a critical temperature Tc , where χ ∼ |T − Tc |−γ [24], consistent with standard thermodynamic scaling. Similarly, the ensemble-averaged token expectation, 1 X (i) ⟨t⟩ = t (4) Ns L i,σ σ serves as an order parameter [25]: in the high-temperature (disordered) phase it tends toward zero; in the low-temperature (ordered) phase, deterministic generation yields a nonzero ensemble average, analogous to the spontaneous breaking of O(N ) symmetry below Tc . 2
Intrinsic Dimension. The TwoNN method estimates the intrinsic dimension Id of a data manifold from local distance ratios [26]. For each point, defining µ = r2 /r1 , local uniformity implies f (µ) = Id µ−Id −1 , yielding − ln 1 − F (µ) = Id · ln µ , (5) where F (µ) is the cumulative distribution function. The intrinsic dimension acts as an unsupervised phase-transition detector [15]: Id is small in the ordered phase, approaches the total degrees of freedom in the disordered phase, and exhibits non-monotonic features at criticality.
4
Experiments and Main Results
We use the Qwen3 family [27] (0.6B–32B), fixing output length to 300 tokens. Prompts are drawn from English Wikipedia; controls include Chinese Wikipedia, jokes, poems, novels, and nonsensical texts. All generations use “no-think” mode (i.e., extended reasoning is disabled). For each temperature we generate Ns = 1,000 samples and compute the ensemble average over token positions to obtain the order parameter. It should be noted that the results we presented are embeddings in different dimensions(Table 1). Susceptibility. Figure 1(a) shows χ(T ) for different model scales; Figure 1(b) shows results across prompt categories. A pronounced peak appears near Tc , with power-law scaling χ ∼ (T − Tc )−γ and γ ≈ 0.1 on both sides (Figure 2. We also present the fitting results for different sizes in the appendix(Figure 7). The curves do not collapse across models or prompts, indicating that parameter count and prompt type act as distinct effective fields. Convergence with increasing ensemble size and sequence length is confirmed in Figure 3. 8
7
Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B
4
Susceptibility
Susceptibility
6
Qwen3-32B-wiki Qwen3-32B-wiki_Chinese Qwen3-32B-poem Qwen3-32B-joke Qwen3-32B-novel Qwen3-32B-nosence
6 5 4 3
2 0
2
4
6
Temperature
8
2 0
10
2
4
(a)
6
8
Temperature
10
(b)
Figure 1: Susceptibility χ vs. temperature T for (a) different model scales using Wikipedia prompts (1000 samples, 280 tokens). All models exhibit a peak near Tc ≈ 1.4, with larger models showing higher peak susceptibility. (b) Different prompt categories generated by Qwen3-32B (1000 samples, 300 tokens).
6 4 2
0
5
Temperature (a)
10
T < Tc A=8.422±9.024
T > Tc A=7.762±7.981 Data (T > Tc) Fit, =0.100 Linear fit slope=0.302
6 × 100 4 × 100 3 × 100 2 × 100
Data (T < Tc) Fit, =0.100 Linear fit slope=0.380
6 × 100
(log scale)
(dof = 51)
(log scale)
Data Fit region Fit, =0.100 Tc = 1.407
χν2 = 0.203
Susceptibility
8
4 × 100 3 × 100
10 2
10 1
T - Tc (log scale) (b)
10 1
Tc - T (log scale) (c)
Figure 2: Power-law fits of susceptibility near Tc for Qwen3-32B. (a) Full χ(T ) curve with the fit region highlighted; (b) log-log plot for T > Tc ; (c) log-log plot for T < Tc . Both sides yield a critical exponent γ ≈ 0.1, consistent with power-law divergence.
3
6
6
2 samples 100 samples 200 samples 500 samples 1000 samples
2
4
Susceptibility
Susceptibility
4
100 tokens 150 tokens 200 tokens 250 tokens 300 tokens
5 3 2 1
0 0
2
4
6
Temperature
8
10
0
2
(a)
4
6
Temperature
8
10
(b)
Figure 3: Stability analysis of the susceptibility signal. (a) Convergence with increasing ensemble size Ns from 2 to 1000 samples; (b) Convergence with increasing sequence length from 100 to 300 tokens. Order Parameter Structure. Applying PCA to the ensemble-averaged order-parameter vector across temperatures, we find that generated texts concentrate along a single semantic direction below Tc , with a sharp directional change at criticality (Figure 4). This confirms the phase transition and suggests that, at low temperatures, the LLM effectively selects tokens approximating a unique semantic target.
0
6
20
4
0.250 0.225 0.200
T=0.00 2 T=10.00 100 0
40 0
Principal Component Value
8
100 |<t>| PC1 PC2 50 PC3 0
0.275
|<t>|
20
Temperature
PC2 (10.7%)
10
0.175 0.150
PC1 (82.5%) (a)
50 0.0
2.5
5.0
Temperature
7.5
10.0
(b)
Figure 4: Order parameter structure via PCA. (a) Projection onto the first two principal components (PC1 explains 82.5%, PC2 explains 10.7%), colored by temperature. Below Tc , ensemble outputs cluster tightly along a single direction; above Tc , they disperse. (b) Magnitude |⟨t⟩| and the first three principal component values vs. temperature, showing a sharp transition near Tc . Intrinsic Dimension. We apply the TwoNN method to sentence-level embedding vectors (obtained via average pooling over token embeddings) of Ns = 1,000 texts at each temperature (Figure 5). The intrinsic dimension Id reaches a minimum near Tc and rises rapidly above it: at low T the configuration space is restricted; near Tc it enters a critical regime; at high T the system explores the full configuration space. Fits are excellent at all nonzero temperatures (see Appendix A).
5
Conclusion
We have presented a statistical-field framework for LLM-generated text, defining susceptibility and an order parameter in the token embedding space. Systematic variation of the softmax temperature reveals phase-transition-like behavior: a divergent susceptibility peak near Tc , a sharp change in the order parameter accompanied by dimensional collapse onto a single semantic direction, and a minimum in the TwoNN intrinsic dimension in the same critical region. These mutually consistent signatures validate the feasibility of statistical-field analysis for probing the collective structure of LLM outputs. Future directions include applying renormalization-group methods [28] to capture long4
250 Intrinsic Dimension
200 150 100 50 0
2
4
6
Temperature
Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B 8 10
Figure 5: Intrinsic dimension Id vs. temperature for five Qwen3 model scales. All models exhibit a minimum near Tc ≈ 1.3–1.5, after which Id rises sharply and saturates at high T , consistent with the onset of a disordered phase where the configuration space expands. range correlations, and leveraging the critical behavior identified here as a diagnostic for trustworthy AI—for instance, the sharp onset of the disordered phase may serve as a quantitative indicator of regimes in which LLM outputs become unreliable.
Acknowledgement We thank Drs. Kai Nakaishi, Jan Pawlowski, Enrico Rinaldi, Gabriele Di Ubaldo, and Sho Yokoi for helpful discussions. We thank the DEEP-IN working group at RIKEN-iTHEMS for support in the preparation of this paper. XG and HR is supported by the National Natural Science Foundation of China under Grant No. 12035007. LX is supported by JSPS KAKENHI Grant No. 25H01560, and JST-BOOST Grant No.JPMJBY24H9.
References [1] Steven Pinker. The language instinct: How the mind creates language. Penguin uK, 2003. [2] Noam Chomsky. Syntactic structures. Walter de Gruyter, 2002. [3] Melanie Mitchell and David C Krakauer. The debate over understanding in ai’s large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023. [4] Colin Scheibner, Lindsay M. Smith, and William Bialek. Large language models and the entropy of english. ArXiv, abs/2512.24969, 2025. [5] Ryosuke Takahira, Kumiko Tanaka-Ishii, and Łukasz D˛ebowski. Entropy rate estimates for natural language—a new extrapolation of compressed large-scale corpora. Entropy, 18(10):364, 2016. [6] Wentian Li. Mutual information functions of natural language texts. Santa Fe Institute Santa Fe, NM, USA, 1989. [7] Kai Nakaishi, Yoshihiko Nishikawa, and Koji Hukushima. Critical phase transition in large language models. arXiv preprint arXiv:2406.05335, 2024. [8] Youran Sun and Babak Haghighat. Phase transitions in large language models and the o(n) model. arXiv preprint arXiv:2501.16241, 2025. [9] Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4):045002, 2019. [10] Daniel J Amit and Victor Martin-Mayor. Field theory, the renormalization group, and critical phenomena: graphs to computers. World Scientific, 2005. 5
[11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [12] Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Zhi Jin, and Hong Mei. Hot or cold? adaptive temperature sampling for code generation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 437–445, 2024. [13] Henry W Lin and Max Tegmark. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299, 2017. [14] Cheng-Shang Chang. A simple explanation for the phase transition in large language models with list decoding. arXiv preprint arXiv:2303.13112, 2023. [15] T. Mendes-Santos, X. Turkeshi, M. Dalmonte, and Alex Rodriguez. Unsupervised learning universal critical behavior via the intrinsic dimension. Physical Review X, 11(1), February 2021. [16] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838, 2018. [17] Christopher M Bishop and Hugh Bishop. Deep learning: Foundations and concepts. Springer Nature, 2023. [18] Zhen Yang, Yingxue Zhang, Fandong Meng, and Jie Zhou. Teal: Tokenize and embed all for multi-modal large language models. arXiv preprint arXiv:2311.04589, 2023. [19] Sandeep Mehta, Darpan Shah, Ravindra Kulkarni, and Cornelia Caragea. Semantic tokenizer for enhanced natural language processing. arXiv preprint arXiv:2304.12404, 2023. [20] Roman Egger. Text representations and word embeddings: Vectorizing textual data. In Applied data science in tourism: Interdisciplinary approaches, methodologies, and applications, pages 335–361. Springer, 2022. [21] Sidney Coleman, Roman Jackiw, and HDavid Politzer. Spontaneous symmetry breaking in the o (n) model for large n. Physical Review D, 10(8):2491, 1974. [22] Bertrand Eynard and Charlotte Kristjansen. Exact solution of the o (n) model on a random lattice. Nuclear Physics B, 455(3):577–618, 1995. [23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [24] Leo P Kadanoff. Spin-spin correlations in the two-dimensional ising model. Il Nuovo Cimento B (1965-1970), 44(2):276–305, 1966. [25] Jean Zinn-Justin. Quantum field theory and critical phenomena, volume 171. Oxford university press, 2021. [26] Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140, 2017. [27] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [28] Kenneth G Wilson and John Kogut. The renormalization group and the ϵ expansion. Physics Reports, 12(2):75–199, 1974.
6
A
TwoNN Fitting
We present the data and fitting plot for the TwoNN method applied to Qwen3-32B in the appendix to show more details.
ln( )
0.1
00.0
-ln(1-F)
00.00 0.05 ln( )
T=1.438, d=87.48, R 2 =0.994
5
00.00
0.05
ln( )
5
-ln(1-F)
-ln(1-F)
5
00.00
Data Fit: d=111.61
0.05
ln( )
ln( )
10
ln( )
10
0.1
ln( )
7
0.1
-ln(1-F) -ln(1-F) -ln(1-F) -ln(1-F)
ln( )
0.1
ln( )
ln( )
00.0
00.0
ln( )
0.1
Data Fit: d=100.79
00.00
0.05
ln( )
T=1.562, d=115.34, R 2 =0.995
5
0.1
ln( )
T=1.488, d=100.79, R 2 =0.997
T=1.550, d=114.69, R 2 =0.930 Data Fit: d=114.69
0.1
Data Fit: d=79.20
5
0.1
ln( )
T=1.400, d=79.20, R 2 =0.993
5
Data Fit: d=85.16
10
Data Fit: d=59.57
00.0
T=1.462, d=85.16, R 2 =0.971
00.0
T=1.538, d=107.07, R 2 =0.972
00.0
0.1
0.1
ln( )
T=1.312, d=59.57, R 2 =0.998
5
Data Fit: d=67.09
00.0
0.1
Data Fit: d=59.79
00.0
T=1.387, d=67.09, R 2 =0.986
5
Data Fit: d=107.07
-ln(1-F)
-ln(1-F) -ln(1-F) -ln(1-F) -ln(1-F) -ln(1-F)
ln( )
Data Fit: d=88.85
00.0
ln( )
T=1.512, d=111.61, R 2 =0.998
00.0
0.1
T=1.450, d=88.85, R 2 =0.990
5
00.00 0.05
T=1.500, d=104.05, R 2 =0.998 Data Fit: d=104.05
Data Fit: d=87.48
-ln(1-F)
Data Fit: d=76.91
00.0
0.1
ln( )
0.1
0.1
ln( )
T=1.238, d=59.79, R 2 =0.995
5
Data Fit: d=54.33
5
Data Fit: d=67.28
ln( )
00.0
T=1.300, d=54.33, R 2 =0.988
T=1.363, d=67.28, R 2 =0.997
5
ln( )
ln( )
Data Fit: d=52.91
-ln(1-F)
5
T=1.413, d=76.91, R 2 =0.995
5
Data Fit: d=63.67
-ln(1-F)
-ln(1-F)
00.0
T=1.350, d=63.67, R 2 =0.993
0.1
0.1
ln( )
T=1.150, d=52.91, R 2 =0.991
5
Data Fit: d=56.26
00.0
0.1
T=1.288, d=56.47, R 2 =0.973
00.0
00.0
T=1.212, d=56.26, R 2 =0.997
5
Data Fit: d=57.92
-ln(1-F)
Data Fit: d=66.05
ln( )
Data Fit: d=56.47
0.1
ln( )
0.1
ln( )
T=1.062, d=57.92, R 2 =0.998
-ln(1-F)
T=1.337, d=66.05, R 2 =0.992
10
0.1
ln( )
00.0
0.1
-ln(1-F)
ln( )
5
Data Fit: d=58.44
00.0
00.0 0.1
-ln(1-F)
T=1.262, d=58.44, R 2 =0.996
5
00.0
0.1
00.0 5
Data Fit: d=55.42
5
Data Fit: d=54.25
0.1
ln( )
T=1.137, d=55.42, R 2 =0.990
T=1.200, d=54.25, R 2 =0.979
5
ln( )
00.0
0.1
ln( )
Data Fit: d=51.09
5
Data Fit: d=53.42
5
-ln(1-F)
-ln(1-F)
00.0
T=1.250, d=55.04, R 2 =0.962 Data Fit: d=55.04
ln( )
T=0.800, d=51.09, R 2 =0.993
T=1.050, d=53.42, R 2 =0.995
-ln(1-F)
ln( )
5
0.1
00.0
0.1
Data Fit: d=57.68
00.0
0.2
Data Fit: d=56.30
5
T=1.188, d=57.68, R 2 =0.997
-ln(1-F)
-ln(1-F)
ln( ) Data Fit: d=54.17
00.0
10
00.0
0.1
T=1.163, d=54.17, R 2 =0.998
-ln(1-F)
5
5
ln( )
T=1.113, d=56.30, R 2 =0.980
-ln(1-F)
00.0
Data Fit: d=54.40
-ln(1-F)
5
T=1.100, d=54.40, R 2 =0.996
-ln(1-F)
-ln(1-F)
Data Fit: d=57.77
-ln(1-F)
T=1.087, d=57.77, R 2 =0.988
00.0
0.1
Data Fit: d=44.26
5
Data Fit: d=53.58
5
ln( )
T=0.600, d=44.26, R 2 =0.995
T=1.038, d=53.58, R 2 =0.995
-ln(1-F)
00.0
0.1
ln( )
00.0
0.2
ln( ) Data Fit: d=54.75
-ln(1-F)
-ln(1-F)
00.0
5
T=1.012, d=54.75, R 2 =0.992
5
Data Fit: d=37.98
-ln(1-F)
Data Fit: d=55.40
T=0.400, d=37.98, R 2 =0.997
Data Fit: d=25.13
00.0
0.1
ln( )
T=1.000, d=55.40, R 2 =0.995
5
T=0.200, d=25.13, R 2 =0.988
-ln(1-F)
00.0
5
-ln(1-F)
Data Fit: d=27.80
-ln(1-F)
T=0.000, d=27.80, R 2 =-1.195
-ln(1-F)
5
00.00
Data Fit: d=115.34
0.05
ln( )
ln( )
5
0 0.000 0.025 ln( )
-ln(1-F)
-ln(1-F)
5
ln( )
0 0.025 0.000 ln( )
-ln(1-F)
-ln(1-F)
5
5
ln( )
T=3.000, d=196.84, R 2 =0.993
0 0.000
5
0.025
ln( )
0 0.000
T=6.000, d=194.37, R 2 =0.997
0 0.000
5
0.025
ln( )
0.025
Data Fit: d=204.39
0 0.000
ln( )
Data Fit: d=187.62
ln( )
0 0.000 0.025 ln( )
-ln(1-F)
-ln(1-F)
-ln(1-F)
-ln(1-F)
0 0.000
-ln(1-F)
5
0.025
ln( )
5
Data Fit: d=204.29
ln( )
ln( )
T=5.000, d=205.82, R 2 =0.997
5
0 0.000
ln( )
ln( )
T=10.000, d=188.78, R 2 =0.993
5
0.025
Data Fit: d=205.82
0 0.000 0.025
T=9.000, d=214.12, R 2 =0.991 Data Fit: d=214.12
Data Fit: d=193.89
0 0.000 0.025 5
0 0.000 0.025
0.025
ln( )
T=2.800, d=193.89, R 2 =0.997
T=4.500, d=204.29, R 2 =0.996
T=8.000, d=199.99, R 2 =0.996 Data Fit: d=199.99
-ln(1-F)
-ln(1-F)
0.025
ln( )
0 0.000 0.025 5
0.025
5
Data Fit: d=195.26
0.05
Data Fit: d=189.11
0 0.000
T=2.600, d=195.26, R 2 =0.994
T=4.000, d=187.62, R 2 =0.998
5
ln( )
ln( )
ln( )
T=1.988, d=189.11, R 2 =0.992
5
0 0.000 0.025
0.05
Data Fit: d=182.05
00.00
T=1.962, d=188.68, R 2 =0.997 Data Fit: d=188.68
ln( )
T=1.900, d=182.05, R 2 =0.986
5
ln( )
5
Data Fit: d=200.81
0 0.000
T=7.000, d=204.39, R 2 =0.995
-ln(1-F)
-ln(1-F)
5
Data Fit: d=194.37
Data Fit: d=206.47
Data Fit: d=190.77
Data Fit: d=181.59
0 0.000 0.025
T=2.400, d=200.81, R 2 =0.998
T=3.500, d=206.47, R 2 =0.995
-ln(1-F)
-ln(1-F)
5
Data Fit: d=196.84
ln( )
ln( )
5
0 0.000 0.025
0.025
Data Fit: d=182.01
00.00
T=1.887, d=181.59, R 2 =0.995
5
0 0.000 0.025
T=2.200, d=186.17, R 2 =0.999 Data Fit: d=186.17
ln( )
T=1.950, d=190.77, R 2 =0.993
5
0 0.000 0.025
T=2.000, d=195.95, R 2 =0.987 Data Fit: d=195.95
0 0.000
T=1.938, d=187.99, R 2 =0.992 Data Fit: d=187.99
Data Fit: d=181.09
ln( )
T=1.812, d=182.01, R 2 =0.992
5
0 0.000 0.025
T=1.863, d=181.09, R 2 =0.996
5
0 0.000 0.025
T=1.913, d=182.92, R 2 =0.992 Data Fit: d=182.92
Data Fit: d=183.40
0 0.025 0.000
-ln(1-F)
ln( )
ln( )
Data Fit: d=177.21
0.05
Data Fit: d=173.12
5
ln( )
ln( )
T=1.738, d=173.12, R 2 =0.995
-ln(1-F)
5
0 0.025 0.000
ln( )
Data Fit: d=162.38
T=1.800, d=177.21, R 2 =0.996
5
0 0.000 0.025
T=1.850, d=183.40, R 2 =0.997
-ln(1-F)
-ln(1-F)
5
Data Fit: d=181.41
0.025
Data Fit: d=187.68
ln( )
00.00
-ln(1-F)
T=1.837, d=181.41, R 2 =0.995
5
0.1
0 0.000 0.025
T=1.788, d=187.68, R 2 =0.993
Data Fit: d=147.07
5
-ln(1-F)
ln( )
0 0.000
5
ln( )
T=1.650, d=147.07, R 2 =0.997
T=1.712, d=162.38, R 2 =0.996
-ln(1-F)
0 0.000 0.025
Data Fit: d=169.86
Data Fit: d=162.29
-ln(1-F)
5
00.0
-ln(1-F)
Data Fit: d=176.36
ln( )
0 0.000 0.025
T=1.762, d=169.86, R 2 =0.988
-ln(1-F)
-ln(1-F)
5
0.05
-ln(1-F)
ln( )
T=1.750, d=176.36, R 2 =0.992
Data Fit: d=129.41
10
T=1.700, d=162.29, R 2 =0.998
5
0 0.000 0.025
T=1.637, d=129.41, R 2 =0.843
-ln(1-F)
ln( )
0.05
Data Fit: d=164.26
-ln(1-F)
00.00
5
-ln(1-F)
-ln(1-F)
5
T=1.688, d=164.26, R 2 =0.993
-ln(1-F)
Data Fit: d=148.14
00.00
-ln(1-F)
T=1.663, d=148.14, R 2 =0.993
ln( )
0.05
-ln(1-F)
ln( )
00.00
Data Fit: d=133.13
5
-ln(1-F)
0.05
T=1.613, d=133.13, R 2 =0.965
-ln(1-F)
00.00
Data Fit: d=127.92
5
-ln(1-F)
-ln(1-F)
5
T=1.600, d=127.92, R 2 =0.997
-ln(1-F)
Data Fit: d=129.17
-ln(1-F)
T=1.587, d=129.17, R 2 =0.993
Data Fit: d=188.78
0 0.000
0.025
ln( )
Figure 6: By evaluating the ratio of the second-nearest neighbor distance to the nearest neighbor distance among these data points and fitting its distribution, the derived slope yields the intrinsic dimension.
8
B
Configurations of Qwen3 models
We include a table to illustrate their varying semantic spaces across different embedding dimensions. Model
Hidden size
Layers
Attention heads (Q / KV)
Qwen3-0.6B 1024 28 16 / 8 Qwen3-1.7B 2048 28 16 / 8 Qwen3-4B 2560 36 32 / 8 Qwen3-8B 4096 36 32 / 8 Qwen3-14B 5120 48 40 / 8 Qwen3-32B 8192 64 64 / 8 Table 1: Architectural parameters of the Qwen3 series models.
Testing scaling laws at different sizes
Susceptibility
4 2 00
5
Temperature
T > Tc A=5.809±6.577
6 × 100 4 × 100 3 × 100 2 × 100
100
10
(a)
00
5
Temperature
10
Susceptibility
6 4 2 00
5
Temperature (g)
(c)
Data (T < Tc) Fit, =0.100 Linear fit slope=0.580
100
100
10 2
10 1
10 2
T - Tc (log scale)
10
(f)
T < Tc A=5.898±8.353
T > Tc A=11.045±15.010
101
100
10 1
Tc - T (log scale)
(e)
Data Fit region Fit, =0.100 Tc = 1.289
10 1
Tc - T (log scale)
T < Tc A=8.008±9.460
Data (T > Tc) Fit, =0.100 Linear fit slope=0.516
Data (T > Tc) Fit, =0.100 Linear fit slope=0.751
(log scale)
8 (dof = 42)
100
6 × 10 1
T > Tc A=8.327±9.595
(d) χν2 = 0.307
T - Tc (log scale)
Data (T < Tc) Fit, =0.100 Linear fit slope=0.582
(log scale)
2
10 1
(log scale)
4
10 2
4 × 100 3 × 100 2 × 100
(b)
Data Fit region Fit, =0.100 Tc = 1.343
(dof = 45)
Susceptibility
6
χν2 = 0.221
T < Tc A=4.874±5.567
Data (T > Tc) Fit, =0.100 Linear fit slope=0.438
(log scale)
Data Fit region Fit, =0.100 Tc = 1.405
2
ν
(dof = 48)
Data (T < Tc) Fit, =0.100 Linear fit slope=0.446
(log scale)
6 χ = 0.097
(log scale)
C
100
10 1
T - Tc (log scale) (h)
10 2
10 1
Tc - T (log scale) (i)
Figure 7: Scaling laws tests at different sizes. Panels (a), (b), and (c) show exponent fitting for a system size of 200 tokens; panels (d), (e), and (f) for 400 tokens; panels (g), (h), and (i) for 500 tokens.
9