Research Article
Sky sphere representation in language models Aleksandr Berdnikov*1,2 , Yevgeny Liokumovich3 1
Fields Institute 2 Principles of Intelligence 3 University of Toronto 2026
arXiv:2607.27092v1 [cs.LG] 29 Jul 2026
Abstract We analyze whether language models of size ∼100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-source models do have such a representation, and it often even surfaces to the top principal components on prompts that ask questions like “what is close to this object in the night sky”. In all but one model this representation showed significant scores in LOO testing, containing up to 65-85% of variance (R2 -score) and having median angular error down to 12◦ − 21◦ . We verify that our representation is not a simple leak from a correlated flat representation. To our knowledge, this representation is the first example of a curved high-dimensional irreducible feature manifold. Codes used in the paper are published at github.com/l3erdnik/Decodable-sky keywords AI, Mechanistic Interpretability, Feature Manifold.
1.
INTRODUCTION
Mechanistic interpretability tools can be roughly divided into two groups: linear methods, such as linear probes (Alain and Bengio, 2016; Belinkov, 2022) and sparse autoencoders (SAEs) (Bricken et al., 2023; Cunningham et al., 2024), and methods that analyze activations with deep learning systems, such as natural language autoencoders (NLAs) (Fraser-Taliente et al., 2026) and activation oracles (Karvonen et al., 2025). The downside of the second group is opaqueness. The downside of the first is that linear methods miss important non-linear structures present in the representation space. Of particular interest, therefore, are feature manifolds: low-dimensional submanifolds of the activation space that correspond to a specific concept. Feature manifolds offer a middle ground between the two groups of tools: they capture non-linear structure while remaining explicit and human-inspectable. For example, when asked “what day is 5 days after Friday?”, LLMs represent the days of the week (and, likewise, the months of the year) on a circle and solve this modular arithmetic problem by rotating it (Engels et al., 2025). Similarly, when
Figure 1. Representation of selected 38 night sky objects in the residual stream of Mistral Large 2 LLM at layer 66. Each color corresponds to a single object point cloud.
deciding where to break a line in fixed-width text, a model tracks the running character count on a curved one-dimensional manifold, which attention heads then twist to compare against the line width (Gurnee et al., 2026). Both are examples of one-dimensional feature manifolds. In this paper we demonstrate the existence of a spherical feature manifold in the activation space
* Corresponding Author: Aleksandr Berdnikov <[email protected]>
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
of LLMs, corresponding to the positions of stars and constellations in the night sky (Figures 1 and 2). Our prompts mention the objects only by their proper names and avoid mentioning any coordinates explicitly, but rather ask what is close to the object in the sky. Thus, a spherical arrangement in activation space is likely not a trivial echo of coordinate strings seen in training, but rather a reflection of the notion of proximity for the night sky objects. Across seven open-source models of 32B–235B parameters, we find that the position of an object on the celestial sphere is linearly decodable from the top principal components of the residual stream on proximity focused prompts: in all but one model, leave-one-out validation yields R2 scores of up to 65–85% and median angular errors down to 12◦ –21◦ , and the representation rises into the top ∼8 principal components. Previously discovered geometric representations in language models have been either flat (continents and USA states in (Gurnee and Tegmark, 2024), color space in (Abdou et al., 2021)) or one-dimensional (months and historical years in (Engels et al., 2025) and (Karkada et al., 2026), sentence length in (Gurnee et al., 2026)]). To our knowledge, this is the first example of a high-dimensional non-flat irreducible feature manifold (that is, is not a product of 1-dimensional ones, as defined in (Engels et al., 2025)). 2. 2.1.
Figure 2. Constellation regions in stereographic projections and corresponding point clouds from Mistral Large 2 layer 66 residual stream. Northern hemisphere on top.
METHODOLOGY AND DATA PCA components that carry the best fit (see Appendix D).
General pipeline
We feed various prompts (from a fixed list) about night sky objects into an LLM in auto-complete mode (no thinking), and extract top 128 principal components of the distribution of the residual stream vectors in every layer. We find linear regression fit (using top n = 4, 8 or 128 PC-components) to the true coordinates of the objects on the unit sphere of the night sky. For validation on object X it is left out of the data for the regression fit. We then record the accuracy of how well it predicts the unseen object X across the whole dataset: the R2 score (variance explained by the regression model) and median angular error. We also record the
2.2.
Input Data
We use 188 astronomical objects of 3 types: 85 brightest stars, all 88 official constellations and 15 arbitrary miscellaneous objects like prominent galaxies, nebula and clusters. The coordinates of the constellations are defined as their area centroids (projected back on the unit sphere) — a choice that is somewhat arbitrary, but, we argue, inconsequential to our findings. For more, see Appendix B. We feed these objects into LLMs wrapped in 25 prompts of the following type: 2
Sky sphere representation in language models
1
Mistral Large 2
0.9
GLM-4.5-Air
0.8
Qwen3-235B
0.7
Qwen3-32B
0.6
0
10
20
30
40
50
60
70
80 0 10 20 30 40 50 60 70 80 90
Mistral Large 2 R2
0.5 0.4
Mixtral-8x22B
0.3
0
0.2 0.1
PCA
0 0 10
GLM-4.5-Air
20
Qwen3-235B
30
Qwen3-32B
40
Mixtral-8x22B gpt-oss-120b
50
0
80
20
30
40
50
60
70
80
layer 1
0
10
20
30
40
50
60
70
80 0 10 20 30 40 50 60 70 80 90
Llama-3.3-70B
60 70
10
128
Mistral Large 2
Llama-3.3-70B
R
2
90 0
α
4 8
Figure 3. R2 -score (top) and median angular error (bottom) as a function of normalized layer depth for predicting the coordinates on the night sky of an object from residual stream (top-8 PCA) using linear regression on stream data from other objects
PCA
0
10
20
30
40
50
60
70
80
128
• the closest constellation to X is • in star atlases X lies close to • stargazers spot X right beside
layer Figure 4. R2 -score (black) and median angular error (gray) in LOO validation for sky fitting when using top-8 (thin) and top-4 (thick) PCA components. Colored charts show which PC’s correlate with the optimal projection (blue&purple) and with the object type (yellow)
where X is replaced by a given object, so the whole dataset is of size 25 × 188 = 4700. The prompts focus on the apparent proximity on the night sky but avoid explicit mentions of coordinates in any way. For the whole prompt set and more details see Appendix C 2.3.
α
4 8
Llama-3.3-70B gpt-oss-120b
1
3. 3.1.
RESULTS LLMs comparison
Figure 3 shows significant decodability of the position on the celestial sphere form residual stream in most tested models. Note that the poor performance of gpt-oss in this test does not reflect poorly on its actual abilities: when asked directly for stellar coordinates, it locates over 70% of objects to within 1◦ , similar to other models with thinking in our set, see Appendix D.3.
Models
We used the following open-souse models in our investigation: • Qwen3-32B • Qwen3-235B • gpt-oss-120b • Llama-3.3 • Mixtral-8x22B • Mistral Large 2 • GLM-4.5-Air
3.2.
Representation prominence in PCA
Figure 4 shows that the fitting embedding of the celestial sphere for small layer depth is weak and spread along many PCA components (light and purple), but as the depth increases it strengthens and rises to top∼8 PCA components (dark and blue). In many models it is only surpassed by the direction (yellow) that separates stars from
For more information on those see Table 1. 3
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
1
0
10
20
30
40
50
60
70
80
model dimension separately. We find that the distinctive discontinuous dimension of the flat 2D chart comes with 0 (or even < 0) coefficient in top-8 PCA space (e.g. Figure 6), which excludes the possibility of the flat chart representation there (see Appendix E.1).
1
Mistral Large 2 R
2
0
0
Another alternative representation conceivable for the stars is a genuine 3D-representation. 2 However, since distances to the stars alone span Figure 5. R -score in LOO validation for sky fitting on generic prompts (see Table 4) compared to the top-8 fit for proximity-focused prompts ∼ 3 orders of magnitude (in our set), the (gray shade). The signal is almost 0 in top-8 components (thick) and is radial component, if present, would have to be weak and decaying in top-32 (thin). non-linear. We analyze the radial component, 0 10 20 30 40 50 60 70 80 but don’t find a significant (or even consistent 2 across models) correlation with the distance to Mistral Large 2 the star (Appendix E.2). This rules out our 1 representation being secretly a 3D Euclidean Σβ one (even with adjusted, but still clear radial dependence). We find, however, that the radial 0 component is influenced by other factors, most 0 10 20 30 40 50 60 70 80 notably, the type of the object (constellations are layer placed further, stars are placed closer), and, to Figure 6. Weights of various dimensions of a spherical (yellow) and flat a lesser extent, by its frequency in texts and, 2D (blue) models in RDM analysis of top-8 PCA space. A particular pair possibly, brightness (Figure 11). of dimensions, that are highly correlated between the two models, makes 0
10
20
30
40
50
60
70
80
layer
their attribution ambiguous. If they are excluded from both models (bottom thick lines), the spherical model stays strong, but the 2D map drops to 0 and below: its discontinuous “longitudal” coordinate is not observed on its own.
4.
We only analyzed the emergence of the representation in the residual stream. That leaves several aspects unclear. Is the mechanism that constructs the representation more interpretable than pure memorization? Are the directions of the representation tied to meaningful unembeddings? Given that the models know celestial coordinates way better, than the accuracy of our representation (see Figure 8a), what is its role for the models? Our representation was extracted with proximity-focused prompts, reasoning that they could fish out “textual proximity”, that, hopefully, is heavily influenced by proximity on the night sky. Similar logic has been used in (Karkada et al., 2026) to explain geometric structures in activation spaces by the metric derived from co-occurence in the training data. How well is our example explained by this paradigm?
constellations and leaves the sky dome mostly in PCs 2-4 (see Appendix F). We note that the prominence and clarity of this signal relies on our prompts being focused on night sky proximity. Figure 5 shows that on more generic prompts the signal weakens and sinks to lower PCA ranks. Similar charts for other models and their technical details are in Appendix D. 3.3.
OPEN QUESTIONS
Competing representations
The unit sphere representation of the night sky in not the only possible one. In (Gurnee and Tegmark, 2024) the model used for the continents and USA states was not globe-based, but a flat map with altitude and latitude as coordinates. Similarly, the sky may be represented as a flat 2D-chart instead of as a unit sphere. Separating the two is not entirely straightforward, since some of their dimensions are highly correlated. We use a modified RDM-analysis by measuring how much the metric on test objects (that we observe in the residual stream) is attributable to each
5.
STATEMENT ON AI USE
AI agent Claude Opus 4.8 was used to code and analyze experiments. 4
Sky sphere representation in language models
6.
2025. URL https://arxiv.org/abs/ 2405.14860.
ACKNOWLEDGEMENTS
This work would not be possible without cluster access generously offered by Cursor (https://cursor.com/).
Kit Fraser-Taliente, Subhash Kantamneni, Euan Ong, Dan Mossing, Christina Lu, Paul C Bogdan, et al. Natural language autoencoders produce unsupervised explanations of LLM activations. Transformer Circuits Thread, 2026. URL https://transformer-circuits.pub/ 2026/nla/index.html.
We are grateful to Arul Shankar and Jacob Tsimerman for valuable discussions and encouragement. A.B.’s research was supported by Principles of Intelligence (https://princint.ai/).
GLM-4.5 Team. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/abs/2508. 06471.
References Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. In Proceedings of the 25th conference on computational natural language learning, pages 109–132, 2021.
Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations, volume 2024, pages 2483–2503, 2024. Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task. arXiv preprint arXiv:2601.04480, 2026.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.
Albert Q. Jiang et al. Mixtral of experts, 2024. URL https://arxiv.org/abs/ 2401.04088.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/ 2023/monosemantic-features.
Dhruva Karkada, Daniel J Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Symmetry in language statistics shapes the geometry of model representations. arXiv preprint arXiv:2602.15029, 2026. Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, and Samuel Marks. Activation oracles: Training and evaluating LLMs as general-purpose activation explainers. arXiv preprint arXiv:2512.15674, 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/ 2309.08600.
Dominik Kundel. Openai harmony response format. URL https: //developers.openai.com/cookbook/ articles/openai-harmony.
Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. In International Conference on Learning Representations,
Llama Team, AI @ Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/ abs/2407.21783. 5
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
Mistral AI. Mistral large 2 (instruct, 2411). https://huggingface.co/mistralai/ Mistral-Large-Instruct-2411, 2024a. Blog: https://mistral.ai/ news/mistral-large-2407/. Accessed 2026-07-16. Mistral AI. Cheaper, better, faster, stronger: Mixtral 8x22b. https://mistral.ai/ news/mixtral-8x22b/, 2024b. Model: https://huggingface.co/mistralai/ Mixtral-8x22B-Instruct-v0.1. Accessed 2026-07-16. OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/ 2508.10925. Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
6
Sky sphere representation in language models
A.
MODELS
Model huggingface repo used Qwen3-32B, (Qwen Team, 2025) Qwen/Qwen3-32B 9216db5 Qwen3-235B, (Qwen Team, 2025) Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 f07f63f gpt-oss-120b, (OpenAI, 2025) openai/gpt-oss-120b b5c939d Llama-3.3, (Llama Team, AI @ Meta, 2024) unsloth/Llama-3.3-70B-Instruct (unsloth mirror) 99cd0d2 Mixtral-8x22B, (Mistral AI, 2024b), (Jiang et al., 2024), mistralai/Mixtral-8x22B-Instruct-v0.1 cc88a6c Mistral Large 2, (Mistral AI, 2024a) mistralai/Mistral-Large-Instruct-2411 ba78820 GLM-4.5-Air, (GLM-4.5 Team, 2025) zai-org/GLM-4.5-Air a24ceef
Type
Size
Layers
Dense
32B tot
64
MoE
235B tot 22B act 117B tot 5B act 70B tot
94
141B tot 39B act 123B tot
56
106B tot 12B act
46
MoE Dense MoE Dense MoE
36 80
88
Table 1. Models used in this work
Table 1 lists the open-source models used for most of tests in this work. Weights were loaded with transformers via AutoModelForCausalLM.from pretrained(repo, revision=<commit>). Each revision was the repository’s main HEAD at access time (July 2026). B.
OBJECTS
Our test set consists of 3 groups. The first contains the 85 stars of apparent magnitude 2.5 or brighter under their proper names, which, together with their coordinates, are taken from IAU WGSN catalog. The second group has 15 miscellaneous, arbitrarily chosen night sky objects like prominent galaxies, nebula and clusters (see Table 2), whose coordinates are pulled via astropy’s SkyCoord.from name method. GALAXIES Andromeda Galaxy Triangulum Galaxy Whirlpool Galaxy Sombrero Galaxy Large Magellanic Cloud
NEBULAS Orion Nebula Crab Nebula Ring Nebula Lagoon Nebula Carina Nebula Tarantula Nebula
CLUSTERS Pleiades Hercules Cluster Omega Centauri Beehive Cluster
Table 2. Miscellaneous objects included in this study
Lastly, we often include 88 official constellations. For purposes of our analysis we define and compute the “location” of constellations on the night sky as their area centroids: ∼ 2 · 105 points are sampled uniformly over the unit sphere, those that fall within the constellation (determined by astropy’s get constellation) are averaged, and the result is projected back onto the unit sphere. We have two arguments to excuse the arbitrary nature of this choice. The first is that successful models in our tests don’t perform worse on constellations compared to objects with less subjective point representatives (this analysis is not included), so the data couldn’t have been spoiled too much by our choice of designation. The second argument is that in our tests we got median angular error to be 7
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
12◦ − 21◦ at best, so this noise would likely hide any imperfections of our choice anyway, given that the typical radius of a constellation is 10◦ − 15◦ , so any reasonable ways to pick a center would differ by a much smaller amount. C.
PROMPTS
The prompts we used for most tests are listed in Table 3. The set of prompts was generated by a combination of composing, weeding out and adjusting by hand, and using Claude to expand existing sets. Originally, the prompts were more diverse, but the better performance was shown if • the prompt asked about the nearest vicinity (not more general geometric relations) • it made clear that it talks about specifically the night, in some way. Thus, only prompts of this type made the final list. in the night sky X is not far from in the night sky X shares space with in the night sky X is a short hop from in the sky charts X is right next to in the night sky X is close beside astrophotographers find X by first locating in the night sky X is right next to the neighboring constellation to X is the closest constellation to X is
on the celestial sphere X is near in star atlases X lies close to in the night sky X appears near on star charts X sits beside sky watchers locate X next to in sky atlases X is adjacent to the brightest star near X is stargazers spot X right beside
sky atlases place X close to on sky maps X appears beside in sky atlases X neighbors on star maps X lies beside sky charts show X beside nearby in the sky to X is in the night sky X borders sky maps place X near
Table 3. Default list of prompt templates
To show the significance of prompt phrasing being focused on night sky proximity, we run the same test on a more generic set of prompts listed in Table 4. The results (Appendix D.4) show that, in this case, the representation appears weaker, never touches the top-8 PCA components (despite residing there entirely under default prompts), and decays quicker. the age of X is estimated to be the discovery of X is due to in mythology, X represents X was first recognized by the name of X comes from last time X was seen in X was first observed by
X is X emits X is known for X is studied by X is believed to X was named after
X was cataloged by X is classified as X is described in X is mentioned by brightness of X is X is an example of
the magnitude of X is X is composed mainly of the temperature of X is the luminosity of X is astronomers use X for X is important for
Table 4. Generic prompts
D.
CHARTS
Here we explain the analysis summarized in Figures 4 and 5 and provide them for other models. All 25 × 188 prompts are fed into the model in the auto-completion (no thinking) regime and for each layer we record the top 128 PCA components of the residual stream on the last token of the prompt. D.1.
Plots
The presence of the sky sphere representation is validated with the following leave-one-out (LOO) analysis. The model prediction for an object X is set via linear regression (using all objects but X) of top-4 or top-8 PCA space onto the “true” night sky globe space — R3 where all objects, are placed on the unit sphere S 2 ⊂ R3 according to their true celestial coordinates. The accuracy of that prediction is measured with R2 -score (variance explained by the model) and median angular error α. 8
Sky sphere representation in language models
1
0
10
20
30
40 0 10 20 30 40 50 60 70 80 90
GLM-4.5-Air R
2
0
1
0
10
20
30
40
50
60
70
80
90 0 10 20 30 40 50 60 70 80 90
Qwen3-235B α
R
2
0
4 8
4 8
30
40
0
10
20
30
40
layer 1
0
10
20
30
40
60
70
80
90
layer
50
60
70
80 0 10 20 30 40 50 60 70 80 90
Mistral Large 2 R
50
2
0
1
0
10
20
30
40
50
60 0 10 20 30 40 50 60 70 80 90
Qwen3-32B α
R
2
0
40
50
60
70
80
0
10
20
30
layer 1
0
10
20
50
60
layer
30
40
50 0 10 20 30 40 50 60 70 80 90
Mixtral-8x22B R
40
2
0
1
0
10
20
30 0 10 20 30 40 50 60 70 80 90
gpt-oss-120b α
R
2
0
40
50
0
10
20
layer 1
0
10
20
30
40
layer 50
60
70
80 0 10 20 30 40 50 60 70 80 90
Llama-3.3-70B R
30
2
0
1
0
10
gpt-oss-120b α
R
20
30 0 10 20 30 40 50 60 70 80 90
*
2
0
40
50
60
70
80
0
layer
10
20
30
128
30
128
20
PCA
PCA
10
α
4 8
4 8
0
128
30
128
20
PCA
PCA
10
α
4 8
4 8
0
128
30
128
20
PCA
PCA
10
α
4 8
4 8
0
128
20
128
10
PCA
PCA
0
α
layer
Figure 7. R2 -score (black) and median angular error (gray) for top-8 (thin) and top-4 (thick) PCA components for a sky sphere fit. Colored charts show which PC’s correlate with the optimal projection (blue&purple) and with the object type (yellow) (see details in text Appendix D). The last gpt-oss-120b∗ chart for a special test for this outlier model, see Appendix D.3
9
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
D.2.
Density PCA charts
Below the plots we show charts that illustrate the location of the sky sphere representation within PCA directions (blue and purple). Specifically, we center and whiten the sky data (so that their coordinates ck are distributed with mean 0 and unit variance I3 ), and record X s(l, i) = Cov2 (ck , (wp)li ) 1≤k≤3
where (wp)li is the i-th principal component p of l-th layer scaled by a weight w that either uniformly scales the layer so that the top variance is 1, or whitens the data so that every componentp has unit variance. The first version sets the red channel and the second sets the green via formula 1 − s(l, i). The blue channel is defined like the green one (with whitened PCA), but instead of correlation with celestial coordinates ck we correlate (wp)li to the “type” function that is 1 if the object X in the prompt is a star, -1 if X is a constellation, and 0 if X is neither. That means that the colors on the chart imply the following: Color Dark blue Light blue Dark purple Light purple Solid Yellow Light yellow White
Correlation with Sky Sphere, Strong Sky Sphere, Weak Sky Sphere, Strong Sky Sphere, Weak Object Type, Strong Object Type, Strong Nothing
PCA strength Strong Strong Weak Weak
Table 5. Meaning of colors in charts like Figure 7
D.3.
gpt-oss Anomaly
Figures 3 and 7 show that in gpt-oss-120b the sky sphere is much less decodable than in models of comparable size. We note that it doesn’t mean that “it is way worse at knowing celestial coordinates”; in fact, all models with thinking in our test managed to locate most non-constellation objects to within 1◦ (Figure 8a) — way better than 10 − 20◦ error we get in our tests. This leaves many questions regarding the role of the representation we found in model’s thinking, but also leaves its absence in gpt-oss unexplained. One way in which it differs from other models is that it was trained on the harmony response format (see (Kundel)), rather than raw text, so our raw prompts were out of distribution in that regard. However, when we wrapped prompts in the appropriate format, the decodability representation dropped even lower (Figure 8b, same as last chart in Figure 7) .001° 1
.01°
0
10
gpt-oss-120b
Qwen3-235B ang_error
.1°
20
30 0 10 20 30 40 50 60 70 80 90
*
R2
GLM-4.5-Air 1° gpt-oss-120b
0
Qwen3-32B 10
20
30
80
90
100 0
(a) Accuracy on prompts “Give the right ascension and declination of X with no commentary.”, log of angular error sorted in increasing order (results closer than .001◦ not shown). Some curves don’t span the whole 100 tested non-constellation objects since on some obscure stars models have ran out of thinking budget, eliminating them from the pool.
Figure 8
10
10
20
30
layer (b) Chart from Figure 7 for gpt-oss on prompts with proper harmony response format.
128
40 50 60 70 rank of the object
PCA
90° 180°
4 8
10°
α
Sky sphere representation in language models
D.4.
Charts for proximity-focused vs generic prompts
Here we illustrate the effect that the phrasing of the prompts has on decodability of sky sphere representation. Note that the apples-to-apples comparison is to be done between the gray shade graph (sky proximity prompts) and thick black line (generic prompts), both being restricted to the same top-8 PCA space. In this case the signal for generic prompts stays essentially at 0. The R2 -score only rises to the thin black line when the domain is expanded to top-32 PCA space, and even then it is far from the results on the focused prompts, and decays in later levels. 1
0
10
20
30
40
1
1
0
GLM-4.5-Air R
R
0
0 10
20
30
40
50
60
70
80
90
40
50
60
70
80
90
20
30
2
0
40
0 0
10
20
30
layer 1
0
10
20
30
40
layer 50
60
70
80
1
1
0
Mistral Large 2 R
R
0
0 10
20
30
10
20
0
10
20
40
50
60
70
30
40
50
10
20
1
1
0 30
40
0
10
50
60
0
10
20
30
40
50
60
70
80
40
50
60
70
80
1
2
0
50
0 0
layer 1
40
Llama-3.3-70B
0 20
30
1
layer
R
10
60
0 0
2
0
50
0
80
Mixtral-8x22B R
40
2
layer 1
30
Qwen3-32B
2
0
1
Qwen3-235B
2
0
10
10
20
30
layer
20
30
20
30
1
gpt-oss-120b R
2
0
0 0
10
layer Figure 9. R2 LOO scores for decodability of position on the sky sphere from residual stream of prompts focused on sky proximity (gray shade, top-8 PCA only) and generic prompts (thick black - on top-8 PCA components, almost 0; thin black - on top-32 PCA components).
11
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
E.
TESTING ALTERNATIVE REPRESENTATIONS
We argue that the representation we found is indeed based on a spherical feature manifold, rather than a leak from a conceptually different but correlated representation. To that end, we pit our model against two other possible representations of the night sky objects. We present evidence that they are essentially absent from the top-8 PCA where the spherical representation resides. E.1.
Flat chart
Since the location on the night sky is typically documented as RA/Dec1 coordinates, that are night sky equivalents of longitude and latitude, one might expect a representation that is linear in those coordinates (a flat 2-D map). This is the representation analyzed in (Gurnee and Tegmark, 2024) for the case of continents on the globe and states of USA. It is not trivial to separate this model from the spherical one, since they are highly correlated2 . To distinguish them, we use RDM analysis (compare the distances imposed by each model to those observed in the data). In short, we see that the RA-coordinate (the most distinctive feature of RA/Dec representation) is absent in top-8 PCA (bottom blue line in Figure 10 is ≤ 0). Here are the details. We exclude from the pool the constellations that cross the RA = 0◦ ≡ 360◦ discontinuity. For each layer l we take top-8 PCA space of the residual stream3 , scale it to make the top variance be 1, and compute the RDM matrix M defined via Mij = |vi − vj |2 where vi is the vector representing object i (averaged over prompt templates). We use the square of the distance so that such RDM would be additive for direct sums of representations, which we rely on. We perform a linear regression M = βx Mx + βy My + βz Mz + βRA MRA + βDec MDec + η (1) where η is the minimized error, Mχ is the RDM matrix defined similarly for one of the coordinates (normalized to variance 1), and βχ is the corresponding parameter.
x = sin(Dec)
Sphere model y = cos(Dec) sin(RA)
z = cos(Dec) cos(RA)
2D flat chart RA Dec
Table 6. Coordinate components of tested models
Since x and Dec are almost identical, the split between βx and βDec is numerically unstable and meaningless in practice. To circumvent this issue, we only use them merged as βx^ := βx + βDec Dec and record the total β score for each model with and without this merged term. The results (Figure 10) clearly show that without the inseparable βx^ , the flat model is more than non-existent: βRA ≤ 0 Dec (lower thick blue line). On the other hand, the spherical model is prominent even without its βDec component (thick yellow line). This shows that regardless of which model we attribute the ambiguous ^ altitude component x Dec, the flat 2D representation cannot be present in the data, since otherwise we would see its unique feature — the discontinuous latitude RA, — and we do not.
1 Right Ascension and Declination. 2 Dec is almost identical to x = sin (Dec) of the sphere and RA is correlated with y = sin(RA) cos(Dec) of the sphere. 3 Fitting linearly the RA/Dec model mostly put it aligned with the sphere representation at the top, so we tested whether RA/Dec shares the space with it in the top rather than sitting separately in lower components.
12
Sky sphere representation in language models
0
10
20
30
40
0
10
20
30
40
50
60
70
80
90
2
2
GLM-4.5-Air
Qwen3-235B 1
1
Σβ
Σβ
0 0
10
20
30
0
40
0
10
20
30
40
layer 0
10
20
30
40
50
60
70
80
90
layer 50
60
70
80
0
10
20
30
40
50
60
2
2
Mistral Large 2
0
10
20
30
Qwen3-32B
40
50
60
70
1
1
Σβ
Σβ
0
0
80
0
10
20
30
layer 0
10
20
30
40
50
60
layer 40
50
0
10
20
30
40
50
60
70
80
2
2
Mixtral-8x22B
Llama-3.3-70B 1
1
Σβ
Σβ
0 0
10
20
30
40
0
50
0
layer 0
10
20
10
20
30
40
50
60
70
80
layer 30 2
gpt-oss-120b 1
Σβ 0 0
10
20
30
layer Figure 10. Prominence (sums of β’s in equation 1, Σβ) of the spherical (yellow) and flat 2D (blue) sky representations in top-8 PCA space of the residual stream, either excluding both βx and βDec (thick bottom lines) or including them both (thin top lines).
13
Aleksandr Berdnikov, Yevgeny Liokumovich (2026)
E.2.
3D chart, radial analysis
Another conceivable alternative representation is something akin to a genuine 3D map of the celestial objects that properly accounts for the distance to them rather than projecting them onto a unit sphere. However, expecting a literal undisturbed 3D map in this case is unreasonable, since the distances to the stars in our set span ∼ 3 orders of magnitude, including galaxies adds several more, and constellations just don’t have a meaningful notion of distance to them. Even so, it raises the question of whether the radial direction in the observed representation does encode the distance to the object in some warped way. And if not — whether it encodes or is influenced by some other factors. Figure 11 shows the correlations of the radial direction with various parameters, with black bars indicating significance of the correlation (p = 0.05 corresponds to half-height bar). We see that (Spearman) correlations with distance (dist∗S ) are weak and disparate with low significance (only Llama barely climbs past p = 0.05). Given the textual origin of the data, the textual frequency (zipf) is the next factor that we test against, with more pronounced results, especially for constellations (zipf □ ) rather than stars (zipf ∗ ), placing more mentioned objects slightly closer. Brightness is a parameter that is both physical and reflecting how noticeable it is for people. We see mostly positive, but weak and low significance correlations of radial direction with brightness (-mag, negative apparent magnitude). The only really robust correlations we found were with the object type, placing constellations farther (I(□) > 0) and stars closer (I(∗) > 0). level
α
Qwen3-235B
88
12.0
Mistral Large 2
65
12.7
GLM-4.5-Air
42
13.4
Qwen3-32B
61
15.9
Llama-3.3-70B
64
17.9
Mixtral-8x22B
51
21.5
distS *
zipf *
zipf
-mag
*
I()
I(*)
I (x )
1
0
-1
Figure 11. Correlations of radial direction with various functions (see Table 7) at the level with least median angular error α. Black bars represent significance of the correlation, height is 1 − pln(.5)/ ln(.05) (p = 0.05 gives height 0.5).
To do this analysis, we pick the level with the best median angular error α in LOO analysis on top-8 PCA subspace (computed as in Appendix D), and discard from the pool the objects whose MSE for the angle is more than double of the average, to eliminate outliers and get a cleaner signal. We use linear regression to project the top-8 PCA space onto the 3-dimensional space that hosts the unit sphere sky model. For the remaining objects we compute correlations of the length of their projection with the following functions listed in Table 7. Label dist∗S zipf∗ zipf□ −mag∗ I(□) I(∗) I(x)
Function correlated Distance, from SIMBAD trigonometric parallax; Spearman correlation Log frequency in text corpus, from wordfreq.zipf frequency Log frequency. . . , “ constellation” stripped Brightness as negative apparent magnitude, from SIMBAD Indicator function of “constellation” type Indicator function of “star” type Indicator function of “other” type Table 7. Functions correlated with radial length.
14
Domain Stars only Stars only Constellations only Stars only All All All
Sky sphere representation in language models
F.
PCA COMPONENTS AHEAD OF SKY SPHERE
In many models the sky sphere representation rises to the very top of PCA components (blue in Figure 7) on proximity focused prompts. However, even in the best examples it falls short from crystallizing as the top-3 components (see for example top-8 PCA of Mistral model, Figure 12). This is mostly due to the direction (yellow) that separates concepts “star” and “constellation” prevalent in our test data (see Appendix D for the definition of the yellow direction). Another way to analyze the competitor direction is to pick the top-4 components for a layer, where they are saturated with the sky sphere model, like level 66 in Mistral model, and isolate the direction orthogonal to the sky model (fig. 13a). The prompts whose object X is a constellation (blue) land on one side and those where X is a star (red) land on the other, with miscellaneous objects (black) spread in between. The three notable exceptions (highlighted in green) correspond to the cases where the template itself contains the word “constellation” or “star”, landing them firmly in the corresponding group. Two softer outliers (highlighted in gray) are the templates • “in the night sky X borders”, tilts towards constellations, • “astrophotographers find X by first locating”, tilts towards stars, which is in line with our interpretation: a vague relation to the concept produces a mild bias. PCA
0
10
20
30
40
layer
50
60
70
80
Figure 12. Partial copy of Mistral chart from Figure 7 with analyzed layers 66 and 83 marked.
(a) Mistral Level 66 top-4 PCA components minus the sky sphere: object is STAR, CONSTELLATION, NEITHER
(b) Mistral Level 83 top-2 PCA components minus the “type” direction: last word is IS, BESIDE, NEAR, other
Figure 13. Top PCA directions ahead of the sky representation in Mistral Large 2.
While the “star”-“constellation” direction is the only unsurpassed rival to the sky sphere in many models, we can also clearly see another direction surface up in layers 79-87 in Mistral. Plotting the direction, orthogonal to “star”-“const” direction in layer 83, reveals its meaning (fig. 13b): it is focused on the last word of the prompt. We note that while we provided the plots only for one layer in each case, the plots for all layers 54-75 and 79-87, respectively, are essentially the same.
15