Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy

[2608.27813] Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy Skip to main content Search Submit Donate Log in Computer Science > Computation and Language arXiv:2608.27813 (cs) [Submitted on 28 Aug 2026] Title: Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy Authors: Juan Pablo Vigneaux , Mary Kennedy , Khalil Iskarous , Robert Frank , Matilde Marcolli View PDF HTML (experimental) Abstract: Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space. Comments: 29 pages (9 main text, 18 appendix), 20 figures, 7 tables. Code and data: this https URL Subjects: Computation and Language (cs.CL) MSC classes: 68T50 (Primary) 68T07, 94A17 (Secondary) ACM classes: I.2.7; I.2.6; G.3 Cite as: arXiv:2608.27813 [cs.CL] (or arXiv:2608.27813v1 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2608.27813 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Juan Pablo Vigneaux [ view email ] [v1] Fri, 28 Aug 2026 01:17:18 UTC (10,008 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Current browse context: cs.CL < prev | next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy#779728
• LatCom: Cross-Agent Latent Compression for Efficient Multi-Agent Collaboration#1132218
• Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses#798920
• Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses#797506
• The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese#620371