arXiv:2605.27259v1 [cs.LG] 26 May 2026
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
Sridhar Mahadevan∗ Adobe Research San Jose, CA [email protected]
Abstract We propose Kan Extension Transformers (KETs) as a unifying categorical framework for a diverse group of Transformer implementations. The core claim is that a Transformer layer can be viewed as a weighted structured extension operator: standard attention is the singleton-neighborhood case, Geometric Transformer style incidence mixing is a sparse edge-restricted case, and KET is the higher-order simplicial case. This lens also clarifies a bridge to diffusion-style completion. When the extension operator acts on detached predictive carriers instead of teacherforced hidden states, it becomes a valid self-conditioning mechanism that exposes noncausal structure without leaking gold future tokens. We include a comprehensive experimental validation of 12 different Transformer implementations varying across strict-causal and predict-detach regimes on Penn Treebank, WikiText-2, and WikiText-103. In the strict-causal setting, quadratic KET is the strongest model among the compared causal architectures on WikiText-2 and WikiText-103. Across all datasets, however, the largest gains come from the predict-detach regime rather than from changing the neighborhood family alone.
1
Introduction
A foundational problem in machine learning, already visible in Gold’s formalization of language identification in the limit (Gold, 1967), is generalization: extending information from a small or local domain to a larger structured domain. Category theory reframes this as the extension of a functor, a structured map on both objects and arrows, along a structural map (Mac Lane, 1971; Riehl, 2017; Richter, 2020). Once source values and target structure are organized functorially, left and right Kan extensions provide principled extension operations. Attention extends token evidence into contextualized representations (Vaswani et al., 2017), while diffusion-style models refine partial or noisy structure through denoising (Ho et al., 2020). KETs give a common categorical language for both: once we choose source neighborhoods and values on them, a contextual update is a weighted extension from local objects to token positions. Attention uses singleton neighborhoods, Geometric Transformer style mixing uses sparse local incidences, and KET uses simplices and their incidences. This viewpoint matters most when we ask what values are extended. Ordinary teacher-forced hidden states can leak gold future information through noncausal neighborhoods. Detached predictive carriers computed from causal prefix states instead make the same operator a valid self-conditioning ∗ Academic affiliation: Research Professor, University of Massachusetts, Amherst; See webpage at https://people.cs. umass.edu/~mahadeva/Site/About_Me.html
Preprint.
t1
t2
t3
t4
(a) Token neighborhood: causal chain t4
t2 t3
t1
(b) Topological neighborhood: learned geometric adjacency t3 t4
σ123 t1
t2
e24
e12 (c) Simplicial neighborhood: higher-order objects Figure 1: Three neighborhood systems. (a) Attention uses token-level neighborhoods. (b) TopoCoend adds learned geometric edges in a latent space. (c) KET adds higher-order simplices, so aggregation may occur over edges, faces, and motifs rather than only tokens.
mechanism: the operator is unchanged, but observed hidden states are replaced by predicted-anddetached surrogates. We focus the paper around three claims. 1. KET is best understood as a weighted left-Kan-style extension over simplicial neighborhoods, with attention and incidence mixing as special cases. 2. Predict-detach gives a clean causal boundary: richer noncausal neighborhoods become usable only when the transported values are detached predictions rather than teacher-forced future states. 3. We provide a comprehensive experimental validation of 12 different Transformer models on three canonical benchmark domains: Penn Tree Bank (PTB), WikiText-2, and WikiText-103, strict-causal quadratic KET is competitive and strongest on the two larger corpora, but the dominant empirical effect comes from the predict-detach regime itself. 1.1
Background Materials
This paper builds on six years of previous research on a categorical foundation for Artificial General Intelligence (AGI) – now available as a 600-page book complete with a Lean-4 verification of the theoretical results – which contains a deeper theoretical study of Kan Extension Transformers (Mahadevan, 2025a). There is a chatbot CLIFF implemented using a novel deep learning language called F UNCTORFLOW that provides implementation details of the Kan Extension Transformers (KETs) described in this paper (Mahadevan, 2025b). KET models were used in two state of the art deep causal research systems, D EMOCRITUS (Mahadevan, 2025c) and P ROMETHEUS (Mahadevan, 2026). Supplementary Materials give the categorical details (Section 9), information-regime analysis for GT (Section 16), and reproducibility details.
2
Neighborhood Systems and Kan-Style Aggregation
The model families studied in this paper differ less in the form of their update rule than in the structure of the neighborhoods over which they aggregate. Standard attention aggregates over tokens. TopoCoend aggregates over learned topological neighborhoods. Kan Extension Transformers aggregate over simplicial objects such as tokens, edges, and higher-order motifs. This section gives a common categorical language for these choices. Figure 1 illustrates the distinction: standard attention follows token-level neighborhoods, TopoCoend learns geometric adjacency in a latent space, and KET adds higher-order simplicial source objects such as edges and faces. 2
2.1
A Common Setup
Let T = {0, . . . , S − 1} denote the target token positions in a sequence window of length S. A model update at token t is built from a source neighborhood system N , whose objects may be tokens, edges, simplices, or learned geometric neighbors. We write X : N → Vect for the assignment of feature vectors to source objects. Here X(σ) may be a token state, edge embedding, or higher-order simplex value, and its contribution to target token t is determined by a structural weight W (t, σ) arising from attention scores, topological kernels, or simplicial incidence. This yields a common weighted aggregation form Z σ∈N X h′t ≈ W (t, σ) ⊗ X(σ) ≈ w(t, σ) V (σ),
(1)
σ∈N
where the coend notation emphasizes that the update is a structured weighted extension rather than an arbitrary sum (see Section 9 for more details on coend calculus). Equation (1) is the computational form shared by attention, TopoCoend, and KET. What changes from one model family to another is the choice of N and the origin of the weights W (t, σ). 2.2
Token Neighborhoods: Attention
For standard self-attention, the source neighborhood system contains only token objects: Nattn = {0, . . . , S − 1}. Each source object is a token s, and the update at target token t takes the familiar form X h′t = w(t, s) Vs . s
In categorical language, this is a weighted left-Kan-style extension in which source and target are both token positions; KET and TopoCoend generalize it by enriching the source neighborhood system. 2.3
Topological Neighborhoods: TopoCoend
TopoCoend keeps the source objects at the token level but changes the neighborhood relation. Instead of using only positional adjacency or dense token-to-token attention, it learns a low-dimensional geometric representation zt = π(vt ), where vt is either a hidden state or a detached predictive carrier. A fuzzy k-nearest-neighbor graph is then constructed in the latent space of the zt ’s. The resulting Ntopo is a learned geometric graph whose weights Wtopo (t, s) are induced by topological proximity rather than sequence position alone. The corresponding update is X h′t = ht + wtopo (t, s) Vs . (2) s
TopoCoend is thus weighted aggregation over a learned topological neighborhood category. 2.4
Simplicial Neighborhoods: KET
KET changes not just the weights but the source objects themselves: • 0-simplices: tokens t, 3
• 1-simplices: edges such as (t − 1, t), • optional higher simplices: faces, motifs, or larger spans. Let Nsimp denote this simplicial indexing category. Each simplex σ has value X(σ), and its contribution to token t is controlled by Wsimp (t, σ). The generic KET update is therefore X h′t = ht + wsimp (t, σ) V (σ). (3) σ∈Nsimp
Thus KET aggregates over higher-order source objects, not just individual tokens, generalizing both attention and local geometric mixing. 2.5
Quadratic and Incidence-Restricted KET
The simplicial update in Eq. (3) admits two important realizations. Quadratic KET. In the global or quadratic variant, every token may aggregate from every simplex: X h′t = ht + w(t, σ) V (σ), w(t, σ) = softmax(Q⊤ (4) t Kσ ). σ∈Nsimp
This is a second attention-like kernel, now defined over simplices rather than only over tokens. Its complexity is typically O(S 2 ) in sequence length. Incidence-restricted KET. In the incidence-restricted variant, only simplices incident to a token may contribute: X h′t ≈ ht + ϕ V (σ) . (5) σ∋t
For edge-only models this becomes et = ψ([vt−1 , vt ]),
(6)
h′t = ht + ϕ(et )
(causal incidence).
(7)
This reduces the complexity to O(S) and makes the connection to Geometric Transformers (GT) more transparent. 2.6
Relationship to Geometric Transformers
The geometric branch of a Geometric Transformer is an incidence-restricted Kan-style update in which the message map is implemented by a local convolutional or message-passing operator. This yields the following hierarchy: • Attention: weighted extension over tokens. • TopoCoend: weighted extension over learned topological neighborhoods. • Incidence-restricted KET: weighted extension over incident simplices. • Quadratic KET: weighted extension over all simplices with a learned global kernel. • GT: an efficient incidence-restricted special case of KET. This hierarchy explains the later expressive and runtime tradeoffs. It also clarifies our limited diffusion claim: predictive or denoising carriers turn the same transport rule into structured self-conditioning over partially specified future content.
3
Predict-Detach as the Causal Boundary
3.1
Three Information Regimes
The experiments compare three regimes (for a more detailed discussion of information regimes, please see Section 16 in the Supplementary Materials). 4
Strict-causal. Both neighborhoods and values are restricted to prefix-valid information. This is the standard autoregressive setting. Gold noncausal. The model is allowed to mix teacher-forced future hidden states through noncausal neighborhoods. This regime is invalid for language modeling but is useful as a leakage diagnostic. Predict-detach. The model may use noncausal neighborhoods, but the transported values are detached predictions derived from causal states. This is the regime we claim is valid and useful. 3.2
Predictive Carriers
For token t, let ℓt be the next-token logits produced from its causal hidden state. We build a predictive carrier êt = detach(softmax(ℓt /T ) E) , (8) where E is the embedding matrix and T is a temperature. The KET layer may then transport êt rather than teacher-forced hidden states. Future positions can therefore exchange predicted content through simplicial neighborhoods even though they never exchange teacher-forced future hidden states. 3.3
Why Detach Matters
There are two distinct reasons for the detach in (8). No gold-future values. The carrier is computed from a prefix-valid predictive state, not from a hidden state that already encodes the true future under teacher forcing. No leakage gradient. Detaching blocks the auxiliary branch from becoming a backdoor through which the model could cheaply encode targets into the very carriers later consumed by noncausal aggregation. This makes predict-detach a useful modal boundary: noncausal structure is allowed only through predicted surrogates that are causally generated and then frozen before transport. Operationally, the mechanism resembles internal prompt repetition or one-step denoising. The model writes down a guess about nearby future content and then reuses that guess as structured context, but it never gets to backpropagate through an illicit gold-future channel.
4
Experiments comparing Information Regimes
4.1
Setup
We evaluate KET on Penn Treebank, WikiText-2, and WikiText-103 with context length 128, L = 2, and d = 256, except where the block-completion depth sweep is explicitly marked otherwise. Headline tables compare a causal Transformer, GT, quadratic KET, and incidence-restricted KET; Figure 2 overlays TopoCoend trajectories. Self-conditioned runs use predictive carriers, and goldnoncausal models are retained only as leakage diagnostics. The supplementary repository contains harnesses, configs, summaries, logs, and checkpoints. 4.2
Strict-Causal Results
Table 1 shows the strict-causal comparison. On PTB the plain Transformer remains strongest, while on WikiText-2 and WikiText-103 quadratic KET is best among the tabulated causal architectures. The TopoCoend curve in Figure 2 remains in the same broad causal band. Incidence-restricted KET is slightly weaker but close on the larger datasets, consistent with an efficient sparse approximation to the richer quadratic neighborhood system. 4.3
Predict-Detach Results
Table 2 isolates the regime effect. Replacing hidden-state carriers by detached predictive carriers improves every KET architecture: quadratic KET drops from 133.37 to 31.43 on PTB, 156.42 to 5
Table 1: Strict-causal test perplexity. Lower is better. Model PTB WT2 WT103 124.47 127.17 133.37 137.19
Transformer GT-Causal KET-Quad-C KET-Inc-C
163.92 157.74 156.42 161.12
232.52 215.69 210.30 213.76
Table 2: Predict-detach ablation. Model PTB WT2 WT103 KET-Quad-C KET-Quad-PD KET-Inc-C KET-Inc-PD
133.37 31.43 137.19 6.54
156.42 38.23 161.12 19.08
210.30 51.89 213.76 47.17
GT-PD
1.05
1.59
12.84
38.23 on WT2, and 210.30 to 51.89 on WT103; the incidence-restricted variant improves even more on PTB and WT2. The strongest self-conditioned baseline is GT-PD, while TopoCoend improves only modestly. Thus the largest empirical gain comes from the information regime itself; the neighborhood family determines how that gain is expressed. 4.4
Leakage Diagnostic
When noncausal neighborhoods receive teacher-forced future hidden states, test perplexity collapses toward 1. These leakage diagnostics confirm that causal validity is determined not by the graph alone, but by which values may flow through it.
5
Experiments on Block Denoising as Structured Completion
A first generalization of standard autoregressive prediction is to predict not a single next token, but an entire future block at once. Instead of producing only xt+1 , the model predicts xt+1:t+B , where B is the block size. The corresponding prediction map is f : Ct −→ ΣB ,
(9)
B
where Ct is the causal context at position t and Σ denotes the B-fold product of the vocabulary object: ΣB = Σ × Σ × · · · × Σ. Categorically, direct block prediction therefore replaces the usual single-token target Σ by a structured product object. The model must map the context Ct into a tuple of future tokens in one step rather than constructing that future sequentially. Direct block prediction is challenging: a different approach is to change not the architecture, but the information regime. Instead of predicting the future block from context alone, we provide the model with a corrupted or partially specified version of that block and ask it to reconstruct the original: f : (Ct , x̃t+1:t+B ) −→ xt+1:t+B ,
(10)
where x̃t+1:t+B is a noisy, masked, or otherwise corrupted version of the future block. The corrupted block already carries partial structure, and the model learns to extend that partial object to a full consistent one. This suggests a complementary viewpoint to left-Kan aggregation: where left Kan extensions capture the aggregation of evidence from local sources, denoising naturally points toward a right-Kan intuition, in which the task is to find a value or structure that is compatible with multiple local constraints at once. We now report on results with block denoising, a diffusion-style completion task in which the model predicts a length-4 future block directly or reconstructs it from a corrupted version. Figures 3 and 6
PTB
WikiText-2
WikiText-103
Figure 2: Validation perplexity across datasets. The headline tables quantify Transformer, GT, and KET comparisons; the plots additionally show TopoCoend for qualitative reference. On PTB, the Transformer baseline is strongest among the tabulated strict-causal models. On WikiText-2 and WikiText-103, quadratic KET is the strongest tabulated strict-causal model. Self-conditioned GT/KET variants improve sharply, TopoCoend changes more modestly, and explicit future-information leakage diagnostics collapse.
7
Autoregressive Prediction
Direct Block Prediction
Denoising Completion
→ xt+1
Ct →
(Ct , x̃) →
(xt+1 , . . . , xt+B )
(xt+1 , . . . , xt+B )
Ct → Σ
Ct → ΣB
(Ct , x̃) → x
Ct
Figure 3: Three information regimes for language modeling. Direct block prediction maps context into the product object ΣB ; denoising completion instead fills a partially specified future block. 4 frame the comparison in Table 3: direct block prediction is a one-shot map into Σ4 , whereas denoising is a structured completion problem. This completion process admits a natural simplicial analogy. A block of tokens may be viewed as a simplex whose vertices correspond to token positions and whose internal relations encode coherence across the block. A corrupted block x̃ then behaves like a partially specified simplex: some vertices or relations are present, while others are missing or uncertain. In simplicial homotopy theory (May, 1992), such partial simplices are often described as horns. To fill a horn is to extend a partial simplex to a full simplex. A simplicial set is called a Kan complex when every horn admits such a filler. 5.1
Setup
We ran a structured-completion harness with matched causal Transformer and incidence-KET backbones on PTB, WikiText-2, and WikiText-103. To test whether the block objectives change with depth, we report higher-depth sweeps at L = 8 and L = 16 with d = 64. All block experiments use context length T = 128 and block size B = 4. The direct objective learns Ct −→ Σ4 , while the denoising objective learns (Ct , x̃t+1:t+4 ) −→ xt+1:t+4 .. For denoising, corruption is applied independently to positions in the future block using an 8-step schedule p(s) = 0.05 + 0.45 ·
s−1 , 7
s ∈ {1, . . . , 8},
so the corruption rate ranges from 0.05 to 0.50. We report first-token perplexity and block perplexity averaged over all four target offsets. 5.2
Main Results
Table 3 shows two patterns. Denoising is dramatically easier than direct block prediction across datasets and depths. Greater depth mainly helps the hard direct objective: from L = 8 to L = 16, KET first-token PPL drops from 152.25 to 143.14 on PTB, 202.89 to 174.55 on WT2, and 574.06 to 472.29 on WT103. Denoising is already low-perplexity, so depth and backbone effects are modest. These results echo predict-detach. Direct block prediction asks the model to select an element of Σ4 from context alone; denoising supplies a partial future object and asks for completion. That information-regime change dominates the backbone choice, reducing first-token perplexity by one to two orders of magnitude.
6
Related Work
We summarize related work along three lines. Diffusion and denoising Transformers pair refinement objectives with Transformer architectures (Ho et al., 2020; Peebles and Xie, 2023; Li et al., 2022); our connection is conceptual rather than score-based. Higher-order, hypergraph, and topological attention models enrich the aggregation domain (Hajij et al., 2022; Goh et al., 2022; Battiloro et al., 2024; Zhang et al., 2022), and sheaf-based neural models show that algebraic-topological structure changes diffusion and message passing (Bodnar et al., 2022; Barbero et al., 2022; Hajij et al., 2025). Our contribution is to use Kan extensions as a unifying language for attention, incidence mixing, learned topological neighborhoods, and predict-detach completion. 8
Table 3: Structured language modeling with direct block prediction and block denoising (B = 4, context = 128, d = 64) on PTB, WikiText-2, and WikiText-103. Dataset
Model
L
Test first-PPL
Test block-PPL
PTB PTB PTB PTB
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
8 8 8 8
155.95 152.25 4.21 4.21
332.38 327.62 4.56 4.55
PTB PTB PTB PTB
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
16 16 16 16
144.14 143.14 4.17 4.18
316.65 315.25 4.54 4.55
WikiText-2 WikiText-2 WikiText-2 WikiText-2
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
8 8 8 8
194.11 202.89 4.42 4.44
362.19 369.77 4.80 4.81
WikiText-2 WikiText-2 WikiText-2 WikiText-2
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
16 16 16 16
177.87 174.55 4.35 4.35
343.04 341.37 4.73 4.72
WikiText-103 WikiText-103 WikiText-103 WikiText-103
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
8 8 8 8
539.92 574.06 5.83 5.85
839.24 859.31 6.07 6.09
WikiText-103 WikiText-103 WikiText-103 WikiText-103
TF-Block-4 KET-Block-4 TF-Denoise-4 KET-Denoise-4
16 16 16 16
491.12 472.29 5.69 5.74
796.10 782.34 5.96 6.01
Corrupted Block as a Horn xt+2
Completed Simplex xt+2
denoising / horn filling xt+1
xt+3
xt+1
partial structure x̃ known faces, missing completion
xt+3
completed structure x full simplex / coherent block
Figure 4: A simplicial view of denoising language modeling. A corrupted block is a partial simplex, or horn; denoising fills the missing relations to produce a coherent completion.
7
Discussion and Future Directions
The experiments support a more specific interpretation of KET than a generic “higher-order attention helps” claim. In the strict-causal setting, neighborhood design matters: quadratic KET is the strongest of the compared causal models on WT2 and WT103, while the incidence variant remains competitive with lower structural cost. Across all runs, however, the dominant effect comes from the information regime rather than from the neighborhood family alone. 7.1
Limitations and Future Directions
Although we compare 12 Transformer models, the study remains modest in scale and centered on perplexity. We therefore view the main contribution as a practical design lens and causal framing, not a final verdict on large-scale architectures. Future work should scale neighborhood design and information regime separately, testing richer simplicial families, adaptive simplex selection, and longer-context sparse approximations. 9
Hybrid systems are another direction: KET provides prescribed combinatorial structure, while TopoCoend learns geometric neighborhoods. The structured-completion perspective also extends beyond the small block-denoising setting: predict-detach, blockwise denoising, infilling, and other partial-information objectives can be treated as neighboring points in one design space.
8
Conclusion
We introduced KET as a categorical unification of Transformer sequence models in which a layer extends local information from a chosen source category of neighborhoods to target token positions. Attention is the singleton-neighborhood case, incidence-style geometric mixing is a sparse local case, and KET is the simplicial higher-order case. The same framework clarifies when noncausal neighborhoods can be used for self-conditioning: only through detached predictive carriers rather than teacher-forced hidden states.
Supplementary Materials We give a more detailed theoretical explanation for KET models here. We also provide an accompanying supplementary reproducibility code repository that contains the exact experiment harnesses, configs, logs, summaries, and checkpoints used for the reported language-model comparisons and structured completion pilots.
9
Kan Extensions as Universal Structured Extension
Machine learning research over the past six decades, ever since Gold’s formalization of language identification in the limit (Gold, 1967), has been centered on the scientific principle of approximating functions over sets. Category theory (Mac Lane, 1971) fundamentally alters our perspective of machine learning, moving from functions to functors, structured mappings of categories that map both the objects and the arrows of an input category to an output category. Instead of investigating a smorgasboard of methods for an arbitrary extension of a function on points, we instead redefine the problem in terms of an extension of a functor along a structural map. Once the source values and the target structure are organized functorially, universal constructions become available: left and right Kan extensions define principled ways to extend local information across a larger domain relative to a specified structure. This perspective is central to the architectures developed in this paper. A Transformer layer, a geometric mixing layer, or a higher-order message-passing rule may all be viewed as structured extension operators. The key question is not merely which values are mixed, but along what structure and according to which universal principle they are extended. 9.1
Kan Extension Definition
We summarize the main ideas underlying Kan Extensions, leaving the details to standard textbook treatments (Mac Lane, 1971; Riehl, 2017; Richter, 2020). Let F : C → E,
K:C→D
be functors. The functor F assigns values to objects of the source category C, while K explains how C sits inside, or maps into, the larger target category D. We would like to construct a functor H:D→E that extends F along K. Definition 1 The left Kan extension of F along K, denoted LanK F , is a functor LanK F : D → E together with a natural transformation η : F ⇒ (LanK F ) ◦ K 10
such that for any other pair G : D → E, there exists a unique natural transformation
γ : F ⇒ G ◦ K,
α : LanK F ⇒ G with γ = (αK) ◦ η. Dually, the right Kan extension RanK F is the universal extension characterized by maps ϵ : (RanK F ) ◦ K ⇒ F. F
C
E η
LanK F K G
D Succinctly, we can informally rephrase the definitions as: A left Kan extension is the most universal way to build values on D from data on C. A right Kan extension is the most universal way to complete values on D so that they remain compatible with C. For the purposes of this paper, the left Kan extension will play the leading role, since it captures aggregation and contextualization. The right Kan point of view will become increasingly important when we turn to denoising, completion, and refinement. 9.2
Pointwise Formulas: Colimits and Limits
Kan extensions become especially intuitive when written pointwise. For an object d ∈ D, the value (LanK F )(d) is built from all source objects c ∈ C that map into d. Formally, one considers the comma category (K ↓ d), whose objects are arrows Kc → d in D, and whose morphisms are commuting triangles induced by morphisms in C. If the relevant colimits exist, then the left Kan extension is computed pointwise by (LanK F )(d) ∼ = colim F. (K↓d)
(11)
Dually, the right Kan extension is built from all ways in which d must be compatible with source objects. Writing (d ↓ K) for the comma category whose objects are arrows d → Kc, one obtains the pointwise formula (RanK F )(d) ∼ = lim F. (d↓K)
(12)
These two formulas already capture the conceptual distinction that will matter throughout the paper: • Lan uses a colimit: it glues together many local contributions. • Ran uses a limit: it finds a jointly compatible value satisfying many constraints. For machine learning readers, it is often enough to remember the mnemonic: left Kan = aggregation from evidence, straints. 11
right Kan = completion under con-
9.3
Interpolation and Completion as Motivating Examples
The difference between sets and categories can be seen in two familiar problems. Interpolation. Suppose we know values f (xi ) = yi at a finite set of sample points. As a settheoretic problem, there is no canonical extension to all query points x: one may choose a constant predictor, a piecewise linear interpolant, a polynomial, or a kernel smoother. The ambiguity does not disappear in category theory, but it is reframed. One specifies a structural relation W (i, x) saying how each sample xi contributes to a query point x, and the extension is then computed relative to that structure. In finite enriched settings, the resulting left Kan extension takes the weighted form X (LanW F )(x) ≈ W (i, x) F (i). i
Thus the Kan extension is not a magical elimination of modeling choices; it is the universal extension relative to the chosen structure. Completion and denoising. Now suppose the problem is not to interpolate from samples but to complete a partially observed or noisy state. Again, many completions may be possible in principle. What selects among them is the compatibility structure: which local constraints must the completed state satisfy? The right Kan viewpoint formalizes exactly this situation. Instead of aggregating evidence from contributing sources, one computes a value that is as compatible as possible with all the relevant local views. This is the categorical intuition behind denoising, structured completion, and diffusion-style refinement. These two examples foreshadow the main architectural claim of the paper: attention behaves like a learned weighted left Kan extension, while self-conditioned denoising and refinement point toward a right-Kan interpretation. 9.4
Coends as Weighted Aggregation
The pointwise left Kan formula in Eq. (11) can often be written in a more computationally useful form using a coend. For readers unfamiliar with ends and coends, the essential intuition is simple: a coend behaves like a structured weighted sum, with the indexing category enforcing the appropriate identifications. The notation
Z c H(c, c)
should be read informally as “sum over all c, while identifying contributions that are equivalent under the morphisms of the indexing category.” In the settings relevant here, this behaves like a weighted pooling operation. If F (c) is a feature vector and W (c, t) is a compatibility weight, then the coend has the computational flavor X (coend)(t) ∼ W (c, t) F (c), c
possibly with identifications imposed by the structure of the category. This is exactly the pattern that appears in attention, geometric message passing, and Kan-style pooling: values are aggregated from many structured sources, and the weights encode how strongly each source contributes to the target. 9.5
Kan Extensions via Coends
A standard enriched formula expresses the pointwise left Kan extension as a coend: Z c∈C ∼ D(Kc, d) ⊗ F (c). (LanK F )(d) = 12
(13)
Here D(Kc, d) is the hom-object measuring how Kc contributes to d, and ⊗ denotes the enrichment, which in our neural setting behaves like scalar weighting of feature vectors. Equation (13) therefore says: To compute the value at d, gather contributions from every source object c, weight them by how strongly Kc points to d, and aggregate them universally. Dually, in enriched settings the right Kan extension may be written in end form as Z (RanK F )(d) ∼ [D(d, Kc), F (c)], =
(14)
c∈C
where the end expresses compatibility with all incoming constraints. We will not need this formula computationally in most of the paper, but it is conceptually important: left Kan extensions are naturally coend-like and aggregation-oriented, while right Kan extensions are end-like and compatibilityoriented. 9.6
Interpretation for Transformer Architectures
Equation (13) has a direct neural interpretation. • The source objects c correspond to tokens, edges, or higher-order simplices. • The target object d is typically a token position whose representation is being updated. • The hom-objects D(Kc, d) play the role of compatibility or neighborhood weights. • The values F (c) are the feature vectors carried by the source objects. Under this interpretation, a familiar update h′d =
X
wd,c Vc
c
is precisely a discretized enriched Kan/coend computation. The architectural families in this paper can then be read in a common language: • Transformers aggregate over token neighborhoods; attention is a weighted left-Kan-style pooling rule on a sequence. • Kan Extension Transformers (KET) make the source category explicit, lifting from tokens to simplices and then extending back to token updates via simplicial incidence. • TopoCoend Transformers retain the weighted coend form but define the neighborhood category through learned topological geometry rather than fixed simplicial structure. Thus KET and TopoCoend should not be viewed as unrelated alternatives. They are two realizations of the same broader idea: structured contextualization as universal weighted extension. 9.7
Why This Matters for the paper
This section provides the conceptual bridge from ordinary geometric mixing to the models introduced later. • It explains why attention and geometric aggregation can be treated uniformly as structured extension operators. • It clarifies why KET naturally lives in the language of Kan extensions: the model extends values from simplicial structure back to tokens. • It explains why TopoCoend lives naturally in the language of coends: the model aggregates over learned neighborhoods using weighted geometric compatibility. • It prepares the later analysis of information regimes by distinguishing aggregation from compatibility and extension from leakage. The main lesson is that Kan extensions and coends are not merely abstract categorical decorations. They provide the right universal vocabulary for describing how local token information is transported, aggregated, and reorganized across structured domains. 13
10
From Autoregression to Structured Completion
In the categorical view of Transformer-style architectures presented in this paper, attention, geometric mixing, simplicial aggregation, and topological neighborhood pooling all appear as instances of structured extension operators. Kan Extension Transformers (KET) make this explicit: a layer can be understood not merely as a hand-designed mixing rule, but as a principled way of extending local token representations across a richer neighborhood system. Our experiments also highlight a central empirical fact: information regime often matters more than architectural detail. In light of this, we can ask a more fundamental question: What is the right prediction regime for language modeling once we stop assuming that generation must proceed one token at a time? In standard autoregressive language modeling, the model learns a map f : Ct −→ Σ,
(15)
where Ct is the causal context at position t and Σ is the vocabulary object. This formulation has been extraordinarily successful, but it builds in a strong modeling choice: the future is generated as a sequence of isolated single-token decisions. From the categorical perspective, there is nothing inevitable about this restriction. Autoregression is one particular way of extending a local context into a future prediction, but it is not the only one. More generally, language modeling may be viewed as a problem of structured extension or structured completion. Instead of predicting one token, a model may attempt to predict an entire future block at once, or it may attempt to complete a partially specified future structure. These two possibilities correspond to different information regimes: • Direct block prediction: learn a map Ct −→ ΣB , which predicts a block of B future tokens in one shot. • Denoising block completion: learn a map (Ct , x̃t+1:t+B ) −→ xt+1:t+B , which completes a corrupted or partially specified future block. The difference between these regimes is not merely cosmetic. Direct block prediction asks the model to construct an entire structured future from context alone. Denoising instead treats the future block as a partially specified object and asks the model to complete it compatibly. The first regime is closer to a one-shot extension problem, while the second is closer to a structured completion problem. We prefer the latter formulation in this paper. From the KET perspective, this shift is especially meaningful. We emphasized above that left Kan extensions as a principled language for aggregation and contextualization. Here we begin to see the complementary side of the story: language modeling is not only about aggregating evidence from the past, but also about completing a future structure under internal compatibility constraints. Autoregressive next-token prediction, direct block prediction, and denoising completion should therefore be seen as distinct information regimes for the same underlying structured prediction problem. The central empirical claim of this paper is that these information regimes differ dramatically in difficulty and effectiveness. Across datasets and architectures, direct block prediction proves much harder than denoising completion, and denoising often yields much lower perplexity than direct block generation. This suggests that language modeling is better understood not merely as one-step prediction, but as a problem of structured completion. Kan-style architectures tell us how to extend and organize information over structured domains. This paper asks what kind of target structure should be predicted. Once that question is posed, the design space widens substantially: the target need not be a single token, but may be a block, a corrupted future configuration, or a partially specified linguistic object. The choice among these regimes turns out to be one of the most important determinants of empirical performance. 14
11
Direct Block Prediction as a Product-Object Target
A first generalization of standard autoregressive prediction is to predict not a single next token, but an entire future block at once. Instead of producing only xt+1 , the model predicts xt+1:t+B , where B is the block size. The corresponding prediction map is f : Ct −→ ΣB , (16) where Ct is the causal context at position t and ΣB denotes the B-fold product of the vocabulary object: ΣB = Σ × Σ × · · · × Σ. Categorically, direct block prediction therefore replaces the usual single-token target Σ by a structured product object. The model must map the context Ct into a tuple of future tokens in one step rather than constructing that future sequentially. This formulation is natural, but it is substantially harder than ordinary next-token prediction. The target space now has cardinality |Σ|B , so the model must select a coherent block from a combinatorially larger output space. Equivalently, it must infer not only which tokens should appear, but which joint configuration of tokens is compatible with the context. Direct block prediction is still an extension problem, but it is a particularly demanding one: the model must extend the causal context directly into a fully specified future structure, without any partial information about that structure being provided as input. This is precisely what distinguishes it from denoising completion, where the future block is only partially specified and the task becomes one of structured refinement rather than one-shot generation. In practice, our experiments show that direct block prediction performs surprisingly poorly even when paired with strong architectures. This suggests that the main difficulty is not architectural weakness, but the intrinsic hardness of predicting a coherent product-structured target in a single step.
12
Denoising as Structured Completion
A different way to change the language-modeling objective is to change not the architecture, but the information regime. Instead of predicting the future block from context alone, we provide the model with a corrupted or partially specified version of that block and ask it to reconstruct the original: f : (Ct , x̃t+1:t+B ) −→ xt+1:t+B ,
(17)
where x̃t+1:t+B is a noisy, masked, or otherwise corrupted version of the future block. This changes the task in an important way. Direct block prediction asks the model to generate a fully specified future object from scratch. Denoising instead presents the future as a partially specified structure and asks the model to complete it. Some coordinates are already present, others are uncertain or corrupted, and the model must infer a coherent full block compatible with both the causal context and the partial future evidence. From a categorical perspective, this is closer to a completion problem than to a one-shot generation problem. The corrupted block already carries partial structure, and the model learns to extend that partial object to a full consistent one. This suggests a complementary viewpoint to left-Kan aggregation: where left Kan extensions capture the aggregation of evidence from local sources, denoising naturally points toward a right-Kan intuition, in which the task is to find a value or structure that is compatible with multiple local constraints at once. This perspective helps explain why denoising is often easier than direct block prediction. The model is not forced to choose a block in ΣB from context alone. Instead, it solves a constrained completion problem in which part of the future structure is already given. Empirically, this distinction turns out to matter greatly: denoising block prediction consistently outperforms direct block prediction across the architectures studied in this paper. For this reason, the experiments below suggest that language modeling is often better viewed as a problem of structured completion rather than pure one-shot generation. 15
13
Simplicial Structure and Horn Completion
The denoising regime admits a natural simplicial interpretation. A corrupted future block is not merely noisy data; it is a partially specified structured object. Some components of the future are given, others are masked, corrupted, or uncertain, and the model is asked to infer a coherent completion. This makes denoising fundamentally different from direct block prediction. The task is no longer to generate a full product-structured target from scratch, but to complete an already partially present configuration. A useful geometric analogy is provided by simplicial homotopy theory. A horn is a partial simplex in which enough faces are present to specify most of the structure, but one face or relation remains missing. To fill the horn is to extend this partial object to a full simplex. A simplicial set is called a Kan complex if every horn can be filled May (1992). We do not claim that a denoising language model literally defines a Kan complex in the strict homotopy-theoretic sense. Rather, the horn-filling picture provides the right structural intuition: the model receives a partial future object and must construct a completion compatible with the local information already present. This perspective is especially natural from the KET viewpoint developed in this paper. Left Kan extensions organized the aggregation of contextual evidence from structured neighborhoods. Here, denoising introduces the complementary intuition of compatibility-based completion. The corrupted block already imposes constraints on the space of valid futures, and the model must fill in the missing coordinates in a way that respects both the causal context Ct and the internal relations within the block itself. In this sense, denoising is closer to a right-Kan-style completion problem than to a one-shot left-Kan-style aggregation problem. The horn-filling view also helps explain the empirical advantage of denoising over direct block prediction. Direct block prediction asks the model to select a point in the full product object ΣB from context alone. Denoising, by contrast, constrains the search space by presenting a partial object whose completion must satisfy local compatibility conditions. This additional structure makes the task easier and more stable. Empirically, as the experiments show, that difference in information regime matters far more than small architectural variations. For this reason, denoising language modeling is best viewed not simply as another decoding trick, but as a structured completion problem with a natural simplicial interpretation. The horn-filling picture makes precise the sense in which the model is asked to complete a partial future rather than emit a future from nothing.
14
Partial Information, Horn Filling, and Structured Completion
The contrast between direct block prediction and denoising block prediction may now be stated more precisely. In direct block prediction, the model learns a morphism f : Ct −→ ΣB , mapping a causal context Ct directly into the B-fold product object of future tokens. The target is a fully specified block, and the model must select a coherent point of ΣB in a single step. Categorically, the prediction problem is therefore posed entirely in terms of a map from context to a complete structured target. In denoising block prediction, by contrast, the model is given a corrupted or partially specified future block x̃ and learns a map f : (Ct , x̃) −→ xt+1:t+B . The crucial difference is that the domain now contains partial information about the target object itself. The model is not asked to choose a future block from scratch. Instead, it must complete a partially specified structure in a way that is compatible with both the context and the information already present in x̃. This distinction can be expressed diagrammatically as Ct
ΣB
versus 16
(Ct , x̃)
ΣB .
The codomain is the same in both cases, but the domain is richer in the denoising regime. That extra structure dramatically changes the learning problem. Direct block prediction searches over the full product object, whereas denoising searches over a constrained family of completions consistent with a partial specification. 14.1
Horn Filling as a Structural Analogy
This completion process admits a natural simplicial analogy. A block of tokens may be viewed as a simplex whose vertices correspond to token positions and whose internal relations encode coherence across the block. A corrupted block x̃ then behaves like a partially specified simplex: some vertices or relations are present, while others are missing or uncertain. In simplicial homotopy theory, such partial simplices are often described as horns. To fill a horn is to extend a partial simplex to a full simplex. A simplicial set is called a Kan complex when every horn admits such a filler. We do not claim that a denoising language model literally constructs a Kan complex in the strict homotopy-theoretic sense. Rather, the horn-filling picture provides the correct structural intuition: the model is given a partial future object and must construct a coherent completion. This perspective sharpens the empirical lesson of this paper. Denoising is easier than direct block prediction not merely because it is “given more input,” but because it solves a more structured problem. The corrupted block already narrows the space of admissible futures, and the model must fill in what is missing rather than generate everything from nothing. 14.2
Relation to KET
KET and related architectures provide a principled language for structured extension: local token information is extended across simplicial or topological neighborhood systems by universal aggregation mechanisms. We are in a sense generalizing attention from the structure of the aggregator to the structure of the target. From this perspective: Ct → Σ is ordinary next-token prediction, Ct → ΣB is direct prediction of a product-structured future, and (Ct , x̃) → xt+1:t+B is structured completion in a category of partial information. This is the deeper connection to the broader program of the book. Intelligence is often not best understood as one-shot emission of fully specified outputs, but as the ability to extend partial structures into coherent global structures within an appropriate category of information. Under that reading, denoising language modeling is not a minor variant of decoding. It is a concrete instance of a much more general principle of structured completion.
15
Predict–Detach as a Modal Boundary
The predict–detach mechanism plays a central role in the valid self-conditioned regimes studied in this paper. Operationally, it is simple: the model computes a predictive carrier and then applies detach() so that the carrier may participate in forward computation without transmitting gradients backward through the predictive branch. Conceptually, however, this is more than an implementation trick. It introduces a separation between two kinds of structure: the forward semantics of the model and the learning semantics induced by reverse-mode differentiation. This section gives a categorical formulation of that separation. The key idea is that predict–detach acts as a modal boundary: it preserves forward information flow while annihilating the associated gradient flow. This viewpoint makes precise why predict–detach can support noncausal structural aggregation without collapsing into gold future-token leakage. 17
15.1
Forward and Backward Semantics
A neural layer has two mathematically distinct aspects. Forward semantics.
A forward operator transforms representational states: f : X → Y,
where X and Y may be token sequences, hidden states, predictive distributions, or embedding carriers. Backward semantics. Under reverse-mode automatic differentiation, the same operator induces a backward map on cotangent or gradient objects: f ∗ : ∂Y → ∂X. Thus every learned computation carries both a forward action and a backward action. For the purposes of this paper, it is convenient to package these together into a paired semantics f 7→ (f, f ∗ ). 15.2
Detach as a Modal Operator
The detach operation preserves the forward component of an operator but kills its backward component. Definition 2 (Detach modality) For an operator with paired semantics (f, f ∗ ), define □(f, f ∗ ) := (f, 0). Equivalently, □ leaves the forward map unchanged while replacing the backward map by the zero morphism. This formalizes the implementation-level meaning of detach(): the carrier remains available to subsequent layers as a forward signal, but it cannot serve as a path for gradient propagation. Interpretation. The modality □ is not a change to what the model computes in the forward pass. It is a change to which paths are visible to learning. In this sense, predict–detach imposes a modal boundary between semantic use and gradient credit assignment. 15.3
Predict–Detach as a Composite
Let Pred : H → E denote the predictive carrier map from hidden states H to embedding space E: Pred(h) = softmax(W h/T ) E. The predict–detach carrier is then the composite PredDetach = □ ◦ Pred. Thus a hidden state first generates a predictive embedding and that embedding is then passed into later geometric or simplicial aggregation only through the detached channel: Pred
□
H −−−→ E − → E. In paired semantics, this becomes (Pred, Pred∗ )
7→
(Pred, 0).
So the predictive carrier participates in forward aggregation, but the predictive branch itself does not receive gradient contributions from that later aggregation. 18
15.4
Information Regimes Revisited
This modal viewpoint clarifies the three information regimes used throughout the paper. Strict causal regime. All forward morphisms respect temporal order. No future-position object contributes to the update at time t. This is the ordinary autoregressive regime. Gold noncausal regime (invalid). Future-position hidden states computed under teacher forcing are allowed to contribute directly to the update at time t. Since those hidden states encode gold future tokens, this introduces an illegitimate information path. Predict–detach regime. Future-position carriers may appear structurally in the neighborhood system, but only through predictive values passed through the detach modality. Thus the forward computation may exploit richer noncausal structure, while the learning dynamics remain shielded from direct gradient flow through that channel. The essential distinction is therefore not just whether the neighborhood graph is noncausal. It is whether the noncausal channel carries gold future state or only detached predictive state. 15.5
Why Predict–Detach Preserves Causal Validity
The causal legitimacy of predict–detach follows from the fact that the carrier is generated from prefix-conditioned logits and then insulated from backward influence. Let
êt = detach softmax(ℓt /T ) E ,
ℓt = Wo ht .
Since ht is causal, êt is also a function only of the prefix x≤t . Even if the neighborhood system later treats êt as part of a nonlocal geometric or simplicial structure, that structure is enriched only by prefix-derived predictions rather than by gold future tokens. This is precisely what fails in the gold noncausal regime: there the carrier itself already encodes teacher-forced future information. 15.6
Diagrammatic View
The contrast between an ordinary predictive branch and a detached one can be summarized diagrammatically as H
Pred
H PredDetach E
E ⇝
∂H
Pred∗
∂E
∂H
0
∂E
The left diagram represents an ordinary predictive carrier with both forward and backward semantics. The right diagram represents the detached carrier: the forward morphism remains, while the backward morphism is zeroed out. 15.7
Relation to KET and TopoCoend
This modal interpretation applies uniformly across the models of the paper. • In quadratic KET, detached predictive carriers may be lifted to simplex values and then aggregated globally. • In incidence-restricted KET, detached carriers may be passed along local simplicial incidence relations. • In TopoCoend, detached carriers may be embedded into a learned topological coordinate system and used to define fuzzy neighborhood aggregation. Thus predict–detach is not specific to one architecture. It is a general semantic device for permitting richer structural forward computation without introducing invalid gradient paths. 19
15.8
Conceptual Summary
The main point of this section is that predict–detach should not be viewed merely as a frameworkspecific trick. It has a clean categorical meaning: predict–detach is a modal boundary that preserves forward semantic carriers while discarding their backward learning morphisms. This perspective explains why self-conditioned noncausal aggregation can be valid, why gold noncausal aggregation is not, and why the same device can be used uniformly across GT, KET, and TopoCoend architectures.
16
Information Regimes in Geometric Transformers
In this section, we study two questions: architecture class and information regime. The key distinction is that a model may keep causal self-attention while still receiving auxiliary side-channel information through a geometric branch. We therefore classify comparisons by whether auxiliary morphisms factor through prefix-only objects (strict causal regime C), are built from detached endogenous selfpredictions (self-conditioned regime E), or introduce exogenous future-informative paths (augmentedcontext regime A).
16.1
Categorical View of Information Regimes
Let I≤t denote the prefix index category up to position t. A strict autoregressive model is a functor over I≤t , and all update paths to ŷt factor through prefix objects (Regime C). Auxiliary augmentation is represented by an added morphism αt : At → Ht . If αt is provided by an external futureinformative channel, it defines Regime A. If αt is computed endogenously from prefix-derived hidden states and detached before reuse, we label it Regime E. We analyze six practical variants: • transformer_causal • transformer_future_hint • gt_causal • gt_noncausal • gt_pred_next_detach • gt_pred_prev_causal_detach and report each one with an explicit regime label and side-channel description. FutureHint vs PredictiveHint. transformer_future_hint and gt_pred_next_detach can produce similar validation/perplexity trajectories, but they are not the same mechanism. transformer_future_hint is an exogenous hint pathway (A), while gt_pred_next_detach is endogenous self-conditioning (E): the hint is generated by the model from prefix-derived states and injected after a detach step. This distinction is important for fair interpretation of gains and for categorical factorization arguments. 16.2
Kan Extension Transformer as Self-Conditioned Completion
Let I≤t be the prefix index category and F≤t : I≤t → Vect the prefix-state functor. At step t, we consider two coupled operations: x bt+1 ≈ Ranιt (F≤t )(t + 1), and hgeo ≈ t
colim
I≤t ∪{t+1}
F≤t ∪ x bt+1 ,
where ιt : I≤t ,→ I≤t+1 is the canonical inclusion. The first line is a right-Kan-style predictive completion (next-token estimate from prefix information), while the second is a left-Kan/colimit-style geometric aggregation that reuses this completion as an auxiliary object. 20
Operationally, this yields a self-conditioned Kan extension transformer: 1. predict a next-token distribution from prefix-derived hidden states; 2. map this prediction into an embedding-level auxiliary branch; 3. aggregate hidden and auxiliary streams through the geometric mixer. This explains the observed transition regime of predictive-hint GT variants. Early in training, x bt+1 is noisy and behavior remains close to strict-causal controls. As prediction quality improves, the induced geometric update approaches the behavior of augmented-context GT variants. In this sense, the model interpolates between strict causal computation and augmented-context behavior through an endogenous Kan completion, rather than by directly injecting observed future tokens. We distinguish three regimes: 1. strict causal: no auxiliary completion channel; 2. self-conditioned Kan completion: auxiliary channel built from x bt+1 ; 3. exogenous augmented channel: auxiliary channel uses externally provided futureinformative signal. 16.3
The Geometric Transformer
The Geometric Transformer augments Transformer-style global mixing with a geometric branch defined over local simplicial structure. In sequence tasks, token indices are 0-simplices and adjacent pairs form the 1-skeleton. The key update in this paper is to separate architecture from information budget by explicitly modeling causal versus augmented-context variants. The architectural diagram on the next page clarifies this distinction: (a) Baseline Transformer (b) GT with the earlier (non-causal) geometric mixer (c) GT-augmented (d) Predicted-hint GT. Categorical Interpretation of Auxiliary Channels The central issue is not whether attention is causal, but whether the full update diagram at index t includes extra morphisms that are futureinformative. We formalize this at the diagram level. For a token sequence, let Xi denote the representation object at position i (after token and positional embedding, and any prefix-local processing up to that stage). Let Ht denote the decoder hidden state object used to produce the logit at position t, and let ŷt denote the output-logit object at position t. Thus, the base causal chain X0 → X1 → · · · → Xt → Ht → ŷt is an object-level shorthand for the prefix computation in the autoregressive model. Base causal diagram. Let I≤t be the prefix index category (objects are positions 0, . . . , t with arrows i → j for i ≤ j). A strict autoregressive decoder defines a functor Ft : I≤t → Vect, and logits ŷt are computed from the colimit object induced by this prefix-only diagram. In this setting, all computational paths factor through I≤t . Auxiliary augmentation.
Suppose we add an auxiliary object At and morphism αt : At → Ht ,
where Ht is the decoder state at position t. This extends the diagram from a prefix-only shape to a larger category Iet . Definition 3 (Admissible Causal Augmentation) An auxiliary map αt is admissible causal iff it factors through the prefix diagram, i.e., there exists Gt in the image of Ft and maps At → Gt → Ht with Gt constructed only from objects indexed by ≤ t. 21
Output (a) Transformer (causal baseline)
Output
Add & Norm
Add & Norm
Feed Forward
Feed Forward
Add & Norm
Add & Norm
Causal Self-Attention
Causal Self-Attention
Input (tokens + pos)
Input (tokens + pos)
(b) GT-causal (geo_causal=1)
Geometric Mixer Causal Conv1D
Output (c) GT-augmented (geo_causal=0)
Add & Norm
Feed Forward
Add & Norm
Geometric Mixer Symmetric Conv1D
Future-info path may exist
Causal Self-Attention
Input (tokens + pos) (d) Predicted-hint GT variants
Residual + Norm
Geometric Conv Branch
Mode C: pred_prev_causal_detach hint at t ← pt−1 E
Predicted token dist. pt = softmax(Wout ht )
Mode E: pred_next_detach hint at t ← pt E
Decoder hidden state ht
22 Figure 5: Geometric Transformer architectural variants studied in this paper.
X0
···
X1
Xt
Ht
ŷt
(a) Base causal diagram (prefix-only).
···
X0
Xt
Gt
γt
Ht
ŷt
αt
βt
At (b) Admissible causal augmentation: αt = γt ◦βt .
···
X0
Xt
Ht
ŷt
αt
Xt+1
At
(c) Future-informative augmentation: At depends on Xt+1 .
Figure 6: Information-regime diagrams for GT/Transformer updates. Left: strict causal prefix functor. Middle: admissible auxiliary channel (Definition 3) factors through a prefix-derived object Gt , so the update remains in Regime C. Right: future-informative auxiliary channel (Definition 4) keeps the same type αt : At → Ht , but At now depends on future-indexed objects, placing the model in Regime A. Model family: GT / Transformer variants
Regime C Strict causal
transformer_causal gt_causal gt_pred_prev_causal_detach
Regime E Endogenous self-conditioning
Regime A Augmented context
gt_pred_next_detach
gt_noncausal transformer_future_hint
Figure 7: Evaluation taxonomy by information budget. Architecture comparisons are fair only within a regime; Regime E is reported separately from both strict-causal controls and explicit augmented-context leakage baselines.
Definition 4 (Future-Informative Augmentation) An auxiliary map αt is future-informative iff no such factorization exists through prefix-only objects, equivalently the induced update depends on objects indexed by > t in Iet . Operationally, this means causal self-attention by itself is insufficient to guarantee strict causal evaluation: the geometric branch can alter the information regime depending on the source of its inputs.
Variant
Auxiliary map type
Regime
transformer_causal gt_causal gt_pred_prev_causal_detach gt_noncausal transformer_future_hint gt_pred_next_detach
none beyond prefix functor causal local map (left-padded) shifted self-conditioning (t ← t−1) symmetric local map (can mix t±1) explicit future-informative hint path same-index detached predicted-hint path
C C C A A E
Table 4: Model-to-regime mapping via auxiliary morphisms. The deciding factor is whether the auxiliary map factors through the prefix diagram. 23
16.4
Language-Model Instantiation
For token sequence length L: • token positions 1, . . . , L are 0-simplices; • adjacency (i, i + 1) defines 1-simplices; • global operator is causal self-attention; • geometric operator is a local 1D mixer (causal or symmetric depending on regime). In the leakage-controlled variants we parameterize the geometric branch input source: ut ∈ {ht , detach(pt E), detach(pt−1 E)} , pt = softmax(Wout ht /τ ). The strict causal self-conditioned mode uses only detach(pt−1 E). Which GT family is ablated here? All variants in this leakage protocol are GT-Lite style instantiations (Transformer + local depthwise Conv1D geometric mixer). They do not include GT-Full simplicial transport blocks or GT-MoE routing. This is intentional: the protocol isolates informationregime effects without adding capacity/routing confounds from the larger GT families. 16.5
Revised LM Interpretation
The revised empirical picture is: • In Regime C, gt_causal and transformer_causal gt_pred_prev_causal_detach tracks this regime.
are
comparable;
• In Regime E, gt_pred_next_detach can move far below Regime C and, at larger width, approach the low-perplexity band of Regime A without becoming the same mechanism. • In Regime A, gt_noncausal and transformer_future_hint can show near-collapse in perplexity. Therefore, large gains in Regime A are reported as augmented-information effects rather than architecture-only gains, while Regime E is treated separately as endogenous self-conditioning. Reporting standard. Each LM table below includes an explicit regime label (C/E/A); the reproducibility materials record the corresponding geo_causal flag, geometric input source, and shift policy for predicted hints. 16.6
GT LM Results by Regime
This section reports the PTB and WikiText-2 causal-ablation runs directly, and refers to the main paper for the corresponding WikiText-103 comparison, grouped by regime rather than by model name alone. The interpretation standard is: 1. make architecture claims only within the same information regime; 2. interpret cross-regime gains as information-budget effects. WikiText-2 (L=2, d=256) snapshot. The current WikiText-2 run shows the expected three-way split: strict-causal variants remain close to each other, the endogenous self-conditioning regime drops sharply below them, and the explicit augmented-context variants stay in the same low-perplexity band. In this table, all gt_* entries are GT-Lite style models (Transformer + local depthwise Conv1D geometric branch), not GT-Full or GT-MoE. PTB (L=2, d=256) snapshot. PTB shows the same qualitative pattern: strict-causal variants cluster together, the endogenous self-conditioning regime nearly matches the leakage baselines, and the explicit augmented-context variants collapse perplexity. 24
Model transformer_causal gt_causal gt_pred_prev_causal_detach transformer_future_hint gt_pred_next_detach gt_noncausal
Regime
Val PPL
Test PPL
C C C A E A
178.013 169.707 174.075 2.073 1.624 1.051
163.918 157.744 159.774 2.023 1.593 1.051
Table 5: WikiText-2 causal-ablation results grouped by information regime.
Model transformer_causal gt_causal gt_pred_prev_causal_detach transformer_future_hint gt_pred_next_detach gt_noncausal
Regime
Val PPL
Test PPL
C C C A E A
134.767 137.550 134.735 1.509 1.102 1.075
124.474 127.165 126.499 1.380 1.054 1.050
Table 6: PTB causal-ablation results grouped by information regime.
The endogenous self-conditioning variant (gt_pred_next_detach). The most informative behavior in these tables is the transition profile of gt_pred_next_detach: it often begins close to strict-causal models and then shifts toward augmented-context performance as training proceeds. This supports an endogenous self-conditioning view: the model uses a detached predicted-token embedding (computed from prefix-derived states) to modulate the geometric branch. No future token is directly injected, but the internal predictive signal can become strong enough to emulate some benefits of augmented-context updates. Interpretive significance. This suggests a productive gray zone between strict-causal and fully augmented regimes. Rather than treating all side channels as equivalent, we distinguish: (i) explicit future-informed channels, and (ii) inference-derived endogenous self-conditioning channels. The latter appears to be a key mechanism worth further study in GT-style architectures. Smaller-model check (L=2, d=64). At lower width, gt_pred_next_detach still improves strongly over Regime C baselines but no longer reaches Regime A performance. On PTB, test perplexity is 16.567 for gt_pred_next_detach, compared with ∼ 208–222 (Regime C) and ∼ 1.144–4.940 (Regime A). On WikiText-2, it reaches 44.050, between ∼ 288–306 (Regime C) and ∼ 1.421–10.160 (Regime A). This indicates the transition behavior is likely capacity-dependent. Dataset
Model
Regime
Test PPL
PTB (d = 64) PTB (d = 64) PTB (d = 64) PTB (d = 64) PTB (d = 64) PTB (d = 64)
transformer_causal gt_causal gt_pred_prev_causal_detach gt_pred_next_detach transformer_future_hint gt_noncausal
C C C E A A
222.238 207.721 216.909 16.567 4.940 1.144
WikiText-2 (d = 64) WikiText-2 (d = 64) WikiText-2 (d = 64) WikiText-2 (d = 64) WikiText-2 (d = 64) WikiText-2 (d = 64)
transformer_causal gt_causal gt_pred_prev_causal_detach gt_pred_next_detach transformer_future_hint gt_noncausal
C C C E A A
305.511 288.030 295.857 44.050 10.160 1.421
Table 7: Small-model (L = 2, d = 64) causal-ablation snapshot.
25
Capacity-transition index.
Define
transition_gain =
PPLcausal_best − PPLpred_next , PPLcausal_best − PPLaug_best
where causal_best is the best strict-causal control and aug_best is the best augmented-context control. This measures how close gt_pred_next_detach gets to augmented-context performance. Dataset
d
Causal best
Pred-next
Transition gain
PTB WikiText-2 PTB WikiText-2
256 256 64 64
124.474 157.744 207.721 288.030
1.054 1.593 16.567 44.050
1.000 0.997 0.925 0.851
Table 8: Capacity dependence of gt_pred_next_detach: near-augmented performance at d = 256, but intermediate performance at d = 64.
WikiText-103. The corresponding WikiText-103 comparison is reported in the main paper, so we do not duplicate a second table here. The same qualitative pattern persists there: Regime C variants cluster together, Regime E remains substantially better than Regime C, and Regime A stays lowest because it admits future-informative side channels. Interpretation and Scope The main empirical lesson is that the Geometric Transformer benefits should be interpreted as structure-aligned inductive bias, not universal dominance. In regime-matched causal comparisons, gains are modest and task-dependent. In augmented-context settings, large gains can occur for multiple model families due to extra information pathways.
17
Algorithmic and Implementation Details
This appendix collects algorithmic and implementation details used in the experiments. 17.1
Quadratic Kan Extension Block
Algorithm 1 Quadratic Kan Extension Block Require: token states h0:S−1 , simplex set I, value base v, optional causal mask Ensure: updated states h′0:S−1 1: Construct simplex values V (σ) for all σ ∈ I 2: Construct simplex keys Kσ = WK V (σ) 3: for t = 0 to S − 1 do 4: Qt ← WQ ht 5: It ← {σ ∈ I : max(σ) ≤ t} if causal mask is enabled; otherwise It ← I 6: w(t, σ)P ← softmaxσ∈It (Q⊤ t Kσ ) 7: mt ← σ∈It w(t, σ) V (σ) 8: h′t ← LN ht + MLP(mt ) 9: end for 10: return h′0:S−1 This block implements global simplicial pooling and scales quadratically in sequence length under the present construction. 17.2
Incidence-Restricted Kan Block
This block implements the linear-time approximation in which only incident edges contribute to each token update. 26
Algorithm 2 Incidence-Restricted Kan Block (Edge-Only) Require: token states h0:S−1 , value base v0:S−1 , regime flag Ensure: updated states h′0:S−1 1: for t = 1 to S − 1 do 2: et ← ψ([vt−1 , vt ]) 3: end for 4: for t = 0 to S − 1 do 5: mt ← 0 6: if t ≥ 1 then 7: mt ← mt + ϕ(et ) 8: end if 9: if regime is noncausal and t + 1 ≤ S − 1 then 10: mt ← mt + ϕ(et+1 ) 11: end if 12: h′t ← LN(ht + mt ) 13: end for 14: return h′0:S−1
17.3
Predict–Detach Carrier Construction
For self-conditioned regimes, the value base is not taken directly from teacher-forced hidden states at future positions. Instead we construct detached predictive carriers: êt = detach softmax(ℓt /T ) E , ℓt = Wo ht . (18) These carriers may then be used in either quadratic or incidence-restricted Kan aggregation, as well as in TopoCoend neighborhood construction. 17.4
Leakage Test Procedure
To verify that a regime does not exploit gold future-token leakage, we use the following diagnostic. 1. Randomly shuffle the target tokens xt+1 within a batch. 2. Train the model for a short schedule under the candidate regime. 3. Observe whether perplexity remains sensible or collapses. In explicitly gold-noncausal regimes, perplexity collapses toward 1 even under shuffled targets, indicating that the model is exploiting an invalid information path. In predict-and-detach regimes, perplexity remains high under shuffling, supporting the claim that no direct gold-future leakage is present. 17.5
Hyperparameters and Training Details
The released experiments fall into two families. Main causal language-model comparisons. The summary tables and plots under plots/PTB_Spark, plots/Wiki-2-Spark, and plots/Wiki103_Spark are produced by scripts/run_lm_causal_comparison_kan_topocoend.py and the corresponding training scripts it invokes. These runs use: • optimizer: AdamW • learning rate: 3 × 10−4 • weight decay: 10−5 • batch size: 32 • context length: 128 27
• number of layers: 2 • hidden dimension: 256 • attention heads: 4 • training iterations: 5000 • reporting cadence: every 100 steps • evaluation cadence: every 1000 steps Self-conditioned regimes use temperature T = 1.0 unless otherwise noted. TopoCoend uses topo_k = 16 and topo_dim = 16. The released LM figures are single-seed runs, consistent with the checklist statement that we do not report multi-seed error bars. Structured completion runs. The block-completion results are produced by scripts/ket_experiments/ket_unified_harness.py and scripts/ket_experiments/train/train_ket_block.py. The reported PTB/WT2/WT103 higher-depth block comparisons use: • matched Transformer and incidence-KET backbones • optimizer: AdamW • learning rate: 3 × 10−4 • weight decay: 10−2 • context length: 128 • hidden dimension: 64 • layers: 8 and 16 • attention heads: 4 • block size: 4 • training steps: 2000 • evaluation cadence: every 250 steps • denoising steps: 8 • evaluation cap: 100 held-out batches These structured-completion runs use the local benchmark text files under scripts/ket_experiments/data/ and apply the same GPT-2 tokenizer across the matched backbones. Gradients are clipped to norm 1.0. The released block runs record seeds 7, 11, 17, 19, and 1337 in the accompanying all_runs.csv files. 17.6
Hardware and Runtime Measurement
The figures and tables in the main LM comparison were generated from the released DGX Spark runs in plots/*_Spark. Additional pilot runs were also exercised on Apple MPS hardware during development. The structured-completion harness records the same runtime fields in scripts/ket_experiments/results/logs/. The hardware families used in the released artifacts are: • Apple MPS hardware • NVIDIA DGX Spark (Blackwell GPU) Runtime is measured end-to-end by wall-clock time. Iterations per second are computed as iters/sec =
number of iterations . total wall-clock seconds
Quadratic KET benefits disproportionately from modern GPU acceleration, which significantly narrows its runtime gap relative to incidence-based or baseline models. 28
18
Broader Impact
This paper describes a new family of Kan Extension Transformers, which are useful in building structured language models. The applicability of this framework extends to all of language models, and consequently carries the usual societal risk of deployment of models that might hallucinate or behave in ways that are undesirable. The focus of this paper was building the basic methodology, and not on fine-tuning the model to obtain satisfactory performance in particular regimes.
References Barbero, F., Bodnar, C., Sáez de Ocáriz Borde, H., Bronstein, M. M., Veličković, P., and Liò, P. (2022). Sheaf neural networks with connection laplacians. In Topological, Algebraic and Geometric Learning Workshops, volume 196 of Proceedings of Machine Learning Research, pages 28–36. Battiloro, C., Testa, L., Giusti, L., Sardellitti, S., Di Lorenzo, P., and Barbarossa, S. (2024). Generalized simplicial attention neural networks. IEEE Transactions on Signal and Information Processing over Networks, 10, 1–16. Bodnar, C., Di Giovanni, F., Chamberlain, B. P., Liò, P., and Bronstein, M. M. (2022). Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. In Advances in Neural Information Processing Systems, volume 35, pages 18527–18541. Goh, C. W. J., Bodnar, C., and Liò, P. (2022). Simplicial attention networks. Gold, E. M. (1967). Language identification in the limit. Information and Control, 10(5), 447–474. Hajij, M., Zamzmi, G., Papamarkou, T., Miolane, N., Guzmán-Sáenz, A., Ramamurthy, K. N., Birdal, T., Dey, T. K., Mukherjee, S., Samaga, S. N., Livesay, N., Walters, R., Rosen, P., and Schaub, M. T. (2022). Topological deep learning: Going beyond graph data. Hajij, M., Bastian, L., Osentoski, S., Kabaria, H., Davenport, J. L., Dawood, S., Cherukuri, B., Kocheemoolayil, J. G., Shahmansouri, N., Lew, A., Papamarkou, T., and Birdal, T. (2025). Copresheaf topological neural networks: A generalized deep learning framework. arXiv. Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems. Li, X. L., Thickstun, J., Gulrajani, I., Liang, P., and Hashimoto, T. B. (2022). Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, volume 35, pages 4328–4343. Mac Lane, S. (1971). Categories for the Working Mathematician. Springer-Verlag, New York. Graduate Texts in Mathematics, Vol. 5. Mahadevan, S. (2025a). Categories for AGI. Book manuscript. Mahadevan, S. (2025b). CLIFF_CatAgi: Categories for AGI local research interface. GitHub repository. Mahadevan, S. (2025c). Large causal models from large language models. Mahadevan, S. (2026). Prometheus: Automating deep causal research integrating text, data and models. May, J. (1992). Simplicial Objects in Algebraic Topology. University of Chicago Press. Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205. Richter, B. (2020). From Categories to Homotopy Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press. 29
Riehl, E. (2017). Category Theory in Context. Aurora: Dover Modern Math Originals. Dover Publications. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008. Zhang, H., Liu, X., and Zhang, J. (2022). HEGEL: Hypergraph transformer for long document summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10167–10176.
30