Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers Haris Riaz and Hyungji Kim and Mihai Surdeanu Department of Computer Science University of Arizona Tucson, AZ, USA § Code

Abstract

TG

82

1

SyntaxGym score

arXiv:2608.06111v1 [cs.CL] 6 Aug 2026

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate where and how the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly— while for encoders it is best added directly to the input embeddings, composing with each encoder’s native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3% while simultaneously reducing perplexity by 9.0% over a base model with no syntactic supervision—a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2% over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.

80 78

TreeReg

Pushdown LM SiPE (ours) PLM

TPT

76 74 72

Transformer-XL (no syntactic supervision)

0 parser-free

1 single parse

// 300 beam marginalization

Inference cost (parse trees evaluated per sentence) Figure 1: Our Syntax-Informed Positional Embeddings (SiPE) move the Pareto frontier between syntactic supervision and inference cost. Each point plots a method’s SyntaxGym score (Table 2) against the number of parse trees it evaluates per sentence at inference. Joint syntactic LMs (TG (Sartran et al., 2022), Pushdown LM (Murty et al., 2023), PLM (Qian et al., 2021)) recover p(x) by marginalizing over ≈300 candidate parses; parser-free methods (TreeReg (Nandi et al., 2025), Tree-Planted Transformers (Yoshida et al., 2024)) inject syntax only during training and discard the parser at runtime, making inference cheap but sacrificing some syntactic knowledge. SiPE conditions on a single parse, retaining most of the benefit of full marginalization at 1/300th of its inference cost.

the impact of LLMs is substantial, several linguistic simplifications were introduced in the underlying architecture to improve scalability. One key choice was the use of positional embeddings (PE) to encode each token’s location within a sequence. While this approach preserved computational efficiency and scalability, it reduced positional information to a simple ordering mechanism—sufficient to indicate where tokens appear, but insufficient to capture the syntactic relations that form the scaffolding for downstream semantics. That is, without explicitly modeling syntactic compositionality, semantic frames (Baker et al., 1998; Kingsbury and

Introduction

The transformer architecture (Vaswani et al., 2017) forms the foundation of modern LLMs. While

1

Figure 2: Left: From bottom to top, the figure shows the correspondence between dependency arcs, binary head tree (BHT), and hexatags (left-shifted) for the sentence “she watches funny videos.” Right: Our prior-injection method for absolute positional embeddings (input-pathway injection). From bottom to top, each token (at its first subword; subsection A.1) selects one row from each of four lookup tables: the token embedding (|V | × D; omitted from the figure for brevity), the non-terminal tag T embedding EN νp (5 × D), the terminal tag embedding Eτp (2 × D), and the positional embedding (N × D). The selected rows are combined via an injection operation ⊕ (addition by default; we ablate concatenation, weighted-addition and addition to skip connection in Appendix E) into the syntax-informed input xp (Equation 2). The transformer maps these inputs to contextual states hi , and at each masked position i ∈ M three prediction heads recover the token xi , terminal tag τi , and non-terminal tag νi , yielding L = LMLM + LT + LNT ; one masked position is shown, and head colors match the embedding rows they supervise. For decoders, next-token prediction replaces MLM: the token head at position i predicts token i+1 at every position, while the tag heads are applied only at first-subword positions and predict the tags of the next tagged position (subsection A.1).

Palmer, 2002) are more likely to be extracted incorrectly, increasing the risk of misinterpretation for LLM-based agents. Consider, for example, this hypothetical agent request: “Please move this large file to another folder.” To execute the requested action correctly, the move predicate must be associated with its arguments: the object to be moved (file) and the destination (folder). Absolute positional encodings (Devlin et al., 2019; Liu et al., 2019), provide no signal to link this predicate to its arguments. Relative positional encodings, e.g., DeBERTa (He et al., 2021) and Transformer-XL (Dai et al., 2019), weaken this signal. For instance, the relative distance between move and folder is large (six words), even though a direct oblique syntactic dependency connects the verb to its modifier (De Marneffe et al., 2021). Partially due to this shallower representation, language models (LMs) still exhibit weak compositional generalization (Guo et al., 2020). Prior work injects hierarchical inductive bias by constraining self-attention using constituency or dependency structure, often via quadratic token-token interac-

tions and hard-coded attention masks (Sartran et al., 2022; Zhao et al., 2024; Murty et al., 2023; Qian et al., 2021; Xie et al., 2021). This is a complex process that requires considerable changes in the transformer architecture and which increases the attention mechanism’s computational overhead (e.g., Pushdown LMs (Murty et al., 2023) maintain a recursively updated stack over the sequence that reshapes the attention pattern at every step). The main contributions of our paper are: (1) We introduce Syntax-informed Positional Embeddings (SiPE) that augment a model’s positional pathway with a lightweight prior derived from linearized dependency trees—coarse directional indicators relating each token to its syntactic governor (§3), learned during pre-training via an auxiliary indicator-prediction objective alongside the LM loss. (2) SiPE composes with all three dominant positional encoding families—absolute (Liu et al., 2019), relative (He et al., 2021; Dai et al., 2019), and rotary (Warner et al., 2024)—without modify2

ing self-attention or any other Transformer component.

et al., 2025). However, most methods encode syntax by constraining self-attention via pairwise token interactions and hard-coded attention masks, adding parameters and training complexity (Xie et al., 2021; Sartran et al., 2022; Qian et al., 2021; Murty et al., 2023). We instead sidestep this by encoding dependency structure as two lightweight prior vectors derived from Hexatagging, and apply these priors to the positional embedding pathway. Modern Transformers encode position through three dominant schemes: absolute embeddings added to the input residual (Liu et al., 2019), relative embeddings indexed by offset i − j inside attention (He et al., 2023; Dai et al., 2019), and rotary embeddings that rotate queries and keys by position-dependent angles (Su et al., 2023). We show that our method seamlessly composes with all three PE types (subsection 3.1).

(3) For encoders, SiPE improves BLiMP (Warstadt et al., 2023) syntactic generalization across all three families (up to +2.3% relative for ModernBERT), with gains compounding under continued pre-training on BLLIP-LG (+3% for DeBERTav3 on out-of-domain evaluation), indicating the prior generalizes beyond its pre-training distribution (Table 3). The same models also improve on GLUE (Wang et al., 2019) at both small and large scale (tables Table 4, 5 and 6). (4) For autoregressive decoders like TransformerXL, injecting through the positional pathway alone is the strongest recipe: it achieves the best SyntaxGym (Hu et al., 2020) score of any configuration of our method (80.60, +10.3 relative improvement over the no-syntax baseline) while cutting perplexity by 9.0% (relative), and drives a significant +8.2% relative improvement on GLUE for the same model (Table 6).

2

Related Work, Background, and Notations

Prior work shows that pretraining on formal languages or injecting tree-based signals can improve syntactic generalization and data efficiency (Hu 1 Dependency tags are predicted in parallel by a lightweight DeBERTa-v3-base classifier whose overhead on top of the encoder’s is linear in input length; see Appendix B. Implemented in-house with a multitask-learning encoder, two per-token tag heads: https://clulab.org/processors/

3

⇒

⇒

⇒

⇒

→

(5) As shown in Figure 1, prior syntactic LMs sit at two extremes: parsing at inference (expensive marginalization over many parse trees) or only during training (discarding syntax at runtime, with weaker syntactic generalization). SiPE moves the Pareto frontier between these by conditioning on a single parse at inference. The injection of syntactic information adds no asymptotic cost over selfattention,1 and unlike most prior syntactic LMs which jointly model syntax and language, SiPE leaves downstream usage unchanged—models architecturally remain compatible with continual pretraining, supervised fine-tuning and reinforcement learning. Empirically we outperform parser-free approaches—Tree-Planted Transformers (Yoshida et al., 2024) and TreeReg (Nandi et al., 2025)—on both BLLIP-LG perplexity and SyntaxGym (Table 2).

→

Hexatagging. In this work, we leverage Hexatagger (Amini et al., 2023), a “parsing-as-tagging” dependency parser that assigns each token two types of discrete syntactic labels (analogous to position or subword IDs): (a) the position of its syntactic governor (which can further be decomposed into two subtypes) and (b) the dependency relation. This formulation can be mapped to simple embedding lookup tables that can be linearly combined with token embeddings without modifying self-attention. Hexatagger maps a projective dependency tree to a binary head tree (BHT, a special form of constituency tree) by binarizing the structure and labeling each internal node with L/R to indicate whether the span head lies in the left or right subtree, then linearizes the BHT via an in-order (left-corner) traversal. Each visited node yields a tag encoding its attachment direction (left vs. right child), and for non-terminals, the head-direction (L vs. R). This yields a fixed projective tag inventory with terminal tags and , plus non-terminal tags L , R , L , and R (Figure 2, left). This avoids unbounded (length-dependent) tag sets whose cardinality grows with the input length; instead each token’s tag can be predicted independently with a linear classifier. In our implementation, we further append a dedicated EOS non-terminal (via a left shift of between-token non-terminals) to obtain one non-terminal label per token.1 In this work, we omit dependency relation labels, finding that terminal/non-terminal hexatags alone provide sufficient syntactic inductive bias to improve downstream language modeling performance.

3

Approach

3.3

Both training and inference operate on a hexatagged sequence: the input is first tagged by our parser (subsection A.1). Encoders train with the standard MLM objective; decoder training is standard next-token prediction, as each token carries its own tags and truncating the sequence at any prefix leaves the retained tags intact. 3.1

To test prior injection on autoregressive decoders, we use Transformer-XL (Dai et al., 2019) (hereon referred to as TXL), which is one of the earliest “LLM-like” architectures carrying a sinusoidal relative-position encoding directly in the attention score and is a common syntactic-LM baseline (Sartran et al., 2022; Zhao et al., 2024). Following prior work, we disable cross-segment caching (so relative positions apply only within the current window) and replace the adaptive softmax with a tied linear projection, making TXL a purely causal decoder. For a query at position i and key at position j in head n of layer ℓ, the attention score splits into a content term and a position term:

Deriving a Positional Syntactic Bias

An inductive bias can enter the PE layer in three ways (Figure 2): (a) entangled with the token embedding, applied to the input before any positional or attention information is introduced; (b) composed with the default PE scheme under some operation; or (c) disentangled, a separate additive term in the attention score that modulates each head. Linearizing the binary head tree into discrete tags (the hexatagging procedure of section 2) yields two embedding tables, one per tag type, composable with the PE mechanism under any of these operations. At position p they contribute:
mp · ETτp + EN (1) νp

(ℓ)

Ai,j,n =

(ℓ)

(ℓ)

(ℓ)

qi,n + un , kj,n + qi,n + vn , WR ri−j,n {z } | {z } | (ℓ) (ℓ) ACi,j,n (content) BDi,j,n (position)

(3) where ri−j,n is the sinusoidal vector for offset i−j, WR projects it into head n’s dhead -dimensional subspace, and un , vn are the learned content and position bias vectors. Notably, this content-conditioned relative-position term remains competitive at frontier scale: the concurrent 975B-parameter openweights model Inkling (Thinking Machines Lab, 2026) abandons RoPE for a relative positional bias of the same query–offset-embedding form, citing better quality and length extrapolation. Because BD depends only on i−j, it carries no information about which tokens occupy positions i and j or their syntactic role. We supply that missing signal from the hexatag embeddings ETτj + EN νj and study where in the score it should enter, across five injection sites treated in turn below.

where ET ∈ R|T |×d and EN ∈ R|N |×d are learned tables (|T |=2 terminals, |N |=5 non-terminals), and mp ∈ {0, 1} is the first-subword mask, equal to 1 only at each word’s first subword so the hexatag is injected once per word (subsection A.1). Tables are randomly initialized and trained jointly with the LM objective. Throughout the paper, every occurrence of ETτp + EN νp is implicitly multiplied by mp ; we omit the mask from later equations for readability, so continuation subwords contribute no tag term anywhere. 3.2

Transformer-XL: Syntactic Priors in an Autoregressive Decoder with Relative PE

RoBERTa: Absolute Positional Embeddings

3.3.1 Input-Side Injection (ADD) The simplest approach adds the tag embeddings to the token embedding, so the prior enters the input residual at layer 0 and reaches the score only indirectly, through the content projections WQ , WK that build the query and key:

We begin with the simplest case. Adopting strategy (a), hexatag priors are injected directly into the input embedding, which combines a learned absolute position vector pp with the token, segment, and tag terms:
seg T N xp = etok (2) p + p p + e p + m p · E τp + E ν p

T N h(0) = etok p p + mp (Eτp + Eνp )

(4)

with mp ∈ {0, 1} masking tag-free positions. The relative-position encoding itself is untouched; the prior entangles with lexical content rather than with position.

The tag term sits at the same level as pp , so the model learns to attend to a richer composite input without architectural conflict2 . 2 However, the input embedding is not the only place in the transformer where we can inject these priors; see appendix E and figure 9.

4

chosen: a query at i adds the same syntactic bias toward key j whether or not it already attends to the offset i−j. This is the opposite of the multiplicative coupling (Eq. 6), where that same bias is scaled by BD, and therefore drops to nearly zero when the query does not attend to the offset (i.e., when BD ≈ 0). In our experiments, we find that this ungated form is weaker (refer to Table 2 and ablations in Table 8, appendix A.3) than the multiplicatively entangled bias, which indicates that the bias from SiPE helps most when it augments an attention preference the position pathway has already established, rather than when it acts on its own.

3.3.2 Position-Pathway Injection (PP-SiPE) The approach which we find works best is to leave the input residual clean and route the tag straight into the position term. Using a layer-specific pro(ℓ) jection WE , we map the tag at key j into head n’s subspace and take its inner product with the (ℓ) same effective query qi,n + vn that appears in the position term BD (Eq. 3):  (ℓ)  (ℓ) (ℓ) ci,j,n = √d1 qi,n + vn , WE (ETτj + EN ) n ν j head (5) where [ · ]n selects head n’s dhead -dimensional block and where ETτj and EN νj are the terminal and nonterminal hexatag embeddings at key j. Eqs. 3 and 5 are deliberately parallel: the one effective (ℓ) query qi,n + vn poses two questions — “how well do I align with this offset?” (BD) and “how well do I align with this tag?” (c). The second is therefore a syntactic counterpart of the first, on the same per-head scale. We couple it to the position term multiplicatively, leaving the content term AC untouched:
e(ℓ) = AC(ℓ) + 1 + c(ℓ) · BD(ℓ) A i,j,n i,j,n i,j,n i,j,n

3.3.4

Injecting Syntactic Bias in Self-Attention Motivated by nearly all prior work which imposes syntactic constraints on attention weights as either a hard or soft bias, (Strubell et al., 2018; Murty et al., 2023; Zhao et al., 2024; Omote et al., 2019; Xie et al., 2021; Yoshida et al., 2024), we propose our hexatag-derived bias applied to attention logits and the attention output, leaving the input residual, AC, and the positional pathway BD untouched (0) (hp = etok p ). The mechanism follows Shaw et al. (2018), who augment attention with learned key-side and valueside vectors. We project the summed hexatag emtag,(ℓ) bedding at key j two ways: through WK to form a key bias that is added to the attention score e(ℓ) , and through Wtag,(ℓ) to form a value bias A i,j,n V

(6) In our syntactic evaluation, we observe the multiplicative form (Eq. 6) to outperform all other approaches (refer to experiments in table 2 and ablations in table 7, appendix A.2): its correction term c · BD gates by offset alignment, applying the syntactic adjustment in proportion to how strongly the query already attends to that offset. Where the query-offset alignment is strong (|BD| large), the tag exerts a large pull; where it is near zero, the tag exerts almost none, so syntax modulates an existing positional preference rather than acting in isolation.3 The offset embedding ri−j,n itself is never modified.

(ℓ)

that is added to the attended output zi,n : e(ℓ) = A(ℓ) + √ 1 A i,j,n i,j,n d

head

(ℓ)

zi,n =

3.3.3 Fully-Disentangled Injection A third option completely disentangles the tag from both pathways. Rather than multiplying c with BD, we add it as an independent third component of the logit, alongside the content and position terms and on the standard attention scale: e(ℓ) = AC(ℓ) + BD(ℓ) + c(ℓ) . A i,j,n i,j,n i,j,n i,j,n

X

(ℓ)

tag,(ℓ)

qi,n , [WK

(ETτj + EN νj )]n

(8)

  (ℓ) (ℓ) tag,(ℓ) αi,j,n vj,n + [WV (ETτj + EN νj )]n

j

(9) (ℓ) where αi,j,n are the softmax weights, and [ · ]n se-

lects multi-head attention head n’s block. Two things differ from the multiplicative alignment (ℓ) score ci,j,n of Eq. 5. First, that score was built (ℓ)

from the effective query qi,n + vn and scaled the position term BD, whereas this key bias uses the (ℓ) plain query qi,n and is added directly to the logit. Second, it is keyed on the tag at the single key position j rather than on a relative pair (i, j), which keeps the added cost linear (rather than quadratic) in sequence length. The projections may be learned

(7)

(ℓ)

Now ci,j,n depends on the query and the tag but (ℓ)

not on BDi,j,n , so its size is fixed once the tag is (ℓ) 3 e ≈ A at initialWe initialize WE small, so c ≈ 0 and A ization: training starts from the vanilla Transformer-XL score and learns the syntactic correction into it.

5

per layer (the (ℓ) superscript) or tied to a single pair shared across all layers; in our experiments we find the former (per-layer variant) outperforms the latter (shared variant) on syntactic generalization tasks, both on its own and when combined with the inputside injection of Eq. 4 (Appendix A.4).

the relative-position mechanisms (its bucketing, lookup table, and projection matrices) are left untouched. Tag priors and relative position thus interact only through the content stream, which is what makes the injection portable across positionalencoding families. More details, including how the prior propagates to all three score terms, is given in Appendix A.5.

3.3.5

Injecting Syntax into Position and Attention Simultaneously The final variant pairs our two strongest single-site injections: SiPE on the positional pathway (the multiplicative coupling, Eq. 6) and the per-layer attention-side biases. Combining the two, however, hurts: the joint variant scores below either injection strategy used on its own (Table 2, Appendix A.4). Injecting the same hexatag signal as both a positional and attentional bias is thus redundant rather than complementary; for Transformer-XL, the prior is best supplied once, through the positional pathway. Having established where and how in the score the prior should be combined, we next ask which layer it should enter from: at least for the decoder model in our experiments, syntax is best infused from layer 1 onward (section 5). 3.4

3.4.2 Position-Pathway Injection (PP-SiPE) Mirroring our Transformer-XL injection, we do not add the prior to the input embeddings (the token representations enter the network unchanged) and instead route it only into the relative-position term pos Sij of the attention score, leaving the content term (ℓ)

con untouched. A per-layer projection W Sij E maps the summed hexatag embedding at key j into head space, and we form a tag–query alignment against the content query:

cij = √1d

h

pos

con S̃ij = Sij + (1 + cij ) Sij .

DeBERTa-v3 (He et al., 2023) carries no inputlevel position embedding; positional information enters only at the attention score, through disentangled relative attention. For a query at i and key at j, the score splits into a content term and two relative-position terms:

3.5

|

{z pos Sij

(13)

ModernBERT: Injecting the Prior into Rotary Position Encoding

ModernBERT (Warner et al., 2024) like many other Large language models (Dubey et al., 2024; DeepSeek-AI, 2024; Yang et al., 2025; Kamath et al., 2025) delivers positional information through rotary positional embeddings (RoPE) (Su et al., 2023), which rotate the content query and key by position-dependent angles after the content projections, so that relative-position dependence emerges from the geometry of two absolute rotations. Each head dimension is split into frequency pairs indexed by t; pair t is rotated by θp,t = p ωt at absolute position p, with frequency ωt and a 2×2 rotation R(·). Because RoPE carries no input-level position term, we again consider two injection sites: the input residual, and the rotation angle itself.

⊤ r + Kjc Qrδ(i,j)⊤ , Qci Kjc⊤ + Qci Kδ(i,j)

| {z } con Sij

(12)

The coefficient cij scales the relative-position score in proportion to how strongly the query aligns with the key’s syntactic tag, so syntax sharpens an existing positional preference rather than acting on its own, exactly as in the multiplicative TransformerXL variant (Eq. 6).4

DeBERTa-v3: Injecting the Prior under Disentangled Relative Attention

Sij =

(ℓ)

Qci , WE (ETτj + EN νj ) ,

}

(10) where δ(i, j) is the bucketed relative offset and Qc , K c are the content query and key. As with the relative-PE models above, we consider two injection sites: the input residual, and the position term S pos . 3.4.1 Input-Side Injection Because DeBERTa has no absolute position embedding, the tag prior becomes the only positionallylocalized signal at the input layer:
tok seg T N h(0) (11) p = ep + ep + mp · Eτp + Eνp .

(ℓ)

4 WE is initialized small so that cij ≈ 0 and S̃ij ≈ Sij at initialization; training starts from the unmodified DeBERTa score and learns the syntactic correction into it.

The prior propagates through WQ and WK into the content terms of the attention score, while 6

BLIMP Example ✓ The keys to the cabinet are on the table. ✗ The keys to the cabinet is on the table.

Model

PLM (GPT-2) 75.10 (Qian et al., 2021) Transformer Gram- 73.50 mar (TransformerXL) (Sartran et al., 2022) Pushdown LM 75.60 (GPT-2) (Murty et al., 2023) TreeReg (GPT-2) 74.80 (Nandi et al., 2025)

Table 1: Examples from the BLIMP dataset (top) and SyntaxGym (bottom). In both datasets, p(✓) > p(✗), but in BLIMP this probability is computed for the complete sentence, whereas in SyntaxGym it is computed only over the region of interest.

3.5.1 Input-Side Injection Since RoPE has no input-level position embedding, the tag prior is added at the input residual exactly as in DeBERTa (Eq. 11, without the segment term, which ModernBERT does not use), and propagates into qp , kp before rotation; the rotary mechanism itself is untouched.

↓ -1.63

↑ +9.73

↓ -2.39 18.40 ↑ +1.23

82.50

↑ +12.87

↑ +0.40 19.90 ↓ -6.82

82.30

↑ +12.60

↓ -0.66 22.30 ↓ -19.70 80.00

↑ +9.45

Table 2: Language modeling and syntax evaluation of decoder LMs; all our SiPE variants (Ours) use Transformer-XL as the decoder model, and each baseline’s model is noted in parentheses. BLiMP and SyntaxGym are higher-is-better, while PPL (perplexity, computed on BLLIP-LG test split (Charniak et al., 2000)) is lower-is-better. ∆ columns report relative improvement or degradation against vanillaTransformer-XL (tokens). Best scores for each dataset are bolded.

The effective angle between query m and key j at frequency t is then (m − j) ωt + (∆θm,t − ∆θj,t ), so the tag enters as a syntactic phase shift on top of the positional one: the rotation that already encodes relative distance is nudged by how the two positions’ hexatags relate. Since the tag enters only as a rotation angle, it leaves the query and key magnitudes unchanged and perturbs only their direction (Appendix A.6).

Experimental Settings and Datasets

Constituency Parsing ↓ -0.27 29.80 ↓ -59.96 80.20

–

Syntax-Informed Embeddings with Hexatagging (Ours) SiPE (positional pathway) Input-side Injec- 73.82 ↓ -1.97 16.16 ↑ +13.26 76.97 ↑ +5.31 tion Position74.01 ↓ -1.71 16.95 ↑ +9.02 80.60 ↑ +10.29 pathway Injection Fully74.72 ↓ -0.77 16.66 ↑ +10.57 78.72 ↑ +7.70 disentangled Injection Attention Bias Projections 73.60 ↓ -2.26 16.73 ↑ +10.20 76.35 ↑ +4.46 shared across layers Projections 74.66 ↓ -0.85 16.24 ↑ +12.83 78.10 ↑ +6.86 Learned Per-Layer SiPE + Attention Bias Input-side Injec- 74.16 ↓ -1.51 15.89 ↑ +14.71 78.47 ↑ +7.36 tion Position74.44 ↓ -1.14 16.52 ↑ +11.32 77.97 ↑ +6.68 pathway Injection

which is added to the rotary angle before rotation, for both the query at position m and the key at position j:
q̃m,t = R m ωt + ∆θm,t qm,t , (15)
k̃j,t = R j ωt + ∆θj,t kj,t . (16)

4.1

Syntax ∆ Gym ↑ SyntaxGym

Tree-Planted Transformers (TPT) (Yoshida et al., 2024) TPT [zero] — — 47.50 ↓ -154.97 71.70 ↓ -1.90 TPT [rand.] — — 47.40 ↓ -154.43 69.00 ↓ -5.60 TPT [seq.] — — 47.30 ↓ -153.89 70.10 ↓ -4.09 TPT [dep.] — — 47.70 ↓ -156.04 77.10 ↑ +5.49 TPT [cons.] — — 45.50 ↓ -144.23 75.80 ↑ +3.71 TPT [bin.] — — 45.60 ↓ -144.77 73.00 ↓ -0.12

3.5.2 Position-Pathway Injection (PP-SiPE) Mirroring the Transformer-XL and DeBERTa injections, we can also route the prior into the position pathway instead of the input embedding, which for RoPE is the rotation angle. Using the summed hexatag embedding at position p, a per-layer projection (ℓ) Wδ produces a per-frequency phase offset
(ℓ) ∆θp,t = Wδ (ETτp + EN (14) νp ) t ,

Experimental Results

∆ PPL

No Inductive Bias Transformer-XL 75.30 – 18.63 – 73.09 (tokens) GPT-2 (tokens) 72.20 ↓ -4.12 21.60 ↓ -15.94 71.90 (Radford et al., 2019)

SyntaxGym Example ✓ The farmer near the clerks knows many people. ✗ The farmer near the clerks know many people.

4

BLiMP ∆ PPL ↑ BLiMP ↓

Xie et al., 2021; Omote et al., 2019; Yoshida and Oseki, 2022; Zhao et al., 2024; Murty et al., 2023). We argue these gains should also be assessed extrinsically, on downstream tasks where syntax is indirectly useful, and organize our experiments accordingly. Intrinsic evaluation: We test syntactic understanding on BLiMP (Warstadt et al., 2023) and SyntaxGym (Hu et al., 2020), both targeted minimalpair benchmarks contrasting grammaticality (Ta-

Prior work injecting syntax into Transformers focused mainly on intrinsic evaluation—directly testing syntactic understanding (Sartran et al., 2022; 7

Model

Acc. (%)

Our models on WikiText RoBERTa-base RoBERTa + SiPE (Input pathway) DeBERTa-base DeBERTa + SiPE (Input pathway) ModernBERT-base ModernBERT + SiPE (Input pathway) Transformer-XL Transformer-XL + SiPE (Positional Pathway)

(Liu et al., 2019; He et al., 2023; Warner et al., 2024) and Transformer-XL for 1M steps on a 50Mtoken, offline-hexatagged slice of WikiText-103 (Merity et al., 2016).7 We evaluate these models on a hexatagged BLiMP (Warstadt et al., 2023): using encoders, for each minimal pair we prefer the sentence with higher pseudo-log-likelihood (PLL) (Salazar et al., 2020) under the masked-LM objective; for the Transformer-XL decoder we instead use sentence log-likelihood (the sum of nexttoken log-probabilities). Accuracy is scored against BLiMP’s gold labels.

∆ (%)

70.68 – 71.68 ↑ +1.41 67.40 – 68.93 ↑ +2.27 49.94 – 51.10 ↑ +2.32 53.56 – 53.44 ↓ -0.22

OOD Eval: WikiText → BLLIP-LG RoBERTa-base 71.29 – RoBERTa + SiPE (Input pathway) 72.62 ↑ +1.87 DeBERTa-base 69.01 – DeBERTa-base + SiPE (Input pathway) 71.12 ↑ +3.06 ModernBERT-base 51.25 – ModernBERT-base + SiPE (Input path53.41 ↑ +4.21 way)

Extrinsic evaluation: To assess the downstream impact of injecting syntactic information directly in the transformer architecture, we carry out experiments on the GLUE benchmark (Wang et al., 2019), which contains a suite of real-world NLP tasks. We finetune our three encoder models as well as Transformer-XL (which were all previously pre-trained with SiPE on Wikitext) on each task using standard GLUE hyperparameters for each model (refer to Tables 4, 5 and 6 for results) and table 11 in appendix D.

Table 3: BLiMP accuracy results for models trained with SiPE. For each model family we report its most performant GLUE variant: input-pathway SiPE injection for the encoders and positional-pathway SiPE injection for the TransformerXL decoder; the OOD experiments likewise use the most performant GLUE variant of each family. For encoders, we use PLL-based accuracy. ∆: relative change w.r.t. the corresponding base model. OOD section: models pre-trained on WikiText, then continuously pre-trained on BLLIP-LG (Charniak et al., 2000). ↑ improvement, ↓ degradation. Bold: best relative improvement with our method.

4.2

Key Takeaways

Positional-pathway injection is the strongest recipe for a relative-PE decoder. On Transformer-XL, injecting the SiPE prior into the positional pathway gives our best decoder result— SyntaxGym 80.60 (+10.29 over the vanilla token baseline) with perplexity cut from 18.63 to 16.95 (Table 2). This carries through to GLUE, improving the TXL macro average from 68.17 to 73.78, a dramatic +8.2% relative gain (Table 6). Using only a single parse tree at inference, SiPE also surpasses parser-free approaches, TreeReg (Nandi et al., 2025) and all Tree-Planted Transformer variants (Yoshida et al., 2024), on both SyntaxGym and BLLIP-LG perplexity, cutting perplexity by roughly a quarter against the strongest such baseline (16.95 vs. 22.30 for TreeReg) and by more than half against the Tree-Planted Transformer variants (16.95 vs. 45.5–47.7) while matching or exceeding their syntactic generalization.

ble 1). BLiMP requires assigning higher probability to the grammatical sentence overall, whereas SyntaxGym compares probabilities only in the critical region where the ungrammaticality occurs. Following prior work (Zhao et al., 2024; Sartran et al., 2022), we pretrain Transformer-XL on BLLIP-LG (Charniak et al., 2000) with their hyperparameters, using the training splits of Hu et al. (2020).5 We report BLiMP accuracy via sentence log-likelihood, SyntaxGym via the standard suitelevel voting protocol, and BLLIP-LG test perplexity to confirm language-modeling ability is preserved. For our models, test sentences are hexatagged once by the parser before scoring, so our perplexity is conditioned on this single fixed parse, p(x | T̂ (x)), whereas joint syntactic LMs report a marginal p(x) approximated by summing over many candidate parses.6 In a second set of experiments, we pretrain three encoders: RoBERTa, DeBERTa, ModernBERT

For encoders, the simpler input-embedding injection is better, with gains that strengthen 7

Our goal is to study syntactic embeddings on real downstream tasks, so we pretrain on WikiText and then fine-tune on GLUE. For encoders, we also re-use the same checkpoints for syntactic evaluation as a separate BLLIP-LG pretraining run with its own hyperparameter tuning was beyond our academic budget.

5

We do not use the dataset’s syntactic annotations; we hexatag the sentences with an in-house parser (URL hidden for blind review). 6 This is the trade-off SiPE makes by design: a single parse at inference instead of expensive marginalization (Figure 1).

8

RoBERTa (base) Task

Base

+SiPE

DeBERTa (base) Base

+SiPE

+PP-SiPE

ModernBERT (base) Base

+SiPE

+PP-SiPE

31.41

31.77

27.22

26.09

24.97

25.00

24.45

23.26

(±2.02)

(±0.71)

(±2.37)

(±2.14)

(±2.02)

(±0.83)

(±1.85)

(±2.83)

CoLA

87.23

87.22

87.31

87.88

87.50

85.09

85.86

84.25

(±0.79)

(±0.52)

(±0.30)

(±0.54)

(±0.19)

(±1.01)

(±0.79)

(±1.38)

SST-2

83.24

83.36

84.02

84.95

84.88

83.53

83.09

82.45

(±0.21)

(±0.11)

(±0.31)

(±0.15)

(±0.13)

(±0.13)

(±0.45)

(±0.94)

QQP

77.65

78.24

68.07

68.88

67.94

62.51

61.83

65.79

(±0.50)

(±0.27)

(±0.08)

(±0.24)

(±0.29)

(±1.01)

(±0.70)

(±0.39)

QNLI

73.03

74.05

71.25

71.85

71.74

71.29

72.24

71.15

(±0.33)

(±0.17)

(±0.14)

(±0.14)

(±0.30)

(±0.03)

(±0.35)

(±0.11)

MNLI

65.76

64.66

62.94

63.06

64.14

62.21

63.06

63.90

(±0.74)

(±0.56)

(±1.23)

(±1.73)

(±1.68)

(±0.45)

(±2.13)

(±1.02)

RTE

79.11

81.41

76.20

76.98

76.33

79.67

81.21

79.49

(±0.30)

(±0.13)

(±0.56)

(±0.24)

(±1.32)

(±0.30)

(±0.78)

(±0.06)

STSB

80.96

82.02

80.49

81.11

83.45

79.74

82.82

81.56

(±0.44)

(±0.85)

(±0.84)

(±0.76)

(±0.71)

(±1.43)

(±1.08)

(±1.07)

MRPC

72.30

72.84

69.69

70.10

70.12

68.63

69.32

68.98

(±0.33)

(±0.42)

(±0.25)

(±0.39)

(±0.22)

(±0.08)

(±0.11)

(±0.37)

Macro

Table 4: GLUE evaluation of baseline encoders vs. SiPE augmentation, over 3 seeds (std in parentheses). Baselines span three PE schemes: RoBERTa-base (absolute), DeBERTa-v3-base (relative), ModernBERT-base (RoPE). SiPE is injected via two routes: input pathway (into token embeddings) and positional pathway (into positional information; relative/RoPE only). Best variant per task in bold. SiPE improves every architecture on the macro average—RoBERTa +0.75%, DeBERTa-v3 +0.59%, ModernBERT +1.0% relative.

RoBERTa (large) ModernBERT (large) Task CoLA SST-2 QQP QNLI MNLI RTE STSB

Base

+SiPE

Base

Transformer-XL

+SiPE

Task CoLA

30.71

30.36

25.06

24.58

(±0.51)

(±0.88)

(±1.21)

(±0.87)

88.95

87.50

84.44

85.47

(±0.35)

(±0.09)

(±0.52)

(±0.36)

84.31

85.10

82.54

82.55

(±0.18)

(±0.24)

(±0.39)

(±0.55)

80.18

82.68

61.99

62.11

(±0.18)

(±0.33)

(±0.52)

(±0.41)

74.72

74.43

72.03

72.36

(±0.10)

(±0.31)

(±0.53)

(±0.22)

65.94

64.62

63.54

61.49

(±0.45)

(±1.06)

(±0.29)

(±2.27)

81.89

82.34

79.03

80.05

(±0.59)

(±0.38)

(±0.41)

(±0.35)

MRPC 80.92

83.70

81.86

82.00

(±1.00)

(±1.15)

(±1.24)

(±0.76)

Macro 73.45

73.84

68.81

68.83

(±0.16)

(±0.13)

(±0.13)

(±0.35)

SST-2 QQP QNLI MNLI RTE STSB MRPC Macro

We did not pre-train DeBERTa-large with SiPE (input pathway/positional pathway), nor ModernBERT-large with SiPE (positional pathway), due to limited academic compute and budget constraints.

Base

+PP-SiPE

13.80

26.26

(±0.26)

(±0.22)

84.14

88.80

(±0.11)

(±0.44)

83.83

85.64

(±0.12)

(±0.21)

76.00

82.28

(±0.26)

(±0.29)

68.07

75.22

(±0.32)

(±0.14)

63.42

67.51

(±1.70)

(±0.29)

76.90

81.80

(±0.17)

(±0.12)

79.20

82.75

(±0.21)

(±0.26)

68.17

73.78

(±0.17)

(±0.05)

Table 6: GLUE results for the decoder-only Transformer-XL (TXL) baseline and its SiPE (positional pathway) variant (blue tint); best per task in bold. SiPE improves every task and gains +8.2% relative on the macro average (68.17→73.78), showing the positional pathway best augments TXL’s relative positional encoding.

Table 5: GLUE evaluation results for large encoder models.

out of distribution. Across all three encoder families—absolute (RoBERTa), disentangledrelative (DeBERTa-v3), and rotary (ModernBERT), input-pathway SiPE consistently outperforms the base model on GLUE (Table 4); the positionalpathway route also helps for the relative and rotary

schemes but yields slightly smaller gains, so for encoders mixing the prior directly with input embeddings is sufficient and most reliable. All three encoders likewise improve on BLiMP—+1.41% (RoBERTa), +2.27% (DeBERTa-v3), +2.32% (ModernBERT) (Table 3)—and these gains grow 9

Syntax-injection depth sweep 80

SyntaxGym accuracy Word-level PPL (right)

17.1 16.8

78 75

16.5

72 16.2

70

15.9

68

than being introduced in later layers. Complementing this view of where syntax should enter, Appendix C examines how the injected prior manifests in attention on BLiMP object–verb agreement: the encoders redistribute verb→object attention to mirror syntactic adjacency, whereas Transformer-XL surfaces the prior only weakly in its attention maps and instead converts it into the largest downstream semantic (GLUE) gains of any model we train.

Word-level perplexity (lower is better)

SyntaxGym accuracy (%)

82

1 2 4 6 8 10 12 14 k = first injected layer (k=1: all 16 layers injected; higher k later, fewer layers)

Figure 3: Layerwise SiPE injection sweep on TransformerXL. Injecting from layer k=1 (all layers) is best; skipping the first layer already drops SyntaxGym accuracy sharply, and later entry points degrade syntactic generalization further.

6

Conclusion

under continued pre-training on BLLIP-LG after WikiText, rising to +1.87%, +3.06%, and +4.21% respectively, indicating the prior generalizes beyond its original distribution rather than overfitting. 8

We proposed a simple strategy to infuse syntactic information in any transformer architecture. Our proposed modification is lightweight, adding only O(1, 000) parameters per model via an additive prior embedding, yet it yields reliable gains in both intrinsic and extrinsic evaluations and multiple encoder and decoder transformer variants.

5

Limitations

Where Should Syntax Enter? A Layerwise Injection Study

Our approach assumes access to hexatag annotations at inference time: each input sequence must be tagged by a dependency parser before being passed to the syntax-infused encoder. While this adds a preprocessing step, the tagger is lightweight and fast in practice (we use DeBERTa-v3-base as the hexatagger, 184M parameters) and runs efficiently relative to encoder pre-training and downstream finetuning. The approach also inherits the hexatagger’s errors, so its benefit may be smaller in domains where the parser is less accurate. We consider only coarse-grained syntactic priors derived from directional terminal and non-terminal tags. Preliminary pre-training experiments incorporating full dependency-relation labels (DEPREL) did not yield meaningful gains on GLUE (Appendix E, Figures 7 and 8), but we did not investigate this further. Our study of where to inject the prior is broad but not exhaustive. We study the main injection sites and coupling choices for each positional scheme, but under a fixed academic compute budget we could not pre-train every conceivable combination, particularly for the relative and rotary attentionside variants, where the design space is large. The configurations we report are therefore the strongest we found rather than a guaranteed global optimum, and our per-dataset results with RoBERTa (Figure 8) show that no single setting dominates across all tasks.

To understand at which depth syntax should interact with semantics, we sweep the entry point of the positional pathway SiPE bias (PP-SiPE) in Transformer-XL. For a model with N layers, we inject the bias from layer k onwards: layers 1, . . . , k−1 use the default positional information, while layers k, . . . , N receive the SiPE bias. We vary k ∈ {2, 4, 6, 8, 10, 12, 14} and compare injecting syntactic information in every layer (from k=1 onwards), measuring syntactic generalization (SyntaxGym) and word-level perplexity on BLLIPLG (Figure 3). Two findings emerge. First, syntax is most impactful at the very first layer: full injection (k=1) is clearly best, and skipping just the first layer (k=2) already causes a sharp drop in syntactic generalization: SyntaxGym falls from 80.6 to 73.5. Second, and more broadly, syntactic information matters most in the lower layers of a decoder like Transformer-XL: injecting it in later layers yields consistently weaker syntactic generalization. Together, these results indicate that for autoregressive models the positional pathway should carry syntactic structure from the earliest layer onward, rather 8

The small BLiMP drops are expected: BLiMP scores the full sentence, so language-modeling ability dominates over the ungrammatical span alone (Zhao et al., 2024). Most baselines in Table 2 (PLM, Transformer Grammars, TreeReg, most TPT variants) likewise fall below the vanilla token baseline on BLiMP.

10

Perhaps the biggest limitation, which we plan to address in future work, is fast autoregressive text generation with the decoder. Because the injected prior is conditioned on hexatags, generating each new token requires re-tagging the sequence produced so far. Our hexatagger is fast, so this is feasible in principle, but it is not compatible with standard KV-caching: as the sequence grows, the parser may revise the tags of earlier tokens, changing their injected representations and invalidating the cached keys and values. Efficient incremental decoding under a per-step syntactic prior is therefore an open research-engineering problem, and the direction we consider most important for future work. Finally, our experiments are limited to small models, English text, and the pre-training budgets used here. Whether the gains hold at larger scale, in other languages, or under substantially longer pretraining, especially with billion parameter LLMs remains to be seen; its unclear how the relative benefit of an explicit syntactic prior will scale as model and data scale increase.

Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. The llama 3 herd of models. Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020. Hierarchical poset decoding for compositional generalization in language. Advances in Neural Information Processing Systems, 33:6913– 6924. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decodingenhanced bert with disentangled attention. Preprint, arXiv:2006.03654. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. 2020. A systematic assessment of syntactic generalization in neural language models. Preprint, arXiv:2005.03692. Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases. Preprint, arXiv:2502.19249.

References

Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gael Liu, and 191 others. 2025. Gemma 3 technical report. ArXiv, abs/2503.19786.

Afra Amini, Tianyu Liu, and Ryan Cotterell. 2023. Hexatagging: Projective dependency parsing as tagging. Preprint, arXiv:2306.05477. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.

Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1989–1993.

Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJ corpus release 1. LDC2000T43. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. Preprint, arXiv:1901.02860.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.

Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255– 308.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. 2023. Pushdown layers: Encoding recursive structure in transformer language models. Preprint, arXiv:2310.19089.

DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.

Ananjan Nandi, Christopher D. Manning, and Shikhar Murty. 2025. Sneaking syntax into transformer language models with tree regularization. Preprint, arXiv:2411.18885.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman,

11

Yutaro Omote, Akihiro Tamura, and Takashi Ninomiya. 2019. Dependency-based relative positional encoding for transformer nmt. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 854–861, Varna, Bulgaria. INCOMA Ltd.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint, arXiv:1804.07461. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. Preprint, arXiv:2412.13663.

Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3735–3745, Online. Association for Computational Linguistics.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2023. Blimp: The benchmark of linguistic minimal pairs for english. Preprint, arXiv:1912.00582.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

Yikuan Xie, Wenyong Wang, Mingqian Du, and Qing He. 2021. Transformer with syntactic position encoding for machine translation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1536–1544, Held Online. INCOMA Ltd.

Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022. Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale. Transactions of the Association for Computational Linguistics, 10:1423–1439.

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. Preprint, arXiv:1803.02155.

Ryo Yoshida and Yohei Oseki. 2022. Composition, attention, or both? In Findings of the Association for Computational Linguistics: EMNLP 2022, page 5822–5834. Association for Computational Linguistics.

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguisticallyinformed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038, Brussels, Belgium. Association for Computational Linguistics.

Ryo Yoshida, Taiga Someya, and Yohei Oseki. 2024. Tree-planted transformers: Unidirectional transformer language models with implicit syntactic supervision. In Findings of the Association for Computational Linguistics ACL 2024, page 5120–5134. Association for Computational Linguistics.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.

Yida Zhao, Chao Lou, and Kewei Tu. 2024. Dependency transformer grammars: Integrating dependency structures into transformer language models. Preprint, arXiv:2407.17406.

Thinking Machines Lab. 2026. Inkling: Our openweights model. https://thinkingmachines.ai/ news/introducing-inkling/. Blog post; model weights at https://huggingface.co.

A

Preliminaries

A.1

Subword-Level Tag Injection

Hexatag annotations are produced at the word level: every word wi in a sentence carries a terminal tag ti and a nonterminal tag ni (drawn from small vocabularies T and N , respectively, where |T | = 2 and |N | = 5). The language model, however, operates on subword tokens: the tokenizer may split wi

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

12

into one or more subwords si,1 , . . . , si,ki . We must therefore specify how a word-level hexatag is associated with its corresponding subword positions. We adopt a assign hexatag to first-subword only strategy: the tag pair (ti , ni ) is attached to the position of the first subword si,1 , and the remaining subwords si,2 , . . . , si,ki receive no tag (in our implementation this is equivalent to receiving a tag ID of −100). Concretely, for each subword position p in the flattened input sequence, we define the following mask:   1 if p is the first subword mp = (17) of some word wi ,   0 otherwise.

its injected tag embedding is masked along with it (the mp term is zeroed), so the auxiliary head must infer the tag from surrounding context rather than copy it from its own input. The remainder of this section details the design choices and trade-offs for each injection strategy we study. These fall into a few families: adding the syntactic bias to the input embeddings; entangling it directly with the positional encoding; keeping it disentangled and adding it as a separate term to the attention score; injecting it as a separate attention-side bias; and combining any of these. We examine these strategies across four model architectures that span the major positional-encoding schemes: RoBERTa (encoder, absolute positional embeddings), DeBERTa-v3 (encoder, relative positional embeddings), ModernBERT (encoder, rotary positional embeddings), and Transformer-XL (autoregressive decoder, relative positional embeddings).

At subword positions with mp = 1, the prior contributes a learned tag embedding to augment the subword embedding; at positions with mp = 0 (continuation subwords), no prior is added, and the subword embedding is used as-is. Tag information still reaches continuation subwords, but only indirectly, through self-attention in subsequent layers. This design has two practical benefits: (i) The total tag embedding signal injected per word is invariant to its subword length ki , avoiding overamplification for words that break into many subwords (ii) it gives a clean one-to-one correspondence between words and tagged subword positions, which is required by the auxiliary tag-prediction objective: this objective is a per-position classification head that predicts the (terminal, nonterminal) tag pair, and it only contributes loss at positions where mp = 1. The assign hexatag to first-subword only is also the simpler choice here. The natural alternative would be to assign the same hexatag to all ki subwords of a word, which would require supervising all pieces, contributing ki loss terms per word and thus over-weighting words that fragment into many subwords. We leave this variant to future work. The indicator mp thus controls both prior injection and supervision, giving each word exactly one tag signal at the input and one prediction at the output. The auxiliary objective takes one of two concrete forms in our experiments. In autoregressive models, the head at position p predicts the tags (τp+1 , νp+1 ) of the next tagged position, teacherforced from gold left context. In masked language models, the head at position p predicts (τp , νp ) at masked positions only; wherever a token is masked,

A.2

SiPE Position-Pathway Design Choices for Transformer-XL

The multiplicative position-pathway injection of section 3.3 has two design axes. Layer sharing controls whether a single projection WE is reused (ℓ) across all layers or each layer learns its own WE . (ℓ) Projection target controls the dimension WE projects the tag into before its inner product with the query forms the alignment score c: either a single attention head’s dhead -dimensional subspace (a small map that places the tag in the same space as that head’s query, so c is formed by the same query that produces BD and lands on the same per-head scale; the one projected vector is then shared across all heads), or the full model dimension dmodel (a larger map whose output is split into nhead blocks, giving each head its own projected tag). Table 7 reports all four combinations. For Transformer-XL, the best configuration we find is the per-layer projection into the full model dimension (74.01 BLiMP, 80.60 SG). Neither axis dominates alone: the gain comes from their pairing. Per-layer projections matter because the tag correction is scaled by each layer’s own BD, so a (ℓ) separate WE lets each layer specialize how syntax modulates its positional preference. The fulldimension target matters because it gives the tag a richer map than a single head’s subspace allows. Notably, the better projection target flips with the coupling: the multiplicative coupling here works

13

Sharing

Projection

Shared Per-layer Shared Per-layer

dhead dhead dmodel dmodel

BLiMP ↑

SG ↑

73.95 74.15 73.95 74.01

78.80 75.84 75.09 80.60

axis has an obvious right answer a priori, we ran all four combinations and report them in Table 8.

Table 7: Multiplicative position-pathway injection across the two design axes: layer sharing (shared vs. per-layer WE ) and projection target (dhead head subspace vs. dmodel full dimension). Per-layer projection into the full model dimension gives the best average across BLiMP and SyntaxGym (SG); this is the configuration we adopt as SiPE (positional pathway) for Transformer-XL.

Projection

Shared Shared Per-layer Per-layer

dhead dmodel dmodel dhead

BLiMP ↑

SG ↑

74.17 75.03 73.51 74.72

77.72 76.10 77.20 78.72

Table 8: Fully-disentangled injection across the two projection axes: layer sharing (a single WE shared across (ℓ) all layers vs. a per-layer WE ) and projection target (mapping the tag into a single head’s dhead -dimensional subspace vs. the full model dimension dmodel , reshaped into per-head blocks). Per-layer projection into the head subspace gives the best average across BLiMP and SyntaxGym (SG); this is the configuration reported in the main results.

better with dmodel , whereas the fully-disentangled coupling of Appendix A.3 works better in the dhead head subspace. The best target therefore depends on how the prior is coupled to the position pathway, not on the projection axis in isolation. Beyond these two axes, the form of the coupling itself matters: multiplicative coupling scales the position term by the tag alignment (Eq. 6), while the disentangled form adds the alignment as a standalone logit term (section 3.3.3, Eq. 7). The multiplicative variants are the strongest overall, but the ordering is informative: disentangled injection already outperforms input-side injection (Eq. 4), and multiplicative coupling outperforms both. Entangling the prior with position thus helps progressively more as the coupling becomes gated by the query’s existing offset alignment, so that syntax sharpens a positional preference the model has already formed rather than adding an unconditional bias. A.3

Sharing

The best disentangled configuration is per-layer projection into the head subspace (74.72 BLiMP, 78.72 SG; Table 8). In terms of syntactic generalization performance, it outperforms input-side injection (73.82 BLiMP, 76.97 SG) but lags behind the multiplicative coupling (§3.3.2). We thus find that the least to most performant design choices for entangling the syntactic prior are: input-side < fully disentangled < multiplicative. A.4

Attention-Side and Combined Injection Design Choices for Transformer-XL

Relation to the Shaw formulation: The key and value biases of Eqs. 8–9 adapt Shaw et al. (2018), V who add learned per-pair vectors aK ij and aij to the key-side logit and the value-side aggregation, P K and z += V . We change eij += q⊤ a α a i i ij j ij ij two things. First, our bias is indexed by the key position j alone rather than by the relative pair (i, j), so the tag at j contributes a single per-key vector that every query sees; this is what keeps the added cost linear in sequence length rather than quadratic. Second, the bias is conditioned on hexatag identity rather than relative offset, and is learned jointly with the language model. The terminal and nonterminal tag embeddings are projected separately and summed, and a position-validity mask mj zeroes the contribution wherever the tag label is -100. The (ℓ) tag bias is formed from the unmodified query qi , (ℓ) not qi + un as in the content term, so it bypasses the global content bias un of Dai et al. (2019). And since we set mem_len = 0, it spans the same keys as intra-segment attention, so no padding is needed for memory positions.

Fully-Disentangled Syntax Injection Design Choices for Transformer-XL

The fully-disentangled variant of section 3.3.3 adds the tag–query alignment c to the attention logit as a standalone term (Eq. 7), leaving both the content term AC and the position term BD exactly as in standard Transformer-XL. It is governed by the same two design axes (layer sharing and projection target) as the multiplicative position-pathway variant of Appendix A.2, the only difference being that there c multiplies BD whereas here it is added (ℓ) as a standalone logit term: WE may map the tag into the full model dimension (then reshaped into per-head blocks) or directly into a single head’s dhead -dimensional subspace, and it may be shared across all layers or learned per layer. Since neither 14

and Wc2p , Wp2c project content vectors into this relative-position subspace.9

This places our design between the dependencydistance bias of Omote et al. (Omote et al., 2019) and the root-to-node path embedding of Xie et al. (Xie et al., 2021). Both put syntactic structure inside attention rather than in the residual stream, but our bias is keyed on hexatag identity rather than a tree-distance proxy, is a single per-key term rather than a pairwise one, and is applied in a causally masked unidirectional language model rather than a bidirectional encoder.

Input-side injection: where priors enter, and where they do not. Under input-side injection (Eq. 11) each input vector carries the summed tag prior, so the queries and keys carry it into the score through the content stream: 
 seg T N qi = WQ etok i + e i + m i · Eτ i + E νi , (19) 
seg T N kj = WK etok j + e j + m j · E τj + E ν j .

Sharing across layers. The one design considtag tag eration here is whether WK and WV are shared across all layers or learned per layer. In the shared form a single pair is reused at every layer; in the per-layer form each of the L layers owns its own pair, giving L independent key and value maps and letting each layer specialize its tag-to-attention routing, in the same way the content projections WQ , WK , WV already specialize per layer. The per-layer form is the stronger of the two on syntactic evaluations, both for the attention-side injection on its own and for its combination with the inputside injection of Eq. 4, so we use it wherever the attention-side biases appear. A.5



(20) Reading Eqs. 18–20 together makes the design (0) (0) visible. The tag priors sit inside hi and hj and therefore propagate through WQ , WK into all three score terms via the content projections. The relative-position table, by contrast, appears only in c2p and p2c, and is purely a function of the offset. The two streams enter through entirely separate pathways and meet only when their score terms are summed to form the attention logit Aij ; the relative-position components themselves — the bucketing, the rδ table, and the Wc2p , Wp2c projections — are left entirely unmodified. Tag priors interact with relative position through the content stream alone, which is what makes the design portable across positional schemes. The tag tables ET , EN are constructed similarly to RoBERTa: uniformly initialized, masked by the first-subword indicator mp , and added inside the input layer norm.

DeBERTa-v3: Disentangled Relative Positional Embeddings and Tag Injection

DeBERTa-v3 (He et al., 2023) does not use absolute positional embeddings at the input level. Instead it delivers positional information through disentangled relative attention: each pair of positions (i, j) contributes three distinct terms to the attention score, only one of which depends on token content alone. This raises a natural question for our setting: where do tag priors enter when there is no input-level position embedding to sit alongside?

Position-pathway injection: The positionpathway variant of section 3.4.2 leaves the input clean and instead multiplies the two relativeposition terms (c2p and p2c, grouped as S pos in Eq. 10) by the tag–query alignment cij of Eq. 12, leaving the content-to-content term untouched. Here cij is formed from the content query Qci (ℓ) and the per-layer projection WE of the summed hexatag embedding, and the relative-position table and the Wc2p , Wp2c projections are left unmodified. This is the direct DeBERTa analogue of the multiplicative Transformer-XL injection: the prior

Background: disentangled attention. For a query at position i and a key at position j, the score decomposes into a content-to-content (c2c), a content-to-position (c2p), and a position-to-content (p2c) term, Aij = q⊤ kj + q ⊤ i Wc2p rδ(i,j) | i{z } | {z } c2c

c2p

+ k⊤ j Wp2c rδ(j,i) , | (0)

{z

p2c

(18)

9 The buckets δ collapse pairs of positions with similar offsets into shared rows of the relative-position table, following (Shaw et al., 2018): small offsets {−3, −2, −1, 0, 1, 2, 3} each receive their own bucket, while larger offsets are progressively merged — e.g. {+9, . . . , +15} might collapse into a single bucket — so the table stays small while preserving fine-grained discrimination near the diagonal.

} (0)

where qi = WQ hi and kj = WK hj . The relative-position embeddings rδ(·,·) are pulled from a learned table indexed by bucketed signed offsets, 15

scales the position pathway and leaves the content pathway untouched with the only difference being that DeBERTa’s position pathway carries two terms (c2p and p2c) rather than Transformer-XL’s single BD. A.6

ModernBERT: Rotary Injection Details

Portability of input-side injection: The unifying observation across the three encoders is that input-level injection commutes with whatever positional mechanism the model already uses, whether absolute (RoBERTa), relative (DeBERTa), or rotary (ModernBERT).

Figure 4: Verb→object attention on BLiMP (Warstadt et al., 2023) for RoBERTa-base vs. RoBERTa-base+SiPE (input pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On the subset of Causative examples where SiPE assigns higher PLL to the grammatical sentence but the base model does not, verb→object attention is higher under SiPE in 90% of cases (Appendix C).

(ℓ)

Identity at initialization. With Wδ → 0 we have ∆θ → 0 in Eq. 14, so Eqs. 15–16 reduce to plain RoPE: training starts from an unmodified ModernBERT, exactly as in the Transformer-XL and DeBERTa variants.

B

Time Complexity

SiPE adds two small embedding tables for terminal and non-terminal tags, ET ∈ R2×d and EN ∈ R5×d , totaling 7d parameters per pathway (∼5K at d = 768, ∼7K at d = 1024), well under 0.01% of the LM’s total parameters10 . Looking up and adding these embeddings costs O(Ld) per sequence, which is lower-order than self-attention’s O(L2 d) cost. Position-pathway variants introduce tag–query interactions with O(L2 d) complexity, matching the asymptotic cost of self-attention. SiPE therefore preserves the underlying Transformer’s asymptotic complexity. At inference, a single auxiliary parser pass supplies the hexatags, while two lightweight token-level heads add only linear overhead beyond the parser encoder.

C

Figure 5:

Verb→object attention on BLiMP for ModernBERT-base vs. ModernBERT-base+SiPE (input pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On Causative examples where SiPE is correct and the base model is not, the SiPE model assigns higher verb→object attention in 17/20 (85%) cases, mirroring the RoBERTa pattern despite ModernBERT’s rotary positional encoding.

its SiPE counterpart. Verb→object attention is averaged across all heads; the layers we average over differ by architecture. For Transformer-XL, our layerwise injection study (section5) shows that syntax matters at every layer, so we average over all of its layers. We have not run the corresponding sweep for the encoders, so we instead follow Tenney et al. (2019), who find that “BERT’s intermediate layers encode a rich hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle and semantic features at the top,” and therefore average over each encoder’s middle layer band.11 Across all three families, SiPE attains higher overall BLiMP accuracy than the corresponding base model. To understand how this syntactic information influences attention, we restrict the analysis to the subset of minimal pairs where the SiPE model is correct and the base model is not. We manually inspect 20 examples from this subset. We find

Analyzing Attention Patterns on Object–Verb Agreement Tasks

A natural desideratum for SiPE-pretrained models is that they should up-weight attention between tokens that are linearly distant but syntactically adjacent (distance 1 in the dependency tree). We probe this on the BLiMP Causative split for three model architectures spanning the three positional encoding families: RoBERTa-base (absolute), ModernBERT-base (rotary), and Transformer-XL (relative), each against 10 For comparison, a single attention layer in RoBERTa-base contains ∼2.4M parameters; the priors add less than 0.3% of a single layer’s parameters.

11

Layers 5–8 for the 12-layer RoBERTa-base and layers 9–12 for the 22-layer ModernBERT-base; all layers 1–16 for the Transformer-XL we pre-train.

16

LR

WD

RoBERTa-base/large & DeBERTa-v3-base/large all tasks 1e−5 0.1 ModernBERT-base/large (per task) CoLA 8e−5 1e−6 SST-2 8e−5 1e−5 MRPC 5e−5 5e−6 STS-B 8e−5 5e−6 QQP 5e−5 5e−6 MNLI 5e−5 5e−6 QNLI 8e−5 5e−6 RTE 5e−5 1e−5

Figure 6: Verb→object attention on BLiMP for Transformer-XL vs. Transformer-XL+SiPE (positional pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1), averaged over all 16 layers. On Causative examples where SiPE assigns higher sentence loglikelihood to the grammatical sentence but the base model does not, SiPE places higher verb→object attention in 11/20 (55%) cases: the same direction as the encoders (Figs. 4 and 5), but a much weaker majority. The green dashed line (right axis) marks the verb→object surface distance.

Table 11: GLUE fine-tuning hyperparameters. Only ModernBERT’s LR/WD vary by task; RoBERTa and DeBERTa-v3 use LR 1e−5, WD 0.1 throughout. All runs: 20 epochs, AdamW, ϵ = 10−6 .

E

Throughout the paper, each token carries just two hexatag priors: a terminal tag ti = ETτi and a nonterminal tag ni = EN νi , read from small learned tables (2 × D for the terminals, 5 × D for the non-terminals). But is this the richest syntactic signal we could inject? A natural alternative is the token’s full dependency-relation label (DEPREL; 40 labels such as nsubj and dobj), which would come from a much larger 40 × D table, giving a per-token embedding depreli . This opens two questions we study together: whether the richer DEPREL prior helps, and, once it is combined with the terminal and non-terminal priors, how all of these vectors should be fused with the token and positional embeddings, by addition, concatenation, a learned interpolation weight, or injection deeper into the network. We explore both questions as a small neuralarchitecture search over injection strategies. To keep it tractable we run the search on the simplest positional scheme, absolute positional embeddings, using RoBERTa-base as the encoder (the case illustrated in Figure 9). The strategies we explore are collected in Table 12 and split into two studies at different pretraining scales.

that RoBERTa+SiPE places higher verb→object attention than its base model in 90% of cases (Figure 4) and ModernBERT+SiPE does so in 85% of cases (Figure 5). Transformer-XL moves in the same direction but far more weakly: only a bare majority (11/20 (55%)) of such cases shift attention toward the object (Figure 6), where the encoders push it almost every time. Moreover, in over 80% of the subset of 20 examples we examine, the object is nonadjacent to the verb, indicating a redirection of attention weights to the syntactically relevant object, even when it is linearly distant from the verb. This suggests that SiPE effectively leverages syntactic information to guide attention in a way that captures long-range dependencies, which is crucial for understanding complex sentence structures. The weak Transformer-XL effect poses an open question. The encoders appear to spend most of the syntactic prior directly on attention, redistributing mass to mirror syntactic adjacency. TransformerXL barely does so in its attention maps, yet it converts the same signal into the largest downstream gain of any model we train (+8.2% relative on GLUE; Table 6). This suggests the decoder routes more of the enhanced syntactic knowledge beyond attention, towards semantic processing.

D

Alternative Embedding Combination Strategies

Fusion mechanism (1M steps). We first fix the priors to ti and ni and vary only how they enter the model, pretraining RoBERTa-base for 1M steps (top block of Table 12, Table 9). The simplest option wins: ADD_T_NT, which just adds the two priors to the input embedding with no extra parameters, attains the best macro-average. CONCAT_T_NT is the weakest (with large drops on QNLI and CoLA), and injection into the model’s residual (skip) connection (ADD_T_NT (Residual

Hyperparameters

Table 11 reports the task-specific fine-tuning hyperparameters used for each backbone. Unless noted otherwise, all models use AdamW with (β1 , β2 ) = (0.9, 0.999) and ϵ = 10−6 .

17

70.19

ADD T+NT 69.63

CONCAT POS+NT+T+DR

69.61

CONCAT POS+DNT ADD DNT α

69.60 69.48

ADD DNT 69.32

CONCAT POS+NT+T

69.28

ADD T+NT+DR ADD T+NT α

69.18 −0.5

0.0

0.5

1.0

1.5

2.0

Figure 7: Average GLUE performance across prior-injection variants for RoBERTa-base, shown as the improvement over the no-prior baseline (68.64 ± 0.61, shaded band at zero); bar-end labels give absolute scores and whiskers denote one standard deviation over 3 seeds. The strongest downstream performance is achieved by directly adding terminal and non-terminal prior embeddings to the absolute positional embedding model (ADD T+NT), outperforming concatenation, weighted addition, and variants using full dependency labels (the residual-injection variant is evaluated separately in Table 9). In these experiments, we set α = 0.5.

Figure 8: GLUE performance averaged across 3 random seeds per dataset under various prior-injection variants for RoBERTa-base. The strongest downstream performance is achieved by directly adding terminal and non-terminal prior embeddings to the absolute positional embedding model, outperforming concatenation, weighted addition, and variants using full dependency labels (the residual-injection variant is evaluated separately in Table 9).

Connection)) yields only modest gains. Replacing the plain addition with a learned linear interpolation between the input embedding and the prior, ADD_T_NT (α): x′i = α xi + (1 − α)(ti + ni ), never improves over ADD_T_NT at any α initialization (Table 10). Adding capacity or depth to the fusion, therefore, does not help; direct addition at the embedding layer is both the cheapest and the strongest.

ken’s DEPREL label and combine it with the terminal and non-terminal priors through the bottomblock strategies of Table 12. Because this study is more expensive, every variant, including a rerun baseline and ADD_T_NT, is pretrained for 500k steps for an apples-to-apples comparison (Figures 7, 8). Every prior-injection variant improves over the no-prior baseline on macro-average, yet ADD_T_NT still attains the highest mean. Layering DEPRELS on top of ti + ni (ADD_T_NT_DR, CONCAT_POS_NT_T_DR) does not close the gap, and swapping the terminal tag for the DEPREL

Adding dependency relations (500k steps). We then enrich the prior itself: we embed each to-

18

Task

Baseline

ADD_T_NT

CONCAT_T_NT ADD_T_NT (Residual Connection)

CoLA SST-2 QQP QNLI MNLI RTE STSB MRPC

31.41 (±2.02) 87.23 (±0.79) 83.24 (±0.21) 77.65 (±0.50) 73.03 (±0.33) 65.76 (±0.74) 79.11 (±0.30) 80.96 (±0.44)

31.77 (±0.71) ↑ 87.22 (±0.52) ↓ 83.36 (±0.11) ↑ 78.24 (±0.27) ↑ 74.05 (±0.17) ↑ 64.66 (±0.56) ↓ 81.41 (±0.13) ↑ 82.02 (±0.85) ↑

26.85 (±1.06) ↓ 87.27 (±0.35) ↑ 82.60 (±0.22) ↓ 70.79 (±0.70) ↓ 72.89 (±0.31) ↓ 64.74 (±1.45) ↓ 79.36 (±0.26) ↑ 82.76 (±0.72) ↑

32.35 (±0.79) ↑ 87.42 (±0.19) ↑ 83.13 (±0.15) ↓ 75.47 (±0.54) ↓ 73.90 (±0.14) ↑ 65.46 (±0.45) ↓ 80.41 (±0.47) ↑ 81.06 (±0.96) ↑

Average

72.30

72.84 ↑

70.91 ↓

72.40 ↑

Table 9: RoBERTa performance on GLUE under alternative strategies for combining syntactic priors. ADD_T_NT adds the terminal prior ti and non-terminal prior ni directly to the token representation (i.e., additive fusion at the embedding layer). CONCAT_T_NT concatenates the original embedding (including standard positional information) with ti and ni , then applies a learned linear down-projection to the model dimension before feeding the encoder. ADD_T_NT (Residual connection) injects (ti + ni ) into the residual stream before the multi-head self-attention sublayer. Results are averaged over 3 random seeds. Green arrows indicate improvements over the baseline and red arrows indicate degradations. Overall, ADD_T_NT yields the strongest macro-average (72.84%), while concatenation is the weakest configuration (70.91%), with particularly large drops on QNLI and CoLA. Task

Baseline

α = 0.1

α = 0.5

α = 0.9

CoLA SST-2 QQP QNLI MNLI RTE STSB MRPC

31.41 (±2.02) 87.23 (±0.79) 83.24 (±0.21) 77.65 (±0.50) 73.03 (±0.33) 65.76 (±0.74) 79.11 (±0.30) 80.96 (±0.44)

29.40 (±1.62) ↓ 86.93 (±0.34) ↓ 83.04 (±0.08) ↓ 77.08 (±0.73) ↓ 72.58 (±0.02) ↓ 66.25 (±0.18) ↑ 78.38 (±0.03) ↓ 82.77 (±0.09) ↑

29.89 (±0.71) ↓ 86.81 (±0.29) ↓ 83.01 (±0.11) ↓ 73.72 (±0.22) ↓ 73.50 (±0.37) ↑ 64.98 (±0.62) ↓ 79.64 (±0.51) ↑ 81.28 (±1.51) ↑

30.37 (±0.54) ↓ 88.13 (±0.06) ↑ 83.66 (±0.48) ↑ 76.45 (±2.00) ↓ 73.07 (±0.05) ↑ 66.07 (±1.44) ↑ 78.25 (±0.14) ↓ 80.42 (±0.04) ↓

Average

72.30

72.05 ↓

71.60 ↓

72.05 ↓

Table 10: Effect of initializing the interpolation gate α for RoBERTa under the ADD_T_NT setting. We initialize α ∈ {0.1, 0.5, 0.9} and allow it to be updated during pretraining via backpropagation; all results report downstream GLUE performance after pretraining. Despite task-level variability (notably on QNLI), none of the α initializations improves upon the baseline on average, and all are weaker than the simple additive fusion.

is no better, whether added directly (ADD_DNT), concatenated (CONCAT_POS_DNT), or interpolated through a learned weight, ADD_DNT (α): x′i = α xi + (1 − α)(depreli + ni ). Richer syntactic signal, at least in this form, does not translate into stronger downstream performance.

coder; the optimal injection strategy for relative (DeBERTa-v3, Transformer-XL) and rotary (ModernBERT) positional schemes, and under larger pretraining budgets, remains open. We therefore adopt ADD_T_NT as a simple and robust default, and leave a systematic study of injection strategies across positional schemes to future work.

Summary and scope. Taken together, these experiments identify ADD_T_NT, direct addition of the terminal and non-terminal priors at the input embedding, as the strongest injection strategy for absolute positional embeddings on the RoBERTa encoder. Two considerations bound the generality of this finding. First, the per-dataset results (Figure 8) show that no single variant is optimal on every task; the relative ordering of strategies varies across GLUE tasks, and ADD_T_NT is strongest in the macro-average rather than uniformly. Second, our search covers only the absolute-PE en19

Strategy

Injected input representation x′i

Description

Fusion mechanism (terminal + non-terminal tags) ADD_T_NT x′i = xi + ti + ni CONCAT_T_NT

x′i = W [ xi ; ti ; ni ]

ADD_T_NT (Residual Connection)

hi

ADD_T_NT (α)

x′i = α xi + (1 − α)(ti + ni )

(1)

= LN xi + ti + ni + Attn(x)i

Adding dependency relations (depreli , 40 labels) ADD_T_NT_DR x′i = xi + ti + ni + depreli CONCAT_POS_NT_T_DR

x′i = W [ xi ; ni ; ti ; depreli ]

ADD_DNT ADD_DNT (α)

x′i = xi + depreli + ni x′i = α xi + (1 − α)(depreli + ni )

CONCAT_POS_DNT

x′i = W [ xi ; depreli ; ni ]

Add both priors to the input embedding (default; no extra parameters). Concatenate with the input embedding, then down-project to D. Add the priors to the skip connection of the first attention sublayer. Linearly interpolate the input embedding with the (terminal + non-terminal) prior; learned α initialized in {0.1, 0.5, 0.9}. Add all three priors to the input embedding. Concatenate all priors with the input embedding, then down-project. Swap the terminal tag for the DEPREL. Interpolated counterpart of ADD_DNT; learned α initialized in {0.1, 0.5, 0.9}. Concatenate (DEPREL replacing the terminal), then down-project.

Table 12: Injection strategies we explore for absolute positional embeddings on RoBERTa-base. xi = ei + pi is the standard input embedding (token ei plus positional pi ), x′i the representation actually fed to the encoder, and W a learned projection back to the model dimension D. ti /ni are the terminal/non-terminal hexatag priors and depreli the dependency-relation prior. For the residual variant we write the first attention sublayer explicitly, h(1) = LN(x + Attn(x)): the priors are added on the skip branch, so unlike ADD_T_NT they do not enter the self-attention computation itself. The top block fuses only the terminal and non-terminal tags; the bottom block additionally brings in DEPRELS.

20

Figure 9: Injection points for syntactic priors in a Transformer encoder. We study where and how to inject hexatag-derived priors (terminal ti , non-terminal ni , and optionally dependency-relation labels depreli ) into RoBERTa-base. Three architectural locations are considered: (i) at the input embedding, alongside the token, segment, and positional embeddings; (ii) at the residual (skip) connection entering the first self-attention sublayer; and (iii) implicitly in the attention computation, by injecting at the input and propagating through WQ , WK .

21

Related documents

• RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings#1128821
• Official Review · ICLR 2025#787185
• Distilling Sequential Computation in Transformer Language Models#1063444
• Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction#1155775
• Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs#1077772