arXiv CSopen access

On the Role of Directionality in Structural Generalization

On the Role of Directionality in Structural Generalization Zichao Wei Saarland University

Abstract Several SLOG test categories explicitly involve directional distinctions (modifier position shifts, argument extraction positions), yet AM-Parser, the previous SOTA, uses an AM algebra whose operations do not encode direction. We redesign the symbolic backend around CCG directed types (deterministic CKY + single linear decoder, 30K learnable parameters). Under the same BERT-base encoder, the system achieves 75.9 ±6.4% LF exact match, surpassing AM-Parser (70.8±4.3%). Per SLOG’s own category group­ ings, gains are highly directional: the CCG system outperforms AM-Parser on all 5 posi­ tion-shift categories (+29.9pp), while AM-Parser outperforms on all 6 recursive-depth categories. Replacing the encoder with DeBERTa-v3-large yields 90.7±4.9%, with the largest encoder gains in recursive-depth categories, complementary to directionality’s gains. Directional representa­ tions shift the bottleneck from the symbolic layer (AM-Parser’s 0% category ceiling) to the neural layer, which improves with encoder upgrades.

1

Introduction

Structural generalization is a central challenge in semantic parsing (Lake and Baroni, 2018): a sys­ tem must apply compositional rules learned from training data to structural combinations never seen during training. On SLOG, a benchmark specifically targeting structural generalization (Li et al., 2023), end-to-end neural models (T5 (Raf­ fel et al., 2020), LLaMA (Touvron et al., 2023)) achieve approximately 40% (Yao and Koller, 2022). Prior work attributes this not to insufficient ca­ pacity but to a structural deficiency in seq2seq architectures for compositional generalization: end-to-end models learn surface correlations in the training distribution rather than composable rules (Kim and Linzen, 2020).

The neuro-symbolic system AM-Parser (Groschwitz et al., 2018; Lindemann et al., 2019) (70.8±4.3%) addresses this by using a BERT supertagger to predict AM algebra types, then letting AM algebra rules perform combinatory reasoning. AM-Parser validates the neuro-sym­ bolic paradigm, but its success has obscured an unexamined design choice: the AM algebra was designed for AMR semantic graph parsing, where word order is irrelevant, so its operations deliber­ ately do not encode direction (Weißenhorn et al., 2022a). Of SLOG’s 17 test categories, however, 5 explicitly require directional distinctions (PP/RC modifier position shifts, argument extraction po­ sitions). This raises a question: does a directional representation yield a consistent advantage on these categories? This paper argues that on structural gener­ alization categories involving directional distinc­ tions, representations that encode the direction of combinatory operations have a systematic advantage. The slash direction of Combinatory Categorial Grammar (CCG (Steedman, 2001)) provides the most direct encoding of direc­ tionality: 𝑆 \ NP (takes subject on the left) and 𝑆/ NP (takes object on the right) distin­ guish positions through the combinatory rules themselves, requiring no additional mechanism. Following the neuro-symbolic paradigm of AMParser, we redesign the symbolic backend around CCG directed types: deterministic CKY performs combinatory reasoning, and a single linear layer performs type prediction (30K learnable para­ meters). The contribution of directionality is tested through per-category group analysis: if directionality is the cause, its effect should fall precisely on position-shift categories, not on re­ cursive-depth categories. Contributions: (1) New SOTA on SLOG; per SLOG’s own category groupings, directionality’s gains concentrate on position-shift categories

(5/5) and are absent for recursive depth (0/6). (2) Two explanations for why directionality works: it reduces the supertagger’s discriminative burden (engineering), and it encodes a fundamental cue in the word-order-to-structure mapping (Levelt, 1989; Hawkins, 1994; Hawkins, 2004) (linguistic). (3) Encoder scalability: 75.9% → 79.9% → 90.7% across three encoder tiers, with gains comple­ mentary to directionality’s.

2

Background and Related Work

2.1

Structural Generalization and COGS/ SLOG

COGS (Kim and Linzen, 2020) is a compositional generalization benchmark for semantic parsing with 21 generalization categories: 18 are lexical generalization (substituting new words into ex­ isting structures) and only 3 are structural gener­ alization, limiting its coverage of structural phe­ nomena. SLOG (Li et al., 2023) was designed to address this gap by adding 17 structural general­ ization categories covering relative clauses (RC), wh-questions, and center embedding, establish­ ing the current standard evaluation for structural generalization. SLOG uses the same training set as COGS (32,755 samples), but its gen set contains 17,000 test samples (1,000 per category). The training data contains no relative clauses or wh-questions; these structures appear only in the test set. This design forces systems to handle entirely novel syntactic structures with zero training signal, testing the algebraic closure of the symbolic rep­ resentation: the type system must reserve type slots for unseen structures. SLOG’s 17 categories are grouped by the original paper into four sections: §2.1 recursive depth (6 categories, testing generalization over nesting depth), §2.2 modifier position (4 cate­ gories, testing PP/RC modifier shifts from object side to subject side), §2.3 extraction position (1 category, testing argument extraction position within RCs), and §2.4 wh-questions (6 categories, with mixed properties). This grouping naturally separates direction-relevant from direction-irrel­ evant dimensions: modifier position and extrac­ tion position categories explicitly test positional directionality, while recursive-depth categories are unrelated to directionality.

2.2

AM-Parser and Neuro-Symbolic Meth­ ods

AM-Parser (Groschwitz et al., 2018; Lindemann et al., 2019) decomposes semantic parsing into two steps: (1) a neural network predicts AM algebra types for each word (supertagging), and (2) AM algebra rules perform combinatory reasoning. This design ensures that structural generalization derives from the compositional closure of alge­ braic rules rather than from specific structural combinations present in training data. AM-Parser demonstrated the potential of the neuro-symbolic paradigm on COGS (Weißen­ horn et al., 2022a; Petit et al., 2023), and Weißen­ horn et al. (2022b) further argued that compo­ sitional generalization requires compositional parsers. Recently, Lindemann et al. (2024) showed that pre-training on synthetic syntactic transfor­ mations can strengthen the structural inductive bias for semantic parsing. AM-Parser achieves 70.8±4.3% on SLOG, the previous strongest sys­ tem. The AM algebra was originally designed for AMR semantic graph parsing, where AMR is an abstract, word-order-independent semantic representation. Its apply and modify operations therefore do not encode direction. This is a rea­ sonable and successful design for AMR tasks. When AM-Parser was applied to COGS and SLOG, this directionless combinatory system was inherited directly. Because AM-Parser substan­ tially outperforms end-to-end methods on SLOG (70.8% vs. 40%), the question of whether direc­ tionality is a better choice has not been explored. 2.3

Combinatory Categorial Grammar (CCG)

CCG (Steedman, 2001) is a lexicalized grammar formalism whose defining feature is encoding directionality in its types: • 𝑆 \ NP: requires an NP to the left to produce S (backward function application) • 𝑆/ NP: requires an NP to the right to pro­ duce S (forward function application) Directionality in CCG is not an add-on but the es­ sential definition of its combinatory operations. The same NP retains its type regardless of posi­ tion; slash direction distinguishes which side of the verb a constituent occupies. For tasks requir­ ing structure recovery from ordered sentences (as SLOG tests), CCG’s directed types directly encode positional information. The AM algebra’s

modify does not distinguish left from right; when positional distinctions are needed, this burden shifts to the supertagger or to additional features (such as +dist (Weißenhorn et al., 2022a)). We do not use the full CCG combinatory cal­ culus. The system retains forward and backward function application (𝐴/𝐵 + 𝐵 → 𝐴 and 𝐵 + 𝐴 \ 𝐵 → 𝐴) but simplifies type raising and com­ position into atomic types with explicit merge rules, reducing the inventory to 26 categories suited to COGS/SLOG (including 2 disambigua­ tion types; see §3.2).

3

Method

3.1

Overall Architecture

The system adopts the same neuro-symbolic par­ adigm as AM-Parser: neural supertagger + sym­ bolic combinatory backend. The difference lies in the choice of type system and combinatory rules. The system consists of three components: 1. Neural supertagger: A frozen BERT (De­ vlin et al., 2019) encodes the input sentence into contextualized word vectors. Function words (articles, “was”, “did”, etc.) are then stripped, retaining only content words. This aligns the word sequence with the CCG type sequence, since function words do not carry CCG types in the COGS/SLOG formalization. Gumbel-Softmax (Jang et al., 2017) then discretizes the continuous BERT embeddings into one of 𝐾 = 32 codebook entries, and a linear readout head (ℝ64 → ℝ26 ) maps codebook embeddings to logits over 26 CCG types. Total: 30K learnable parameters (excluding the frozen BERT). 2. Deterministic CKY composition: Bot­ tom-up CKY parsing is applied to the pre­ dicted CCG type sequence using standard CCG combinatory rules (forward/backward function application). This step is purely symbolic, deterministic, and non-learnable. It corresponds to the role of the AM alge­ bra in AM-Parser (performing combinatory reasoning), but uses CCG’s forward/back­ ward function application rules. 3. Semantic edge extraction: Predicate-ar­ gument edges (agent, theme, recipient, etc.) are deterministically extracted from the CKY parse tree to produce a logical form. This step is likewise purely symbolic.

3.2

CCG Type System

We use 26 CCG types to cover all syntactic structures in COGS and SLOG. Of these, 20 base types (NP, S, 𝑆 \ NP, (𝑆 \ NP)/ NP, passives, prepositions, etc.) cover all 21 COGS categories; 4 extension types (RC_THAT, TV_GAP, S_GAP, WH) cover the relative clauses and wh-ques­ tions added by SLOG; 2 disambiguation types (DIT_REC, RC_THAT_REC) resolve role ambigu­ ities in ditransitive verbs and RC indirect-object extraction. CCG types are deterministically derived from gold-standard logical forms (LFs). The training target is the final state of the CKY parse trajec­ tory (the head word survives; all other positions collapse to ∅). 3.3

Evaluation Metric

We report LF exact match: the system applies deterministic CKY parsing and semantic edge ex­ traction to the predicted CCG type sequence and compares the extracted predicate-argument edge set against the edge set extracted from the gold LF. COGS/SLOG LFs are conjunctions of predi­ cate-argument relations (e.g., sleep.agent(x, cat)), so edge-set comparison is naturally insen­ sitive to conjunct order and variable naming. The reformatted LF exact match reported by AM-Parser (Li et al., 2023) achieves the same nor­ malization by sorting conjuncts and re-indexing variables; the two are equivalent in their criterion of semantic correctness: both determine whether the system correctly recovers all predicate-argu­ ment relations. Pipeline fidelity is 99.92% (see Appendix A); the 0.08% loss stems from inher­ ent ambiguities in the CCG type system and is included in the reported figures. 3.4

Training Details

• Data: SLOG training set (32,755 samples), 100% CKY coverage • Optimizer: AdamW (lr=10−3 , weight decay= 10−4 ), cosine schedule • Gumbel temperature: annealed from 1.0 to 0.1 over 50 epochs • Parameters: 30K (excluding frozen BERT) • Training time: 8 minutes (single GPU) • Model selection: All hyperparameters are fixed across seeds; no validation-based model selection or early stopping is used, ensuring no information leaks from the test distribution into model selection. Results for

AM-Parser

Ours (BERT)

Ours (DeBERTa)

Encoder

BERT-base

BERT-base

DeBERTa-v3-large

Type system

AM algebra (undirected)

CCG (directed)

CCG (directed)

Symbolic composition

AM algebra apply/modify

CKY fwd/bwd application

CKY fwd/bwd application

Decoder

Symbolic logic + edge predictor

Single linear layer

Single linear layer

0% categories

2

0

0

Overall%

70.8±4.3

75.9±6.4

90.7±4.9

Table 1: Architectural comparison of three neuro-symbolic systems. The symbolic composition layer is a purely symbolic, non-learnable operation; the decoder converts type sequences into logical forms. Category

Ours (DeBERTa)

Ours (BERT)

AM-Parser

T5

LLaMA

PP recur. (depth 3)

99.1±1.2

96.8±2.6

100.0±0.0

93.1±1.9

98.9±0.6

PP recur. (depth 5–12)

97.8±2.1

94.3±3.2

100.0±0.0

16.6±1.0

20.6±1.0

CP recur. (depth 3)

81.9±29.8

18.8±34.4

100.0±0.0

60.9±2.1

98.1±0.7

CP recur. (depth 5–12)

64.8±26.3

17.7±34.7

100.0±0.0

5.3±0.4

12.1±0.7

Center emb. (depth 3)

100.0±0.0

99.7±0.3

100.0±0.0

64.1±19.1

50.7±5.7

Center emb. (depth 5–12)

99.3±1.8

80.6±8.3

99.5±0.4

0.0±0.0

0.0±0.0

PP_modif_iobj

99.3±0.8

97.6±1.7

90.4±8.1

53.8±1.4

71.2±4.2

PP_modif_subj

94.5±4.8

92.8±4.0

57.6±8.1

0.8±0.5

28.9±3.5

RC_modif_iobj

95.7±5.8

78.4±10.6

74.4±6.4

36.6±2.1

55.0±2.1

RC_modif_subj

52.0±4.1

61.9±11.6

55.8±8.4

0.2±0.2

29.5±3.4

RC_iobj_extracted

97.8±2.3

96.9±3.0

0.0±0.0

0.0±0.0

2.5±3.2

Q_subj_active

91.9±7.2

77.8±13.3

99.8±0.6

98.1±1.7

93.3±6.0

Q_subj_passive

99.2±2.5

85.9±16.5

100.0±0.1

100.0±0.0

15.3±17.5

Q_dobj_ditransV

99.6±1.3

96.3±5.0

29.4±33.5

98.5±0.9

8.6±5.7

Q_iobj_ditransV

99.7±0.8

71.4±26.4

41.4±42.4

0.4±0.7

73.5±18.4

Q_modified_NPs

84.5±5.4

73.9±5.8

55.6±12.5

36.8±0.4

20.8±2.4

Q_long_mv

84.7±15.1

48.4±19.1

0.0±0.0

24.9±5.1

3.0±4.7

Overall

90.7±4.9

75.9±6.4

70.8±4.3

40.6±1.0

40.1±1.8

Table 2: SLOG gen set results (LF exact match mean±std). Our system reports results over 10 seeds; other systems report reformatted exact match (5 seeds) from (Li et al., 2023) Table 5. Vanilla TF = Transformer trained from scratch (Vaswani et al., 2017). Highest value per row in bold.

3.5

10 seeds report the final checkpoint at epoch 50.

4

Experimental Results

4.1

Main SLOG Results

System Architecture Comparison

Table 2 reports per-category accuracy for each system on the SLOG gen set. Our system reports LF exact match (10 seeds); other systems report reformatted exact match (5 seeds) from (Li et al., 2023) Table 5. Under the controlled condition (BERT-base), the system achieves 75.9±6.4%, surpassing AMParser (70.8±4.3%). With DeBERTa-v3-large, the system reaches 90.7±4.9%, exceeding AM-Parser by nearly 20 percentage points, with 9/17 categories averaging above 97%. The encoder

Table 1 compares the architectural configura­ tions of three neuro-symbolic systems. All three share the BERT supertagger + symbolic combina­ tory backend paradigm; differences concentrate in the representational properties of the type system (directionality) and the complexity of the decoder.

Figure 1: Mean accuracy by SLOG category group. On §2.2 modifier position and §2.3 extraction position, the CCG system outperforms AM-Parser; on §2.1 recursive depth, AM-Parser has the advantage. The largest gains from the encoder upgrade (DeBERTa-v3-L) appear in the recursive-depth group.

upgrade systematically narrows the gap with AM-Parser under BERT-base: most recursivedepth categories approach or surpass AM-Parser under DeBERTa, with only CP recursion and RC_modif_subj still showing a clear deficit. 4.2

Directional Separation by SLOG Cate­ gory Groups

If directionality is the cause of the performance gain, its effect should fall precisely on directionrelevant categories, not distribute uniformly. To test this prediction, we strictly follow SLOG’s original category groupings (Li et al., 2023): §2.1 recursive depth (6 categories, testing gener­ alization over nesting depth, unrelated to direc­ tionality), §2.2 modifier position (4 categories, testing PP/RC modifier shifts from object side to subject side, directly related to directionality), §2.3 extraction position (1 category, testing argu­ ment extraction position within RCs, directly related to directionality), and §2.4 wh-questions (6 categories, with mixed properties). Using SLOG’s own groupings eliminates any influence of grouping methodology on the conclusions. Figure 1 shows mean accuracy by group. Under controlled conditions (BERT-base), the difference between the CCG system and AMParser exhibits a precise directional separation: the CCG system outperforms AM-Parser on all 5 position-shift categories (§2.2 + §2.3) by an

average of +29.9pp and underperforms on all 6 recursive-depth categories (§2.1) by an average of −31.9pp. The absolute magnitudes of these two deltas are nearly equal, indicating that the effect of directionality is not a general improvement but a directional one: it improves the directionrelevant dimension and does not help on the irrelevant dimension. This symmetry implies that the effect of directionality constitutes a precise redistribution across the two dimensions, not a random fluctuation within a general improve­ ment. The wh-question group (§2.4) provides fur­ ther evidence. The 6 wh-categories can be divided, by a priori criteria independent of experimental results, into two subgroups: Q_dobj_ditransV, Q_iobj_ditransV, Q_long_mv, and Q_modified_NPs involve argument extrac­ tion from non-subject positions, where direc­ tionality directly determines argument role iden­ tification; Q_subj_active and Q_subj_passive in­ volve simple subject extraction, where the argu­ ment is the subject itself and no positional ambi­ guity arises. Within the same system, the CCG system leads by large margins on the directionrelevant subcategories (+30.0 to +66.9pp) and AM-Parser leads on the direction-irrelevant sub­ categories. This within-group comparison con­ trols for all system-level confounds: the decoder and training strategy are identical across the two

subgroups, making it a cleaner test of the direc­ tional effect than cross-group comparisons. We acknowledge that the two systems differ in ways beyond the type system, including decoder architecture (single linear layer vs. edge predic­ tor), training strategy (Gumbel-Softmax anneal­ ing vs. AM training), and type granularity (26 types vs. 50 types). More fundamentally, replac­ ing the representation is a system-engineering operation: directionality is not an optional para­ meter of CCG but the definition of its combina­ tory rules (see Appendix B). It cannot be isolated as an independent variable for ablation. Just as one cannot “add directionality” to the AM algebra without rewriting the entire combinatory calcu­ lus, one cannot “remove directionality” from CCG without deconstructing the entire formal system. We accept this limitation candidly. This limitation does not, however, block the test of directionality. The logic of the test relies not on ablation but on pattern alignment: if direc­ tionality is the cause, improvements should fall precisely on direction-relevant categories. The data show exactly this pattern: the CCG system leads on all 5 position-shift categories and trails on all 6 recursive-depth categories. We acknowl­ edge that confounds could also produce nonuniform effects, and this pattern alone cannot ex­ clude alternative explanations. Directionality is, however, the most parsimonious explanation for the observed pattern: the advantage aligns pre­ cisely with the dimension where directionality is most relevant, not with the dimension where decoder architecture or training strategy would most plausibly have an effect. The within-group differentiation of the wh-question categories, where a priori classification aligns precisely with experimental results within the same system, further narrows the space of alternative explana­ tions. Directionality is the most parsimonious explanation for this full set of evidence, requiring the fewest additional theoretical assumptions. Replacing BERT-base with DeBERTa-v3-large shifts the largest encoder-upgrade gains to recur­ sive-depth categories (+22.5pp), complementary to the gains from directionality: directionality resolves position shifts, and the encoder resolves recursive depth. The two belong to different dimensions and are complementary rather than competitive.

4.3

Encoder Scalability

AM-Parser scores 0.0±0.0 on two categories (5 seeds) (Li et al., 2023), constituting an archi­ tectural ceiling that cannot be improved by upgrading the encoder. All categories under the CCG system are empirically non-zero, so encoder improvements are not truncated by the compo­ sition layer. We verify this with three encoder tiers (all other hyperparameters fixed): BERTbase 75.9±6.4% → ModernBERT-base (Warner et al., 2025) 79.9±3.7% → DeBERTa-v3-large (He et al., 2023) 90.7±4.9%. DeBERTa-v3-large pushes 9/17 categories above 97%, including the two AM-Parser 0% categories (RC_iobj_extracted 97.8%, Q_long_mv 84.7%). Detailed per-category encoder analysis and a shared variance structure analysis are in Appendix C and G.

5

Discussion

The preceding sections demonstrated the direc­ tional separation pattern: directionality precisely targets position-shift categories. The following three subsections explain the causes of this pattern, and the final subsection discusses its methodological implications. 5.1

Why Directionality Matters: An Engi­ neering Argument

In a neuro-symbolic system, the type system is the sole interface between the neural predictor and the symbolic reasoner: the predictor’s output is a type sequence, and the reasoner’s input is also a type sequence. The design of this inter­ face determines not only what each component processes but also whether they can collaborate effectively. Any information that must be represented in the system has only two possible locations: it can reside in the non-learnable combinatory rules or in the learnable type labels. Directionality deter­ mines how information is distributed between these two locations. Consider PP modification as an example. In the AM algebra, PP modification of the object (“the cat beside a car”) and PP modification of the subject (“the cat beside a car was chased,” with the PP on the left side of the verb) require the supertagger to assign different type labels to the two cases, because the AM algebra’s modify operation does not distinguish left from right. In CCG, the PP’s type in both cases is NP \ NP, and directionality is automatically distinguished by

Figure 2: Accuracy comparison for PP and RC modifier position pairs (subject side vs. indirect-object side). AM-Parser shows large directional asymmetry on both modifier pairs (Δ33, Δ19); the CCG system (BERT) substantially reduces the asymmetry for PP modifiers. The subject side of RC modifiers is weaker across all systems.

the slash direction at composition time (forward vs. backward application). The type inventory does not expand, and the supertagger’s discrimi­ native burden does not increase. PP modification is just one instance. Argument structure (subject vs. object), modifier attach­ ment (left vs. right), and wh-extraction (forward vs. backward) are all naturally distinguished by slash direction in CCG. With the same BERT and training data, the CCG system outperforms the AM system not because BERT is stronger, but be­ cause its discriminative burden is lighter. Figure 2 shows this for PP and RC modifier position pairs. The authors of AM-Parser have indirectly confirmed the impact of missing directionality. Weißenhorn et al. (2022a) reports that PP modi­ fication direction requires an additional distance encoding (+dist feature, a decoder-layer patch that does not belong to the AM algebra itself) to distinguish the linear position of a modifier relative to the head. The existence of +dist is itself evidence: positional information was omitted by the AM algebra and had to be externally supplied. In CCG, slash direction is the definition of the combinatory operations: positional information is built into the representational protocol, requir­ ing no patch.

5.2

Why Directionality Matters: A Linguis­ tic Argument

We contend that directionality is effective at the system-design level because it reflects a structural property of language production and comprehension. Natural language as a spoken medium is a one-dimensional linear signal: words must be produced sequentially in time. The conceptual structure (who did what to whom, what modifies what) is relational in the speaker’s mind and has no linear order. The process of converting con­ ceptual structure into a one-dimensional word sequence is called linearization (Levelt, 1989), language comprehension is its inverse, recover­ ing multi-dimensional conceptual structure from one-dimensional word order. Hawkins (Hawkins, 1994; Hawkins, 2004) provides independent ar­ guments from the comprehension side: cross-lin­ guistic word-order regularities are systematically governed by structural-recognition efficiency, and human parsers prefer word orders that allow the fastest identification of phrasal structure. In this framework, directionality (which side of the head an argument appears on) is the property of word order most directly relevant to structural generalization: it determines whether a parser can distinguish grammatical relations from local positional information alone.

The benchmark design of SLOG (testing struc­ tural generalization through position shifts: PP moving from the object side to the subject side, wh-constituents extracted from their base posi­ tion to sentence-initial position) is in essence testing whether a parsing system has processed the directional cues in the mapping from linear word order to structure. CCG’s slash direction is a precise formalization of this cognitive process. This explains why the experimental gains from directionality fall precisely on position-shift cat­ egories: these categories are exactly the points in the word-order-to-structure recovery process where directional information is most critical. The AM algebra’s directionless design is reason­ able for AMR semantic graph tasks, but on SLOG tasks that require recovering structure from lin­ ear word order, abandoning directionality means abandoning the most fundamental structural cue in the comprehension process. 5.3

Synthesis: The Role of Directionality

The engineering and linguistic arguments con­ verge: directionality is a key property of the representational protocol between the neural and symbolic components. It determines what infor­ mation is learned by the neural layer and what is given by symbolic rules. Its effect on positionshift categories is directional (5/5) and absent for recursive depth (0/6). Its effect on scalability is architectural: it removes the 0% category ceiling and places the performance bottleneck on the improvable supertagger (§4.3). Directionality is not an ablatable parameter; it is the definition of CCG’s combinatory operations (see Appendix B). The hypothesis is falsifiable: if subsequent work finds a non-directional representation that matches CCG on position-shift categories, our claim is challenged. Conversely, if introducing directional encoding into other formalisms yields consistent gains, directionality’s role is validated across frameworks. We welcome such tests.

6

Conclusion

This paper tests the role of directionality in structural generalization through controlled ex­ periments on SLOG. The effect is directional: improvements concentrate on all 5 position-shift categories (+29.9pp) and are absent for all 6 recursive-depth categories (−31.9pp). The effect is also architectural: directional representations eliminate the 0% category ceiling present in AM-

Parser, shifting the bottleneck to the neural layer, which improves with encoder upgrades (75.9% → 90.7%). Directionality and encoder strength are complementary, not substitutive. On structural generalization involving positional distinctions, directional representations have a consistent and substantial advantage.

7

Limitations

The system generates logical forms through a deterministic pipeline (CCG types → CKY → semantic edge extraction). Pipeline fidelity is 99.92% (13/17,000 inconsistencies stem from in­ herent ambiguities in the CCG type system). This pipeline is not fully comparable in output format to seq2seq methods that directly generate LFs. We note, however, that LF exact match itself has known evaluation issues: T5 achieves 47.2% plain exact match on Q_dobj_ditransV but 98.5% after reformatting (Wu et al., 2023), suggesting that LF format differences can exaggerate the apparent degree of generalization failure. CCG type match avoids this problem but introduces the degree of freedom inherent in type system design. The system uses a frozen BERT-base encoder (110M parameters) for supertagging. BERT’s pretraining data may contain patterns similar to SLOG test structures, so pre-training leakage cannot be entirely ruled out. This confound is shared by the entire supertagger paradigm and does not affect the core argument of this paper (directional vs. non-directional comparison), but it underscores the need for further work to isolate pre-training effects. The argumentative framework of this paper relies on CCG’s concept of directionality. These concepts have been empirically validated within the COGS/SLOG domain, but whether they gen­ eralize to broader structural generalization eval­ uations remains an open question. The variance analyses of the two systems (ours and AM-Parser) are based on different numbers of seeds (10 vs. 5), which may affect the precision of variance estimates. The qualitative conclusion about shared variance structure (both systems exhibit categories where standard deviation ap­ proaches or exceeds the mean) does not depend on an exact match in seed count. Code and trained models will be released upon acceptance. The encoder replacement experiments were conducted only under the CCG system. AM-

Parser’s evaluation pipeline is not fully opensourced (code is open but pre-trained weights and evaluation scripts are not available), preventing us from reproducing AM-Parser’s results under the same encoders. The core comparison under the BERT-base control condition is unaffected by this limitation.

References Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-Training of Deep Bidirectional Transformers for Language Un­ derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech­ nologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR Dependency Parsing with a Typed Semantic Algebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol­ ume 1: Long Papers), pages 1831–1841, Melbourne, Australia. John A. Hawkins. 1994. A Performance Theory of Order and Constituency. Cambridge Studies in Linguistics 73. Cambridge: Cambridge University Press. John A. Hawkins. 2004. Efficiency and Complexity in Grammars. Oxford, New York: Oxford University Press. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa Using ELECTRA-style Pre-Training with Gradient-Disen­ tangled Embedding Sharing. In The Eleventh Inter­ national Conference on Learning Representations. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categor­ ical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Represen­ tations, ICLR 2017. Najoung Kim and Tal Linzen. 2020. COGS: A Composi­ tional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process­ ing (EMNLP), pages 9087–9105, Online. Brenden M. Lake and Marco Baroni. 2018. Gener­ alization without Systematicity: On the Composi­ tional Skills of Sequence-to-Sequence Recurrent Networks. In Proceedings of the 35th International Conference on Machine Learning, pages 2879–2888. Willem J. M. Levelt. 1989. Speaking: From Intention to Articulation. The MIT Press. Bingzhi Li, Lucia Donatelli, Alexander Koller, Tal Linzen, Yuekun Yao, and Najoung Kim. 2023. SLOG:

A Structural Generalization Benchmark for Seman­ tic Parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3213–3232, Singapore. Matthias Lindemann, Jonas Groschwitz, and Alexan­ der Koller. 2019. Compositional Semantic Parsing across Graphbanks. In Proceedings of the 57th An­ nual Meeting of the Association for Computational Linguistics, pages 4576–4585, Florence, Italy. Matthias Lindemann, Alexander Koller, and Ivan Titov. 2024. Strengthening Structural Inductive Bi­ ases by Pre-Training to Perform Syntactic Transfor­ mations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11558–11573, Miami, Florida, USA. Alban Petit, Caio Corro, and François Yvon. 2023. Structural Generalization in COGS: Supertagging Is (Almost) All You Need. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1089–1101, Singapore. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21(140):1–67. Mark Steedman. 2001. The Syntactic Process. Language, Speech and Communication. Cambridge, Mass.: MIT Press. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. http:// arxiv.org/abs/2302.13971. arXiv: 2302.13971 [cs]. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Pro­ cessing Systems. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lin­ guistics (Volume 1: Long Papers), pages 2526–2547, Vienna, Austria. Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022a. Compositional Generalization with a Broad-Coverage Semantic Parser. In Proceedings of

the 11th Joint Conference on Lexical and Computa­ tional Semantics, pages 44–54, Seattle, Washington. Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, and Alexander Koller. 2022b. Compositional Generaliza­ tion Requires Compositional Parsers. https://arxiv. org/abs/2202.11937v1. Zhengxuan Wu, Christopher D. Manning, and Christopher Potts. 2023. ReCOGS: How Incidental Details of a Logical Form Overshadow an Evalua­ tion of Semantic Interpretation. Transactions of the Association for Computational Linguistics 11:1719– 1733. Yuekun Yao and Alexander Koller. 2022. Structural Generalization Is Hard for Sequence-to-Sequence Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates.

A CCG-to-LF Pipeline Fidelity Verifica­ tion Of the full 17,000 SLOG gen samples, 16,861 con­ tain predicate-argument edges and 139 do not. All edge-free samples come from Q_subj_active (79) and Q_subj_passive (60), all of which are intran­ sitive wh-questions (e.g., “Who slept?”) whose LFs contain only a single predicate and one argu­ ment, producing no comparable edge pairs. For these samples, evaluation degenerates to a pure type-prediction correctness check (trivial edgeset match). On the 16,861 samples with edges, gold CCG type sequences processed through determinis­ tic CKY parsing and semantic edge extraction achieve 99.92% agreement (16,848/16,861) with gold LF edges. The remaining 13 inconsistent samples all belong to Q_modified_NPs and stem from an inherent ambiguity in the CCG type sys­ tem: transitive verbs and ditransitive verbs miss­ ing one argument share the type (𝑆 \ NP)/ NP. This ambiguity does not affect any conclusion of this paper.

B On the Infeasibility of a Directionality Ablation One might expect a directionality-removal abla­ tion: remove slash direction within the CCG framework and observe whether performance drops. We explain here why this ablation is tech­ nically infeasible at every level of the system. Type inventory. The 26 types (Appendix D) encode direction in their definitions: 𝑆 \ NP (type 3) means “takes subject on the left,” and (𝑆 \ NP)/ NP (type 4) means “takes object on the right, then subject on the left.” Removing direction would require merging types that differ only by slash direction, reducing the inventory and destroying the distinctions the supertagger learns to predict. CKY merge rules. The 37 merge rules (Ap­ pendix E) are keyed on ordered pairs (𝐿, 𝑅): the left span’s type and the right span’s type. Directionality enters through this ordering. For example, ((S \ NP)/ NP, NP) fires when the transitive verb is on the left and its object is on the right, assigning the theme role. The reverse pair (NP, 𝑆 \ NP) fires when the subject NP is on the left and the verb on the right, assigning the agent role. These are different rules with different re­ sults and different role assignments. Making the

merge table symmetric (accepting both orderings with the same result) would collapse agent and theme into a single undifferentiated role, elimi­ nating the information needed for semantic edge extraction. Semantic edge extraction. Role labels (agent, theme, recipient, nmod) are read directly from the merge rules during CKY backtracking. There is no separate role prediction component. Removing direction from the merge rules re­ moves the source of role information entirely. Restoring role assignment would require intro­ ducing a learned role predictor (an additional neural component), which would in substance reintroduce directionality through a different route: the predictor would need to learn, from po­ sitional context, the same left-right distinctions that slash direction encodes declaratively. In short, directionality is not a parameter of the system but the foundation on which the type inventory, merge rules, and edge extraction are jointly built. A “non-directional CCG” is not a variant of CCG but a different formal system requiring redesign of all three compo­ nents. This paper therefore employs cross-system category-level pattern analysis, comparing the per-category performance of two complete rep­ resentational systems (one directional, one nondirectional) under controlled conditions, rather than ablation experiments, to test the contribu­ tion of directionality. Directionality consistently outperforms the non-directional system on all 5 position-shift categories and consistently under­ performs on all 6 recursive-depth categories. This directional separation constitutes direct evidence for directionality as a representational property.

C Full Multi-Encoder Comparison Table 3 reports the 10-seed overall results for all encoders we tested. The relationship between encoder scale and performance is not monotonic: ELECTRA-large (69.9%) and ModernBERT-large (69.8%) show significant degradation relative to their base versions, RoBERTa-large (76.9%) is slightly better than RoBERTa-base (73.9%), while DeBERTa-v3-large (90.7%) substantially surpasses all base models. The degradation cases indicate that the 1024→32 Gumbel-Softmax bot­ tleneck is unfavorable for the representation spaces of some large models. The mechanism behind DeBERTa-v3-large’s success warrants further investigation (one hypothesis is that its

Encoder

Overall%

Std

BERT-base (control)

75.9

±6.4

ModernBERT-base

79.9

±3.7

DeBERTa-v3-large

90.7

±4.9

ELECTRA-base

77.3

±4.0

RoBERTa-large

76.9

±3.8

DeBERTa-v3-base

75.1

±5.4

RoBERTa-base

73.9

±7.4

ELECTRA-large

69.9

±6.4

ModernBERT-large

69.8

±5.4

Table 3: Multi-encoder 10-seed results (overall LF exact match mean±std). All configurations use the same Gumbel-Softmax bottleneck (K=32) and CKY composition.

disentangled attention design (He et al., 2023) may make position-direction information more linearly separable in the embedding space). Per-category analysis reveals that gains are highly non-uniform: the highest-variance cate­ gories under BERT-base benefit most from the encoder upgrade. CP recursion jumps from 18.8 ±34.4% (BERT-base) to 81.9±29.8% (DeBERTav3-large); Q_long_mv from 48.4±19.1% to 84.7 ±15.1%; PP_modif_subj from 92.8% to 94.5%. RC_modif_subj drops from 61.9% to 52.0%, a cat­ egory that is weak across all systems (AM-Parser 55.8%), suggesting that subject-side relative clauses pose a generalization challenge beyond directionality. No single category drops to zero under any encoder. Under DeBERTa-v3-large, 9/17 categories average above 97%, including the two AM-Parser 0% categories (RC_iobj_extracted 97.8%, Q_long_mv 84.7%). Overall variance does not decrease monotoni­ cally with encoder strength: ±6.4 → ±3.7 → ± 4.9. DeBERTa-v3-large’s slightly higher variance relative to ModernBERT-base (±4.9 vs. ±3.7) is driven by two low-scoring seeds on CP recur­ sion (seed 2: CP_3 2.7%, seed 1: Q_long_mv 43.4%), while these seeds perform normally on the remaining 15 categories, indicating that the discretization boundary remains highly sensitive to deep nesting structures. Even the worst seed (82.1%) exceeds AM-Parser by 11 percentage points.

D Complete CCG Type Inventory Table 4 lists all 26 CCG types used in this paper. Types 1–19 cover the 20 base categories for COGS (21 generalization categories). Types 20– 23 extend coverage to SLOG’s relative clauses and

wh-questions. Types 24–25 are disambiguation types that resolve role ambiguities in ditransitive verbs and RC indirect-object extraction. We retain forward and backward function application but replace type raising and compo­ sition with atomic types carrying explicit merge rules. Standard CCG correspondences: wh-words map to WH (standard: 𝑆𝑞 /(𝑆 \ NP)); relative pronouns map to RC_THAT (standard: (NP \ NP)/(𝑆/ NP)). Type raising (NP → 𝑆/(𝑆 \ NP)) and composition (𝐴/𝐵 + 𝐵/𝐶 → 𝐴/𝐶) are not used.

E Complete CKY Merge Rules Table 5 lists all 37 deterministic merge rules used by the CKY parser. Each rule specifies: the left and right input types, the result type, any predicateargument edge extracted (role and direction), and which side retains the head. These rules, together with the type inventory in Appendix D, fully specify the deterministic pipeline from CCG type sequences to logical forms. Directionality is encoded in the ordered pair: rule (𝐴, 𝐵) fires only when 𝐴 occupies the left span and 𝐵 the right span. The same two types in reversed order either do not match any rule or match a different rule with a different result and role assignment. This is the concrete mechanism by which slash direction enters the system.

F Worked Example: From CCG Types to Logical Form This section traces the complete deterministic pipeline on a concrete sentence, from raw input to extracted predicate-argument edges.

ID

Name

CCG notation

Description

0

∅

1

NP

NP

Noun phrase

2

S

S

Sentence

3

S_NP

𝑆 \ NP

Intransitive verb (takes subject left)

4

SNP_NP

(𝑆 \ NP)/ NP

Transitive verb (takes object right, subject left)

5

PP_NP

PP / NP

Preposition (takes complement right)

6

NP_NP

NP \ NP

Nominal modifier (attaches left to NP)

7

DIT

DIT

Ditransitive verb stem

8

CCOMP

CCOMP

Clausal complement verb

9

THAT_S

THAT_S

Complementizer “that”

10

PN

PN

Proper noun (treated as NP)

11

XCOMP

XCOMP

Raising/control verb

12

UNACC

UNACC

Unaccusative verb (single theme argument)

13

PASS_TV

PASS_TV

Passivized transitive verb

14

PASS_DIT

PASS_DIT

Passivized ditransitive verb

15

PASS_REC

PASS_REC

Passivized ditransitive (recipient promoted)

16

BY

BY

Passive agent marker “by”

17

BY+

BY+

BY after absorbing its NP complement

18

TO

TO

Dative marker “to”

19

TO+

TO+

TO after absorbing its NP complement

20

RC_THAT

RC_THAT

Relative pronoun (theme/agent gap)

21

TV_GAP

TV_GAP

Transitive verb with object gap

22

S_GAP

S_GAP

Sentence with extracted argument

23

WH

WH

Wh-word

24

DIT_REC

DIT_REC

Disambiguation: ditransitive, recipient reading

25

RC_THAT_R

RC_THAT_R

Disambiguation: RC with recipient gap

Void (absorbed position)

Table 4: Complete CCG type inventory (26 types). ID 0 (∅) marks absorbed positions after CKY merges.

Input sentence: “The dog chased the cat be­ side a car.” Step 1: Function word stripping. Articles (“The”, “the”, “a”) and other function words are removed. These words do not carry CCG types in the COGS/SLOG formalization; removing them aligns the word sequence with the CCG type sequence. Content words retained: Posi­ tion 0

Posi­ tion 1

Posi­ tion 2

Posi­ tion 3

Posi­ tion 4

dog

chased

cat

beside

car

Step 2: CCG type assignment. The supertagger predicts one of 26 CCG types for each content word. The gold (target) type sequence is: dog NP

chased

cat

beside

car

(𝑆 \ NP)/ NP NP PP / NP NP

Step 3: Deterministic CKY parsing. Bottomup CKY applies the merge rules from Appendix E. The derivation proceeds from the innermost (rightmost) constituents outward: Merge 1 (span [3,4]): PP / NP + NP → NP \ NP beside + car. Head = car (R). No edge extracted. The preposition absorbs its complement, form­ ing a nominal modifier. Merge 2 (span [2,4]): NP + NP \ NP → NP cat + [beside car]. Head = cat (L). Edge: R→ L nmod → cat.nmod(cat, car). The modifier attaches to the noun. Direction­ ality: NP\NP attaches leftward (backward applica­ tion). Merge 3 (span [1,4]): (𝑆 \ NP)/ NP + NP → 𝑆 \ NP chased + [cat beside car]. Head = chased (L).

Left

Right

Result

Edge

Head

Core argument structure (𝑆 \ NP)/ NP

NP

𝑆 \ NP

L→R: theme

L

NP

𝑆 \ NP

S

R→L: agent

R

NP

UNACC

S

R→L: theme

R

DIT

NP

(𝑆 \ NP)/ NP

L→R: recipient

L

DIT_REC

NP

𝑆 \ NP

L→R: recipient

L

Modification and complementation PP / NP

NP

NP \ NP

R

NP

NP \ NP

NP

R→L: nmod

L

CCOMP

S

𝑆 \ NP

L→R: ccomp

L

THAT_S

S

S

XCOMP

𝑆 \ NP / UNACC

𝑆 \ NP

L→R: xcomp

L

R

Passive constructions NP

PASS_TV

S

R→L: theme

R

NP

PASS_REC

S

R→L: recipient

R

PASS_DIT

NP

PASS_REC

L→R: theme

L

PASS_TV

BY+

PASS_TV

L→R: agent

L

PASS_TV

TO+

PASS_TV

L→R: recipient

L

PASS_REC

BY+

PASS_REC

L→R: agent

L

𝑆 \ NP

TO+

𝑆 \ NP

L→R: recipient

L

BY

NP

BY+

R

TO

NP

TO+

R

Relative clauses and wh-questions (19 rules) NP

TV_GAP

S_GAP

RC_THAT

𝑉any

NP \ NP

R

RC_THAT_R

𝑉any

NP \ NP

R

WH

S / 𝑆 \ NP / UNACC / PASS_TV / PASS_REC

S

R

R→L: agent

R

Table 5: Complete CKY merge rules (37 rules). L→R: left span assigns role to right span’s head. R→L: right span assigns role to left span’s head. Head: which span’s head survives. 𝑉any abbreviates the set {𝑆 \ NP, UNACC, PASS_TV, PASS_REC, PASS_DIT, S, S_GAP}.

Edge: L→R theme → chase.theme(chased, cat). The transitive verb absorbs its object on the right (forward application). The slash direction / NP determines that the right-side constituent is the theme. Merge 4 (span [0,4]): NP + 𝑆 \ NP → 𝑆 dog + [chased cat beside car]. Head = chased (R). Edge: R→L agent → chase.agent(chased, dog). The subject NP is consumed on the left (back­ ward application). The slash direction \ NP deter­ mines that the left-side constituent is the agent. Step 4: Edge collection. The three edges ex­ tracted during CKY backtracking constitute the logical form:

chase.agent(chased, dog) ∧ chase.theme(chased, cat) ∧ cat.nmod.beside(cat, car) This edge set is compared against the gold LF edge set for evaluation (LF exact match). The entire pipeline from type prediction to LF is deterministic: no learned component participates after the supertagger’s type prediction in Step 2. Role of directionality. In Merges 3 and 4, the role assignment (theme vs. agent) is determined entirely by which side of the verb the NP appears on: right → theme (forward application), left → agent (backward application). In a non-direc­ tional system, both merges would produce the same undifferentiated NP+verb combination, and distinguishing agent from theme would require an additional learned component.

G Shared Variance Structure Per-category data reveal a pattern beyond overall accuracy. Both systems (despite using different type representations) exhibit categories where the standard deviation approaches or exceeds the mean. Our system shows 34.4% standard de­ viation on CP recursion, with individual seeds jumping from 0% to 99.6%. AM-Parser shows 33.5% standard deviation on Q_dobj_ditransV and 42.4% on Q_iobj_ditransV, with standard deviations approaching or exceeding the means themselves. The choice of representation changes where variance falls (ours on CP recursion, AM-Parser’s on Q_dobj), but does not eliminate variance. The two systems share the same paradigm (BERT supertagging + symbolic combinatory backend), the same training data, and differ only in type representation, yet both exhibit high-variance categories. This suggests that supertagger initial­ ization sensitivity on certain categories cannot be eliminated by representation choice alone: directionality can relocate instability but cannot remove it. This pattern implies that initialization sensitivity is an inherent cost of the neuro-sym­ bolic paradigm’s separation of statistical learning from symbolic reasoning, a cost that representa­ tion choice can redistribute but not eliminate.