arXiv CS open access

SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

S AGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models Hoda Fakharzadehjahromy1 Emil Wiman1 Andreas Bueff1 Hafsteinn Einarsson2 Fredrik Heintz1 1 Linköping University 2 University of Iceland

arXiv:2608.06179v1 [cs.LG] 6 Aug 2026

Abstract Preference optimisation has proven effective for improving large language models but typically relies on costly human preference annotations. Extending these methods to morphologically rich, low-resource languages remains challenging because such annotations are scarce. We present S AGA (Score-weighted Adaptive Generation Alignment), a parserguided preference optimisation framework that replaces human labels with dependency-parser supervision. S AGA converts parser judgements into preference pairs for ∆-DPO, combines parser quality with lexical diversity in a composite reward, filters low-information pairs using a reward-gap criterion, and monitors reward hacking to maintain reliable supervision. Across Danish, Icelandic, and Norwegian Bokmål using GPT-SW3-1.3B, S AGA consistently improves grammatical quality without requiring human preference labels. Danish parse success increases from 69.0% to 93.8%, Icelandic achieves a +4.5 percentage-point improvement on an independent Stanza evaluation (three-run mean +3.3 percentage points (pp)) while native speakers prefer S AGA outputs in 80% of pairwise comparisons, and Norwegian Bokmål improves by +28 percentage points. These results demonstrate that parser-derived supervision is a practical alternative to human preference annotation for grammatical alignment in low-resource languages where high-quality dependency parsers are available.

Introduction

Grammatical correctness is a prerequisite for usable text generation, yet it remains a persistent weakness of small, low-resource language models. Consider a Nordic news-completion task: given the fragment “Þingið samþykkti frumvarpið um” (Icelandic: “The parliament approved the bill on”), a fine-tuned GPT-SW3-1.3B model frequently produces continuations with incorrect case agreement. These errors are invisible to standard token-level metrics but immediately apparent to native speakers. The standard remedy would be preference optimisation. Reinforcement learning from human

Figure 1: Parser-derived supervision replaces human preference labels: the incorrect case (Ég) fails dependency parsing, whereas the correct case (Mig) is accepted, providing a preference signal without human annotation.

feedback (RLHF) and Direct Preference Optimization (DPO) (Rafailov et al., 2023) have transformed alignment for English, but both rest on the same costly foundation: large collections of human preference judgements. For low-resource languages this foundation is largely missing: qualified annotators are scarce, annotation budgets rarely extend to minority languages, and preference datasets at the scale English alignment assumes do not exist. Human preference data, rather than model capacity, is therefore the primary bottleneck for aligning language models in these communities. Yet many of these languages already possess high-quality dependency parsers, developed over decades of linguistic research and distilling treebank annotation into fast, reproducible judgements about morphology and syntax. Rather than collecting new preference labels, we ask whether these existing resources can provide the supervision needed for preference optimisation. This question moti-

vates S AGA: the parser’s verdict on a generated continuation (the wrong case fails to parse, the correct case passes — Figure 1) provides a parserderived preference signal without requiring human preference labels, at no annotation cost. Parser-derived supervision, however, raises three challenges. Reward hacking: a model can learn surface patterns that fool the parser while producing incoherent text (Pang et al., 2023; Gao et al., 2023). Alignment tax: optimising a singledimension reward may degrade the general fluency acquired during pretraining (Lin et al., 2024). Data scarcity: low-resource base models generate weak preference candidates; Icelandic, for example, constitutes just 2.7% of the Nordic Pile pretraining corpus (Ekgren et al., 2024). S AGA addresses these challenges through composite rewards, quality-aware pair filtering, and lightweight regularisation validated by ablation. Our contributions are: • S AGA: a parser-supervised preference optimisation framework that converts on-policy parser judgements into ∆-DPO preference pairs, requiring no human preference labels. • Multilingual evidence that parser-derived supervision improves grammatical quality across Danish, Icelandic, and Norwegian Bokmål, with gains confirmed by both automatic metrics and native-speaker evaluation. • A reward hacking diagnostic that identifies oracle/parser divergence, lexical collapse, and repetition as key failure modes, together with composite rewards and lightweight regularisation that make parser supervision reliable.

Related Work

RL from verifiable feedback. RLHF (Christiano et al., 2017; Ouyang et al., 2022) trains LMs from human judgements, but annotation is expensive. Verifiable rewards give exact signals for math (Cobbe et al., 2021; Shao et al., 2024) and code; grammar is a more challenging target because parser-based rewards capture only one aspect of linguistic quality and may therefore be exploited. Reward hacking (Skalse et al., 2022; Gao et al., 2023; Pang et al., 2023) and alignment tax (Lin et al., 2024) are the main practical risks; our composite reward, detection module, and BAPO address both directly.

Preference optimization. DPO (Rafailov et al., 2023) recasts RLHF as a classification loss over (chosen, rejected) pairs, avoiding an explicit reward model. The weakness is pair quality: pairs close to the decision boundary produce inconsistent gradients, and static datasets cannot track a changing policy. Recent variants remove the reference model (Meng et al., 2024) or generate pairs iteratively from the current policy (Chen et al., 2024). We instead generate preference pairs from parser judgements over on-policy outputs and filter by quality gap: the reward margin between chosen and rejected responses strongly predicts training quality (Lambert et al., 2025), and filtering lowmargin pairs consistently improves convergence (Liu et al., 2024). We adopt ∆-DPO because it naturally supports on-policy preference generation and quality-gap filtering, both of which are central to S AGA; pairs whose gap falls below a perlanguage threshold δ ∈ [0.10, 0.15] are discarded (Appendix A). Parser-reward alignment. Using automatic parsers as reward signals for generative quality improvement has been explored for English. Existing parser-based methods primarily focus on English or use parsers only as evaluation signals rather than as the primary supervision source for preference optimisation. Zhang et al. (2026) propose CroCo, which uses cross-lingual signals to score withinlanguage preference pairs — our cross-lingual warm-start is complementary, transferring syntactic alignment via model weights rather than via a scoring signal. Multi-objective alignment methods balance competing reward signals dynamically (Yang et al., 2024); we instead use a fixed-weight composite of parse quality and lexical diversity, keeping reward weighting simple, interpretable, and consistent across languages. Parsing infrastructure and Nordic language resources. Universal Dependencies (Nivre et al., 2020) harmonises syntactic annotation across 90 languages. Stanza (Qi et al., 2020) provides UDtrained parsers for 66 languages; we use its Icelandic model trained on IcePaHC as an independent IS evaluation metric, independent of the Greynir reward parser (Section 5.2). SpaCy models trained on DDT (Johannsen et al., 2015) and the Norwegian UD treebank (Øvrelid and Hohle, 2016) serve as training oracles for DA and NB. For the Nordic setting we build on dedicated pretrained models (Ekgren et al., 2024; Silo AI, 2024) and evaluation

infrastructure (Nielsen, 2023; Samuel et al., 2023; Snæbjarnarson et al., 2022).

The S AGA Framework

3.1

Pipeline Overview

3.4

Figure 2 illustrates the S AGA pipeline. Each pass generates K ∈ {8, 16} candidates per prompt and scores them with the composite reward (Eq. 1). Preference pairs with ∆r≥δ are retained for ∆DPO training, optionally regularised with BAPO and KL-SFT where ablations show benefit (Section 6; Table 6). The merged checkpoint initializes the next iteration. 3.2

Multi-Reward Composite Score

Let g denote a generated continuation for prompt p. The composite reward is: (1)

R(g, p) = α rparse (g) + β rdiv (g)

Parse reward rparse : mean fraction of tokens assigned a valid dependency arc (spaCy for DA/NB/SV; Greynir (Thorsteinsson et al., 2019) for IS). For IS, Greynir additionally emits an integer sentence score s normalised as σ(s/τ ); calibration details are in Appendix A; unscored parses map to 0.5. Diversity reward rdiv : MATTR (window 100 tokens) (Covington and McFall, 2010), penalising degenerate repetition. We use MATTR because its sliding-window formulation provides a lengthnormalised estimate of lexical diversity that is less sensitive to generation length than type-token ratio. Weights: (α, β)=(0.80, 0.20) for DA pass 1, IS, NB; (0.65, 0.35) for DA iter 2+/SV (see the MATTR-weight ablation in Section 6). 3.3

Cross-Lingual Warm-Start

Danish and Swedish share V2 word-order constraints and cognate vocabulary. A S AGA-trained DA checkpoint transfers dependency-consistent constructions to SV via checkpoint initialization alone, without requiring parallel data or a crosslingual scoring signal, unlike CroCo (Zhang et al., 2026), which uses cross-lingual supervision to rank within-language candidates. Concretely, for target language Lt , we initialize ∆-DPO from the source-language merged checkpoint: (L )

t θDPO = arg min LDPO θ, θ(Ls ) ; D(Lt )

This reduces the SV pipeline to a single ∆-DPO pass. Experimental results are presented in Section 5.4.

(2)

Base-Anchored Policy Optimization

Standard DPO can suppress the probability of grammatical continuations absent from the training pairs. BAPO (Lee et al., 2024) addresses this via a onesided hinge applied here to the chosen response yw : LBAPO (θ) = LDPO (θ) +λB ED max 0, log π0 (yw |x)−log πθ (yw |x) (3) The penalty is applied only when the updated policy assigns lower probability to the chosen response than the frozen base policy. Unlike Lee et al. (2024), who anchor on a separate base response, we anchor on the chosen response. This preserves capabilities represented in the preference pairs but does not protect behaviours outside them. Perlanguage λB settings appear in Table 7; IS ablation in Section 5.2. 3.5

Automated Reward-Hacking Detection

Before evaluation, every trained model is screened for three failure modes. (1) Oracle–parser divergence: a large gap between the training oracle and an independent parser indicates that the model is gaming the reward rather than improving grammatically. (2) MATTR collapse: a large drop in lexical diversity from base indicates degenerate, repetitive outputs. (3) Repetition: high 4-gram self-BLEU confirms the collapse. Thresholds are calibrated using baseline models: oracle/parser gap >0.20, MATTR drop >0.15, or 4-gram repetition >0.15 (Appendix L). Models failing any check are marked ‡ in tables.

Experimental Setup

Base models. GPT-SW3-1.3B (AI-Sweden/gpt-sw3-1b3) serves as the primary backbone for all three languages (DA, IS, and NB). Training data. For DA: Danish Wikipedia (20k samples, DDT-filtered (Johannsen et al., 2015)). For IS: MIM-GOLD corpus (Loftsson et al., 2010) (20k samples, Greynir-filtered). For NB: HPLT nob_Latn (de Gibert et al., 2024) (20k samples, quality-filtered at q≥0.65). Candidates are scored

SAGA: Score-weighted Adaptive Generation Alignment Parser-guided Preference Optimisation without Human Labels

(i)

Figure 2: S AGA pipeline. (1) Sample: πθ (frozen π0 + LoRA) draws K∈{8, 16} completions per prompt (DA K=8; IS/NB K=16; T ∈{0.7, 1.1}, rep. penalty 1.3); DA→SV warm-start initializes SV. (2) Score & Filter: R=αrparse +βrdiv ranks outputs; ∆-DPO gap filter retains pairs with ∆r≥δ (per-language; DA 0.15, IS 0.15, (i+1) NB 0.10). (3) Update: LDPO (+ λB LBAPO where beneficial); merged πθ seeds the next iteration.

using Equation (1), and pairs with ∆r < δ are discarded. Evaluation prompts are held out from training: n=200 for Oracle PS, n=400 for Stanza PSR. Full hyperparameters and per-language configurations are in Appendix A. Hyperparameter selection. DPO hyperparameters (β=0.1, lr, batch size) follow standard practice (Rafailov et al., 2023). Per-language parameters (δ, λB , iteration count) were selected by monitoring the independent evaluation metric (Stanza PSR for IS; held-out oracle parse score for DA/NB) and stopping once an additional training pass no longer improved the validation metric; these choices were fixed before the final reported numbers were computed. The same held-out evaluation prompts were used for model selection and final reporting, which may introduce mild selection bias. We discuss this limitation in Section 7. For IS, the IcePaHC Stanza evaluation is fully disjoint from training data by corpus and time period.

Evaluation. Parse quality. Parse Success Rate (PSR: fraction of sentences with ≥1 valid arc) and mean parse score via Stanza Universal Dependencies (Qi et al., 2020; Nivre et al., 2020) (independent parser, no shared rules with training oracles), Greynir, and spaCy. Lexical diversity. MATTR (window 100 tokens). Perplexity (PPL) is reported as a secondary diagnostic.

Results

5.1

Danish

S AGA raises DA PSR 69% → 87.3% in a single pass (n=200; Wilson 95% CI [82.0%, 91.2%]) and to 93.8% after a second iterative pass; DA continues to 99.5% over four passes (ablation anchor, Table 6; validated across 6 seeds, mean 96.9%, std 1.1pp). The oracle/Stanza gap is 0.147 (below the 0.20 rejection threshold) and independent Stanza UD evaluation rises 86% → 99%, indicating that the

Table 1: Danish results. PSR uses SpaCy da_core_news_lg (training oracle); held-out Stanza UD PSR: base 86%, iter 2 99%, confirming SpaCy PSR is not inflated by circular evaluation. Gap = SpaCy / Stanza parse-quality gap (>0.20 = reward hacking, ‡). † p<0.001 vs. base (two-proportion z-test, n=200; same seed for base and S AGA). “+ parse only” PSR of 61% is pass-1 (single training pass); the ablation’s “parse score only” at 77.0% (Table 6) is the iter-4 version of the same condition, confirming that even with more iterations, parse-score-only reward triggers hacking and underperforms S AGA. Wilson 95% CI: pass-1 [82.0, 91.2]%, iter-2 [89.6, 96.4]%. Training continues past iter-2: the full four-iteration system reaches 99.5% PSR (Score 0.747, PPL 19.8), reported as the ablation anchor in Table 6.

Table 2: Icelandic results (GPT-SW3-1.3B, MIMGOLD 20k, δ=0.15). Stanza PSR = Stanza UD parse success on IcePaHC (n=400; primary independent metric; different parser and domain from training oracle; † n=200 for pass 2). Score = Stanza parse-quality score on IcePaHC (n=400). MATTR = lexical diversity. Oracle PS∗ = Greynir parse success rate (n=200; also the training reward; reported as corroborating metric). IcePaHC is a 10th–18th century treebank; its archaic morphology causes a ≈0.35 Greynir/Stanza gap before training. The oracle/Stanza gap stays flat after training (0.465→0.462), inconsistent with reward gaming. Best Stanza PSR in bold; S AGA canonical result shaded. Human pairwise (Table 5): 80% native-speaker preference (N =129, p<0.001). System

System

PSR Score MATTR Gap

GPT-SW3-1.3B base 69.0 + parse only‡ 61.0 + S AGA pass 1 (ours) 87.3† + S AGA iter 2 93.8†

0.341 0.312 0.581 0.734

0.903 0.831 0.909 0.906

— >0.20 0.147 0.100

gains generalize beyond the training parser. Ablation details and component analysis are in Section 6. 5.2

Icelandic

Icelandic is the most challenging alignment target because rich morphological case inflection makes grammatical errors immediately visible to the parser. Consequently, the parser reward is highly informative but also more susceptible to reward exploitation than in the other languages. S AGA is trained on 20k MIM-GOLD sentences with δ=0.15 (ablation in Appendix B) and uses no human preference labels at any stage. Three independent signals indicate genuine improvement. First, native speakers prefer S AGA outputs in 80% of pairwise comparisons (N =129, p<0.001). The Fable-5 judge reaches a similar conclusion (83%, n=25, p<0.01), although human evaluation remains the primary evidence. Both evaluations are driven primarily by fluency (∆Gr ≈ 0). Second, independent Stanza PSR improves by +3.3pp across three runs (79.1% ± 1.0pp; best run +4.5pp, seed 42; IcePaHC n=400). Third, BÍN morphological recognition improves by +0.8pp (p=0.059). Greynir Oracle PS, the training reward, increases by +18.3pp and is reported as corroborating rather than primary evidence.

GPT-SW3-1.3B base + S AGA pass 1 (ours) + S AGA pass 2

Stanza PSR Score MATTR Oracle PS∗ 75.8 80.3 72.0†

0.364 0.404 0.393

0.840 0.912 0.917

70.4 88.7 85.4

Independent evaluation. Independent evaluation is particularly important because Greynir provides both the training reward and the oracle evaluation. Agreement across Stanza, nativespeaker preferences, and BÍN provides complementary evidence that the observed gains reflect grammatical improvement rather than optimisation of a single parser. Stanza (IcePaHC, n=400): 75.8% → 80.3% (three-run mean 79.1% ± 1.0pp; best run +4.5pp, seed 42). The oracle–Stanza gap remains essentially unchanged (0.465 → 0.462), arguing against reward hacking. BÍN (n=3 000): +0.8pp. MATTR 0.840 → 0.912; 4-gram repetition 0.048 → 0.004. Greynir Oracle PS (threeseed): 88.5% ± 0.9pp. Figure 3 illustrates why independent evaluation is necessary. Although sDPO reaches Oracle PS 82.7%, its Stanza PSR regresses to 71.8% (below the 75.8% base), showing that the training reward can be optimised without improving syntactic quality. S AGA (−BAPO) is the only method that improves both metrics simultaneously. The δ sweep and RL ablations (Appendix B, C) further support robustness: vanilla DPO collapses to 59.7% Oracle PS, while removing BAPO improves both Oracle PS (+3.6pp) and Stanza PSR (+5.0pp), indicating that BAPO is not universally beneficial. Human judgment. Native-speaker pairwise evaluation (n=43 pairs; pair-level sign test, p<0.001) prefers S AGA in 80% of non-tied comparisons (64% overall). A Fable-5 judge evaluated 25 independently sampled IS pairs (A/B randomised

Score (%)

Oracle PS (%) — training reward Stanza PSR (%) — independent Reward hacking detected

88.7

80.3

↓Stanza base Stanza

Table 3: Cross-lingual transfer: DA→IS. Oracle PS = Greynir parse success rate (n=200; primary eval). The DA→IS warm-start initialises GPT-SW3-1.3B from the DA merged checkpoint and applies IS Greynir reward directly (no KL-SFT, δ=0.15). From-base runs ∆-DPO (δ=0.15, −BAPO, seed 42) on IS data only. System

Oracle PS Score PPL

GPT-SW3-1.3B base From-base ∆-DPO (δ=0.15, −BAPO) DA→IS warm-start (ours)

50 Base

DPO

GRPO

sDPO SAGA −BAPO

IS training method

Figure 3: IS RL method comparison (GPT-SW31.3B). Circles = Oracle PS (training reward, n=200); squares = Stanza PSR (independent, n=400). sDPO reaches 82.7% Oracle PS yet Stanza PSR regresses to 71.8% (below the 75.8% base, dotted line), indicating the training reward can be maximised without genuine syntactic improvement. S AGA −BAPO is the only method where both metrics exceed the base simultaneously (88.7% / 80.3%). Orange diamonds: reward hacking detected. Full results in Table 9 (Appendix C).

per pair, seed 42; one judgment per pair; T =0.1; Appendix M) and reached the same conclusion, preferring S AGA in 83% of non-tied comparisons (p<0.01). IS is the most morphologically complex language in this study, where parser and human notions of grammaticality could plausibly diverge most. Instead, native speakers show the strongest preference for the parser-trained model across all three languages, with the automated judge providing independent corroboration. Likert analysis indicates that the gains are driven primarily by fluency (∆Fl. +0.51), with little change in grammaticality ratings (∆Gr. +0.01); the syntactic claim is supported by Stanza and the morphological trend by BÍN. Analysis. Iteration budget as a per-language hyperparameter. A second ∆-DPO pass degrades IS performance (Oracle PS 85.4%, −3.3pp relative to pass 1), with independent Stanza evaluation declining as well, whereas DA gains +6.5pp from a second pass. This asymmetry suggests that morphologically rich languages saturate more quickly: once parser-exploitable errors have largely been corrected, continued optimisation can reduce generalisation. Accordingly, pass 1 (−BAPO) is the canonical IS configuration. Iteration counts should therefore be selected using an independent evalu-

70.4 88.7 80.9

0.364 0.404 0.422

17.0 19.5 20.3

Table 4: Norwegian results. PSR and Score use SpaCy NB (nb_core_news_lg); Gap = oracle/Stanza gap (‡ = > 0.20). Trained on HPLT nob_Latn (20k samples), δ=0.10. † p<0.001 vs. base (two-proportion z-test, n=200). Held-out Stanza UD PSR (different prompt set): base 89.0%, S AGA 97.5% (+8.5pp, p<0.001), confirming improvement generalises beyond SpaCy oracle. Wilson 95% CI for S AGA SpaCy PSR: [90.4, 96.9]%. System

PSR

Score MATTR

GPT-SW3-1.3B base 66.5 0.333 + S AGA (ours) 94.5† [90.4, 96.9]% 0.645

0.900 0.929

Gap — 0.156

ation metric rather than the training reward alone. Grammar specialisation also incurs a measurable trade-off: IS factual recall on nqii decreases from 24.5% to 14.3% (Section 7). Cross-lingual transfer. Table 3 compares training IS from scratch with DA→IS warm-start initialisation. Training from scratch achieves the highest Oracle PS (88.7%), whereas the warm-start reaches 80.9% while improving overall quality (0.422 vs. 0.404). The warm-start therefore trades maximum parse performance for improved Nordic fluency, making it attractive when fluency is prioritised over parse accuracy. 5.3

Norwegian

We train on 20k HPLT nob_Latn samples (de Gibert et al., 2024) using δ=0.10 and no human preference labels. S AGA improves NB PSR 66.5% → 94.5% (+28.0pp, p<0.001; 95% Wilson CI [90.4%, 96.9%]) and the parse score 0.333 → 0.645 (Table 4). Held-out Stanza UD evaluation confirms the improvement (89.0% → 97.5%, +8.5pp, p<0.001). Lexical diversity also improves: MATTR 0.900 → 0.929 and 4-gram repetition 0.022 → 0.003. The oracle–Stanza gap remains low (0.156), well below the 0.20 rejection threshold, indicating grammatical improvement without evidence of reward hacking or lexical

Table 5: Pairwise quality evaluation (A/B positions randomised per pair for all evaluations, seed 42). Win% excludes ties; W/T/L are judgment-level counts (NB: pair count shown). IS: 3 annotators/pair; DA: 5 Prolific speakers, 3/pair; NB‡ : 5 annotators/pair; IS LLM: 1 judge/pair. ∆Gr./Fl. = mean Likert 5-pt grammar/fluency gain (S AGA vs. base); IS (human) Likert: Gemini 2.5 Flash automated† ; IS (Fable-5) Likert: rated by the same Fable-5 judge§ ; DA/NB Likert: Prolific native speakers. p-values: pair-level sign test, one-sided. Lang.

Comparison

IS (human) IS (Fable-5) DA (human) NB (human)‡

S AGA vs. base 83/25/21 S AGA vs. base 19/2/4 S AGA vs. base 59/46/27 S AGA vs. base 49 pairs

W/T/L Win% 80% 83% 69% 60%

∆Gr.

∆Fl.

<0.001 +0.01† <0.01 +0.04§ <0.001 +0.70 <0.01 +0.12

+0.51† +0.84§ +0.50 +0.12

collapse. Results on alternative backbones are reported in Appendix J. The canonical 94.5% PSR is measured on in-distribution HPLT prompts; on Wikipedia prompts (Appendix J), performance decreases to 74.2%, reflecting the expected domain shift, but remains above the corresponding base model (68.1%). NB human evaluation uses GPT-SW3-1.3B. Because the base model already achieves a high grammaticality score (4.70/5.0 Likert), there is limited headroom for perceptible improvement, explaining the smaller preference margin relative to Danish and Icelandic. 5.4

Cross-Lingual Transfer to Swedish

Initialising ∆-DPO from a DA-trained checkpoint and fine-tuning on Swedish Wikipedia reaches 97.0% PSR, statistically indistinguishable from training from the base model (97.5%; p=0.76). However, the warm-start preserves fluency significantly better (PPL 29.7 vs. 41.7, p<0.001, paired bootstrap, n=500), indicating that parse-aligned initialisation mitigates the fluency regression observed when training directly from the base model. Full results and the transfer figure are provided in Appendix I. 5.5

Pairwise Preference Evaluation

Across all three evaluated languages, native speakers prefer S AGA outputs over the base model (Table 5), indicating that parser-derived supervision improves text quality as judged by humans rather than parsers alone. Likert ratings suggest that the gains are driven primarily by fluency. IS (3 native speakers, n=43 pairs; pair-level sign test, p<0.001): 80% prefer S AGA excluding ties

(64% overall), the strongest preference observed across the three languages despite Icelandic being the most morphologically complex setting. Likert ratings show almost no change in grammaticality (∆Gr. +0.01) but a substantial improvement in fluency (∆Fl. +0.51). DA (5 Prolific native speakers, 3/pair, n=44 pairs; pair-level sign test, p<0.001): 69% prefer S AGA excl. ties (45% overall; ∆Gr. +0.70, ∆Fl. +0.50); S AGA is preferred in all ten threeannotator subsets (Krippendorff α=0.12, 3-way). NB (5 Prolific native speakers, n=49 pairs, GPT-SW3-1.3B): 60% prefer S AGA (p<0.01; ∆Gr. +0.12). The smaller preference margin is consistent with the near-ceiling grammaticality baseline (4.70/5.0 Likert).

Ablation Study

S AGA’s premise is that parser-derived rewards can replace the human preference labels that lowresource languages lack. The core algorithm — composite reward and ∆-gap pair filter — is active in every language; BAPO and KL-SFT are optional per-language regularisers enabled only where ablations show consistent benefit. Table 6 isolates each component’s contribution on Danish (iter-4 anchor, 99.5% PSR); IS and NB saturate after a single pass, while DA continues to improve through iter-4 (Appendix J). Reward design. Removing the diversity reward (β=0) raises PSR to 98.5% but widens the oracle– parser gap to 0.22, triggering the reward-hacking diagnostic. With the diversity term, the gap stays near 0.10 and MATTR improves across all languages (DA 0.903 → 0.909, IS 0.840 → 0.912, NB 0.900 → 0.929). Replacing the graded reward with binary 0/1 parser verdicts yields PSR = 69% with severe hacking (MATTR 0.90 → 0.66, 4-gram repetition 32.7%, gap 0.29). Figure 4 summarises this trade-off: graded parser rewards trace the Pareto frontier, whereas all ablations fall below it. Regularisation. KL-SFT is unnecessary during the first training pass (−2.3pp) but improves stability across later iterations. Removing it in iter-4 reduces PSR by 3.5pp and increases PPL by 7.5. BAPO is language dependent. IS performs best without BAPO (λB =0): removing it improves Oracle PS by 3.6pp and Stanza PSR by 5.0pp. DA and NB benefit from λB =0.05. The DA isolation experiment (Table 6) illustrates this trade-off. Re-

Table 6: Ablation on Danish (top), KL-SFT weight sensitivity on NB (middle), and fidelity-reward results for IS and NB (bottom). Full S AGA iter-4 vs. componentremoved variants. ‡ reward hacking detected (MATTR drop >0.15, n-gram repetition >0.15, or oracle/Stanza gap >0.20). IS rows use Stanza PSR; IS base = 75.8%; NB rows use SpaCy PSR. For NB, KL-SFT at all λ>0 values degrades performance (HPLT corpus distribution is misaligned with the parsing task); λ=0 (no warm-up) is the NB canonical result. NB PPL for λ=0.10/0.05 rows not measured (—). Variant

PSR Score PPL

Full S AGA iter-4 − KL-SFT (λ=0) − ∆-DPO filter − MATTR (β=0)‡ parse score only‡

99.5 96.0 95.5 98.5 77.0

19.8 27.3 32.7 32.5 12.2

0.10 0.11 0.09 0.22 0.27

MATTR weight = 0.275 MATTR weight = 0.350

97.0 0.653 24.9 97.0 0.659 26.4

0.11 0.10

Reward signal comparison (DA, 1.3B) Binary-label DPO (0/1 parser verdicts)‡ 69.0 0.382 54.3

0.29

BAPO isolation (DA, iter-2) SAGA iter-2 + BAPO (λB =0.05) SAGA iter-2 − BAPO (λB =0)

0.747 0.721 0.660 0.686 0.385

Orc.Gap

93.8 0.764 19.9 98.3 0.866 26.8

0.117 0.035

NB KL-SFT weight sensitivity (HPLT corpus, 1.3B) NB S AGA (no warm-up, λ=0) 94.5 0.645 21.8 NB + KL-SFT warm-up (λ=0.25) 69.5 0.403 19.2 NB + KL-SFT warm-up (λ=0.10) 31.4 0.258 — NB + KL-SFT warm-up (λ=0.05) 28.8 0.272 —

0.156 0.248 0.056 0.016

Fidelity reward ablation IS + fidelity‡ NB + fidelity

69.0 0.358 37.2 0.43 0 candidate pairs — no training

Grammar quality (Oracle PSR, %, ↑)

Higher grammar & diversity %

SAGA iter 4

SAGA iter 2

fro

nti

Pa re

−KL-SFT

−∆ filter

NB KL-SFT warm-ups (λ>0) degrade PSR across all tested values (94.5% → 69.5%/31.4%/28.8% for λ=0.25/0.10/0.05; Table 6). Lower KL weights produce larger degradations because weaker regularisation during warm-up permits greater HPLT-specific SFT drift, making subsequent ∆-DPO optimisation harder. Direct ∆-DPO from base (λ=0) is therefore the NB canonical. Removing the ∆ filter costs −4pp PSR on DA. On IS, the unfiltered on-policy variant (δ=0.00, Appendix B) achieves comparable Stanza PSR (80.8% vs. 80.3%) under the fixed-compute sweep, and is thus competitive per-language. However, the same configuration is not robust across settings: vanilla DPO regresses Oracle PS to 59.7% on IS and collapses on DA summarisation transfer (PS 0.015). The filter therefore provides consistent robustness across evaluated settings, whereas unfiltered optimisation exhibits substantial regressions in two of the four evaluation scenarios. Summarisation transfer and scale ablations: Appendix K and J.

Conclusion

With BAPO, the full iter-4 pipeline reaches both higher final PSR (99.5%) and lower PPL (19.8), improving parsing and fluency simultaneously.

−MATTR (β=0)‡

SAGA iter 1 Binary-label DPO‡ (MATTR = 0.66, PSR = 69%)

SAGA Base

DA base

Ablation (‡ hacked)

55 0.895

0.905

0.915

0.925

0.935

Lexical diversity (MATTR, ↑)

Figure 4: Grammar-diversity Pareto frontier for Danish ablations. x: lexical diversity (MATTR; ↑ = more varied). y: oracle parse success rate (%, SpaCy; ↑ = more grammatical). S AGA iterations (filled circles) trace the frontier: each pass improves both axes. Ablations that remove reward components lie below it. Binary-label DPO (0/1 parser verdicts; MATTR 0.66, PSR 69%) collapses diversity far outside the axis range.

moving BAPO yields higher iter-2 PSR (98.3% vs. 93.8%) and parse score (0.866 vs. 0.764), but substantially worse fluency (PPL 26.8 vs. 19.9).

Parser-derived supervision provides an effective grammaticality signal for preference optimisation wherever a high-quality dependency parser is available. Across three typologically diverse Nordic languages, S AGA improves parse success by 4.5–28 percentage points while preserving lexical diversity, and native speakers consistently prefer its outputs (60–80% pairwise preference; all p≤0.01). The ablations further show that the composite reward, ∆-gap filtering, and oracle–parser gap monitoring are all necessary for robust alignment. Training requires no human preference labels: preference data are constructed automatically by generating candidate outputs and scoring them with the parser. The alignment comes at a factual-recall cost (nqii EM 24.5% → 14.3%), making S AGA most appropriate where surface grammaticality is the primary concern. More broadly, reliable symbolic evaluators may provide practical supervision for low-resource alignment tasks where human preference annotation is prohibitively expensive.

Limitations Evaluation and generalisation. DA and NB parse quality is measured with the same SpaCy parser family used during training, although heldout Stanza UD evaluation confirms the observed improvements. For IS, where Greynir provides the training reward, we additionally evaluate using an independent Stanza PSR (+3.3pp three-run mean), native-speaker pairwise judgements (80%, n=43, p<0.001), and BÍN morphological recognition (+0.8pp, directional). These complementary evaluations reduce—but do not eliminate—the possibility that parser-specific biases influence the reported gains. Grammar-specialised alignment also reduces factual recall on nqii (24.5% → 14.3%), consistent with known reward-overoptimisation effects (Pang et al., 2023; Gao et al., 2023). Human evaluation remains relatively small (n=43 IS pairs, n=44 DA pairs); the Fable-5 judge (n=25) is corroborating evidence only, not a substitute for human evaluation. IS Likert ratings use Gemini 2.5 Flash; DA and NB use Prolific nativespeaker ratings. Scope, data, and reproducibility. Training corpora are news and broadcast text. Generalisation beyond this domain is untested. NB training fragments share 3.8% substring overlap with evaluation prompts (shared Norwegian Wikipedia source). Generated continuations are uncontaminated (η<0.02; Appendix E), so PSR (measured on model outputs) is unaffected. IS reports threeseed results (Stanza PSR 79.1% ± 1.0pp; Oracle PS 88.5% ± 0.9pp). DA iter-4 was validated across 6 seeds (mean PSR 96.9%, std = 1.1pp). Wilson confidence intervals for DA and NB lie entirely above the corresponding base-model upper bounds, supporting the observed gains. A single S AGA pass takes approximately 2 to 4 hours on a single NVIDIA RTX PRO 6000 Blackwell GPU.

Ethical Considerations Our models generate text in Nordic languages and should not be deployed without human review in high-stakes settings. The underlying base models (GPT-SW3, Viking, and NorMistral) are trained on public web data and may reproduce societal biases. S AGA neither introduces new training data nor amplifies these biases. Parser-based rewards reflect formal dependency grammar standards (UD/Greynir), which may encode prestige-

variety norms. They should not be applied without considering dialectal or historical variation, as illustrated by the Greynir/IcePaHC domain gap in IS. Prolific annotators were compensated at the platform-standard rate and provided informed consent. IS Likert ratings were produced by Gemini 2.5 Flash under Google’s terms of service; no human data were collected for this evaluation.

References Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 6621–6642. PMLR. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Michael A. Covington and Joe D. McFall. 2010. Cutting the Gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema RamírezSánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECCOLING 2024), pages 1116–1128, Torino, Italia. ELRA and ICCL. Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, and Magnus Sahlgren. 2024. GPT-SW3: An autoregressive language model for the Scandinavian languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7886–7900, Torino, Italia. ELRA and ICCL. Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on

Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866. Anders Johannsen, Héctor Martínez Alonso, and Barbara Plank. 2015. Universal dependencies for Danish. In Proceedings of the 14th International Workshop on Treebanks and Linguistic Theories, pages 157–167. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, et al. 2025. Tülu 3: Pushing frontiers in open language model post-training. In Proceedings of the Second Conference on Language Modeling (COLM). Gihun Lee, Minchan Jeong, Yujin Kim, Hojung Jung, Jaehoon Oh, Sangmook Kim, and Se-Young Yun. 2024. BAPO: Base-anchored preference optimization for overcoming forgetting in large language models personalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6804–6820, Miami, Florida, USA. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. Mitigating the alignment tax of RLHF. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580–606, Miami, Florida, USA. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-Reward: Bag of tricks for reward modeling in LLMs. arXiv preprint arXiv:2410.18451. Hrafn Loftsson, Jökull H. Yngvason, Sigrún Helgadóttir, and Eiríkur Rögnvaldsson. 2010. Developing a PoStagged corpus using existing tools. In Proceedings of the 7th SaLTMiL Workshop on Creation and Use of Basic Lexical Resources for Less-Resourced Languages (LREC 2010), pages 53–60, Valletta, Malta. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. SimPO: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems, volume 37. Dan Saattrup Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Lilja Øvrelid and Petter Hohle. 2016. Universal Dependencies for Norwegian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1579–1585, Portorož, Slovenia. European Language Resources Association (ELRA). Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. 2023. Reward gaming in conditional text generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4746–4763, Toronto, Canada. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Rafael Rafailov, Archit Sharma, Eric Mitchell, Chelsea Finn, Stefano Ermon, and Christopher D. Manning. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36. David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja Øvrelid, Egil Rønningstad, Elina Sigdel, and Anna Palatkina. 2023. NorBench – a benchmark for norwegian language models. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 618–633. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Silo AI. 2024. Viking: A family of Nordic large language models. Silo AI / LumiOpen, HuggingFace. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, volume 35. Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Jónsson, Vilhjálmur Thorsteinsson, and Hafsteinn Einarsson. 2022. A warm start and a clean crawled corpus — A recipe for good language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4356– 4366, Marseille, France. European Language Resources Association.

Vilhjálmur Thorsteinsson, Hulda Óladóttir, and Hrafn Loftsson. 2019. A wide-coverage context-free grammar for Icelandic and an accompanying parsing system. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1397–1404, Varna, Bulgaria. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewardsin-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 56276–56297. Mike Zhang, Ali Basirat, and Desmond Elliott. 2026. CroCo: Cross-lingual contrastive preference tuning on self-generations. Preprint, arXiv:2605.26293.

Per-Language Configuration

Table 7 summarises the reward weights, regularisation, initialisation, and pair-gap threshold used in all reported experiments; γ=0 (fidelity term disabled) throughout. ∆-DPO hyperparameters: lr 3×10−5 , batch 32, 2 epochs, βDPO =0.1, repetition penalty 1.3. BAPO: λB =0.05 for DA/NB (fluency preservation), λB =0 for IS/SV. Reference model = π0 (base GPT-SW3-1.3B); each configuration uses seed 42 unless noted. Table 7: Per-language S AGA hyperparameters. DA uses separate configurations for pass 1 and later iterations; SV is initialised from the DA iter-3 checkpoint.

Lang.

KL λ

λB

DA (pass 1) DA (iter 2+) IS NB SV

0.80 0.65 0.80 0.80 0.65

0.20 0.35 0.20 0.20 0.35

– 0.10 – – –

0.05 0.05 0 0.05 0

0.15 0.15 0.15 0.10 0.15

Icelandic δ-Threshold Ablation

Table 8 sweeps the ∆-DPO pair-selection threshold δ for Icelandic (no KL-SFT; single pass on MIMGOLD). The δ=0.00† row is a plain DPO baseline that keeps S AGA’s on-policy pair generation but removes the gap filter, min-score filter, and BAPO anchor (λB =0, min_chosen_score = 0); it is distinct from the vanilla DPO of Table 9, which trains on static clean-vs-corrupted pairs with no on-policy generation. All δ>0 rows are full S AGA (BAPO λB =0.05, min-score threshold = 0.30).

Table 8: Icelandic δ-threshold ablation (GPTSW3-1.3B, no KL-SFT). PSR = Stanza UD parse success; Oracle = Greynir training-signal PSR; Gap = Oracle−Stanza. ‡ = gap >0.20 (reward-hacking threshold). † = plain DPO baseline: no BAPO anchor (λB =0), no min-score filter. All δ>0 rows are full S AGA with all three components. Bold marks best among full S AGA rows (δ>0, all three components active); δ=0.20 (shaded) is the sweep optimum: it achieves the highest Stanza PSR (79.8%, above base 75.8%) and Score (0.435) in this fixed-compute controlled sweep. The full production canonical run (Table 2) uses δ=0.15 and achieves Oracle PS 88.7% with Score 0.404 on IcePaHC (Stanza PSR 80.3%). The δ=0.00† row is a plain-DPO baseline lacking the gap filter and BAPO: a separate method included for deconfounding, not a competing S AGA configuration. Every trained model exceeds the hacking threshold (‡); Gap values do not discriminate plain-DPO from S AGA rows, consistent with oracle/Stanza gap reflecting Icelandic domain shift, not parser gaming (Section B). Oracle PSR Score MATTR

δ †

0.00

85.6

80.8

0.451

0.10 0.15 0.20 0.25 0.40

76.2 81.8 83.5 83.7 83.1

74.0 74.8 79.8 77.5 76.0

0.380 0.386 0.435 0.399 0.394

Gap

4-gram

0.924

0.400

‡

0.003

0.899 0.897 0.918 0.886 0.922

0.497‡ 0.494‡ 0.436‡ 0.525‡ 0.429‡

0.011 0.015 0.004 0.018 0.002

Plain DPO (δ=0.00) achieves Oracle PS 85.6% and Score 0.451 (highest in both columns within this sweep); because every δ>0 sweep row also carries the BAPO anchor, the sweep confounds the gap filter with BAPO, and the BAPO-free comparison at the end of this section isolates the filter’s effect. Among S AGA configurations (δ ≥ 0.10), Oracle PS is roughly flat at δ=0.20 to 0.25 (83.5% to 83.7%) and lower at the extremes. At δ=0.25 the Oracle PS is marginally higher (83.7%) but Score drops (0.399 vs. 0.435), indicating that stricter filtering over-prunes training pairs. At δ=0.10 the large candidate pool includes near-trivial contrasts that add noise. All IS trained models flag the oracle/Stanza gap (> 0.20), but this reflects genuine domain shift rather than reward hacking. Quantitatively, the oracle/Stanza gap stays approximately flat after S AGA training (0.465 base → 0.462 trained on the same n=200 held-out Oracle eval prompts): Stanza quality improves by +18.6 pp while Oracle PS improves by +18.3 pp, meaning both metrics move in parallel. A model engaged in reward hacking would show an increasing gap; the flat trajectory is inconsistent with that

pattern. In the controlled fixed-compute sweep, δ=0.20 achieves the highest Stanza PSR (79.8%, above base 75.8%) and Score (0.435) among full S AGA variants (δ>0, all three components active; Table 8). The plain-DPO δ=0.00† baseline posts higher Stanza PSR (80.8%) and Score (0.451), but it lacks the gap filter and BAPO anchor that define S AGA; it is a separate method included to deconfound the gap-filter contribution. For IS specifically, adding the gap filter without BAPO improves Oracle PS substantially over plain DPO (88.7% vs. 85.6%, +3.1pp) while maintaining comparable IcePaHC Stanza PSR (80.3% vs. 80.8%, −0.5pp). Adding BAPO (λB =0.05) reduces both Oracle PS (to 85.1%, −3.6pp) and Stanza PSR (to 75.3%, −5.0pp), suggesting the BAPO constraint overregularises the IS model for this morphologically complex language. Among S AGA variants, δ=0.25 has marginally higher Oracle PS (83.7%) but lower Stanza PSR (77.5%) and Score (0.399).1

Icelandic RL Method Comparison

Table 9: Icelandic RL method comparison (GPT-SW31.3B). Oracle PS = Greynir parse success rate (n=200; same parser used as reward; held-out prompts disjoint from training). Stanza PSR = independent Stanza IcePaHC parse success rate (n=400; 10th–18th c. archaic text, cross-domain; non-reward metric). Gap = Greynir oracle − Stanza quality (flag >0.20 indicates reward hacking; base Gap = 0.465 reflects IcePaHC domain shift, not gaming; S AGA −BAPO Gap = 0.462, lower than base, inconsistent with gaming). ‡ = Gap >0.20 (reward hacking flag; base also flags due to IcePaHC domain shift). Bottom two rows: controlled BAPO isolation (δ=0.15, all else equal). Best Oracle PS in bold; S AGA canonical result shaded. Method

Start Oracle PS Stanza PSR

Gap

Score MATTR

Base GPT-SW3-1.3B DPO‡ GRPO‡ sDPO‡

— Base Base Base

70.4 59.7 74.8 82.7

75.8 76.5 74.8 71.8

0.465 0.415 0.488 0.491

0.364 0.321 0.384 0.360

0.840 — — —

S AGA (∆-DPO, δ=0.15, +BAPO) Base S AGA (∆-DPO, δ=0.15, −BAPO) (ours) Base

85.1 88.7

75.3 80.3

0.466 0.409 0.462 0.404

— 0.912

Table 9 compares RL training approaches on IS GPT-SW3-1.3B evaluated with Greynir Oracle PS (the training reward; independent evaluation in Section 5.2). Greynir’s false-negative rate on 300 gold modern-Icelandic sentences is 11.7% (verbless parliamentary fragments), and the base model’s 70.4% 1 Fixed-compute sweep for controlled δ comparison (Section B). The canonical run (δ=0.15, λB =0) uses a larger compute budget, achieving Oracle PS 88.7% and Stanza PSR 80.3% (Table 2). δ=0.15 is shared with DA for crosslanguage consistency; NB uses δ=0.10.

Oracle PS sits far below the ≈88% gold-corpus parse ceiling, indicating that most base-model failures reflect genuine grammar errors rather than coverage gaps. Vanilla DPO from base (static cleanvs-corrupted preference pairs, no on-policy generation) regresses IS Oracle PS to 59.7% (−10.7pp vs. base 70.4%), indicating that off-policy preference learning without S AGA’s on-policy pair construction and gap filter is harmful; the onpolicy no-filter baseline behaves differently (Table 8, δ=0.00† ). S AGA (−BAPO, λB =0) starting from the same base achieves 88.7% Oracle PS (+18.3pp), a +29.0pp advantage over vanilla DPO. The controlled BAPO ablation (bottom two rows in Table 9) isolates λB as the sole variable: −BAPO outperforms +BAPO by +3.6pp Oracle PS (88.7% vs. 85.1%) and +5.0pp Stanza PSR (80.3% vs. 75.3%), suggesting BAPO over-constrains IS syntactic learning at this δ.

Grammar-Diversity Pareto Frontier

Figure 4 (Section 6) plots oracle parse success rate against MATTR lexical diversity for the Danish ablation suite. S AGA iterations (filled circles) trace the Pareto frontier: each training pass improves both grammar and diversity simultaneously. Ablations that remove or weaken the composite reward fall strictly below the frontier: removing the diversity term raises PSR to 98.5% but widens the oracle– parser gap to 0.22, and binary-label DPO collapses lexical diversity to MATTR 0.66 with severe output repetition (4-gram 32.7%, gap 0.29). Graded composite rewards thus prevent the grammar–diversity trade-off, rather than merely shifting which dimension degrades.

Contamination Analysis

We measure contamination using 50-character substring windows (η; suspicious >0.1, dirty >0.5). DA: η=0, 0% dirty for prompts and continuations (uncontaminated). NB: training and eval prompts both draw from Norwegian Wikipedia (28% full-sentence overlap), but models receive 2 to 6-word fragments and generate novel continuations (η<0.04 on fragments, 0% dirty on generated text), so prompt overlap does not inflate PSR. IS: MIM-GOLD (modern news/web) vs. IcePaHC (10th to 18th century historical prose); corpora disjoint by genre, register, and period; empirical η=0. Contamination therefore does not confound any reported result.

Cross-Domain Stanza UD Generalization

Stanza UD Parse Success Rate (%)

Held-out Stanza UD generalization (independent of SpaCy training oracle; both p < 0.001) 100 80

99.0%

86%

+13.0pp

89%

97.5%

+8.5pp

20 0

Danish (DA) Base

Norwegian (NB) SAGA -DPO (ours)

Figure 5: Held-out Stanza UD parse success rates for Danish and Norwegian Bokmål, evaluated on independent treebanks distinct from the SpaCy training oracle. S AGA improves Stanza PSR by +13.0pp (DA) and +8.5pp (NB), both p<0.001, suggesting that SpaCy oracle gains generalise to an independent dependency parser and are not an artifact of circular evaluation.

Compute Efficiency of ∆-DPO

∆-DPO decouples generation from gradient updates. All K=16 completions per prompt are sampled in one offline vLLM pass (0.11 h for IS 356M), scored CPU-parallel by the dependency parser, and the DPO loss applied once over surviving pairs (0.08 h); the total is 0.19 h. Online methods (GRPO, sDPO) interleave GPU generation and weight updates at every training step, requiring 4.8 h for comparable or lower final parse score.

Figure 6: Parse score vs. training time for IS (GPT-SW3356M) on a single A100-40GB GPU. ∆-DPO (0.19 h; offline vLLM generation + one DPO pass) lies on the Pareto frontier: higher parse score than online methods (GRPO: 4.81 h; sDPO: 4.76 h) at 25× lower wall-clock cost.

Figure 7: Parse reward vs. cumulative generation tokens (IS, GPT-SW3-356M). ∆-DPO (⋆) is placed at its total token budget; its eval score of 0.768 lies above both training curves with no training–eval gap, while GRPO and sDPO show reward over-optimisation (Gao et al., 2023).

Figure 6 shows the resulting Pareto frontier: ∆-DPO achieves parse score 0.768 vs. GRPO 0.623 in 25× less wall-clock time. Figure 7 complements this with a token view: GRPO and sDPO plateau near

0.67 training reward at ∼4M tokens, yet their held-out parse scores are only 0.62–0.61, indicating reward over-optimisation. ∆-DPO’s eval score of 0.768 exceeds both training curves without any training–eval gap, consistent with offline DPO’s reduced exposure to greedy reward over-optimisation.

Qualitative Examples

All examples are drawn from the pairwise evaluation set. Icelandic (IS): GPT-SW3-1.3B base vs. S AGA ∆-DPO Fragment

Base (GPT-SW3-1.3B)

S AGA (ours)

Löbbuðum því aftur niðrí bæ þar (We við vorum að fá okkur kaffi og spjallaði við náðum að fá okkur kaffi áður en við wandered back downtown where . . .) um daginn og veginn. (we were getting fórum inn á flugvöll. (we managed to coffee and chatted [sg. verb, pl. subject] get coffee before heading to the airport.) about this and that.) Skrifstofa menningarmála hefur eftir- stefnumótunaráætlunar. (policy-policy lit með framkvæmd stefnumótunar og planning. [root-word repetition]) (Cultural Affairs Office oversees policy. . . )

áætlanagerðar í samræmi við lög og reglugerðir sem settar eru á grundvelli laganna. (planning in accordance with laws and regulations established under the statute.)

Danish (DA): GPT-SW3-1.3B base vs. S AGA ∆-DPO Fragment

Base (GPT-SW3-1.3B)

S AGA (ours)

I samtlige ministerier at der skal være et et sammenhængende ar- at finde måder, hvorpå man kan undgå, at prøver man (In all min- bejdsmiljø. (there should be a a [doubled ansatte oplever det som belastende og krænkistries one tries to . . .) article] coherent work environment.) ende. (find ways to prevent employees from experiencing it as burdensome and demeaning.) En ung sovjetisk soldat får sitt livs chock när han upptäcker att det inte er en kniv i ryggen, men han stikker den af for (A young Soviet soldier . . .) sant. (the shock of his life when he discovers at komme væk fra slagmarken. (a knife in the it is not true. [code-switched to Swedish]) back, but flees to escape the battlefield.) Norwegian Bokmål (NB): GPT-SW3-1.3B base vs. S AGA ∆-DPO Fragment

Base (GPT-SW3-1.3B)

S AGA (ours)

Lisboa-traktaten, som skulle redde de (The Lisbon Treaty, which was to save the . . .)

europeiske unionen etter den mislykkede Irak europeiske sosialdemokratiene fra den nylibkrigen og vise at EU er et fredsprosjekt. (Eu- erale bølgen og styrke EU i en konkurranseutropean Union [missing definite article] af- satt verden. (European social democracies ter the failed Irak war [should be one com- from the neoliberal wave and strengthen the pound] and show EU is a peace project.) EU in a competitive world.)

Et tomt blikk var det første merke til. (notice. [minimal; no new infor- merke til da hun kom inn i stua etter å ha moren la (A blank stare mation]) funnet sønnen livløs. (when she entered the was the first thing the living room after finding her son lifeless.) mother noticed . . .)

Cross-Lingual Transfer to Swedish

Table 10: Swedish results. PSR and Score use Stanza UD (sv); PPL on Wikipedia-SV. The DA→SV warm-start initializes GPT-SW3-1.3B from the DA iter-3 checkpoint. From-base runs ∆-DPO from the raw GPT-SW3-1.3B checkpoint with SV data only. Gemma4-4B is an untuned baseline. Both trained models reach ≈97% PSR; the warm-start achieves this with substantially lower perplexity (29.7 vs. 41.7), indicating better fluency preservation. System

PSR Score PPL

GPT-SW3-1.3B base Gemma4-4B base From-base ∆-DPO DA→SV warm-start (S AGA) (ours)

87.0 89.5 97.5 97.0

0.383 0.437 0.654 0.635

— — 41.7 29.7

100.0 97.5 95.0

0.60

92.5

+10pp

90%

90.0 87.5

0.635

0.65

Parse score

Parse success rate (%)

0.70

97%

87%

0.55 0.50 0.437

0.45

85.0

0.40

82.5

0.35

80.0 GPT-SW3-1.3BGemma4-4B DA SV base base (SAGA, ours)

0.30 GPT-SW3-1.3BGemma4-4B DA SV base base (SAGA, ours)

0.383

Stanza UD (sv), n = 200. DA SV uses GPT-SW3-1.3B initialised from DA iter-3.

Figure 8: Cross-lingual transfer to Swedish. Warm-start from DA-trained ∆-DPO achieves 97.0% PSR (PPL 29.7), non-significantly different from the from-base control on PSR (97.5%, p=0.76) while preserving fluency significantly better (PPL 29.7 vs. 41.7, p<0.001, paired bootstrap, n=500).

Table 10 and Figure 8 show the full Swedish transfer results. A DA-trained ∆-DPO warm-start achieves 97.0% PSR (PPL 29.7), non-significantly different from a from-base control on PSR (97.5%, p=0.76) while preserving fluency significantly better (PPL 29.7 vs. 41.7, p<0.001, paired bootstrap, n=500). The from-base control achieves comparable parse success but at the cost of significant fluency regression (PPL 41.7 vs. 29.7 for warm-start; vs. base model PPL 29.8), consistent with the interpretation that DA-to-SV warm-start’s primary benefit is fluency preservation rather than PSR gain.

Model Scale and Architecture

Table 11: Norwegian S AGA pass 1 across model architectures. PSR = SpaCy nb_core_news_lg oracle parse success rate, n=400, measured on reward-hacking check prompts (out-of-distribution random Wikipedia sentences). This differs from Table 4, which uses n=200 HPLT in-distribution prompts; Viking-13B scores 98.0% under that in-distribution protocol. Nordic = Nordic-corpus pretraining; Gap = oracle/Stanza gap (‡ = >0.20). † = HPLT nob_Latn corpus, δ=0.20; others use default δ=0.10. Bottom block: Phi-4 trained on DA, evaluated on NB zero-shot (cross-lingual transfer). Model

Params Nordic PSR Gap

GPT-SW3 GPT-SW3† NorMistral Viking (Silo AI, 2024) Phi-4

1.3B 6.7B 7B 13B 14B

✓ ✓ ✓ ✓ ×

74.2 75.5 70.6 69.6 68.6

0.156 0.178 0.101 0.128 0.100

Cross-lingual zero-shot (DA-trained → NB eval) Phi-4 (DA→NB) 14B × 62.1 0.092

Table 12: Danish S AGA pass 1 across model architectures. Nordic = Nordic-corpus pretraining. Gap = oracle/Stanza gap. Model

Params Nordic PSR Gap

GPT-SW3 GPT-SW3 Viking (Silo AI, 2024) Phi-4

1.3B 6.7B 13B 14B

✓ ✓ ✓ ×

87.3 84.5 85.8 76.8

0.147 0.197 0.148 0.145

Table 13: Icelandic S AGA pass 1 across model scales. Oracle PS = Greynir parse success (training oracle; n=200). Nordic = Nordic-corpus pretraining. Base rates: 1.3B 70.4%, Viking-13B 64.5%, Phi-4 55.0%. ‡ = oracle/Stanza gap >0.20. Model

Params Nordic Oracle PS

GPT-SW3 GPT-SW3 Viking (Silo AI, 2024) Phi-4

1.3B 6.7B 13B 14B

✓ ✓ ✓ ×

88.7 83.0 83.1 86.6‡

Across DA (Table 12) and IS (Table 13), Nordic pretraining outweighs raw model scale: Phi-4 (14B, multilingual) achieves only 76.8% DA PSR while GPT-SW3-1.3B (Nordic) reaches 87.3%. PSR is non-monotonic within the Nordic family: GPT-SW3-1.3B (87.3%) outperforms GPT-SW3-6.7B (84.5%) and Viking-13B (85.8%) for DA; for IS, GPT-SW3-1.3B achieves the highest Oracle PS (88.7%), outperforming GPT-SW3-6.7B (83.0%) and Viking-13B (83.1%), despite Viking-13B’s larger base capacity. These results suggest that targeted Nordic pretraining provides stronger grammatical priors than scale alone for morphologically rich Nordic languages.

Summarization Domain Transfer IS (GPT-SW3-356M)

DA (GPT-SW3-356M)

Stanza

MATTR

Base

0.340

0.072

0.776

SFT

0.195

0.024

0.508

GRPO

0.335

0.061

0.652

DPO

0.015

0.000

0.113

sDPO

0.255

0.037

0.580

∆-DPO (ours)

0.490

0.138

0.925

Stanza

MATTR

Base

0.505

0.158

0.884

GRPO

0.720

0.316

0.939

DPO

0.540

0.179

0.860

sDPO

0.675

0.285

0.915

∆-DPO (ours)

0.990

0.739

0.955

Wikipedia summarisation transfer (N =200; column-normalised within panel).

Figure 9: Summarization domain-transfer heatmap, IS (left) and DA (right) (N =200 per language; colour columnnormalised within each panel). None of these models were trained on summarisation. ∆-DPO obtains the highest parse success (PS), Stanza quality, and lexical diversity (MATTR) in both languages. DA PS rises from 50.5% to 99.0%; IS from 34.0% to 49.0%. Online methods (GRPO, sDPO) transfer substantially less, and plain DPO collapses IS (PS 0.015).

Figure 9 (above) shows the full domain-transfer heatmap. We evaluate domain transfer by prompting each trained model on out-of-domain Wikipedia article summarization, a task absent from the training distribution. For IS (GPT-SW3-356M), ∆-DPO achieves PS = 0.490 versus base 0.340, Stanza 0.138 versus 0.072, and MATTR 0.925 versus 0.776; all baselines (SFT, GRPO, sDPO) fall short, and plain DPO collapses to PS 0.015. For DA (GPT-SW3-356M), ∆-DPO achieves PS = 0.990 versus base 0.505 (+96% relative), with Stanza quality rising from 0.158 to 0.739 and MATTR from 0.884 to 0.955. Online methods (GRPO: 0.720, sDPO: 0.675) achieve high DA PS but lower quality and diversity than ∆-DPO. ∆-DPO’s offline preference signal induces structural grammatical properties that generalise to unseen domains, whereas online methods (GRPO, sDPO) do not achieve the same quality ceiling.

Danish Reward-Hacking Diagnostic

Figure 10 plots PSR versus oracle/Stanza gap for all DA ablation configurations. The shaded zone (gap > 0.20) marks the reward-hacking threshold; configurations in this zone are flagged ‡ in Table 6. Full S AGA iterations stay in the unshaded region, while every ablation that removes a safeguard (parse-only reward, binary labels, no MATTR) crosses into the hacking zone.

Parse success rate (%, ↑)

100

% ideal −KL-SFT

hacking zone

SAGA iter 4

SAGA iter 2

−MATTR (β=0)‡

−∆ filter

SAGA iter 1

Parse-only‡

DA base

SAGA iteration SAGA ablation variant Base model Ablation (‡hacked)

Binary labels‡

60 0.10

0.15

0.20

0.25

0.30

0.35

Oracle/Stanza gap (↓ = no hacking) Figure 10: PSR vs. Oracle/Stanza gap for DA ablation configurations. x-axis: oracle/Stanza gap (lower = no hacking); y-axis: SpaCy PSR (%). The shaded zone (gap > 0.20) marks the reward-hacking threshold. Ablations without safeguards (parse-only, binary labels, −MATTR) all fall in the hacking zone; SAGA iterations move from base toward the upper-left (high PSR, controlled gap) across passes. ‡ = reward hacking detected.

Fable-5 Automated Judge Protocol

Setup. The Fable-5 pairwise judge (claude-fable-5) was used as a cross-evaluator check on the IS human pairwise result (Section 5.5). 25 IS pairs (base vs. canonical S AGA −BAPO) were drawn from MIM-GOLD prompts, independently of the 43-pair native-speaker evaluation set. A/B positions were randomised per pair (seed 42) to control for position bias. Temperature was set to 0.1 to reduce judge stochasticity. A single judge evaluation was run per pair. The verbatim prompt is reproduced below: Judge prompt (verbatim). You are an expert in Icelandic grammar and linguistics. Two language models were given the same Icelandic sentence fragment and asked to complete it. Your task: decide which continuation is more grammatically correct and natural in Icelandic. Fragment (given to both models): {fragment} Continuation A: {cont_a} Continuation B: {cont_b} Judge ONLY the continuations — the fragment is fixed input. Evaluate on: • Grammaticality: correct case (nominative/accusative/dative/genitive), noun-adjective agreement, verb conjugation, and word order • Fluency: naturalness as a native Icelandic speaker would write • Coherence: continuation logically follows from the fragment Then pick a winner. If both are equally good or equally bad, choose “tie”. Respond ONLY with this JSON (no markdown):

{ "winner": "A" or "B" or "tie", "grammaticality_a": <1-5>, "grammaticality_b": <1-5>, "fluency_a": <1-5>, "fluency_b": <1-5>, "reason": "<one sentence in English explaining the key grammatical difference>" }

Results. 19W/2T/4L for S AGA; 83% excl. ties (p<0.01, exact binomial). Likert means: base Gr. 4.28 / Fl. 2.60; S AGA Gr. 4.32 / Fl. 3.44 (∆Gr +0.04, ∆Fl +0.84). These ratings are produced by the same Fable-5 judge and labelled § in Table 5.

SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

Analyze & explore related research in Conceptio

Related documents