L EX F LIP: A Dissociation Diagnostic for Legal Meaning Preservation Metrics Gaurab Baral University of Cincinnati [email protected]
arXiv:2609.05296v1 [cs.CL] 4 Sep 2026
Abstract Does a simplified legal clause still say what the original said? The checks in current use cannot establish that it does: requiring an identical pair to score highest and an unrelated pair lowest moves lexical overlap and legal force together, so any monotone function of token overlap satisfies both. Our remedy is a dissociation, an item holding surface form fixed while legal force moves. We release L EX F LIP, 373 minimal perturbations of Quebec statutory French that reverse legal force while preserving 0.93 of the tokens, with a harness scoring metrics, regressors and prompted judges alike. The seven embedding and BERTScore metrics we test spend only 0.022–0.039 of their identical-to-unrelated range on such an edit, against 0.670 for bidirectional NLI, the one family the identical-pair check would disqualify. On F R J UDGE, against a measured human ceiling of r = 0.597, a bare length feature outscores every semantic metric and has the lowest margin we measure.
1
Introduction
Rewriting a legal text for a lay reader has an unusual failure mode. When a news summary drops a qualifier the reader is mildly misinformed; when an insurance clause drops a qualifier the reader may be uninsured. Moffatt v. Air Canada (2024) held an airline liable for its chatbot’s misstatement of its own policy, and legal research tools sold as hallucination-free still err on 17 to 33% of queries [1]. If simplified legal text is produced automatically, an automatic metric is what checks it. Two checks are in current use [2, 3]: a sentence paired with itself must receive the maximum score, and a sentence paired with an unrelated one the minimum. Both are necessary and both are easy to state. But moving from an identical pair to an unrelated
QUEBEC HIGHWAY SAFETY CODE
L’agent de la paix doit remettre le certificat au conducteur dès qu’il l’a examiné. The peace officer must return the certificate to the driver as soon as they have examined it. one word replaced; token Jaccard 0.88 LEGAL FORCE REVERSED
L’agent de la paix peut remettre le certificat au conducteur dès qu’il l’a examiné. The officer may return it: the duty has become a discretion and the clause states the opposite law.
BERTScore 0.99 ✗ LLM-judge 0.49 ✗ NLI 0.07 ✓ mean over all 373 edits: 0.99, 0.63, 0.26
Figure 1: One L EX F LIP item. Replacing doit with peut turns a duty into a discretion and leaves the sentence otherwise untouched, so a metric reading legal meaning must drop and one reading overlap cannot. The judge’s ten-point rating is rescaled to [0, 1], as in Table 1. Over the whole set the edit costs BERTScore 0.022 of its own identical-to-unrelated range and the judge 0.370, against 0.670 for bidirectional NLI.
pair changes lexical overlap and legal meaning together, in the same direction, at the same time, so any monotone function of token overlap satisfies both by construction. The checks test calibration at two endpoints; they do not test what the metric is reading. The remedy is a dissociation: an item on which the two confounded variables are pulled apart, so that legal force changes while surface form does not. Replace doit (must) with peut (may) and ninety-odd percent of the tokens are untouched, the sentence stays fluent and statutory in register, and an obligation has become a permission (Figure 1). A metric that measures legal meaning must drop sharply; a metric that measures overlap cannot. We release L EX F LIP, 373 such perturbations of Quebec statutory French, with a harness taking
an arbitrary scoring function so that untrained metrics, regressors and prompted judges take the same test,1 and a four-part validation protocol around it.
from raw score to human scale on the training split alone; isotonic regression cannot improve a ranking, and rank statistics move by at most 0.014.
Related work. BLEU does not correlate with meaning preservation once sentence splitting is involved [4], and BERTScore [5] and other embedding metrics score high across all alterations. F R J UDGE [3] is to our knowledge the only annotated resource for legal meaning preservation in any language, and trained metrics fitted to such corpora [2] are our point of departure. Closest in method, Chen and Eger [6] show similarity metrics are not robust to meaning-changing edits while NLI-based metrics repair much of the gap, and Mujahid et al. [7] run the complementary experiment; no prior challenge set we know of targets legal force. We follow Bean et al. [8] and Pacchiardi et al. [9] on construct validity, Deutsch et al. [10] and Graham and Baldwin [11] on statistics, Xu and Jurgens [12] on rating distributions.
R3: a trivial-feature control. Simplifications are usually shorter and the dominant annotated error category is omission, so a rubric deducting score once per identified error yields a label partly predictable from how many words disappeared. We require a supervised baseline over surface features alone, fitted under R2.
2
What Would Validate a Legal Meaning Metric?
The four requirements below are stated for an arbitrary scorer m(o, s) over an original clause o and a candidate simplification s; none presupposes an architecture. R1: a measured ceiling. A correlation with human judgment means nothing until one knows what correlation a human achieves. Where a corpus carries k ≥ 3 ratings per item this is free: hold out one rater, correlate their ratings against the mean of the other k − 1, average over the held-out position. That answers is the metric as good as an expert?; the Spearman–Brown reliability of the krater mean answers is it as good as the aggregate label allows? A metric can sit above the first and below the second without being superhuman, since predicting an average is easier than being a rater. R2: identical supervision. Comparisons here place an unsupervised cosine similarity and a finetuned regressor in one table, decimal-scale the former, and report RMSE for both. Decimal scaling fixes the range but neither the location nor the shape of the distribution, so a similarity occupying [0.7, 1.0] records a large RMSE however well it ranks, and the column measures supervision rather than judgment. We instead fit a monotone map 1
https://github.com/ Nyvora-Vision-Labs/LEXFLIP
R4: dissociation. On naturally occurring pairs surface form and legal force vary together, the confound the endpoint checks inherit. Let s′ be a perturbation of s with token Jaccard J(s, s′ ) ≈ 1 and legal force L(s′ ) ̸= L(s), and u an unrelated sentence of the same register. We report the margin fraction (m(s, s) − m(s, s′ ))/(m(s, s) − m(s, u)), the share of the metric’s own identical-to-unrelated range spent on a legally decisive edit, which charges it against the range it actually has rather than a nominal [0, 1]. A metric treating the flipped sentence as it treats an unrelated one scores 1; a pure overlap function scores near 0. This is not the rate at which (s, s) outranks (s, s′ ), the weaker quantity: a metric can rank every item correctly while moving a thousandth of its range.
3
L EX F LIP
Source sentences come from two Quebec statutes held out from every corpus used here, the Automobile Insurance Act and the Highway Safety Code, and each takes one minimal edit that changes legal force while leaving the sentence intact. The inventory is organised by legal effect, not by linguistic category. Six families make up tier 1, where the edit unambiguously changes who is bound, what is covered or how much: modality (doit ↔ peut), party (l’assuré ↔ l’assureur), polarity (dropping ne . . . pas), quantum (×10), scope (tous les → certains) and temporal (avant ↔ après); tier 2 connective and class shifts are reported separately. The diagnostic would fail if a metric could find the edited item by noticing that it reads badly, so perturbations introducing a grammatical artefact are filtered by a conservative French well-formedness check that flags none of the 400 unmodified sources. The result is 373 pairs (310 tier 1) with mean token Jaccard 0.93, none below 0.75. For each s with perturbation s′ we score (s, s), (s, s′ ) and (s, u) for an unrelated u from the same
Metric NLI-bidirectional (min) NLI-bwd (no omission) NLI-fwd (no hallucination) LLM-judge (deepseek-chat, k=5) surface: token Jaccard LaBSE BERTScore-FlauBERT mE5-base Para-mUSE-mpnet Sentence-CamemBERT BERTScore-CamemBERTv2-Recall BERTScore-CamemBERTv2 surface: −| |s| − |o| |
identical L EX F LIP unrelated discrim.% margin 0.784 0.784 0.784 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
0.265 0.432 0.469 0.632 0.928 0.972 0.979 0.975 0.981 0.983 0.983 0.982 0.997
0.010 0.042 0.043 0.006 0.089 0.292 0.393 0.266 0.312 0.316 0.240 0.193 0.718
93.3 77.2 77.7 86.3 99.2 100.0 100.0 100.0 100.0 100.0 100.0 100.0 11.3
0.670 0.473 0.424 0.370 0.079 0.039 0.035 0.034 0.028 0.024 0.023 0.022 0.011
Table 1: L EX F LIP, min–max normalised per metric. discrim. is the share of items ranked correctly (identical > edited), margin the fraction of the identical-to-unrelated range spent on the legal edit.
statutes, min–max normalise per metric so the two probes anchor the range, and take the margin from the three means. Only French-capable metrics are administered, since scoring Englishonly ones on French confounds coverage with quality. The set covers three families, named in Table 1: BERTScore [5] over CamemBERTv2 and FlauBERT, in F1 and recall-only variants since recall targets omission; four sentence-embedding cosines; and bidirectional NLI with mDeBERTaXNLI [13], where failure of o ⇒ s signals hallucination and of s ⇒ o omission, the two dominant categories of the legal error taxonomy. We add token Jaccard and the surface features of R3.
4
Results
4.1
The Diagnostic
Every similarity-based metric fails, and fails almost completely (Table 1). The seven embedding and BERTScore variants spend between 0.022 and 0.039 of their working range on an edit that reverses the law: BERTScore over CamemBERTv2 places a sentence saying the opposite of the original at 0.982, where the original sits at 1.000 and an unrelated statute at 0.193. Token Jaccard, at 0.079, is in the same regime, which is the point: these are overlap functions, exactly as Section 2 says they must be. Their discrimination rates are uninformative for the same reason, the seven ranking the identical pair first 100% of the time by a margin that carries no decision. Bidirectional NLI does not fail: it spends 0.670 of its range on the edit, ranks correctly on 93.3% of items, and places a flipped sentence nearer the unrelated pair than the identical one. The existing checks would have excluded it. Note where NLI’s identical score sits: 0.784, and
Perturbation
n mean margin
drop ne . . . pas 21 avant ↔ après 9 amount × 10 21 doit → peut 82 peut → doit 111 tous les → certains 42 et ↔ ou 53 assureur ↔ assuré 14
0.333 0.272 0.256 0.241 0.167 0.153 0.069 0.035
All 12 rules
0.155
373
Table 2: Margin over the 13 scorers of Table 1; selected rules, same-direction pairs pooled. All 12 rules are in the supplementary material.
not as a normalisation artefact. In raw terms mDeBERTa assigns a mean entailment of 0.780 to a sentence paired with itself, clearing the 99% bar the identical-pair convention requires on only 1.3% of items, against 100% for every BERTScore and embedding metric. The metric that best tracks legal meaning is the one convention would disqualify. Saturation on identical input is easy to build, every overlap function having it by construction: report it, but do not filter on it. Nor is any rule reliably detected (Table 2), the best averaging 0.333 and the party swap 0.035. 4.2
The F R J UDGE Corpus
The diagnostic answers R4. We run the other three on F R J UDGE [3], 297 insurance clauses simplified by gpt-4-turbo-2024-04-09 and rated by five law students on legal meaning (1– 10), then ask whether the two kinds of evidence agree. The public release drops annotator identity, making R1 impossible downstream, so we rebuild the corpus from the raw Prodigy export, recovering the five annotators (pseudonymised A–E) and reproducing the published pairwise agreement of 25.96%.
r
Metric Unsupervised, French-capable BERTScore-CamemBERTv2 BERTScore-FlauBERT LaBSE mE5-base NLI-bidirectional (min)
0.291±0.09 0.482±0.08 0.409±0.06 0.358±0.08 0.395±0.07
ρ RMSEraw RMSEcal over% 0.326 0.459 0.402 0.356 0.405
2.62 2.16 2.33 2.52 3.74
2.36 2.12 2.21 2.26 2.22
49.3 49.4 47.2 48.0 46.6
Trivial surface features −| |s| − |o| | (length difference) 0.641±0.04 0.624 Token Jaccard 0.303±0.06 0.331 Surface-Ridge (10 features, supervised) 0.621±0.05 0.588
3.25 3.34 1.89
1.85 2.34 1.89
47.0 48.0 45.8
Prompted LLM judge LLM-judge (deepseek-chat, k=5)
0.755±0.05 0.753
2.93
1.57
43.9
Human ceiling (one annotator)
0.597 0.620
–
3.27
–
Table 3: Calibrated comparison on the F R J UDGE test split (89 items), mean over 10 seeds; all ten unsupervised metrics are in the supplementary material. RMSEraw is the decimal-scaling protocol in current use, RMSEcal the isotonic map of R2.
Identical supervision, and a trivial control. Calibration collapses most of the RMSE gap uncalibrated comparisons record (Table 3): the metrics were not as far from human judgment as such tables suggest, merely on a different scale. And a ridge over word counts and token overlap reaches r = 0.62, at the human ceiling and above every semantic metric we test, the best reaching r = 0.482. Only two scorers beat it, and neither is semantic: the prompted judge below, and the bare length difference at r = 0.641. This says less about the corpus than about what the annotation rewards, score being reduced once per identified error and the most frequent error type, omission, correlating strongly with becoming shorter. A prompted judge. A prompted model given the annotators’ rubric [15] is the natural competitor to a fine-tuned 112M-parameter encoder. We score
embedding / BERTScore surface feature
entailment prompted LLM judge
NLI-bidirectional
0.6 NLI-bwd
LexFlip margin
Disagreement and the ceiling. Krippendorff’s α for legal meaning is 0.104 under the nominal coefficient, the one conventionally reported here, and 0.325 under the ordinal one appropriate to an ordered scale [14]. What remains is structural: the annotators separate into two stable populations, B, C and D averaging 7.61 against 4.34 for A and E, and 63% of pairs span at least 6 points. Averaging leaves a target with standard deviation 2.40, the condition under which a regressor predicting the middle scores well. Correlating each annotator against the mean of the other four gives a ceiling of 0.597 (RMSE 3.27); exceeding it is not superhuman, the bound for predicting that mean being its own reliability, 0.811 √ by Spearman–Brown, so a metric can approach 0.811 = 0.90.
NLI-fwd
0.4
LLM-judge
0.2 token Jaccard −| |s| − |o| |
0.0 7 embedding / BERTScore metrics margin 0.02–0.04
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Pearson r with the human label (calibrated)
Figure 2: The two orderings come apart. Horizontally, calibrated correlation with the human label, the quantity a conventional comparison selects on; vertically, the share of working range spent on an edit that reverses the law.
each pair with deepseek-chat under the F R J UDGE scale, a ten-point judgment deducting one point per identified legal error, in its own request so the judge never sees the probe condition, K = 5 times. Under R2 it reaches r = 0.755±0.054 , above the length feature and our ceiling (a redraw gave 0.726). On L EX F LIP it scores the identical pair at 10.00 and the unrelated pair at 1.06, flawless on the two checks jointly satisfiable by overlap, then awards a clause whose legal force has been reversed 6.69 out of ten, a margin of 0.370. The rubric, not the model, limits this: with the model fixed the margin moves from 0.421 under the rubric-faithful prompt to 0.687 under a bare instruction stripping the four-type taxonomy, against 0.002 for a redraw. Score is reduced once per identified error, so a clause reversal is one incoherence and the penalty is capped near the top of the scale. One behaviour survives every configuration: peut for doit scores 4.72 while the reverse scores 8.02, though both change the law by the same construction, so the
judge tracks legal severity rather than legal equivalence.
5
Discussion and Conclusion
Over the 13 scorers carrying both a calibrated r and a margin the rank correlation between them is ρ = 0.24 (95% interval [−0.39, 0.75]), an absence of resolution rather than a demonstrated null; the extremes are the durable part. The length feature outscores every semantic metric and has the lowest margin of anything we test (0.011), while bidirectional NLI sits mid-table on correlation and alone reacts to the edit. The empty upper-right quadrant of Figure 2 is the finding. A metric claiming to measure legal meaning should therefore report three checks, not two: identical pairs (saturation), unrelated pairs (bottoming out), and L EX F LIP (does it move when, and only when, the law moves?). The first two are jointly satisfiable by lexical overlap; the third is not, so a low margin is a fixable deficiency rather than a limit of automatic evaluation. Only legal sensitivity resists calibration: rank on it, and build the next metric on a repaired entailment objective.
Limitations Our L EX F LIP labels are definitional rather than annotated: we assert that replacing doit with peut changes legal meaning rather than having lawyers rate each perturbation. That is defensible for tier-1 edits, reported separately from tier 2, but a small expert study would be stronger. The edits are single and template-generated, so a system could be tuned to them without acquiring general legal sensitivity, which is why the diagnostic is a necessary condition and not a benchmark to optimise. The same caveat bears on our best result. Our perturbations are largely negation, modality, quantifier and antonym contrasts, close to what NLI training data is built around, so bidirectional NLI’s margin of 0.670 may reflect familiarity with their form rather than sensitivity to their legal consequence. Yang et al. [16] give the general version: a counterfactual edit isolates the variable of interest only against a meaning-preserving paraphrase control, which we do not run and the next version of the diagnostic should carry. Finally, the diagnostic and the corpus comparison run on different text, insurance forms against two statutes, and answer different questions; the judge results rest on one model under a rubric
reconstructed from the published description, a load-bearing assumption given that wording alone moves the margin by 0.267; and the ceiling is computed on all 297 items while model correlations run on 89-item splits, so “model > ceiling” carries the sampling uncertainty of both.
Ethical Considerations F R J UDGE’s five annotators are pseudonymised as A–E throughout, our released code pseudonymises annotator identifiers on load, and we ship the pseudonymisation script for anyone redistributing a reconstruction. Beauchemin et al. [3] caution that their metric informs a practitioner about legal meaning preservation rather than supplying a complete juridical analysis; our results sharpen that into a rule. A metric that cannot distinguish doit from peut must not gate whether a simplified clause is safe to publish, and the appropriate deployment is triage, the ranking of clauses for human review, rather than certification. Nothing here constitutes legal advice. The perturbed sentences are deliberate misstatements of Quebec law, labelled as such in the distribution, and are not statements of the law.
References [1]. Magesh, Varun, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. “Hallucination-free? Assessing the reliability of leading AI legal research tools.” Journal of Empirical Legal Studies 22, no. 2 (2025): 216–242. [2]. Beauchemin, David, Horacio Saggion, and Richard Khoury. “MeaningBERT: Assessing meaning preservation between sentences.” Frontiers in Artificial Intelligence 6 (2023): 1223924. [3]. Beauchemin, David, Michelle Albert-Rochette, Richard Khoury, and Pierre-Luc Déziel. “JUDGEBERT: Assessing legal meaning preservation between sentences.” In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 92–118. 2025. [4]. Sulem, Elior, Omri Abend, and Ari Rappoport. “BLEU is not suitable for the evaluation of text simplification.” In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 738–744. 2018. [5]. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. “BERTScore: Evaluating text generation with BERT.” arXiv preprint arXiv:1904.09675 (2019). [6]. Chen, Yanran, and Steffen Eger. “MENLI: Robust evaluation metrics from natural language inference.” Transactions of the Association for Computational Linguistics 11 (2023): 804–825. [7]. Mujahid, Zain Muhammad, Dustin Wright, and Isabelle Augenstein. “Stress testing factual consistency metrics for long-document summarization.” In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31914–31933. 2026. [8]. Bean, Andrew M., Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi et al. “Measuring what matters: Construct validity in large language model benchmarks.” Advances in Neural Information Processing Systems 38 (2026). [9]. Pacchiardi, Lorenzo, Marko Tesic, Lucy G. Cheke, and José Hernández-Orallo. “Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers.” arXiv preprint arXiv:2410.11672 (2024). [10]. Deutsch, Daniel, Rotem Dror, and Dan Roth. “A statistical analysis of summarization evaluation metrics using resampling methods.” Transactions of the Association for Computational Linguistics 9 (2021): 1132–1146. [11]. Graham, Yvette, and Timothy Baldwin. “Testing for significance of increased correlation with
human judgment.” In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 172–176. 2014. [12]. Xu, Yinuo, and David Jurgens. “Beyond consensus: Perspectivist modeling and evaluation of annotator disagreement in NLP.” arXiv preprint arXiv:2601.09065 (2026). [13]. Laurer, Moritz, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. “Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and BERT-NLI.” Political Analysis 32, no. 1 (2024): 84–100. [14]. Krippendorff, Klaus. Content Analysis: An Introduction to Its Methodology. TPB, 1996. [15]. Enguehard, Joseph, Morgane Van Ermengem, Kate Atkinson, Sujeong Cha, Arijit Ghosh Chowdhury, Prashanth Kallur Ramaswamy, Jeremy Roghair et al. “LeMAJ (Legal LLM-as-aJudge): Bridging legal reasoning and LLM evaluation.” In Proceedings of the Natural Legal Language Processing Workshop 2025, pp. 318–337. 2025. [16]. Yang, Zihao, Mosh Levy, Yoav Goldberg, and Byron C. Wallace. “Compared to what? Baselines and metrics for counterfactual prompting.” arXiv preprint arXiv:2605.01048 (2026).