arXiv CSopen access

Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification

Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification

1

Jingxi Qiu1,2 , Zeyu Han2 , Cheng Huang1,† ZenWeave AI, 2 Georgetown University, † Corresponding Author [email protected], [email protected]

arXiv:2605.26663v1 [cs.CL] 26 May 2026

Abstract

predict NEI for reasons that have little to do with whether the evidence is actually sufficient. Prior work on fact-verification artifacts has focused on the claim side. Schuster et al. (2019) show that FEVER can be partially solved by claimonly classifiers, and adversarial or contrastive verification resources further show that standard evidence-aware accuracy can hide brittle decision rules (Thorne and Vlachos, 2019; Schuster et al., 2021). A separate line studies evidence sufficiency by removing parts of otherwise-valid evidence (Atanasova et al., 2022; Vladika et al., 2025). We study a complementary failure mode—how the negative evidence condition is built in the first place—and argue that this construction silently determines what a verifier learns and what an aggregate NEI-F1 can hide. Figure 1 illustrates the mechanism. Easy NEI can be solved by recognizing absence, format, or topic mismatch; hard NEI keeps the evidence related to the claim but incomplete, which can induce false support when the model mistakes overlap for sufficiency. NEI-CAP targets this gap by making construction family part of the evaluation record rather than treating NEI as a constructionfree class. NEI-CAP makes the construction explicit. Each NEI example carries the family of evidence condition that produced it, which lets us audit each family for shortcut features and stress-test whether a model trained on one family recognizes insufficiency in another. We instantiate the protocol on SciFact-style scientific verification (Wadden et al., 2020), with FEVER (Thorne et al., 2018), HoVer (Jiang et al., 2020), and the broader factverification literature as context (Augenstein et al., 2019; Aly et al., 2021). The headline finding is a transfer failure that aggregate NEI-F1 cannot detect. A DeBERTa verifier trained on placeholder NEI reaches perfect matched-placeholder NEI-F1 across five seeds, yet

Evidence absence is not evidence insufficiency, but fact verification benchmarks can make them observationally similar. The N OT E NOUGH I N FORMATION (NEI) label is often operationalized through different evidence conditions, and that choice silently determines what a verifier learns, and what its score can hide. We introduce NEI-CAP, a construction-aware diagnostic protocol for insufficient-evidence evaluation. Each NEI example carries the construction family that produced it; NEI-CAP audits shortcut cues, validates hard cases through human adjudication, and tests whether competence transfers across constructions. We instantiate the protocol in SciFact-style scientific verification, with FEVER and HoVer as bounded external controls. Across these settings, NEI competence does not transfer reliably: models trained on shortcut-prone constructions fail to recognize semantically related insufficient evidence, and mixed-construction training narrows but does not close the gap. Fixed-claim diagnostics further show that the evidence condition shifts confidence in the reference S UP PORT/R EFUTE label, not only NEI recall, so an aggregate NEI score can hide which problem a model has actually solved.

1

Introduction

A fact verification system labels a claim as supported, refuted, or Not Enough Information (NEI) when the available evidence is inconclusive (Thorne et al., 2018; Wadden et al., 2020; Jiang et al., 2020). The NEI label is meant to be evidenceconditioned: for a claim c and an evidence set E, it means that E does not establish c either way. Building those negative evidence sets, however, is itself a design step that no formal definition covers. An empty field, an off-topic passage, a high-overlap retrieval miss, and a non-rationale sentence drawn from a cited document can all be labelled NEI, and a verifier trained on one of these constructions can 1

Easy NEI

Hard NEI

Claim The Great Wall of China is located in Brazil.

Claim The Great Wall of China is visible from the Moon.

Evidence Example 1

Related Evidence (same topic) The Great Wall of China is located in northern China. It is over 21,000 km long.

<No evidence provided> Evidence Example 2 Quantum entanglement in particle physics.

NEI-CAP Construct Hard NEI

Audit Artifacts

Validate Hard NEI

Stress-test Models

placeholder random BM25 cited same-doc missing-hop

length overlap source position placeholder-rate

human adjudication

train-test matrix

contamination audit

multi-model

Missing Relation

fixed-claim

relevant? complete?

Learned Shortcut Rules

NEI low overlap

Model Output

NEI

decisive?

No evidence about visibility from the Moon.

Sufficient?

High term overlap → looks supportive

NEI

Model Output

Model Output

NEI

SUPPORT

Figure 1: Conceptual illustration of NEI construction artifacts. Easy NEI constructions, such as placeholders or unrelated passages, can teach absence, format, or topic-mismatch shortcuts. Hard NEI keeps the evidence semantically related but incomplete; a verifier may therefore overpredict S UPPORT from overlap rather than recognize insufficiency. NEI-CAP records the construction family, audits shortcuts, validates hard examples, and stress-tests whether NEI competence transfers. Examples are schematic; experiments use the SciFact, FEVER, and HoVer constructions described in Sections 3–4.

scores zero NEI-F1 on BM25 near-miss and cited non-rationale evaluation; the collapse replicates on RoBERTa and SciBERT. Probability mass shifts to S UPPORT and R EFUTE rather than to NEI, so the failure is not a calibration artifact. Training on random-irrelevant NEI fares only marginally better, showing that the problem extends beyond placeholder detection. Mixed-construction training narrows but does not close the gap. A fixedclaim diagnostic further shows that swapping the evidence shifts confidence in the reference S UP PORT or R EFUTE label, not only NEI recall. The construction choice therefore affects the verifier on the full three-way task, not only the NEI corner.

2

Related Work

2.1

Fact Verification Benchmarks

Fact verification is commonly formulated as a threeway classification problem: given a claim and an evidence set, predict whether the evidence supports, refutes, or is insufficient to verify the claim. FEVER introduced a large-scale Wikipedia benchmark with S UPPORTED, R EFUTED, and N OT ENOUGH I NFO labels (Thorne et al., 2018); SciFact extended the formulation to expert-written scientific claims that require retrieving evidencecontaining abstracts and rationales (Wadden et al., 2020); and HoVer added many-hop evidence retrieval, where verification can depend on facts spread across multiple Wikipedia articles (Jiang et al., 2020). MultiFC broadens fact-checking to real-world multi-domain claims (Augenstein et al., 2019); HealthFC focuses on evidence-backed medical claims (Vladika et al., 2023); FEVEROUS adds structured table evidence (Aly et al., 2021); and VitaminC creates contrastive claim– evidence pairs that require sensitivity to small factual changes (Schuster et al., 2021). Rationalecentered resources further ask whether systems identify the evidence used to support predictions (DeYoung et al., 2020). Across these benchmarks the NEI label is treated as a fixed third class, but how its evidence side is built is left to each

We make three contributions. First, we recast NEI as a construction-sensitive evidence condition rather than a single negative label. Second, we introduce NEI-CAP: a diagnostic protocol that treats the construction family as an explicit evaluation variable, audits its shortcut surface, and validates hard cases through human adjudication. Third, our SciFact, FEVER, and HoVer experiments show that shortcut-prone training fails to transfer to semantically related insufficient evidence, and that multi-seed and mixed-construction protocols do not remove the need for construction-stratified reporting. 2

benchmark’s discretion—and that choice is not made part of the evaluation protocol. 2.2

against supporting sources (Min et al., 2023), and in retrieval-augmented generation or grounding evaluation, where generated claims must be supported by provided context (Niu et al., 2024; Jacovi et al., 2025). These studies motivate evidencesensitive evaluation, but they usually treat the negative or unsupported condition as already given or derived from a valid one. NEI-CAP works in the opposite direction: it asks how the insufficient evidence set was built in the first place, and how that construction determines what a verifier can be said to have learned.

Dataset Artifacts and Behavioral Evaluation

NLP benchmarks routinely contain artifacts that let models score well without learning the intended capability. In natural language inference, hypothesis-only classifiers can recover the label from premise-free input (Gururangan et al., 2018; Poliak et al., 2018), and controlled challenge sets such as HANS show that high benchmark accuracy can mask reliance on lexical or syntactic heuristics (McCoy et al., 2019). Similar shortcut effects appear beyond NLI, including argument reasoning artifacts (Niven and Kao, 2019); more broadly, shortcut learning is a known failure mode of modern neural systems (Geirhos et al., 2020). In fact verification, Schuster et al. (2019) show analogous claim-side cues in FEVER and demonstrate that claim-only baselines remain competitive against evidence-aware models; FEVER2.0-style adversarial work further stresses robustness to perturbations (Thorne and Vlachos, 2019). A broader line of work uses contrast sets, counterfactually augmented data, and behavioral testing to expose brittle shortcut reliance (Kaushik et al., 2020; Gardner et al., 2020; Ribeiro et al., 2020). These studies mostly examine claim-side, hypothesis-side, or local decision-boundary artifacts. NEI-CAP carries the same diagnostic stance to the evidence side and asks what shortcuts the construction of NEI evidence itself can teach. 2.3

3

NEI-CAP: Construction-Aware NEI Evaluation

Figure 1 motivates NEI-CAP as a way to separate shortcut recognition from evidence-insufficiency recognition. This section formalizes that idea with a construction variable, a compact taxonomy of evidence conditions, and the operational workflow in Protocol 1. 3.1

Evidence-Conditioned NEI

A verification instance is (c, E, y), where c is a claim, E = {e1 , . . . , ek } is its evidence set, and y ∈ {S UPPORT, R EFUTE, NEI}. The NEI label is a property of the pair (c, E) rather than of the claim alone (Thorne et al., 2018; Wadden et al., 2020): whether the evidence is insufficient depends on what evidence is provided. We make this dependence explicit by extending each example with a construction variable, x = (c, E, y, z, g),

Evidence Sufficiency and Missing Evidence

where z records the family of NEI evidence condition that produced E and g is a grouping identifier that keeps variants of the same claim within a single split. The model never receives z or g; they are diagnostic interventions, used only for auditing, splitting, and stratified reporting.

A claim can be true or false in the world while the available evidence is still insufficient to settle it, so evidence sufficiency is a distinct question from veracity prediction. Atanasova et al. (2022) make this question operational by removing parts of otherwise-valid evidence and asking whether fact-checking models notice the omission, and rationale evaluation benchmarks ask whether models identify supporting passages rather than only predicting labels (DeYoung et al., 2020). Work on missing counter-evidence similarly argues that evidence availability and sufficiency are assumptions built into fact-checking datasets (Glockner et al., 2022). The same question appears in factuality evaluation for generation, where long-form claims are decomposed into atomic facts and checked

3.2

NEI Construction Families

NEI-CAP separates shortcut-prone constructions from semantically related insufficient evidence. The former expose format, topic, position, or retrieval shortcuts; the latter test whether evidence that remains related to the claim is still recognized as insufficient. This follows the same motivation as contrast-set and behavioral evaluation: a benchmark should expose when a model succeeds through an unintended decision rule rather than the 3

intended capability (Gardner et al., 2020; Ribeiro et al., 2020; Geirhos et al., 2020). Table 1 lists the compact taxonomy used in the rest of the paper; full definitions, metadata fields, and shortcut-risk dimensions are in Appendix A. Family

Evidence condition

Role

Placeholder Random irrelevant Position-biased BM25 near-miss Cited non-rationale Same-document Fixed-claim Missing-hop

Fixed/empty no-evidence marker Unrelated evidence Predictable non-rationales High-overlap insufficient evidence Cited but non-rationale evidence Same-source non-rationale evidence Same claim, changed evidence Multi-hop evidence with a required fact removed

Format shortcut anchor Topic-mismatch anchor Position/source audit Hard NEI Hard NEI Source-controlled NEI Evidence-substitution diagnostic External multi-hop control

and R EFUTE portions of the task are held comparable across variants, while only the NEI evidence condition changes. This lets us train a verifier under one NEI construction and evaluate it under another. Splits are group-disjoint, keyed by the original claim or claim–document grouping, so construction variants of the same claim never straddle train and test partitions. 4.2

Candidate hard NEI examples are useful only if they are actually insufficient: BM25 nearmiss, cited non-rationale, and same-document nonrationale evidence can carry implicit support or refutation that the construction rule does not catch. NEI-CAP therefore separates candidate hard NEI from human-adjudicated hard NEI. We use two validation resources. The SciFact hard-NEI audit adjudicates candidate BM25/cited near-miss examples from the construction suite; the fixed-claim/same-document audit covers examples where the claim or source document is held fixed while the evidence condition changes. Two PhD annotators produced consensus adjudications; AI-assisted checks were used only as secondary support. The full protocol and label schema are in Appendix D. The primary human-hard model evaluation uses the held-out test split of the SciFact humanadjudicated hard-NEI audit: 54 validated hard-NEI examples after group-disjoint splitting. The larger audit pool estimates label validity, while the heldout subset supports model evaluation. Appendix G reports the full evaluation table and Appendix B documents the sampling trail.

Table 1: Compact NEI-CAP construction taxonomy.

3.3

Diagnostic Protocol

Protocol 1 lists the five stages that produce the construction-stratified evidence reported in Sections 5–6. The stages share a common output discipline: each one returns a typed artifact that the next stage can consume without re-deriving anything from the raw text. Protocol 1: NEI-CAP diagnostic workflow Input: claim–evidence examples (c, E, y) and construction rules. Output: audit tables, adjudicated subsets, construction-stratified metrics, and claim boundaries. 1. Construct. Assign construction family z and group ID g. 2. Audit. Measure evidence-side shortcut features by label and construction. 3. Validate. Adjudicate candidate hard NEI used for central claims. 4. Stress-test. Evaluate single- and mixed-construction training under construction-stratified tests. 5. Report. Release construction-specific metrics, uncertainty, and claim boundaries.

4

Data, Validation, and Experimental Setup

SciFact is our primary setting (Wadden et al., 2020): it requires evidence-sensitive verification over scientific abstracts, where reference evidence and semantically related but insufficient evidence routinely coexist in the same document. We use FEVER and HoVer as bounded external controls (Thorne et al., 2018; Jiang et al., 2020). 4.1

Human-Adjudicated Hard NEI

Validation resource SciFact BM25/cited hard-NEI audit Fixed-claim/same-document audit

n

Valid NEI

Contam.

250 122

89.2% 96.7%

10.8% 3.3%

Hard outcome 195 hard (87.4% of valid) 114 hard (96.6% of valid)

Table 2: Human validation summary. Contamination includes examples adjudicated as actually supported, actually refuted, ambiguous, or invalid. The hardoutcome column reports the absolute number of humanadjudicated hard NEI examples that pass the audit, with the in-parentheses rate computed against valid NEI. Full adjudication details are in Appendix D.

SciFact Construction Suite

The main SciFact suite instantiates the construction families in Table 1: placeholder, random irrelevant, position-biased, BM25 near-miss, and cited non-rationale. BM25 near-miss examples are obtained with BM25 retrieval (Robertson and Zaragoza, 2009) and then filtered to retain highoverlap but insufficient evidence. The S UPPORT

4.3

Models and Metrics

Our primary verifier is DeBERTa-v3-base (He et al., 2021), with RoBERTa-base and SciBERT as secondary backbones (Liu et al., 2019; Beltagy et al., 2019). We use these as diagnostic probes rather 4

than proposed architectures: training variants differ only in the NEI construction family used during training. Mixed-construction regimes combine the single-construction variants into an easy mixture (placeholder, random irrelevant, positionbiased), a hard mixture (BM25 near-miss, cited non-rationale), and a balanced mixture over all five families. Multi-seed experiments use seeds 13, 17, 23, 29, and 37.

SciFact NEI-F1 under construction shift

For three-way verification, we report Macro-F1 and class-specific F1, with particular attention to NEI-F1. For human-validated hard-NEI subsets, every evaluated example is adjudicated as NEI, so Macro-F1 is not informative; we instead report NEI recall, false support rate, false refute rate, and mean predicted class probabilities. Hyperparameters, checkpoint selection, seed aggregation, and metric details are in Appendices E and L.

4.4

0.67

0.56

0.72

0.65

0.68

Cited

0.51

0.58

0.77

0.77

0.60

PH

0.00

0.00

1.00

0.95

0.00

Pos

0.00

0.00

1.00

1.00

0.06

0.8

0.6

0.4

NEI-F1

Training NEI construction

1.0

BM25

0.2

0.44

0.29

0.99

0.99

0.99

25

d ite

PH

s Po

nd

Rand

0.0

BM

C

Ra

Evaluation NEI construction

Figure 2: SciFact NEI-F1 under train/test construction shifts. Cell values are seed-aggregated means from the primary construction matrix; Appendix F reports full numeric matrices and Macro-F1. Matched easy NEI can yield high scores without transferring to semantically related insufficient evidence.

External Controls The point is not that placeholder evidence is realistic; it is an intentionally shortcut-prone anchor. The broader finding is that different easy constructions teach different shortcuts—format absence, position/source bias, or topic mismatch—whereas the target of NEI-CAP is hard NEI: semantically related evidence that remains insufficient.

FEVER provides a Wikipedia verification control with a N OT E NOUGH I NFO label (Thorne et al., 2018), and HoVer provides a multi-hop control in which evidence sufficiency can depend on more than one supporting fact (Jiang et al., 2020). We use FEVER as a non-toy subset control and HoVer as a candidate missing-hop control. Together they test whether the construction-aware perspective travels beyond SciFact; neither is intended as a full-data or human-validated hard-NEI evaluation in its own right. Full details are in Appendix J.

5

Results

5.1

Matched NEI Performance Does Not Imply Transfer

5.2

Human-Adjudicated Hard NEI Exposes Easy-Training Failure

Human validation does not rescue easy NEI training. Table 3 shows that placeholder and positionbiased training yield 0.000 recall on humanadjudicated hard NEI. Random-irrelevant training also performs poorly (0.216 recall), confirming that topic-unrelated NEI does not teach hardinsufficiency recognition. Errors are dominated by false S UPPORT, and BM25 near-miss and cited non-rationale training partially recover recall.

Figure 2 shows the SciFact train/test construction matrix. A placeholder-trained verifier obtains perfect matched-placeholder NEI-F1, but falls to 0.000 on BM25 near-miss and cited nonrationale NEI. Position-biased training shows the same hard-construction collapse, while randomirrelevant training is less extreme but still shortcutprone: it is nearly solved under matched evaluation yet transfers poorly to BM25 and cited hard constructions. Full numeric matrices are in Appendix F.

Train NEI

Recall

False SUP

False REF

BM25 near-miss Cited non-rationale Random irrelevant Placeholder Position-biased

0.691 0.679 0.216 0.000 0.000

0.142 0.074 0.574 0.870 0.907

0.167 0.247 0.210 0.130 0.093

Table 3: Evaluation on the held-out SciFact humanadjudicated hard-NEI test split (n = 54). Since all examples are validated as NEI, we report NEI recall and error rates rather than Macro-F1; Appendix B gives the sampling trail from the larger audit pool.

5

Appendix G further shows that easy training assigns near-zero NEI probability to validated hard NEI while reallocating probability mass to S UP PORT and R EFUTE; the failure is therefore not just low recall but high-confidence wrong answers. 5.3

the collapse as a training-dynamic artifact of any one backbone. Model DeBERTa RoBERTa SciBERT

Mixed Training Helps but Remains Construction-Stratified

BM25

Cited

Hard recall

Easy mixture

0.379

0.273

0.178

Hard mixture

0.662

0.652

0.770

Balanced mixture

0.802

0.803

0.915

PH→BM25

PH→Cited

Drop

1.000 1.000 1.000

0.000 0.000 0.000

0.000 0.000 0.000

1.000 1.000 1.000

5.5 Source-Controlled Diagnostics Are Useful but Bounded Fixed-claim diagnostics extend the finding past NEI recall: when the same claim is paired first with reference evidence and then with humanadjudicated insufficient evidence, the probability that the verifier assigns to the reference S UP PORT /R EFUTE label drops on the insufficient side. Construction choice therefore affects how confidently the model commits to the non-NEI labels, not only whether it predicts NEI. Same-document hard NEI provides an additional source-controlled view that retains human adjudication; a shallowfeature audit shows it remains constructionally distinct from BM25/cited hard NEI, so we report it as its own diagnostic family rather than as a universal hard-NEI score. Full fixed-claim and samedocument diagnostics are in Appendix I.

Takeaway Shortcutheavy mixture remains weak Hard training improves recall Best, but still stratified

Table 4: DeBERTa mixed-construction training summary. BM25 and Cited are construction-stratified NEIF1; Hard recall is one-class recall on the SciFact humanadjudicated hard-NEI subset. Easy mixture includes random-irrelevant and position-biased NEI, so it improves over placeholder-only training but remains weak on hard insufficient evidence.

5.6

External Controls Are Bounded

FEVER and HoVer extend the diagnostic scope to Wikipedia and to multi-hop evidence. In both, a placeholder-trained baseline reaches NEI-F1 of 1.000 on placeholder evaluation and 0.000 on BM25 (FEVER) or missing-hop (HoVer) evaluation, mirroring the SciFact pattern at a different model scale and on a different domain. We use FEVER as a non-toy subset shortcut control and HoVer as a candidate missing-hop control; both are narrower in scope than the SciFact suite. Full external-control results are in Appendix J.

The balanced mixture improves the hard-side metrics, but the full stratified matrix in Appendix H still shows different performance profiles across placeholder, random-irrelevant, BM25, and cited conditions; mixed training is therefore a stronger stress test, not a replacement for constructionaware reporting. 5.4

PH→PH

5 5 5

Table 5: Five-seed placeholder-to-hard robustness. PH denotes placeholder NEI. Scores of 0.000 are exact: in all five seeds, placeholder-trained models never predicted NEI on the BM25 or cited hard test sets, yielding zero NEI-F1.

Mixed training is a stronger test because real benchmarks rarely contain a single NEI construction. Table 4 summarizes DeBERTa under an easy mixture, a hard BM25/cited mixture, and a balanced all-family mixture. Easy-mixture training remains weak on hard constructions; hard and balanced mixtures improve recall but still produce constructionstratified profiles. Mixed training changes the failure profile—it does not make NEI constructionfree. Train regime

Seeds

Placeholder-to-Hard Collapse Is Stable Across Seeds and Backbones

The placeholder-to-hard collapse is not specific to DeBERTa or to one seed. Across five seeds for DeBERTa, RoBERTa, and SciBERT, each backbone reaches perfect matched-placeholder NEI-F1 and zero NEI-F1 on BM25 near-miss and cited nonrationale evaluation when trained on placeholder NEI (Table 5). That the same exact 1.000→0.000 pattern reproduces across three architecture families and fifteen training runs rules out interpreting

6

Analysis and Discussion

6.1

NEI Is a Family of Evidence Conditions

Hard NEI should not be treated as one uniform category. BM25 near-miss, cited non-rationale, samedocument non-rationale, fixed-claim hard NEI, and missing-hop evidence remove different shortcuts and stress different behaviors. Construction-level 6

audits show that these families differ in evidence length, claim–evidence overlap, coverage, and retrieval metadata. These audits are not auxiliary checks: they are the mechanism by which NEICAP distinguishes evidence-insufficiency evaluation from shortcut-sensitive construction recognition. Surface differences do not invalidate the constructions; they show why construction family must be reported instead of hidden inside aggregate NEI-F1. Random irrelevant evidence is especially important: it can teach topic mismatch rather than insufficiency, demonstrating that the problem is broader than placeholder detection. 6.2

construction-specific signatures. Future benchmarks should therefore treat the construction of insufficient evidence as part of their evaluation protocol, not as an implementation detail.

7

NEI is often treated as a single negative label, but the way it is built decides what a verifier can actually learn. In SciFact-style verification, a placeholder-trained verifier earns a perfect matched-placeholder NEI-F1 yet scores zero on evidence that is topically related but insufficient—a pattern that reproduces across three architectures, five seeds, and two external datasets. NEI-CAP makes the construction explicit. By attaching every NEI example to the family of evidence condition that produced it and validating the hard cases through human adjudication, the protocol turns insufficient-evidence evaluation from a single number into a construction-stratified report. The construction of insufficient evidence belongs alongside the labels and metrics that define any fact-verification benchmark. An aggregate NEI-F1 that hides how its NEI examples were built cannot tell a reader whether the model has learned to recognize insufficiency or simply to recognize the artifact.

Mixed Training Changes the Profile, Not the Reporting Requirement

Mixed-construction training addresses a legitimate concern about single-construction probes. If construction sensitivity disappeared under mixed training, the single-construction matrix would be less relevant. It does not disappear. Easy-mixture training remains weak on hard constructions; hard and balanced mixtures improve hard-NEI recall but still produce construction-stratified profiles. Mixed training is therefore not a replacement for NEICAP; it is a stronger test of whether constructionaware reporting remains necessary. NEI-CAP’s output is not a better aggregate score; it is a reporting discipline. Every result we report carries the construction family that produced it, the audit metrics that describe its shortcut surface, and the validation status of any hard examples it relies on. 6.3

Conclusion

Limitations NEI-CAP is a diagnostic protocol, not a complete solution to evidence-insufficiency reasoning. It identifies construction sensitivity and provides audit tools, but strong NEI-CAP results do not guarantee general-purpose sufficiency reasoning. Different domains may require additional construction families and annotation guidelines. Our strongest evidence comes from SciFactstyle scientific verification. FEVER and HoVer are bounded controls: FEVER is a subset control, and HoVer uses candidate missing-hop constructions rather than human-validated hard NEI. SciFact does not natively provide a construction-stratified training split; reconstructing one in model-ready form would require additional preprocessing outside the scope of this paper. Human validation reduces label-validity risk but does not eliminate it. BM25 near-miss, cited nonrationale, and same-document examples can carry implicit support, refutation, or ambiguity. Final labels are the consensus of two PhD annotators,

What Fixed-Claim Diagnostics Can and Cannot Show

Fixed-claim evidence substitution separates two abilities: recognizing semantically related evidence as insufficient, and assigning the reference label when decisive evidence is present. The probabilitydrop metric shows that replacing reference evidence with insufficient evidence lowers confidence in the reference label, so construction affects S UP PORT /R EFUTE verification confidence as well as NEI recall. The same-document artifact audit further strengthens the taxonomy argument. Samedocument hard NEI is human-adjudicated, but shallow features nearly separate it from BM25/cited hard NEI and other construction families. Human validity and constructional distinctness are different properties: even human-valid hard NEI can carry 7

reached after adjudicating a small number of preconsensus disagreements; these disagreements concerned fine-grained sub-label boundaries within the contamination categories rather than the binary hard-NEI versus contaminated distinction that the paper relies on. Appendix D gives the full protocol. Fixed-claim and same-document diagnostics are decomposition tests, not proofs of clean counterfactual evidence use. Same-document hard NEI is human-adjudicated but constructionally distinct under shallow-feature audits, so we report it as a source-controlled diagnostic family rather than as an artifact-free universal hard-NEI score. Our model probes are pretrained encoder crossencoders with secondary-backbone checks. We use them because they support controlled constructionspecific training, seed replication, and prediction logging. Recent factuality, retrieval-augmented, and grounding benchmarks evaluate whether generated atomic facts or long-form responses are supported by reliable sources, retrieved passages, or provided documents (Min et al., 2023; Niu et al., 2024; Jacovi et al., 2025). Extending NEI-CAP from fact-verification labels to those generative evaluation settings is future work.

9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458. Association for Computational Linguistics. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, and 7 others. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665–673. Max Glockner, Yufang Hou, and Iryna Gurevych. 2022. Missing counter-evidence renders NLP factchecking unrealistic for misinformation. arXiv preprint arXiv:2210.13865.

References Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact extraction and VERification over unstructured and structured information. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence. Transactions of the Association for Computational Linguistics, 10:746–763.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradientdisentangled embedding sharing. arXiv preprint arXiv:2111.09543.

Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidencebased fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4685–4697, Hong Kong, China. Association for Computational Linguistics.

Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, and 7 others. 2025. The FACTS grounding leaderboard: Benchmarking LLMs’ ability to ground responses to longform input. arXiv preprint arXiv:2501.03200.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the

Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and

8

claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460, Online. Association for Computational Linguistics.

Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics.

Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.

Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3419–3425, Hong Kong, China. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. Association for Computational Linguistics.

James Thorne and Andreas Vlachos. 2019. Adversarial attacks against fact extraction and verification. arXiv preprint arXiv:1903.05543. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

Juraj Vladika, Ivana Hacajova, and Florian Matthes. 2025. Step-by-step fact verification system for medical claims with explainable reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 805–816, Albuquerque, New Mexico. Association for Computational Linguistics.

Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10862– 10878, Bangkok, Thailand. Association for Computational Linguistics.

Juraj Vladika, Phillip Schneider, and Florian Matthes. 2023. HealthFC: Verifying health claims with evidence-based medical fact-checking. arXiv preprint arXiv:2309.08503.

Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664. Association for Computational Linguistics.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912. Association for Computational Linguistics. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3:333–389.

9

A

NEI-CAP Construction Taxonomy

A.2

This appendix expands the compact taxonomy in Table 1. NEI-CAP treats NEI not as a constructionfree negative label, but as an evidence-conditioned label whose interpretation depends on how the evidence set is paired with the claim.

Construction Families

Family

Construction rule

Shortcut risk

Validation status

Paper role

Placeholder

Replace evidence with a fixed or empty no-evidence marker. Pair claim with evidence from unrelated claims or documents. Select non-rationale evidence from predictable locations. Retrieve highoverlap evidence that remains insufficient. Use evidence from claim-associated cited documents that is not the rationale. Use non-rationale evidence from the same source document.

Format, length, absence.

Constructed only.

Shortcut baseline.

Topic match, overlap.

mislow

Constructed only.

Topicmismatch baseline.

Sentence position, source distribution. Lexical overlap mistaken for support.

Not primary humanvalidated. Humanadjudicated in SciFact audit. Humanadjudicated in SciFact audit. Humanadjudicated in fixedclaim/samedocument audit. Humanadjudicated in fixedclaim/samedocument audit. Candidateonly in HoVer.

Artifact stress test.

Random irrelevant

Positionbiased

A.1

Taxonomy Axes

BM25 nearmiss

We characterize each NEI construction family along six axes: evidence availability, topical relatedness, source control, lexical or entity overlap, evidential completeness, and validation status. These axes separate absence-based shortcuts from semantically related insufficient evidence.

Cited nonrationale

Samedocument nonrationale

Evidence availability. Some NEI examples contain no substantive evidence or use a fixed placeholder. Others contain non-empty evidence that is related to the claim but still insufficient. Topical relatedness. Random irrelevant evidence may be insufficient because it is off topic. BM25 near-miss, cited non-rationale, samedocument non-rationale, and missing-hop examples are instead related to the claim but incomplete.

Source relevance mistaken for sufficiency. Residual topic/source cues.

Fixed-claim hard NEI

Pair the same claim with reference evidence and validated insufficient evidence.

Evidence-side differences may still contain shallow cues.

Missinghop control

Remove a required supporting fact from multi-hop evidence.

FEVER subset control

Construct bounded Wikipedia verification controls with alternative NEI evidence.

Partial evidence overinterpreted as sufficient. Datasetspecific lexical or retrieval artifacts.

Subset control, not humanvalidated hard NEI.

Hard NEI.

Hard NEI.

Sourcecontrolled hard NEI.

Evidencesubst. diagnostic.

External multi-hop control. External shortcut control.

Table 6: Full NEI-CAP construction taxonomy. The taxonomy separates shortcut-prone baselines, candidate hard NEI, human-adjudicated hard NEI, and bounded external controls.

A.3 Source control. Same-document non-rationales and fixed-claim diagnostics reduce source and topic shortcuts by holding the source document or claim fixed.

Construction Strength

We use three construction-strength categories. Easy NEI. Placeholder, random irrelevant, and position-biased examples are shortcut-prone by design. They are useful probes of artifact sensitivity but should not be treated as strong evidencesufficiency tests.

Lexical and entity overlap. Near-miss examples preserve lexical or entity overlap while removing decisive support, testing whether models mistake overlap for sufficiency.

Candidate hard NEI. Candidate hard examples are constructed to be semantically related but insufficient. Before adjudication, they may contain semantic contamination.

Evidential completeness. Partial and missinghop examples contain some relevant information while omitting a necessary relation, condition, or reasoning step.

Human-adjudicated hard NEI. Humanadjudicated hard examples are candidate hard examples labeled as truly insufficient rather than actually supportive, actually refuting, ambiguous, or invalid. These subsets support the strongest hard-NEI claims in the paper.

Validation status. NEI-CAP distinguishes constructed examples, candidate hard NEI, and humanadjudicated hard NEI. The strongest hard-NEI claims are restricted to human-adjudicated subsets. 10

A.4

B.2

Terminology and Claim Boundaries

Avoid

Use instead

Reason

hard-negative

hard NEI example

gold-truth

gold label / reference label reference-evidence side insufficient-evidence side

Avoids retrieval/contrastive ambiguity. Standard terminology.

gold-side hard-side counterfactual proof HoVer human-validated FEVER full validation

fixed-claim evidencesubstitution diagnostic HoVer candidate missinghop control FEVER subset control

More precise. Defines evidence condition. Avoids causal overclaim. No HoVer human audit. Bounded external probe.

Table 7: Terminology mapping used throughout the paper.

Field

Type

Description

example ID claim ID group ID source data

string string string string

claim evidence label construction

string string/list categorical categorical

split document ID sentence IDs

categorical string/list string/list

retrieval method retrieval rank BM25 score sentence position validation status

string integer float integer/list categorical

adjudicated label

categorical

Unique claim–evidence instance ID. Original claim ID. Group key for group-disjoint splitting. SciFact, FEVER, HoVer, or derived resource. Claim text. Evidence text or list of evidence units. S UPPORT, R EFUTE, or NEI. Placeholder, BM25 near-miss, samedocument, etc. Train, development, test, or audit. Source document IDs when available. Sentence or rationale IDs when available. Retrieval or sampling method. Rank of retrieved evidence. Retrieval score, when applicable. Sentence position within document. Not validated, candidate, valid NEI, contaminated, or ambiguous. Final adjudicated label if available.

Table 8: Recommended NEI-CAP manifest schema. Construction metadata is required to report results by evidence condition rather than by label alone.

The taxonomy supports three bounded claims: NEI evaluation is construction-sensitive; easy NEI can inflate apparent competence; and hard insufficient evidence should be validated when it supports central claims. It does not imply that NEI-CAP solves evidence sufficiency, that all hard-NEI families are equivalent, or that fixed-claim substitution proves clean counterfactual evidence use.

B

Manifest Schema

B.3

SciFact Suite

The core SciFact suite contains five variants: placeholder, random irrelevant, position-biased, BM25 near-miss, and cited non-rationale. The S UPPORT and R EFUTE portions remain comparable across variants; only the NEI evidence condition changes. This design supports train/test construction-shift evaluation.

Dataset Construction and Manifests

B.4

Group-Disjoint Splitting

This appendix documents the dataset assets and manifest requirements behind NEI-CAP. The main paper reports only compact descriptions; the appendix records the construction metadata needed for reproducible construction-aware evaluation.

Construction variants derived from the same claim or claim–document group must not leak across train and test. NEI-CAP therefore uses group-disjoint splitting keyed by claim, claim–document pair, or fixed-claim substitution group where applicable. This prevents a model from exploiting claim memorization across evidence variants.

B.1

B.5

Example Representation

Human-Adjudicated Assets

Human-adjudicated assets are separated from automatically constructed candidate data. The SciFact audit validates BM25/cited candidate hard NEI; the fixed-claim/same-document audit validates sameclaim and same-document diagnostics. These assets estimate semantic contamination and define human-adjudicated hard-NEI subsets for model evaluation.

Each example is represented as:

xi = (ci , Ei , yi , zi , gi , mi ),

where ci is the claim, Ei is the evidence set, yi ∈ {S UPPORT, R EFUTE, NEI} is the label, zi is the construction family, gi is a group identifier, and mi stores provenance and audit metadata. The construction variable zi is not given to the model; it is used for splitting, auditing, and reporting.

B.6

Human-Hard Evaluation Sampling Trail

The SciFact human-audit pool estimates label validity, while the model evaluation uses only the held-out group-disjoint test split. Table 9 records 11

B.9

the paper-facing sampling trail used to interpret the n = 54 hard-NEI evaluation in Table 3. Stage

The accompanying release links each reported result to its source dataset, construction family, split policy, model configuration, seed set, prediction artifact, and evaluation output. The full machinereadable asset map is provided with the released manifests.

Count

Candidate BM25/cited hard-NEI audit pool Human-valid NEI after adjudication Human-adjudicated hard-NEI subtype Held-out group-disjoint model-evaluation split

250 223 195 54

Table 9: Sampling trail for the SciFact human-hard model-evaluation subset. The larger audit pool estimates label validity; the held-out split supports model evaluation.

B.7

C

FEVER and HoVer are included as bounded external controls. FEVER provides a non-toy Wikipedia subset control. HoVer provides a candidate missing-hop control for multi-hop insufficiency. Neither is treated as human-validated hard NEI.

C.1

C.2

The additional experiments add construction-level split statistics for each paper-facing family. The audit records label counts, claim and document group counts, evidence length, sentence count, placeholder rate, claim–evidence overlap, coverage, retrieval metadata, duplicate counts, and missingfield checks. Table 10 gives a compact test-split view for the core SciFact suite; full CSV artifacts will be released with the accompanying code and data package.

Placeholder Random irrelevant Position-biased BM25 near-miss Cited non-rationale

SUP/REF/NEI

Avg. tok.

Coverage

177 177 177 177 177

76/40/61 76/40/61 76/40/61 76/40/61 76/40/61

174.7 236.8 181.7 251.4 246.9

0.444 0.474 0.501 0.630 0.590

Audited Features

We audit evidence length, number of evidence sentences, claim–evidence lexical overlap, claim– evidence coverage, placeholder rate, sentence position, source concentration, and discourse markers. These features do not determine semantic validity; they identify shortcut risk.

B.8 Construction Split Statistics and Leakage Audit

n

Artifact Audit Statistics

NEI-CAP audits construction families before interpreting model performance. The audit asks whether NEI examples can be separated using superficial evidence-side features rather than evidencesufficiency reasoning.

External Controls

Variant

Asset Map

Construction-Level Summary n

Avg. sent.

Avg. tok.

Coverage

SciFact BM25/cited hard NEI Same-document hard NEI Fixed-claim hard NEI Placeholder NEI

195

11.49

216.56

0.483

Humanadjudicated

114

1.04

23.93

0.233

114

1.23

23.93

0.233

61

1.00

2.00

0.005

Random irrelevant NEI

61

9.70

182.46

0.092

Humanadjudicated Humanadjudicated Constructed only Constructed only

Construction group

Status

Table 11: Artifact audit summary by construction group. Placeholder NEI has extremely short evidence and nearzero coverage; human-adjudicated hard NEI contains substantive evidence.

C.3 Table 10: Compact test-split statistics for the SciFact construction suite. Each variant keeps comparable S UP PORT /R EFUTE /NEI label counts while changing the NEI evidence condition.

Overlap and Marker Statistics

Construction group SciFact BM25/cited hard NEI Same-document hard NEI Fixed-claim hard NEI Placeholder NEI Random irrelevant NEI

The leakage audit reports zero claim-group overlap across train/development/test for the core SciFact construction variants and zero constructionvariant cross-split leakage. Document overlap can occur because the same scientific paper may be relevant to multiple claims; we report it as sourcedistribution metadata rather than claim leakage.

Overlap count

Jaccard

Context marker

Method marker

6.24

0.053

0.210

0.226

3.06

0.098

0.316

0.140

3.06

0.098

0.316

0.140

0.05 1.31

0.005 0.013

0.000 0.148

0.000 0.213

Table 12: Claim–evidence overlap and marker statistics. Random irrelevant NEI contains long evidence but low overlap, indicating a topic-mismatch shortcut rather than evidence insufficiency.

12

C.4

D.3

Hard-NEI Subtypes

Reporting Metrics

We report: Group

Broad topic

Near-miss

Partial

62 50

102 49

31 15

SciFact BM25/cited hard NEI Fixed-claim/same-document hard NEI

valid NEI =

Table 13: Subtype distribution for human-adjudicated hard NEI. Hard NEI is heterogeneous rather than a single condition.

C.5

and: contam. =

Interpretation

D.4

Valid NEI rate Contamination rate Actually supported rate Actually refuted rate Ambiguous/invalid rate Hard subtype rate among valid NEI Topic-unrelated rate among valid NEI

D.5

Final adjudicated labels Human-validated NEI Human-validated hard NEI Same-claim hard NEI Same-document hard NEI

Human validation determines whether a candidate hard-NEI example is truly insufficient. The resulting labels are used to estimate semantic contamination and to define human-adjudicated hard-NEI subsets for evaluation.

Definition Evidence does not support or refute the claim. Evidence supports the claim. Evidence refutes the claim; reported as R E FUTE in paper-facing terminology. Status cannot be determined reliably. Claim or evidence is malformed, missing, unreadable, or out of scope.

actually_supported actually_contradicted ambiguous invalid_or_unreadable

95% high

0.852 0.072 0.028 0.028 0.000 0.826 0.076

0.928 0.148 0.088 0.084 0.000 0.919 0.164

Rows

Contam.

Hard rows

Hard rate

122 118 114 114 114

0.0328 0.0000 0.0000 0.0000 0.0000

114 114 114 114 114

0.966 0.966 1.000 1.000 1.000

Table 16: Fixed-claim/same-document validation summary. The 114 hard-NEI examples support same-claim and same-document diagnostics.

D.6

Annotation Labels

truly_insufficient

95% low

0.892 0.108 0.056 0.052 0.000 0.874 0.117

Fixed-Claim and Same-Document Validation

Subset

Validation Goal

Label

Observed

Table 15: SciFact hard-NEI audit validation over 250 candidate hard-NEI examples. Intervals are bootstrap 95% intervals.

Hard insufficient evidence can be noisy. Retrieved near-misses, cited non-rationales, and samedocument non-rationales may contain implicit support, implicit refutation, or ambiguity. NEI-CAP therefore separates automatically constructed candidate hard NEI from human-adjudicated hard NEI.

D.2

SciFact Hard-NEI Audit

Metric

Human Validation Protocol and Adjudication

D.1

Nsup + Nref + Namb/invalid . Naudited

AI-assisted checks, where used, are treated only as triage or secondary support, not as human validation.

The audit supports three conclusions. First, placeholder NEI exposes strong format and absence cues. Second, random irrelevant NEI mainly tests topic mismatch. Third, human-adjudicated hard NEI is heterogeneous across BM25/cited, samedocument, and fixed-claim settings. These findings motivate construction-specific reporting rather than aggregate NEI-F1 alone.

D

Ninsufficient , Naudited

Annotator and Adjudication Protocol

The revised protocol documentation clarifies that two PhD annotators participated in human validation and that final labels are consensus adjudications. Annotators independently evaluated claim plus candidate evidence for semantic sufficiency. Blinded packets removed LLM labels, gold labels, model predictions, and construction provenance from primary annotation fields. AI-assisted checks were used only as secondary support or triage and are never reported as standalone human validation. Pre-consensus agreement between the two annotators was high. The small number of disagreements concerned fine-grained sub-label boundaries

Table 14: Human validation label schema. Contamination includes actually supported, actually contradicted/refuted, ambiguous, invalid, and unreadable cases.

13

E.3

within the contamination categories. The hardNEI subsets used for paper-facing model evaluation therefore depend on a coarser decision than the one on which annotators occasionally diverged. Disagreements were resolved through joint adjudication; the resulting consensus labels are the labels used throughout the paper.

The primary matrix trains one verifier for each NEI construction and evaluates it on all five constructions. Each cell asks whether performance transfers across evidence conditions while S UPPORT and R EFUTE portions remain comparable. E.4

D.7

Boundary

E.5

Experimental Setup and Model Details

Task Format

E.6

All neural experiments are three-way claim– evidence classification:

The model receives claim and evidence text. Construction family is not provided as model input; it defines train/evaluation variants and reporting groups.

E.7

Value

Primary model Input root Train variants

deberta-v3-base NEI-CAP SciFact suite placeholder, cited, random irrelevant, BM25 near-miss, position-biased placeholder, cited, random irrelevant, BM25 near-miss, position-biased test 384 3 16 2.0 × 10−5 0.01 0.10 enabled development Macro-F1 1 bf16 if available, otherwise fp32 13, 17, 23 save best checkpoint enabled

Evaluation variants Evaluation split Max length Epochs Batch size Learning rate Weight decay Warmup ratio Class weighting Early stopping metric Patience Precision Seeds Checkpoint policy Prediction logging

Additional Experiments

Additional experiments add targeted checks without overwriting the primary outputs. The new runs use five seeds, {13,17,23,29,37}, for expanded placeholder-to-hard uncertainty and mixedconstruction training. Mixed-construction regimes include an easy mixture over shortcut-prone constructions, a hard mixture over BM25/cited constructions, and a balanced mixture over all five SciFact construction families. These experiments test whether construction sensitivity persists under more realistic training distributions; they are not treated as new model architectures.

Primary SciFact Configuration

Setting

Robustness Classification

For secondary-backbone SciFact probes, construction sensitivity is considered replicated when placeholder matched NEI-F1 is high, placeholder-tohard drop is large, and at least one hard construction has non-trivial matched NEI-F1. These thresholds are diagnostic for this study, not universal standards.

y ∈ {S UPPORT, R EFUTE, NEI}.

E.2

Metrics

For three-way verification, we report accuracy, Macro-F1, class-specific F1, and especially NEIF1. For one-class human-hard evaluation, MacroF1 is not meaningful. We report NEI recall, false S UPPORT rate, false R EFUTE rate, and mean predicted class probabilities.

NEI-CAP uses models as diagnostic probes of construction sensitivity, not as proposed architectures. E.1

Secondary Backbones

RoBERTa-base and SciBERT are secondary robustness probes. They reuse the same construction suite and train/evaluation variants. Secondary models supplement, but do not replace, the locked primary DeBERTa matrix.

The paper may claim that the SciFact hard-NEI audit and the fixed-claim/same-document audit provide human-adjudicated hard-NEI subsets. It should not claim that human validation eliminates all ambiguity, that candidate-only examples are valid without adjudication, or that FEVER/HoVer controls are human-validated hard NEI.

E

Construction Matrix

E.8

Result Provenance

Each reported result should be traceable to a dataset manifest, construction family, model configuration, seed set, prediction file, evaluation output, and paper table or figure. Aggregate scores without construction metadata are insufficient for NEI-CAP.

Table 17: Primary SciFact construction-matrix configuration.

14

F

Full SciFact Construction Matrices

examples are validated as NEI, Macro-F1 is not reported.

This appendix reports the full locked DeBERTa SciFact construction-shift matrices. F.1

G.1

NEI-F1 Matrix

Train NEI

BM25

Cited

Placeholder

Position

Random

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.675 0.508 0.000 0.000 0.445

0.561 0.578 0.000 0.000 0.294

0.725 0.768 1.000 1.000 0.995

0.652 0.768 0.947 1.000 0.992

0.676 0.599 0.000 0.063 0.995

Recall and Error Rates

Train NEI

Recall

95% low

95% high

False SUP

False REF

n

BM25 near-miss Cited non-rationale Random irrelevant Placeholder Position-biased

0.691 0.679 0.216 0.000 0.000

0.593 0.562 0.123 0.000 0.000

0.790 0.790 0.321 0.000 0.000

0.142 0.074 0.574 0.870 0.907

0.167 0.247 0.210 0.130 0.093

54 54 54 54 54

Table 21: Primary DeBERTa evaluation on humanadjudicated hard NEI. False REF maps internal contradiction errors to the paper-facing R EFUTE label.

Table 18: Full SciFact construction-shift matrix for NEIF1. Rows are train constructions; columns are evaluation constructions.

G.2

Predicted Probabilities

Train NEI

F.2

Macro-F1 Matrix

Train NEI

BM25

Cited

Placeholder

Position

Random

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.529 0.357 0.277 0.262 0.474

0.479 0.386 0.277 0.262 0.415

0.559 0.474 0.684 0.665 0.716

0.530 0.474 0.657 0.665 0.715

0.532 0.399 0.276 0.285 0.716

BM25 near-miss Cited non-rationale Random irrelevant Placeholder Position-biased

G.3

Placeholder Position-biased Random irrelevant BM25 near-miss Cited non-rationale

Matched

BM25

Cited

Hard drop

1.000 1.000 0.995 0.675 0.578

0.000 0.000 0.445 0.675 0.508

0.000 0.000 0.294 0.561 0.578

1.000 1.000 0.625 0.057 0.035

Table 20: Matched and hard-evaluation NEI-F1 comparisons. Hard drop compares matched performance against average BM25/cited performance.

F.4

H

Mean P (SUP)

Mean P (REF)

n

0.431 0.458 0.304 0.013 0.014

0.324 0.318 0.417 0.540 0.551

0.277 0.266 0.371 0.448 0.435

54 54 54 54 54

Interpretation

Multi-Model Robustness

This appendix documents secondary-backbone robustness and five-seed revision checks. RoBERTa and SciBERT are used as diagnostic probes; they do not replace the locked primary DeBERTa matrix.

Interpretation

The matrix shows that NEI is not constructionfree. Placeholder and position-biased training yield high matched performance but collapse on hard constructions. BM25 and cited training do not yield perfect matched scores, but they transfer more stably across hard constructions.

G

95% high

0.366 0.371 0.129 0.012 0.013

Placeholder and position-biased training fail to recognize validated hard NEI. Their errors are dominated by false S UPPORT, and their mean P (NEI) is near zero while P (Support) and P (Refute) absorb most probability mass. Random-irrelevant training also performs poorly, showing that topicunrelated NEI does not teach the intended hardinsufficiency behavior. BM25 and cited training partially recover hard-NEI recognition but do not solve the task.

Drop Summary

Train NEI

95% low

0.398 0.416 0.212 0.013 0.014

Table 22: Mean predicted probabilities on humanadjudicated hard NEI. Easy training assigns near-zero NEI probability to validated hard insufficient evidence and reallocates probability mass to S UPPORT and R E FUTE.

Table 19: Full SciFact construction-shift matrix for Macro-F1. Aggregate performance can appear strong under easy matched evaluation while hiding hard-NEI failure.

F.3

Mean P (NEI)

H.1

Five-Seed Placeholder-to-Hard Drop

The main paper reports the exact five-seed placeholder-to-hard comparison. Across DeBERTa, RoBERTa, and SciBERT, placeholdertrained verifiers obtain PH→PH NEI-F1 of 1.000 and PH→BM25/Cited NEI-F1 of 0.000 for all five seeds. We omit the duplicate table here and use this appendix for expanded secondary-backbone, mixed-training, and same-document diagnostics.

Human-Validated Hard-NEI Evaluation

This appendix reports full evaluation on the heldout SciFact human-adjudicated hard-NEI test split (n = 54). The larger audit pool and sampling trail are documented in Appendix B; since all evaluated 15

H.2

H.4

Secondary Backbones on Human-Adjudicated Hard NEI

Model

Train NEI

Recall

False SUP

False REF

Mean P (NEI)

SciBERT SciBERT SciBERT SciBERT SciBERT

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.838 0.875 0.000 0.007 0.099

0.070 0.063 0.574 0.838 0.520

0.092 0.062 0.426 0.156 0.381

0.710 0.692 0.011 0.024 0.114

RoBERTa RoBERTa RoBERTa RoBERTa RoBERTa

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.703 0.638 0.000 0.009 0.186

0.116 0.109 0.867 0.932 0.586

0.181 0.253 0.133 0.060 0.227

0.404 0.435 0.004 0.014 0.191

Same-Document Robustness

Model

Train NEI

Recall

False SUP

False REF

SciBERT SciBERT SciBERT SciBERT SciBERT

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.482 0.731 0.000 1.000 0.216

0.231 0.143 0.509 0.000 0.415

0.287 0.126 0.491 0.000 0.368

RoBERTa RoBERTa RoBERTa RoBERTa RoBERTa

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

0.816 0.871 0.526 1.000 0.950

0.023 0.000 0.412 0.000 0.029

0.161 0.129 0.061 0.000 0.020

Table 23: Secondary-backbone evaluation on humanadjudicated hard NEI. Placeholder-trained RoBERTa and SciBERT both collapse on validated hard insufficient evidence.

Table 26: Secondary-backbone same-document evaluation. Same-document hard NEI behaves differently from SciFact BM25/cited hard NEI, reinforcing hardNEI heterogeneity.

H.3

H.5

Mixed-Construction Training

The secondary models support the claim that construction sensitivity persists beyond DeBERTa. They do not show that all model families behave identically, and they do not replace the primary SciFact matrix.

We evaluate mixed-construction training regimes. Table 24 summarizes the DeBERTa revision results. Easy mixtures remain weak on hard conditions, while hard and balanced mixtures improve hardNEI recognition. These results reduce the artificiality of single-construction training but preserve the need for construction-stratified reporting.

I Regime

BM25

Cited

SciFact hard

Easy mixture Hard mixture Balanced mixture

0.379 0.662 0.802

0.273 0.652 0.803

0.178 0.770 0.915

Easy mixture Hard mixture Balanced mixture

PH

Position

Random

BM25

Cited

0.989 0.754 0.903

0.989 0.729 0.903

0.960 0.694 0.901

0.379 0.662 0.802

0.273 0.652 0.803

Same-Claim and Same-Document Diagnostics

Fixed-claim/same-document diagnostics control the claim or source document while varying evidence. They are decomposition tests, not proofs of clean counterfactual evidence use.

Table 24: DeBERTa mixed-construction training summary. BM25 and Cited are construction-stratified NEIF1; SciFact hard is one-class NEI recall on humanadjudicated hard NEI. Same-document results are reported separately because that family is constructionally distinct.

Regime

Boundary

I.1

Setup

The same-claim diagnostic pairs the same claim with reference evidence and with humanadjudicated insufficient evidence. The samedocument diagnostic draws insufficient evidence from the same source document as the reference evidence. We report hard-NEI recall, false S UPPORT/R EFUTE rates, reference-side accuracy, probability-drop success, strict swap success, and the mean drop in the reference-label probability. The reference label is S UPPORT or R EFUTE, so this metric tracks non-NEI verification confidence rather than only NEI recall.

Table 25: Full DeBERTa mixed-construction stratified NEI-F1 matrix. The balanced mixture improves all conditions but still yields a construction-specific profile: easy conditions remain higher than BM25/cited hard conditions. PH denotes placeholder.

16

I.2

Primary DeBERTa Fixed-Claim Diagnostics

Train NEI

Hard recall

Swap success

Mean ∆ ref

0.661 0.988 0.898 1.000 0.947

0.535 0.743 0.737 0.728 0.807

0.017 0.023 0.112 0.034 0.124

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

BM25 near-miss Cited non-rationale Placeholder Position-biased Random irrelevant

Hard recall

False SUP

False REF

0.661 0.988 0.898 1.000 0.947

0.310 0.009 0.085 0.000 0.053

0.029 0.003 0.018 0.000 0.000

0.997 1.000 1.000 1.000 –

0.997 1.000 1.000 1.000 –

J

completed completed completed completed underpowered

Interpretation

External Controls: FEVER and HoVer

FEVER and HoVer are bounded external controls. They test whether construction-aware diagnostics are useful beyond SciFact, but they are not humanvalidated hard-NEI benchmarks.

Secondary-Backbone Same-Claim Diagnostics

Model

Train NEI

Ref. acc.

Hard recall

Prob. drop

Strict swap

SciBERT SciBERT SciBERT SciBERT SciBERT

Placeholder Cited non-rationale Random irrelevant BM25 near-miss Position-biased

0.599 0.488 0.556 0.529 0.006

0.000 0.731 0.216 0.482 1.000

0.494 0.827 0.602 0.734 0.883

0.000 0.365 0.140 0.289 0.006

RoBERTa RoBERTa RoBERTa RoBERTa RoBERTa

Placeholder Cited non-rationale Random irrelevant BM25 near-miss Position-biased

0.401 0.108 0.091 0.135 0.009

0.503 0.833 0.956 0.763 1.000

0.643 0.614 0.760 0.605 0.538

0.137 0.053 0.079 0.050 0.009

J.1

FEVER Subset Control

Baseline

Train NEI

Claim+evidence TF-IDF Claim+evidence TF-IDF Claim+evidence TF-IDF Evidence-only TF-IDF Length/overlap logistic regression

Placeholder BM25 near-miss Random irrelevant BM25 near-miss BM25 near-miss

Placeholder

BM25

Random

PH→BM25 drop

1.000 0.614 0.237 0.826 0.772

0.000 0.373 0.414 0.313 0.406

0.000 0.327 0.476 0.274 0.536

1.000 0.241 -0.177 0.513 0.366

Table 31: FEVER external subset control. Placeholdertrained shallow models solve placeholder NEI while failing on BM25 near-miss and random-irrelevant evaluation.

Table 29: Secondary-backbone same-claim diagnostics. Results motivate the bounded interpretation: hard-side recognition and reference-side verification can diverge.

I.5

Same-doc vs BM25/cited hard Same-doc vs placeholder Same-doc vs random irrelevant Same-doc vs constructed BM25/cited Matched feature audit

Status

Same-claim diagnostics separate two abilities: recognizing that semantically related evidence is insufficient, and assigning higher confidence to the reference label when decisive evidence is present. The positive mean reference-label deltas in Table 27 show that substituting insufficient evidence lowers confidence in the original S UPPORT/R EFUTE label. A model may still succeed on the hard-NEI side while remaining weak on the reference-evidence side. Therefore, fixed-claim evidence substitution is a useful diagnostic but not a proof of clean counterfactual evidence use.

Table 28: Primary DeBERTa same-document hard-NEI diagnostics. The hard subset is shared with the fixedclaim diagnostic, so hard recall matches Table 27; samedocument evaluation does not define swap success or reference-label probability drop, and instead reports error rates on the insufficient-evidence side.

I.4

Macro-F1

I.6

Primary DeBERTa Same-Document Hard-NEI Diagnostics Train NEI

Acc.

Table 30: Same-document artifact audit. Samedocument hard NEI is human-adjudicated but constructionally distinct; it should be reported as a sourcecontrolled diagnostic family, not as artifact-free universal hard NEI.

Table 27: Primary DeBERTa fixed-claim diagnostics. The same claim is paired with reference evidence and with insufficient evidence, so swap success and mean reference-label probability drop are defined.

I.3

Comparison

Same-Document Artifact Audit J.2

We audit whether the same-document hard-NEI family has a shallow construction signature. A shallow feature classifier nearly separates samedocument hard NEI from several other construction groups using evidence length, sentence count, overlap, and coverage features. Table 30 reports the compact audit result. The matched length/coverage audit is underpowered, so we do not overinterpret matched-subset comparisons.

HoVer Candidate Missing-Hop Control

Train

Eval

Missing-hop Missing-hop Placeholder Placeholder

Missing-hop Placeholder Missing-hop Placeholder

NEI-F1

NEI recall

False SUP

neval

0.835 0.973 0.000 1.000

0.756 1.000 0.000 1.000

0.097 0.000 0.431 0.000

7,978 7,978 7,978 7,978

Table 32: HoVer candidate missing-hop control. Values are rounded to three decimals. HoVer rows are candidate-only and are not human-validated hard NEI.

17

J.3

K.3

Boundary

FEVER supports a non-toy subset-level shortcutcontrol claim. HoVer supports a candidate-only multi-hop construction-sensitivity claim. Neither supports full cross-dataset human-validated hardNEI generalization.

K

Value

Dataset Claim

SciFact fixed-claim/same-document T cell receptor/CD3 microdomains are required to induce the immunologic synapse. Same-claim reference-side weakness Position-biased / 13 NEI NEI P (reference) = 0.001; P (NEI | Ehard ) = 0.998 The hard side is recognized as NEI, but the referenceevidence side is missed, motivating the same-claim caveat.

Why it matters

Table 34: Case study: fixed-claim reference-side weakness.

This appendix provides qualitative examples from row-level artifacts. We include only cases with available claim/evidence text and, where model behavior is discussed, saved predictions and probabilities. We do not reconstruct unavailable cases from aggregate statistics.

K.4

Case Selection Policy

Cases are selected only when: the claim and evidence are available, prediction and probability fields are available when model behavior is discussed, and human validation status is available when the case is described as human-adjudicated hard NEI. FEVER and HoVer cases are construction examples or candidate controls unless humanadjudicated.

K.2

Field

Construction Train/seed Ref. pred. Hard pred. Prob.

Diagnostic Case Studies

K.1

Reference-Side Weakness

Same-Document Human-Validated Hard NEI

Field

Value

Dataset Claim

SciFact fixed-claim/same-document Removal of H3K9me3 improves reprogramming efficiency in human somatic cell nuclear transfer experiments. Aberrant epigenetic reprogramming can cause developmental defects in somatic cell nuclear transfer embryos. Human-validated hard NEI NEI P (NEI) = 0.998; P (Support) = 0.001; P (Refute) = 0.001 Same-document evidence controls source/topic better than random irrelevant evidence while remaining insufficient.

Evidence cerpt Status Prediction Prob.

ex-

Why it matters

Table 35: Case study: same-document human-validated hard NEI.

Fixed-Claim Hard-Side NEI Recognition K.5

Field

Value

Dataset Claim

SciFact fixed-claim/same-document Cholesterol loading induces KLF4 expression in vascular smooth muscle cells, resulting in pro-inflammatory cytokine expression. Same-claim / same-document human-validated hard NEI Random irrelevant / 17 S UPPORT NEI P (reference) = 0.594; P (NEI | Ehard ) = 0.978 The model changes prediction when evidence changes while the claim is fixed, supporting hard-side insufficiency recognition.

Construction Train/seed Ref. pred. Hard pred. Prob. Why it matters

HoVer Candidate Missing-Hop Control

Field

Value

Dataset Claim

HoVer Brett Herron’s team competes in the Pro14 and the European Rugby Champions Cup. Missing-one-supporting-fact candidate control Candidate-only, not human-validated HoVer placeholder / 13 S UPPORT P (Support) = 0.981; P (NEI) = 0.000; P (Refute) = 0.018 Placeholder-trained behavior can fail on topically related missing-hop evidence.

Construction Status Train/seed Prediction Prob. Why it matters

Table 33: Case study: fixed-claim hard-side NEI recognition.

Table 36: Case study: HoVer candidate missing-hop control.

18

K.6

Unavailable Row-Level Cases

Case type

Unavailable reason

intervals where available, computed by resampling evaluation examples or groups and recomputing the metric.

SciFact hard-audit placeholder failure on humanvalidated BM25/cited hard NEI SciFact BM25/cited recovery case

Row-level model predictions for final hardNEI examples are unavailable; aggregate bootstrap tables exist only.

L.4

Row-level paired recovery predictions for placeholder versus BM25/cited hard NEI are unavailable in primary outputs.

For same-claim diagnostics, reference-label probability drop is:

Table 37: Unavailable case-study reasons. We do not invent row-level cases when prediction artifacts are missing.

L

∆i = Pθ (yref | ci , Eiref ) − Pθ (yref | ci , Eihard ). Probability-drop success is the fraction of pairs with ∆i > 0. Strict swap success additionally requires the reference-evidence side to be predicted as the reference label and the insufficient-evidence side to be predicted as NEI.

Statistical Testing and Uncertainty Estimation

This appendix defines metrics and uncertainty procedures for construction matrices, one-class hardNEI evaluation, fixed-claim diagnostics, and prediction coverage checks. L.1

Fixed-Claim Metrics

L.5

Prediction Coverage

Prediction coverage is:

Three-Way Verification Metrics

For standard verification, models predict: coverage = y ∈ {S UPPORT, R EFUTE, NEI}.

Rows with missing or duplicate predictions are not treated as complete.

We report accuracy, Macro-F1, class-specific F1, and especially NEI-F1. Construction-shift matrices emphasize NEI-F1 because the experimental manipulation changes the NEI evidence condition. L.2

Scope group

Expected

Predicted

Coverage

114 114 114 885

114 114 114 885

1.000 1.000 1.000 1.000

Fixed-claim reference side Fixed-claim insufficient side Same-document hard side SciFact matrix regression

One-Class Human-Hard Metrics

For human-validated hard-NEI subsets, all evaluated examples are adjudicated as NEI. Macro-F1 is therefore not informative. We report: NEI recall =

npredicted . nexpected

Table 38: Prediction coverage summary. Values summarize the repeated complete-coverage pattern across train variants and seeds.

#predicted NEI , #validated NEI

L.6

Npred=SUP , Nvalidated NEI Npred=REF False Refute = . Nvalidated NEI

False Support =

Reporting Rules

NEI-CAP follows these rules: report construction family for every NEI result; report Macro-F1 only when all three labels are meaningful; do not report Macro-F1 on one-class hard-NEI subsets; document prediction coverage before interpreting fixedclaim diagnostics; and mark FEVER/HoVer controls as bounded unless human validation is available.

We also report mean predicted probabilities for NEI, S UPPORT, and R EFUTE. L.3 Seed Aggregation and Bootstrap Intervals The primary construction matrix uses seeds 13, 17, and 23. Revision robustness and mixedconstruction experiments use seeds 13, 17, 23, 29, and 37. Where seed-level outputs are available, we report mean, standard deviation, minimum, maximum, and bounded intervals. Human validation and human-hard evaluations report bootstrap 95%

M

Reproducibility, Release, and Claim Boundary Checklist

This appendix records release readiness, lockedoutput policy, and paper-facing claim boundaries. 19

M.1

M.4

Release Audit

Check

Passed

Critical

Key reports exist

yes

yes

Paper tables exist and are non-empty Run registry entries exist

yes

yes

yes

yes

Fixed-claim labels are preserved No LLM labels marked human No candidate-only HoVer rows marked humanvalidated No Macro-F1 on one-class human subsets Primary SciFact matrix primary outputs unchanged External dependencies documented Known limitations documented Unavailable cases documented

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

no

yes

yes

yes

no

Claim

Strength

Allowed wording

Forbidden wording

NEI-CAP is constructionaware SciFact NEI is constructionsensitive Easy NEI inflates performance Human validation confirms hard subsets

Strong

Do not claim it solves fact verification. Do not claim all errors have the same cause. Do not claim every model uses only shortcuts. Do not call LLM triage human validation.

Same-claim is diagnostic

Limited

Protocol for constructing, auditing, and stresstesting NEI evaluation. Scores vary substantially across construction conditions. Shortcut-like constructions can inflate apparent competence. Audited BM25/cited and Fixed-claim/samedoc hard-NEI subsets are human-adjudicated. Supports hard-side recognition with reference-side caveats.

One-class hard-NEI evaluations exclude Macro-F1. Locked primary matrix is preserved. Upstream data dependencies documented. Limitations recorded.

HoVer is external control

Limited

FEVER is external control Hard NEI is heterogeneous

Limited

Missing row-level case reasons tracked.

Release is reproducible with limitations

Strong

Detail Experiment reports and summaries are present. Paper-facing tables are generated. Core experiment entries are tracked. Final fixed-claim/samedocument labels path exists. Reports use boundary language. HoVer marked human_validated=no.

Table 39: NEI-CAP release audit checklist. Critical checks concern result traceability, label provenance, metric validity, and claim boundaries.

Moderate

Strong

Moderate

Candidate-only multihop constructionsensitivity probe. Non-toy subset-level shortcut control. Different construction families stress different behaviors. Includes reproducibility instructions and claim boundaries.

Do not claim clean counterfactual evidence-use success. Do not call HoVer human-validated. Do not call FEVER full-data validation. Do not collapse topic-unrelated cases into hard NEI. Do not report toy, failed, or smoke outputs as paper-facing.

Revision-Stage Release Audit and Responsible Reporting

The additional experiments add response evidence without changing primary outputs or creating new human labels. The final audit records completed split statistics, mixed training, expanded multiseed checks, same-document artifact audit, humanprotocol documentation, no Macro-F1 on one-class hard subsets, no LLM-only human validation, and no candidate-only HoVer human validation. NEICAP is an audit and reporting protocol, not a blanket declaration that a dataset is invalid.

Locked Outputs and Traceability

Locked evidence includes the primary SciFact construction matrix, SciFact human-audit and hardNEI evaluation, fixed-claim/same-document adjudicated labels, secondary-backbone robustness, FEVER subset controls, HoVer candidate missinghop controls, and derived paper tables/figures. Each reported result should be traceable to a source dataset, construction family, split policy, model configuration, seed set, prediction artifact, and evaluation output.

M.3

Strong

Table 40: Claim strength and wording boundaries.

M.5

M.2

Claim Strength Matrix

Release Package Contents

The accompanying release will include construction manifests, group-disjoint split files, humanadjudicated labels, paper-facing prediction logs, evaluation scripts, artifact-audit scripts, and tablegeneration scripts. Upstream datasets are referenced and documented but not redistributed. Rows or case studies unavailable in primary outputs are documented rather than reconstructed from aggregate statistics. 20

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500