arXiv CSopen access

Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement

Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement

arXiv:2605.05103v1 [cs.CL] 6 May 2026

Nicholas S. Kersting [email protected] Oracle Corporation

Vittorio Castelli [email protected] Oracle Corporation

Xinzhu Wang [email protected] Oracle Corporation

Chieh Ting Yeh [email protected] Oracle Corporation

Saad Taame [email protected] Oracle Corporation

Abstract We introduce the **Concept Field** of a text corpus: a local drift field with pointwise uncertainty, estimated in sentence-embedding space from the deltas between consecutive sentences. Given a candidate sentence transition, we score its agreement with the field by ζ, the mean absolute z-distance between the observed delta and the field’s local Gaussian estimate. The score is black-box (no model internals), corpus-attributable (every score traces to nearby corpus sentences), and admits a direct probabilistic reading. We support the computation with the introduction of a **Vector Sequence Database (VSDB)** that stores embeddings together with sequence-position and next-delta metadata. We evaluate this approach on two large-scale settings: hallucination-style groundedness detection over the U.S. Code of Federal Regulations, and novelty detection over Project Gutenberg. Using controlled LLM-generated rewrites, Concept Fields achieve strong selective classification performance under a grounded / ungrounded / unsure triage policy, which unlike retrieval-centric baselines have similar coverage-risk behavior across both domains, supporting a probability-based interpretation that transfers across domains. We also sketch how divergence and curl of the Concept Field, computed on dense clusters, surface qualitatively meaningful semantic patterns (logic sources, sinks, and implicit topics), which we offer as hypothesis-generating rather than as a quantitative result. Concept Fields provide a fast, lightweight, and interpretable signal for groundedness and novelty, complementary to LLM-as-judge and whitebox detectors.

1

Motivation and Introduction

In practice, LLM outputs are often used where users care about factual support. The literature usually calls failures "hallucinations," but we focus on the broader notion of groundedness indicating whether an output is supported by a target corpus. This framing is useful for both compliance tasks (where deviation is bad) and creativity tasks (where deviation can be good). Many existing methods are either white-box (model-internal)[1, 2, 3], grey-box (token probabilities/perplexity)[4, 5, 6, 7], or black-box but calibration-heavy (e.g., LLM-as-judge, perturbation stability thresholds, cosine cutoffs)[8, 9, 10, 11]. These approaches can be expensive, architecture-dependent, or difficult to interpret operationally. Our approach, on the other hand, is fully black-box, corpus-grounded, and yields a directlyinterpretable probability metric. Let a corpus be represented as embedded sequences: {(t1 , . . . , tn ) | ti ∈ Σ} → {(v1 , . . . , vn ) | vi ∈ Rd }.

(1)

This is the raw data from which all language modeling starts. Auto-regressive modeling, in particular, focuses on predicting the next vector in the sequence given all the previous ones, i.e., finding the vn+1 which maximizes the conditional probability P(vn+1 |(v1 , v2 , . . . , vn )) which we can cast in a functional language via the probability density P over the next vector in the sequence given the last and all the previous vectors: P (vn+1 ) = f (vn ; (v1 , v2 , . . . , vn−1 ))

(2)

This general class of functions which use the full information contained in the raw data includes the transformer[12], which of course provides the most accurate language modeling to date but also suffers in speed/memory and loss of explainability. The next simplest model would be P (vn+1 ) = f (vn ; h)

(3)

where h represents one or more "hidden" states which compresses all the information in (v1 , v2 , . . . , vn−1 ), as is the case in RNN models such as LSTM, and finally P (vn+1 ) = f (vn )

(4)

where the probability of the next vector is dependent only on the last vector. This is actually just a vector field, and represents the most basic language modeling possible. In the case where the vectors represent token embeddings, Eqn. (4) is just a bi-gram probability density averaged over the whole corpus. In this work, rather than taking the vectors in Eqn. (4) to represent tokens, we have them represent whole sentences as per concept embedding models such as SONAR[13]; making the slight adjustment to have the vector field represent the probability density of a ‘delta’ to the next sentence vector, P (sn+1 − sn ) = f (sn )

(5)

we obtain a deterministic drift field defining a ‘flow’ in concept embedding space. As we obtain this field at any point from averaging over a distribution of corpus deltas, the field defines a local ‘direction’ to the next most likely concept given by the maximum of this distribution with uncertainty given its finite width; we might therefore say we are representing the corpus as a "concept vector field" which carries inherent pointwise uncertainty. Some concepts for example may be very broad and support a wide range of possible continuations in the corpus, whereas others may be more narrowly constrained. Now a sequence of sentences from an LLM, also being a sequence of concept embeddings, can be directly overlaid on this Concept Field, the statistical agreement between the two naturally accounted for by the component-wise mean absolute z-distance ζ between the sequence deltas and the vector field values; this carries a probability interpretation assuming a local working approximation of elliptical covariance.1 As a directly-traceable probabilistic measure, small ζ indicates local agreement with corpus flow; large ζ indicates departure. Figure 1 schematically illustrates our setup. While we use "hallucination" where needed for literature alignment, our primary object of consideration is groundedness. We will describe the usage of a Concept Field to detect clearly grounded vs clearly ungrounded corpus sample rewrites, introducing a practical storage/indexing layer, the Vector Sequence Database (VSDB), to support this computation at scale. We then demonstrate the method on a 2D toy system and on two large text corpora, one for factual/legal knowledge (CFR[14]), and one for general literature (Gutenberg[15]). Contributions: 1. Concept Field via VSDB. A corpus-level representation of sentence-sequence flow with local uncertainty, implemented through a VSDB. 2. Selective groundedness detection. ζ-based triage into grounded / ungrounded / unsure, with strong error rates on accepted cases. 3. Cross-domain use. The same mechanism supports both compliance-style groundedness checks (CFR) and novelty-style deviation checks (Gutenberg). 4. Corpus geometry. Qualitative evidence that divergence/curl extrema in dense regions of the Concept Field correspond to meaningful semantic trends. 1 Modern embedding spaces like SONAR are trained with smooth objectives, partially justifying this assumption. See

Appendix for further details.

2

Figure 1: General design: raw text is split into sentence sequences, embedded as concept vectors, and stored in a VSDB. The induced Concept Field supports groundedness scoring by mean absolute z-distance and exploratory geometric analysis.

2

Related Work

Black-box hallucination detection through self-consistency methods such as SelfCheckGPT [16], SAC3 [17], and FactSelfCheck [18] rely on multiple generations. By contrast, our method scores a single generated sequence against an external corpus field. Geometric output-space approaches [19, 20, 21, 22] are closest in spirit; our key difference is corpus-wide multi-step flow with explicit local uncertainty. Other works consider sequence comparison in embedding space, including DTW-style methods [23, 24, 25, 26] which are fundamentally pairwise. In contrast, our approach compares a query against a field induced by the whole corpus. Chamfer-style matching [27, 28] and learned sequence distance models [29] are complementary. Geometry/topology analysis of embeddings includes work on manifold/topological structure [30, 31]. Our contribution differs by modeling directed sentence transitions and using divergence/curl as exploratory corpus diagnostics. Finally, our work is based on the use of SONAR [13] (and note OmniSONAR [32]) with FAISS [33]; field interpolation uses IDW [34].

3

Vector Sequence Database (VSDB)

A VSDB stores variable-length sequences of fixed-dimensional vectors V[1] , V[2] , . . . , with V[k] = {v1 , . . . , vn }, vi ∈ Rd . Examples in d=1 might include stock ticker time series, single-track musical scores, and seismographs. Spatial trajectories are good examples of 2- or 3-d vector sequences, which provide our pedagogical example in Sec. 4, and for the remainder of this paper vectors are high-dimensional sentence embeddings and sequences are paragraph-level sentence chains. Regardless of the specific application, for each stored vector we keep sequence-local metadata:  
V SDB ≡ vi , (IDj , pi,j , ∆i,j ) | vi ∈ V[q]j , (6)  vp+1 − vp , p < m ∆i,j = (7) null, p=m where p is the position in sequence j labeled by IDj . 3

This design supports: 1. fast nearest-neighbor lookup (we use FAISS[33]), 2. sequence retrieval/reranking (ID frequency + Chamfer cosine-distance between sequences), 3. local delta aggregation (0th-order field estimate summing the deltas for each vector), 4. IDW interpolation for Concept Field estimation from nearby anchors (1st-order field estimate from Euclidean weighting) We use a minimal implementation with heuristic settings sufficient for this study. IDW interpolation, in particular, uses simple inverse 1/d weighting with cutoffs for nearest-neighbor count (topN ) within distance (dmax ). To compute the z-distance deviation metric of a delta D against from the IDW Concept Field value µ, we take a component-wise average over the largest k = topNζ components of the latter: k 1 X Di − µi . (8) ζ(D) = k i=1 σ̃i This gives freedom to adjust sensitivity to random, semantically uncorrelated directions which invariably arise in high-dimensional embedding space. We address tuning of these hyperparameters in Section 5.

4

Toy Example: 2D Ballistics

The principles and implementation details of computing a vector field from a VSDB are best illustrated in a low dimensional example. Here we consider the textbook physics treatment of ballistic trajectories in 2D with no air resistance and constant gravity pulling downwards. Supposing we launch a series of projectiles from the origin (x, y) = (0, 0) with the same initial speed but at launch angles varying uniformly from 0 (i.e., just skimming the ground) to 90 (straight upwards), see Fig. 2a. Recording the position of each projectile at uniform time intervals, we obtain a 2D vector sequence [(0, 0), (x1 , y1 ), ..., (xn , yn )] describing each trajectory from the time of launch to its last position before hitting the ground. Figure 2b shows graphically what some of these vector sequences look like, which of course fall on parabolic paths. We will store these in a VSDB to enable efficient testing of a query trajectory discretized in the same manner. √ In this simulation we fix a value of gravity at g = 9.81 and launch speed v = 2g (MKS units understood in this and the following), confining the projectiles to a region in the x-y plane with 0 <= x <= 2 and 0 <= y <= 1 , regardless of launch angle. We simulate each trajectory with a discrete time-step Euler-method based on the first order differential equations y ′ (t) = v sin θ − gt ′ and x′ (t) = v cos θ, defining launch angle θ = arctan xy ′ (0) (0) and using time steps of 0.001 which results in a sufficiently long vector sequence for most angles. We generate 1000 trajectories with launch angles uniformly spread between 0 and 90 degrees, and use these sequences to compute the underlying IDW vector field of the trajectories at any point by averaging over the instantaneous velocities of the topN neighbor points lying on other trajectories. The parameters of this setup include N=10, a nearest-neighbor distance cut-off of dmax = 0.03, and simple inverse 1/d weighting, though probably similar values would also work. Sampled on a grid with spacing 0.05, this gives the blue vector field shown in Fig. 2c. Note that near the origin where many trajectories cross (refer to Fig. 2b) the vector field rapidly changes in a noisy fashion, a hallmark of the inherent uncertainty in that region. This uncertainty fades away towards the edge of the kinematic envelope (also a parabola) where all trajectories tend to flow together. Aside from visual inspection, we could also quantitatively identify this ‘laminar’ region of the vector field by its minimal divergence and curl. Likewise, the origin (0,0) is a point of maximal divergence as all trajectories point away from it by construction. Notice that beyond the kinematic envelope the field is not defined by virtue of our finite value of dmax , and we would say the field is ‘out of corpus’ here. Fig. 2c also shows sampled velocity vectors of a query trajectory (the red arrows) along its path, and we can compare this with the underlying vector fields’ values, using Eqn. 8 (with k = 2) to test statistical agreement. We compute ζ = 0.277, indicating statistical agreement at the nominal ∼ 78% level. Note here we do not attain 100% from the irreducible uncertainty in the trajectory passing 4

through the inherently noisy region near the origin, and we can take it as ‘good enough’ agreement with the underlying vector field (it is common practice in experimental physics, for example, to take exclusion thresholds closer to 3σ ≈ 0.1% or higher). Now for a query trajectory generated with additional physics (we add air resistance) in Fig. 2d, we see the trajectory departs from a parabolic shape and thus expect statistical disagreement; indeed we measure ζ = 133, corresponding to essentially 0% statistical agreement with the underlying vector field, so we may infer different generating physics. Possible physical applications aside2 , the usefulness of this toy example is to provide a direct analogy to the more complex situation with higher-dimensional text embeddings: in constructing a VSDB from sequences of natural language sentence embeddings, we will encounter the same concepts of the IDW vector field and its geometrical features, as well as ζ of a query sequence as a proxy for agreement with the corpus Gestalt.

(a) Projectile setup with launch angle θ.

(b) Sample of the trajectories generated with base physics used to construct VSDB.

(c) IDW field comparison of trajectory with same physics.

(d) Comparison of a trajectory with air resistance.

Figure 2: Corpus vector field for 2D ballistics.

5

The Concept Field of a Text Corpus

The construction of a high-dimensional text corpus Concept Field will proceed in a similar manner to the 2D example we looked at above with several additional complications: now the ‘trajectories’ consist of sequences of English sentences, possibly of varying length, contained in paragraphs, requiring sentence and paragraph splitting; such trajectories are very sparsely distributed in 1024-d 2 Although this is just a toy example, one may imagine a realistic scenario in, say, space exploration, where scientists observe trajectories of debris emitted from an exoplanetary volcano and from the shapes of such try to infer geo-atmospheric conditions such as local gravity, air resistance, and wind speed by reference to trajectories from similar volcanoes on Earth.

5

space; the IDW field we considered cannot be constructed on a grid but rather on a point-by-point basis: it may be highly irregular and insufficiently supported in most of space. Despite these caveats, we will show the procedure still makes sense and yields practical results. We first check whether the field interpolates plausibly between nearby sequences. Creating a VSDB from two sentence sequences, one a more intense version of the other, we compare: 1. The pure average embedding decoded by SONAR, and 2. IDW-field drift starting from the same initial point. In Figure 3 we show one instance of this exercise (the Appendix contains additional ones). As we see, both produce coherent intermediate semantics, supporting the practical use of both SONAR and local IDW field interpolation.

Figure 3: Interpolation check. Left/right columns are mild/intense source sequences; middle columns show SONAR-decoded embedding averages and IDW-field drift outputs. As another check, on a realistic corpus (CFR 2016) field-following from an in-corpus sentence remains close to the source sequence while local support is strong; once support weakens, significance (measured by the average ratio of the delta to its standard deviation) drops and drift becomes unstable: see Figure 4.

Figure 4: Field-following examples. Top: start from an in-corpus CFR sentence and track a supported drift. Bottom: start from an arbitrary sentence and follow the local field where defined. Significance values are in white. Finally, we performed several sanity checks on the approximate Gaussian distribution of deltas of nearest semantic neighbors in a VSDB which the reader may refer to in the Appendix. 5.1

Testing Groundedness

Now as our main experiment, we evaluate two different uses of ζ for groundedness: 1. CFR groundedness (deviation is "hallucination"): we consider the last 26 years of the publicly-available CFR data (comprising about 70M sentences/1B tokens) as a source of U.S. legal truth. Encoding each block of sentences as a vector sequence, we build a VSDB sharded by year and test whether LLM-reworded pairs3 of neighboring sentences are reliably 3 It is for efficiency’s sake that we test pairs of sentences rather than entire sequences. In practice of course the latter are

built of the former and one can aggregate ζ.

6

identified as either faithful to the original meaning or not based on ζ of these sentence pairs alone. 2. Gutenberg groundedness (deviation is "novelty"): we encode the english language portion of the Gutenberg 75k corpus (comprising about 57k books/5B tokens) as a source of literature. Again we test LLM-reworded pairs of neighboring sentences to see if large ζ reliably identifies the creative aspect of the sentences. In both settings we use LLM-reworded sentence pairs as controlled stress-test data, with human spot-checking for quality control. This dataset is intentionally not a benchmark from the wild; it is designed to test whether ζ separates clearly grounded from clearly ungrounded transformations across very different corpora in the same fashion. 5.1.1

CFR Groundedness (Hallucination-Style Detection)

As the CFR is a year-by-year corpus, with laws in one year overriding all laws in previous years, we therefore focus on testing laws from individual years. Randomly choosing 10 years between 2000-2025 as a validation set and the rest for testing, we use an LLM (GPT-OSS-120B) to rewrite a random selection of 1k neighboring sentence pairs from each year as either a grounded example (keeping semantics intact) or ungrounded example (changing semantics significantly). Appendix 6 has details and examples of these transformations. The same LLM is used to post-filter sentence pairs that don’t meet the (un)groundedness criteria and human spot-checking is done at all stages to check quality of this procedure, which disqualified about 25% of the generated examples. We then check each sentences pair’s delta (i.e., the difference of the embeddings s2 − s1 ) against the corpus Concept Field value as measured at s1 , expressing the deviation via Eqn. 8. Full description of the experimental steps, including tuning of they hyperparameters (p, topN, topNζ , dmax ) on the Validation Set, are described in the Appendix. While the core usage of ζ is a probability of unambiguous groundedness, useful for example within complex agentic workflows that compound a risk score from multiple interactions, it is also useful as a simple classifier of hallucination by setting user-defined thresholds for clearly grounded examples (ζ < ζlow ), ungrounded examples (ζ > ζhigh ) and everything else "unsure." Which values one wishes to use depend on use case, but coverage (defined by the percentage of examples assigned a positive or negative label) may differ accordingly. For example, in Figure 5a below we show how the F1 score and coverage for hallucination detection in the CFR Test Set (similarly generated on the other 16 years) varies depending on what we designate as unsure within the [ζlow , ζhigh ] interval. Since for hallucination the cost of a False Negative is usually much higher than a False Positive, a choice such as (ζlow , ζhigh ) = (1, 3) corresponding roughly to thresholds of (32%, 99.7%) for hallucination probability, gives R, F 1, AU C > 96% with about 56% coverage (44% of the examples would fall in the unsure category). However, as noted no choice is forced and one can just use this technique with the probability interpretation intact as a feed into standard risk analysis. 5.1.2

Gutenberg Novelty Detection

A very similar analysis applies to this use case, with however ungroundedness being a ‘good’ thing: a sentence pair with a positive label (for novelty) will thus have a large ζ with respect to the corpus field (or even be ‘out of corpus’). Of the roughly 57k English works in the Gutenberg data, we use half (28k) for validation and the rest for testing. The Appendix again contains more details, but we note that the hyperparameters of our setup optimize to the same values used for the CFR; this, together with the very similar coverage/F1 tradeoff heatmap in Figure 5b, provides strong evidence for corpus agnosticity of our technique. 5.1.3

Baseline Comparisons

In considering competitive baselines to using the Concept Field, we intentionally discount LLMbased techniques on the basis of their being typically much more expensive, slower, and inherently untrustworthy. Comparing only to baselines which are equivalently fast, inexpensive to deploy, and corpus-attributable, we settle on a few classifiers in Table 1, applied to the CFR hallucination detection: the first two remove the Concept Field and just use the nearest neighbor in the VSDB using either Euclidean (L2) or cosine as distance metrics between the deltas. The third is a vanilla VDB chunked on all pairs of sentences in the corpus, so one is comparing embeddings of sentence 7

Figure 5: Threshold sensitivity heat maps for (a) Hallucination detection in the CFR, and (b) Novelty detection in the Gutenberg corpus. Despite being very different corpora, the coverage-performance trade-off is similar (AURC= 0.03 and 0.08, respectively).

Table 1: Comparison of the Concept Field to various baselines with thresholds (lo,hi) tuned to give maximum performance at 50% coverage of the original set of CFR 15k+15k pos+neg test samples. Precision is defined on the ungrounded "hallucination" examples. AURC is computed over all possible lo/hi thresholds. Here we see VSDB-based techniques generally outperform a vanilla VDB solution. Model lo hi TP TN FP FN P R F1 MCC VSDB + Field 1.0 3.75 10479 3974 316 152 0.971 0.986 0.978 0.923 VSDB + top1(L2) 0.04 0.15 10917 4045 293 148 0.973 0.987 0.980 0.929 VSDB + top1(cos) 0.01 0.14 7557 7218 66 566 0.991 0.930 0.960 0.920 VDB + top1(cos) 0.01 0.13 3566 10600 6 1185 0.998 0.751 0.857 0.821

pairs with cosine distance to determine groundedness. Choosing the optimal performance at 50% coverage" point for comparison, we note that the VDB-based technique generally performs the worst, capturing the smallest number of Positive (hallucination) examples, and tends to err quite heavily on the False Negative side – this is in keeping with the well-known deficiency of cosine distance to capture negation inside a larger context (see Appendix for examples). VSDB-based techniques, on the other hand, err in the other direction by having slightly more False Positives — but for risk-averse applications this is actually preferable. Otherwise the VSDB-based techniques all have comparable performance. However, only the Concept Field technique retains robustness across different corpora due to its directly interpretable probability-based thresholds (see for example very similar performance trends in Fig 5a-b); Euclidean L2 or cosine-distance based techniques must always be calibrated on a corpus-by-corpus basis since the scale of these metrics changes with the scope and specific embedding of the corpus data. We empirically verify this is the case computing the corresponding 50% coverage performances for the Gutenberg Test Set (see Table 10 in Appendix) where the Concept Field thresholds and performance is about the same, while the other baselines vary wildly. 5.2

Geometric Features

Now that we have seen how a corpus of embedding sequences gives rise to a vector field, the question arises as to geometric features of this vector field with semantic implications. Divergence and curl, for example, are frequently used to describe interesting regions of a vector field. For example, in familiar examples of vector fields in 2D and 3D such as wind maps, one may be interested in convergent regions (high absolute divergence and low curl), laminar regions (low divergence and curl) and vortices (high curl), representing regions of wind sources/sinks, consistent fronts, and circular flow. 8

AUC 0.984 0.986 0.973 0.919

AURC 0.039 0.037 0.040 0.036

The same geometrical measures of divergence and curl are interesting for high dimensional vector fields in semantic embedding space as well, as there are very tangible semantic implications. Densely populated regions of high divergence, for example, indicate some concept that all other logic flows tend to start at (logic sources) or end at (logic sinks). In regions of low divergence and curl, the vector field is highly linear, that is vector sequences in this region all tend to point in the same direction, indicating ‘common reasoning’. We are instinctively familiar with this as certain sentences in common speech are almost always continued in a parallel fashion, e.g., “Excuse me?” -> “Yes?” and “Can I ask a question?” -> “Sure.”. Regions of high curl, on the other hand, are semantically interesting as regions where vectors flow around some central concept, literally ‘beating around the bush’. This may indicate avoidance regions in the VSDB where certain topics are conspicuously missing; and for an LLM trained on the data in such a VSDB, it would also indicate regions where the LLM is likely to hallucinate for lack of training data in that region. − − → − → → − → Although in principle one could compute divergence ▽ · f and curl ∇ × f from the IDW-vector → − field f computed on a grid, this is not feasible in a high number of dimensions. Instead, we work with the raw sequences in the VSDB to give an approximation in regions where there is enough data to support it. Focusing on the densest semantic clusters of the VSDB where statistics are strongest, we may compute the following: 1. Divergence Analysis: → −T→ −

− vi is the delta to • Compute the total estimated divergence as D = Nd Σi v|ri i |r2i where → → − − − −c ri = → pi − → the ‘next’ vector of pi , the concept embedding of the i-th sentence, and → → is the radial vector pointing from the centroid to the i-th concept embedding with −c being the position of the cluster centroid. – The clusters with large positive D have flow away from the centroid, indicating some ‘logic source’, i.e., common starting points or memes – The clusters with large negative D indicate flow towards the centroid, indicating ‘logic sinks’, i.e., common ending points or conclusions 2. Curl Analysis: −T → −→

−T → −→

• Compute the ‘Total Angular Momentum’ of the cluster as M = Nd Σi ri vi |r−i |v2i ri

• The Frobenius Norm ||M|| of the matrix M is proportional to the total circulation about the centroid. • Clusters with large ||M|| exhibit the most circular flow about their centroid, which may thus represent an ‘implicit topic’. • The clusters with both ||M|| ∼ 0 and D ∼ 0 indicate flow in one direction across the cluster, indicating ‘logic flows’, i.e., common concept-concept logic In this paper, these geometric findings are qualitative and hypothesis-generating; representative examples are provided in the Appendix.

6

Discussion and Conclusions

We have introduced the Concept Field of a text corpus in this work, a conceptually simple yet powerful method to test groundedness with respect to the corpus providing both a probabilistic score and explainability/traceability to the supporting pieces of the corpus. Beyond what we exhibited as hallucination and novelty detection of LLM outputs, the technique enjoys numerous other applications we did not have space to address, such as prompt injection defense, corpus-corpus comparison for conflicts, compliance testing, and general corpus assessment based on geometric features such as common logic flow, holes, and singularities. Future work will address these. As the present default go-to for hallucination detection is LLM-as-judge, we argue that testing groundedness with a Concept Field becomes a serious competitor, if not an essential accessory, in particular being a technique that does not depend on the blackbox internals of an inherently non-explainable entity. On a practical level as well, a corpus Concept Field is faster (once a VSDB is indexed), cheaper (no API token usage) and lightweight (commodity CPU suffices, and one can choose to store the field values separately from the origin sequences, giving lightweight portability), and enjoys explainability (every concept vector is related to its nearest corpus sentence). Concepts 9

are also language agnostic, so one corpus Concept Field simultaneously fulfills the same role for testing LLMs specialized for various world languages. Our treatment has been abbreviated and basic by intention, leaving the research community room to improve on it in various aspects. In particular, one can explore other concept embedding schemes besides SONAR and better interpolation schemes than simple IDW. Some aspects of our approach are fundamentally data-dependent: clearly with more data, the Concept Field fills space more densely and covers more semantic ground, and the quality of data likewise determines how efficiently this happens: corpora based on internally logically consistent content such as Congressional Law fill out embedding space more reliably, while noisier data from the free Internet is probably far less efficient. We imagine that as this technique gains favor in the community, people and institutions will cultivate vetted self-consistent corpora and their associated Concept Fields for purposes of testing compliance with law, ethics, and accepted world knowledge — such are useful not only for testing the outputs of LLMs, curbing the ubiquitous hallucination problem, but also any sequence of concepts (even from humans or other corpora). Even in this initial form, the approach is practically useful as a complementary trust signal alongside other safeguards, and is applicable beyond text wherever data can be represented as vector sequences.

APPENDIX A. SONAR embeddings Clearly our agenda is dependent on the choice of embedding scheme: a poorly-trained embedding will not exhibit meaningful correlations between distance in embedding space and semantics. We therefore explicitly check that SONAR, our choice for this work, is a reasonable scheme. The first way to check this is to define a linear interpolation Z between sequences X and Y: Z = αX + (1 − α)Y , and check semantic stability for 0 < α < 1. We do in fact explicitly test across a number of sentence pairs that taking the linear interpolation between the two sentences (concept-embeddings) in each pair we obtain intermediate semantics: • X : ’I like to run in the park.’ • Y : ’I like to walk in the library.’ • alpha=0.00: Z: ’I like to run in the park.’ • alpha=0.25: Z: ’I like to run in the park.’ • alpha=0.50: Z: ’I like to walk in the park.’ • alpha=0.75: Z: ’I like to walk in the library.’ • alpha=1.00: Z: ’I like to walk in the library.’ • X: ’The food at this restaurant is terrible and the service is slow.’ • Y: ’The cuisine at this eatery is excellent and the service is prompt.’ • alpha=0.00: Z:’The food in this restaurant is terrible and the service is slow.’ • alpha=0.25: Z:’The food in this restaurant is terrible and the service is slow.’ • alpha=0.50: Z:’The food in this restaurant is excellent and the service is slow.’ • alpha=0.75: Z: ’The cuisine at this eatery is excellent and the service is quick.’ • alpha=1.00: Z: ’The cuisine at this eatery is excellent and the service is prompt.’ A similar conclusion occurs for generating points randomly in between the two vectors, i.e., Zi = (Xi + Yi )/2 + d/2 ∗ ŝ, with d ≡ |X − Y | : • X: ’The meticulous planning for the ambitious cross-continental journey ensured every potential obstacle was identified and systematically mitigated, leading to an utterly smooth and successful outcome.’ • Y: ’The haphazard preparation for the challenging cross-continental journey failed to account for several major logistical hurdles, which inevitably resulted in chaotic delays and an ultimate failure of the expedition.’ 10

• Z: (1% closer to Y) ’The meticulous preparation for the pernicious intercontinental journey was not aware of any substantial obstacle arrangements, which were consequently timeless and resulted in the ultimate failure of the expedition.’ B. Embedding Perturbation Stability of SONAR (1024-d) As a quick sanity check, we perturb the ’delta’ between a pair of sentences in a Gaussian fashion and decode with SONAR. Original sentence pair • s1: “The annualized and present value estimates of monetized costs and benefits over the 10-year period from 2023 through 2032 using three percent and seven percent discount rates are summarized below.” • s2: “The annualized, monetized costs (2020 USD) of the provisions in the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7.7 to $87 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are largely the same.” Table 2: Noise statistics for 10 perturbations in 1024-dimensional embedding space. idx 1 2 3 4 5 6 7 8 9 10

noise1_l2 0.03114 0.03157 0.03166 0.03284 0.03216 0.03212 0.03134 0.03228 0.03229 0.03111

noise2_l2

noise1_mean

noise2_mean

noise1_std

noise2_std

0.03295 0.03202 0.03173 0.03225 0.03104 0.03159 0.03248 0.03077 0.03206 0.03160

−5

−6

0.000972 0.000986 0.000988 0.001026 0.001005 0.001004 0.000979 0.001008 0.001009 0.000972

0.001030 0.001000 0.000991 0.001008 0.000970 0.000987 0.001014 0.000961 0.001001 0.000986

−4.92 × 10 −3.70 × 10−5 5.13 × 10−5 1.18 × 10−5 3.29 × 10−5 −1.64 × 10−5 −1.43 × 10−5 2.64 × 10−5 −9.69 × 10−6 1.18 × 10−5

−5.47 × 10 2.71 × 10−5 8.61 × 10−7 1.52 × 10−5 1.49 × 10−5 2.52 × 10−5 2.68 × 10−5 −1.74 × 10−5 4.23 × 10−5 −4.65 × 10−5

Decoded perturbed sentences (all 10) 1. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The monetized, annualized costs (2020 dollars) of the provisions in the NPRM (if finalized as proposed) are estimated to be $29 million (range: $77–$87 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are largely the same.” 2. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three to 7 per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $77–$87 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” 3. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three to 7 per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7.7 million to $8.7 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” 4. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are 11

summarized below.” s2: “The monetized, annual $20 million costs (provisions in the NPRM) if finalized (as proposed) are estimated to be $29 million: $7.70 million (from $7.87 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are largely the same.” 5. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $77–8.7 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” 6. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7.7 million to $8.7 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are largely the same.” 7. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The monetized, annual costs (2020 dollars) of the provisions in the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7.7 to $8.7 million) using a three percent discount rate; the estimated monetized costs with seven percent discount rate are largely the same.” 8. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $77–$87 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” 9. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7.7 million to $8.7 million) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” 10. s1: “The annualized and present value estimates of monetised costs and benefits over the 10-year period from 2023 to 2032 using three per cent and seven per cent discount rates are summarized below.” s2: “The annualised, monetized costs (2020 dollars) of the provisions of the NPRM (if finalized as proposed) are estimated to be $29 million (range: $7,700 to $8,700) using a three percent discount rate; the estimated monetized costs using a seven percent discount rate are roughly the same.” Similar experiments with several noise levels up to 0.1 likewise yielded expected increasing perturbations of the semantics. C. Local Gaussian Calibration Check We evaluated whether the local transition statistics used by our score are well calibrated under an independent per-coordinate Gaussian noise model. For an anchor sentence embedding x, we retrieve its local neighbors from the vector-sequence database and collect the corresponding transition deltas: δj = xj+1 − xj . 12

(9)

Neighbors are filtered by a maximum distance threshold and weighted by inverse-distance weighting: wj ∝

1 , (dj + ϵ)p

(10)

where dj is the distance from the anchor to neighbor j. We choose p = 1 in this work and ϵ = 10−8 , up to a maximum distance of dmax = 0.3. For each anchor, the collected local deltas are randomly split into a training set and a held-out test set. On the training set, we estimate a coordinate-wise local Gaussian model: µi =

X

wj δj,i ,

(11)

wj (δj,i − µi )2 .

(12)

j

σi2 =

X j

A small floor σmin is applied for numerical stability: σ̃i = max(σi , σmin ).

(13)

For each held-out delta δjtest , we compute the absolute standardized residual: zj,i =

test δj,i − µi . σ̃i

(14)

This directly matches the score used in the main experiments, which averages absolute per-coordinate standardized deviations: k

ζ(D) =

1 X Di − µi . k i=1 σ̃i

(15)

Here k is the number of selected embedding coordinates. We then check whether the held-out standardized residuals are calibrated as draws from a standard normal distribution. For central coverage level q ∈ {0.50, 0.80, 0.95}, the expected threshold is:   1+q tq = Φ−1 , (16) 2 where Φ is the standard normal CDF. The empirical coverage is: ĉq = Pr (|Z| ≤ tq ) ,

(17)

estimated over held-out coordinates and held-out deltas. An anchor is counted as passing if: |ĉ0.50 − 0.50| < 0.05,

(18)

|ĉ0.80 − 0.80| < 0.05,

(19)

|ĉ0.95 − 0.95| < 0.03.

(20)

We report the percentage of anchors passing, the median empirical coverage errors across anchors, and the number of anchors skipped due to insufficient local deltas. 13

Algorithm 1 Local per-coordinate Gaussian calibration check Require: VSVDB database, anchors A, neighbor count N , dimension count k, distance threshold r Require: Train fraction α, IDW exponent p, standard deviation floor σmin 1: for each anchor x ∈ A do 2: Retrieve top-N neighbors of x 3: Collect neighbor deltas δj = xj+1 − xj 4: Keep only neighbors with distance dj ≤ r 5: if too few deltas remain then 6: Mark anchor as skipped 7: continue 8: end if 9: Select k embedding coordinates, using the largest |xi | coordinates 10: Randomly split deltas into train and test sets 11: Compute IDW weights wj ∝ (dj + ϵ)−p on train deltas 12: for each selected coordinate i do P 13: Estimate µi = j wj δj,i P 14: Estimate σi2 = j wj (δj,i − µi )2 15: Set σ̃i = max(σi , σmin ) 16: end for 17: for each held-out delta δjtest and coordinate i do test 18: Compute zj,i = (δj,i − µi )/σ̃i 19: end for 20: for q ∈ {0.50, 0.80, 0.95} do 21: Set tq = Φ−1 ((1 + q)/2) 22: Estimate ĉq = mean[1{zj,i ≤ tq }] 23: end for 24: Mark anchor as passing if all coverage errors are below tolerance within error bars 25: end for 26: return fraction of anchors passing and median coverage errors

Applying to only a 1k-book sample of the Gutenberg corpus, for example, we find the median empirical coverages ĉq to be (0.46 ± 0.05, 0.74 ± 0.05, 0.82 ± 0.03), which is not very far from Gaussian except on the tails where there are slightly more events than expected. Increasing sample size improves results overall, e.g., for a 28k-book sample of the same corpus, we get coverages of (0.44 ± 0.05, 0.74 ± 0.05, 0.90 ± 0.03), so we expect Gaussian shape is the limiting behavior for larger corpora. D. Additional Sentence Interpolations

Figure 6: Interpolation check. Left/right columns are mild/intense source sequences; middle columns show SONAR-decoded embedding averages and IDW-field drift outputs.

14

Figure 7: Interpolation check. Left/right columns are mild/intense source sequences; middle columns show SONAR-decoded embedding averages and IDW-field drift outputs. E. Fundamental issue with cosine similarity Consider the following pair of very similar sentences which in fact only differ by two words: • "However, additional funding may be available in Fiscal Year 2001 to award a limited number of grants under the College and University Affiliations Program to enable current and former Fulbright scholars to build on their experiences as individual Fulbright grantees through broadened institutional cooperation." • "However, additional funding may very well be available in Fiscal Year 2001 to award a limited number of grants under the College and University Affiliations Program to enable current and former Fulbright scholars to build on their experiences as individual Fulbright grantees through broadened institutional cooperation." The cosine similarity of their SONAR embeddings is 0.96. If we negate one of them, • "However, additional funding may be available in Fiscal Year 2001 to award a limited number of grants under the College and University Affiliations Program to enable current and former Fulbright scholars to build on their experiences as individual Fulbright grantees through broadened institutional cooperation." • "However, additional funding may not be available in Fiscal Year 2001 to award a limited number of grants under the College and University Affiliations Program to enable current and former Fulbright scholars to build on their experiences as individual Fulbright grantees through broadened institutional cooperation." the similarity is not only still high, at 0.98, but even higher than the affirmative case! This illustrates a well-known problem in cosine-similarity measures, especially on small but semantically important changes within larger blocks of text. F. Groundedness in the CFR Raw CFR XML-encoded files are available on a year-by-year basis [14]. As paragraph boundaries are hard to detect in this corpus, we choose to chunk by every 5 sentences. We use spacy[35] for sentence splitting and normalization. • Experimental Procedure – positive examples 1. extract pairs of sequential sentences from CFR sequences 2. reword with LLM to change slightly; use LLM-as-judge with human spot-checking to keep only pairs that do not significantly change semantics; * : Generative Prompt: “Generate a new second sentence that preserves the meaning of the original second sentence while making only a slight wording change. - Keep the new second sentence very close to the original wording and avoid broad paraphrases, extra details, or a no-op copy.” 15

* : Quality-check Prompt: “Accept only if: - the rewritten second sentence is coherent with the first sentence - it preserves the meaning of the original second sentence - it makes only a slight wording change rather than a broad paraphrase - it is not identical to the original second sentence Reject if any of those are not true.” 3. check ζ of each re-worded pair 4. for each re-worded pair, find closest pair (by similarity with first sentence) in VSDB, and its ζ ; if ζ > ζhigh , consider this pair ‘too noisy’; i.e., in actual inference application one would not trust this region of the Concept Field, tantamount to the noisy region of the ballistic plots in our 2D example. 5. compare test sequence ζ with that of closest sequence: if they are both less than ζlow , mark as True Negative (for hallucination). If ζ > ζhigh , False Positive. Otherwise "unsure" (this lowers coverage). – negative examples 1. extract pairs of sequential sentences from CFR sequences (possibly same as above) 2. reword with LLM to change second sentence significantly; use LLM-as-judge with human spot-checking to keep only sequences that have significantly changed semantics; * : Opposite-type example · : Generative Prompt: “Generate a new second sentence that expresses the opposite or a clear contradiction of the original second sentence. - Make the contradiction clear in the new second sentence without changing the first sentence." · : Quality-check Prompt: “Accept only if: - the rewritten second sentence is coherent with the first sentence - it clearly expresses the opposite or a contradiction of the original second sentence - it is not identical to the original second sentence Reject if any of those are not true.” : * Warped-type example · : Generative Prompt: “Generate a new second sentence that radically warps the meaning second sentence while keeping it related. - Make the warping clear in the new second sentence without changing the first sentence." · : Quality-check Prompt: “Accept only if: - the rewritten second sentence is coherent with the first sentence - it clearly expresses a warped version of the original second sentence - it is not identical to the original second sentence Reject if any of those are not true.” 3. check ζ of each re-worded pair 4. for each re-worded pair, find closest pair (by similarity with first sentence) in VSDB, and its ζ; if ζ > ζhigh , consider this pair ‘too noisy’. 5. compare test sequence ζ with that of its closest sequence: if they are both less than ζlow , mark as False Negative (for untruthfulness). If ζ > ζhigh , True Positive. – Compute F1 Score * P = TP/(TP + FP) * R = TP/(TP + FN) * F1 = 2P*R/(P + R) – Other metrics (MCC, AUC, Cross Entropy) follow standard procedure. – We also compute Area under the Risk Curve (AURC), measuring the trade-off between error-rate and coverage (percent of examples not in the "unsure" bucket). Tables 3,4 show examples of our positive and negative generated examples. We chose 10 years of the CFR as a validation set: 2000, 2001, 2005, 2008, 2011, 2013, 2017, 2020, 2022, and 2025. Choosing 1k positive and 10k negative example pairs from each year, we compared observed deltas with measurements of the IDW Concept Field for that year, with hyperparameter choices for the number of nearest neighbors (topN ), number of top components compared for ζ calculation (topNζ ), and maximum neighbor distance (distmax ). Below in Table 5 is an example analysis for one set of hyperparameter choices (here we choose ζlow = ζhigh = 3): 16

Ground Truth Sequence ["A discussion of work to be performed during the next reporting period.","Between scheduled reporting dates the grantee(s) also shall immediately inform the Grant Officer’s Technical Representative (GOTR) of significant developments affecting their ability to accomplish the work."] ["We believe that these proposals for enhanced local access to OCA information would help to realize the benefits of public disclosure of OCA information identified in the benefit assessment and would help satisfy the public’s interest in access at the local level to information about the sources of chemical accident risks that could affect them directly.","We anticipate that members of the public seeking OCA information held by LEPCs and local fire departments would be more likely to ask about the other information available from LEPCs under EPCRA regarding chemical hazards in the community."] ["We see no special concerns warranting an exception to this policy.", "Based on these results we conclude that this waste does not pose risk to human health and the environment at levels that warrant listing."]

Transformed Positive Sequence ["A discussion of work to be performed during the next reporting period.", "Between scheduled reporting dates, the grantee(s) must also promptly notify the Grant Officer’s Technical Representative (GOTR) of any significant developments that affect their ability to complete the work."] ["We believe that these proposals for enhanced local access to OCA information would help to realize the benefits of public disclosure of OCA information identified in the benefit assessment and would help satisfy the public’s interest in access at the local level to information about the sources of chemical accident risks that could affect them directly.","We anticipate that members of the public seeking OCA information held by LEPCs and local fire departments would be more likely to inquire about the other information available from LEPCs under EPCRA regarding chemical hazards in the community."] ["We see no special concerns warranting an exception to this policy.", "Based on these results, we conclude that this waste does not pose a risk to human health and the environment at levels that warrant listing."]

ζGT 4.66 · 10−6

ζtransf 1.04

1.07 · 10−5

0.48

3.59

3.72

Table 3: Examples of Ground Truth sequences from CFR transformed with an LLM to very similar sequences, which are labeled as Positive for groundedness. The first two rows show typical examples from this procedure, while the last row shows a rejected example where the Ground Truth is in too noisy a region of concept embedding space. This is typical when the sentence concept is very simple and has many different continuations in the corpus.

As we can see there is little variation between the performance on any given year, showing the stability of the technique. Ablation analysis over suitable combinations of these hyperparameters yielded the following results in Table 6 compiled over all years of the validation set: G. Creative departure from the Gutenberg corpus • Experimental Procedure – as creativity assistant, essential difference from hallucination: we are looking for confirmation that an idea flow is not in the corpus, something LLM cannot do – negative examples 1. extract pairs of sequential sentences from Gutenberg sequences 2. reword with LLM to change both sentences slightly; use LLM-as-judge with human spot-checking to keep only pairs that do not significantly change semantics; * : Generative Prompt: “You will be given a sentence pair. Rewrite both sentences with slight wording changes while preserving their meanings. Requirements: Preserve the meaning of the first sentence. - Preserve the meaning of the second sentence. - Make only slight wording changes rather than broad paraphrases. Do not leave either sentence identical to the original." * : Quality-check Prompt: “Accept only if: - the rewritten first sentence preserves the meaning of the original first sentence - the rewritten second sentence preserves the meaning of the original second sentence - both rewritten sentences remain coherent with each other as a pair - both are only slight wording changes 17

Ground Truth Sequence ["(7) Describe the language capabilities of staff proposed for this study.", "The proposal clearly describes the (1) qualifications, commitment, and epidemiologic skills and experience of the project director and his/her ability to devote adequate time and effort to provide effective leadership; (2) qualifications and experience of other staff involved in the project to accomplish the proposed activity, and their commitment and time they will devote; (3) successful experience the project director and staff have in managing, coordinating and conducting similar or related projects; (4) a study coordinator with epidemiologic training and experience who is able to devote at least 50 percent of his or her time to this project; and (5) facilities, space, and equipment necessary for conducting the project."] ["The need to ensure an adequate supply of generation usually was met through requirements imposed by states on franchise utilities to build or buy adequate power resources to meet demand consistently.", "Today, however, in states such as California, the adequacy of local power resources depends, not just on state requirements, but also on whether market prices are sufficient to elicit adequate supplies, through construction or otherwise."] ["Because the Exchange administers the Pilot Fee Structure as part of its rules, the Commission requests that the Exchange provide within 45 calendar days a thorough description of each fee that is permissible under the Pilot Fee Structure.", "The description should clearly identify the circumstances in which a distribution intermediary may assess a particular fee."]

Transformed Negative Sequence ["(7) Describe the language capabilities of staff proposed for this study.","The proposal fails to describe any qualifications, commitment, or epidemiologic skills of the project director or staff, provides no evidence of their experience, does not identify a study coordinator with epidemiologic training, and omits information about the necessary facilities, space, or equipment."]

ζGT 4.29 · 10−6

ζtransf 14.85

["The need to ensure an adequate supply of generation usually was met through requirements imposed by states on franchise utilities to build or buy adequate power resources to meet demand consistently.","Today, in states such as California, the adequacy of local power resources depends solely on state requirements, with market prices playing no role in securing sufficient supplies."]

2.57 · 10−5

5.05

["Because the Exchange administers the Pilot Fee Structure as part of its rules, the Commission requests that the Exchange provide within 45 calendar days a thorough description of each fee that is permissible under the Pilot Fee Structure.","The description must detail the mysterious, almost mythical circumstances that allow a distribution intermediary to conjure and impose a fee at will."]

4.49 · 10−8

6.14

Table 4: Examples of Ground Truth sequences from CFR transformed with an LLM to very different sequences, labeled as Negative for groundedness. The first two rows show typical examples of simple negation from this procedure, while the last row shows a more warped transformation.

rather than broad paraphrases - neither rewritten sentence is identical to its original sentence Reject if any of those are not true.” 3. check ζ of each re-worded pair: if ’out of corpus’ or ’insufficient statistics’, mark as False Positive. 4. for re-worded pair, find closest pair (by similarity with first sentence) in VSDB, and its ζ ; if ζ > ζhigh , consider this pair "too noisy" 5. compare test sequence ζ with closest sequence: if they are both less than ζlow , mark as True Negative (for novelty). if ζ > ζhigh , False Positive. – positive examples 1. extract pairs of sequential sentences from Gutenberg sequences (possibly same as above) 18

Year

TP

TN

FP

FN

P

R

F1

MCC

AUC

LogLoss

2000 2001 2005 2008 2011 2013 2017 2020 2022 2025

618 583 619 625 578 585 605 495 572 585

885 872 846 835 833 835 855 824 793 828

61 62 58 85 66 60 71 68 89 81

138 174 174 128 159 161 163 134 149 148

0.91 0.90 0.91 0.88 0.90 0.91 0.89 0.88 0.87 0.88

0.82 0.77 0.78 0.83 0.78 0.78 0.79 0.79 0.79 0.80

0.86 0.83 0.84 0.85 0.84 0.84 0.84 0.83 0.83 0.84

0.76 0.72 0.73 0.74 0.72 0.73 0.72 0.72 0.70 0.72

0.94 0.92 0.93 0.93 0.93 0.93 0.93 0.93 0.93 0.93

1.37 1.41 1.35 1.55 1.45 1.32 1.41 1.57 1.46 1.45

Total

5865

8406

701

1528

0.89

0.79

0.84

0.73

0.93

1.43

Table 5: Year-by-year results for parameter choice (topN=10, topNζ =10, distmax =0.3) on the Validation Set. topn

dmax

topNζ

explanation

TP

TN

FP

FN

P

R

F1

MCC

AUC

CE Loss

5 10 50 10 10 10

0.1 0.3 0.5 0.3 0.3 0.3

10 10 10 3 50 1024

small neighborhood defaults larger neighborhood reduced sensitivity medium sensitivity full sensitivity

558 4946 4953 4560 5087 5095

359 2645 2654 3246 2438 2366

33 238 235 434 124 120

8 84 87 258 55 58

0.944 0.954 0.955 0.913 0.976 0.977

0.986 0.983 0.983 0.946 0.989 0.989

0.965 0.968 0.969 0.929 0.983 0.983

0.912 0.911 0.912 0.833 0.948 0.947

0.975 0.979 0.979 0.949 0.989 0.989

0.804 0.680 0.675 1.150 0.459 0.440

Table 6: Ablation across parameters of the IDW averaging scheme on the Validation Set for groundedness in the CFR. Here only sentence pairs with ζ less than 0.5 or greater then 3 are accepted, all others being too "unsure." The inverse power p was set to 1 for all tests as preliminary analysis indicated essentially no difference from p > 1.

2. reword with LLM to change first sentence slightly and second sentence radically; use LLM-as-judge with human spot-checking to keep only sequences that have significantly changed semantics; * : Generative Prompt: “You will be given a sentence pair. Rewrite both sentences. Requirements: - Rewrite the first sentence with only a slight wording change while preserving its meaning. Do not leave it the same as the original. - Generate a new second sentence that radically warps the meaning." * : Quality-check Prompt: “Accept only if: - the rewritten first sentence preserves the meaning of the original first sentence - the rewritten first sentence is only a slight wording change rather than a broad paraphrase - the rewritten sentence is different from the original first sentence Reject if any of those are not true.” 3. check ζ of each re-worded pair: if ’out of corpus’ or ’insufficient statistics’, mark as True Positive. 4. for re-worded pair, find closest pair (by similarity with first sentence) in VSDB, and its ζ ; if ζ > ζhigh , consider this pair "too noisy" 5. compare test sequence ζ with closest sequence: if they are both less than ζlow , mark as False Negative (for novelty). if ζ > ζhigh , True Positive. – Compute F1 Score * P = TP/(TP + FP) * R = TP/(TP + FN) * F1 = 2P*R/(P + R) – Other metrics (MCC, AUC, Cross Entropy) follow standard procedure as before. As for creativity detection, we generate negative examples by lightly adjusting the ground truth sequences, as shown below in Table 7. For creativity detection, we generate positive examples by radically transforming the first sentence, as shown below in Table 8. An ablation analysis similar to that for the CFR above yields similar conclusions: 19

Ground Truth Sequence ["In returning now to the adventures of Siegfried there is little more to be described except the finale of an opera.",Siegfried, having passed unharmed through the fire, wakes Brynhild and goes through all the fancies and ecstasies of love at first sight in a duet which ends with an apostrophe to ’leuchtende Liebe, lachender Tod!’, which has been romantically translated into ’Love that illumines, laughing at Death,’ whereas it really identifies enlightening love and laughing death as involving each other so closely as to be usually one and the same thing."] ["While teachers may be real authorities in subject-matter, they can never be anything more than assistants in the self-development of their students.","They should more openly assume this subordinate position, placing the primary responsibility upon the learner; they would then be less likely to subordinate the inner growth of the student, which it is their highest function to aid, to the mere acquisition of knowledge."] ["The moments passed.", "I heard the whistle of the approaching train."]

Transformed Negative Sequence ["Returning now to Siegfried’s adventures, there is little else to describe beyond the opera’s finale.", "Siegfried, having emerged unscathed from the flames, awakens Brynhild and experiences the whims and raptures of love at first sight in a duet that concludes with an apostrophe to ’leuchtende Liebe, lachender Tod!’, romantically rendered as L̈ove that illumines, laughing at Death,’ though it actually denotes that enlightened love and laughing death are so intertwined that they are often regarded as the same."]

ζGT 1.95 · 10−5

ζtransf 0.618

["Although teachers can be genuine experts in their subject, they are at most helpers in their students’ selfdevelopment.", "They ought to more openly adopt this subordinate role, putting the main responsibility on the learner; then they would be less inclined to subordinate the student’s inner growth—its highest purpose to support - to mere knowledge acquisition."]

4.21 · 10−5

1.08

["The moments went by.", "I heard the whistle from the approaching train."]

6.20

1.45

Table 7: Examples of Ground Truth sequences from the Gutenberg Corpus transformed with an LLM to very similar sequences, labeled as Negative examples for creativity. The first two rows show typical examples from this procedure, while the last row shows a rejected example where the Ground Truth is in too noisy a region of embedding space.

H. Threshold Sensitivities on Corpora As mentioned in the main text, the Concept Field technique enjoys a certain stability across different corpora whereas other threshold-dependent techniques must be calibrated on a case-by-case basis. I. Divergence and Curl Analysis Examples: • high positive divergence (logic source) in English Premier League Commentary[36] – group size: 20 P – (cos(t)/r2 ) = −18.24, Avg = −18.24 – centroid semantic: “Can either side find a winner to the end?” – example sentences in cluster: * “Can either side snatch a lead just ahead of half-time? ” −→ EOS * “Can either side find a winning goal, or are we set for a stalemate? ” −→ EOS * “Can either side now find a late winning goal? ” −→ EOS • high negative divergence (logic sink) in English Premier League Commentary – group size: 283 P – (cos(t)/r2 ) = −14679.69, Avg = −51.87 – centroid semantic: “GOOOOAAALL!! It is” – example sentences in cluster: * “GOALLLLLL! ” −→ “WHAT A START FOR BURNLEY! 1-0!” 20

Ground Truth Sequence ["There are some things that may not be discussed directly, and the conduct of life at a modern university–which is a reflection of life in the greater world– is one of these.", "Perry Blackwood and Ham did most of the talking, while Ralph, characteristically, lay at full length on the window-seat, interrupting with an occasional terse and cynical remark very much to the point."] ["The most popular works of fiction, such as leave nothing to our imagination.", "And to this craving after prose, who would not be lenient, that has at all known life, with its usual predominance of our lower and less courageous selves, our constant hankering after the cosey closed door and line of least resistance?"]

Transformed Positive Sequence ["Behind the polished façade of campus life, hidden truths fester in silence, turning the modern university into a clandestine arena where society’s unspoken experiments unfold.","Perry Blackwood and Ham did most of the talking, while Ralph, characteristically, lay at full length on the window-seat, interrupting with an occasional terse and cynical remark very much to the point."] ["Even the most obscure, censored tomes insist on dictating every detail of our thoughts.", "And to this craving after prose, who would not be lenient, that has at all known life, with its usual predominance of our lower and less courageous selves, our constant hankering after the cosey closed door and line of least resistance?"]

ζGT 4.21 · 10−5

ζtransf 3.14

5.07 · 10−5

4.59

Table 8: Examples of Ground Truth sequences from the Gutenberg Corpus transformed with an LLM to very different sequences, labeled as Positive for creativity. topn

dmax

ζtopN

explanation

TP

TN

FP

FN

P

R

F1

MCC

AUC

CE Loss

10 10 10

0.3 0.3 0.3

10 50 1024

reduced sensitivity medium sensitivity full sensitivity

534 570 179

114 83 170

90 67 60

14 0 13

0.856 0.895 0.749

0.974 1.00 0.932

0.911 0.944 0.831

0.631 0.704 0.675

0.888 0.947 0.874

0.542 0.367 0.762

AUC 0.958 0.999 0.999 0.999

AURC 0.0.08 0.001 0.001 0.001

Table 9: Ablation across parameters of the IDW averaging scheme on the Validation Set for groundedness in the Gutenberg Corpus. Here only sentence pairs with ζ less than 1.0 or greater then 1.5 are accepted, all others being too "unsure." We did not vary other parameters of the setup as we believe the CFR experiment established the best values.

* “GOOOOOOAAAAAAAALL! ” −→ “NEWCASTLE TAKE THE LEAD THROUGH SCHAR!” “WHAT A GOAAALLLLLLLLLLLLLLLL! ” −→ “UNITED STRIKE FIRST!” * • high positive divergence (logic source) in CFR – group size: 10 P – (cos(t)/r2 ) = −82.54, Avg = −11.79 – centroid semantic: “The specification of complementarity field GPLD-1 in reference to the claimed geomagnetic performance standard TMP-07 includes an 8°C reference field limited in consideration of spatial telemetry reference magnitude of the relevant geomagnetic field; an 8°C reference field for the field, and an 8°C reference field for

Table 10: Comparison of the Concept Field to various baselines with thresholds (lo,hi) tuned to give maximum performance at 50% coverage of the original Gutenberg set of 9k+9k pos+neg test samples. Precision is defined on the ungrounded "hallucination" examples. AURC is computed over all possible lo/hi thresholds. Here we see the distance-based techniques enjoy a high performance on this artificial test set but with very different threshold values. Also the Concept Field technique captures many more Positive examples. Model lo hi TP TN FP FN P R F1 MCC VSDB + Field 1.2 3.7 4158 2366 332 401 0.926 0.912 0.919 0.785 VSDB + top1(L2) 0.08 0.28 2449 5464 2 0 0.999 1.00 0.999 0.999 VSDB + top1(cos) 0.01 0.21 2880 4505 3 0 0.999 1.00 0.999 0.999 0.02 0.19 1068 6662 0 1 0.999 1.00 0.999 0.999 VDB + top1(cos)

21

the field in consideration of the information and simulation variability of the reference field.” – example sentences in cluster: * “The supplemental GMD event definition in proposed Reliability Standard TPL007-2 contains a non-spatially-averaged reference peak geoelectric field amplitude component of 12 V/km, in contrast to the 8 V/km figure in the spatially-averaged benchmark GMD event definition. ” −→ “As NERC explains in its petition, the supplemental GMD event will be used to “represent conditions associated with localized enhancement of the geomagnetic field during a severe GMD event for use in assessing GMD impacts.”” “With respect to the calculation of the reference peak geoelectric field amplitude * component of the benchmark GMD event definition, Order No. 830 expressed concern with relying solely on spatial averaging in Reliability Standard TPL-007-1 because “the use of spatial averaging in this context is new, and thus there is a dearth of information or research regarding its application or appropriate scale.”” −→ “NERC states that proposed Reliability Standard TPL-007-2 enhances currentlyeffective Reliability Standard TPL-007-1 by addressing reliability risks posed by GMDs more effectively and implementing the directives in Order No. 830.” “The supplemental GMD event definition contains a higher, non-spatially-averaged * reference peak geoelectric field amplitude component than the benchmark GMD event definition (12 V/km versus 8 V/km). ” −→ “These three new requirements largely mirror existing Requirements R4, R5, and R6 that currently apply, and would continue to apply, only to benchmark GMD vulnerability and transformer thermal impact assessments.” • high negative divergence (logic sink) in CFR – group size: 6155 P – (cos(t)/r2 ) = −17763.35, Avg = −34.43 – centroid semantic: “Comments to submit are to be received by or before February 18, 2019.” – example sentences in cluster: * “Written comments and information are requested and will be accepted on or before July 23, 2018.” −→ “Interested persons are encouraged to submit comments using the Federal eRulemaking Portal at” “Rebuttal comments should be submitted by October 26, 2018.” −→ “Although * there do not appear to be any issues relevant to approval or disapproval which would be facilitated by an oral presentation of views, data, and arguments, the Commission will consider, pursuant to Rule 19b-4, any request for an opportunity to make an oral presentation.” “Comments and related material must be received by the Coast Guard on or before * July 9, 2018.” −→ “You may submit comments identified by docket number USCG-2018-0580 using the Federal eRulemaking Portal at ” • high curl (implicit topic) in Gutenberg – group size: 708042 – ||M|| = 1095.98 – centroid semantic: “The Commission is aware of the difficulties encountered by the Member States in implementing the guidelines.” – example sentences in cluster: * “And now we will leave the earth and look at the pictures of airships and the wonderful planes, which, although improvements are made every day, are already capable of flying with an almost incredible speed and with a security which only a little while ago would have been considered quite impossible.” −→ “Today air travel is not only a possibility but a realized fact, and it is difficult to realize that it has occurred in the last twenty years, and that before then the practical flying machines were unknown.” * “In these matters the Balance of Power is not less vital for international life and for the evolution of true cosmopolitan ideals than it is in mere Politics.” −→ “And if 22

we stand up in a battle for the lesser races it is not merely because they are small and need defence but because an element of right, a part of the civilization we intend to conquer, is with them and is part of their heritage.” “Therefore, a negotiated solution for the final status of Jerusalem could be of a * character different from that of the rest of the West Bank.” −→ “The Gaza Strip is currently governed by Israeli military authorities and Israeli civil administration; it is American policy that the final status of the Gaza Strip will be determined by negotiations between the parties concerned; these negotiations will determine how this area should be governed.”

(a) Part of a highly negatively divergent region of the Gutenberg Corpus.

(b) A region of the CFR where div and curl are minimal, hence indicating common logic flow.

(c) A region of the CFR with high curl.

Figure 8: Examples of corpus regions with extreme divergence and/or curl, shown in 2D UMAP plots.

References [1] T. Doan. Hallucination mitigation for faithful retrieval-augmented generation. Aaltodoc (Thesis), 2025. Introduces MAESTRO, a hybrid strategy combining "Decoding by Contrasting Layers" (DoLa) with attention-based "Over-trust Penalty and Retrospection-Allocation" (OPERA) to reduce hallucinations in RAG. [2] Yining Wang, Mi Zhang, Junjie Sun, et al. Mirage in the eyes: Hallucination attack on multimodal large language models with only attention sink. arXiv preprint arXiv:2501.15269, 2025. While focusing on attacks, this paper details white-box mitigation through the manipulation of "attention sinks" and hidden embeddings to ensure image-text relevance in multimodal models. [3] Yue Zhang, Yafu Li, Leyang Cui, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv/MIT Press Direct, 2025. A foundational survey that taxonomizes white-box challenges, specifically noting how a model’s "vague knowledge boundary" and internal states complicate the explanation and mitigation of errors. [4] Yihao Xue, Kristjan Greenewald, Youssef Mroueh, and Baharan Mirzasoleiman. Verify when uncertain: Beyond self-consistency in black box hallucination detection. arXiv preprint arXiv:2502.15845, 2025. Contrasts black-box self-consistency with white-box methods that utilize token-level logits and intermediate representations (referencing prior work like Varshney et al. 2023 and Zou et al. 2023). [5] Y. Xu et al. Factselfcheck: Fact-level black-box hallucination detection for llms. ACL Anthology (Findings of EACL), 2026. Provides a 2026 update on the white-box vs. black-box divide, noting that white-box methods analyze internal states (e.g., Azaria and Mitchell 2023) to achieve more "universal" detection across different model architectures. [6] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024. A landmark paper defining the "grey-box" approach of using token-level probabilities to cluster responses by meaning rather than syntax. 23

[7] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Detecting hallucinations in LLMs. In International Conference on Learning Representations (ICLR), 2023. The predecessor to the 2024 Nature paper; focuses on using entropy over the final logits to quantify epistemic uncertainty. [8] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. [9] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Kong Ruigi, and Chenguang Chen. GEval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12432–12448, 2023. Key reference for using multi-step reasoning (CoT) in the evaluation prompt. [10] Anu Pradhan, Alexandra Ortan, Apurv Verma, and Madhavan Seshadri. LLM-as-a-judge: Rapid evaluation of legal document recommendation for retrieval-augmented generation. In ResearchGate / Bloomberg Technical Reports, 2025. Analyzes Inter-Rater Reliability (IRR) between human legal experts and LLM judges in RAG pipelines. [11] Nicholas S. Kersting, Mohammad Rahman, Suchismitha Vedala, and Yang Wang. Harmonic llms are trustworthy, 2024. [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 5998–6008, 2017. [13] Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. SONAR: Sentence-level multimodal and language-agnostic representations. arXiv preprint arXiv:2308.11466, 2023. [14] US Government. Code of federal regulations. https://www.govinfo.gov/bulkdata/CFR/. [15] Project Gutenberg. Gutenberg corpus. https://www.kaggle.com/datasets/ lokeshparab/gutenberg-books-and-metadata-2025. [16] Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource blackbox hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9004–9017. Association for Computational Linguistics, 2023. [17] Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Kumar Sricharan. SAC3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [18] Y. Xu et al. FactSelfCheck: Fact-level black-box hallucination detection for LLMs. Findings of the European Chapter of the Association for Computational Linguistics (EACL), 2026. arXiv preprint arXiv:2503.17229. [19] Emanuele Ricco et al. Hallucination detection: A probabilistic framework using embeddings distance analysis. arXiv preprint arXiv:2502.08663, 2025. [20] CERT Research. Geometric hallucination detection via directional consistency in embedding space. https://cert-framework.com/docs/research/dc-paper, 2026. January 2026. [21] CERT Research. A geometric taxonomy of hallucinations in LLMs. arXiv:2602.13224, 2026.

arXiv preprint

[22] others. Geometric uncertainty for detecting and correcting hallucinations in LLMs. arXiv preprint arXiv:2509.13813, 2025. [23] Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. In IEEE Transactions on Acoustics, Speech, and Signal Processing, volume 26, pages 43–49, 1978. 24

[24] Xiao-Ying Liu, Yi-Ming Zhou, and Rong-Shu Zheng. Sentence similarity based on dynamic time warping. In Proceedings of the International Conference on Semantic Computing (ICSC), pages 250–256. IEEE, 2007. [25] Guangyou Zhu and Carlos A. Iglesias. Measuring similarity of academic articles with semantic profile and joint word embedding. Tsinghua Science and Technology, 22(6):619–632, 2017. [26] others. Similarity measure of time series based on angle-distance penalized metric dynamic time warping. Scientific Reports, 2025. [27] Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39–48, 2020. [28] Laxman Dhulipala et al. MUVERA: Multi-vector retrieval via fixed dimensional encodings. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. [29] others. General neural embedding for sequence distance approximation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025. [30] Alexander Jakubowski, Milica Gasic, and Marcus Zibrowius. Topology of word embeddings: Singularities reflect polysemy. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), pages 103–113. Association for Computational Linguistics, 2020. [31] Ada Uchendu et al. Unveiling topological structures in text: A comprehensive survey of topological data analysis applications in NLP. arXiv preprint arXiv:2411.10298, 2025. Updated June 2025, surveying 95 papers. [32] OmniSONAR Team, João Maria Janeiro, et al. Omnilingual SONAR: Cross-lingual and crossmodal sentence embeddings bridging massively multilingual text and speech. arXiv preprint arXiv:2603.16606, 2026. [33] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, PierreEmmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. arXiv preprint arXiv:2401.08281, 2024. Revised October 2025. [34] Donald Shepard. A two-dimensional interpolation function for irregularly-spaced data. Proceedings of the 1968 23rd ACM National Conference, pages 517–524, 1968. [35] . Spacy. https://spacy.io/. [36] Pranav Karnani. English premier league - match commentary. https://www.kaggle.com/ datasets/pranavkarnani/english-premier-league-match-commentary?select= 23_24_match_details.csv.

25