Beyond Accuracy: Measuring Logical Compliance of Predictive Models Guillaume Delplanque, Pierre Genevès, Nabil Layaïda, Zephirin Faure
arXiv:2606.20208v1 [cs.AI] 18 Jun 2026
June 19, 2026 Abstract Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.
1
Introduction
Machine learning algorithms are commonly evaluated through predictive performance metrics: the central question is usually how closely their predictions match a reference ground truth. This focus extends even to advanced systems that explicitly exploit structured knowledge, including graphbased neuro-symbolic models [6, 28] and relational learning systems [11, 3]. These systems are still most often assessed through the prism of predictive accuracy, using task-specific metrics such as mean reciprocal rank (MRR) and Hits@k [2] for link prediction, mean absolute error (MAE) [23] and R2 [16, 5] for regression, or accuracy and precision [17] for classification [20, 18]. However, these metrics effectively quantify predictive performance, but they often fail to assess whether model outputs satisfy logical or domain-specific constraints. In many high-stakes applications, including healthcare, finance, cybersecurity or autonomous systems, respecting logical consistency and operational rules can be as important as, or even more important than maximizing predictive accuracy alone. A model achieving high accuracy may generate outputs that violate critical constraints, potentially leading to unreliable or unsafe decisions. In this paper, we introduce Rule Violation Score (RVS), a new evaluation metric designed to quantify the extent to which a predictive model respects predefined logical rules independently of its predictive accuracy. We show that two predictive models with similar accuracy scores can exhibit significantly different levels of RVS, highlighting important behavioral differences that remain 1
invisible when relying solely on conventional performance metrics. Our results suggest that RVS provides complementary information to traditional evaluation metrics and should be considered when assessing AI systems deployed in constraint-sensitive environments.
1.1
Motivating example
Consider the dataset D consisting of the three facts shown in Table 1 and the link-prediction query wife(Alice, ?), which asks for Alice’s spouse. Suppose that two predictive models, Model 1 and Model 2, each return a single candidate completion for this query. Known facts (D) Ground-truth answer Prediction of Model 1 Prediction of Model 2
mother(Alice, Bob) father(Christopher, Bob) sister(Alice, Daniel) wife(Alice, Emmanuel) wife(Alice, Christopher) wife(Alice, Daniel)
Table 1: Motivating example Both predictions are incorrect according to standard metrics. However, Model 2 violates the hard constraint ⊥ ← sister(x, y) ∧ wife(x, y), which states that two individuals cannot simultaneously be siblings and spouses, and is assumed to admit no exception. Model 1, by contrast, remains logically consistent. Existing metrics do not capture this distinction. This example illustrates that some predictions should be penalized not only for being incorrect but also for violating symbolic domain knowledge. It motivates the need for evaluation metrics that complement accuracy by measuring the extent to which predictions respect domain knowledge.
1.2
Desirable properties
We identify four desirable properties for logical compliance metrics: (P1) Ground-truth independence. A logical compliance metric should be computable without access to the ground-truth labels used by standard predictive metrics. Its purpose is not to determine whether a prediction is correct, but whether it complies with the rule set under consideration. This makes it possible to distinguish between predictions that are merely wrong and predictions that are also logically inconsistent. (P2) Hard/soft rule distinction. We distinguish two types of logical rules. Hard rules represent constraints that must always hold, whereas soft rules represent statistical regularities, which constitute hints for predictive models. For example, the following rule stating that two people cannot simultaneously be sisters and spouses: ⊥ ← sister(x, y) ∧ wife(x, y) can be seen as hard, whereas the following rule can be seen as soft: wife(x, z) ← mother(x, y) ∧ father(z, y), since co-parenthood does not necessarily entail marriage. A metric should distinguish these two categories. Hard rule violations should be reported and treated as logical failures. Soft rules, by contrast, may already be imperfectly satisfied in the data, so their violations should be evaluated differently. (P3) Dataset-aware comparability. For soft rules, the metric should take into account how often the rule is contradicted in the observed dataset. A model should not be penalized in the same way for contradicting a rule that is almost always satisfied in the data and for contradicting a rule that is frequently contradicted by the data itself. Instead, soft-rule compliance should be assessed relative to the dataset baseline: predictions contradicting a rule less often than the data are more compliant, while those contradicting it more often are less compliant. This also makes scores comparable across models for a given rule. 2
(P4) Model- and data-agnostic computability. The metric should not depend on a specific dataset, predictive model, or rule set. Given observed facts, model predictions, and logical rules, it should be automatically computable by a generic, reusable implementation.
2
Preliminaries
We first introduce the formal notions used throughout the paper, starting with the evaluation setting on which the proposed metric is defined. b R) where D is a finite set of Evaluation setting. An evaluation setting is a triple E = (D, D, b is a finite set of predictions produced by a model, and R is a finite set of logical observed facts, D rules whose compliance is evaluated. b and R are expressed over a common relational vocabulary Data model. We assume that D, D, P. The vocabulary P consists of predicate symbols with fixed arities. If p(n) ∈ P, then p is a predicate symbol of arity n. We distinguish variables from constants. Variables are taken from a set V. Constants are taken from a domain C and may denote entities, labels, or typed numerical b ∪R). values. The set of constants occurring in the evaluation setting is denoted by C = Const(D ∪ D A term is either a variable x ∈ V or a constant c ∈ C. An atom over P is an expression of the form p(t1 , . . . , tn ) where p(n) ∈ P and each ti is a term. A ground atom is an atom whose terms are all constants: p(c1 , . . . , cn ), where c1 , . . . , cn ∈ C. A fact is a ground atom. Both observed facts in D b are therefore ground atoms over the same vocabulary P and the same and predicted facts in D domain of constants C. This representation covers several prediction settings, e.g., link prediction (wife(Alice, Christopher)), node classification (label(v, c)), or attribute regression (price(h, 250)).
Logical rules. We consider logical knowledge expressed as evaluable rules over P. Let V be a set of variables. A rule r ∈ R is written r : ψr ← ϕr where ϕr is the body condition and ψr is the head condition. Both ϕr and ψr are Boolean formulas over P, possibly including relational atoms, Boolean connectives, numerical comparisons, arithmetic or aggregate expressions over finite sets, and user-defined decidable predicates. Let Vϕr and Vψr denote the variables occurring in the body and head conditions, respectively. We require that Vψr ⊆ Vϕr and then define the variables of r as Vr = Vϕr . A positive Horn rule is the special case where ϕr = B1 ∧ · · · ∧ Bm and ψr = H, where the Bi and H are atoms. Groundings. A grounding of r is a function θ : Vr → C. The set of all groundings of r over the constants of the evaluation setting is Θr = {θ | θ : Vr → C}. Applying θ to r yields a ground rule instance ψr θ ← ϕr θ. For a finite set of facts F and an evaluable formula φ, we write F |= φθ when the grounded formula φθ evaluates to true over F . In particular, for a relational ground atom A, confirmation holds by membership: F |= A iff A ∈ F . For a rule r, the set of groundings whose body is satisfied in F is denoted by: BrF = {θ ∈ Θr | F |= ϕr θ}. Body evaluability assumption. RVS is intended to be computable over finite evaluation settings. We therefore impose the following assumption on the rules: rule bodies must be evaluable in finite time. Formally, for every rule r : ψr ← ϕr , every finite set of facts F considered in the evaluation, and every grounding θ ∈ Θr , the truth value of ϕr θ over F must be computable in finite time. This ensures that the set of body-satisfying groundings BrF is well-defined and computable.
3
Confirmed and refuted heads. For each body-satisfying grounding θ ∈ BrF , the grounded head ψr θ may be either confirmed, refuted, or left undecided by the evaluation semantics. It is confirmed if the evaluation setting establishes that ψr θ is true, and refuted if it establishes that ψr θ is false. We define CrF = {θ ∈ BrF | ψr θ is confirmed in F } and VrF = {θ ∈ BrF | ψr θ is refuted in F }. Confirmed groundings satisfy the rule, refuted groundings violate the rule, and other groundings are ignored. We define ErF = CrF ∪ VrF . Refutation mechanisms. Whether a ground atom is true or false depends on the refutation mechanism available. For example, in closed-world settings, a ground atom may be refuted by its absence from the fact set. However, in open-world settings, absence alone is not sufficient to establish falsity. Refutation must then be provided explicitly, for example through negative facts, incompatibility constraints, auxiliary rules, functionality constraints, or user-defined decidable predicates. For example, an incompatibility constraint ⊥ ← brother(x, z) ∧ aunt(x, z) can be used to refute the head brother(x, z) whenever aunt(x, z) is observed. Such auxiliary rules, that may be provided by domain experts, specify when the head of a target rule should be considered false in the evaluation setting. We define Rref as the set of auxiliary refutation rules. Each auxiliary rule rref ∈ Rref refuting the head ψr of a target rule has the form ⊥ ← ψr ∧ B1 ∧ · · · ∧ Bm where the Bi are atoms.
3
Rule Violation Score (RVS) Definition
b R) be an evaluation setting. For each r ∈ R, RVS computes dr , the contradiction Let E = (D, D, rate in D, and pr , the prediction-induced contradiction rate relative to D.
3.1
Dataset contradiction rate
For a rule r ∈ R, the dataset contradiction rate dr measures the fraction of body-satisfying groundings in D that lead to an established contradiction: dr =
|VrD | |ErD |
whenever |ErD | > 0; otherwise dr is undefined. dr is used to partition the rule set into hard and soft rules. By default, r is hard iff dr = 0 and soft otherwise, although users may explicitly declare a rule hard despite observed contradictions. Let Rhard and Rsoft denote the corresponding sets.
3.2
Prediction contradiction rate
b so that contradictions Predictions are evaluated individually, via DA = D ∪ {A} for each A ∈ D, can be associated with specific predictions. For a grounded rule instance rθ, let Atoms(rθ) = Atoms(ϕr θ) ∪ Atoms(ψr θ) denote the ground atoms occurring in ϕr θ or ψr θ. Prediction A partici-
4
pates in θ iff A ∈ Atoms(rθ). We define: BrD,A = {θ ∈ Θr | DA |= ϕr θ, A ∈ Atoms(rθ)}, CrD,A = {θ ∈ BrD,A | ψr θ confirmed in DA }, VrD,A = {θ ∈ BrD,A | ψr θ refuted in DA }, ErD,A = CrD,A ∪ VrD,A . The prediction contradiction rate pr is defined as follows whenever the denominator is non-zero; otherwise pr is undefined: P D,A | b |Vr pr = PA∈D D,A b |Er | A∈D
3.3
Rule-level score
Hard rules.
For r ∈ Rhard ,
RVShard = r
X
|VrD,A |
b A∈D
. The optimal value is 0. For r ∈ Rsoft ,
Soft rules.
RVSsoft = r
pr dr
Values below 1 indicate better compliance than the dataset, 1 indicates identical behavior, and values above 1 indicate worse compliance. Example 1. If dr = 0.01 and pr = 0.03, then RVSsoft = 3: predictions contradict the rule three r times more often than the dataset.
3.4
Global aggregation
Hard rules.
Given user-defined importance weights σr >= 0, RVS
whenever
hard
=
r∈Rhard σr > 0, otherwise RVS
P
hard r∈Rhard σr · RVSr
P
hard
P
r∈Rhard σr
,
is undefined.
Soft rules. We define Wr as a normalization factor corresponding to the fraction of evaluable prediction-induced groundings associated with rule r. The aggregate soft-rule score is then defined as: P
Wr = P
b A∈D
r′ ∈R
|ErD,A |
RVSsoft =
, D,A b |Er′ | A∈D
P
X
Wr · RVSsoft r
r∈Rsoft
Values below 1 indicate better compliance than the dataset, values above 1 indicate worse compliance.
5
3.5
RVS computation
RVS is computed by translating the quantities dr and pr into relational queries over a database b with one table per predicate. For each rule r, the body-satisfying groundings are storing D and D enumerated, the evaluable ones identified, and confirmed/refuted heads counted by SQL queries. These counts directly yield dr , pr , and the resulting RVS. For positive Horn rules, SQL generation is fully automatic: body atoms are compiled into joins, shared variables into equality constraints, and head confirmation or refutation into additional lookups or incompatibility checks. For more general rules, we use rule-specific SQL templates. Details can be found in Appendix C.
3.6
Verification of the desirable properties
RVS respects properties (P1) and (P2), which are both immediate by definition of RVS. (P3): For soft rules, RVS is explicitly dataset-aware because it compares the prediction contradiction rate pr to the dataset contradiction rate dr . The value dr acts as a rule-specific baseline measuring how often the rule is contradicted in the observed data. As a result, RVSsoft = 0 when r the predictions do not contradict the rule, RVSsoft = 1 when they contradict it at the same rate r as the dataset, and values above or below 1 indicate respectively worse or better compliance than the dataset baseline. Moreover, for two models evaluated on the same rule with non-zero scores, the ratio of their scores reflects the ratio of their prediction contradiction rates relative to the same dataset baseline. This makes model comparisons interpretable at the rule level. (P4): RVS depends only on an evaluation setting and applies equally to any predictive system. In practice, for rules that can be translated into relational queries, the required quantities dr and pr can be computed using standard Database Management System (DBMS) operations. Our implementation computes RVS for Horn rules by automatically storing and querying predicates in a relational database.
4
Experiments
Research questions. Our experiments are designed to assess whether RVS provides information that is complementary to standard predictive metrics. We focus on two questions: RQ1 asks whether models with similar predictive accuracy can exhibit different levels of logical compliance; and RQ2 asks whether dr values can help identify and compare inconsistent or ill-posed rules. Datasets and rules. We evaluate RVS on three datasets that differ in domain, scale, and predictive task: two knowledge-graph link-prediction benchmarks and one large relational regression dataset, as summarized in Table 2. Dataset Family FB15k-237 DV3F
Entities 3,007 14,541 392,809
Relations/Tables 12 237 13
Triples/Tuples 19,845 272,115 4,837,087
Test facts 5,681 20,466 2,213
Rules 37 509 4
Table 2: Overview of dataset characteristics.
6
Kind Knowledge graph Knowledge graph Relational database
Family [7] is a knowledge graph for link prediction. We use 36 AMIE+ rules [10], e.g. aunt(A, C) ← sister(A, B) ∧ father(B, C), together with a soft rule father(X, Y ) ← son(Y, X) which was intentionally included as an imperfect rule (because son(Y, X) could also imply mother(X, Y )). This gives 37 rules for evaluation. We also add incompatibility constraints such as ⊥ ← brother(A, B) ∧ sister(A, B). This benchmark provides a controlled setting in which many rules correspond to strong semantic constraints over kinship relations. It is therefore particularly useful for testing whether RVS detects logically implausible predictions even when predictive performance is high. FB15k-237 [21] is a knowledge graph from Freebase [1]. We use 509 NeuralLP rules [26], e.g. teamLocation(X, Y ) ← placeOfBirth(Y, X). Most rules are soft, reflecting statistical regularities. We additionally include automatically generated incompatibility constraints to detect rule violations (Appendix D). This benchmark is larger and exhibits weaker semantic regularities than Family. It therefore tests RVS in a setting where rules mostly express statistical regularities. DV3F [9] is a large open real-estate database made available by the French government. The prediction task is transaction-price regression. We define proximity rules stating that properties with the same type, postal code, and year should have comparable price per square meter. For instance, for houses: |price_square_meter(A)−price_square_meter(B)| < ϵ ← type(A) = ”house”∧ type(B) = ”house” ∧ postalcode(A) = postalcode(B) ∧ year(A) = year(B). This setting extends the evaluation beyond traditional knowledge-graph prediction tasks. Predictive models. The selected models represent several methodological families, as summarized in Table 3. AnyBURL [15] is a rule-based method; CompGCN [22] and GraphSAGE [12] are neural graph models; UniKER [4] and ExpressGNN [27] incorporate logical reasoning; and Rel-LLM [24] augments relational learning with LLM-derived representations. For each experiment, the prediction set D̂ is obtained by applying the model to the test partition. Table 3: Predictive models evaluated in our experiments. Algorithm
Family
Approach
Datasets
AnyBURL CompGCN UniKER ExpressGNN GraphSAGE Rel-LLM
Neuro-symbolic Neural Neuro-symbolic Neuro-symbolic Neural Neural
Rule-based KG embedding Embedding + rules GNN + probabilistic logic GNN GNN + LLM
Family, FB15k-237 Family, FB15k-237 Family FB15k-237 DV3F DV3F
Results. We evaluate whether RVS provides information not captured by standard predictive metrics. The results are organised around the two research questions introduced above. Family. Figure 1 reports the rule-level contradiction rate dr computed on the Family dataset. Most rules exhibit near-zero values, indicating that the dataset is largely consistent with the family relation semantics encoded in the rule set. The main exception is the deliberately imperfect soft rule father(X, Y ) ← son(Y, X), for which dr = 0.467. This confirms that the rule is contradicted in approximately 46.7% of its evaluable groundings, as expected: observing son(Y, X) does not uniquely determine that X is the father of Y , since X may also be the mother. This illustrates the role of dr as a rule-level diagnostic metric: rules that are semantically too broad or poorly calibrated are immediately identifiable before evaluating any predictive model. 7
Table 4 reports Hits@1, MRR, RVSsoft , and RVShard . AnyBURL achieves both the highest predictive performance (MRR = 0.973 and Hits@1 = 0.965) and the highest logical compliance (RVShard = 6.39 and RVSsoft = 2.15). This is expected given its rule-based nature. The comparison between CompGCN and UniKER is more informative. CompGCN achieves a higher MRR and Hits@1 than UniKER (MRR: 0.923 vs. 0.867), yet it produces substantially more hard-rule violations (425 vs. 121). Thus, the model with higher ranking performance is not the one that best satisfies hard logical constraints. RVS therefore exposes a difference in model behavior that is invisible from MRR or Hits@1. This effect is strongly dependent on the rule considered. For example, on the hard rule brother(X, Z) ← brother(X, Y ) ∧ sister(Y, Z), CompGCN produces 1160 violations out of 4626 evaluable prediction-induced groundings, whereas UniKER produces only 29 out of 3059. Conversely, for the rule aunt(X, Z) ← sister(X, Y ) ∧ father(Y, Z), CompGCN produces only 6 violations out of 662, whereas UniKER produces 30 out of 906. Figure 2 shows that CompGCN exhibits large variation across hard rules, while UniKER displays a more homogeneous violation profile. This per-rule view is informative: the RVS score summarises logical compliance, and rule-level scores indicate which logical behaviors are responsible. Method AnyBURL CompGCN UniKER
Hits@1 0.965 0.877 0.797
MRR 0.973 0.923 0.867
RVSsoft 2.15 37.9 34.0
RVShard 6.39 425 121
Table 4: Results on the Family dataset. Bold indicates the best value per metric.
0.05 0
CompGCN UniKER
2,000
0.47 RVS
dr
0.1
1,000 425 121
1 10 20 30 37 Figure 1: dr values for Family rules.
1
10
20
28
Figure 2: Rule-level RVS for Rhard on Family.
FB15k-237. Figure 3 shows that the dr values on FB15k-237 exhibit a much wider range than on Family. This indicates that many rules extracted or defined for this dataset are only weakly supported by the observed facts. For instance, the soft rule area_admin_parent(X, Y ) ← location_contains(Y, X) has dr = 0.88, meaning that 88% of its evaluable groundings are contradicted in the dataset. Some rules even reach dr = 1, meaning that whenever the body holds in the dataset, the head is violated in the dataset. These observations are important for interpreting rule compliance scores. A high prediction violation rate should not be interpreted in isolation: it must be compared with the corresponding dataset-level violation rate (dr ). This is precisely the role of the soft-rule normalisation in RVS. Table 5 compares ExpressGNN, CompGCN, and AnyBURL. ExpressGNN obtains the best MRR (0.49) and Hits@1 (0.43). It also obtains the highest logical compliance, with no observed hard-rule violations (RVShard = 0) and the lowest soft-rule score (RVSsoft = 0.971). Since RVSsoft < 1, ExpressGNN’s predictions are more consistent with the soft rules than the dataset. 8
The contrast between AnyBURL and CompGCN again illustrates the added value of RVS. Their Hits@1 and MRR scores are very close (MRR: 0.327 and 0.335), making them difficult to distinguish using predictive accuracy alone. However, their hard-rule compliance is markedly different: RVShard = 1.44 for AnyBURL versus 49.8 for CompGCN. Thus, two models with nearly identical ranking performance can have substantially different logical behavior. This difference is expected given that AnyBURL relies on explicit rule-based inference, whereas CompGCN is an embedding-based neural model. Additional results are provided in Appendix B.
dr
1
Method Hits@1 MRR RVSsoft RVShard ExpressGNN 0.43 0.490 0.971 0.0 CompGCN 0.245 0.335 0.992 49.8 AnyBURL 0.243 0.327 0.983 1.44
0.5 0
1 10 20 30 Figure 3: Values of dr for 30 sampled rules on FB15k-237.
Table 5: Results on FB15k-237.
DV3F. Table 6 reports the dataset contradiction rates for the four property-type rules. The House rule exhibits the lowest contradiction rate (dr = 0.019), indicating that houses with the same type, postal zone, and year tend to satisfy the price-proximity constraint. The Apartment, Industrial, and Outbuilding rules are more frequently contradicted (dr = 0.203, 0.141, and 0.257, respectively). These differences show that the same rule structure does not have the same empirical validity across property types. RVS therefore provides a dataset-level diagnostic that can guide the refinement of rules, for example by suggesting that additional attributes may be needed for property categories with higher dr . Table 6 also compares GraphSAGE with its LLM-enhanced variant, RelLLM. GraphSAGE achieves better predictive accuracy (R2 = 0.931 vs. 0.915) and lower error (MAE = 31.9 vs. 40.5). However, Rel-LLM achieves better aggregate logical compliance (RVS = 0.91 vs. 1.17). Since the aggregate score of Rel-LLM is below 1, its predictions violate the proximity rules less often than dataset baseline, whereas GraphSAGE violates them more often. The rule-level scores clarify this trade-off. GraphSAGE is more compliant on the House rule (0.732 vs. 0.915), but Rel-LLM is substantially more compliant on the Outbuilding rule (0.857 vs. 1.28), while both models exhibit similar behavior on the Industrial rule. Thus, the LLM-enhanced model does not simply improve or degrade compliance uniformly; rather, it changes the types of regularities that are better preserved in the predictions. dr value GraphSAGE Rel-LLM
R2
MAE
0.931 0.915
31.9 40.5
House 0.019 0.732 0.915
Apartment 0.203 1.01 1.02
Outbuilding 0.257 1.28 0.857
Industrial 0.141 0.015 0.015
RVS (aggregation) 1.17 0.91
Table 6: Rule-level dr and RVS values on DV3F. Bold indicates the best value per metric.
Interpretation of Results RQ1: Predictive accuracy and logical compliance are distinct. Across the three benchmarks, RVS highlights differences between models that are not captured by standard predictive metrics. On Family, CompGCN obtains higher MRR and Hits@1 than UniKER, but produces 9
over three times more hard-rule violations. On FB15k-237, AnyBURL and CompGCN have nearly identical MRR and Hits@1 scores, yet AnyBURL is substantially more compliant with hard rules. On DV3F, GraphSAGE achieves better R2 and MAE than Rel-LLM, while Rel-LLM obtains the better aggregate RVS score. These cases suggest that predictive accuracy alone is not always sufficient to characterise the quality of a model’s outputs in constraint-sensitive settings. RVS provides additional information for incorrect predictions. Conventional metrics measure whether a prediction matches the ground truth, but they do not distinguish between an incorrect prediction that remains plausible and an incorrect prediction that violates domain knowledge. RVS provides this missing dimension: it measures how logically inconsistent predictions are with respect to a specified rule set. Also, in the model comparison, per-rule RVSr further show that compliance is not uniform: CompGCN and UniKER violate different hard rules on Family, and GraphSAGE and Rel-LLM differ across property types on DV3F. RQ2: Rule-level dr values identify problematic or informative rules. The dataset contradiction rate dr provides a model-independent measure of rule-dataset compliance, and each dr value highlights rules that may require revision. For instance, the deliberately imperfect Family rule father(X, Y ) ← son(Y, X) is immediately identified by its high dr value, while FB15k-237 rules with dr close to 1 point to regularities that are poorly supported by the dataset. More broadly, the near-zero dr values observed across Family confirm that the dataset is largely consistent with the intended kinship semantics, whereas the wider distribution of dr on FB15k-237 shows that many of its rules should be interpreted as weak statistical regularities rather than strict constraints. On DV3F, the variation across property types indicates that the same proximity assumption holds more consistently for houses than for apartments, industrial properties, or outbuildings. dr can thus be computed independently of model evaluation, to assess whether a given rule is well-suited to the dataset. Overall, the experiments provide evidence for the central claim of the paper: RVS is complementary to standard predictive metrics. It adds an orthogonal dimension that becomes particularly important when predictions must satisfy logical or domain-specific constraints. In particular, when two models achieve similar predictive accuracy, RVS can reveal substantial differences in logical compliance, allowing practitioners to choose between models with a clearer understanding of the trade-off between accuracy and rule consistency. The results also show that RVS is useful at three levels: comparing models, auditing datasets, and diagnosing individual rules.
5
Related Work
Commonly used predictive performance metrics are task-specific: Hits@k [2] and MRR for link prediction, R2 [16, 5] and MAE [23] for regression, and accuracy, precision [17] and area under curve (AuROC, AuPRC) [8, 19, 14] for classification [20, 18]. All measure how closely predictions match ground-truth targets, but are silent on whether predictions conform to logical constraints. [25] introduces a semantic loss function that encodes symbolic knowledge directly into the training objective of neural networks. The approach augments the standard task-specific loss with a term that penalises predictions inconsistent with a set of logical constraints expressed in propositional logic. The semantic loss is thus a training-time mechanism rather than an evaluation metric: its goal is to produce models whose outputs are more logically consistent by construction. More closely related in spirit is the Sem@k metric introduced in [13] for assessing the semantic validity of a link prediction task. For each query (s, r, ?) or (?, r, o), the metric checks whether the top-k predicted tail entities (o) belong to the range of relation r, and whether the head entities 10
(s) belong to the domain of r. Formally, a prediction is counted as semantically valid if both constraints are satisfied, and Sem@k reports the fraction of semantically valid predictions across all top-k candidates and all queries. Sem@k focuses on a specific task (link prediction) for a specific data model (knowledge graphs) and a specific rule type (domain and range of constants). RVS is more general as it covers a variety of rules and tasks, and applies to broader data models such as relational databases. Moreover, Sem@1 is a special case of RVS under the two rules stating that, for every fact (subject, relation, object) the type of subject and the type of object must be in the domain and range of relation.
6
Conclusion and Future Work
We introduced RVS, an evaluation metric that measures logical rule compliance of a model independently of its predictive accuracy. Starting from four desirable properties identified for a logical compliance metric: ground-truth independence; hard/soft rule distinction; dataset-aware comparability; and model- and data-agnostic computability, we proposed RVS as a concrete metric satisfying these requirements. A key feature of RVS is that it distinguishes hard constraints from soft regularities. For soft rules, the dataset contradiction rate dr provides a model-independent estimate of how often each rule is contradicted in the data. This makes the evaluation dataset-aware: rules that are already weakly supported in the dataset do not unfairly penalise predictive models. At the same time, rule-level scores make it possible to inspect, compare, and rank individual rules, thereby helping to identify ill-posed, overly broad, or inconsistent constraints before and during model evaluation. RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. For rules that can be translated into SQL, its computation can be implemented using a relational DBMS, enabling automatic score computation and integration into existing data-processing pipelines. This makes the metric usable on real relational datasets while preserving a formal interpretation. Experiments on heterogeneous benchmarks, covering knowledge-graph link prediction and relational regression, show that RVS captures information that is not reflected by predictive accuracy alone. Models with similar MRR, Hits@1, MAE, or R2 scores can exhibit substantially different levels of logical compliance and in some cases a model with stronger predictive performance can produce more rule violations. These results support the use of RVS as a diagnostic tool for comparing predictive models, auditing datasets, and analyzing the contribution of individual rules. Future work will investigate how RVS can be used not only as an evaluation metric but also as a training signal. In particular, incorporating rule-violation terms into learning objectives could make it possible to jointly optimize predictive performance and logical compliance, while maintaining the distinction between strict constraints and empirical regularities.
11
References [1] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250, 2008. [2] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787–2795, 2013. [3] Tianlang Chen, Charilaos I. Kanatsoulis, and Jure Leskovec. Relgnn: Composite message passing for relational deep learning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon LacosteJulien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. [4] Kewei Cheng, Ziqing Yang, Ming Zhang, and Yizhou Sun. Uniker: A unified framework for combining embedding and definite horn rule reasoning for knowledge graph inference. Conference on Empirical Methods in Natural Language Processing, pages 9753–9771, November 2021. [5] Davide Chicco, Matthijs J. Warrens, and Giuseppe Jurman. The coefficient of determination r-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science, 7:e623, 2021. [6] Guillaume Delplanque, Luisa Werner, Nabil Layaïda, and Pierre Genevès. A comparative analysis of neurosymbolic methods for link prediction. In Leilani H. Gilpin, Eleonora Giunchiglia, Pascal Hitzler, and Emile van Krieken, editors, Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning (NeSy 2025), 8-10 September 2025, Santa Cruz, CA, USA, Proceedings of Machine Learning Research, pages 674–696. PMLR, 2025. [7] Woodrow W. Denham. The Detection of Patterns in Alyawarra Nonverbal Behavior. 1973. [8] Tom Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861–874, 2006. [9] French state. Dv3f - donnée pour l’analyse des marchés fonciers et immobiliers. https: //doc-datafoncier.cerema.fr/doc/dv3f/?v=13 [saw in june 2026]. [10] Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. Fast rule mining in ontological knowledge bases with amie+. July 2015. [11] Justin Gu, Rishabh Ranjan, Charilaos I. Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, and Jure Leskovec. Relbench v2: A large-scale benchmark and repository for relational data. CoRR, abs/2602.12606, 2026.
12
[12] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017. [13] Nicolas Hubert, Pierre Monnin, Armelle Brun, and Davy Monticolo. Sem@k: Is my knowledge graph embedding model semantic-aware? Conference on Empirical Methods in Natural Language Processing, 2023. [14] Matthew B. McDermott, Haoran Zhang, Lasse Hyldig Hansen, Giovanni Angelotti, and Jack Gallifant. A closer look at AUROC and AUPRC under class imbalance. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 44102–44163. Curran Associates, Inc., 2024. [15] Christian Meilicke, Melisachew Wudage Chekol, Patrick Betz, Manuel Fink, and Heiner Stuckenschmidt. Anytime bottom-up rule learning for large-scale knowledge graph completion: C. meilicke et al. The VLDB Journal, 33(1):131–161, 2024. [16] N. J. D. Nagelkerke. A note on a general definition of the coefficient of determination. Biometrika, 78(3):691–692, 09 1991. [17] David Powers. Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies, 2(1):37–63, 2011. [18] David M. W. Powers. Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies, 2(1):37–63, 2011. [19] Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3):e0118432, 2015. [20] Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437, 2009. [21] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. CVSC, pages 57–66, July 2015. [22] Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition-based multi-relational graph convolutional networks. arXiv preprint arXiv:1911.03082, 2019. [23] Cort J. Willmott and Kenji Matsuura. Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate Research, 30(1):79–82, 2005. [24] Fang Wu, Vijay Prakash Dwivedi, and Jure Leskovec. Large language models are good relational learners. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7835–7854, 2025. [25] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pages 5502–5511. PMLR, 2018. [26] Fan Yang, Zhilin Yang, and William W. Cohen. Differentiable learning of logical rules for knowledge base reasoning. NIPS, 2017. 13
[27] Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. ICLR, February 2020. [28] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: a general graph neural network framework for link prediction. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA, 2021. Curran Associates Inc.
14
A
Hyperparameters
All experiments are conducted on a Linux server equipped with an Intel Xeon Silver 4210R CPU (10 cores, 20 threads, 2.40 GHz), 180 GB of RAM, and an NVIDIA RTX A6000 GPU with 49 GB of VRAM. The full implementation details, configuration files and results are publicly available in our GitHub repository1 . Throughout all experiments, the importance weight σr is set to 1 for every rule r ∈ Rhard .
A.1
Inductive model: UniKER
The UniKER model is trained in TransE mode with a fixed configuration across experiments. Mode: TransE Iterations: 8 Noise threshold: 0.0 Top-k threshold: 0.2 Init flag: 0 (default)
A.2
Rule-based model: AnyBURL
The AnyBURL model is trained using identical hyperparameters for both Family and FB15k-237. WORKER_THREADS: 7 POLICY: 2 REWARD: 5 EPSILON: 0.1 THRESHOLD_CORRECT_PREDICTIONS: 2 THRESHOLD_CONFIDENCE: 0.0001 ZERO_RULES_ACTIVE: false
A.3
Embedding model: CompGCN
Family setting: name: best_model score_func: conve opn: corr FB15k-237 setting: score_func: transe opn: sub gamma: 9 hid_drop: 0.1 init_dim: 200
A.4
GNN model: ExpressGNN (FB15k-237)
slice_dim: batchsize: use_gcn: 1
16 16 1
https://anonymous.4open.science/r/Rule-Violation-Score-585C
15
RVS value
100
49.8
1.44
769
ExpressGNN CompGCN AnyBURL
1
10 hard rules
18
Figure 4: RVS for ExpressGNN, CompGCN, and AnyBURL on the FB15k-237 dataset for each hard rule. num_hops: 1 embedding_size: 128 gcn_free_size: 127 patience: 20 lr_decay_patience: 100 entropy_temp: 1
A.5
GraphSAGE (DV3F)
GraphSAGE is used with the default settings from the original implementation.
A.6
Rel-LLM (DV3F)
epochs: batch_size: learning rate: dropout: LLM: output layer:
A.7
15 128 2e-4 0.1 frozen MLP
Evaluation infrastructure
All evaluation metrics are computed using a PostgreSQL database running inside a Docker container. Predictions are generated in Python and automatically translated into SQL queries.
B
Additional Results
Figure 4 shows the details of RVS for the 18 hard rules for each method in FB15k-237. Figure 5 shows the details of RVS for 30 sampled soft rules for each method in FB15k-237.
C
Practical Computation of RVS
We describe how we compute RVS in practice for any dataset, prediction algorithm, and rule set. Computing RVS requires two preliminary steps: the algorithm must first be run on the test set to
16
RVS value
3
144 80
ExpressGNN CompGCN AnyBURL
2 1 0
1
10
20
30
soft rules Figure 5: RVS for ExpressGNN, CompGCN, and AnyBURL on the FB15k-237 dataset for 30 sampled soft rules. obtain predictions, and both the dataset and the predictions must then be stored in a database, as described below. Database schema. We create one table per predicate symbol appearing in any rule r. We distinguish two settings depending on the data model. • Knowledge graph. The dataset is a set of triples (s, r, o), where s and o are entities and r is a relation. Each relation r becomes a dedicated table with columns subject, object, tag, and model. Observed facts are inserted with tag = 0 and model is left NULL. Predictions are inserted with tag = 1, model set to the algorithm identifier. The facts in the test data are not inserted into these tables. • Relational database. The original schema is preserved as-is. In the table containing the test facts, we add a column tag set to 2 for the test facts and 0 for observed facts. As before, test data are not used to compute RVS. Then, to distinguish the predictions made by the algorithm, we create a new table with columns id, prediction, tag, and model, filled using the same convention as above. Table 7 shows a sample of the aunt table for the Family dataset, illustrating the coexistence of observed facts and predictions from a single model in the same table. Table 7: Sample of the aunt table for the Family dataset. Rows with tag = 0 are observed facts; rows with tag = 1 are model predictions. subject
object
tag
model
1369 1682 1283 2079 2082 2099
1287 512 1271 2088 2086 2089
0 0 0 1 1 1
NULL NULL NULL compgcn compgcn compgcn
RVS computation with SQL. For each rule r, the quantities dr and pr are computed using SQL queries. We automated the generation of these SQL queries for Horn rules. In the case of 17
a knowledge graph, all facts are of the form rel(subject, object). One query is shown in the following example. For other types of rules, we need to define SQL queries manually. Example query Consider the rule r : aunt(X, Z) ← sister(X, Y )∧father(Y, Z) on the Family dataset. To count the confirmed groundings CrD = {θ ∈ BrD | ψr θ is confirmed in D} needed for the computation of dr , we issue: SELECT FROM JOIN JOIN WHERE AND AND AND AND AND
COUNT (*) aunt t0 sister t1 ON t0 . subject = t1 . subject father t2 ON t1 . object = t2 . subject AND t0 . object = t2 . object t0 . model IS NULL t1 . model IS NULL t2 . model IS NULL t0 . subject <> t1 . object t0 . subject <> t0 . object t1 . object <> t0 . object ;
This corresponds to evaluating every grounding of aunt(x, z) ∧ sister(x, y) ∧ father(y, z). The three model IS NULL filters restrict the query to observed facts. The three inequality conditions enforce that the variables X, Y , Z are mapped to distinct constants. To count violating groundings (the numerator of dr ), we adopt the open-world assumption. Thus, we first define a set of relations incompatible with aunt: brother, sister, father, ... Then, for each incomp_rel in this set, we run the above SQL query by substituting aunt with incomp_rel, and sum the resulting counts. To compute pr for a model m, the same query structure is reused with two modifications. First, the condition ( t0 . tag + t1 . tag + t2 . tag ) = 1
is added to ensure that exactly one atom in each grounding originates from the predictions of model m, while the remaining atoms come from observed facts. This makes every prediction contribute independently to the count. Second, the filter t0 . model = ’m ’ OR t1 . model = ’m ’ OR t2 . model = ’m ’
is added to identify the model. t Given dr and pr , the final metrics RVShard and RVSsof are obtained as described in Section 3.3. r r hard soft These are then aggregated to compute RVS and RVS , as defined in Section 3.4.
D
Incompatibility Constraints on FB15k-237
To define the set of refutation rules Rref , it is necessary to identify pairs of relations that are incompatible within the FB15k-237 dataset. We adopt the open-world assumption (OWA), under which absence from the knowledge graph does not imply falsity. Nevertheless, for the purpose of deriving incompatibility constraints, we introduce an additional assumption: if two relations have never been observed to hold simultaneously for the same pair of entities in the dataset, they are considered incompatible. While this assumption enables automatic extraction of such constraints, these may also be specified manually when needed.
18
For each relation reli , we define the set of incompatible relations as n o all_incompat_rels(reli ) = relk ∈ all_relations ∄(A, B) ∈ D 2 , reli (A, B) ∧ relk (A, B) .
To compute this set, we first identify, for every relation reli , the set of relations that are compatible with it. Starting from an empty compatibility set, we iterate over all facts in the dataset. Whenever both reli (A, B) and relk (A, B) are observed in the dataset for any pair of entities (A, B), the relation relk is marked as compatible with reli . After processing the entire dataset, any relation that has not been observed to co-occur with reli for any entity pair is deemed incompatible with reli . These incompatibility constraints are then used to derive refutation rules of the form ⊥ ← reli (X, Y ) ∧ relk (X, Y ), for every reli ∈ all_relations and relk ∈ all_incompat_rels(reli ). Formally: Rref = {⊥ ← reli (X, Y ) ∧ relk (X, Y ) | reli ∈ all_relations and relk ∈ all_incompat_rels(reli )}
19