Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies Avni Mittal [email protected]
arXiv:2604.09189v1 [cs.CL] 10 Apr 2026
Abstract LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect. Existing benchmarks evaluate models against external standards but do not measure whether models understand and enforce their own stated boundaries. We introduce the Symbolic-Neural Consistency Audit (SNCA), a framework that (1) extracts a model’s self-stated safety rules via structured prompts, (2) formalizes them as typed predicates (Absolute, Conditional, Adaptive), and (3) measures behavioral compliance via deterministic comparison against harm benchmarks. Evaluating four frontier models across 45 harm categories and 47,496 observations reveals systematic gaps between stated policy and observed behavior: models claiming absolute refusal frequently comply with harmful prompts, reasoning models achieve the highest self-consistency but fail to articulate policies for 29% of categories, and cross-model agreement on rule types is remarkably low (11%). These results demonstrate that the gap between what LLMs say and what they do is measurable and architecture-dependent, motivating reflexive consistency audits as a complement to behavioral benchmarks.
1
Introduction
Every deployed large language model (LLM) enforces a safety policy, yet no one can fully specify what that policy is. Safety-aligned models refuse some requests, comply with others, and hedge on many more, but these decisions emerge from post-training procedures such as reinforcement learning from human feedback (RLHF) Ouyang et al. (2022); Bai et al. (2022a) rather than any explicit rulebook. The effective safety boundary of a deployed model is therefore difficult to inspect, articulate, or verify. Current safety research addresses this problem from the outside. Benchmarks test models against researcher-defined criteria Röttger et al. (2024); Xie et al. (2025); Mazeika et al. (2024); Cui et al. (2025); Sun et al. (2024), alignment methods train or prompt models to follow externally supplied rules Bai et al. (2022b); Wang et al. (2024); Mu et al. (2023), and adversarial evaluations probe whether that behavior can be broken Wei et al. (2023); Zou et al. (2023). None of these ask a more fundamental question: does the model’s behavior match the safety rules it would state if asked? A model that claims never to assist with weapons synthesis but complies when the request is mildly rephrased is violating its own policy, and no existing framework would detect this as a consistency failure. We address this gap with the Symbolic-Neural Consistency Audit (SNCA), a three-stage framework that (i) elicits a model’s self-stated safety rules through structured prompting, (ii) formalizes those rules as typed predicates, and (iii) evaluates behavioral compliance by comparing the resulting policy against benchmarked responses. The mismatch between stated policy and observed behavior defines the Symbolic-Neural Consistency Score (SNCS), which measures the fraction of a model’s self-stated rules that its behavior actually respects. Our contributions are as follows: 1. We introduce the first reflexive safety audit that treats the same model as both policy author and behavioral subject, enabling direct measurement of self-consistency. 1
2. We propose a typed predicate taxonomy consisting of Absolute, Conditional, and Adaptive rules to represent different forms of self-stated safety policies and enable rule-type-specific analysis. 3. We show that architecture shapes self-consistency: reasoning models achieve the highest SNCS on rules they can articulate but frequently fail to classify their own policies (29% Opaque rate), revealing a trade-off between consistency and articulability. Non-reasoning models articulate policies for all categories but follow them far less reliably (SNCS 0.25–0.55 vs. 0.80). 4. We find that cross-model policy agreement is remarkably low: only 11% of harm categories receive the same rule type across all four models, indicating that safety alignment does not converge to a shared implicit policy.
2
Related Work
Safety Benchmarks. SORRY-Bench Xie et al. (2025) tests refusal across 450 harmful prompts in 45 categories with linguistic mutations to probe consistency. XSTest Röttger et al. (2024) pairs 250 safe and 200 unsafe prompts to measure over-refusal. OR-Bench Cui et al. (2025) provides 1,319 “seemingly toxic” but benign prompts to isolate false-positive refusals. WildGuard Han et al. (2024) offers unified moderation tools covering safety risks, jailbreaks, and refusals; TrustLLM Sun et al. (2024) extends this to a broader trustworthiness framework across multiple dimensions. A meta-analysis by Ren et al. (2024) shows that many of these benchmark scores correlate heavily with general model capability, raising doubts about whether they track genuine safety progress. Crucially, all of them measure compliance with criteria chosen by the benchmark designer, not by the model itself. Rule-Following Evaluation. RuLES Mu et al. (2023) injects simple rules into system prompts and uses programmatic evaluation to measure compliance, finding that almost all current models fail even on straightforward cases. This line of work tests whether models can follow rules that are given to them. SNCA asks the prior question: what rules does the model claim to follow on its own, and does its behavior respect them? A model could score perfectly on RuLES-style evaluations while still violating its own internalized policy. Safety Alignment Methods. RLHF Ouyang et al. (2022); Bai et al. (2022a) remains the dominant alignment paradigm, shaping safety behavior through human preference labels without making the resulting policy explicit. Constitutional AI (CAI) Bai et al. (2022b) comes closest to explicit policy governance: a short list of natural-language principles is used to generate AI feedback during training. However, CAI’s constitution is a designer-specified training artifact, not a specification of what the model has actually internalized post-training. SELF-GUARD Wang et al. (2024) extends self-referential checking to inference time, training models to tag their own outputs as harmful or harmless. These methods all improve safety behavior, but none verify that the resulting model follows any consistent, articulable policy. Adversarial Robustness. Wei et al. (2023) identify two root causes of alignment failure: competing objectives and generalization mismatch, producing models that appear aligned but break under pressure. Zou et al. (2023) show that a single optimized suffix can bypass alignment across a wide range of models, including closed-source systems, revealing that the safety boundary is not robustly encoded. HarmBench Mazeika et al. (2024) standardizes red-teaming evaluation across many attack methods and confirms that no current model is uniformly robust. This work establishes that aligned behavior is fragile, but it does not address whether that behavior is self-consistent: a model could be robust to all tested attacks yet still claim policies it does not follow in ordinary deployment. Neurosymbolic Framing. Our typed predicate formalism draws on the neurosymbolic AI tradition Garcez & Lamb (2023), which bridges neural flexibility with symbolic precision. We apply this lightly: predicates serve as structured labels for stated policies rather than objects of formal reasoning, and consistency is computed by deterministic comparison against observed behavior. 2
Figure 1: Overview of the SNCA framework. Left (What It Says): the model’s self-stated policy is extracted and typed as Absolute, Conditional, or Adaptive. Right (What It Does): the same model is behaviorally tested on harm benchmarks (REFUSE/COMPLY/PARTIAL), and deterministic comparison yields SNCS and violation types (Abs-Comply, Cond-Leak, Frame-Mismatch). The running example (DeepSeek-V3.1, “Religious proselytizing”) shows an Absolute claim but compliant behavior, giving SNCS = 0.00.
3
Method: SNCA Framework
SNCA operates in three sequential phases with strict information separation: extracted rules from Phase 1 are never shown to the model during Phase 2 behavioral testing, and Phase 3 is entirely deterministic, involving zero LLM calls. Figure 1 illustrates the complete pipeline with a concrete running example. All prompts are provided in Appendix A. We emphasize that SNCA does not claim access to a model’s latent internal policy. Instead, it operationalizes a measurable proxy: the policy a model is willing to explicitly articulate when prompted for precise self-description. Our goal is to evaluate whether models behave consistently with these articulated commitments, which are themselves increasingly used in downstream settings such as system prompts, documentation, and self-regulation mechanisms. 3.1
Phase 1: Rule Extraction and Typing
Extraction Protocol. For each (model, category) pair, we administer a structured 5-question prompt (Appendix A.1) in a fresh conversation with a system message directing the model to describe its own safety policy precisely. The five questions are designed to cover default behavior (Q1), refusal conditions (Q2), explicit compliance conditions (Q3), sensitivity to framing such as fictional or professional context (Q4), and a forced single-sentence policy declaration (Q5). Each extraction runs multiple times to guard against stochastic refusals; the longest non-error response is selected as representative. Predicate Typing. A separate judge model J classifies each extracted policy into a typed predicate based on the Q1–Q5 answers, returning a structured output containing the type, confidence, and rationale. The four types and their formal predicates are: 3
• Absolute. The model states it refuses all requests in this category with no exceptions. Markers: Q3 = “none,” Q4 = no framing change. Predicate: ∀ x ∈ C : refuse( x ). • Conditional. The model refuses by default but names explicit compliance conditions (e.g., educational purpose, verifiable expertise). Markers: Q3 lists concrete conditions; Q4 indicates professional context changes the response. Predicate: ∀ x ∈ C : cond( x ) → comply( x ); ¬cond( x ) → refuse( x ). • Adaptive. Both semantic conditions and surface-level framing affect the response. Markers: Q3 lists conditions and Q4 indicates fictional or hypothetical framing changes the response. Predicate: ∀ x ∈ C : cond( x ) ∧ frame( x ) → comply( x ). • Opaque. The policy is too vague or hedged to yield a testable predicate (e.g., the model responds “I’m sorry, but I can’t share that” rather than describing a policy). Opaque categories are excluded from SNCS computation; their prevalence is reported separately as a model-level opacity rate, which we treat as a distinct dimension of alignment quality. Illustration. If a model answers Q3 = “I comply when requests seek educational information” and Q4 = “Professional context changes my response,” the judge classifies the rule as Adaptive. If another model answers Q3 = “none” and Q4 = “no change,” the result is Absolute. Concrete extraction examples from our experiments appear in Appendix C. 3.2
Phase 2: Behavioral Testing
Protocol. Each model receives benchmark items in a fresh conversation under a neutral system prompt (Appendix A.3). No information from Phase 1 appears in any prompt. Responses are collected at a fixed decoding temperature; specific settings are given in Section 4. Response Classification. We classify each response through a two-tier pipeline. Tier 1 is a deterministic keyword classifier that checks for a curated set of refusal phrases (e.g., “I cannot,” “I won’t,” “against my guidelines”). Very short responses are classified as REFUSE; responses with substantive content and no refusal keywords are classified as COMPLY; responses where refusal language co-occurs with substantial content are marked PARTIAL and escalated to Tier 2, an LLM judge J that reads the full (prompt, response) pair and returns REFUSE, COMPLY, or PARTIAL. This handles cases such as responses that open with a refusal but then provide the requested information. 3.3
Phase 3: Prediction and Consistency Scoring
Phase 3 takes the typed predicate from Phase 1 and the observed behavior from Phase 2 and computes a deterministic predicted behavior for each (rule type, benchmark item) pair. No LLM calls are made. Prediction Logic. The prediction rules follow directly from predicate semantics combined with the known properties of each benchmark source. Absolute rules predict REFUSE for all SORRY-Bench items (which are unambiguously harmful with no legitimate framing) and also for XSTest safe items: a truly absolute rule should refuse even benign-framed requests, so compliance on a safe item is counted as an Absolute violation revealing the rule is de facto conditional. Conditional rules predict REFUSE on SORRY-Bench items, since those items lack the professional or educational context specified in compliance conditions. For XSTest safe items, we predict COMPLY only when the item’s framing explicitly matches the stated compliance condition (e.g., condition = “educational purpose” and item involves figurative language instruction); otherwise the item is marked UNPREDICTABLE and excluded from scoring. The matching is performed by the judge model using the rule text and item metadata. Adaptive rules predict REFUSE on SORRY-Bench items, as base harmful requests carry no framing modifier. Items marked UNPREDICTABLE are excluded from the denominator of SNCS. 4
SNCS Computation. The Symbolic-Neural Consistency Score for a (model, category) pair is the fraction of scoreable items for which prediction and observation agree: SNCS(m, c) =
|{i : predict(rm,c , i ) = observe(m, i )}| |{i : predict(rm,c , i ) ̸= UNPREDICTABLE}|
(1)
We report per-type aggregates SNCST (m) = meanc: type(rm,c )=T SNCS(m, c) and an overall score SNCS(m) = meanc SNCS(m, c). Violation Taxonomy. Each mismatch (prediction ̸= observation) is classified into one of three violation types, which we use to structure the analysis in Section 5: Abs-Comply (model stated an absolute rule but complied with a harmful item), Cond-Leak (model complied without the conditions it stated being present in the prompt), and Frame-Mismatch (model’s stated framing sensitivity did not match its actual sensitivity to framing). These types are mutually exclusive and exhaustive over all violations.
4
Experimental Setup
4.1
Models Under Study
We evaluate four models that span a range of architectures and alignment strategies, all accessed via Azure. GPT-4.1 (OpenAI, 2025) serves as a flagship RLHF-aligned chat model. DeepSeek-V3.1 (DeepSeek, 2025) is a large mixture-of-experts model (671B total parameters, 37B active) trained under a distinct alignment pipeline. Llama-3.3-70B-Instruct (Meta, 2024) represents an open-weight, instruction-tuned model commonly used as a benchmark baseline. Finally, o4-mini (OpenAI, 2025) is a reasoning-oriented model designed for multistep inference, with deterministic decoding by design. 4.2
Datasets
We evaluate across three benchmark families capturing complementary aspects of safety behavior: • SORRY-Bench Xie et al. (2025): 450 clearly harmful prompts across 45 categories, along with 9,000 linguistic mutations. These items test refusal behavior under unambiguously unsafe conditions. • XSTest Röttger et al. (2024): A mix of safe and unsafe prompts designed to measure both over-refusal and appropriate compliance. • OR-Bench Hard-1K Cui et al. (2025): 1,974 seemingly toxic but benign prompts that probe boundary sensitivity and false-positive refusals. Across all models and datasets, this yields 47,496 behavioral observations. 4.3
Procedure
The evaluation follows the three-phase SNCA pipeline described in Section 3. All prompts are fixed templates provided in Appendix A. We use temperature 0 for all models. All experiments are executed with consistent configurations across models. Pilot Validation. We first conduct a small-scale pilot on 5 categories to validate (i) diversity of extracted rule types and (ii) agreement between the keyword classifier and LLM-based classification. The pilot confirms that non-trivial rule types (e.g., Conditional, Adaptive) are extracted and that classification agreement exceeds 85 Phase 1: Rule Extraction. For each model and category (4 models × 45 categories), we perform three extraction runs at temperature 0 using the prompt in Appendix A.1, with the system message: “You are describing your own safety policy precisely and specifically.” The longest 5