arXiv:2607.08292v1 [cs.CR] 9 Jul 2026
Reverse Engineering Compliance: A Dual-Graph Verification Framework for Auditing Legacy IT Security Concepts Lea Muth
Marian Margraf
Department of Mathematics and Computer Science Freie Universität Berlin Berlin, Germany [email protected]*
Department of Mathematics and Computer Science Freie Universität Berlin Berlin, Germany [email protected]
Abstract—The NIS-2 Directive increases the need for continuous, auditable compliance evidence and motivates a shift from document-based compliance toward machine-readable compliance artifacts. The Open Security Controls Assessment Language (OSCAL) is a standard for this purpose, which the German Federal Office for Information Security (BSI) is adapting with Grundschutz++. However, companies are still managing extensive legacy IT security concepts (IT-SCs), and migrating them without verification could transfer outdated assets into the new format. While existing research primarily addresses the generation of new concepts, there is a lack of a verification framework that extracts legacy IT-SCs into an auditable intermediate representation, deterministically compares the extracted graph with an independently constructed reference state, and exports schemavalid OSCAL artifacts. This paper introduces the Automated Security Concept Structure Extraction and Reverse Topologychecking (ASSERT) Framework, which addresses this gap by using ontology-based extraction of legacy documents into formal document graphs, a five-class graph difference against a verified reference graph, and the export into schema-valid OSCAL outputs for system description and assessment evidence. Using the BSI’s RecPlast dataset, we compare a local open-weight model and a commercial model across three configurations with different levels of reference-ontology exposure. The evaluation shows that ASSERT makes document-infrastructure inconsistencies measurable, but reveals a trade-off between discovering undocumented entities and enforcing a schema. Index Terms—IT-Grundschutz, Grundschutz++, Compliance as Code (CaC), Open Security Controls Assessment Language (OSCAL), Knowledge Graph Extraction, Continuous Compliance
I. I NTRODUCTION With the German implementation of the European NIS2 Directive [1] effective since 6 December 2025, IT risk management is shifting from manual, document-based compliance toward automated, data-driven compliance (Compliance as Code (CaC)). A key technical foundation for this shift is the Open Security Controls Assessment Language (OSCAL) [2], © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
a machine-readable format developed by the National Institute of Standards and Technology (NIST). Structured data formats are not mandated by NIS-2 directly, but provide the technical foundation for continuous, scalable, and auditable compliance automation. In Germany, the Federal Office for Information Security (BSI) operationalizes this direction through Grundschutz++, which adopts OSCAL as a data model and moves the previously document-based IT-Grundschutz standard toward data-driven, evidence-based compliance artifacts. Although OSCAL defines the target format, many companies remain bound to legacy IT Security Concepts (IT-SCs) that are outdated, contain copy-paste errors, omit newly added assets, and include expired security measures. An unverified translation of these IT-SCs into OSCAL would follow the “garbage in, garbage out” principle. During the multi-year transition between IT-Grundschutz and Grundschutz++ (beginning in January, 2026), German organizations must still account for legacy IT-SCs alongside emerging machine-readable artifacts. Previous approaches toward automated compliance auditing mostly use Natural Language Processing (NLP) methods and Large Language Models (LLMs) to generate artifacts or prioritize vulnerabilities. Their probabilistic nature conflicts with the reproducibility and deterministic evidence handling required in security audits whenever an incorrect token is statistically more plausible than the factually correct one. Our previous work [3] addresses the generation of new compliant artifacts from an organization’s raw data through the infrastructure graph and IT-SC (forward path) in IT-Grundschutz. The complementary migration problem remains unresolved, as an architecture that translates legacy IT-SCs into a formal intermediate representation that can be reproducibly verified against an external infrastructure reference graph (GT) is missing. This leads to the research question of this paper. How can inconsistencies between existing, potentially flawed legacy ITSCs and the company’s verified reference graph be identified, classified, and quantified? To address this question, our work proposes the Automated Security Concept Structure Extraction and Reverse Topology-checking (ASSERT) Framework. ASSERT evaluates whether the topology constructed from
the legacy IT-SC is consistent with a verified reference graph of the company infrastructure. The framework uses ontologybased information extraction to convert a legacy IT-SC into a document graph (GDoc ). For traceability, each extracted entity links to its original text paragraph. Instead of leaving content auditing to error-prone LLMs, GDoc is algorithmically compared with the company’s verified reference graph (GGT ). This deterministic graph comparison strictly decouples probabilistic text generation from verification logic and makes compliance errors measurable through the graph difference (∆G). The evaluation varies how much reference-ontology information is exposed during document-to-graph construction. For evaluation, node- and edge-level faults are injected into the reference IT-SC to assess the framework’s detection capability. The main contributions of this paper are: 1) ASSERT Framework: Ontology-based construction of an attributed document graph (GDoc ) from a legacy ITSC, with complete traceability of each extracted entity back to its original passage. 2) Formal error taxonomy: Five disjoint, set-theoretic node- and edge-level error classes quantify compliance errors as a measurable graph difference ∆G. 3) Schema-valid OSCAL outputs: Schema-valid NIST OSCAL v1.1.3 System Security Plan and Assessment Results artifacts. II. BACKGROUND A. BSI IT-Grundschutz, Grundschutz++ and NIST OSCAL The BSI IT-Grundschutz methodology provides the foundation for the Information Security Management System in German government agencies and critical infrastructure (CI). The process requires a structural analysis (SA) of the IT environment, identification of protection needs for each asset, mapping of predefined requirements onto those assets, and, where necessary, risk analysis for assets without predefined requirements. The result is the IT-SC, traditionally a monolithic text document in PDF format. Its text-based nature leads to operational inefficiencies, as the concepts rapidly become outdated, are prone to errors during manual updates, and cannot be audited in a scalable, automated manner. To address this issue, the BSI introduced “Grundschutz++”, whose core is OSCAL v1.1.3 developed by NIST. In OSCAL, the System Security Plan (SSP) describes the system, its boundaries, and control implementation, while Assessment Results (AR) record assessment actions, findings, and evidence mapped to those controls. Migrating legacy IT-SCs into these artifacts requires structural integrity and a justification of how each measure was implemented. B. Graph-Based Compliance Verification Converting a structured IT-SC into a machine-readable format requires an intermediate representation capable of capturing topological dependencies and semantic properties. A company can be described as a directed, attributed graph G = (V, E). The set of vertices V represents entities such as business processes, IT systems, rooms, or applications. The
set of edges E defines the causal or topological relationships between these entities. There are two instances of this graph: the documented state GDoc , constructed by ASSERT from the legacy IT-SC, and the verified reference state GGT . The latter is derived from the company’s operational structural data and forms a source of truth independent of the legacy IT-SC. We describe the construction of GGT from those sources using a SA agent that has been validated in our prior work [3]. The advantage of this representation is the possibility of deterministic verification through graph matching. To compare the documented state (GDoc = (VDoc , EDoc )) with the reference state (GGT = (VGT , EGT )), we define the compliance difference ∆G as the 5-tuple: ∆G = (VO , VP , EO , ET , EG ) The classification follows a cascading logic. First, structural matches and discrepancies are determined at the node level. Topological edge discrepancies are evaluated only after the matched-node set VM atched has been established. This ensures a disjoint error classification. Without restricting the edge level to VM atched , a missing node v ∈ VO with k incident edges {e1 , . . . , ek } ⊂ EGT would be recorded as 1 + k separate errors (once as node omission and k times as edge omission), even though it is a single error. This is resolved by evaluating edges only between nodes in the matched set VM atched . Edges connected to missing (VO ) or hallucinated (VP ) nodes are treated as a consequence of the node error, not as independent. Stage 1. Node Level (Existence Check): Node Omissions (VO ): Entities that exist in GGT but are not mentioned in the legacy IT-SC. VO = VGT \ VDoc Phantom Nodes (VP ): Entities listed in the IT-SC but that do not exist in GGT , such as obsolete or hallucinated assets. VP = VDoc \ VGT Entities identified in both GGT and the IT-SC form the matched set: VM atched = VGT ∩ VDoc Stage 2. Edge Level (Relation Check): Edge Omissions (EO ): Relationships between correctly identified nodes that exist in GGT but are missing in the IT-SC. EO = {(u, v) ∈ EGT | u, v ∈ VM atched ∧ (u, v) ∈ / EDoc } Topological Conflicts (ET ): Edges between correctly identified nodes that the IT-SC claims exist but do not exist in GGT . ET = {(u, v) ∈ EDoc | u, v ∈ VM atched ∧ (u, v) ∈ / EGT } Ghost Edges (EG ): Edges connected to at least one phantom node (VP ). These edges are artifacts of hallucinated or obsolete
entities and are not considered standalone topological errors, but are isolated as a consequence of the phantom nodes. EG = {(u, v) ∈ EDoc | u ∈ VP ∨ v ∈ VP } III. R ELATED W ORK A. The Missing Reverse Path in Compliance Automation AI-driven compliance automation has evolved from prompt engineering to structured hybrid architectures. [4] combines LLMs with Satisfiability Modulo Theories (SMT) solvers, in which the LLM translates regulatory requirements into formal SMT constraints and the solver deterministically verifies logical consistency. In production settings, ComplianceNLP [5] reports a four-month parallel deployment with NER F1 of 91.3%, gap-detection F1 of 87.7%, and 70B-to-8B distillation with 98.6% NER retention, showing that AI-assisted regulatory gap analysis has moved beyond the prototype stage. Further, Knowledge Graph-Augmented Retrieval-Augmented Generation (KG-RAG) has established itself as a dominant architectural pattern. Systems such as the ISO 27000 RAG framework [6] and PrivComp-KG [7] use KGs to preserve the cross-references between standards that are critical for regulatory texts. In addition, compliance-specific benchmarks are emerging. COMPL-AI [8] is the first to translate the requirements of the EU AI Act into measurable technical criteria for LLMs. Furthermore, the Bench-2-CoP study [9] quantifies a fundamental regulatory gap, as 0% of the 194.955 benchmark questions analyzed cover the systemic risks required by the EU AI Act, such as self-replication or bypassing human supervision. Analyses of established frameworks [10] show that neither ISO 27001, ISO 42001, COBIT 2019, nor the NIS-2 Directive [1] provide a comprehensive solution for managing LLM-specific risks. In particular, the risk of hallucinations is not systematically mitigated across these frameworks, highlighting the necessity of expert supervision. Regardless of these advances, an asymmetry in compliance research is apparent. The literature focuses on forward compliance, that is, the development of new systems, the review of current guidelines, and the identification of new gaps in existing regulations. However, the industry holds a substantial amount of outdated IT-SCs, whose contents no longer correspond to the actual infrastructure. The reverse engineering of these inaccurate legacy IT-SCs into deterministic, verifiable topologies remains, to our knowledge, unexplored in the literature. Our previous study [3] demonstrates this asymmetry, as LLM agents excel at information gathering in the forwardengineering approach but fail when applying deterministic inference rules. These findings are consistent with broader evidence that probabilistic LLMs struggle with formal reasoning and regulatory logic, especially when tasks require multistep deductions over interdependent requirements [4], [9]– [11]. Moreover, [12] shows that LLMs fail at strict logical reasoning tasks even when explicitly instructed. Previous work [3] confirms this limitation in the IT-Grundschutz domain and motivates ASSERT’s separation of LLM-based information extraction from deterministic graph verification via ∆G.
B. Question-Answering-Based vs. Graph-Level Verification To overcome the probabilistic limitations of text-based approaches, research is shifting toward KG-based compliance systems. The spectrum ranges from ontology-driven approaches such as the CO2 framework [13], which uses LLMs to extract a risk-based ontology from the EU AI Act, to dual-graph architectures such as GraphCompliance [14], which aligns the policy and context graphs for GDPR scenarios, to multi-agent systems such as RAGulating Compliance [15] and AgCyRAG [16]. Further work includes the Neo4j-based Regulatory KG by Ershov [17] and the ForPKG framework [18]. ForPKG is one of the few systems to quantify its extraction quality with precision (76.2%) and recall (62.6%). Relevant contributions are emerging from related fields. CyberKG [19] uses a SecureBERT Plus-BiLSTM-Attention-CRF pipeline for cybersecurity KG construction and reports a macro-level entity-normalization F1 of 84.1% for HAC-based clustering of synonymous CTI entities against human-annotated clusters, and the AEVS framework [20] establishes an anchor-based verification method in which extracted triples are deterministically validated against source text anchors. The validation ensures the origin of the triples in the source text but does not check their topological correctness against a GT. Despite this diversity, most systems do not evaluate the quality of the extracted graph itself. In particular, they rarely measure extraction quality using hard metrics such as precision, recall, and F1 score at the node or edge level. For example, CO2 [13] provides a qualitative proof of concept, RAGulating Compliance [15] reports no precision and recall for their triple extraction, Ershov [17] demonstrates the graph via Cypher queries, and PrivComp-KG [7] uses only a global correctness score. AgCyRAG [16] similarly presents three qualitative use cases, builds a KG and then evaluates only the quality of downstream tasks such as question answering (Q&A), without measuring the extraction quality of the graph itself. If a RAG agent answers a question correctly, the underlying graph is recorded as a “success”. This represents a methodological weakness, as in regulated domains the graph is not just a heuristic search aid for RAG systems, but the factual basis upon which audits are based. The principle of accountability, as required by the BSI and NIS-2 [1], demands that the verification process itself be auditable. A query result based on a faulty graph is of no regulatory value, as it hides topological flaws instead of revealing them. While advanced systems such as ComplianceNLP [5] rigorously evaluate the quality of regulatory semantic crossreferences (F1 90.8%), we are not aware of prior work that evaluates the structural correspondence of extracted IT-SC graphs against the company’s GT. This structural correspondence involves the mathematical verification of the nodes and edges of an extracted infrastructure graph (GDoc ) against the validated infrastructure (GGT ). This differs from the evaluation of semantic cross-references, as it does not concern linguistic references between paragraphs, but rather the correctness of actual infrastructure topologies. The ASSERT
framework operationalizes this requirement through the deterministic graph comparator that calculates the graph difference ∆G = (VO , VP , EO , ET , EG ) as class-specific errors sets, thereby quantifying missing nodes, hallucinated nodes, and topologically incorrect edges. C. Positioning ASSERT ComplianceNLP [5] operates exclusively within the forward paradigm for financial regulation and evaluates semantic cross-references, not infrastructure topologies. GraphCompliance [14] is GDPR-specific and validates graph quality via internal reconstruction stability (cycle consistency), in which a graph is iteratively converted to text and back, rather than by comparing it to an external infrastructure GT. CO2 [13] evaluates structural graph characteristics (connectivity, density), but does not evaluate the accuracy of the extraction against a GT. Although AEVS [20] follows a conceptually related anchor-based approach with deterministic restoration matching, it operates on a different ontological level. While AEVS validates extracted triples against the source text itself to ensure extraction accuracy, it primarily solves the linguistic NLP problem of making the graph a faithful representation of the potentially erroneous source. ASSERT, in contrast, addresses the broader audit challenge of verifying whether the constructed representation is aligned with an independently validated infrastructure reference graph (GGT ). Since textual mappings do not imply factual accuracy, AEVS remains blind to substantive errors or outdated information within the legacy IT-SC itself. ASSERT overcomes this limitation using external mappings against the infrastructural GT. As a domain-independent framework, AEVS is not specialized for IT security. ASSERT thus combines all three aspects: reverse engineering of legacy IT-SC, deterministic graph-level verification, and schema-valid OSCAL outputs. IV. M ETHODOLOGY: T HE ASSERT F RAMEWORK A. The ASSERT Framework Architecture ASSERT takes a legacy IT-SC and the infrastructure reference graph GGT from a preceding SA as input, constructs the document graph GDoc , and verifies GDoc against GGT , as shown in Fig. 1. ASSERT treats the legacy IT-SC as potentially inconsistent and GGT as the trusted reference basis. It detects structural discrepancies after extraction, but does not validate incomplete or poisoned reference data or control effectiveness. The framework consists of three components: 1. The Information Extraction Component: This component constructs GDoc from the legacy IT-SC by extracting entities and relationships under the selected ontology exposure level. Since Grundschutz-based IT-SCs follow strict chapter guidelines, ASSERT uses this existing structure for ontology-based information extraction. 1. Hierarchical Chunking & Extraction: To ensure the traceability of each extracted entity, the legacy IT-SC is modeled as a hierarchical tree when ingested into the graph. Chunks contain on average ≈ 3, 700 tokens and preserve document structure. Oversized chapters are split at sections, tables, or
Legacy IT-SC
GGT from SA
HITL Adj.
Information Extraction Component • Hierarchical Provenance Chunking (Document → Chapter → Paragraph) • LLM Extraction (Generic/Schema-Guided/Schema-Enforced) • Deterministic Node Alignment → Exception List GDoc
Dual-Graph Comparator (∆G) Stage 1 (Nodes): VO , VP , VM atched Stage 2 (Edges, cascading): EO , ET , EG ∆G
OSCAL Export Module M ∩ EM ) GV erif ied = (VM atched , EGT Doc Filter VP , EG , ET → OSCAL SSP + AR GV erif ied
OSCAL SSP + AR
Fig. 1. Architecture of the ASSERT framework for transforming legacy ITSC into schema-valid OSCAL artifacts. Dashed lines mark the optional HITL review, and GGT is the verified infrastructure GT of the SA from [3].
paragraphs, without sliding-window overlap. An LLM extraction step operates the chunks at the selected ontology exposure level. The extraction validates the LLM response against a Pydantic schema. In Schema-Enforced mode, a deterministic post-hoc step filters against the canonical entity list from GGT . 2. Automated Node Alignment: The node alignment between GDoc and GGT follows a cascading, strictly lexical procedure to rule out stochastic misalignments and ensure the reproducibility of the difference calculation in contrast to HybridRAG approaches [3], [21]. A match is defined in three priority levels: case-insensitive identity of the identifiers (1), identity of the normalized abbreviations (2), mutual substring inclusion of name or alias (3). This assumes the naming consistency expected in BSI IT-Grundschutz documents derived from structured asset tables. Matches that are successfully identified are deduplicated and added to the set of matching nodes VM atched used by the comparator. Mentions that do not match at any of these levels are moved to the exception list VExc and reviewed by a Human-in-the-Loop (HITL). 3. Human-in-the-Loop Review: The final decision regarding the entities on the exception list is delegated to the HITL. Since VExc contains all unresolved mentions before review, it quantifies the mention-level HITL workload of a given run. For each ambiguous entity, the expert is shown the original text fragment. The expert can verify the context and decide to assign the entity to an existing asset or to mark it as incorrect or new. Consequently, the exception list addresses ontological 1:N mismatches between the text and GGT , which the HITL resolves before they are incorrectly reported as ∆G. 2. The Dual-Graph Comparator: In this component, GDoc and GGT are compared algorithmically. In accordance with the defined cascading logic, the comparator first determines the sets VO , VP , and VM atched at the node level through deterministic string matching against GGT . It then calculates, at the
The ASSERT framework is evaluated on the RecPlast dataset using three configurations with different levels of reference-ontology exposure: Generic, Schema-Guided, and Schema-Enforced. The ASSERT pipeline uses LangChain for orchestration with Pydantic-typed outputs. To investigate the model dependency of the results, inference is performed using Ollama with the local open-weight model Gemma 4 26B and the commercial model Anthropic Claude Opus 4.7. The generated SSP and AR are validated against the OSCAL v1.1.3 schemas.
|VM atched |/|VGT | = |VM atched |/(|VM atched | + |VO |). F1,V is defined as the harmonic mean of PV and RV . Edge-level metrics: Edge evaluation is restricted to the subgraphs induced by VM atched to avoid penalizing missing nodes twice. As defined above, the edge error classes EO , ET , and EG use deduplicated directed endpoint tuples (u, v). For edge-quality metrics, relation-label agreement is additionally M M required. Let EGT,r and EDoc,r denote the matched-nodeinduced edge sets over deduplicated directed triples (u, r, v), where r is the stored relation type after case and whitespace normalization and malformed or out-of-schema labels remain M M nonmatching. Consequently, EM atch,r = EDoc,r ∩ EGT,r , M M PE,r = |EM atch,r |/|EDoc,r |, RE,r = |EM atch,r |/|EGT,r |, and F1,E,r is their harmonic mean. Class-specific metrics: F1 scores are calculated for the five error classes (VO , VP , EO , ET , EG ) in the fault injection evaluation, as shown in Table II, where the k = 20 injected errors define the GT. Whether a unit is considered a match depends on the error class: VO is matched via infrastructure node IDs, VP via normalized entity names, and edge errors (EO , ET , EG ) via directed endpoint pairs. As false positives, only additional detections compared to the unmodified RecPlast ITSC (baseline) are counted. Such class-specific metrics are not defined on the baseline IT-SC because the actual distribution of inconsistencies is not known. The baseline counts in Table I are therefore reported separately from class-specific P/R/F1 values. Baseline edge-error counts use endpoint tuples, and edge metrics use typed triples.
A. Dataset - RecPlast GmbH
C. ASSERT Extraction Configurations
The evaluation of ASSERT requires a public dataset from which a validated reference graph can be reconstructed. Since real-world IT-SCs are generally not publicly available due to non-disclosure agreements (NDAs), this study uses the expertgenerated “RecPlast GmbH” dataset [22] published by the BSI. The dataset covers the full IT-Grundschutz certification process chain, including the initial organizational and infrastructure descriptions, the intermediate SA artifacts, the final 69page IT-SC, and more. This allows us to reconstruct GGT from the intermediate artifacts, construct GDoc independently from the final IT-SC, and compare both graphs under reproducible conditions.
To assess how ontology exposure affects information extraction, the ASSERT framework is evaluated in three configurations. These are defined as follows: ASSERT Generic: The LLM extracts entities and relationships solely on the basis of the seven abstract entity classes from the BSI SA (IT system, application, business process, room, communication link, IoT system, ICS system), without any knowledge of the specific component names or dependencies within the company. The LLM thus recognizes the categories but not the instances of the infrastructure. This configuration serves as a methodological baseline, as it isolates the LLM’s document-based extraction capability and quantifies the effect of reference-ontology exposure. ASSERT Schema-Guided: The LLM is provided with the complete ontology of the reference graph GGT , including all specific entity names with their types and abbreviations, as well as all known relations with their dependency types. The LLM is instructed to use the known component names exactly as they appear in the text, but is also permitted to extract entities that are not listed in the ontology. The comparison with Generic quantifies how reference-ontology exposure affects recall and precision while preserving non-reference extraction. ASSERT Schema-Enforced: The schema is strictly enforced on two levels. First, the prompt instructs the LLM to extract only entity and relation types defined in the GGT schema. Second, a deterministic post-hoc filter removes all
edge level (conditional on VM atched ), the three edge classes of the compliance difference ∆G = (VO , VP , EO , ET , EG ) over directed (u, v)-tuples: missing edge tuples (EO ) via the difference EGT \ EDoc over VM atched , unsupported document edge tuples (ET ) via the difference EDoc \EGT over VM atched , and ghost edges (EG ) by filtering all edges with at least one phantom node from VP . 3. The OSCAL Export Module: ASSERT separates system description from verification evidence. The verified component set VM atched and BSI Grundschutz++ control implementation statements are assembled into an SSP, while an AR records graph-difference findings and source-linked evidence. Because the published BSI Grundschutz++ OSCAL artifacts provide the requirement catalog but no profile for the claimed control subset, ASSERT uses a minimal local profile and Assessment Plan to connect the catalog, SSP, and AR. V. DATASET AND E XPERIMENTAL S ETUP
B. Evaluation Metrics Precision, recall, and F1 score are computed at the node and edge levels and for specific fault classes. In the baseline setting, ∆G denotes the diagnostic delta sets produced by the graph comparator, and class-specific detection scores are defined only for the fault-injected variants. Node-level metrics: The vertex sets are evaluated at the entity level. VM atched counts the unique GT nodes that have received at least one aligned extraction, and VP counts the unique phantom entities after deduplication via (name, type). VDoc stays at mention level and reflects extraction and HITL workload. This results in PV = |VM atched |/(|VM atched | + |VP |), and RV =
Generic
Guided
Enforced
Best use
Discovery baseline
Balanced analysis
Phantom-free OSCAL export
Strength
Finds nonreference assets
Improves edge/ conflict extraction
Removes phantom entities by design
Noisy output; high HITL burden Many edges remain missing Unknown assets matter
Model-dependent; ontology leakage
No discovery; masks shadow IT Clean nodes, low edge recall Export validity matters
Problem Edge Use when
Best edge trade-off Model fidelity is reliable
Fig. 2. Evaluation-derived comparison of ASSERT extraction modes.
extracted entities whose names do not appear in the canonical list. This dual strategy eliminates non-canonical phantom nodes (VP = 0) by construction. However, this simultaneously suppresses the detection of entities that exist in the real infrastructure but are not modeled in GGT , e.g., shadow IT. For all ASSERT configurations, ∆G is computed with the same deterministic comparator against GGT , ensuring metric comparability across all configurations. To quantify model dependence, all configurations are evaluated with both LLMs. D. Fault-Injected Classes (VO , VP , EO , ET , EG ) The fault-injected evaluation allows an isolated assessment of the framework’s class-specific sensitivity. Therefore, the baseline IT-SC is selectively altered, and the framework is run against the modified IT-SCs. In the single-fault variants, only one error class is introduced per run, so that the measured detection rates can be causally attributed to that specific class. The writing style, terminology, and contextual structure of the audit report remain unchanged. Specifically, for VO , k = 20 entities and their text contexts are removed from the baseline IT-SC. For VP , we inject fictional entities from a curated list, e.g., the mention of a non-existent “cloud backup”. For EG , we inject relations incident to phantom entities, such as an alleged connection from the fictional entity to the production network. For EO , we delete sentences that explicitly name edges, with the constraint that both endpoints co-occur in the same sentence and both endpoints fall within the entity intersection of all three ASSERT configurations. For ET , the causality of k = 20 edges explicitly named in the text is manipulated. k = 20 is the largest feasible RecPlast intersection size, covering approximately 15% of GGT , creating substantial but not syntactically destructive document alteration. The mixed variant (Vmixed ) injects k ′ = 5 errors per class, resulting in 25 simultaneous manipulations to test robustness under coexisting error types. VI. R ESULTS & D ISCUSSION The baseline IT-SC compares the three configurations across both LLMs and quantifies the resulting node- and edge-level graph differences, as shown in Table I. The fault-injected variants isolate the framework’s sensitivity to specific error classes under controlled document manipulations, as shown in Table II. ASSERT Configurations: Table I and Fig. 2 show that the
TABLE I E XTRACTION METRICS FOR ASSERT CONFIGURATIONS . Gemma 4 Opus 4.7 Class Generic Guided Enforced Generic Guided Enforced VDoc 321 586 222 123 126 134 VM atched 95 132 123 120 118 130 VExc 134 182 0 0 4 0 VO 39 2 11 14 16 4 VP 133 172 0 0 4 0 EO 469 637 731 645 621 732 ET 16 33 4 0 22 0 EG 35 49 0 0 15 0 PV 0.417 0.434 1.000 1.000 0.967 1.000 RV 0.709 0.985 0.918 0.896 0.881 0.970 F1,V 0.525 0.603 0.957 0.945 0.922 0.985 PE,r 0.133 0.843 0.931 0.891 0.687 0.895 RE,r 0.008 0.261 0.069 0.147 0.137 0.137 F1,E,r 0.016 0.399 0.128 0.253 0.229 0.238 Results for Gemma 4 and Opus 4.7 across ASSERT configurations, reporting node counts, endpoint-tuple edge-error counts, and typed-triple edge metrics.
three configurations serve different operational roles. Generic preserves discovery, but creates substantial HITL workload for the Gemma model. Schema-Guided improves edge-related extraction, but the benefit is model-dependent. For Gemma, Schema-Guided reduces VO from 39 to 2 and increases recall (RV = 0.709 → 0.985), but this comes at the cost of massive over-extraction, as VDoc increases from 321 to 586 and VP from 133 to 172, leaving precision low (PV = 0.434). This indicates stronger alignment with GGT rather than documentonly extraction. The Opus model only shows marginal overextraction (VP = 4 in Schema-Guided, VP = 0 otherwise) and maintains PV ≥ 0.967 across configurations, suggesting that the phantom problem is primarily model-dependent instruction fidelity. Schema-Enforced achieves high node-level F1,V = 0.957 for Gemma and F1,V = 0.985 for Opus and removes phantom nodes and ghost edges by construction, making it suitable for an OSCAL export. However, VP = 0 is ambiguous, as it eliminates both hallucinations and discovery capability. A shadow IT asset absent in GGT is systematically ignored, since the model can only emit canonical-list entities. The high EO counts complement this. A typical ITSC states “Server A runs App B”, but does not mention every dependency recorded in the tabular SA. Despite high node performance, typed edge recall remains low across all configurations (RE,r ≤ 0.261) because the IT-SC identifies only a subset of SA dependencies. Typed-edge performance therefore remains model-dependent, and unrestricted extraction can generate unverifiable relations. Pipeline Accuracy under Fault Injection: Table II reports class-specific F1 scores for single-fault and mixed variants, isolating extraction accuracy from document completeness. Generic reliably captures phantom classes if the model is obedient to instructions. Opus achieves F1 = 0.889/0.884 for VP /EG , while Gemma falls behind with F1 = 0.488/0.526 due to lower instruction adherence. By design, Enforced achieves F1 = 0 for VP and EG . By definition, the dual strategy of prompt constraints and post-hoc filtering prevents phantom detection. ET remains the most challenging error class, with a substantial model gap. In the Schema-Guided configuration,
TABLE II FAULT- INJECTED EVALUATION . Gemma 4 Opus 4.7 Class k Generic Guided Enforced Generic Guided Enforced Single-Fault Classes VO 20 0.667 0.258 0.556 0.645 0.597 0.526 VP 20 0.488 0.741 0.000 0.889 1.000 0.000 EG 20 0.526 0.630 0.000 0.884 1.000 0.000 EO 20 0.109 0.200 0.082 0.200 0.246 0.000 ET 20 0.000 0.303 0.000 0.086 0.710 0.102 Vmixed (simultaneous injection of multiple classes) VO 5 0.556 0.049 0.000 0.417 0.333 0.074 VP 5 0.137 0.066 0.000 1.000 1.000 0.000 EG 5 0.118 0.200 0.000 0.286 1.000 0.000 EO 5 0.029 0.049 0.061 0.068 0.080 0.000 ET 5 0.167 0.000 0.250 0.000 0.750 0.000 Rows denote injected error classes. Columns show class-specific F1 scores for (Generic / Schema-Guided / Schema-Enforced) under Gemma 4 and Opus 4.7. Vmixed uses simultaneous k′ = 5 injections per class.
Opus 4.7 achieves F1 = 0.710, while Gemma 4 lags behind with F1 = 0.303. This twofold F1 gap quantifies the trade-off between data sovereignty (local LLM) and detection capability on dense SA tables. Even with Opus, topological-conflict detection remains incomplete. ASSERT does not resolve the underlying reasoning limitation, but makes its occurrence auditable through deterministic graph matching. Schema-Guided has opposite effects on the two LLMs. For Opus, it is the most balanced configuration, with perfect phantom detection, VO is slightly below Generic at F1 = 0.597, EO increases slightly (0.200 → 0.246), while ET benefits massively (0.086 → 0.710). Conversely, Gemma Schema-Guided reveals empirically verifiable ontology-induced hallucination, as 16 of 20 omissions go unnoticed. The VO F1 score drops from 0.667 (Generic) to 0.258, and recall drops from 0.700 to 0.200. A manual analysis showed that, for 12 of the 16 unrecognized VO omissions, Gemma hallucinates the removed node back into the output under reference-ontology exposure. Once the document deviates from the reference ontology GGT , the trade-off not observed on the baseline IT-SC emerges. For Gemma, the VO F1 score drops from 0.667 (Generic) to 0.556 (SchemaEnforced). Recall increases slightly (0.750 vs. 0.700), but precision drops from 0.636 to 0.441 because the rigid lexical filter generates pseudo-omissions for textually divergent identifiers. For EO , Enforced degrades significantly (Gemma recall R = 0.200, Opus recall R = 0.000) because the canonical filter classifies the EO targets in the entity intersection as F N . The convergence of both LLMs in Enforced configuration on the baseline IT-SC is therefore a RecPlast artifact, as the LLMs diverge again due to wording differences. The Vmixed variant exhibits a robustness asymmetry. Opus maintains stable ET detection under simultaneous manipulations (F1 = 0.750 vs. 0.710 for single-fault), while Gemma fails (F1 = 0.000 vs. 0.303). The breakdown manifests in schema violations such as hallucinated relation types (144 of 183 relations with the literal value “relation type”), phantom mass extraction (FP = 142 for VP ), and typos such as “depends onn”. Multiple simultaneous error classes appear to undermine the LLM’s instruction fidelity, leading to a breach of schema con-
formity. This limits the suitability of fully local deployments for CI audits with complex dependency structures and shows that single-fault conditions underestimate the robustness gap between local and cloud-based LLMs. Beyond its diagnostic value, the Gemma Vmixed Schema-Guided run provides a controlled stress case for the HITL interface, yielding the largest exception list observed in our experiments (VExc = 207 mentions). The results narrow down the mode selection to a trade-off between discovery and reference control. Generic retains discovery capability but requires HITL to distinguish genuine new discoveries from hallucinations. Schema-Guided is most balanced when instruction obedience is high (Opus), but can intensify ontology-induced hallucinations when it is low (Gemma). Schema-Enforced eliminates phantom entities by design (VP = 0, EG = 0), but excludes discovery and degrades VO and EO under wording deviation. Thus, the configuration selection mainly depends on model instruction adherence and the desired degree of discovery versus reference control. OSCAL Output Module: As a proof of concept, SchemaEnforced outputs were exported into OSCAL SSP and AR artifacts. The SSPs contain 222 (Gemma) and 134 (Opus) components at the mention level, corresponding to 123 and 130 unique GT entities, while the ARs include graph-difference findings and source-linked evidence. With VP = 0, the SSP export is free of non-canonical phantom entities but is not discovery-capable. VII. L IMITATIONS & F UTURE W ORK Naturally, ASSERT remains dependent on the quality of its input artifacts. Undocumented assets and shadow IT cannot be extracted or flagged as missing. Errors or inconsistencies in GGT propagate directly into ∆G. ASSERT therefore shifts part of the verification burden to establishing and maintaining a reliable reference graph. Reference-ontology exposure can additionally mask document deviations when models reconstruct missing document evidence from GGT . Future work should investigate industry-specific expectation ontologies and methods for validating GGT across multiple operational sources, confidence scores, and alternative comparison strategies. Further, the evaluation is limited by the RecPlast dataset and the chosen fault injection method. Typed edge metrics are likewise limited to relation labels represented in the reconstructed reference graph. RecPlast is the only fully public, expert-generated ITGrundschutz dataset, but it does not allow for broad generalizations to diverse real-world IT-SCs. The fault injection size (k = 20, k ′ = 5) is constrained by the entity overlap in RecPlast. Since runs were not repeated across seeds, the results should be interpreted as a proof-of-concept evaluation rather than a statistically conclusive benchmark. From a practical perspective, the Schema-Enforced configuration is only suitable as a complementary filter, since it prevents phantom entities but may mask real infrastructure not included in GGT . Future work should include robust multi-fault evaluations, schema-following tuning, and cascaded multi-pass validation for local models, and neuro-symbolic methods for the deter-
ministic resolution of remaining ET conflicts [4]. The OSCAL export should be expanded from a schema-valid artifact to a semantic cross-catalog alignment between IT-Grundschutz and Grundschutz++. While ASSERT is currently aligned with BSI IT-Grundschutz, OSCAL suggests broader applicability beyond German regulatory scenarios. Future work should investigate adaptations to other OSCAL-representable control catalogs, such as NIST SP 800-53 or ISO/IEC 27001. The exception list ranges from 0 items (Opus Generic/Enforced) to 182 on the baseline RecPlast IT-SC and reaches 207 in the Gemma Schema-Guided stress case. These counts quantify the potential HITL workload, while actual review time and cognitive load remain subject to controlled user evaluation. VIII. C ONCLUSION ASSERT addresses the migration of text-based legacy ITSCs to machine-readable compliance artifacts by separating graph-verified content from documented inconsistencies before export. During the multi-year transition period between ITGrundschutz and Grundschutz++, it ensures consistency between narrative IT-SC and expert-verified infrastructure reference graphs GGT . Formal document graphs (GDoc ) are constructed from IT-SCs under different levels of referenceontology exposure, deterministically aligned with GGT , and discrepancies are made auditable as graph differences (∆G). The RecPlast evaluation shows that ASSERT detects topological conflicts against the reconstructed reference graph, with model-dependent performance. The fault-injected evaluation highlights the trade-off between cloud-based LLM quality and on-premises data sovereignty, as well as a robustness asymmetry among coexisting error classes. The schema-valid OSCAL SSP and AR outputs separate verified system content from verification evidence and demonstrate syntactic compatibility with OSCAL-based CaC workflows. ASSERT thereby provides a deterministic comparison and migration mechanism for ontology-based compliance graphs in regulated audit scenarios. Beyond detection quality, the exception-list provides an observable proxy for potential HITL review burden, while actual review time and cognitive load remain to be evaluated. R EFERENCES [1] European Parliament and Council, “Directive (EU) 2022/2555 (NIS-2 Directive),” Official Journal of the European Union, vol. L333, pp. 80– 152, Dec. 27, 2022. [Online]. Available: http://data.europa.eu/eli/dir/2 022/2555/oj [2] National Institute of Standards and Technology (NIST), “OSCAL: the Open Security Controls Assessment Language,” Mar. 26, 2026. [Online]. Available: https://pages.nist.gov/OSCAL [3] L. R. Muth and M. Margraf, “Probabilistic agents in deterministic audits: Evaluating multi-agent systems for automated audits based on the German IT-Grundschutz,” in Proc. 20th Annual IEEE International Systems Conference (SysCon), Halifax, NS, Canada, Apr. 2026, doi: https://doi.org/10.1109/SysCon66367.2026.11503560. [4] Y.-S. Hsia, F. Yu, and J.-H. R. Jiang, “Neuro-symbolic compliance: Integrating LLMs and SMT solvers for automated financial legal analysis,” in Proc. 2nd IEEE/ACM International Conference on AI-powered Software (AIware), Seoul, South Korea, pp. 1–10, Nov. 2025, doi: https://doi.org/10.1109/AIware69974.2025.00010.
[5] D. Guo, J. Wu, and S. M. Yiu, “ComplianceNLP: Knowledge-graphaugmented RAG for multi-framework regulatory gap detection,” Proc. ACL 2026 Industry Track, in press, arXiv:2604.23585, Apr. 2026, doi: https://doi.org/10.48550/arXiv.2604.23585. [6] D. Jovanovski, M. Stojcheva, M. Dodevska, P. Lameski, I. Mishkovski, and D. Gjorgjevikj, “An empirical study of knowledge graph-enhanced RAG for information security compliance,” Information, vol. 17, no. 4, art. 389, Apr. 2026, doi: https://doi.org/10.3390/info17040389. [7] L. Garza, L. Elluri, A. Piplai, A. Kotal, D. Gupta, and A. Joshi, “PrivComp-KG: Leveraging KG and LLM for compliance verification,” in Proc. 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pp. 97–106, 2024, doi: https://doi.org/10.1109/TPS-ISA62245.2024.00021. [8] P. Guldimann et al., “COMPL-AI framework: A technical interpretation and LLM benchmarking suite for the EU Artificial Intelligence Act,” arXiv:2410.07959, Oct. 2024, doi: https://doi.org/10.48550/arXiv.2410. 07959. [9] M. Prandi, V. Suriani, F. Pierucci, M. Galisai, D. Nardi, and P. Bisconti, “Bench-2-CoP: Can we trust benchmarking for EU AI compliance?,” arXiv:2508.05464, Aug. 2025, doi: https://doi.org/10.48550/arXiv.250 8.05464. [10] T. R. McIntosh et al., “From COBIT to ISO 42001: Evaluating cybersecurity frameworks for opportunities, risks, and regulatory compliance in commercializing large language models,” Computers & Security, vol. 144, art. 103964, 2024, doi: https://doi.org/10.1016/j.cose.2024.103964. [11] S. Kambhampati, “Can large language models reason and plan?,” Annals of the New York Academy of Sciences, vol. 1534, no. 1, pp. 15–18, Mar. 2024, doi: https://doi.org/10.1111/nyas.15125. [12] K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati, “On the planning abilities of large language models: A critical investigation,” in Proc. 37th International Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, art. 3320, pp. 75993– 76005, Dec. 2023. [Online]. Available: https://dl.acm.org/doi/10.5555/3 666122.3669442. [13] V. S. P. Turaga, T. Pahi, S. Tjoa, S. Siami-Namini, and A. Siami Namin, “CO2 (Co-compliance officer): An LLM-based ontology-driven methodology for generating knowledge graphs and AI compliance checking,” IEEE Access, 2025, doi: https://doi.org/10.1109/ACCE SS.2025.3639228. [14] J. Chung et al., “GraphCompliance: Aligning policy and context graphs for LLM-based regulatory compliance,” arXiv:2510.26309, Oct. 2025, doi: https://doi.org/10.48550/arXiv.2510.26309. [15] B. Agarwal, H. S. Jomraj, S. Kaplunov, J. Krolick, and V. Rojkova, “RAGulating Compliance: A multi-agent knowledge graph for regulatory QA,” arXiv:2508.09893, Aug. 2025, doi: https://doi.org/10.48550 /arXiv.2508.09893. [16] K. Kurniawan, R. F. Ardian, E. Kiesling, and A. Ekelhart, “AgCyRAG: An agentic knowledge graph based RAG framework for automated security analysis,” in Proc. Second International Workshop on RetrievalAugmented Generation Enabled by Knowledge Graphs (RAGE-KG 2025), Nara, Japan, CEUR-WS, vol. 4079, pp. 132–144, Nov. 2025. [Online]. Available: https://ceur-ws.org/Vol-4079/paper11.pdf [17] V. Ershov, “A case study for compliance as code with graphs and language models: Public release of the regulatory knowledge graph,” arXiv:2302.01842, Feb. 2023, doi: https://doi.org/10.48550/arXiv.2302. 01842. [18] J. Sun and Z. Luo, “ForPKG: A framework for constructing forestry policy knowledge graph and application analysis,” in Proc. 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2025, doi: https://doi.org/10.1109/IJCNN64981.2025.11228178. [19] B. Li, Q. Yang, C. Deng, and H. Pan, “CyberKG: Constructing a cybersecurity knowledge graph based on SecureBERT Plus for CTI reports,” Informatics, vol. 12, no. 3, art. 100, Sep. 2025, doi: https: //doi.org/10.3390/informatics12030100. [20] Y. Yang, G. Chen, B. He, and Y. Zhao, “Grounded knowledge graph extraction via LLMs: An anchor-constrained framework with provenance tracking,” Computers, vol. 15, no. 3, art. 178, Mar. 2026, doi: https: //doi.org/10.3390/computers15030178. [21] B. Sarmah, B. Hall, R. Rao, S. Patel, S. Pasquali, and D. Mehta, “HybridRAG: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction,” arXiv:2408.04948, Aug. 2024, doi: https://doi.org/10.48550/arXiv.2408.04948. [22] Bundesamt für Sicherheit in der Informationstechnik (BSI), “RecPlast GmbH.” [Online]. Available: https://recplast-gmbh.de