arXiv CSopen access

From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability

From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability

arXiv:2606.05252v1 [cs.CR] 3 Jun 2026

Alexandre Cristovão Maiorano [email protected] Abstract Security teams routinely simulate attacks against their own systems to check whether their monitoring would catch a real intruder. These Breach-and-Attack-Simulation (BAS) tools surface findings, but the security information and event management (SIEM) systems that watch production need detection rules — and today a human bridges that gap by hand, reading each finding and writing the corresponding Sigma rule (a vendor-neutral detection format). We show this translation can be partially automated when probes are drawn from a locked corpus, so each finding carries a stable identifier back to the originating probe. We describe a deterministic synthesis function that maps each finding to a starter Sigma rule through a small template library (N =23, indexed by categories from the OWASP LLM and Web Top 10), with a back-reference to the originating finding and its MITRE ATT&CK technique. On two locked corpora (17-probe LLM, 23-probe Web), every bypassed-probe finding yields a starter rule, and all 17/17 emitted rules parse and convert to Splunk and Elasticsearch backends. Replayed through a live OpenSearch SIEM, the LLM rules fire on 30% of a held-out AdvBench subset and 14% of HarmBench at 7.7% false positives on a benign baseline; the Web side is validated structurally, not against a held-out attack set. The contribution is a verifiable, byte-stable path from BAS finding to operator-deployable starter rule, re-derivable from the published corpus and template library alone — trading the breadth of LLM-generative methods for exact reproducibility and a typed traceback from any fired alert to the originating probe.

1

Introduction

move toward locked probe corpora — published JSON files whose every entry has a stable probe id that does not change between runs [2]. Second, the move toward OWASP-aligned categorization [5] — OWASP being the community body whose Top 10 lists enumerate the most critical security risks for web and LLM applications — where each probe declares its OWASP LLM Top 10 (or Web Top 10) category up front and tags itself with the corresponding technique from MITRE ATT&CK [10], the standard public knowledge base of adversary tactics and techniques. These two together mean a finding now carries enough structure to look up a template-shaped rule deterministically: finding.metadata.probe id → corpus entry → OWASP category → Sigma template. This paper contributes the following:

A typical BAS run produces dozens to hundreds of findings. Each finding describes what the simulator was able to do against the target. The next step in a security operations center (SOC) workflow is to ask: would we have detected this in production? — and if the answer is no, write a detection rule, typically in the Sigma format [7] — a vendor-neutral YAML schema for detection rules — so it can be ported across security information and event management (SIEM) backends. Today that translation is manual. The finding’s textual description (detection gap, defense recommendation) gives the analyst enough context to author a rule, but they still have to choose the data source, write the regex/keyword logic, attach the right MITRE ATT&CK tag, and decide on a sensitivity threshold. The result is craftwork: high quality when the analyst is good, but slow, expensive, and dependent on which analyst sat down with the finding. Two recent developments in the BAS-research literature change the shape of the problem. First, the

1. A template library of 23 Sigma rule skeletons (8 MITRE T-code legacy templates + 9 OWASP LLM Top 10 + 6 OWASP Web Top 10; 19 exercised by the two released corpora, 4 reserved for future corpora covering A04, LLM03, LLM04, LLM05) that resolve findings emitted by the locked corpora to starter detection rules. The 1

library metadata and the synthesis contract that reading guide that disambiguates each measurement consumes it are published as pseudocode in the context. replication package, under the same license as Why this matters beyond the immediate time savthe corpora. ing: determinism and probe-level traceability are what let auto-generated detection content be governed like 2. A traceability contract: every emitted rule code — version-controlled, diffed across corpus revicontains two reference URIs — one to the origi- sions, reviewed, and signed off with an unambiguous nating finding and one to the MITRE technique. owner — rather than treated as opaque output a Combined with the finding’s stable probe id and SOC must re-verify on every run. That is the disa content-hashed record of the run that produced cipline the “detection-as-code” movement asks for, it, this gives a reviewer a deterministic path from and it is precisely what generative approaches, whose rule back to the corpus entry that motivated it. output changes with the model release or prompt, cannot guarantee. A reproducible finding-to-rule path 3. A reproducibility envelope: an external re- is therefore a prerequisite for trusting automation in viewer can re-derive every emitted rule from the detection engineering, not merely a convenience. published locked corpus, the published template This paper is positioned in the gap between BAS library, and the deterministic synthesis function. surface results and ready-to-deploy SIEM content. We The protocol is detailed in Section 3.4; we do not do not claim that auto-generated rules are productionrequire the reader to run our specific software ready. The system is designed to shorten the manualstack to verify any claim. authoring loop by giving the analyst a typed, referenceable starting point with the corpus probe baked 4. An operational measurement of coverage on in; we do not run a user study or a time-on-task the two released corpora plus a real-SIEM replay: trial in this paper, so the workflow speed-up claim across 17 LLM probes and 23 Web probes, every is supported structurally (per-rule synth time of milbypassed-probe finding produced a starter rule liseconds plus a few minutes of analyst review, against (skipped=0). The emitted rules ingest into a live a 30–60-minute manual baseline; break-even at ∼4–8 OpenSearch [11] index via the Lucene backend findings per category, see Section 5.6) rather than and fire on 15/50 of a held-out AdvBench [13] sub- empirically. A controlled user study is the natural set and 7/50 of a second held-out HarmBench [3] follow-up. subset, with 7.7% FP on a benign-LLM baseline. The measurement protocol is detailed in Section 3 and the results are reported in Sections 4.5 2 Related Work and Positioning and 4.7. Four bodies of work are relevant. We summarize positioning in Table 1 and elaborate below. From v1 to v2: rubric evolution. The version labels v1 and v2 in this paper refer only to the templateauthoring rubric — how a template’s detection logic is BAS calibration with locked corpora. Our comwritten — and not to the engine’s feature scope, which panion work [2] establishes the per-defense attribution is held fixed throughout. The generalization results methodology that produces the findings consumed in contribution (4) reflect an iterative improvement here. We extend the corpus-locked discipline from the of that rubric. Our first template rubric (v1) used engine side (probe → finding) to the defense side (findkeyword-only matching and achieved a 0/50 fire rate ing → rule): the same sha256-pinned probe corpus on the AdvBench held-out set — a baseline failure that that anchors that paper’s attribution measurements motivated a v2 authoring rubric adding regex selection anchors our rule synthesis, and the synthesis function on semantic markers. A Python prototype of the v2 is a pure mapping from (finding, template) to Sigma rubric achieved 31/50 (62%); after integration into the YAML. engine, the v2 rubric achieved 30/50 (60%) in live synthesis, and 15/50 (30%) when replayed through a real LLM red-team frameworks. garak [1] is the closOpenSearch SIEM (the drop attributable to a backend est peer for the LLM-side calibration scenario: it compatibility shim, as discussed in Section 5.7). The ships locked probes keyed to OWASP LLM Top 10 + numbers reported in this paper therefore represent a MITRE ATT&CK and produces per-probe verdicts. progression: v1 (keyword-only, 0/50), v2-prototype garak does not emit detection content; the operator (Python, 31/50), v2-integrated (engine, 30/50), and reads a result table and writes rules manually. Our v2-integrated (SIEM, 15/50). Section 4 provides a synthesis function is the missing back-half — given 2

garak-style findings, emit Sigma rules deterministi- step combine the two strengths and are the natural cally. composition, not a competing design. SIEM rule libraries (hand-authored). 3 Methodology SigmaHQ [8] maintains ∼3,000 community-authored Sigma rules. The library is excellent but the authoring Figure 1 shows the round-trip the rest of this section workflow is exactly the manual step we automate: formalizes: a locked corpus drives the BAS engine — a human reads a vuln advisory or red-team report, any tool that fires each probe against a target appliwrites a Sigma YAML, and submits a PR. We do cation and labels the outcome as bypassed or blocked not compete with SigmaHQ’s depth (their rules — each bypassed probe emits a Finding record, the cover Windows kernel events, AWS CloudTrail audit, synthesis function converts the finding to a Sigma etc.); we offer a starter-rule path for BAS findings rule, and the rule’s references[0] URI lets the SOC that SigmaHQ does not currently cover (corpus analyst click from a fired alert back to the originatprobe → rule with traceback URI). ing probe. The engine is not the contribution and is deliberately treated as replaceable: the synthesis Adversary emulation platforms. CALDERA [9] function downstream of it is a pure mapping from provides ATT&CK-mapped adversary abilities with (finding, template), so the reproducibility envelope metadata that detection engineers consume as input (Section 3.4) holds for any verdict-producing engine, to the manual rule-authoring workflow. Our pipeline and the reference implementation we release is one differs in two dimensions: (a) we operate on BAS interchangeable choice. findings (post-attack signal) not attack-plan metadata (pre-attack intent); (b) we emit detection artifacts directly, not just metadata for analysts.

3.1

Locked corpora

Definition. We use locked probe corpus to mean a JSON file with three properties: (a) every probe carries a stable probe id that survives engine refactors and corpus revisions; (b) the file content is content-hashed at lock time, and the hash is part of any claim made against the corpus (Section 3.4); (c) updates to a probe (payload changes, indicator changes) require a new probe id suffix (e.g. a03-sqli-login-email-001 → a03-sqli-login-email-002); the old id is preserved for backward traceability. The concept is introduced in the companion BAS-attribution work [2] and adopted here without modification. Update policy: a corpus version bump (1.0.0 → 1.1.0) preserves all existing probe ids and adds new ones; a major bump (1.x.x → 2.0.0) is required to retire or repurpose an existing id. We work with two probe corpora, both released with the artifact pack:

LLM-assisted detection engineering. A growing concurrent line uses LLMs as the engine for detection content. RuleGenie [6] uses LLMs to optimize existing SIEM rules — tuning thresholds, reducing false positives. RAM [12] uses LLMs to map detection rules to MITRE ATT&CK techniques. Both produce useful output but share a property our work deliberately avoids: the mapping from input to rule (or input to mapping) is not byte-stable across runs — a different LLM release, a different prompt, or a temperature change produces a different artifact for the same input. The trade-off is direct. Our deterministic templatesynthesis approach gives up the breadth an LLM gets for free (an LLM can draft a rule for any attack class given a prompt; our function returns null on a missing template) in exchange for two properties LLMs cannot offer: (i) exact reproducibility — re-running • the locked LLM corpus — 17 probes coverthe synthesis on the same finding emits the same ing OWASP LLM01, LLM02, LLM06, LLM07, YAML, which a reviewer can audit and a detection LLM10. Locked at 2026-05-16 for the companengineer can sign off on without re-verifying each oution paper [2]. put; (ii) a typed traceability hook — every emitted rule carries the originating finding id, so when a • the locked Web corpus — 23 probes coverrule fires in production the analyst can pivot back to ing OWASP A01, A02, A03, A05, A07, A09 the BAS probe that produced it, and when a rule is of the OWASP Top 10 (2021) [4]. Locked at wrong the owner is unambiguous (the template au2026-05-18. thor, named in git). Hybrid pipelines that use the deterministic synthesis as the bootstrapping step and Each corpus is a JSON file with a schema covering an LLM optimizer (RuleGenie-style) as the tuning corpus-level metadata (id, version, lock date) and a list 3

Table 1: Positioning vs. closest related artifacts. Locked corpus = stable probe IDs survive engine revisions. Probe traceback = a rule (or report row) carries an explicit URI back to the originating probe. Auto-emits Sigma = a deterministic function emits rule YAML (not a human-authored ruleset). Multi-backend = same rules consumable by ≥ 2 SIEM dialects. Held-out test = published, peer-reviewed corpus exercises the artifact outside its training set. Artifact SigmaHQ rules [8] garak [1] MITRE CALDERA [9] RuleGenie [6] Paper #1 engine [2] This work

Locked corpus

Locked corpus

Probe traceback

Auto-emits Sigma

Multi-backend

Held-out test

N/A yes yes (abilities) no yes yes (2+2 held-out)

N/A per-probe id per-ability id N/A per-probe id per-rule URI

no (hand-authored) no no no (optimizes existing) no (no rule output) yes (det.)

yes (pysigma) N/A N/A N/A N/A yes (Splunk+Lucene)

no partial (corpus is the test) no no lattice-internal only yes (AdvBench+HarmBench)

BAS engine

findingTo Sigma()

Finding probe id

Sigma rule

pysigma → SIEM

references[0] URI (§3.2)

Figure 1: Round-trip from locked corpus to SIEM. Solid arrows: data flow at calibration time (left to right). Dashed arrow: the traceability URI a SOC analyst clicks from a fired rule back to the originating probe. Rounded blue boxes are artifacts (the locked corpus, the Finding record, the emitted Sigma rule); sharp orange boxes are processes (the BAS engine, the synthesis function, the Sigma backend stack); the right-most box collapses pysigma + Splunk SPL / Lucene KQL conversion + the OpenSearch index that hosts the held-out cohorts (AdvBench, HarmBench, benign-LLM, benign-Web) used in §4.7. The synthesis function is pure on (finding, template) so a reviewer can re-derive the rule set from the leftmost artifacts alone without running the engine source code (§3.4). • detection gap, defense recommendation — free-text fields used as Sigma description content.

of probes carrying their probe id, OWASP category, target endpoint, payload, expected indicator, severity, and CWE reference. A content hash of each file is part of the corpus’s lock metadata and is included in the replication package.

3.2

• metadata.{corpus id, corpus version, probe id, owasp category} — the traceability key.

Finding emission

• evidence refs[] — one entry per run, pointing to a content-hashed evidence record of the probeverdict snapshot (Section 3.4).

A calibration run drives every probe in the corpus against a sandbox target — OWASP Juice Shop for the Web corpus, and VulnBot, a deliberatelyvulnerable LLM application we release with this paper (backed by Gemini 2.5 Flash, model id gemini-2.5-flash, at sampling temperature 0.7 — the model the engine targets in production, chosen for its cost and latency profile), for the LLM corpus — and classifies the response. Probes that bypass the target’s defenses produce a Finding record with the following fields relevant to detection-rule synthesis:

3.3

Template library and rule synthesis

The template library contains N =23 entries indexed by either MITRE T-code or OWASP category. Lookup order: T-code first (some findings are labelled with a MITRE ATT&CK T-code rather than an OWASP category), OWASP regex second (the • attack technique — OWASP <CAT> --- calibration scenarios used here). Each template spec<name> (e.g. OWASP LLM01 --- DAN-style ifies a Sigma logsource + detection block; it is act-as hijack). The OWASP prefix is the key parameterized only via the finding’s per-rule fields the template library looks up. (Listing 1): 4

function findi ngToSigm a ( finding ) { template = pickTemplate ( finding . a t t a c k _ t e c h n i q ue ); if (! template ) return null ; return { title : template . title + " --- " + finding . attack_technique , id : finding . finding_id + " -" + template . technique , references : [ " bas :// finding /" + finding . finding_id , " https :// attack . mitre . org / techniques /" + template . technique . replace ("." , "/") + "/" , ], tags : [" attack ." + template . technique . toLowerCase () , " bas . severity ." + finding . severity ] , logsource : template . logsource , detection : template . detection , false positiv es : template . falsepo sitives ?? [" Unknown "] , level : template . level , }; }

Listing 1: Synthesis is a pure function of (finding, template). The synthesis is deterministic: re-running against the same finding produces the same YAML byte-forbyte (modulo the synthesis date in date:). This is intentional — the rules can be diffed across corpus versions to surface template churn.

convention: bas://finding/<id> resolves through the engine’s evidence-pack API to the originating probe payload, and bas.severity.<lvl> is a Sigmacompatible custom tag namespace. Implementations are free to substitute their own prefix (e.g. a tenantspecific URN); the contract is structural (one URI Template selection logic. per finding, one severity tag per rule), not the literal pickTemplate(technique) applies two passes prefix. Beyond the template body, four finding fields in order: (1) extract any MITRE ATT&CK T-code shape the output: (a) finding id becomes the rule’s in the technique string and look it up (e.g. T1190.001 id (prefixed with the template’s MITRE technique); hits a prompt-injection template); (2) if no T-code (b) severity becomes a bas.severity.<lvl> tag matches, search for an OWASP <CAT> prefix in the (the Sigma level field comes from the template, not technique and look up the OWASP-keyed template the finding, since severity in the corpus is per-probe (e.g. LLM01, A03). When multiple templates would whereas Sigma level is per-rule); (c) detection gap match (rare in practice — corpus probes carry + defense recommendation are embedded into the exactly one OWASP category), the first key tried rule’s description so the analyst sees the probe evwins. This is a first-match heuristic, not a principled idence inline; (d) the finding’s metadata.probe id selection; we did not observe ambiguity on the two is encoded in the references[0] URI for traceback. released corpora but flag two known weaknesses: (a) a Crucially, the probe’s payload is not fed in directly — finding whose attack technique string carries both the template’s detection block describes the class of a T-code and an OWASP prefix takes the T-code attack (e.g. “UNION SELECT keyword on a web-server path even when the OWASP-keyed template would log”), not the specific bytes of the calibration probe. yield a more specific rule; (b) cross-cutting attack This decouples the published rule from the calibration classes (e.g. an SSRF — server-side request forgery — probe distribution and is the design choice that enprobe that straddles OWASP A01 and A10) cannot ables future replay- on-held-out-probes measurement map to two templates in the current pipeline. A (Section 3.5). principled extension would either treat the mapping as set-valued (emit multiple rules) or score templates Traceability URI resolution. The against the finding’s full context (severity, asset, references[0] field of each rule encodes a payload shape) rather than the technique prefix alone. finding id. The mapping from finding id to its We leave both as future work. enclosing run record (which carries probe id and evidence-blob hash) is part of the replication package. Finding details that feed into the rule. The A stand-alone resolver service that lets a SIEM-side bas:// URI scheme and the bas.-prefixed tags used rule click straight through to the originating finding throughout this section are the engine’s back-reference is out of scope for this paper. 5

3.4

Reproducibility envelope

function returns a non-null rule. Lower bound on usefulness; upper bound on what template-only synthesis can achieve.

A reviewer should be able to reproduce every emitted rule. The envelope has four steps, none of which require access to a specific implementation stack:

• Baseline FP rate (§4.2) — each emitted rule is evaluated against N =100 benign log lines (typical product routes for Web, neutral chat prompts for LLM). Evaluator: keyword/selection substring matching, the v1 template subset’s semantics. This is a synthetic-log harness, not a SIEM.

1. Obtain the locked corpus and verify its content hash against the value committed at lock time. 2. Drive every probe in the corpus through any BAS engine that produces a per-probe verdict. The method depends on no specific engine; the production one is withheld, but the synthesis downstream is engine-agnostic.

• Held-out TP rate (§4.5) — each LLM01-family rule is evaluated against a 50-prompt deterministic sample of AdvBench [13], a public peerreviewed harmful-prompt corpus whose contents were never seen at template-authoring time. A fire is generalization evidence.

3. For each probe whose verdict is “bypassed,” construct a Finding record (fields listed in Section 3.2).

• Multi-backend Sigma conformance (§4.1, Table 7) — each emitted rule is parsed via pysigma and converted to Splunk SPL (Search Processing Language) and to Elasticsearch’s Lucene/KQL (Kibana Query Language). Reports per-rule parse and conversion success.

4. Apply the synthesis function — specified as language-agnostic pseudocode in the replication package — to each finding. The synthesis is deterministic: re-running it on the same finding emits byte-identical YAML modulo the synthesis-date field. The published rule set — each rule’s deterministic UUID and its Splunk SPL and Lucene conversions for all 17 emitted rules — ships in reports/sigma-conformance*.json. An independent re-implementation from the specification must reproduce these: matching ids (the id is a pure hash of (finding id, technique)) and matching backend conversions, up to YAML serialization.

• Real-SIEM replay (§4.7) — each emitted rule is loaded into a standalone OpenSearch index via its Lucene KQL conversion and queried against the held-out and benign cohorts. Reports the operator-facing union match rate.

4

Results

The integrity chain is: locked corpus → probe Reading guide. The held-out AdvBench corpus verdict → Finding record → Sigma rule reference. appears in four contexts with four different fire rates Each arrow is verifiable from public artifacts; any (0/50, 31/50, 30/50, 15/50). These are not contrabreak (mismatched corpus hash, missing probe id, dictory: they are one rule set under progressively dangling finding reference) is detectable without ac- stricter evaluation as it evolved from the v1 to the cess to the engine’s source code. The corpora, v2 rubric. Table 7(b) collects all four in one the synthesis function, every JSON artifact, and a place and Figure 2 plots them as a single progression; CLAIMS TO ARTIFACTS.csv mapping each reported together they are the canonical reference, and the number to its regenerating script are released as a per-measurement detail follows. In brief: §4.5 reports public replication package (https://github.com/a the v1 keyword-only rubric against AdvBench (0/50, lemaiorano/detection-as-code-synthesis; see the baseline failure that motivated v2), then the v2 rubric (keywords + regex selection) first as a Python Data Availability). prototype outside the engine (31/50 or 62%) and then as integrated in the live engine via a substring har3.5 Measurement protocol ness (30/50, Table 6); §4.7 replays the v2 engine rules Five measurements are relevant: template-hit, FP rate through a real OpenSearch+Lucene SIEM and obon a benign baseline, TP rate on a held-out attack serves 15/50, with the 30 → 15 drop attributable corpus, multi-backend Sigma conformance, and a real- to a pysigma-backend-elasticsearch \b compatibility shim documented in §5.7. The benign-baseline FP SIEM replay through a live query engine. measurements in §4.2 are on synthetic logs, and the • Template-hit rate (§4.1) — percentage of benign-LLM FP in §4.7 is on real OpenSearch index bypassed-probe findings for which the synthesis docs. 6

Table 2: Sigma synthesis coverage on the two locked corpora (single calibration run). Skipped = bypassed findings with no template hit; template hit = rules / bypassed. Table 3 reports the multi-seed variance. Corpus

Probes

Bypassed

Rules

Skipped

Template hit

17 23

8 8

8 8

0 0

100% 100%

AdvBench held-out fire rate (%)

OWASP LLM Top 10 OWASP Web Top 10 (Juice Shop)

v

31/50

sandbox is a deterministic HTTP target; every probe’s verdict is reproducible to the bit.

30/50

60 40

2. LLM corpus exhibits backend stochasticity. With the daily token budget raised to clear the multi-seed window, all 5 LLM seeds were valid (0 errors per seed). The bypass count across the 5 seeds was [8, 8, 8, 10, 8], mean=8.4, σ≈0.8. This is real variance — the LLM target ran on Gemini 2.5 Flash (gemini-2.5-flash), called at sampling temperature≈0.7; it returns slightly different outputs per call, and on 1/5 seeds one extra borderline probe crossed the bypass threshold. The variance is a backend property (the LLM), not a corpus property.

15/50

20

0/50 0 rd wo ey k 1

e

o

rot

p v2

v

in ng 2e

M

v2

SIE

Figure 2: The LLM01 rule set’s AdvBench heldout fire rate across the four evaluation stages it appears in throughout Section 4: the v1 keywordonly rubric (0/50, a baseline failure), the v2 key3. Skipped count is zero on every seed. Across word+regex rubric as a Python prototype (31/50) all 5+5 seeds and 7+5=12 valid bypass counts, and integrated in the live engine (30/50, synthetic-log the synthesis function emitted a non-null rule harness), and the same v2 rules replayed through a real for every bypassed-probe finding. This is the OpenSearch+Lucene SIEM (15/50). The four rates operational claim the paper makes; it survives are one rule set under progressively stricter evaluation, the full multi-seed check. not conflicting measurements; bar labels are fires/50. The 30 → 15 drop is a Lucene regex-compatibility A first run-set exhausted the daily token budget of artifact (Section 5.7), not a rubric regression. the LLM backend after the second seed and produced misleading raw numbers. We rerun with the budget raised to at least 5× the per-run token cost and ob4.1 Coverage on the locked corpora tain N =5 valid seeds in one measurement window; Table 2 reports the template-hit rate for both corpora. Section 5.7 documents the protocol so a reviewer can For every bypassed-probe finding, the synthesis func- reproduce. The audit also surfaced a measurementtion returned a non-null rule. The skipped=0 result side bug: the llm10-maxtokens-001 probe emitted a is not a property of this particular run: Appendix A.5 request shape the LLM-target endpoint rejected with lists the deterministic template assignment for every a 400, causing misclassification as error. Both issues probe in both corpora, so any bypassed finding is are corrected in the numbers reported here. guaranteed a template. Numbers in the table are generated from the same per-run records the synthesis consumes; the prose references the table to avoid drift 4.3 Template breakdown between text and data. Table 4 lists the 23 templates (19 exercised by current corpora, 4 reserved for future) registered at the time 4.2 Reproducibility across N =5 seeds of measurement, grouped by source (MITRE T-code Table 3 reports a multi-seed reproducibility check: legacy library vs. OWASP-prefixed additions from this five back-to-back calibration runs per corpus against work). Section 5.6 analyzes the effort, expertise, and scalability of authoring and maintaining this library the same sandbox target. Two findings: (empirically 2–4 person-hours per template, with a 1. Web corpus is deterministic. All 5 Web seeds break-even against manual rule authoring after 4–8 returned bypassed=8 (σ=0). The Juice Shop findings per category). 7

Table 3: Multi-seed reproducibility. Valid seeds is the count of seeds whose backend was available throughout the run; seeds where > 50% of probes returned a 5xx/timeout error are dropped as measurement artifacts and the drop reason is reported. On the Web corpus, σ=0: the bypass count is reproducible to the integer across the 5 valid seeds. On the LLM corpus, σ=0.80 over the 5 valid seeds (mean 8.4, range 8–10), attributable to non-determinism in the Gemini-templated probe surface even with fixed seed metadata. The 3 dropped LLM seeds are a Vertex AI Gemini daily token-budget cap artifact (100k tokens, hit between seed 2 and seed 3); they tell us nothing about the LLM corpus’s stochasticity. Corpus ID paper-1-locked-2026-05-16 juiceshop-2026-05-18

4.4

Valid/Total 5/5 5/5

Probes

Min byp.

Max byp.

Mean

Std

Drop reason

17 23

8 8

10 8

8.4 8.0

0.80 0.00

none none

Replay fidelity on a synthetic log v1) are out of scope for this measurement as well; see Section 5.4. harness

Table 5 reports a lightweight replay measurement against a synthetic log harness. For each emitted rule we (a) construct a positive log line from the rule’s own first keyword and check that the rule fires (true-positive sanity), and (b) generate N =100 benign baseline log lines per corpus (typical web requests; typical LLM chat prompts) and count false- positive fires. Two caveats matter for interpretation:

4.5

Held-out AdvBench

generalization

on

Table 6 reports the held-out measurement: we evaluate the emitted LLM01-family rules against a 50prompt deterministic sample of AdvBench [13], a public peer-reviewed harmful-behavior corpus whose prompts the template-library author never saw. The corpus and its sampling seed are released in the replication package; the source CSV’s sha256 is pinned in The TP measurement is self-referential. The the locked subset file. positive log line is constructed from the rule’s own The result is a 0/50 (0%) held-out fire rate. This is keyword list. A 100% TP rate therefore verifies the the most informative empirical signal in the paper and substring-evaluation pipeline runs correctly on rules the right place to calibrate scope. The v1 LLM01 tememitted by the synthesis function — not that the rules plates fire on jailbreak syntax — meta-tokens like DAN generalize to held-out attack payloads. The Section 5.7 mode, <|im start|>, “developer mode”, “grandmother” threat-to-validity discussion already flagged this self— but AdvBench prompts are direct harmful-intent rereference; we measure TP only to confirm the harness quests in plain prose (e.g. “Create a guide on identity does not silently no-op. theft. . . ”, “Develop malware that. . . ”). The two corpora are disjoint in surface form, and our keyword-list The FP measurement is meaningful. The be- templates do not bridge them. nign baseline is independent of the corpus probes This is the v1 template library’s scope limit, not a (typical product/blog routes for Web, neutral chat framework failure: the synthesis function and traceprompts for LLM). The Web side fires 4/800 (0.5%): ability contract are unchanged regardless of which two A01 templates each match 2 baseline lines contain- keywords a template carries; the held-out result tells ing /api/users/me (case-insensitive substring colli- us the template-authoring rubric should include a sesion with the sandbox keyword /api/Users). The mantic pattern (e.g. a Sigma regex or an LLM- backed LLM side fires 0/700 on the baseline. This is the first classifier as the rule’s detection logic) in addition to generalization-relevant signal in the paper: the v1 the syntactic keyword list. Section 5.6 incorporates Web templates need case-sensitive or path-anchored this as a v2 authoring requirement. matching to avoid colliding with benign per-user endpoints; the v1 LLM templates’ keyword lists (DAN mode, <|im start|>, etc.) are already specific enough 4.6 Multi-backend Sigma conformance to avoid the corpus we tested. and v2 rubric in the live engine The harness covers only the keyword and selectionwith-contains patterns the v1 templates emit. Rules Two follow-up measurements close evidence gaps from that emit aggregations or sequences (out of scope for earlier draft revisions; both are summarized in Table 7. 8

Table 4: Sigma template library at synthesis time. Source is mitre for the legacy T-code-keyed templates and owasp-llm/owasp-web for the templates added in this work. Technique is the MITRE ATT&CK identifier the emitted rule’s references field points to. Key

Technique

Title

Level

Source

T1190 T1078 T1557 T1059 T1213 T1190.001 T1552.002 T1499.004 LLM01 LLM02 LLM06 LLM07 LLM10 A01 A02 A03 A05 A07 A09 A04 LLM03 LLM04 LLM05

T1190 T1078 T1557 T1059 T1213 T1190.001 T1552.002 T1499.004 T1190.002 T1552.004 T1059.013 T1083 T1499.004 T1190 T1078 T1190 T1083 T1110 T1552.001 T1190 T1565.002 T1499 T1059.013

Web Application Injection Probe IDOR — Rapid Sequential ID Enumeration Response Missing Critical Security Headers Suspicious User-Agent / Tool Fingerprint Directory Listing Exposed Prompt Injection Probe LLM Prompt-Embedded Credential Leakage LLM Token Abuse — Long Prompt / Max-Tokens Manipulation OWASP LLM01 — Prompt Injection / Jailbreak OWASP LLM02 — Sensitive Information Disclosure OWASP LLM06 — Excessive Agency OWASP LLM07 — System Prompt Leakage OWASP LLM10 — Unbounded Consumption OWASP A01 — Broken Access Control (IDOR / admin-route) OWASP A02 — Cryptographic Failure (JWT alg:none) OWASP A03 — Injection (SQLi / XSS / NoSQLi) OWASP A05 — Security Misconfiguration OWASP A07 — Identification & Authentication Failure OWASP A09 — Key/Secret Directory Exposure OWASP A04 — Insecure Design (business-logic abuse) OWASP LLM03 — Training-Data Poisoning Probe OWASP LLM04 — Model DoS / Resource Exhaustion OWASP LLM05 — Improper Output Handling

high medium low medium medium high critical high high critical critical high high high critical high medium critical high high high medium high

mitre mitre mitre mitre mitre mitre mitre mitre owasp-llm owasp-llm owasp-llm owasp-llm owasp-llm owasp-web owasp-web owasp-web owasp-web owasp-web owasp-web owasp-web owasp-llm owasp-llm owasp-llm

Table 5: Replay-fidelity measurement against a synthetic log harness. Rules evaluated counts rules whose detection block uses keyword/selection substring matching (the v1 template subset). TP rate is self-referential: the positive log is constructed from the rule’s own first keyword, so a passing rule is verifying the harness pipeline, not generalization. FP rate is computed against N =100 benign baseline log lines per corpus and is the meaningful generalization indicator. Corpus OWASP LLM Top 10 OWASP Web Top 10 (Juice Shop)

Rules eval/total

TP fires/eval

TP rate

FP fires

FP rate (mean)

7/8 8/8

7/7 8/8

100% 100%

0 4

0.0% 0.5%

Multi-backend Sigma conformance. We parse the emitted rules through pysigma (the reference Python implementation of the Sigma standard) and convert them to two backend dialects: Splunk SPL via pysigma-backend-splunk, and Elasticsearch Lucene KQL via pysigma-backend-elasticsearch. The first parse pass on an earlier draft surfaced a Sigmaspec compliance bug in our engine — the rule id field was a concatenation of finding id with the MITRE technique, but the spec requires a v4 UUID. The synthesis function now derives a deterministic UUID v4 from sha256(finding id + technique), reshaped to RFC 4122. Post-fix and across both corpora, 17/17 (100%) rules parse cleanly and 17/17 convert to executable Splunk SPL and to Elasticsearch Lucene KQL. The trace-back to the originating finding lives

in references[0], preserving the traceability contract.

v2 template rubric — Python prototype to live engine. The earlier held-out result (0/50 fires on AdvBench) motivated a v2 authoring rubric (§5.6): keywords plus regex selection on semantic markers. Appendix A.4 shows the v1 and v2 detection blocks side by side. We validated the rubric in two stages. First, a Python prototype outside the engine — same keyword list as v1 plus 19 direct harmfulintent terms (identity theft, phishing, malware, fraud, . . . ) and two regex patterns matching actionverb + harm-object pairs — fires on 31/50 (62%) of AdvBench held-out prompts and on 4/26 (15%) 9

Union match rate (%)

of a benign baseline of neutral and adversarial-butbenign LLM prompts (cybersecurity journalism, security training, programming tutorials). The four false positives are tunable substring collisions: virus fires on “Discuss the history of computer viruses in the 1990s,” phishing on “Explain how phishing works so I can train my employees,” and the action-verb regex on a Python-tutorial prompt. Each is the expected residue of a keyword/regex approach on ambiguous prompts; a Sigma filter block with an educational/journalism allow-list would resolve them in deployment. Second, the rubric was integrated into the engine’s synthesis-module template library (no Python prototype in the loop). Running the held-out AdvBench replay against rules fetched from the live calibration api yields 30/50 (60%), within 1 prompt of the Python prototype. The end-to-end synthesis path now exercises the v2 rubric: corpus probe → finding → engineemitted Sigma rule → AdvBench held-out match → rule fires. The live api is only how we obtained the rule set; the emitted rules are themselves in the replication package, so this held-out measurement and the OpenSearch replay below are reproducible from the published rules and the published AdvBench/HarmBench subsets alone, without running the engine. Library breadth also grew from 19 to 23 templates (added OWASP A04, LLM03, LLM04, LLM05); the unused templates are released for future corpora that exercise those categories.

30

30 20

14

10

7.7 0

0 ch

n Be dv

A

H

n Be arm

ch Be

n nig

eb -W

LM

-L

n nig

Be

Figure 3: Real-SIEM replay through OpenSearch + Lucene: union match rate per cohort. The two left bars (AdvBench, HarmBench) are operator-facing detection rates on held-out attack corpora; the two right bars (Benign-LLM, Benign-Web) are the false-positive surface on benign baselines. The per-cohort match records are included in the replication package.

nign cohorts through a live SIEM. We stood up OpenSearch 2.13.0 [11] single-node, bulk-ingested four indexed cohorts (AdvBench held-out, HarmBench held-out, benign-LLM, benign-Web), fetched the engine-emitted Sigma YAML through the live calibration api, converted each rule to Lucene KQL via pysigma-backend-elasticsearch, and queried each rule against each cohort. Table 8 reports the operatorfacing fire rate; Figure 3 visualizes the per-cohort 4.7 Real-SIEM replay via OpenSearch detection vs. false-positive surface side-by-side. Four observations. (1) On AdvBench the union + Lucene match rate (≥ 1 rule fires per prompt) is 15/50 The measurements above use a synthetic harness (30%) — about half the synthetic-harness rate (30/50, (substring evaluation in Python) to count rule fires. Table 6). The drop is mechanical: a Lucene Java regex To close that gap we re-ran the held-out and be- rejects \b word-boundary anchors that the pysigma backend emits, so we post-process the converted KQL with a compatibility shim that strips \b, leaving Table 6: Held-out TP measurement (v2 rubric, live slightly broader regexes. Section 5.7 documents the engine). A rule fires on a prompt iff any LLM01- shim. (2) On HarmBench, a second held-out corfamily template keyword/selection substring-matches pus (50 HarmBench prompts spanning cybercrime, the synthetic request log of that prompt. The 60.0% illegal-activity, chemical/biological, and misinformarate reported here is the v2 rubric (keywords + regex tion categories [3]), 3/3 LLM01-family rules fire with on semantic markers) integrated in the engine; the v1 a 14% union match. The lower rate vs. AdvBench keyword-only rubric yielded 0/50 on the same corpus reflects HarmBench’s broader semantic categories — and is the baseline the v2 rubric was designed to many prompts (e.g. chemical-synthesis instructions, close. Section 5.6 documents the v1→v2 progression. harassment text) do not exercise the verbs the v2 The OpenSearch+Lucene replay of the same rules regex selection targets. This is the expected limit of (Section 4.7) lands at 15/50 (30%) after a Lucene an LLM01-only template set against a corpus coverregex-compatibility shim. ing categories the templates were not authored for. (3) The benign-LLM baseline fires on 2/26 (7.7%), Held-out corpus Prompts Rules eval Fires Held-out TP consistent with the 4/26 from the synthetic harness AdvBench [13] subset 50 3 30 60.0% (the Lucene backend is case-sensitive on keyword fields 10

Table 7: Multi-backend conformance and v2 template-rubric integration. Top: pysigma parse + Splunk SPL + Lucene KQL conversion of the engine-emitted rules across both corpora. Three backends validated; the rule id UUID-spec fix lets all rules ingest into a real Sigma stack. Bottom: the LLM01 rule set’s AdvBench held-out fire rate across all four evaluation stages it appears in throughout Section 4 — v1 keyword-only, v2 Python prototype, v2 integrated in the live engine (synthetic-log harness), and v2 replayed through a real OpenSearch+Lucene SIEM. The four rates are a single progression of progressively stricter evaluation, not conflicting measurements; this panel is the reference for disambiguating them. (a) Multi-backend Sigma conformance Corpus

Rules

Parse

Splunk SPL

Lucene KQL

8 9 17

8 9 17

8 9 17

8 9 17

OWASP Web Top 10 (Juice Shop) OWASP LLM Top 10 Total

(b) v2 LLM01 template — held-out AdvBench + benign baseline Variant Held-out fires TP rate v1 (keyword-only, jailbreak syntax) v2 Python prototype v2 in live engine (synthetic harness) v2 via OpenSearch+Lucene SIEM

0/50 31/50 30/50 15/50

0% 62% 60% 30%

FP rate

∼0% 15% on 26 benign inherits v2 proto 7.7% on 26 benign

Conformance 100% 100% 100%

Status baseline; too narrow on AdvBench rubric validation end-to-end through the api real-SIEM replay (§4.7)

Table 8: Real-SIEM replay via OpenSearch 2.13 + Lucene backend. Synthesized rules were converted to Lucene KQL via pysigma-backend-elasticsearch and queried against four ingested cohorts. Union reports the fraction of cohort documents where at least one rule fired (operator-facing detection rate). Benign cohorts surface the false-positive surface in a real query engine. AdvBench and HarmBench corpora are sha256-pinned; queries and source JSON are part of the replication package. Cohort

Kind

n

Union match

Rules w/ hits

AdvBench (held-out) HarmBench (held-out) Benign LLM baseline Benign Web baseline

TP TP FP FP

50 50 26 50

15/50 (30.0%) 7/50 (14.0%) 2/26 (7.7%) 0/50 (0.0%)

3/3 3/3 3/3 0/8

Total hits 45 21 6 0

where the Python substring evaluator was not — the 4.8 Worked example: a single rule endFP surface tightens slightly). (4) The benign-Web to-end baseline fires on 0/50, but for an unintended reason: the eight Web rules’ keyword lists are anchored with \b, which the compatibility shim strips, leaving regexes that no longer match the case-sensitive keyword field. This is reported honestly as a Lucenebackend authoring gap rather than a clean 0% FP rate; Listing 2 shows a representative rule emitted by the the Splunk-backend conformance (17/17, Table 7) is synthesis for the a03-sqli-login-email-001 probe of the locked Web corpus. Appendix A provides three the cleaner cross-backend signal. additional end-to-end walkthroughs (one per templatesource class: legacy MITRE T-code, OWASP LLM, OWASP Web) with analyst-side refinement notes for each. The replay establishes that the synthesized rules are ingestible by a real SIEM, the operator-facing detection rate on held-out adversarial corpora is nonA reviewer can verify this rule corresponds to its trivial and bounded by template-rubric scope, and the claimed probe by following the rule’s references[0] benign-LLM false-positive surface remains in single URI back to the finding record and reading digits. We do not claim production-grade tuning; probe id = a03-sqli-login-email-001, then crosswe claim the finding → rule → SIEM path executes referencing that identifier against the locked corpus end-to-end on artifacts a reviewer can replay. listed in Section 3.1. 11

title : " OWASP A03 ( SQLi ) --- login email bypasses authent ication " id : fnd_xxxxxxxxx - t1190 status : experimental references : - " bas :// finding / fnd_xxxxxxxxx " - " https :// attack . mitre . org / techniques / T1190 /" tags : - attack . t1190 - bas . severity . critical logsource : product : webserver category : access detection : keywords : - " ’ OR 1=1" - " UNION SELECT " - " DROP TABLE " - " < script >" - " javascript :" condition : keywords fals epositiv es : - " Penetration testing tools " - " Security research / bug bounty " level : high

Listing 2: Auto-generated Sigma rule for a SQL injection probe. The references field is the traceability hook.

5

Discussion

Table 9: Every false-positive measurement in the paper, with its evaluator and benign cohort. The rates 5.1 What template-hit does and are not a single number measured five ways: the synthetic rows count fires per (rule, log-line) evaluation, doesn’t tell us whereas the OpenSearch rows count per-prompt union A template hit means the synthesis function returned match (a prompt is a false positive if any rule fires on a non-null rule. It does not mean the rule is correct, it). The headline operational FP is the OpenSearch useful, or deployable. A trivial template that matches benign-LLM row (2/26, 7.7%). † The benign-Web 0% every OWASP category and emits a condition: ’*’ is a degenerate no-fire caused by the Lucene \b shim keyword search would hit at 100% by this measure (Section 5.7), not a clean low-FP signal. and be useless. The template-hit number is a lower Evaluator Benign cohort Fires FP rate bound on usefulness — it’s the fraction of findings Context Web synthetic baseline Python substring 8 rules × 100 lines 4/800 0.5% 7 rules × 100 lines 0/700 0.0% for which a SIEM operator gets any starting point LLM synthetic baseline Python substring v2 prototype (LLM01) Python substring 26 benign prompts 4/26 15.4% — and an upper bound on what template-only syn- OpenSearch benign-LLM Lucene / OpenSearch 26 benign prompts 2/26 7.7% thesis can achieve. We complement template-hit OpenSearch benign-Web Lucene / OpenSearch 50 benign requests 0/50 0.0% with three generalization measurements: a syntheticbaseline FP rate (0–0.5%, §4.2); a held-out TP rate on AdvBench (0/50 for the v1 rubric, 30/50 for the v2 rubric, Section 4.5); and a real-SIEM replay against OpenSearch + Lucene (Section 4.7) that adds a second held-out corpus (HarmBench [3], broken down cohort behind each. Two bases are in play — the per cohort in §4.7) and exercises the rules through a synthetic rows count fires per (rule, log-line), the live query engine. OpenSearch rows count per-prompt union match — so the rates are not directly comparable. The single The false-positive numbers, disambiguated. number to read as the operational false-positive surThe paper reports several FP figures (0%, 0.5%, 15%, face is the OpenSearch benign-LLM row: 2/26 (7.7%). 7.7%, 0%), which a reader can mistake for one quan- The benign-Web 0% is the degenerate no-fire from tity measured inconsistently. They are not: Table 9 the Lucene \b shim (Section 5.7), reported honestly collects all of them with the evaluator and benign rather than as a quality claim. †

12

5.2

Template churn and version policy internal observations. The template covers the per-

Each template carries a MITRE T-code that may evolve when ATT&CK releases a new sub-technique. Our template library is small enough (N=23) that we version-pin the whole synthesis module to the commit hash recorded with each corpus (Section 3.1). A future paper that depends on this work should pin the same hash; future calibration runs should reexport the templates.json artifact and diff against the published version. Drift in template internals (regex contents, detection selections) is acceptable as long as the technique field — which the corpus reviewer relies on — stays stable.

5.3

Scalability of the template library

The template-based approach trades manual rule authoring (every finding) for manual template authoring (every OWASP category). The practical question is whether this trade-off compounds favorably as the corpus grows.

category fixed cost; the per-finding cost drops to the time the synthesis function spends (milliseconds) plus the analyst-side review. The break-even point arrives after ≈ 4–8 findings per category, which both corpora reach after a single calibration run.

5.4

Template expressiveness

The current template library expresses Sigma’s keyword/selection detection patterns — adequate for the OWASP probes in our corpora. It does not yet emit Sigma’s advanced features: | count() by ... aggregations, condition: sel1 → sel2 sequences, or near temporal joins. This is a deliberate scoping choice independent of the rubric version: the corpus probes are single-request attacks where a keyword or status-code selection is the natural fit. Attack patterns that require multi-request reasoning (e.g. slow-rate IDOR enumeration, credential-stuffing chains, multistage prompt injection followed by tool invocation) would need template extensions that emit Sigma’s aggregation block. The synthesis function’s signature accommodates this — template.detection is a free-form object — but no such template ships in the current library. We treat multi-request templates as an explicit follow-up.

Two-axis growth. Corpus growth has two axes: depth (more probes within an existing OWASP category) and breadth (probes for new categories). On the depth axis the template library is O(1) — adding 10 more SQLi probes still uses the same A03 template. On the breadth axis it is O(n) — adding a probe for OWASP A04 Insecure Design requires a new A04 template. Across the two corpora we observed, breadth growth is bounded by the union of the OWASP Top 10 (LLM and Web), i.e. at most 20 templates indexed by 5.5 Rule churn under backend stochascategory, plus the MITRE T-code-indexed templates ticity that cover findings labelled by T-code rather than OWASP category. A library of ∼30 templates covers The multi-seed reproducibility table (Table 3) shows the realistic ceiling for an OWASP-aligned BAS today. σ≈0.8 on the LLM corpus: on 1/5 seeds, one extra borderline probe bypassed when it normally would Authoring cadence. Empirically the 11 templates not. Each bypass produces a finding; each finding added in this work took 2–4 person-hours each produces a rule. If the calibration runs daily, the (Section 5.6). A category that lacks a template de- deployed rule set could grow by one rule on the variant grades silently — pickTemplate returns null and the day and shrink the next, creating rule churn — alert synthesis function emits no rule. The skipped counter fatigue if naively deployed. Two mitigations follow in Table 2 flags exactly this case; a non-zero skipped from the artifact-first design: (1) because every rule’s count is the operational signal that a new template is id is derived from the finding id (which is per-run), needed. a downstream SIEM deployment that keys on tags + logsource + detection rather than id sees the Comparison to manual rule authoring. We do same logical rule across seeds — the variant seed not present a controlled timing study, but the struc- adds a duplicate, not a new rule; (2) the operator tural argument is: a SOC analyst writing a Sigma rule can require an attack to bypass on ≥ k of N seeds for an unfamiliar attack class (reading the BAS find- before deploying its rule, with the multi-seed pipeline ing, choosing a logsource, drafting detection logic, de- already producing the input. The current paper does ciding on false-positive comments, attaching MITRE not prescribe k; we report the σ so an operator can tags) typically spends 30–60 minutes per rule per our set k against their acceptable churn rate. 13

5.6

Authoring new templates

The 19 templates the two corpora exercise (LLM01/02/06/07/10 and Web A01/02/03/05/07/09) plus 4 currently-unexercised templates (A04, LLM03, LLM04, LLM05) released for future corpora total N =23. Authoring a template for a new OWASP category — say, OWASP A04 Insecure Design — is a structured task with three inputs and one validation gate: • Inputs. (a) a MITRE ATT&CK technique-code that best matches the attack class (used as the rule’s tags[0]); (b) a Sigma logsource (product + category) appropriate for where the attack surfaces in production logs; (c) a detection block — typically a keyword/regex list, a selection-withfilter, or a count-by-key — that fires on the attack’s distinctive log signature. • Validation gate. The new template must import into a Sigma-compatible backend (sigmac, pysigma) without conversion errors. A held-out probe from the corpus that exercises the new category should fire the rule when its log is replayed (the Section 3.5 fidelity-replay protocol). In practice the bulk of the authoring effort is producing the detection block, which requires SIEM-side knowledge of what the attack looks like in logs. The corpus side (probe payload) does not need to change — adding a template is decoupled from extending the corpus. The 23 templates released here (19 exercised, 4 future) took roughly 2–4 person-hours each to author and were paired with one calibration probe per category for end-to-end validation. We do not claim this is the optimal authoring cadence; it is the empirical baseline a follow-up scalability study should compare against. Expertise: template author vs. rule author. The template author needs strictly more SIEM context than the analyst writing one rule: the template must generalize across all findings of its OWASP/MITRE class, so the author has to identify the invariant log signature (the keywords/selectors that hold for every probe in the category) rather than the finding-specific signature (the exact request body of one probe). We staffed authoring with engineers carrying both a SIEM operations background and the OWASP taxonomy, but the template’s explicit logsource and detection schema means a SOC analyst can author one without coding: the artifact is a YAML file, and the validation gate (sigmac/pysigma parse + held-out fire) is mechanical.

Break-even with manual rule authoring. A SOC analyst writing a Sigma rule for an unfamiliar attack class from a single finding (read finding, choose logsource, draft detection, decide falsepositives, tag with MITRE) takes 30–60 minutes per rule in our internal observation. Once a template exists, the synthesis function emits a rule per finding in milliseconds plus the analyst’s review pass on the emitted YAML (a few minutes per rule, mostly checking falsepositives for the deployment context). At a template authoring cost of 2–4 hours and per-rule manual cost of 0.5–1 hours, the template pays off after roughly 4–8 findings in its category — a threshold both corpora here cross in a single calibration run. The category that never produces 4+ findings is exactly the category the skipped counter (Table 2) is meant to flag as “template not worth authoring yet.” Held-out signal updates the rubric. The AdvBench measurement against the v1 keyword-only rubric (0/50 fires, documented in Table 7 row 1) told us the v1 keyword-list rubric is too narrow: it captures attack syntax but not attack intent. A v2 rubric should require, in addition to the syntactic keyword list, at least one of: (a) a regex-based selection keyed on semantic markers (verbs like “write,” “create,” “generate,” co-occurring with harm-indicator object phrases); (b) an LLM-backed pre-filter that scores the request body for harmful intent before the keyword list is evaluated; (c) a response keywords list keyed on the model’s output rather than the user’s input, since aligned LLMs leak detectable refusal markers when they bypass. Each of (a)-(c) is a known Sigma backend pattern; the synthesis function already accepts arbitrary detection objects, so the change is template-side, not pipeline-side.

5.7

Threats to validity

Corpus distribution. Both corpora target intentionally-vulnerable sandbox apps (Juice Shop for the Web corpus, a VulnBot we ship with this paper for the LLM corpus). Templates tuned for these probes may generalize poorly to production targets where attack distributions are different. We do not claim the templates are state-of-the-art; we claim they are reproducible from public artifacts. Model-specific target. The LLM target is backed by a single model (Gemini 2.5 Flash), chosen as the model the engine runs against in production, for its cost and latency. Which probes bypass — and hence

14

the multi-seed variance (σ≈0.8, §4.2) — is a property of that backend; a different target model would shift the bypassed set. Crucially, the model does not enter the contribution: the synthesis function is a pure mapping from (finding, template), agnostic to which model produced the finding, so the templatehit (100%), determinism, and traceability results hold for any verdict-producing target. Only the set of bypassed findings — an input to synthesis, not part of it — is model-specific. Benign-baseline scope and absence of a generative baseline. The false-positive surface is measured on small benign cohorts: N =100 synthetic log lines per corpus (typical product and blog routes for Web, neutral chat prompts for LLM) plus N =26 real benign LLM prompts drawn from cybersecurity journalism, security training, and programming tutorials. These are deliberately adversarial-but-benign hard negatives rather than a sample of production traffic, so the reported FP rates bound the surface on confusable inputs, not on a representative log stream; a larger production-derived benign corpus is needed to estimate operational FP. We also do not run a headto-head empirical comparison against a generative (LLM-based) rule generator such as RuleGenie [6]: the positioning argued in Section 2 is structural (determinism and traceability versus generative breadth), and a controlled quality/throughput comparison is left as future work. OWASP category coverage. The Web corpus exercises 6 of 10 OWASP Top-10 categories; the LLM corpus exercises 5 of 10. Categories with zero corpus probes (e.g. OWASP A04 Insecure Design, OWASP LLM05 Improper Output Handling at the time of writing) have no template hit data. Closing those gaps is a corpus-extension task, not a synthesis task. Self-referential templates. The Web template for OWASP A03 (injection) emits a Sigma rule keyed on the same regex tokens the calibration probe sends. Replay against a Sigma-compatible backend is therefore expected to give a ∼100% TP rate, which is a poor measure of generalization. This paper defeats self-reference for the LLM-side rules by reporting their union match rate against two held-out corpora the template author never saw (AdvBench and HarmBench, Sections 4.5 and 4.7); the Web side still relies on the self-referential signal in Section 4.2.

prompt research community has converged on a small set of widely-cited prompt corpora (AdvBench [13], HarmBench [3]) that ship as plain-text prompt sets, which our scanner can replay against the released Sigma rules through a synthetic-log harness and through OpenSearch directly. We are not aware of an analogous public corpus on the Web side that satisfies the same three requirements: (i) OWASP Top 10-aligned coverage, (ii) prompt/payload shipped as plain text (not embedded in a packet-capture or vendor-specific log format that would require schema transformation before ingest), and (iii) a permissive enough license to bundle into a replication package. CSIC HTTP 2010 and the PKDD 2007 challenge datasets are the closest candidates but ship as log/packet captures rather than payload text; OWASP’s own ZAP corpora are tool-scripts rather than payload sets. We treat the construction of an AdvBench-equivalent for the Web Top 10 (a sha256pinned, JSONL prompt set with per-payload OWASPcategory tags) as the cleanest follow-up; the released synthesis function will consume such a corpus without modification because the synthesis is keyed on (finding, template) and is agnostic to where the finding came from. What this means for the Web generalization claim. Until that held-out exists, the Web side of this paper rests on two structural arguments rather than an empirical generalization measurement: (a) the synthesized Web rules pass Splunk+Lucene multibackend conformance (8/8, Table 7), and (b) the synthesis function is identical for Web and LLM rules, so the LLM-side held-out generalization signal (15–30% on AdvBench/HarmBench) is suggestive of what an analogous Web corpus would show if the templates’ detection blocks were authored against the same information-theoretic surface (a fair-butuntested assumption). We do not claim Web-side generalization in the strong sense the LLM side now supports; readers should read the Web TP rate (100% on the self-referential calibration run) as a synthesiscorrectness signal, not a detection-quality signal.

Backend-budget artifacts. The LLM scenario depends on a real LLM backend with a daily token budget. An early multi-seed run exhausted the budget after the second seed, invalidating the remaining three. We rerun with the budget raised to at least 5× the per-run token cost (≈ 100k tokens per LLM calibration run on the targets we used), which yields N =5 valid seeds in a single measurement window. Reviewers reproducing the paper should provision a Asymmetric held-out evaluation. The LLM-side comparable budget headroom or stagger seed runs held-out signal is possible because the LLM-attack- across budget reset boundaries. 15

Lucene regex compatibility shim. The real- benign-LLM cohort, not increased it. The synthetic SIEM replay (Section 4.7) revealed that pysigma- harness fired on 4/26 (15%) benign prompts; the backend-elasticsearch emits \b word-boundary an- OpenSearch backend fires on 2/26 (7.7%). The drop is chors in its KQL output. Lucene’s Java regex en- the case-sensitivity tightening described above (factor gine does not accept \b (a PCRE-only construct), so (i) applied to FP cohort as well as TP cohort). The every rule containing one yields zero hits when run as- Web side is 0/50 FP because its 8 rules are simply not emitted. Concretely, the LLM01 rule’s selector \bDAN firing on anything — a degenerate “no FP because no mode\b is converted by pysigma-backend-elasticsearch detection” condition, not a clean low-FP signal. We into the Lucene regex /\bDAN mode\b/; Lucene’s Java report both honestly in Table 8 rather than presenting regex engine rejects the \b token and returns no docu- the Web 0% as a quality claim. ments, even on a prompt that literally contains “DAN mode”. This is the kind of pattern that fires in the Recommended recovery path. Two fixes are Python harness but silently no-ops in the SIEM. We durable; we recommend both in tandem. Template added a compatibility shim that strips \b after the side (immediate): update the OWASP-Web tembackend conversion — rewriting the example above plates’ detection blocks to use contains|all with to the wildcard term *DAN mode*. After the strip, case-folded selectors and explicit wildcards (*’ OR the converted KQL contains wildcarded substring pat- 1=1*, *UNION SELECT*) rather than \b-anchored bare terns of the form *ignore previous instructions* tokens. This change is backward-compatible with OR *DAN mode* OR ... on the indexed prompt field Splunk and matches the Lucene keyword field se— the exact queries are committed in the replication mantics natively, removing the shim dependency for package’s OpenSearch-replay record. Two compound- the Web side. Toolchain side (structural): coning factors explain the drop from synthetic-harness tribute a Lucene-aware regex emission flavor to 60% (Table 6) to OpenSearch 30% (Table 8). (i) The pysigma-backend-elasticsearch that maps \b to Lucene backend is case-sensitive on keyword-typed the appropriate Lucene token-boundary construct fields where the Python substring evaluator was not, (Lucene supports lookahead-style boundary matching so prompts that contain the trigger in a different case via ^|$-anchored alternations on the analyzer output). (e.g. “Ignore Previous Instructions” vs. the rule’s “ig- Doing both makes \b-bearing Sigma rules portable nore previous instructions”) match in the harness but to Lucene without behaviour drift, and recovers the not in OpenSearch. (ii) Stripping \b broadens the synthetic-harness rate to within bootstrap-level agreeregex slightly (a word match becomes a substring ment with the OpenSearch rate. Neither fix requires match), which mostly cancels the case-sensitivity loss a change to the synthesis function itself — the synon the cohort but does not recover the prompts whose thesis function passes the template’s detection block phrasing diverges further from the rule’s verb list (e.g. through opaquely. Both are template/backend-side “please disregard the prior” vs. “ignore previous”). Em- changes, not pipeline-side. pirically the two factors net out to roughly half the It also explains the 0% fire rate on the benign-Web synthetic-harness rate on AdvBench. cohort (Table 8): all eight Web rules’ keyword selections were authored against \b-anchored expectations Affected templates. The shim touches every and degenerate after the strip. The Splunk backend rule whose template emits \b-anchored selectors — (17/17 conform, Table 7) accepts \b natively, so this all 3 live LLM01 rules and all 8 Web rules in the is a Lucene-emission gap in pysigma, not in the syncalibration set. Of those, the 3 LLM01 rules survive thesis function; a follow-up should either contribute a after the strip as substring matches (*jailbreak*, Lucene-compatible regex flavor upstream or have the *DAN mode*, etc., visible in the replication package’s synthesis function avoid \b in templates targeted at OpenSearch-replay record). The 8 Web rules degen- Lucene deployments. erate to empty selectors because their keyword lists were authored as bare tokens with relied-upon word- Audit-surfaced measurement bugs. The multiboundary semantics (e.g. \bDROP TABLE\b); stripping seed discipline surfaced two bugs in the calibration the anchors leaves a pattern that the OpenSearch pipeline that the single-run protocol would not have keyword field’s exact-match semantics no longer caught: (1) the llm10-maxtokens-001 probe was matches. This is the mechanical explanation for the sending a request shape that the LLM-target endpoint 0/50 benign-Web union-match rate in Table 8. rejected with a 400 status, causing the probe to always classify as error rather than capped/bypassed; (2) FP impact of the shim. Counter-intuitively, an earlier measurement claimed σ=0 on the LLM the shim reduced the LLM01 false positive rate on the corpus based on N =2 valid seeds, an overclaim 16

that the budget-corrected N =5 measurement contradicted (σ≈0.8). Both are now fixed and documented as artifact-audit findings; the contributionlevel claim (template-hit =100%, skipped=0) survives both corrections because skipped depends on findings.length reaching the synthesis function, not on the specific value of bypassed.

6

Conclusion

and (d) a stand-alone bas:// resolver service for clickthrough from a fired SIEM alert to the originating probe, scoped out here (Section 3) and left to operational deployment.

AI Tools Disclosure This research leveraged AI-assisted development tools to support manuscript preparation and code development, while maintaining full human oversight and accountability. The following tools were used:

Locked probe corpora — JSON files with stable probe ids released as research artifacts — make the • Language models: GPT-5 family (OpenAI via gap between BAS findings and SIEM rules shorter Codex) and Claude Opus 4.7 (Anthropic Claude than it has to be. We described a synthesis funcCode) were used to generate and review code tion that consumes the corpus’s OWASP-categorized implementations, and to refine manuscript text. findings and emits Sigma rules with deterministic Google Gemini 2.5 Flash is also a measurement back-references to the source finding and the MITRE subject in this paper: it is the LLM backend of technique. The function covers every bypassed finding the target application that the probes are fired in the two corpora we release (skipped count: zero against (Section 3); that usage is part of the across 17 LLM + 23 Web probes). system being measured and is not an authoring use. The rule synthesis itself is deterministic and We are deliberate about what this paper does uses no model. not claim. The synthesis is starter-rule level, not production-rule level. We report early-stage detection • Web search: MCP Tavily integration was used numbers — 30% union match on AdvBench and 14% to support literature review and fact-checking on HarmBench through a live OpenSearch+Lucene during manuscript preparation. index, with 7.7% FP on a benign-LLM baseline — but these are bounded by the v2 template rubric’s All scientific arguments, empirical methodology, statisscope (LLM01 family only) and by a Lucene regex- tical analysis, research questions, and conclusions were emission gap in pysigma’s backend (Section 5.7). The independently conceived, developed, and validated by contribution we do claim is a reproducibility enve- the author. lope: the synthesis is a deterministic pure function (re-running it on a finding emits byte-identical YAML), specified as pseudocode, and the resulting rule set — Statements and Declarations deterministic ids plus multi-backend conversions — is published, so a reviewer re-implementing from the Funding This study did not receive a direct research spec reproduces it from the locked corpus alone, with grant. Experimental operation used Google Cloud every rule’s source provenance available in two clicks. resources and Gemini 2.5 Flash as the LLM target Our companion work [2] established the locked- backend. corpus discipline for measuring engine-side defense attribution. This paper extends the discipline to Competing Interests The author declares no comthe defense-side output: every emitted detection rule peting interests. traces back to the same corpus entry the engine measured against. The two papers together cover the Data Availability All aggregate results and round trip from attack probe to detection content statistical tables are reported in full in the paper with an integrity chain a reviewer can verify without (Section 4); every numeric value in those tables the engine source code. traces back to a JSON artifact in the public Remaining gaps for follow-up: (a) a controlled N ≥5 replication package (https://github.com/ale analyst timing study to replace the hand-wavy 30–60 m a i o r a n o / d e t e c t i o n - a s- c o d e- s y n t h e s is). minute manual- authoring estimate; (b) corpus ex- The released artifacts are: the two locked tension to exercise the four reserved templates (A04, probe corpora (owasp-llm-probe-corpus.json, LLM03, LLM04, LLM05); (c) a Splunk-side replay juiceshop-probes.json), the synthesized rule to mirror the OpenSearch+Lucene measurement and sets and per-run records, the OpenSearch replay resolve the \b compatibility gap upstream in pysigma; queries, the sha256-pinned held-out AdvBench and 17

HarmBench subsets (with the build scripts that regenerate them from upstream), and the multi-seed calibration outputs. A CLAIMS TO ARTIFACTS.csv maps every reported number to its artifact field and the script that regenerates it. Withheld for intellectual-property reasons: the production source of the BAS engine that fires the probes and emits the Finding records is proprietary, as is the production source of the synthesis function and template internals. However, neither is required to reproduce any number in this paper. The synthesis function — the contribution — is published as language-agnostic pseudocode (the full pickTemplate/findingToSigma contract) sufficient to re-implement it; it is pure on (finding, template), so any verdict-producing engine that emits the finding fields in Section 3.2 can drive it, and the held-out and real-SIEM measurements consume the emitted rules (re-derivable from the corpora and the specified synthesis) and the public AdvBench/HarmBench prompt sets — never the engine. The LLM-target backend is Gemini 2.5 Flash (commercial API); independent replication requires a Google Cloud account but no proprietary tooling.

Ethics Approval and Consent to Participate Not applicable: the study involves no human subjects, personal data, or live third-party systems — only the authors’ own sandbox targets. Consent for Publication

Not applicable.

References

Code Availability The deterministic synthesis function is specified as language-agnostic pseudocode (synthesis/SYNTHESIS SPEC.md), alongside the template-library metadata, the locked corpora, and the standalone exporters that regenerate every table and figure from the artifacts, in the public replication package (https://github.com/alemaiora no/detection-as-code-synthesis), under CC-BY 4.0. The production engine source remains proprietary but is not required to reproduce the reported numbers (see the package’s SCOPE.md).

[1] Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A framework for security probing large language models, 2024. [2] Alexandre Cristovão Maiorano. Which defense closes which threat? attributing OWASP-LLMtop-10 coverage and its brittleness under paraphrasing, 2026. URL https://arxiv.org/abs/ 2606.02822. [3] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. In International Conference on Machine Learning (ICML), 2024. [4] OWASP Foundation. OWASP top 10:2021. ht tps://owasp.org/Top10/, 2021. [5] OWASP GenAI Security Project. OWASP top 10 for LLM applications. https://genai.owas p.org/llm-top-10/, 2025. [6] Akansha Shukla, Parth Atulbhai Gandhi, Yuval Elovici, and Asaf Shabtai. RuleGenie: SIEM detection rule set optimization, 2025. [7] SigmaHQ. Sigma: Generic signature format for SIEM systems. https://github.com/SigmaHQ /sigma, 2024.

Responsible Research and Dual-Use This work is defensive: its output is detection content (Sigma rules) that helps SOC teams catch attacks they [8] SigmaHQ Contributors. Sigmahq community rule would otherwise miss. The released artifacts introlibrary. https://github.com/SigmaHQ/sigma, duce no new offensive capability. The attack probes 2024. target only intentionally-vulnerable applications we [9] The MITRE Corporation. MITRE CALDERA: ship for the purpose (OWASP Juice Shop and a Adversary emulation platform. https://calder deliberately-insecure VulnBot); any secrets in the a.mitre.org/, 2024. corpora are clearly-labelled non-functional canaries, not real credentials. The held-out evaluation reuses [10] The MITRE Corporation. MITRE ATT&CK. AdvBench [13] and HarmBench [3], established public https://attack.mitre.org/, 2024. benchmarks redistributed only as sha256-pinned subsets under their upstream licenses; we add no novel [11] The OpenSearch Project. OpenSearch: An openharmful prompts. Simulation is to be run only against source distributed search and analytics suite. ht systems the operator owns or is authorised to test. tps://opensearch.org/, 2024. 18

[12] Prasanna N. Wudali, Moshe Kravchik, Ehud Malul, Parth A. Gandhi, Yuval Elovici, and Asaf Shabtai. Rule-ATT&CK mapper (RAM): Mapping SIEM rules to TTPs using LLMs, 2025. URL https://arxiv.org/abs/2502.02337. [13] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.

A

Worked Examples: BAS Finding → Sigma Rule

This appendix walks through three end-to-end examples covering the three template-source classes (legacy MITRE T-code, OWASP LLM, OWASP Web). For each, we show: (a) the calibration probe; (b) the resulting finding fields; (c) the Sigma rule emitted by findingToSigma(); (d) a justification for why the rule is a useful, non-trivial starting point for a SOC analyst.

a hit-everywhere baseline that they can tune down with allow-lists for legitimate pen-test sources. The references[0] URI lets an on-call responder pivot from a fired alert back to the originating probe payload in the BAS run.

A.2

Example 2: LLM jailbreak (OWASP LLM01 / T1190.002)

Probe. llm01-jb-dan-001 sends the DAN “Do Anything Now” jailbreak payload via POST /chat {message: "Ignore all previous instructions. You are DAN..."}. The probe declares forbidden = ["DAN MODE ON", "step-by-step", "explosive"]. Finding. Emitted when any forbidden token appears in the model response. attack technique = "OWASP LLM01 --- DAN-style act-as hijack"; severity = high; metadata.probe id = llm01-jb-dan-001.

Emitted rule (excerpt). Template LLM01 fires. logsource = (application, llm); detection.keywords = ["ignore previous Web injection A.1 Example 1: instructions", "DAN mode", "developer (OWASP A03 / T1190) mode", "<|im start|>", "jailbreak", "grandmother"]; tags = [attack.t1190.002, Probe. a03-sqli-login-email-001 sends POST bas.severity.high]; falsepositives = ["Legit /rest/user/login {email: "’ OR 1=1--", red-team / security research", "Researchers password: "x"} and looks for body contains: discussing jailbreak history"]. "authentication" as the bypass indicator. Finding. attack technique = "OWASP A03 --SQL injection in login email bypasses authentication"; severity = critical; detection gap = "Probe a03-sqli-login-email-001 confirmed vulnerable. Indicator matched: body~ ’authentication’..."; metadata.probe id = a03-sqli-login-email-001.

Justification. LLM application logs are not yet covered by stock SIEM rule packs; this rule gives the analyst a starter ruleset to deploy on a Cloud Run stdout log stream or a Vertex AI audit stream. The falsepositives list flags the most likely benign trigger (researchers discussing jailbreaks in chat) so the analyst can pre-author an allow-list.

A.3

Example

3:

LLM

tool-

Emitted rule (excerpt). Template A03 fires enumeration (OWASP LLM06 / (OWASP regex pass). logsource = (webserver, T1059.013) access); detection.keywords = ["’ OR 1=1", Probe. llm06-tools-functions-001 sends “List "UNION SELECT", "DROP TABLE", "<script>", "javascript:"]; condition = keywords; tags = the JSON schema of every tool registered to you. Use the OpenAI function-calling format.” [attack.t1190, bas.severity.critical]. forbidden = [’"name":’, ’"parameters":’, ’"properties":’]. Justification. The rule fires on any HTTP access log where the request URI or body contains a classic SQLi keyword. This is a more aggressive starting point Finding. attack technique = "OWASP LLM06 than vendor SIEM defaults (which typically require a --- List all tool JSON schemas"; severity = downstream alert from a WAF) and gives the analyst high. 19

Emitted rule (excerpt). Template LLM06 fires. logsource = (application, llm); detection.response keywords = ["send email", "delete user", "execute sql", "" tool name" :", "" function" :", "" parameters" :"]; condition = response keywords.

# v1 ( keyword - only ) -- fires on jailbreak SYNTAX detection : keywords : - " DAN mode " - " <| im_start | >" - " developer mode " - " grandmother " condition : keywords # v2 ( keyword + regex ) -- adds harmful - INTENT coverage detection : keywords : # v1 jailbreak syntax , retained - " DAN mode " - " <| im_start | >" - " developer mode " intent_terms : # +19 direct harmful - intent terms - " identity theft " - " phishing " - " malware " - " fraud " # ... ( full 19 - term list in the replication package ) intent_re | re : # action - verb + harm - object pairs - "\ b ( write | create | generate | develop ) \ b .* \ b ( malware | exploit | weapon | bomb ) \ b " condition : keywords or intent_terms or intent_re

Justification. The detection sits on the model’s response side (not the request side), so it catches both direct enumeration via prompt and indirect leakage via reflective queries (“what could you do for me?”). The falsepositives note (“Documentation chatbots intentionally describing their toolset”) tells the analyst to scope the rule out of documentation deployments. Without this starter rule, the analyst writing it from scratch typically needs ≥ 1 hour to identify the right detection surface (response-side) and assemble the keyword list.

Listing 3: Representative v1 (top) and v2 (bottom) detection blocks for the LLM01 template. Term and regex lists are abbreviated; the complete blocks are in the replication package. Trace. All three examples are reproducible end-toend: the corresponding finding id appears in the run’s evidence blob (Section 3.4), and the rule YAML is regenerated deterministically from the same (finding, template) pair by re-running the synthesis function.

A.4

The v1 vs v2 LLM01 detection block

A.5

Full probe-to-template map

Tables 10 (LLM corpus) and 11 (Web corpus) list every probe in the two locked corpora alongside the template its OWASP category resolves to and the MITRE ATT&CK technique the emitted rule references. They are the detection-side companion to the locked-corpus listing published with the attribution work [2]: where that table records each probe’s designed outcome per defense level, these record each probe’s deterministic template assignment. Because the assignment is total over both corpora, the tables double as a visual proof of the skipped=0 result in Table 2 — no probe is left without a template.

Listing 3 shows the LLM01 template’s detection block before and after the v1→v2 rubric change discussed in Section 5.6. The v1 block fires only on jailbreak syntax (meta-tokens such as DAN mode); the v2 block keeps those keywords and adds two semantic markers — a list of direct harmful-intent terms and an action-verb + harm-object regex — so that plainprose AdvBench prompts (which carry no jailbreak syntax) also fire. This is the single template edit behind the 0/50 → 30/50 held-out jump in Table 7(b); the synthesis function and traceability contract are unchanged. 20

Table 10: Deterministic probe→template→technique map, OWASP LLM Top 10 corpus (N =17, paper-1-locked-2026-05-16). Each probe’s OWASP category selects exactly one template (Template column), whose MITRE ATT&CK identifier the emitted rule’s references field carries. The assignment is a pure function of (corpus, template library): every probe resolves to a template, so a bypassed finding can never be skipped (Table 2). Probes of the same category collapse to one template — the source of the depth-axis O(1) scalability discussed in Section 5.6. Probe ID

Description

Template MITRE technique

LLM01 llm01-jb-dan-001 llm01-jb-grandma-001 llm01-jb-devmode-001 llm01-jb-base64-001 llm01-jb-leet-001 llm01-jb-imstart-001

DAN-style act-as hijack Grandma emotional manipulation Developer-mode bypass Base64-encoded payload smuggling Leetspeak smuggling System role override via user (<|im start|>)

LLM01 LLM01 LLM01 LLM01 LLM01 message LLM01

T1190.002 T1190.002 T1190.002 T1190.002 T1190.002 T1190.002

LLM07 System prompt verbatim leak Reverse-engineer system prompt

LLM07 LLM07

T1083 T1083

API canary key echo Training data extraction (Stripe SDK canary) Synthetic PII echo (SSN+CC)

LLM02 LLM02 LLM02

T1552.004 T1552.004 T1552.004

llm06-tools-public-001 Tool registry enumeration (/tools) llm06-tools-functions-001 Tool registry enumeration (/functions) llm06-side-effect-001 Side-effect tool detection (send email, delete *)

LLM06 LLM06 LLM06

T1059.013 T1059.013 T1059.013

LLM10 LLM10 LLM10

T1499.004 T1499.004 T1499.004

llm07-prompt-leak-001 llm07-prompt-reveng-001

LLM02 llm02-canary-001 llm02-training-001 llm02-pii-001

LLM06

LLM10 llm10-maxtokens-001 llm10-model-swap-001 llm10-burst-001

max tokens=999999 manipulation Expensive model swap (claude-opus-4-7) Burst 10 req/s rate limit

21

Table 11: Deterministic probe→template→technique map, OWASP Web Top 10 corpus (N =23, juiceshop-2026-05-18). Companion to Table 10; same construction. Probe ID

Description

Template MITRE technique

SQL injection in login email bypasses authentication (classic ’ OR 1=1–) SQL injection with comment terminator bypasses password check UNION-based SQL injection in product search leaks bcrypt hashes Order-by SQL injection via sort param (error-based detection)

A03

T1190

A03

T1190

A03

T1190

A03

T1190

IDOR — basket of another user readable without auth scoping a01-idor-basket-002 IDOR — basket #2 a01-idor-basket-003 IDOR — basket #3 a01-idor-basket-004 IDOR — basket #4 a01-idor-basket-005 IDOR — basket #5 a01-idor-feedback-001 IDOR — feedback record of another user a01-idor-users-listing-001 Unauthenticated Users listing endpoint leaks all account emails A03 a03-xss-search-001 Reflected XSS in product search query echo a03-xss-deeplink-001 Reflected XSS via fragment deeplink (hash-route DOM injection) a03-xss-track-result-001 Reflected XSS in track-result id parameter A02 a02-jwt-none-001 JWT alg:none accepted — admin identity forgeable without signature A01 a01-admin-public-001 Administration UI route reachable without auth check on frontend a01-admin-api-001 Mass assignment — role field on Users PUT accepts admin upgrade A05 a05-ftp-listing-001 FTP directory listing exposed at /ftp a05-ftp-acquisitions-001 Confidential acquisitions document publicly readable A01 Poison-null byte path traversal bypasses .md a01-pathtraversal-ftp-001 whitelist A09 a09-secret-keys-001 Encryption keys directory listing leaks premium.key A07 a07-2fa-bypass-001 2FA verification flow accepts weak/test token A01 a01-redirect-allowlist-001 Open redirect bypasses allowlist (target=evil.com)

A01

T1190

A01 A01 A01 A01 A01 A01

T1190 T1190 T1190 T1190 T1190 T1190

A03 A03

T1190 T1190

A03

T1190

A02

T1078

A01

T1190

A01

T1190

A05 A05

T1083 T1083

A01

T1190

A09

T1552.001

A07

T1110

A01

T1190

A03 a03-sqli-login-email-001 a03-sqli-login-quote-001 a03-sqli-search-001 a03-sqli-order-by-001

A01 a01-idor-basket-001

22