Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage Maciej Kisielewicz

Michał Kędziora

Ngoc Thanh Nguyen

maciej.kisielewicz@pwr.edu.pl Department of Applied Informatics Wrocław University of Science and Technology Wrocław, Poland

Department of Applied Informatics Wrocław University of Science and Technology Wrocław, Poland

Department of Applied Informatics Wrocław University of Science and Technology Wrocław, Poland

arXiv:2611.08229v1 [cs.CR] 5 Nov 2026

Abstract

up to five orders of magnitude [9]. Until it is refitted, a fast tier may meet such an attack with nothing in its training data, recalibration or thresholds that resembles it; for the fast tier, the attack then belongs to an unseen family. The pipeline catches it only if the fast tier does not close it, and escalation has to stay within a budget on how much benign traffic the slow tier can absorb. Since September 2026 a new class of models has been offered as the fast tier. TypeSafe’s Jev, Cloudflare’s Clef and the open Laya, von and Verdict read an input once and return typed answers, such as a yes or no with a probability, within a few hundred milliseconds [2, 12, 16, 27, 83]; we call them decision models. TypeSafe calls them System One models and claims calibrated probabilities, a property a fast tier needs on attacks unlike any it was fitted on. Studies of these models measure accuracy and calibration on general and security decisions [20, 42, 68], place one in front of a reasoning model [40, 48], and steer them with injected text [29, 39]. To our knowledge, no study evaluates them as the fast tier of web triage on attack families held out of everything a deployment fits, against the detectors and rule sets operators run, within a fixed benign budget. We hold out one attack family at a time from training, recalibration and every threshold, and ask four questions. Are the fast tier’s probabilities trustworthy on the held-out family (RQ1)? Which routing rule gets held-out attacks to the slow tier within a benign budget (RQ2)? What can an adversary achieve by editing an attack or by sending benign requests that the fast tier escalates (RQ3)? What does each fast tier cost in latency, money and slow-tier load (RQ4)? We compare six decision models used zero-shot, three of them hosted, with three supervised detectors trained on the known families (ModernBERT, a fine-tuned Laya and an n-gram model), a local language model read through its token probabilities, the OWASP Core Rule Set (CRS) [52] and a benign-only anomaly baseline. Our main data are real browsing requests from the open-appsec WAF comparison [64] with public attack payloads spliced in, among them Log4Shell (CVE-2021-44228) and Shellshock (CVE-2014-6271). Exploits for 423 vulnerabilities from 2025 and 2026, converted from nuclei templates [57], test recent attacks, and those published after the models’ release are unlikely to be in their pretraining data. Benign twins (exploit requests with the attack replaced by a legitimate value) and re-rendered requests control for request shape. ECML/PKDD 2007 [62] and CSIC 2010 [76] serve as further datasets. Our contributions are:

In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so an attack the fast tier closes is never examined. We hold each of seven attack families out of training, recalibration and thresholds, splice the payloads into parameters of real browsing traffic, and compare zero-shot decision models, which return a probability for a typed question, with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline. Detectors trained on the known families can give much of a held-out family a confident benign verdict: over 10 training runs, ModernBERT gives 37% to 60% of held-out path traversal a probability below 0.05 and escalates only 50% to 81% of it within a 5% benign budget, against nearly all of the same requests when traversal is known. Jev and Clef, two hosted decision models whose budget decisions do not depend on the held-out family, escalate at least 97.6% of every family within the same budget, but their undisclosed training data may contain these public families. Tests on recent exploits need controls for request shape: Jev escalates 37% of benign twins (exploit requests with the attack replaced by legitimate values), and detectors trained on older exploits learn what scanner requests look like. A search guided by Jev’s probability gets 43% of held-out attacks past Jev within 300 queries, but Jev’s answers vary: asked again, it escalates those requests in 60% of its answers.

Keywords two-tier triage, web attack detection, held-out attack families, decision models, calibration, conformal prediction

1

Introduction

Security operations centers commonly triage in tiers. Tier-1 analysts work through a stream of alerts, many of them false positives, and escalate to more experienced tiers the ones they cannot rule out [1, 33]. Recent proposals for automated triage follow the same pattern: a scanner or a fast decision model screens the input, and a large language model (LLM) reviews what it forwards [42, 48, 72]. A language model can examine a suspicious request in depth, but it takes seconds and charges per token [87], too slow and costly for every request of a busy web service. In such a pipeline a fast tier sees all traffic and decides most of it alone, and a slow tier (a generative model, an agentic analysis or a person) examines only what the fast tier escalates. A zero-day attack exploits a vulnerability before its disclosure; Bilge and Dumitraş found such attacks lasting 312 days on average, and disclosure raising the volume of attacks on a vulnerability by

• Supervised detectors confidently miss some held-out families, by amounts that vary between training runs (Sections 5 and 6). Over 10 runs, ModernBERT gives 37% to 60% of 1

Kisielewicz, Kędziora and Nguyen

held-out path traversal a probability below 0.05, although it escalates nearly all of the same requests when traversal is known. The misses persist under other recalibration maps and on percent-decoded requests, while the n-gram model’s escalation of the other families rests largely on how the benchmark encodes payloads. • Used zero-shot, Jev and Clef escalate nearly all of every heldout family (Section 6). They escalate at least 97.6% of every family within a 5% benign budget, and at 0.1% Jev escalates 82% to 97% and Clef 68% to 91%. Their training data may contain these public families, but Jev escalates all 9 exploits published after its release and Clef 4 of the 5 published after its own. • Exploit tests need controls for request shape (Section 6.6). Jev escalates 37% of the benign twins of exploit requests, and detectors trained on older exploits escalate half or more, so much of their gain from the older exploits reflects request shape. • Evasion tests need repeated calls when a fast tier’s answers vary (Section 7.2). A score-guided search gets 43% of heldout attacks past Jev, but, asked again, Jev escalates those requests in 60% of its answers; Clef answers identically when asked again, and the 26% of attacks the search gets past it within 150 queries stay past it. • Flooding, cost and cascades (Sections 7.3 and 8). Padding benign requests makes Clef escalate 97% of them, enough to flood the slow tier. At a 5% budget the slow tier costs more than Jev’s fast tier once an examination costs more than 20 Jev calls. A cascade with the rule set first escalates at least 89% of every held-out family but 73% of the recent exploits.

domain [71]. Others attack these models: one appended opinion flips 12.1% of the decisions of jev-1.13.0 [29], forged inputs bypass 19 to 29% of the phishing emails Jev flags but none of its moderation decisions [39], and prompt injection shifts its probabilities [80, 84]. These studies evaluate the models on fixed test sets or against crafted inputs. Our evaluation differs in four ways. We hold a family out of recalibration and thresholds as well as training, which exposes confident misses, the failure Liu describes [42], in every supervised detector and rarely in Jev or Clef (Section 5). We compare every fast tier within one benign budget (Section 6). Benign twins and a classifier that sees no parameter value, the format control, measure how much of an exploit test’s recall comes from request shape (Section 6.6). We judge a score-guided search’s evasions on repeated calls (Section 7.2), which matters for a model whose answers vary. Zero-day and unseen attacks. A learned detector meets a problem related to zero-day attacks [9]: attacks unlike its training data. For web requests, Kruegel and Vigna [35] and PAYL [79] model normal parameters and payloads, ZeroWall flags requests that an encoder-decoder trained on benign traffic cannot reproduce [74], and RETSINA meta-learns unsupervised detectors across web domains [38]. CADE detects samples of families absent from training [86], OWAD detects and adapts to a shift of normal behavior [26], and Sarhan et al. evaluate network intrusion detectors on attack classes held out from training [67]. Pretrained transformers have been found to degrade less and to stay better calibrated out of distribution [19, 28]; our fine-tuned transformers rank held-out families better than the n-gram model but are no better calibrated on them. Evaluating learned detectors. Sommer and Paxson [70] explained why learned intrusion detectors struggle outside the traffic they were built on, and Axelsson [5] showed that at realistic base rates small false-alarm rates overwhelm analysts; our projections of slowtier load (Section 8.4) follow both. Arp et al. [4] cataloged pitfalls such as spurious correlations, and TESSERACT [54] showed how designs that ignore time inflate results; the template shortcut of Section 4.1 is a spurious correlation in a public web benchmark. WAF-A-MoLE [18] evaded learned web application firewalls by mutating payloads.

Appendix D documents five pitfalls in web attack data, among them a request template that lets a classifier separate a benchmark’s attacks from benign traffic without reading a payload.

2

Related Work

Decision models. Preprints on decision models appeared within weeks of their release. Benchmarks measure their accuracy and calibration on general tasks: Deußer et al. find Jev’s choice probabilities well calibrated but its yes-or-no probabilities in need of a tuned threshold [20], and TypedBench [68], JEVal [21] and Sys1Calv1 [56] measure calibration error, sensitivity to wording and overstated probabilities. Swapping yes and no flips about half of Jev’s answers [61], and the option labels of open decision models can override their definitions [6]. Cloudflare reports Clef’s accuracy, latency and a forecasting Brier score, but no calibration-error or reliability measurement [12, 14]. Closest to our setting, HydroJEV places Jev as the first tier of SCADA alarm triage, ahead of a supervised classifier on event subtypes absent from its labels [48]; JEV-as-a-Judge accepts Jev’s confident verdicts and escalates the rest to a reasoning judge [40]; and Messoudi and Mansar choose risk-controlled thresholds for decision scores [45]. On security decisions, Liu finds that a low average calibration error can hide attacks labeled safe with high confidence [42], and Cyber-Jev, a small model fine-tuned for HTTP-attack, phishing-URL and prompt-injection checks, reports calibration that does not hold outside its training

Conformal evaluation and deferral. Transcend [30] and its successor [8] used conformal evaluation to decide when a malware classifier should no longer be trusted; a conformal guarantee need not hold once exchangeability fails [7], and Li shows that a conformal threshold for intrusion detection loses its coverage under adversarial perturbation [41]. Calibration and its decay under shift are well studied outside security [24, 51]. Selective classification [22] and learning to defer [44, 47] train a model to abstain or hand over, and LLM cascades escalate the queries a cheap model is unsure about [11, 34], optimizing accuracy per unit of cost. Prompt injection and language models in security operations. A model that reads attacker-controlled text can be steered by it: indirect prompt injection compromises LLM-integrated applications [23], injection attacks have been formalized and benchmarked [43], and optimized injections deceive LLM judges [69]. 2

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Surveys map language models in security and in security operations centers [25, 85]. Recent systems triage alerts with cooperating agents [82] or compress LLM reasoning to meet latency budgets [87]. Sun et al. place an LLM after a web scanner and report LLMs alone at 95.4 to 100% recall and 49.6 to 85.0% false positives [72]; they also evaluate how deeply LLMs understand web payloads [73]. Confidence can be read from token likelihoods [31] or verbal statements [75].

close request

slow tier LLM or analyst

Fitted on benign traffic and the known families only Supervised detector: training, recalibration and thresholds. Pretrained model, used zero-shot: recalibration and thresholds. Thresholds: a benign budget 𝑏 , or conformal shares 𝛼 and 𝛽 .

Figure 1: The two-tier pipeline and what a deployment fits for each kind of fast tier. A closed request reaches the application unexamined; only escalated requests reach the slow tier.

Every request goes first to the fast tier, which can close it as benign, raise an alert (report it as an attack without the slow tier), or escalate it; only escalated requests reach the slow tier, an arrangement that follows the dual-process account of judgment [32], from which TypeSafe takes the name of its System One models. For an attack the fast tier does not recognize, the useful outcome is escalation, and because every benign request escalated costs the slow tier time, we bound escalation by a benign budget, the share of benign traffic the slow tier is prepared to see. The fast tier can sit inline, deciding before a request reaches the application, or read a copy of the traffic, where a request it closes has already been served and attacks are detected after the fact. Our escalation measurements apply to both and bound what the pipeline detects, since the slow tier still has to recognize an escalated attack from the same request the attacker wrote.

the edge, and so can requests that carry a recognizable instruction to the classifier. Legitimate requests that merely look unusual cannot be blocked this way, because real customers send them. We measure how often such attack-like benign requests are escalated, which is also the false-escalation load they impose in normal operation, and how often benign requests padded with opaque parameters are escalated. Anyone can query a hosted model, or run an open one with our published questions, and tune an attack against it, as our search does; a detector trained on an operator’s own traffic stays private.

Fast-tier models

4 Methodology 4.1 Data

Cloudflare’s Clef and Clef-flash add a decision head and low-rank adapters, trained with a Brier loss for calibration, to Qwen language models of 27B and 9B parameters [12]; their weights are public, but we query Cloudflare’s hosted service [15]. Table 1 lists the decision models with the fast tiers an operator would otherwise deploy.

3.3

alert escalate

3 Background 3.1 Two-tier triage

3.2

fast tier

Benign traffic. For the 2026 edition of its WAF comparison, the open-appsec project published 1,040,242 HTTP requests that its authors recorded while browsing 692 real websites in 14 categories and using them as customers do: signing up, searching, filling carts and uploading files [64]. A browser records every request a page triggers, so the traffic includes third-party trackers, content delivery networks and analytics alongside the sites themselves. We represent each request in a compact form made of the request line, the Host and Content-Type headers, and the body; the other headers come from the one recording browser or hold session state such as cookies. After duplicates are removed, 773,363 distinct requests remain. We keep those of at most 1,000 characters that carry at least one query, form or JSON-string parameter, which leaves 388,691 requests from 6,259 hosts. Every spliced attack travels in a parameter, so without the parameter condition the mere absence of parameters would mark a request as benign.

Threat model

We consider a web service behind a two-tier pipeline whose fast tier was fitted (trained, recalibrated or given thresholds) on traffic that contains a set of known attack families, and three adversaries. The first holds an attack from a family on which the fast tier was never fitted and succeeds when the fast tier closes it. We reproduce this adversary by holding the family out and measure the share of held-out attacks that each routing rule closes, escalates or alerts on. The second controls the rest of the attack request. It may add a parameter that asserts the request is benign or instructs the classifier, append such text to the attacked value, percent-encode the payload so that the server decodes it unchanged, pad the request so that the payload falls beyond a model’s input window, or carry the payload in a header. We test a fixed battery of such edits and a search, guided by the target model’s probability, over the same kinds of edits (Section 4.5). Resubmitting a request until a nondeterministic model closes it is out of scope, though each copy the model escalates reaches the slow tier. The third wants to exhaust the slow tier. Under a budget rule every escalated request costs the slow tier one examination, whatever it contains, so the adversary needs requests the fast tier escalates. Blatant attacks would do, but they can be blocked or rate-limited at

Attacks. The same project packages the mgm WAF Payload Collection [46] into seven families (Table 8 in Appendix E). The project’s harness sends every payload in the same request, a GET to /?p= or a POST whose body is p=, so a classifier given only the method, Content-Type, Host, path and parameter names, the format control, separates these attacks from benign traffic with an AUROC of 1.000, without reading a single payload (Appendix D). We splice each payload into one real request instead, replacing the value of one parameter, so that attacks and benign traffic share hosts, paths and parameter names; the format control then reaches an AUROC 3

Kisielewicz, Kędziora and Nguyen

Table 1: The fast tiers we compare. Decision models answer a typed question with a probability in one forward pass. Prices are per million input tokens (output is not charged). Questions: how many of our five questions a model answers per request (Section 4.3). Model

Kind

Parameters

Access

Version

Released

Price

Questions

Jev [2, 77] Clef [12, 13] Clef-flash [12, 14] Laya, von, Verdict [16, 27, 83] Qwen2.5-7B [60] ModernBERT [81] Laya (ft) [16] n-gram model OWASP CRS [52] Anomaly baseline

decision model decision model decision model decision models language model supervised supervised supervised rule-based WAF benign-only

not disclosed 27B (Qwen base) 9B (Qwen base) 151M–421M 7.6B (4-bit) 149M 421M 50,000 features rule set per parameter

hosted (TypeSafe) hosted (Workers AI); open weights hosted (Workers AI); open weights local local (Ollama) fine-tuned per fold fine-tuned per fold trained per fold ModSecurity v3 trained on benign traffic

jev-1.13.0 model name only model name only pinned revisions Q4_K_M 3 seeds 1 seed – 4.30.0 and 3.3.2 –

15 Sep 2026 1 Oct 2026 1 Oct 2026 17–19 Sep 2026 19 Sep 2024 base 19 Dec 2024 – – 2 Oct 2026; 30 Jun 2021 –

$0.042 $0.24 $0.038 – – – – – – –

5, one call 5, one call 5, one call 5 1 – 1 choice – – –

of 0.496. Request and parameter are chosen by a hash of the payload, with no regard to what the parameter means, so a traversal payload may land in a search field. Payloads enter query and form values exactly as the harness transmits them, which matters for 1,718 of the 3,000 traversal payloads we keep, whose escapes, such as %c0%ae, decode to bytes that are not valid UTF-8. A JSON string value receives the payload decoded once. In a parameter value, a Shellshock or XML external entity payload tests only whether a fast tier recognizes the string: CGI exports the query string whole, so no value reaches bash as a function definition, and an entity acts only where an application parses the value as XML. Section 6.5 also places Shellshock in headers, which a CGI server exports to bash. Spliced attacks are not held to the benign length cap; leaving out the longer ones moves no model’s held-out escalation by more than 5 points (Appendix E). We leave out the Padding Evasion family that the 2026 edition adds, payloads behind kilobytes of junk, and test padding separately (Section 4.5).

with an attack class such as SQL injection, file inclusion, server-side request forgery, file upload or authentication bypass. An exploit’s class is the first of its tags in a fixed order; tags describe a vulnerability’s impact as well as its payload, so the classes are approximate. We evaluate each template’s expressions as the scanner does, offline with random values drawn from a fixed seed, and set the Host header to a host from the benign traffic. An exploit enters the test when the compact form shows all of it and it carries attack content, which leaves 423 exploits with 813 distinct requests. Of the other 201 exploits, 126 carry part of the attack in a header the compact form drops, and the rest are too long, cannot be reproduced offline, only request a page, or concern an earlier year. Of the 423, 114 (file uploads, authentication bypasses and server-side request forgeries) have no counterpart among the payload families. An exploit counts as escalated when any of its requests is, and as closed when all of them are; the test measures recall only. Thresholds come from the open-appsec folds, outcomes are averaged over the seven folds, and NVD publication dates [49] place each exploit before or after a model’s release.

Attack-like benign traffic. Before running any model we fixed ten patterns that signature rules react to, such as markup, quotes beside SQL keywords, ../, template delimiters and shell separators before a command name. We searched for them in the decoded parameter values of the 64,068 benign requests of the test split left out of the 10,000 sampled for testing (Table 8). The 503 matches form an attack-like benign slice; most come from ordinary page behavior, such as trackers that send ;cat=all or template macros like ${TA_DEVICE_ID} that a page failed to expand.

4.2

Leave-one-family-out folds

Each fold holds out one attack family, which appears only in the test split: training, recalibration and every threshold see benign traffic and the remaining families. Before splitting we set aside 5% of the data, chosen by hash, as a development set, used only to fix the serialization, to check the question wording on Jev, Laya, von and Verdict, the four decision models of the analysis plan (Section 4.6), and von’s controller setting (Appendix E), and to screen a model trained before Log4Shell (Section 9.2); the wording, written before any run, did not change. The remaining items are split 60/20/20 into training, calibration and test by a hash of their text; for openappsec an attack’s split is decided by its payload, so a payload never reaches two splits. Cross-site scripting, path traversal and command execution, the three families with at least 1,000 held-out test attacks, form the main folds (Table 8); SQL injection, Log4Shell, XML external entities and Shellshock are case studies with Wilson intervals. What a hold-out removes depends on the fast tier (Figure 1). A supervised detector never meets the family in training, recalibration or thresholds. A pretrained model is fitted only through its recalibration map and thresholds; the map is monotone and the budget threshold is a quantile of benign scores, so its budget-rule decision

ECML/PKDD 2007. For comparison with earlier work we also use the 50,116 requests of the ECML/PKDD 2007 discovery challenge [62], 35,006 valid and the rest attacks of seven curated types, six of which we hold out in turn (Appendix C). CSIC 2010. As a second benign source we use the normal traffic of CSIC 2010 [76], synthetic requests to one web shop, in the compact form and under the length and parameter rules of the main corpus, which leaves 18,620 distinct requests; the same payloads are spliced in the same way. Exploits for recent vulnerabilities. The payload lists are public and old enough that they may have reached a pretrained model’s training data. As a test less exposed to pretraining, we converted into requests the nuclei templates [57] for vulnerabilities given a CVE identifier in 2025 or 2026, keeping the 624 templates tagged 4

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

on a request is the same whichever family is held out. Whether a family is new to a pretrained model depends on its pretraining, which the providers do not document in full, so the hold-out tests what fitting to known families does to a fast tier, and the exploit tests (Section 4.5) probe the pretrained models further.

4.3

at levels 1 to 4. We use release 4.30.0 and release 3.3.2 of 30 June 2021, from before Log4Shell; “the rule set” means release 4.30.0 at paranoia level 1 unless we say otherwise. Every request is sent in its compact form with the same fixed browser headers, and a request that nginx rejects before scoring gets the highest score. The anomaly baseline is fitted on benign training requests only, after Kruegel and Vigna [35]: for every parameter name it models the length of the decoded value (Chebyshev bound) and its character distribution (chi-square against the idealized distribution), and a request scores the largest anomaly among its parameters. Neither baseline is trained on attacks, so holding a family out changes neither’s budget decisions. The rule set’s conformal rule takes its attack threshold from the known families’ calibration attacks and is reported with the other models’; the anomaly baseline is reported as escalation within the benign budgets only.

Model settings

Decision models. All six decision models answer the same frozen question set. The primary question is “Does this HTTP request attempt to attack, exploit, probe or abuse the web application?” The set adds its inverse (“Is this HTTP request ordinary, legitimate use of the web application?”), two paraphrases (“Is this HTTP request malicious?” and “Would a security analyst flag this HTTP request as an attack?”) and one choice among legitimate use, an attack, and a malformed but harmless request. A model’s attack probability is its probability of yes to the primary question (for von and Verdict, see Appendix E). We left out Decider and Bespoke Nimble, built on language models of about 2B and 9B parameters [42], a class Clef-flash and Qwen2.5-7B cover. We also left out Cyber-Jev [71], a classifier fine-tuned on web attacks that include our families, which makes it a supervised detector.

4.4

Recalibration and routing

Every model’s score is recalibrated per fold by monotone Platt scaling [50, 55] on its logit, fitted on the calibration split with the two classes weighted to equal mass; a negative fitted slope is replaced by a constant, so an inverted model is reported as uninformative and never flipped. On the recalibrated score we compare four routing rules. Budget. Escalate when the score exceeds the (1 − 𝑏)-quantile of benign calibration scores, for budgets 𝑏 of 1 and 5% (and 0.1% on the full benign pools, Section 4.5); a request below it is closed. Ties, frequent in models that round or saturate their outputs, are broken by a deterministic function of the request text, so a budget is always spent exactly; at 5% the tie-breaker settles at most 1.4% of test decisions, except for Laya (15%); the rule set, whose scores are integers (up to 6.5%); and Verdict, whose recalibrated score is constant. Conformal dual threshold. Class-conditional conformal sets [3, 78] admit the label attack when the score is at least 𝑡 low , the 𝑘-th smallest of the 𝑛 calibration attack scores, 𝑘 = ⌊𝛼 (𝑛 + 1)⌋, and the label benign when it is at most 𝑡 high , the mirror threshold on benign scores with share 𝛽. A request is closed when only benign is admitted, alerted when only attack is, and escalated when both or neither are; the empty set marks a score between the two classes’ thresholds. For attacks exchangeable with the calibration attacks, the rule closes at most a share 𝛼 in expectation; we use 𝛼 = 5% and 𝛽 = 1%. A held-out family is not exchangeable with the known ones, so we measure the share of it the rule closes [7]. Disagreement and combinations. Escalate when either the score or the spread of the four yes-or-no answers exceeds its threshold, or when either of two models’ scores does; the two thresholds share the budget so that together they escalate exactly the budget of benign calibration traffic. A cascade instead lets a first stage (a supervised detector or the rule set) forward the share of traffic it ranks highest to Jev, which decides on those requests. The budget and conformal rules depend only on the order of the scores, so recalibration changes none of their decisions, except where it reduces a model to a constant (Verdict); calibration matters for the confident-miss rate and when a model’s own probability is used as a threshold (Section 5). When the known classes separate, so that 𝑡 high < 𝑡 low , the conformal rule closes exactly the requests

Supervised detectors. The n-gram model is a logistic regression over TF-IDF weights of the 50,000 most frequent character 3- to 5-grams within whitespace-delimited tokens, trained on the full training split of each fold. ModernBERT-base [81] is fine-tuned per fold for one epoch on 8,000 benign training requests chosen by hash and every known-family training attack (batches of 16, learning rate 3 · 10−5 , at most 512 tokens, bf16), with seeds 17, 23 and 42, its main runs. Training on the GPU is not bit-for-bit reproducible, so even a repeated seed gives a new model; we trained ModernBERT on the traversal and Log4Shell folds 10 and 7 times in all (Section 5). Analyses that score new texts (exploits, controls, manipulations, small budgets) use one retraining of seed 17 whose checkpoints we kept, its kept run; of the 10 runs on the traversal fold it escalates the most held-out traversal (81%), so using it errs in ModernBERT’s favor. Elsewhere ModernBERT’s results on the folds are the mean over its main runs unless we name a run. Laya (ft) is Laya finetuned per fold with its own training loop [16], for one epoch on ModernBERT’s training selection with one seed, and asked the primary question as a two-option choice with neutral option labels, as its card recommends; it runs on open-appsec only. Nearly all payloads fit the input windows: the payload of 1.2% of held-out attacks (0.9% of traversal) ends beyond ModernBERT’s 512 tokens. Local language model. Qwen2.5-7B-Instruct [60] runs in Ollama 0.40.2 with Q4_K_M quantization, its chat template and default system prompt, at temperature 0. It answers the primary question followed by “Answer with one word: yes or no.” Its attack probability is P(yes)/(P(yes)+P(no)) over the first generated token, each word’s probability summing its case and spacing variants among the 20 most likely tokens. Rule-based and benign-only baselines. The CRS [52], the open rule set for ModSecurity, scores every request in ModSecurity v3 in detection-only mode with all rules up to paranoia level 4. At the end of request processing, an extra rule logs the inbound anomaly score each paranoia level contributes, so one pass gives the score 5

Kisielewicz, Kędziora and Nguyen

at or below the benign (1 − 𝛽)-quantile: the requests a budget of 𝛽 leaves unescalated. We verified this equality on every such fold, which is every fold for Jev, ModernBERT and Laya (ft).

4.5

version, scored the rest of the pools twelve days after the folds; asked again, they answered 200 fold requests identically. Exploit tests and their controls. In a temporal test we trained ModernBERT and the n-gram model, as a deployment with older exploit data would, on all seven families plus the attack-carrying requests of exploits for CVEs from 2015 to 2024 (2,190 exploits converted from the same nuclei snapshot with the same rules, 80% for training and 20% for calibration by a hash of the CVE). An ablation used the seven families alone; both are tested on the 2025– 2026 exploits. The exploits come from a scanner’s templates, so they differ from browsing traffic in shape (admin and API paths, random tokens, a callback host) as well as in attack content, and three controls separate the two. Benign twins: in 62 exploit requests whose attack sits in a parameter value, we replaced that value with a legitimate one (a search term, a file or user name, a samesite URL), keeping everything else, the encoding included; uploads and most authentication bypasses, whose attack is the file or the request itself, cannot be twinned. Re-rendering: new random values and callback subdomain, or neutral ones (word-like tokens and the callback under the request’s own host). Shape only: the format control of Section 4.1, which reads no parameter value, trained and calibrated like the temporal test’s detectors.

Robustness and realism tests

Manipulation battery. We took 704 held-out attacks (200 from each main fold and all of Log4Shell, by hash) and 300 benign test requests, and applied five non-adaptive edits fixed before any model scored them: a parameter asserting that the request is a routine health check and not an attack; a parameter addressed to the classifier, asking it to answer that the request is legitimate; the same assertion appended to the attacked value; every literal character of the payload percent-encoded, which a server decodes to the same bytes; and about 3,000 characters of opaque parameters placed before the payload. Each edited request is scored at its fold’s clean thresholds, and ModernBERT’s kept run is compared with its own clean decisions. Adaptive search. For 100 of the battery’s held-out attacks (30 each of traversal, cross-site scripting and command execution, 10 of Log4Shell) a best-first random search in the manner of WAFA-MoLE [18] keeps the five candidates the target scores lowest, mutates one with one or two edits, and queries the target once per new candidate, until a candidate falls below its fold’s 5% budget threshold or the queries run out. The edits keep the attack’s effect where added parameters go unread and the server decodes percentencoding: added parameters (a benign assertion, an instruction to the classifier or tracking-style padding); percent-encoding part of the payload; and, per family, ./ segments in a path, changed letter case in HTML tag and attribute names, empty quotes or carets in a shell word, and ${lower:x} lookups in Log4Shell. The adversary reads only the model’s probability, as a public interface returns it. Jev, Clef-flash and Qwen2.5-7B get 300 queries per attack and Clef 150, on the first 79 attacks of a rotation over the families, where our spending cap for Clef ended the run; we leave out the 4 searches the cap cut short. Requests found against one model are then scored by the others, and those that got past Jev or Clef are asked again 5 times.

Prospective exploits. An exploit for a vulnerability that NVD published after a model’s release is unlikely to be in that model’s pretraining data, though publication can follow first disclosure, and Clef may have changed since its release. From a second nuclei snapshot, of 23 October 2026 [58], we took the templates for CVEs that NVD published after 15 September 2026, Jev’s release, and kept those that pass the rules above: 9 exploits, 5 of them published after Clef’s release on 1 October.

4.6

Metrics and statistics

We read each test split as two slices, benign requests with knownfamily attacks and benign requests with the held-out family, both weighted to equal class mass. Equal mass is a reference prior: at a base rate of one attack in a thousand, 𝑝 = 0.9 at equal mass corresponds to a posterior of about 0.9%, so routing is set by benign budgets. A confident miss is an attack given a probability below 0.05 at equal mass, which corresponds to a likelihood ratio below 1:19; the probability is the recalibrated one unless a table gives the model’s own. Recalibrated scores are summarized by the square root of the debiased squared L2 calibration error of Kumar et al. [36] over 15 equal-mass bins, which corrects for bin size; each model’s own probabilities by the L1 expected calibration error [53], the quantity calibration claims usually refer to. We report calibration error with the AUROC, because a constant score has zero calibration error. Every escalation rate is reported with the realized escalation of benign traffic. Three comparisons are primary and confirmatory; every other analysis is exploratory. We fixed them in an internal, unregistered analysis plan written before the first leave-one-family-out run; Clef, Clef-flash, Laya (ft) and ModernBERT’s seeds 23 and 42 came after it. For every model but the two baselines, P1 is the calibration error on the held-out slice minus that on the known slice, and P2 the same difference in confident-miss rate; for the six zero-shot

Headers. Log4Shell reaches a vulnerable logger through any header an application logs, and Shellshock reaches bash through headers a CGI server exports. We rebuilt every request in full from the recording, with the browser’s other headers but without cookies or credentials, and placed each held-out Log4Shell and Shellshock payload, decoded once, three ways in its benign context: in the parameter as before, as the User-Agent, and in an added X-Api-Version header. Thresholds come from 2,000 benign calibration requests in full form, and as many test requests give the realized benign rate. Our supervised detectors read the compact form, which has no such headers, so in this test they see neither header placement. Small budgets. A 0.1% budget would rest on the ten highest of the folds’ 10,000 calibration scores. We therefore scored every benign request of the calibration and test splits, 73,491 and 74,068, with the hosted decision models, the n-gram model, ModernBERT’s kept run, the rule set and the anomaly baseline, and set a 0.1% budget on the full calibration pool. Clef and Clef-flash, served without a 6

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Table 2: Primary comparisons P1 and P2 (Section 4.6), main open-appsec folds: held-out minus known calibration error and confident-miss rate (points) after recalibration, with 95% intervals, and AUROC on known and held-out families. ModernBERT: main run, seed 17. † : nonzero after Holm’s correction; ‡ : added after the analysis plan, a Holm family of its own.

known

held

P1: calib. error

P2: conf. miss

0.998 0.997 0.996 0.995 0.931 0.933

+0.019 [0.011, 0.023]† +0.026 [0.018, 0.033]† +0.012 [ − 0.003, 0.020]

+0.4 [0.1, 0.8] +0.3 [ − 0.1, 0.7] − 0.1 [ − 0.5, 0.3]

Qwen2.5-7B

0.971

0.971

+0.008 [ − 0.001, 0.016]

+0.3 [ − 0.2, 0.8]

ModernBERT Laya (ft)‡ n-gram model

0.998 0.959 0.995 0.906 0.987 0.829

+0.217 [0.209, 0.228]† +0.180 [0.170, 0.190]† +0.190 [0.177, 0.200]†

+20.8 [20.0, 21.6]† +29.0 [28.0, 30.0]† +20.7 [19.8, 21.6]†

Jev Clef‡ Clef-flash‡

known families

attack rate

Jev

5.1

held-out family

ModernBERT

n-gram

1.0 0.5 0.0 0.0

0.5

1.0 0.0

calibrated p

0.5

1.0 0.0

calibrated p

0.5

1.0

calibrated p

Figure 2: Observed attack rate against recalibrated probability, pooled over the main folds, for known-family attacks (filled) and held-out attacks (hollow); ModernBERT: main run, seed 17. Points above the diagonal at low probability are attacks the model calls benign with confidence.

5.2

Other maps and decoded requests

The misses do not depend on Platt scaling: with temperature scaling or isotonic regression, wherever the map stays calibrated on the known families, the supervised detectors give 20% to 33% of heldout attacks a probability below 0.05 and the hosted decision models at most 1.3% (Table 17). Nor do they come from how the benchmark encodes payloads (Appendix D). Retrained with seed 17 on requests whose query and form values are all percent-decoded, ModernBERT gives 61.6% of held-out traversal a probability below 0.05, against 59.1% for its seed-17 main run, and escalates at least as much of held-out cross-site scripting and command execution as that run (99.8% and 97.8%). The n-gram model does lean on the encoding: refitted the same way, it still escalates 87% to 97% of known-family attacks but only 8% to 61% of a held-out family other than traversal, against 36% to 94% before.

decision models, P3 compares held-out escalation at a 5% budget under the disagreement rule against the budget rule. P1 and P2 are pooled as the mean over main folds, with 95% intervals from a stratified Bayesian bootstrap [65] of 10,000 draws; P3 pools the paired decisions of all main folds in an exact McNemar test. Holm’s correction runs over the primary comparisons of the models in the analysis plan and, separately, over those of the three models added after it; the intervals themselves are not adjusted. We also report P2 per main family (Table 21) and, because a family’s attacks are often variants of a few payloads, with a cluster bootstrap that gives one weight to all requests sharing a payload skeleton (the payload decoded until stable, lower-cased, and every run of characters other than letters and digits collapsed to one space).

5

Supervised detectors miss with confidence

Recalibrated on the known families, as a deployment would be, the supervised detectors are calibrated on those families and miscalibrated on held-out ones. Their calibration error and confident-miss rate grow significantly (P1 and P2 in Table 2); outside the tested comparisons, they also grow on each main family alone, most on traversal (Table 21), and by similar amounts in ModernBERT’s other two main runs (Table 22). ModernBERT’s main runs separate held-out traversal from benign traffic with an AUROC of 0.889 on average, yet give 56% of these attacks a probability below 0.05, and 38% of held-out Log4Shell; on the traversal fold their 5% budget threshold lies below 0.05, so the budget rule still escalates some of these confident misses (Appendix E). The share depends on the training run: over 10 runs that differ only in the seed or the GPU’s nondeterminism, it ranges from 37% to 60% for traversal, and over 7 runs from 12% to 51% for Log4Shell (Table 12), yet every run on the traversal fold escalates at least 98.4% of known-family attacks. Models that fit the training data equally well can differ widely on data unlike it [17]; an ensemble that averages all runs’ probabilities stays within these ranges on traversal (Appendix E). A decision model fine-tuned on the known families fails the same way: Laya (ft) ranks held-out traversal at an AUROC of 0.739 and gives 76% of it a probability below 0.05, whereas zero-shot Laya gives none. The n-gram model ranks held-out traversal barely above benign traffic (AUROC 0.553). Figure 2 shows how the misses arise: on known families the supervised detectors lie close to the diagonal, but on a held-out family many attacks score in the range the recalibration map associates with benign traffic, so the map calls them benign with high confidence even when they score above most benign requests.

Held-out − known

AUROC Model

largely can, though for them the hold-out withholds a family only from recalibration and thresholds.

Calibration on Held-Out Attack Families (RQ1)

5.3

Routing by a benign budget uses only the order of the scores (Section 4.4), but an analyst or a fixed threshold reads the probability itself. On held-out families the low probabilities of the supervised detectors cannot be trusted, and those of the hosted decision models

Hosted models stay nearly calibrated

Before recalibration, Jev’s and Clef’s own probabilities are as well calibrated on held-out families as on known ones (expected calibration error 0.08 and 0.07; Table 10 in Appendix E). A fixed threshold of 0.5 does not carry over between benign sources: Jev scores 96% 7

Kisielewicz, Kędziora and Nguyen

to 100% of every held-out family above it, but also 91.8% of the synthetic benign requests of ECML/PKDD 2007 (Appendix C). Jev and Clef rank held-out families nearly as well as the known ones (AUROC at least 0.993 for both on every fold); after recalibration their calibration errors grow by 0.019 and 0.026, significant but small, and their confident-miss rates by less than half a point (Table 2). Clef-flash ranks attacks less sharply (held-out AUROC 0.895 to 0.996), and neither its calibration error nor that of Qwen2.5-7B changes significantly. Asked zero-shot, the small open models carry little information about attacks: with the primary question Laya and von reach a held-out AUROC below 0.80 on every fold, and Verdict ranks attacks below benign traffic, so recalibration reduces it to a constant (Table 24).

6

traversal

escalated (%)

other five families

CRS 4.30.0 ModernBERT

n-gram

50 0 1 0.1

5

1 0.1

5

1 0.1

5

1 0.1

5

1 0.1

benign budget (%)

Figure 3: Held-out attacks escalated at benign budgets of 5, 1 and 0.1%, with thresholds set on all 73,491 benign calibration requests (Section 4.5). ModernBERT: its kept run (Section 4.3).

Escalating Held-Out Attacks (RQ2) 6.3

No guarantee for a held-out family

On known families the conformal rule closes about the 5% of attacks its guarantee allows or fewer, with a small excess for Clef-flash (Table 4) and for Laya (Appendix E). When the known classes separate, as for Jev, ModernBERT and Laya (ft) on every fold, the rule closes exactly what a 1% budget leaves unescalated (Section 4.4): Jev’s rule closes at most 2.5% of any held-out family, and ModernBERT’s 65% of held-out traversal. The rule’s benign load, escalated or alerted, depends on how well the known classes separate, from 0.9% of benign traffic for Jev to 32.6% for Clef-flash. Unlike the budget rule, the conformal rule escalates a request whose score falls between the two classes’ thresholds, an empty prediction set; this is how ModernBERT escalates 57% of held-out cross-site scripting at a benign load under 1%.

Supervised detectors lose traversal

For Jev and Clef, as for every zero-shot model, the hold-out changes no budget decision (Section 4.2). The other decision models escalate less within the 5% budget (Table 3): Clef-flash 68% to 97% of each family and the small open models at most 23% (Table 24). The supervised detectors lose held-out traversal, and the same requests show that holding traversal out causes the loss. Of the traversal attacks in the other folds’ test splits, where traversal is a known family, ModernBERT escalates 99.8%, and 55.4% once traversal is held out; Laya (ft) falls from 99.3% to 31.1% and the n-gram model from 97.3% to 2.1%. Over its 10 training runs ModernBERT escalates 50% to 81% of held-out traversal, although it comes within two points of Jev on four of the seven families. The fast tiers trained on no attack data cannot lose a family this way: at paranoia level 1 the rule set escalates 81% to 100% of each family and the anomaly baseline 46% to 94%; higher paranoia levels mostly escalate less within the budget (Table 15). With CSIC 2010 as the benign traffic, Jev and Clef escalate at least 99.8% of every held-out family, and the supervised detectors again fall short on traversal, though less (Table 18).

6.2

Clef

100

5

Under the plain budget rule at a 5% budget, Jev and Clef escalate at least 97.6% of every held-out family while the supervised detectors lose much of traversal; at a 0.1% budget Clef escalates as little as 68% of a family. A conformal rule’s guarantee does not cover a heldout family, and disagreement between questions or models never escalated significantly more attacks. On recent exploits, much of what a recall test credits is request shape.

6.1

Log4Shell

Jev

6.4

Disagreement and novelty signals

The third primary comparison is negative. At an equal benign budget, escalating on confidence or on the spread among the four yes-or-no answers caught fewer held-out attacks than confidence alone for every zero-shot decision model (P3 in Table 22). Across three pairs of models and the seven held-out families, no union escalated significantly more attacks than its better model alone (Appendix E). A novelty score helps ModernBERT on traversal: with the smallest Mahalanobis distance of its embedding to its benign and known-family training classes [37] as a second signal sharing the budget, the kept run escalates 87% of held-out traversal instead of 81%; no other family moves by more than 4 points (Table 19 in Appendix E).

Smaller budgets

6.5

Analysts may not be able to examine 5% of benign traffic. At a 0.1% budget set on every benign calibration request (Figure 3), Jev still escalates 82% to 97% of each held-out family. Clef escalates only 68% to 91%, least of traversal, and Clef-flash 8% to 63% (Table 13). ModernBERT’s kept run escalates 6% of traversal and 31% of Log4Shell, and the n-gram model 0% and 4%. Release 4.30.0 of the rule set then escalates more held-out cross-site scripting and Log4Shell than Jev or Clef; release 3.3.2 escalates only 11% of Log4Shell, since most of these payloads score 5, the weight of one critical rule, and the threshold now exceeds that score. With 1% budgets set on the calibration split, ModernBERT’s main runs escalate 35% of traversal.

Log4Shell and Shellshock, replayed

With the other six families known, the supervised detectors’ conformal rules close 17% (Laya (ft)) to 80% (n-gram model) of held-out Log4Shell, and those of Jev and Clef 2% and 3% (Table 4). Within the budget Jev and Clef escalate 100% and 99% of it, and 100% and 99% of the payloads that nest the lookup, such as ${${::-j}ndi:. . . }. Shellshock is easier: the hosted models, Qwen2.5-7B and the supervised detectors each escalate at least 91% of it, though in a parameter it tests only recognition of the string (Section 4.1). For the rule set the replay can also be faithful in time. Release 3.3.2, from June 2021, before Log4Shell was disclosed in December, has no rule for JNDI lookups, but its level-1 rule for Unix 8

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Table 3: Share of held-out attacks (%) escalated when each fast tier may escalate 5% of benign traffic, per held-out family; the realized share of benign test traffic escalated (Benign); and the share of the 423 recent exploits escalated (Section 6.6). ModernBERT: mean over its three main runs, exploits: its kept run (Section 4.3); –: not run on the exploits; CRS: paranoia level 1. The last four families are case studies with fewer than 1,000 held-out attacks; for an observed 100% the 95% Wilson lower bound is 85% (Shellshock), 90% (XXE), 96% (Log4Shell) and 99% (SQL injection). Model

XSS

Trav.

Cmd.

SQLi

L4Sh.

XXE

ShSh.

Benign

Exploits

Jev Clef Clef-flash

100 100 70

99 98 76

98 98 74

99 99 68

100 99 81

100 100 97

100 100 91

5.2 4.5 5.1

98 97 92

Qwen2.5-7B

88

82

89

90

85

94

96

4.9

82

ModernBERT Laya (ft) n-gram model

98 94 88

56 32 3

98 97 86

94 93 68

82 98 36

100 100 94

100 100 91

5.1 4.8 4.6

68 – 41

CRS 4.30.0 CRS 3.3.2 Anomaly baseline

100 100 63

86 85 46

95 96 56

81 88 55

100 100 60

97 100 94

87 96 70

5.0 5.3 4.5

66 64 11

Table 4: Share of held-out attacks (%) closed by the conformal rule fitted on the known families (𝛼 = 5%, 𝛽 = 1%); known: share of known-family test attacks closed; Load: benign test traffic escalated or alerted (%, main folds). ∗ : above 5% with 𝑝 < 0.001 (exact binomial). Model

Table 5: Controls for the exploit tests (Section 4.5): recent exploits escalated (%); 62 exploit requests / their benign twins escalated (%); change under a neutral rendering (points); exploits published after 15 September 2026 escalated (of 9, averaged over folds and rounded). Rows below the line: trained with the 2015–2024 exploits as well; No values: the format control. ModernBERT above the line: its kept run.

known XSS Trav. Cmd. SQLi L4Sh. XXE ShSh. Load

Jev Clef Clef-flash

1.3 3.5 5.9

0 1 6

2 9∗ 5

2 4 5

2 4 11∗

2 3 5

0 0 0

0 0 4

0.9 1.5 32.6

Qwen2.5-7B

4.8

4

9∗

4

4

8

0

0

16.6

ModernBERT Laya (ft) n-gram model

1.2 5 2.3 14∗ 4.7 10∗

65∗ 74∗ 90∗

4 6 18∗

11∗ 18∗ 38∗

43∗ 17∗ 80∗

0 0 6

0 0 4

0.9 1.0 6.7

CRS 4.30.0 CRS 3.3.2

4.8 4.5

17∗ 29∗

4 3

13∗ 11∗

0 0

0 0

0 4

7.0 22.7

0 0

Model

shell expressions (rule 932130) matches ${. . . } in parameters. As shipped, blocking at an anomaly score of 5, it blocks 100% of the held-out Log4Shell payloads carried in a parameter, at 2.4% of benign requests. The nuclei template for Log4Shell also places the payload in seventeen headers, User-Agent and X-Api-Version among them [57], and rule 932130 does not read headers: there the same rule set blocks 2% of the payloads, while release 4.30.0 blocks them in every placement as shipped. In requests with headers, where Shellshock can reach bash (Table 11 in Appendix E), Jev escalates every placement of both families and Clef at least 99.0%.

6.6

Exploits

Twins Neutral After

Jev Clef Clef-flash Qwen2.5-7B ModernBERT n-gram model CRS 4.30.0

98 97 92 82 68 41 66

100/37 98/27 90/13 84/2 67/1 39/3 74/2

− 1.4 − 1.9 − 4.7 − 1.4 − 0.5 − 0.4 − 0.2

9 8 9 7 6 3 6

n-gram, temporal ModernBERT, temporal No values, temporal

94 87/52 99 100/89 93 100/100

− 3.1 0.0 –

– – –

benign requests: net of its twins it escalates 63 points more of the exploit requests, fewer than the rule set (73) or Qwen2.5-7B (82). Rendering the exploits again changes little (at most 1.7 points with new random values and 5 with neutral ones). The controls also change what the temporal test shows. With the older exploits in training, the n-gram model escalates 94% of the recent exploits, against 34% in the ablation, and 71% of the authentication bypasses (Table 14), but also 52% of the benign twins (none in the ablation). ModernBERT, trained the same way, escalates 89% of the twins (2% in the ablation), and the format control, which never sees a parameter value, escalates 93% of the recent exploits after the same training. These detectors learned what scannergenerated exploit requests look like, which a test of recall alone would credit as learning the attacks. Of the exploits published after each model’s release, Jev escalates all 9 (95% exact lower bound 66%) and Clef 4 of 5. On all recent exploits their conformal rules close 5% and 7% on average over folds, but not evenly: 23% and 47% of the 35 authentication bypasses (exact binomial 𝑝 < 0.001 against 5% for both). Many of the exploits Jev’s rule closes look like ordinary use of an application’s interface

Exploit tests need shape controls

Within the 5% budget Jev escalates 98% of the 423 exploits and Clef 97% (Table 3; by attack class in Table 16). Qwen2.5-7B, released before every dated exploit [59], escalates 82%, while the fast tiers trained on no attack data do worse on the exploits than on the payload families. The benign twins (Section 4.5) show how much of this is shape (Table 5). Jev escalates 37% of the twins and Clef 27%, against a 5% budget, while the rule set, Qwen2.5-7B and the supervised detectors escalate at most 2.6%. Jev scores every exploit request above its twin (paired AUROC 1.000; rule set 0.895), but its threshold, set on browsing traffic, lies below these scanner-shaped 9

Kisielewicz, Kędziora and Nguyen

Table 6: Held-out attacks / benign requests (%) escalated at each model’s clean 5% thresholds, unedited (clean) and with a parameter asserting the request is benign (note) or instructing the classifier (instr.), the assertion appended to the attacked value (value), the payload percent-encoded (enc.), or about 3,000 characters of padding before it (pad). ModernBERT: its kept run (Section 4.3). Model

clean note

enc.

pad

Jev Clef Clef-flash

99/5 99/5 100/100 99/18 99/10 98/4 99/25 100/99 100/91 94/7 72/5 72/7 79/24 76/9 55/10

instr.

value

99/43 100/97 77/38

Qwen2.5-7B

88/4

23/0

61/0

24/0 76/23

84/29

ModernBERT n-gram model

93/5 98/56 58/5 91/59

99/77 98/90

98/54 96/80 88/58 57/8

43/50 15/0

95/3 96/6 53/3

95/3 96/5 75/75

CRS 4.30.0 95/4 CRS 3.3.2 96/5 Anomaly baseline 53/3

95/3 96/4 53/3

choices [84]: the search got past Jev for 56% of the attacks Jev first scored below 0.93, 35% of those from 0.93 to 0.96 and 19% of those from 0.96. The requests it found score 0.16 to 0.22, just below the threshold, and Jev’s answers vary: asked 5 more times, Jev escalated them in 60% of its answers. Of the 43 requests found, 18 stayed past Jev in most answers and 4 in all; the others passed on a favorable answer. Clef answered identically when asked again: it lost 26% of 72 attacks within 150 queries, Jev 28% of the same attacks, and every request that got past Clef still did on the repeated calls. Clef-flash and Qwen2.5-7B lost every attack they escalated, after a median of 7 queries each. The successful requests transfer poorly: none that evaded Qwen2.5-7B evades Jev or Clef, and 5% of those that evaded Jev evade Clef. The search reconciles our battery with JevAdvBench, in which one appended opinion flips 12.1% of Jev’s general decisions [29]: no fixed edit moves Jev’s mean recalibrated score of an attack by more than 0.026, while a search accumulates small moves. Smaller budgets help the search, because their thresholds lie higher: its best request falls below Jev’s 1% threshold (Figure 3) for 71% of the 98 attacks Jev escalates at that budget, and below its 0.1% threshold for all 89 it escalates there. Against Clef’s thresholds it succeeds for 75% of 68 attacks at 1% and 97% of 60 at 0.1%.

91/4 95/3 93/5 95/5 53/3 100/100

(Table 23 in Appendix E): a token request or a password reset is hard to judge without the application’s state, and a harmless file sent to a vulnerable upload form carries no attack content. One is a miss on visible attack content, an unauthenticated call to an agent server’s terminal_execute tool with the command id (CVE-2026-59726), which Jev scores 0.17.

7

7.3

Evasion and Flooding (RQ3)

Against the second and third adversaries of Section 3.3, fixed edits lower Jev’s and Clef’s escalation of attacks by at most a few points, while an injected assertion cuts that of Qwen2.5-7B, and padding that of the supervised detectors, by half or more. A search guided by the model’s probability gets past Jev mostly with requests at its threshold, which Jev often escalates when asked again; against Clef, which answers repeated calls identically, its successes hold. Padded benign requests, which Clef nearly always escalates, can fill the slow tier.

7.1

Fixed edits

Jev escalates at least 98.8% of the attacks under every fixed edit, against 99.4% unedited (Table 6). The assertions, the instruction and padding raise Clef’s escalation from 98% to as much as 100%, but encoding every character of the payload lowers it to 94%. Qwen2.5-7B follows the injected text: the benign assertion lowers its escalation of attacks from 88% to 23% as a parameter and to 24% appended to the attacked value. The rule set escalates 95% of the attacks clean and under each text edit, and 91% encoded. Padding lowers the supervised detectors’ escalation from 93% to 43% (ModernBERT) and from 58% to 15% (n-gram model). On benign traffic the same edits can raise escalation: Jev and Clef escalate 100% and 99% of benign requests that carry the instruction (Clef-flash 24%), and ModernBERT at least 49.9% of the benign requests under every edit.

7.2

Flooding the slow tier

The attack-like benign requests of Section 4.1 are escalated more often than ordinary traffic by the hosted models, Qwen2.5-7B and the supervised detectors, from 8.2% of them by Clef to 33.5% by the n-gram model, against the budget’s 5%. This is the false-escalation load such traffic imposes, and it sets the cost of flooding the slow tier with requests the edge cannot easily refuse (Section 3.3). A slow tier that handles ten requests per second fills with about 200 ordinary requests per second, but with 122 attack-like ones against Clef, 99 against Jev and 30 against the n-gram model. An adversary who can query a public model can keep only the requests it escalates: of the 51 that Jev escalates, Clef escalates 49%, against 8% of attacklike requests overall. ModernBERT’s seed-17 main run, which the adversary cannot query, escalates 14% to 33% of them depending on the fold, against 15% to 23% overall. Padding is cheaper still. With about 3,000 characters of opaque parameters added, Clef escalates 97% of benign requests and ModernBERT 50% (Table 6). The same slow tier then fills at about 10 padded requests per second against Clef and 23 against Jev, while the n-gram model and both releases of the rule set escalate at most 5% of them. An edge rule that refuses requests of that length would also refuse the 7.0% of the distinct recorded requests (Section 4.1) that are longer than the shortest padded one (3,281 characters).

8

The Cost of Running a Fast Tier (RQ4)

A hosted decision model adds a few hundred milliseconds to each request, so it suits triage of a copy of the traffic better than inline use (Section 3). A cascade with the rule set as cheap first stage calls Jev on a fifth of traffic but escalates fewer of the recent exploits than Jev alone.

Score-guided search

Of the 99 held-out attacks Jev escalated, the search of Section 4.5 got 43% past it within 300 queries (19% within 100). Success depends on the initial score, as for prompt injection against Jev’s action 10

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

8.1

Latency and throughput

but each benign one is a false alarm: Jev raises 30,969 a day and ModernBERT 2,664.

Over the 72,095 live calls of our leave-one-family-out experiments, made one at a time, Jev answered in 300 ms at the median and 473 ms at the 99th percentile, most of it the network round trip of 205 ms from our client, which a client closer to the service would largely avoid. Clef and Clef-flash answered 55,842 calls each in 375 and 372 ms at the median and 1,103 and 936 ms at the 99th percentile, over a round trip of 17 ms, almost all of it the service’s own (Figure 4 in Appendix E). Clef-flash, about five times as fast as Clef at the median in Cloudflare’s benchmark [12], matched it. Locally, on one Apple M5 laptop, ModernBERT answers in 21 ms at the median and the n-gram model in under a millisecond. With 1 to 16 concurrent clients Jev’s throughput rose from 3.4 to 58.3 requests per second without a throttled request; a service with ten million requests a day averages about 116 per second, more at peaks.

8.2

9 Discussion 9.1 Lessons for operators Evaluate a fast tier on families held out from everything it fits, over several training runs. Every detector we trained on the known families escalated far fewer of the same traversal requests once traversal was held out, and for ModernBERT how many fewer changed from one training run to the next. Treat a supervised detector’s confident benign verdicts on new attacks with caution. Its recalibration map is learned from the families it already knows, and on a new family it can turn moderate scores into confident benign verdicts, for a fine-tuned decision model as well as for ModernBERT. Pair exploit tests with benign requests of the same shape. Detectors trained on older exploits escalated many benign twins of recent exploits. Route by a benign budget. A conformal rule’s guarantee covers only attacks exchangeable with those it was fitted on. Where the known classes separate, the rule closes what a 1% budget closes; where they overlap, its benign load ranged from 1.5% (Clef) to most of the traffic (the small open models). Test a cascade’s first stage on held-out families and on recent exploits (Section 8.3). Assume an adversary can query a public fast tier, and judge evasions on repeated calls. A search guided by the model’s own probability got held-out attacks past Jev and Clef that fixed edits did not, but most of its successes against Jev did not hold up when the requests were asked again, and the requests it found rarely evaded another model. Budget per service. Benign escalation varies by service: of the 21 hosts with at least 50 benign test requests, Jev escalated no request of 11 but 88% of the requests of one, whose client polls an endpoint with a stray {} in its query string. Expect content-judging fast tiers to miss more of the attacks that look like ordinary use. For most fast tiers authentication bypasses were the recent exploits they escalated least: such requests often carry nothing suspicious, and whether one is legitimate depends on the application’s state. Decide what a hosted fast tier may see and what happens when it fails. A hosted model receives every request body, sign-ups and shopping carts among them, which raises data-protection questions a self-hosted model avoids; we saw no throttling up to 16 clients but did not test outages.

Cost per request

With the five questions of our set, a request of the folds costs Jev 609 input tokens on average, or $25.59 per million requests at the published price, and Clef and Clef-flash 701 tokens, or $168.35 and $26.66 per million. A service with ten million requests a day would pay about $93,000 a year to put Jev in front of all of its traffic and $614,000 for Clef. On 200 requests asked only the primary question, which is all the budget and conformal rules use, Jev reads 470 tokens instead of 608, about $72,000 a year, and its answer moves by at most 0.10. Our benign requests have at most 1,000 characters; longer ones cost more in proportion to their tokens. Under a budget rule the slow tier examines a share 𝑏 of benign traffic, so it costs more than the fast tier once one examination costs more than 1/𝑏 fast-tier calls: 20 Jev calls ($0.0005) at a 5% budget, 100 ($0.003) at 1% and 1,000 ($0.03) at 0.1%.

8.3

Cascades

A cascade (Section 4.4) cuts the fast tier’s cost: a cheap first stage screens all traffic and forwards only the requests it ranks highest to Jev, whose threshold is set so that the cascade as a whole escalates 5% of benign calibration traffic (4.2% to 5.5% of benign test traffic). Forwarding the top 20% of traffic, the cascade escalates 19% of heldout traversal with the n-gram model as first stage and 79% with ModernBERT (main run, seed 17), against 99% for Jev alone. The rule set, trained on no attack data, makes a first stage that forwards the top 20% by its anomaly score; the cascade then escalates at least 89% of every held-out family (traversal: 97%) while calling Jev on 20% of benign traffic. On the recent exploits the same cascade escalates 73%, against 98% for Jev alone, and 29% of the authentication bypasses, 80% of which the rule set scores zero on every request, like most benign traffic.

8.4

9.2

Limitations

Exposure during pretraining. A held-out family is unseen by everything a deployment fits, but perhaps not by a pretrained model: the payload lists, Log4Shell and Shellshock are public, and none of the providers we test documents its training data in full. Our results for zero-shot models therefore measure behavior on attack families absent from the deployment’s own data, the position of an operator who fits a fast tier on local traffic, and may overstate how they would meet an attack class absent from every pretraining corpus. So far only 9 exploits postdate the models’ release, and

Slow-tier load at a realistic base rate

Rates measured on balanced test sets translate into very different daily volumes once attacks are rare [5]. For a service with ten million requests a day, one in a thousand of them an attack from a held-out family, the traversal fold’s rates under the conformal rule send 60,507 requests a day to the slow tier with Jev as the fast tier and 99,261 with ModernBERT. Jev closes 248 of the 10,000 attacks and ModernBERT 6,509. Alerts do not reach the slow tier, 11

Kisielewicz, Kędziora and Nguyen

they belong to known attack classes. A replay with a model trained before Log4Shell’s disclosure failed the screen we set before running it, an AUROC of at least 0.80 for cross-site scripting and SQL injection on the development set: T0 [66] ranks both below benign requests (AUROC 0.19 and 0.23), which recalibration turns into a constant (Section 4.4). Jev and Clef escalate at most 5 points less of encoded payloads than of unedited ones, and of the exploit classes no payload family covers than of all recent exploits, though only 86% of authentication bypasses.

queries or other edits may do better, and a defender that rate-limits or hides scores may make it harder. Hosted models change. Asked 1,813 of our requests again, Jev changed at most 1.6% of the routing decisions. Our 500 fixed canary requests, sent weekly from 6 October to 3 November, moved by at most 0.010 on average. We pinned jev-1.13.0; by 27 October the unpinned alias returned its successor jev-1.14.0 [77]. It escalates 98.4% of held-out traversal against 99.3%, both at a benign rate of 5.2%, but moved the canary probabilities by 0.067 on average and pushed 19 canary decisions across 0.5, so an operator who follows the alias must recalibrate after each change. Cloudflare gives no version, so Clef may change unannounced.

Families as units. Only three families leave enough held-out attacks for main folds, and they differ: ModernBERT gives 56% of held-out traversal a probability below 0.05 but 3% of cross-site scripting. Our intervals describe these families only, and the rate on a new family may differ.

10

Conclusion

In a two-tier pipeline the fast tier decides alone on everything it does not escalate, so how it treats attack families it was never fitted on bounds how many new attacks the pipeline can catch. Holding families out of everything a deployment fits, we found that detectors trained on the known families gave large parts of some held-out families a confident benign verdict, by amounts that varied between training runs. Jev and Clef, whose budget decisions the hold-out cannot change, escalated at least 97.6% of every held-out family within a 5% benign budget. Benign twins showed that detectors trained on older exploits had learned what scanner requests look like, and a search guided by Jev’s probability got 43% of held-out attacks past it within 300 queries but only 18% past it in most of 5 repeated calls. Our protocol can be reused on new fast-tier models, with the data checks of Appendix D: hold each family out of training, recalibration and thresholds over several training runs, spend a fixed benign budget, pair exploit tests with benign twins, attack with fixed and adaptive edits, judge evasions on repeated calls, and count the held-out attacks the fast tier lets through.

Training runs. ModernBERT’s held-out results vary between training runs with identical data (Section 5), so the three main runs understate their spread; Laya (ft) was trained once, and its 76% confident misses on traversal may vary as much. Splits. Known-family payloads are split by their text, so encoding variants of one payload can sit in both training and test and flatter known-family results, and with them the gap to held-out families; a held-out family is excluded whole and cannot leak this way. Benign requests to the same endpoint with different values also land in different splits: 68% of benign test requests share host, method and path with a training request; splits by payload skeleton and by host would test how much either matters. Spliced attacks and injection points. Splicing gives every attack a real host, path and parameter names, and the format control shows that these cues no longer separate the classes. A spliced request still carries one payload in one parameter of a request recorded for other purposes, while real attacks often arrive in sequences and with their own headers. The header test covers two families, the exploit test leaves out the 126 exploits that carry part of the attack in a header the compact form drops, and our filters, at most 1,000 characters and at least one parameter, leave out long bodies and requests without parameters. Labels come from construction, so some payloads may be inert in their context.

References [1] Bushra A. Alahmadi, Louise Axon, and Ivan Martinovic. 2022. 99% False Positives: A Qualitative Study of SOC Analysts’ Perspectives on Security Alarms. In 31st USENIX Security Symposium (USENIX Security 22). 2783–2800. https://www. usenix.org/conference/usenixsecurity22/presentation/alahmadi [2] Diogo Almeida. 2026. Introducing System One Models & Jev. TypeSafe AI blog, https://typesafe.ai/blog/introducing-system-one-models-and-jev. 15 September 2026; accessed 2026-10-08. [3] Anastasios N. Angelopoulos and Stephen Bates. 2023. Conformal Prediction: A Gentle Introduction. Foundations and Trends® in Machine Learning 16, 4 (March 2023), 494–591. doi:10.1561/2200000101 [4] Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and Don’ts of Machine Learning in Computer Security. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 3971–3988. [5] Stefan Axelsson. 2000. The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security 3, 3 (Aug. 2000), 186–205. doi:10.1145/357830.357849 [6] Seyedarmin Azizi, Erfan Baghaei Potraghloo, and Massoud Pedram. 2026. Labels Override Definitions in Jev-Style Typed Decision Models. doi:10.48550/ARXIV. 2610.02586 [7] Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2023. Conformal prediction beyond exchangeability. The Annals of Statistics 51, 2 (April 2023), 816–845. doi:10.1214/23-aos2276 [8] Federico Barbero, Feargus Pendlebury, Fabio Pierazzi, and Lorenzo Cavallaro. 2022. Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 805–823. doi:10.1109/sp46214.2022.9833659 [9] Leyla Bilge and Tudor Dumitraş. 2012. Before we knew it: an empirical study of zero-day attacks in the real world. In Proceedings of the 2012 ACM conference on

Benign traffic. The main benign traffic comes from one browsing campaign recorded through one browser for the 2026 edition of the open-appsec comparison, a browser’s view of many sites, which may differ from one application’s inbound traffic, and our second source, CSIC 2010, is synthetic. The results depend on the benign traffic: on ECML’s synthetic benign requests, which the hosted models find suspicious, Jev escalates only 10% of held-out LDAP injection (Appendix C). Small supervised detectors. Our supervised detectors are small (the n-gram model, ModernBERT with 149M parameters, Laya (ft) with 421M); we did not test whether a large model fine-tuned on the known families, such as Qwen2.5-7B or Clef-flash, would also close held-out attacks with confidence. The adaptive search. The search uses one set of edits, at most 300 queries and the model’s probability, and judges an evasion by the fast tier’s decision, not by an application; an adversary with more 12

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Computer and communications security (CCS ’12). ACM, 833–844. doi:10.1145/ 2382196.2382284 [10] Matteo Bitussi and Roberto Doriguzzi-Corin. 2026. X-WAD: eXplainable Web Anomaly Detection. doi:10.48550/ARXIV.2608.27172 [11] Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. doi:10. 48550/ARXIV.2305.05176 [12] Michelle Chen, Alex Reneau, and Kevin Flansburg. 2026. Introducing Clef: our open-source decision models, and new RL fine-tuning platform. https: //blog.cloudflare.com/clef-decision-models/. Cloudflare blog, 1 October 2026. [13] Cloudflare. 2026. Clef. https://huggingface.co/Cloudflare/clef. Model card. [14] Cloudflare. 2026. Clef-Flash. https://huggingface.co/Cloudflare/clef-flash. Model card. [15] Cloudflare. 2026. Workers AI models: clef and clef-flash. https://developers. cloudflare.com/workers-ai/models/clef/ and https://developers.cloudflare.com/ workers-ai/models/clef-flash/. Accessed 19 October 2026. [16] ConvAI Innovations. 2026. Laya. Hugging Face model convaiinnovations/laya, revision 7b928d82. Apache-2.0. Accessed 2026-10-08. [17] Alexander D’Amour, Katherine Heller, Dan Moldovan, et al. 2022. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research 23, 226 (2022), 1–61. https://jmlr.org/papers/v23/201335.html [18] Luca Demetrio, Andrea Valenza, Gabriele Costa, and Giovanni Lagorio. 2020. WAF-A-MoLE: evading web application firewalls through adversarial machine learning. In Proceedings of the 35th Annual ACM Symposium on Applied Computing (SAC ’20). ACM, 1745–1752. doi:10.1145/3341105.3373962 [19] Shrey Desai and Greg Durrett. 2020. Calibration of Pre-trained Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 295–302. doi:10. 18653/v1/2020.emnlp-main.21 [20] Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa. 2026. Evaluating and Benchmarking the System One Model Jev. doi:10.48550/ARXIV.2609.37647 [21] Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. 2026. General Decision Models: Benchmarking and Insights Beyond Jev. doi:10.48550/ARXIV.2610.03935 [22] Yonatan Geifman and Ran El-Yaniv. 2017. Selective Classification for Deep Neural Networks. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. [23] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec ’23). ACM, 79–90. doi:10.1145/3605764.3623985 [24] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70). PMLR, 1321–1330. [25] Ali Habibzadeh, Farid Feyzi, and Reza Ebrahimi Atani. 2026. Large Language Models for Security Operations Centers: A Comprehensive Survey. Journal of Electrical and Computer Engineering 2026, 1 (Jan. 2026). doi:10.1155/jece/3383674 [26] Dongqi Han, Zhiliang Wang, Wenqi Chen, et al. 2023. Anomaly Detection in the Open World: Normality Shift Detection, Explanation, and Adaptation. In Proceedings 2023 Network and Distributed System Security Symposium (NDSS 2023). Internet Society. doi:10.14722/ndss.2023.24830 [27] heman10x. 2026. Verdict (RLCD ModernBERT, 151M). Hugging Face model heman10x/rlcd-modernbert-151m, revision 8af24969. Apache-2.0. Accessed 2026-10-08. [28] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained Transformers Improve Out-of-Distribution Robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2744–2751. doi:10.18653/v1/2020.acl-main.244 [29] Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang. 2026. JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models. doi:10.48550/ARXIV. 2609.31142 [30] Roberto Jordaney, Kumar Sharad, Santanu K. Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting Concept Drift in Malware Classification Models. In 26th USENIX Security Symposium (USENIX Security 17). 625–642. https://www.usenix.org/conference/usenixsecurity17/ technical-sessions/presentation/jordaney [31] Saurav Kadavath, Tom Conerly, Amanda Askell, et al. 2022. Language Models (Mostly) Know What They Know. doi:10.48550/ARXIV.2207.05221 [32] Daniel Kahneman. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux. [33] Leon Kersten, Tom Mulders, Emmanuele Zambon, Chris Snijders, and Luca Allodi. 2023. ‘Give Me Structure’: Synthesis and Evaluation of a (Network) Threat Analysis Process Supporting Tier 1 Investigations in a Security Operation

Center. In Nineteenth Symposium on Usable Privacy and Security (SOUPS 2023). USENIX Association, Anaheim, CA, 97–111. [34] Varun Kotte. 2026. UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing. doi:10.48550/ARXIV.2605.18796 [35] Christopher Kruegel and Giovanni Vigna. 2003. Anomaly detection of web-based attacks. In Proceedings of the 10th ACM conference on Computer and communications security (CCS ’03). ACM, 251–261. doi:10.1145/948109.948144 [36] Ananya Kumar, Percy S. Liang, and Tengyu Ma. 2019. Verified Uncertainty Calibration. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc. [37] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc., 7167–7177. [38] Peiyang Li, Ye Wang, Qi Li, Zhuotao Liu, Ke Xu, Ju Ren, Zhiying Liu, and Ruilin Lin. 2023. Learning from Limited Heterogeneous Training Data: Meta-Learning for Unsupervised Zero-Day Web Attack Detection across Web Domains. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23). ACM, 1020–1034. doi:10.1145/3576915.3623123 [39] Xiang Li, Aichun Huang, Mingming Zhang, and Zheli Liu. 2026. Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers. doi:10.48550/ARXIV.2610.17820 [40] Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2026. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. doi:10.48550/ARXIV.2609.26550 [41] Zhenpeng Li. 2026. Robust Conformal Intrusion Detection via Traffic-Aware Calibration and Attack-Orbit Invariance. doi:10.48550/ARXIV.2609.19241 [42] Yixuan Liu. 2026. Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation. doi:10.48550/ARXIV.2609. 33401 [43] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association. [44] David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc. [45] Soundouss Messoudi and Youness Mansar. 2026. Calibration and Risk-Budgeted Decisions with System One Models. doi:10.48550/ARXIV.2610.19811 [46] mgm security partners. 2022. WAF Payload Collection. https://github.com/mgmsp/WAF-Payload-Collection. Accessed 2026-10-07. [47] Hussein Mozannar and David Sontag. 2020. Consistent Estimators for Learning to Defer to an Expert. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 7076–7087. [48] Tianwei Mu, Shengyan Jiang, Mingzhe Yuan, Qing Luo, Min Xiao, Wenhong Wang, Jun Li, and Manhong Huang. 2026. HydroJEV: A one-second, trainingfree screen for cyber-attack and fault attribution in water distribution networks. doi:10.48550/ARXIV.2610.02048 [49] National Institute of Standards and Technology. 2026. National Vulnerability Database CVE API 2.0. https://services.nvd.nist.gov/rest/json/cves/2.0. Accessed 2026-10-09. [50] Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML ’05). ACM Press, 625–632. doi:10.1145/1102351.1102430 [51] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc. [52] OWASP CRS Project. 2026. OWASP CRS: The OWASP Core Rule Set. https: //github.com/coreruleset/coreruleset. Releases 4.30.0 (2 October 2026) and 3.3.2 (30 June 2021). [53] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence 29, 1 (Feb. 2015). doi:10.1609/aaai.v29i1.9602 [54] Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2019. TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 729–746. [55] John C. Platt. 2000. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large-Margin Classifiers, Alexander J. Smola, Peter Bartlett, Bernhard Schölkopf, and Dale Schuurmans (Eds.). The MIT Press, Cambridge, MA. [56] Riccardo Porcedda. 2026. Jev thinks “I don’t know”, but doesn’t say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration. doi:10.48550/ARXIV.2609.35342 [57] ProjectDiscovery. 2026. Nuclei Templates: Community curated list of templates for the nuclei engine to find security vulnerabilities in applications. https: //github.com/projectdiscovery/nuclei-templates, commit c8664a1. MIT License. Snapshot of 2026-10-09. 13

Kisielewicz, Kędziora and Nguyen

[58] ProjectDiscovery. 2026. Nuclei Templates: Community curated list of templates for the nuclei engine to find security vulnerabilities in applications. https: //github.com/projectdiscovery/nuclei-templates, commit 1f4c3f6. MIT License. Snapshot of 2026-10-23. [59] Qwen Team. 2024. Qwen2.5: A Party of Foundation Models! https://qwenlm. github.io/blog/qwen2.5/. Published 19 September 2024. [60] Qwen Team, An Yang, Baosong Yang, Beichen Zhang, et al. 2024. Qwen2.5 Technical Report. doi:10.48550/ARXIV.2412.15115 [61] Amir Rafe and Subasish Das. 2026. Benchmarking System One decision models against trained classifiers and language models for automated decision gates. doi:10.48550/ARXIV.2610.00346 [62] Chedy Raïssi, Johan Brissaud, Gérard Dray, Pascal Poncelet, Mathieu Roche, and Maguelonne Teisseire. 2007. Web Analyzing Traffic Challenge: Description and Results. In ECML PKDD 2007 Discovery Challenge. Warsaw, Poland. https://hallirmm.ccsd.cnrs.fr/lirmm-00168955 [63] Tomás Sureda Riera, Juan-Ramón Bermejo Higuera, Javier Bermejo Higuera, José-Javier Martínez Herraiz, and Juan-Antonio Sicilia Montalvo. 2022. A new multi-label dataset for Web attacks CAPEC classification using machine learning techniques. Computers & Security 120 (2022), 102788. doi:10.1016/j.cose.2022. 102788 [64] Boris Rozenfeld. 2026. Best WAF Solutions in 2026: Real-World Comparison. https://www.openappsec.io/post/best-waf-solutions-in-2026-real-worldcomparison. open-appsec blog, 18 January 2026; datasets and tooling at https://github.com/openappsec/waf-comparison-project. [65] Donald B. Rubin. 1981. The Bayesian Bootstrap. The Annals of Statistics 9, 1 (Jan. 1981), 130–134. doi:10.1214/aos/1176345338 [66] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations (ICLR). arXiv:2110.08207. [67] Mohanad Sarhan, Siamak Layeghy, Marcus Gallagher, and Marius Portmann. 2023. From zero-shot machine learning to zero-day attack detection. International Journal of Information Security 22, 4 (March 2023), 947–959. doi:10.1007/s10207023-00676-0 [68] Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron, and Sebastian J. Vollmer. 2026. TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models. doi:10.48550/ARXIV.2610.11392 [69] Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based Prompt Injection Attack to LLMas-a-Judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security (CCS ’24). ACM, 660–674. doi:10.1145/3658644.3690291 [70] Robin Sommer and Vern Paxson. 2010. Outside the Closed World: On Using Machine Learning for Network Intrusion Detection. In 2010 IEEE Symposium on Security and Privacy. IEEE, 305–316. doi:10.1109/sp.2010.25 [71] Subham. 2026. Cyber-Jev: a small calibrated decision model for application-layer security checks. https://huggingface.co/sdmlai/cyber-jev. Model card, version 0.1 (research preview), 29 September 2026. [72] Hao Sun, Yibin Yao, Chaohai Xie, and Yuqun Lin. 2026. Case-Level Verification in Scanner-LLM Cascades: Overcoming the Alert Aggregation Bottleneck to Expand the FRR-TPR Trade-off Space. doi:10.48550/ARXIV.2610.08406 [73] Hao Sun, Yibin Yao, Chaohai Xie, and Yuqun Lin. 2026. An Evaluation of the Semantic Understanding Capabilities of Large Language Models for Web Attack Payloads. doi:10.48550/ARXIV.2610.06507 [74] Ruming Tang, Zheng Yang, Zeyan Li, et al. 2020. ZeroWall: Detecting Zero-Day Web Attacks through Encoder-Decoder Recurrent Neural Networks. In IEEE INFOCOM 2020 - IEEE Conference on Computer Communications. IEEE, 2479–2488. doi:10.1109/infocom41043.2020.9155278 [75] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 5433–5442. doi:10.18653/v1/2023.emnlp-main.330 [76] Carmen Torrano Giménez, Alejandro Pérez Villegas, and Gonzalo Álvarez Marañón. 2010. HTTP Data Set CSIC 2010. https://www.tic.itefi.csic. es/dataset/. Spanish National Research Council (CSIC). [77] TypeSafe AI. 2026. Models. https://docs.typesafe.ai/models. Documentation, accessed 22 October 2026. [78] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2022. Algorithmic Learning in a Random World (2nd ed.). Springer International Publishing. doi:10. 1007/978-3-031-06649-8 [79] Ke Wang and Salvatore J. Stolfo. 2004. Anomalous Payload-Based Network Intrusion Detection. In Recent Advances in Intrusion Detection (RAID 2004) (Lecture Notes in Computer Science, Vol. 3224). Springer Berlin Heidelberg, 203–222. doi:10. 1007/978-3-540-30143-1_11 [80] Shang Wang, Tianqing Zhu, Huajie Chen, Jiayang Li, Meng Yang, and Bo Liu. 2026. Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use. doi:10.48550/ARXIV.2610.04985

[81] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al. 2025. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2526–2547. doi:10.18653/v1/2025.acllong.127 [82] Bowen Wei, Yuan Shen Tay, Howard Liu, Jinhao Pan, Kun Luo, Ziwei Zhu, and Chris Jordan. 2025. CORTEX: Collaborative LLM Agents for High-Stakes Alert Triage. doi:10.48550/ARXIV.2510.00311 [83] wfzyx. 2026. von. Hugging Face model wfzyx/von, revision 498ceba3. Apache2.0. Accessed 2026-10-08. [84] Tiantong Wu and Wei Yang Bryan Lim. 2026. Decision Hijacking: Prompt Injection Attacks on Jev’s Typed Probabilistic Decisions. doi:10.48550/ARXIV. 2609.28613 [85] Hanxiang Xu, Shenao Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and Haoyu Wang. 2025. Large Language Models for Cyber Security: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology (2025). doi:10.1145/3769676 [86] Limin Yang, Wenbo Guo, Qingying Hao, Arridhana Ciptadi, Ali Ahmadzadeh, Xinyu Xing, and Gang Wang. 2021. CADE: Detecting and Explaining Concept Drift Samples for Security Applications. In 30th USENIX Security Symposium (USENIX Security 21). USENIX Association. [87] Guangze Zhao, Yongzheng Zhang, Changbo Tian, Dan Xie, Hongri Liu, and Bailing Wang. 2025. Information-Dense Reasoning for Efficient and Auditable Security Alert Triage. doi:10.48550/ARXIV.2512.08169

A

Ethical Considerations

This study involved no live systems. Every attack in our experiments comes from public sources: the payloads of the mgm WAF Payload Collection (a GPL-3.0 repository whose README notes that most payloads carry their sources’ MIT license), as packaged by the open-appsec WAF comparison project, and the exploit requests of the nuclei templates (MIT). Payloads were inserted as text into recorded requests, and no request reached a live application: the models scored text, and the rule set ran in a local container with no application behind it. The out-of-band callback domain of the templates was replaced by a reserved .example name, and in one control by a subdomain of the request’s own host, as text only. The benign traffic is a public recording (Apache-2.0), used without cookies or other session headers (Section 4.1), and our artifact contains no text of the recorded requests. SR-BH 2020 (CC0) serves only for the label audit, which reports counts and row indices. Requests, attack payloads included, were sent as text to two hosted services under their providers’ terms, among them the edited attacks of the adaptive search (about 22,300 queries to Jev and 9,500 to Clef). TypeSafe’s terms assign outputs to the customer and do not restrict benchmarking; Cloudflare’s terms permit benchmark tests whose disclosure includes the information needed to replicate them, which our artifact provides. We report model versions and dates so that the providers can reproduce our observations. Three results could inform an attacker: the held-out families that a fast tier closes with confidence, the rate of legitimate requests that fills a slow tier, and the share of attacks a score-guided search gets past a hosted model. They rest on public payloads and on edits of known kinds; we publish the search code but not the requests it found. The study involved no human participants.

B

Open Science

The artifact, available at https://anonymous.4open.science/r/ escalate-or-close-artifact-3794/, contains the code for every experiment; the analysis plan written before the first leave-one-familyout run and the log of every later deviation; fold manifests as hashes of each item’s text; per-item predictions of every model 14

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

on every fold; the cached answers of the zero-shot models keyed by text hash, without request text; and the scripts that rebuild every table, figure and number in this paper from these files. Datasets are not redistributed: download scripts fetch the original releases and verify them against SHA-256 checksums, so the artifact ships no payload data; the nuclei templates are fetched at two pinned commits, c8664a1 of 9 October and 1f4c3f6 of 23 October 2026 [57, 58], and verified by a digest over every template. CVE publication dates come from the NVD API [49]; this product uses data from the NVD API but is not endorsed or certified by the NVD. Open models are pinned to Hugging Face revisions with file hashes and Jev to jev-1.13.0 (the drift check to jev-1.14.0); Workers AI serves Clef and Clef-flash by name without a version, and we called them on 19, 22, 23 and 28 October 2026, Clef again on 30 October to repeat the adaptive search’s requests, and both on 31 October to score every benign calibration and test request. The rule set runs from pinned container images, with release 3.3.2 verified by its SHA-256. Re-running the hosted models requires API keys, and Jev’s answers to identical calls vary by up to 0.21 in probability (Section 7.2); every reported number can be regenerated without a key from the cached answers and the stored latency measurements.

C

which limits its value as a benchmark for held-out attack families. The local language model behaves in the same way: it scores 99.5% of the benign ECML requests above 0.5 and escalates 6% to 7% of each held-out family, and the small open models escalate 5% to 10%.

D

The benchmark’s attacks share one template. Because the harness sends every payload to /?p= or in a body p=, a classifier given only the method, Content-Type, Host, path and parameter names separates the harness’s attacks from benign traffic with an AUROC of 1.000. This concerns learned detectors evaluated on these requests; it need not affect the product comparison the benchmark was built for. On spliced attacks, which share host, path and parameter names with benign requests, the same classifier reaches an AUROC of 0.496. The harness’s encoding is shared across families. The payloads keep the harness’s percent-encoding, which escapes characters that browsing traffic rarely escapes: an escaped parenthesis (%28) appears in 0.2% of benign test requests but in 67% to 100% of the cross-site scripting, command-execution, SQL-injection and Shellshock payloads, and in 0% of the traversal payloads, whose escapes include overlong byte sequences such as %c0%af. No spliced value contains a character a browser always escapes, and no benign training value contains ../, even decoded. A character n-gram model reads the shared escapes as attack evidence: refitted on requests whose query and form values are percent-decoded, it escalates 15% of held-out cross-site scripting instead of 88% and 14% of command execution instead of 86%, while ModernBERT’s results barely change (Section 5).

Results on ECML/PKDD 2007

Table 7: ECML/PKDD 2007: held-out AUROC / share of heldout attacks escalated at a 5% benign budget (%), per held-out family. Model

LDAP OS cmd.

SQLi

XPath

XSS

Jev Clef Clef-flash

0.78/10 0.78/15 0.68/14

0.97/84 0.88/53 0.96/80 0.92/62 0.85/44 0.95/78 0.82/41 0.70/19 0.84/44

0.91/48 0.95/74 0.74/19

0.96/81 0.94/69 0.91/60

Qwen2.5-7B Laya von Verdict

0.76/7 0.53/6 0.58/7 0.39/6

0.87/7 0.48/5 0.48/10 0.61/5

0.81/7 0.53/6 0.64/9 0.25/5

0.91/7 0.53/5 0.53/5 0.33/5

ModernBERT 1.00/100 n-gram model 1.00/99

Trav.

0.76/6 0.86/7 0.50/5 0.53/5 0.51/7 0.63/10 0.44/5 0.34/5

Five Pitfalls in Web Attack Data

Each of these pitfalls could have distorted the measurement.

SR-BH 2020 labels a vulnerability scan as normal. Of the 542,991 distinct requests of the SR-BH 2020 honeypot corpus [63], 503,747 come from one client whose traffic includes an OWASP ZAP active scan; of its requests labeled normal, 46.1% carry an attack payload, against 2.8% of the other clients’. The mislabeling that Bitussi and Doriguzzi-Corin estimate at about 13.8% [10] concentrates there, and we do not evaluate on SR-BH 2020.

1.00/99 0.95/83 0.99/98 1.00/100 1.00/100 0.97/90 0.92/79 0.99/96 1.00/100 0.97/95

We drop the 2,180 attacks labeled with two types, keep serverside includes (16 requests, too few to hold out) as a known family, and hold out the other six families (1,825 to 2,279 requests) in turn; each leaves 1,745 to 2,172 test requests. On ECML, ModernBERT and the n-gram model lead (Table 7). ModernBERT escalates at least 82.9% and the n-gram model at least 78.6% of every held-out family, a result that likely reflects the corpus more than the models: most of its attack families share a cue that its benign requests lack (Appendix D). The hosted decision models read the same benign requests as suspicious. Zero-shot, Jev scores 91.8% of the benign ECML test requests above 0.5 (median 0.80), against 0.6% of the real benign traffic of open-appsec (median 0.07), and Clef 86.9%. A monotone recalibration does not reorder requests, so it cannot separate attacks from benign requests that a model scores as high, and Jev, Clef and Clef-flash escalate only 10%, 15% and 14% of heldout LDAP injection. ECML thus favors models that learn its benign vocabulary over models that judge each request by its content,

ECML/PKDD 2007’s attack families share a cue. In four of its six held-out families, 70% to 99% of the attacks contain a run of three or more encoded spaces (+++), against 0.1% of benign requests, so a model trained on the known families can flag a held-out one by cues they share: the n-gram model detects each held-out family with an AUROC of 0.92 to 1.00. Its benign requests in turn carry attack-like noise, an encoded quote in 12% of them, which a model that judges content finds suspicious (Appendix C). We report ECML for comparison with prior work and draw our conclusions from open-appsec. Scanner templates do not map one to one onto exploit requests. A nuclei template is often a probe followed by the exploit, builds its payload with helper functions such as base64, or only fetches a page to read a version string. An earlier conversion of ours let a visible probe stand in for an exploit step the compact form hides, 15

Kisielewicz, Kędziora and Nguyen

Table 10: Each model’s own probability before recalibration: expected calibration error and confident misses (Section 4.6) on known and held-out (held) families, and the share of benign requests given 𝑝 > 0.5. Open-appsec, mean over the main folds and, for ModernBERT, its main runs.

replaced helper calls by random text and counted plain page requests as attacks; on it, Jev escalated 88% of the exploits within a 5% benign budget and its conformal rule closed 22%, against 98% and 5% on the faithful conversion.

E

Additional Results ECE

Table 8: open-appsec attack families: distinct payloads (crosssite scripting and path traversal capped by hash), held-out attacks in the test split of the family’s own fold, and the fold’s role. Benign requests: 221,634 for training, 10,000 each for calibration and test. Family Cross-site scripting Path traversal Command execution SQL injection Log4Shell XML external entities Shellshock

Payloads

Held-out attacks

3,000 3,000 1,234 458 110 35 24

2,839 2,858 1,162 427 104 35 23

Model

Fold main main main case study case study case study case study

Plain ../ %u escapes Overlong UTF-8 URL-encoded Other encodings

..///boot.ini %3F.%uF025boot.ini ..%c0%af..%c0%afboot.ini 0x2e0x2e%5Cetc%5Cissue ..%c0%qfboot.ini

76 100 69 99 34 99 39 100 49 99

0.08 0.07 0.19

0.08 0.07 0.19

0.0 0.4 4.6

0.0 0.4 4.0

0.6 1.7 2.7

Qwen2.5-7B

0.08

0.07

4.6

4.9

11.7

ModernBERT Laya (ft) n-gram model

0.01 0.02 0.13

0.18 0.18 0.42

1.7 3.3 14.4

30.0 32.9 55.3

0.2 0.4 0.0

Log4Shell

Shellshock

Benign par. UA XAV par. UA XAV

Jev Clef Clef-flash

4.4 100 100 4.3 99 100 4.4 75 90

100 100 100 100 100 100 86 83 74

100 100 61

Qwen2.5-7B

4.5

86

65

73

96

35

52

CRS 4.30.0 CRS 3.3.2 Anomaly baseline

5.1 100 4.0 100 4.6 60

67 3 4

66 100 2 96 4 70

83 96 0

65 96 0

values only, almost none in a header. For two requests of the battery and the header test, both over 4,000 characters, none of Qwen2.57B’s 20 most likely first tokens was yes or no; they are left out of its rates.

𝑛 MB Jev Qwen 790 415 552 246 855

Jev Clef Clef-flash

Model

Table 9: Held-out path-traversal attacks (traversal fold) by payload form: an example as transmitted, the number of held-out attacks, the share ModernBERT gives a recalibrated probability below 0.05 (MB, %, mean over its main runs), and the shares Jev and Qwen2.5-7B escalate within a 5% benign budget (%). The example is the shortest payload of its form that ModernBERT gives a probability below 0.05 in every main run. Example

Benign

held 𝑝 > 0.5 (%)

Table 11: Held-out Log4Shell and Shellshock attacks (%) escalated by placement in requests with headers: in a parameter (par.), as User-Agent (UA) or in an added X-Api-Version header (XAV), at a 5% budget set on 2,000 benign requests in the same form; Benign: realized rate on as many benign test requests (%).

Attack families. Table 8 lists the open-appsec families and their sizes. Spliced attacks are not held to the benign length cap: 26% of the held-out XML external entities and 0.0% to 2.9% of each other family’s held-out attacks exceed 1,000 characters, and leaving them out moves no model’s held-out escalation by more than 5 points.

Form

Conf. miss (%)

known held known

90 78 78 86 78

Table 12: ModernBERT’s held-out traversal and Log4Shell attacks escalated at a 5% benign budget / given a probability below 0.05 (%), per training run: the main runs (seeds 17, 23 and 42), the kept run (seed 17, kept), a further retraining of seed 17 (repeat), and further seeds. Every run uses the same data and settings.

Traversal by payload form. Table 9 breaks ModernBERT’s confident misses on held-out traversal down by the form of the payload. Plain dot-dot sequences are missed as well: ModernBERT gives 76% of them a probability below 0.05, while Jev escalates at least 98.7% of every form.

Run

Probabilities as they come and placements in headers. Tables 10 and 11 give the results of Sections 5 and 6.5 in full. Within a 5% budget set on requests with headers, release 4.30.0 of the rule set escalates only 66% to 67% of the Log4Shell payloads in a header: most score exactly 5, the threshold, and the tie-breaker splits them. Jev and Clef escalate the header placements at realized benign rates of 4.4% and 4.3%; Qwen2.5-7B escalates less in a header than in a parameter, and the anomaly baseline, which models parameter

seed 17 seed 23 seed 42 seed 17, kept seed 17, repeat seed 1 seed 2 seed 3 seed 4 seed 5 16

Traversal

Log4Shell

53/59 61/53 53/55 81/37 55/52 79/37 65/53 50/60 67/45 56/56

74/42 89/32 82/39 96/18 – 96/12 88/20 60/51 – –

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

ModernBERT across training runs. Table 12 lists every run. In the main runs the traversal fold’s 5% budget threshold lies below 0.05 (0.023 to 0.031), so 9% to 14% of held-out traversal attacks are escalated although they are confident misses. On the traversal fold, known-family escalation stays between 98.4% and 99.4%. An ensemble that averages all runs’ probabilities escalates 75% of heldout traversal within the budget, 77% with the runs’ disagreement as a second signal, and 98% of Log4Shell, yet still gives 38% of traversal a probability below 0.05.

distribution, because its conditional field is empty whenever the model abstains. The payload of 2.0% of held-out attacks ends beyond Verdict’s input window, and Laya and von cut none. They escalate little more of a held-out family than the budget’s share of benign traffic, and where their classes overlap their conformal rules escalate or alert on most benign traffic. The conformal guarantee on known families. Every model but Laya and Clef-flash closes at most 5.3% of known-family attacks on average and in no fold significantly more than 5% (exact binomial 𝑝 ≥ 0.11). Laya closes 6.5% (5.5% to 6.8% per fold, 𝑝 from 0.002 to 0.25) and Clef-flash 5.9% (5.4% to 6.5%, 𝑝 from 0.02 to 0.27). The guarantee holds on average over draws of the calibration split, and the folds share most of their calibration attacks, so they are not independent checks of it.

Table 13: Share of held-out attacks (%) escalated at a 0.1% benign budget whose threshold is set on all 73,491 benign calibration requests; Benign: realized share of the 74,068 benign test requests escalated (%). ModernBERT: its kept run. Model

Table 15: The CRS at paranoia levels 1 to 4: share of held-out attacks (%) escalated at a 5% benign budget, and the realized benign escalation (%).

Benign XSS Trav. Cmd. SQLi L4Sh. XXE ShSh.

Jev Clef Clef-flash

0.08 0.10 0.10

86 78 10

82 68 10

88 84 8

92 83 20

92 82 11

97 89 63

91 91 30

ModernBERT n-gram model

0.09 0.11

68 50

6 0

94 76

79 54

31 4

97 94

100 83

CRS 4.30.0 CRS 3.3.2 Anomaly baseline

0.09 0.07 0.09

96 94 39

49 37 32

46 86 33

47 35 30

100 11 30

60 86 63

65 74 43

Release, level XSS Trav. Cmd. SQLi L4Sh. XXE ShSh. Benign

Small budgets. Table 13 gives the 0.1% budget of Section 6.2 per family.

𝑛 n-gram, 7 fam. n-gram, +old MB, 7 fam. MB, +old

File inclusion Code execution XSS SQL injection File upload Auth. bypass SSRF Other

89 85 67 56 44 35 35 12

58 15 72 29 23 6 3 0

97 95 99 93 98 71 94 100

93 61 97 91 48 17 17 58

100 92 81 32

86 41 27 17

95 77 68 41

81 73 61 37

100 100 89 53

97 100 100 89

87 65 57 35

5.0 4.5 4.4 4.8

3.3.2, 1 3.3.2, 2 3.3.2, 3 3.3.2, 4

100 95 78 29

85 39 29 16

96 85 76 42

88 81 63 37

100 12 9 12

100 91 97 77

96 70 57 26

5.3 4.6 4.7 4.8

The rule set at every paranoia level. Each level adds rules that match benign requests as well as attacks, so within a budget the higher levels mostly escalate less (Table 15). As shipped, blocking at an anomaly score of 5, release 4.30.0 blocks 7.1% of benign test requests at level 1 and 32.5% at level 2, 87% to 100% of each held-out family at level 1, and 70% of the recent exploits; release 3.3.2 blocks 2.4% of benign requests and 81% to 100% of each family at level 1. In requests with headers, release 4.30.0 at level 1 blocks 8.1% of benign requests.

Table 14: Recent exploits escalated (%) by attack class at the temporal test’s 5% budget, by the n-gram model and ModernBERT (MB) trained on the seven families alone (7 fam.) or with the 2015–2024 exploits as well (+old). Attack class

4.30.0, 1 4.30.0, 2 4.30.0, 3 4.30.0, 4

100 100 100 98 98 94 100 100

Other questions for the small open models. With any of the five questions of our set, Laya, von and Verdict reach held-out AUROC at most 0.77 on the main folds, and Jev at least 0.99. Laya’s card presents it as a base to specialize and warns that its yes-or-no answers can follow their option labels [16]; its answer to our choice question ranks held-out attacks only slightly better than its yes-orno answer (AUROC 0.56 to 0.76, against 0.56 to 0.71, on the main folds). Verdict says true more readily for benign requests whatever the proposition, so its answer to whether a request is legitimate points the right way, but weakly (AUROC 0.57 to 0.64).

The temporal test by attack class. With the older exploits in training both detectors escalate nearly every class (Table 14), authentication bypasses included for ModernBERT. For server-side request forgeries the n-gram model learned the generator’s callback domain, which the older exploits share: under the neutral rendering, with the callback under the request’s own host, it escalates 63% of them instead of 94%.

Recent exploits by attack class. Table 16 gives the exploit test of Section 6.6 by attack class.

The small open decision models. Table 24 gives the results of Laya, von and Verdict, used zero-shot with our questions at Hugging Face revisions 7b928d8, 498ceba and 8af2496. For von we run one forward pass per question with its chain-of-options controller disabled, which on the development set never changed an answer; for Verdict we take P(true)/(P(true)+P(false)) from the full output

Latency against escalation. Figure 4 sets each fast tier’s latency (Section 8) against its held-out escalation. Recent exploits. Jev escalates 98% of the exploits and its conformal rule closes 5% (95% Wilson intervals 96% to 99% and 3% to 7% over the 423 exploits, each exploit’s outcome averaged over the 17

Kisielewicz, Kędziora and Nguyen

Table 16: Exploits for vulnerabilities from 2025 and 2026 by attack class: share (%) escalated within a 5% benign budget / share closed by the conformal rule, averaged over the seven folds (Section 4.1). MB: ModernBERT’s kept run; CRS: 4.30.0, level 1. Attack class

𝑛

Jev

File inclusion Code execution XSS SQL injection File upload Auth. bypass SSRF Other

89 100/0 100/0 97/1 85/25 85 96/5 94/7 82/6 65/43 67 100/0 100/0 96/0 93/15 56 100/2 98/4 95/0 90/14 44 100/11 100/3 80/7 53/61 35 86/23 86/47 31/33 15/91 35 97/6 97/6 54/14 22/85 12 100/8 100/8 67/17 60/43

67/31 28/72 74/21 34/66 27/70 14/81 12/87 5/94

All

423

41/57 66/32

98/5

Clef Qwen

97/7

MB n-gram

82/7 68/40

9 exploits unescalated; it escalated all 4 exploits published after its release, all among the 9 of Section 4.5. ModernBERT, released before every dated exploit, changes by −8.3 points per year (𝑝 = 0.01) and Qwen2.5-7B by −6.9 (𝑝 = 0.04), so a date trend reflects how exploits change over time as well as what a model may have seen. Disagreement and unions of two models. For every zero-shot decision model but Laya, the spread among the four yes-or-no answers separates held-out attacks from benign requests with an AUROC between 0.23 and 0.44, below chance; for Laya it points the right way, but weakly (AUROC 0.57 to 0.58). Across the three model pairs (Jev with the n-gram model or ModernBERT’s seed-17 main run, and Laya with the n-gram model) and seven held-out families, the smallest exact McNemar 𝑝 for a gain of a union over its better model was 0.50.

CRS 92/9 56/40 94/6 91/8 48/49 11/84 20/75 25/76

Table 17: Confident misses (%) on known / held-out families, mean over the main open-appsec folds, with each model’s own probability (none), Platt scaling (used throughout), temperature scaling and isotonic regression, each fitted on the calibration split at equal class mass. ModernBERT: its main runs. ∗ : the map leaves a calibration error above 0.1 on known families.

held-out attacks escalated

1.0 0.8 0.6 0.4 0.2 0.0 10−1

100

101

102

103

latency per request (ms) Jev Clef Clef-flash ModernBERT

ModernBERT on CPU n-gram model Laya von

Verdict Qwen2.5-7B hosted, less round trip

Model

None

Jev Clef Clef-flash

0.0/0.0 0.4/0.4 4.6/4.0∗

Platt Temperature Isotonic

Qwen2.5-7B

4.6/4.9∗

1.3/1.6

0.9/1.0

0.9/1.2

ModernBERT, seed 17 1.9/33.9 seed 23 1.6/27.5 seed 42 1.7/28.7 Laya (ft) 3.3/32.9 n-gram model 14.4/55.3∗

0.9/21.6 1.0/19.8 1.2/20.9 3.3/32.3 0.9/21.6

1.5/27.8 1.3/21.9 1.3/23.3 3.3/32.9 0.0/3.6∗

1.1/26.1 1.0/19.9 1.2/25.9 1.7/25.2 1.3/25.5

0.4/0.8 0.8/1.1 0.9/0.8

0.6/0.9 0.9/1.2 0.4/0.4∗

0.5/1.3 0.8/1.2 1.1/1.0

Other recalibration methods. Table 17 gives the confident misses under each map.

Figure 4: Latency of one request against the share of held-out attacks escalated within a 5% benign budget (open-appsec). Markers give the mean over the seven held-out families and vertical lines their range; horizontal lines run from the median latency to the 99th percentile. Hosted models: one client, network round trip included (filled) and removed (hollow). Local models: one request at a time on an Apple M5 laptop (10-core GPU, 24 GB), the n-gram model and the hollow ModernBERT marker on its CPU; ModernBERT: mean over its main runs. Decision models answer the five questions of our set, Qwen2.5-7B one.

Table 18: CSIC 2010 normal traffic as the benign source (3,549 calibration and 3,615 test requests): share of held-out attacks (%) escalated at a 5% benign budget per family, and the realized benign rate (%). ModernBERT: one run (seed 17) trained on the CSIC folds.

seven folds); counting only the requests that carry attack content, which leaves out scanner steps such as fetching a page, it escalates 97%. For each exploit its conformal rule closes, Jev gives the most suspicious request a probability between 0.12 and 0.39. The small open models escalate between 7% and 16% of the exploits, and Qwen2.5-7B escalates 57% of those in classes that no payload family covers. Within an attack class, Jev’s escalated share changes by −2.5 points per year of publication date (permutation 𝑝 = 0.08, uncorrected), a test with little power because Jev leaves only about

Model

XSS Trav. Cmd. SQLi L4Sh. XXE ShSh. Benign

Jev Clef Clef-flash

100 100 96

100 100 94

100 100 96

100 100 93

100 100 93

100 100 100

100 100 100

4.1 5.0 5.3

Qwen2.5-7B

36

29

48

47

52

54

52

4.8

ModernBERT, seed 17 n-gram model

100 100

83 48

99 97

100 95

100 100

100 100

100 100

4.9 5.5

CRS 4.30.0 CRS 3.3.2 Anomaly baseline

100 100 99

78 84 81

94 97 91

76 88 85

100 100 98

91 100 100

83 96 100

5.1 4.8 5.0

A second benign source. Table 18 gives the CSIC 2010 test of Section 6.1. Clef-flash escalates at least 93% of every held-out family. 18

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

Table 20: AUROC, calibration error and confident misses after recalibration (Section 4.6) on known and held-out (held) families, open-appsec, mean over the main folds and, for ModernBERT, its three main runs. † : as in Table 2.

ModernBERT’s run, trained on the CSIC folds, gives 25% of heldout traversal a probability below 0.05, and the n-gram model 59%. Qwen2.5-7B, which scores 32.4% of CSIC’s benign requests above 0.5, escalates only 29% to 54% of each family.

AUROC

Table 19: ModernBERT’s kept run with a Mahalanobis novelty score: held-out attacks (%) escalated at a 5% benign budget by the attack probability alone, the novelty score alone, and either of the two, their thresholds sharing the budget; Benign: realized rate of the last rule (%); AUROC: of the novelty score, held-out attacks against benign test requests. Held out Cross-site scripting Path traversal Command execution SQL injection Log4Shell XML external entities Shellshock

Model Jev Clef Clef-flash

Probability Novelty Either Benign AUROC 95 81 98 95 96 100 100

87 83 43 93 85 89 22

95 87 98 95 92 100 100

4.6 5.0 5.3 4.8 4.7 5.1 4.9

0.958 0.948 0.902 0.974 0.946 0.977 0.894

known

Calib. error

held known

0.998 0.997 0.996 0.995 0.931 0.933

Conf. miss (%)

held known 0.02†

0.00 0.01 0.04† 0.02 0.03

0 1 1

held 1 1 1

Qwen2.5-7B

0.971 0.971

0.04

0.05

1

2

ModernBERT Laya (ft) n-gram model

0.999 0.959 0.995 0.906 0.987 0.829

0.01 0.22† 0.03 0.21† 0.02 0.21†

1 3 1

21† 32† 22†

The primary comparisons in full. Table 20 gives the levels behind Table 2, Table 22 the primary comparisons of every model with P3, and Table 21 P1 and P2 for each main family alone. Traversal accounts for most of each supervised detector’s growth in both, while the hosted models’ confident-miss rates change by at most 1.2 points on any main family. Resampling payload skeletons instead of requests (Section 4.6) widens the P2 interval of ModernBERT’s seed-17 main run to +18.9 to +23.2 points, still excluding zero for every supervised detector.

A novelty-detection baseline. The novelty score is the smallest Mahalanobis distance of the input to the run’s classification layer from the means of its training classes, benign traffic and each known family, with one shared covariance (Table 19).

19

Kisielewicz, Kędziora and Nguyen

Table 21: P1 and P2 for each main held-out family alone: held-out minus known calibration error and confident-miss rate (percentage points) after recalibration, with 95% intervals from the stratified Bayesian bootstrap, neither adjusted nor tested. Symbols as in Table 22. Cross-site scripting Model

Path traversal

P1

Jev Clef‡ Clef-flash‡

P2

P1

+0.022 [0.005, 0.025] − 0.5 [ − 1.0, − 0.1] +0.024 [0.011, 0.040] − 0.9 [ − 1.7, − 0.4] +0.027 [0.002, 0.044] − 0.4 [ − 1.4, 0.3]

+0.028 [0.018, 0.035] +0.046 [0.030, 0.055] +0.002 [ − 0.021, 0.022]

+0.005 [ − 0.011, 0.019] +0.069 [0.048, 0.094] +0.012 [ − 0.008, 0.036] –

Qwen2.5-7B Laya von Verdict ModernBERT, seed 17 seed 23§ seed 42§ Laya (ft)‡ n-gram model

+0.132 [0.115, 0.153] +0.119 [0.103, 0.141] +0.124 [0.107, 0.145] +0.120 [0.100, 0.138] +0.067 [0.044, 0.085]

− 0.4 [ − 1.4, 0.3] +0.004 [ − 0.012, 0.018] 0.0 [0.0, 0.0] − 0.019 [ − 0.036, − 0.001] 0.0 [0.0, 0.0] +0.036 [0.015, 0.061] – – +1.8 [0.8, 2.7] +1.8 [0.8, 2.7] +2.9 [1.7, 4.0] +10.2 [8.3, 12.0] +2.8 [1.8, 3.8]

+0.488 [0.475, 0.505] +0.452 [0.438, 0.470] +0.439 [0.425, 0.456] +0.378 [0.361, 0.393] +0.400 [0.382, 0.416]

Command execution P2

P1

P2

+0.5 [ − 0.1, 0.9] +0.008 [ − 0.005, 0.018] +1.2 [0.5, 2.1] +0.9 [0.1, 1.5] +0.007 [ − 0.006, 0.025] +1.0 [0.1, 1.9] +0.4 [ − 0.1, 0.9] +0.007 [ − 0.014, 0.027] − 0.2 [ − 1.0, 0.5] +1.5 [0.5, 2.5] 0.0 [0.0, 0.0] 0.0 [0.0, 0.0] – +58.1 [56.2, 60.0] +51.8 [49.8, 53.7] +54.2 [52.2, 56.1] +72.4 [70.4, 74.3] +51.5 [49.5, 53.4]

+0.015 [0.000, 0.030] − 0.1 [ − 1.0, 0.7] +0.029 [0.008, 0.050] 0.0 [0.0, 0.0] +0.034 [0.014, 0.057] 0.0 [0.0, 0.0] – – +0.031 [0.017, 0.048] +0.038 [0.024, 0.053] +0.024 [0.010, 0.042] +0.042 [0.028, 0.058] +0.102 [0.081, 0.123]

+2.4 [1.4, 3.5] +2.9 [1.8, 4.2] +2.0 [1.0, 3.1] +4.4 [2.9, 6.1] +7.8 [6.2, 9.5]

Table 22: Primary comparisons on the main open-appsec folds (Section 4.6) for every model and ModernBERT’s three main runs: P1 and P2 as in Table 2, with unadjusted 95% intervals, and P3, the held-out attacks gained / lost by escalating on confidence or disagreement against confidence alone at an equal 5% budget, with the Holm-adjusted 𝑝 of the exact McNemar test. † , ‡ : as in Table 2; § : outside the analysis plan, not tested. –: P3 needs the four yes-or-no answers of a decision model (Qwen2.5-7B answers one question, the supervised detectors none); Verdict is recalibrated to a constant. Model

P1: calib. error

P2: conf. miss (points)

P3: gained / lost

P3: Holm 𝑝

Jev Clef‡ Clef-flash‡

+0.019 [0.011, 0.023]† +0.026 [0.018, 0.033]† +0.012 [−0.003, 0.020]

+0.4 [0.1, 0.8] +0.3 [−0.1, 0.7] −0.1 [−0.5, 0.3]

1 / 12 8 / 79 29 / 1,018

0.027 <0.001 <0.001

Qwen2.5-7B Laya von Verdict

+0.008 [−0.001, 0.016] +0.026 [0.015, 0.039]† +0.027 [0.016, 0.041]† –

+0.3 [−0.2, 0.8] 0.0 [0.0, 0.0] 0.0 [0.0, 0.0] –

– 126 / 193 90 / 390 27 / 153

– 0.002 <0.001 <0.001

ModernBERT, seed 17 seed 23§ seed 42§ Laya (ft)‡ n-gram model

+0.217 [0.209, 0.228]† +0.203 [0.195, 0.214] +0.196 [0.188, 0.207] +0.180 [0.170, 0.190]† +0.190 [0.177, 0.200]†

+20.8 [20.0, 21.6]† +18.8 [18.0, 19.6] +19.7 [18.8, 20.5] +29.0 [28.0, 30.0]† +20.7 [19.8, 21.6]†

– – – – –

– – – – –

Table 23: The four exploits described in the text, closed by Jev’s conformal rule in every fold: the exploit’s request that Jev scores highest (Host header omitted), Jev’s probability for it, and the outcome under the 5% budget rule, the same in every fold. CVE

Request Jev scores highest (Host omitted)

Jev 𝑝

Budget

CVE-2026-42018 CVE-2025-58434 CVE-2026-59726

POST /access/api/v1/aws/token/ {} POST /api/v1/account/forgot-password {"user":{"email":"admin"}} POST /mcp {"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name": "ruflo__terminal_execute","arguments":{"command":"id"}}} GET /media/amasty_checkout/k/x/kxmbikzs.txt

0.22 0.38 0.17

closed esc. closed

0.29

esc.

CVE-2026-53787

Table 24: The small open decision models, zero-shot: held-out AUROC over the seven folds (lowest to highest), held-out attacks escalated at a 5% benign budget per family (%), the realized benign rate (%), the recent exploits escalated (%), and the conformal rule’s benign load (escalated or alerted, main folds, %). Model

AUROC

XSS

Trav.

Cmd.

SQLi

L4Sh.

XXE

ShSh.

Benign

Exploits

Load

Laya von Verdict

0.56–0.79 0.59–0.71 0.21–0.44

10 10 5

7 7 4

13 19 5

11 7 6

16 7 3

23 0 14

13 4 9

4.7 4.8 4.5

9 16 7

83.7 83.5 95.5

20

Related documents

• Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents#386750
• Prismata: Confining Cross-Site Prompt Injection in Web Agents#352992
• Towards a Unified Misuse Monitoring Benchmark#1216464
• The LLM-Attacker: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems#1386694
• The LLM-Attacker: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems#1374971