arXiv CSopen access

CyberCane: Neuro-Symbolic RAG for Privacy-Preserving Phishing Detection with Formal Ontology Reasoning

arXiv:2604.23563v1 [cs.CR] 26 Apr 2026

CyberCane: Neuro-Symbolic RAG for Privacy-Preserving Phishing Detection with Formal Ontology Reasoning Safayat Bin Hakimα∗ Aniqa Afzalα Qi Zhaoα Vigna Majmundarα Pawel Slobodaβ Houbing Herbert Songα α

University of Maryland, Baltimore County

β

Bowie State University

Abstract Privacy-critical domains require phishing detection systems that satisfy contradictory constraints: near-zero false positives to prevent workflow disruption, transparent explanations for non-expert staff, strict regulatory compliance prohibiting sensitive data exposure to external APIs, and robustness against AI-generated attacks. Existing rule-based systems are brittle to novel campaigns, while LLMbased detectors violate privacy regulations through unredacted data transmission. We introduce CyberCane, a neuro-symbolic framework integrating deterministic symbolic analysis with privacy-preserving retrieval-augmented generation (RAG). Our dual-phase pipeline applies lightweight symbolic rules to email metadata, then escalates borderline cases to semantic classification via RAG with automated sensitive data redaction and retrieval from a phishing-only corpus. We further introduce PhishOnt, an OWL ontology enabling verifiable attack classification through formal reasoning chains. Evaluation on DataPhish 2025 (12.3k emails; mixed human/LLM) and Nazario/SpamAssassin demonstrates a 78.6-point recall gain over symbolic-only detection on AI-generated threats, with precision exceeding 98% and FPR as low as 0.16%. Healthcare deployment projects a 542× ROI; tunable operating points support diverse risk tolerances, with open-source implementation at https://github.com/sbhakim/Cybercane. Keywords: Phishing detection, neuro-symbolic AI, privacy-preserving RAG, formal ontology reasoning

1

Introduction

The human element contributes to over 60% of data breaches [29], with phishing representing a primary initial access vector across industries, yet privacy-critical domains face unique detection challenges. Healthcare exemplifies these constraints: systems must balance security with uninterrupted access to time-critical patient communications—misclassified appointment reminders or prescription notifications directly compromise care. Older adults demonstrate significantly reduced ability to discriminate phishing from legitimate emails, with detection performance declining with age [22, 8]. Recent generative AI advances amplify this threat across all sectors, enabling adversaries to craft convincing domain-specific terminology and adapt to detection systems [6]. Privacy-critical detection systems face four contradictory requirements, which we illustrate through healthcare as a motivating example. First, false positives must approach zero—in healthcare, each false alarm can potentially delay critical medical communications. Second, explanations must be ∗

Corresponding author: [email protected].

transparent and actionable for IT staff who often lack specialized security training [7]. Third, systems must preserve privacy—regulations like HIPAA (Health Insurance Portability and Accountability Act) prohibit protected information transmission to unauthorized third parties, constraining cloud AI services [24]. Fourth, detection must accommodate vulnerable populations with reduced digital literacy. Existing approaches struggle to satisfy these constraints simultaneously. Rule-based systems are precise but brittle to novel attacks and lack semantic understanding for context-aware detection. Deep learning models generalize better but are black boxes unsuitable for verified explanations and audit trails. Recent LLM-based detectors achieve high accuracy—ChatSpamDetector reaches 99.7% [15] and KnowPhish achieves 92.5% accuracy with 97.8% precision [18]—but cannot be directly deployed in privacy-critical domains due to privacy concerns around transmitting unredacted email content to external API providers. This work addresses three research questions: (i) Does the dual-phase architecture achieve healthcare-grade precision (>95%) with near-zero FPR, satisfying the core requirement of privacycritical deployment? (ii) Does Phase 2 RAG provide statistically significant improvement over symbolic-only detection while supporting tunable operating points for diverse workflows? (iii) Does formal ontology reasoning produce verifiable explanations, and do operational economics justify deployment in privacy-critical domains? We introduce CyberCane, a neuro-symbolic framework integrating symbolic rules with privacypreserving RAG (Fig. 1). Phase 1 applies deterministic analysis to identify technical violations; Phase 2 performs semantic classification via RAG with automated sensitive data redaction and retrieval from a phishing-only corpus. Retrieved examples ground LLM explanations while preventing data exposure. CyberCane addresses these requirements through architectural design: tunable thresholds achieve near-zero FPR (Req. 1), PhishOnt ontology provides verifiable explanations (Req. 2), sensitive data redaction maintains HIPAA compliance (Req. 3), and operating modes accommodate diverse workflows (Req. 4). Contributions. We make the following contributions: (i) Privacy-preserving neuro-symbolic architecture: Dual-phase pipeline combining symbolic rules with RAG through HNSW retrieval and sensitive data redaction, achieving tunable operating points. (ii) PhishOnt ontology: Web Ontology Language (OWL)-based reasoning framework enabling verifiable attack classification and formal explanation chains. (iii) Comprehensive evaluation: Statistical validation through bootstrap confidence intervals and DataPhish 2025 benchmark (n=2,337) spanning template-based to GPT-generated attacks. (iv) Deployment viability: Cost-benefit analysis for healthcare deployment demonstrating 542.0× ROI, with framework applicable to other privacy-critical domains.

2

Related Work

Phishing detection evolved from blacklist-based filtering [2] to ML systems combining heuristics [1, 25] and authentication protocols [14, 5, 16]. Recent LLM-based detectors [15, 18] present deployment challenges in privacy-critical domains: limited verification, email content transmission raises regulatory concerns (e.g., HIPAA), and labeled data may not exist for emerging threats. Neuro-symbolic AI combines neural perception with symbolic reasoning [9, 12] addressing fundamental limitations. Privacy-critical domains face unique challenges from life-critical systems and vulnerable populations [7, 8] with strict regulatory constraints [24]. To address these challenges while maintaining interpretability, recent work has explored retrieval-augmented generation (RAG) [26, 10, 11], though existing applications inadequately address privacy in regulated domains. We extend RAG with architectural privacy-by-design through sensitive data redaction before embedding. Prior phishing defenses span blacklist/heuristic pipelines and authentication checks, which are interpretable but brittle to domain compromise and new campaigns [2, 25, 14, 16]. LLM-based detectors raise deployment challenges where sensitive data exposure and auditability are critical [15, 24]. Retrieval-augmented security systems improve traceability by grounding outputs in cited evidence [26, 10]. However, prior work rarely couples retrieval with strict privacy bound2

Email Input

Phase 1: Deterministic symbolic analysis DNS/MX Verification

SPF/DMARC signals

URL & content cues (urgency/PHI)

Domain mismatch

Symbolic Indicators Privacy boundary: redaction before external model calls

Phase 2: Neural RAG classification external API

If Phase 1 = phishing (score≥5), final=phishing regardless of Phase 2

external API

Embedding (text-embedding-3-small)

Vector search (pgvector HNSW)

Retrived examples (k≈8)

Phishing corpus (~2,300)

LLM reasoning (GPT-4.1-mini)

Confiidence Score + Explanation

Human review and Feedback

Symbolic Phase Phishing (>50%) (sim ≥0.88 or P1=review & avg≥0.82)

Suspicious(25-50%) Human Review

→

(sim ≥0.75 or avg≥0.72)

Safe (<25%)

Neural Phase External API Human Review Privacy Boundary

Figure 1: CyberCane dual-phase architecture. Phase 1 applies symbolic rules to email metadata, computing deterministic scores. Emails passing Phase 1 undergo sensitive data redaction before Phase 2 neural analysis using RAG with a phishing-only corpus, producing similarity-driven AI scores and multi-layered explanations.

aries or conservative operating points needed in regulated environments [7, 24]. Human-factor evidence that vulnerable populations are more susceptible to phishing supports the need for transparent, low-FPR decision support [8]. CyberCane combines these elements with privacy-first redaction, phishing-only retrieval, and conservative thresholds tailored to diverse operational workflows.

3

System Architecture

CyberCane implements a dual-phase detection pipeline where emails progress sequentially through deterministic symbolic analysis (Phase 1) followed by neural RAG-based classification (Phase 2). Fig. 1 illustrates the complete architecture. The system is implemented as a FastAPI backend (Python 3.11) with PostgreSQL 17 using pgvector for HNSW vector indexing, integrated with OpenAI APIs (text-embedding-3-small, gpt-4.1-mini [21]), and a Next.js 15.0.3 frontend for human review. Design rationale. Sequential processing prioritizes low-latency, zero-cost symbolic filtering before invoking external APIs. Phase 1 catches obvious phishing (missing DNS records, domain mismatches) with instant technical explanations IT staff can verify independently; Phase 2 adds semantic analysis when needed. Operational review rates and Phase 1 baseline metrics are reported in Appx. B. 3.1

Phase 1: Deterministic Symbolic Analysis

Phase 1 applies lightweight symbolic rules across five categories: DNS/authentication checks (MX, SPF [14], DMARC [16]), domain mismatches, URL obfuscation patterns, urgency language, and credential requests. Rules employ tuned weights (wi ∈ {1, 2, 3}) producing risk scores that map to verdicts: s < 2 (benign), 2 ≤ s < 5 (review), s ≥ 5 (phishing). Detected indicators are passed to ontology reasoning (Algorithm 1) for formal attack classification. Appx. A (Algorithm 2) details complete rule specifications with regex patterns and validation logic. 3.2

Ontology-Guided Symbolic Inference

We introduce PhishOnt, an OWL ontology encoding phishing attack knowledge as description-logic axioms over Phase 1 indicators (missing MX/SPF/DMARC, domain mismatch, URL obfuscation, urgency, credential requests, freemail senders). Algorithm 1 formalizes the neuro-symbolic reason3

Algorithm 1: Ontology-Guided Attack Classification Input: Phase 1 indicators I, ontology Ω, threshold θ = 0.3 Output: Attack types A and reasoning chains E Function OntologyAttackClassification(I, Ω) /* Normalize indicators and map to ontology properties P ←∅ for each indicator i with I[i] = true do P ← P ∪ {MapProperty(i)} A←∅ for each attack type τ ∈ Ω do (R∀ , R∃ ) ← Ω.GetAxioms(τ ) m∀ ← CountMatch(R∀ , P ) m∃ ← HasAny(R∃ , P ) den ← |R∀ | + 1(R∃ ̸= ∅) /* Axiom satisfaction from universal and existential coverage c ← (m∀ + m∃ )/den if c ≥ θ ∧ (R∃ = ∅ ∨ m∃ = 1) then A ← A ∪ {(τ, c)}

*/

*/

E ← GenerateChain(P, A, Ω) return (A, E)

ing mechanism: Phase 1 indicators map to ontology properties (missing_mx → hasMissingMX), enabling formal attack classification via axiom satisfaction. The reasoner computes multi-label attack types with confidence scores (c = m/r where m is satisfied constraints, r is required constraints) and generates verifiable reasoning chains linking detected properties to inferred attack classes. This formal provenance distinguishes our approach from black-box LLM classifiers, providing transparent explanations required for healthcare audit trails. Coverage and attack-type prevalence are reported in Section 5.2. 3.3

Phase 2: Neural RAG Pipeline

Emails escalated from Phase 1 undergo privacy-preserving RAG analysis. The pipeline: (1) PHI redaction masks sensitive fields (emails, SSN, credit cards) via regex before external API calls; (2) embedding generation encodes redacted content using text-embedding-3-small (1536-d); (3) vector retrieval queries phishing-only corpus (2,297 examples) via HNSW [19], returning top-k = 8 neighbors with cosine similarities; (4) LLM reasoning generates tagged explanations ([AUTH], [URL], [SIMILARITY], [ONTOLOGY]) using gpt-4.1-mini conditioned on redacted email, Phase 1 indicators, ontology-inferred attack types (Algorithm 1), and retrieved examples. Critically, the AI score derives from similarity statistics—not LLM generation—ensuring reproducible, grounded verdicts. The complete neuro-symbolic integration workflow is formalized in Appx. A (Algorithm 3), with implementation details including prompt engineering and threshold calibration. Decision logic. If Phase 1 verdict is phishing, final verdict is phishing. Otherwise: top similarity ≥ 0.40 (or Phase 1 needs_review + avg top-3 ≥ 0.35) yields phishing; top similarity ≥ 0.25 (or avg top-3 ≥ 0.17) yields needs_review; lower similarity inherits Phase 1 verdict. This prevents neural overrides of absent technical violations while capturing semantic similarity. Ontology-inferred attack types (Algorithm 1) are passed to the LLM as structured context, enabling [ONTOLOGY]tagged explanations that cite formal reasoning chains rather than heuristic pattern matching. Privacy architecture. Three data categories transmit to external APIs: (1) redacted text embeddings (1536-dim vectors); (2) LLM prompts with redacted content, symbolic indicators, ontologyinferred attack types, and retrieved example snippets (pre-screened); (3) ontology reasoning chains derived from local OWL inference without external transmission. Redaction is empirically active: across 2,337 DataPhish 2025 test emails, 50.7% contained at least one redacted PII field (mean 1.63 items/email); LLM-generated emails averaged 1.73 items/email versus 1.30 for human-written, yet detection F1 remained 0.987 for both groups, confirming that semantic classification is preserved under active PII masking (detailed counts in Appx. F.6). Critically, ontology classification operates entirely offline on Phase 1 indicators, avoiding external API dependency for formal explanations. Residual risks include quasi-identifier inference from surviving context, embedding leakage under inversion attacks [17], and organizational compliance requirements (business associate agreements, workforce training) beyond technical controls. Organizations with zero-trust requirements can deploy on-premises models (sentence-transformers, Llama 3) at cost of reduced accuracy and increased infrastructure complexity. 4

Retrieval corpus bias mitigation. The phishing-only corpus creates structural positive bias. Three mechanisms mitigate: (1) Phase 1 filters obvious phishing before RAG, ensuring borderline cases enter neural analysis; (2) conservative similarity thresholds (≥ 0.40 for phishing classification) require strong semantic overlap; (3) LLM receives symbolic indicators (may be empty for legitimate emails), ontology attack types, and retrieved examples, with explicit prompting to assess attack characteristics beyond superficial similarity. The measured FPR=0.16% suggests limited false escalation despite corpus composition. We empirically evaluate three hypotheses: (H1) dual-phase detection achieves healthcare-grade precision (>95%) with minimal FPR; (H2) RAG improves over deterministic-only baselines; (H3) operational economics justify deployment.

4

Experimental Setup

Dataset. We evaluate on three corpora: (1) Nazario phishing corpus [20] and SpamAssassin [3] (2006-era, 7,274 emails); (2) DataPhish 2025 [27] (12,300 emails, ∼75% LLM-generated, ∼25% human-written). Table 3 (left) summarizes splits. Only phishing training rows are indexed for retrieval. Random seed 42 ensures reproducibility. Synthetic healthcare validation data (Appx. B) is generated via a multi-model pipeline with provenance tracking, deduplication, and contamination checks; a manifest and dataset card are produced for auditability. Metrics. We report precision (fraction of phishing predictions correct—important for minimizing false alarms), recall (fraction of phishing detected), F1-score (harmonic mean), and false positive rate (FPR—fraction of legitimate emails misclassified). Privacy-critical contexts prioritize FPR minimization to prevent workflow disruption. Baselines. Majority class baseline (predicts most frequent label) and TF-IDF logistic regression (unigram/bigram features, trained on train split) provide text-only comparisons. Detailed baseline results in Appx. F. We evaluate a GPT-4 Direct baseline using OpenAI’s gpt-4o-mini model (costeffective variant representing realistic organizational deployment) to quantify privacy and cost implications of unfiltered commercial LLM usage. This baseline transmits full unredacted email text with a zero-shot prompt (“Is this email phishing?”), tracking API costs and sensitive information exposure rates via regex pattern detection (email addresses, phone numbers, SSNs, credit cards, dates of birth). Protocol. Thresholds tuned on validation split. Bootstrap confidence intervals (1,000 resamples for CI estimation; 10,000 samples for McNemar significance testing) estimate statistical stability. All experiments use OpenAI text-embedding-3-small (1536-dim) and gpt-4.1-mini (temperature=0.2) [21].

5

Results

We present results organized to address our three hypotheses (H1–H3): overall performance and baseline comparisons (H1, H2), ontology coverage, explanation quality, operating point flexibility, and economic viability (H3). 5.1

Overall Performance

Table 3 (right) compares Phase 1 (deterministic, threshold=2) against RAG (k = 8) across datasets. On Nazario/SpamAssassin (n=1,110), RAG raises precision to 99.5% while increasing recall to 37.2% and reducing FPR to 0.16%. On DataPhish 2025 (n=2,337; human+LLM mix), Phase 1 recall is 20.5% while RAG reaches 99.1% recall at 98.2% precision. Detailed cohort and modelsource breakdowns appear in Appx. Table 18, and retrieval quality in Appx. C. Comparison with privacy-constrained baselines. Table 4 compares CyberCane against three baselines under explicit privacy accounting. TF-IDF logistic regression trained on PII-redacted text (same pii.py pipeline as CyberCane, PHI exposure = 0%) achieves 98.8% precision and 97.4% recall on the Nazario/SpamAssassin test split—but at 6× higher FPR (0.98% vs. 0.16%), with no explainability, no formal attack taxonomy, and no tunable operating points. This is the fair privacyconstrained bar; CyberCane’s value is not raw recall but precision at near-zero FPR plus verifiable PhishOnt reasoning chains required for healthcare audit trails. GPT-4 Direct achieves 99.0% recall 5

Table 1: DataPhish 2025 robustness: Phase 2 RAG detection consistency across 18 creator sources and 6 emotion categories (n=2,300; 37 of the 2,337 test emails lack creator attribution and are excluded from persource rows but included in overall Table 3). F1 range 0.976–1.000 across all creator groups. Group

N

Prec.

Recall

F1

Creator Source

Human-written LLM-generated (17 models)

559 1,741

97.5% 98.4%

99.5% 99.1%

98.5% 98.7%

LLM Model

DeepSeek-Chat OpenAI (GPT family) GPT-4o DeepSeek Chat (v2) Amazon Nova / Grok / Codestral

602 596 62 62 156

99.1% 97.1% 95.4% 100.0% 100.0%

99.1% 99.3% 100.0% 97.5% 100.0%

99.1% 98.2% 97.6% 98.7% 100.0%

Emotion

Urgency Curiosity Authority Neutral Fear Greed

1,177 976 908 888 655 432

98.7% 98.2% 98.8% 95.0% 99.4% 99.2%

99.2% 99.3% 100.0% 99.0% 99.2% 98.2%

98.9% 98.7% 99.4% 97.0% 99.3% 98.7%

Overall

Phase 2 RAG

2,300

98.1%

99.2%

98.7%

Note. All groups are evaluated with identical thresholds (top_sim≥0.40, avg_top3≥0.35). F1 ranges from 0.976 to 1.000 across 18 creator sources, demonstrating consistent detection regardless of authoring model or rephrasing style. Phase 1 recall for the same set is 7.7–39.9% (emotion-dependent); Phase 2 closes all blind spots to ≥97% F1.

but exposes 53.2% of emails as sensitive data, violating HIPAA regardless of detection accuracy, and flags ∼600 legitimate emails daily at 5.9% FPR.

Robustness across creator sources. Table 1 reports detection consistency across 18 email creator sources in the DataPhish 2025 test set (n=2,300; 37 emails without creator attribution excluded from per-source rows). Phase 2 RAG achieves F1 between 0.976 and 1.000 across all groups— human-written (F1=0.985), GPT-family (F1=0.982), DeepSeek-Chat (F1=0.991), and frontier models including Grok 4 and Amazon Nova (F1=1.000). Performance is statistically indistinguishable between human-written and LLM-generated emails (F1=0.985 vs. 0.987), demonstrating that CyberCane’s semantic embedding space is invariant to authoring model. Emotion-stratified analysis shows Phase 2 closes all Phase 1 blind spots: Neutral-tone attacks (near-zero Phase 1 recall) reach F1=0.970; Altruism and Greed reach F1≥0.987. Critically, all DataPhish evaluations apply the same pii.py PHI-masking step before retrieval and prompting—yet LLM-generated emails score no lower than human-written (F1=0.987 vs. 0.985), confirming that regex redaction does not introduce a detection gap under AI-augmented attack conditions. Phase 2 as layered defense against rule evasion. A structurally important property emerges from stratifying DataPhish 2025 results by Phase 1 outcome. Of the 1,589 phishing emails in the test set, 79.5% (n=1,264) scored zero in Phase 1—attacks with valid DNS infrastructure and no detectable urgency cues that entirely bypass symbolic rules. Phase 2 RAG correctly classified 99.0% of these Phase 1-evaded emails, leaving a combined two-phase miss rate of only 0.8%. Recovery was consistent across emotion categories: Neutral-tone attacks (95.8% Phase 1 evasion rate) reached 99.0% Phase 2 recovery; Greed-tone (97.2% evasion) 98.2%; Authority-tone (64.1% evasion) 100.0%. This demonstrates the intended architectural property: the symbolic layer provides instant verifiable verdicts for flagrant technical violations, while the neural layer provides semantic recovery for attacks that deliberately avoid rule triggers. 5.2

Ontology Coverage and Attack Taxonomy

Table 2 summarizes PhishOnt activation on the test split. The ontology fires on 85.0% of emails and 76.8% of phishing, providing multi-label symbolic explanations; high benign activation (91.7%) reflects shared structural indicators (e.g., URLs, authentication patterns) present in both legitimate and malicious emails. Critically, PhishOnt complements Phase 1/2 detection logic by generating verifiable reasoning chains (e.g., CredentialTheft = hasCredentialRequest AND hasMissingMX) for explanations rather than standalone verdicts. On activated emails, PhishOnt assigns 2.66 attack labels on average with mean confidence 53.6%, reflecting overlapping threat patterns. Confidence discriminability. While activation rates are similar across classes, confidence scores differ systematically: phishing activations carry a mean confidence of 54.6% versus 52.8% for benign activations (+1.84pp). More tellingly, only 14.5% of benign activations exceed the phishing mean confidence—meaning 85.5% of benign activations score below the typical phishing confidence level. This confirms that PhishOnt is discriminative at the confidence level despite similar 6

Table 2: PhishOnt ontology coverage and attack-type prevalence on the test split (n=1,110). (B) Attack types (prevalence)

(A) Coverage

Split

Count

Cov.

Overall 946/1110 85.2% Phishing 383/495 77.4% Benign 563/615 91.5%

Attack type

Prev.

Attack type

Prescription Fraud High-Confidence Phishing Technical Attack Credential Theft

62.3% 58.8% 36.8% 34.1%

Insurance Verification Phish 13.2% Appointment Scam 11.4% Social Engineering Attack 8.2% URL-Based Attack 1.5%

Prev.

Table 3: Dataset composition (left) and performance across corpora (right). DataPhish 2025 shows 78.6pp recall gain of Phase 2 RAG over Phase 1 symbolic-only within CyberCane’s pipeline on AI-generated threats. (B) Test performance

(A) Dataset splits

Source

Train Val Test (Benign/Phish) (Benign/Phish) (Benign/Phish)

Nazario SpamAssassin DataPhish 2025

0 / 1095 2863 / 1202 4932 / 3063

0 / 234 613 / 257 269 / 131

Dataset

Method

Prec.

Recall

F1

FPR

Phase 1 83.0% 17.8% 0.293 2.9% Nazario/SpamAssassin RAG 99.5% 37.2% 0.541 0.16%

0 / 236 615 / 259 711 / 1589

DataPhish 2025

Phase 1 93.4% 20.5% 0.336 RAG 98.2% 99.1% 0.987

— —

binary activation rates. The system is architecturally designed so that this confidence signal feeds structured context into Phase 2 RAG rather than producing standalone verdicts, which explains why the binary activation rate is not the operative discriminability measure. 5.3

Explanation Groundedness

To evaluate whether LLM-generated explanations cite verifiable evidence, we conducted A/B testing on 60 random test emails, comparing explanation quality with and without ontology context. Each explanation bullet receives a tag ([AUTH], [URL], [URGENCY], [CONTENT], [SIMILARITY], [ONTOLOGY]) and should cite specific evidence from Phase 1 analysis, retrieval context, or ontology inference. We classify support status as SUPPORTED (cites specific evidence), UNSUPPORTED (generic without citation), or UNKNOWN (no evidence available to cite). Table 5 demonstrates ontology integration enables verifiable explanations—all 41 ONTOLOGYtagged bullets (100%) cite correct attack types and reasoning chains from formal inference. In comparison, heuristic tags show variable support rates (AUTH: 65.8%, URGENCY: 22.2%, CONTENT: 25.0%), reflecting challenges verifying free-text LLM outputs against rule-based indicators. SIMILARITY achieves 100% support by referencing concrete similarity scores. This demonstrates that formal symbolic reasoning produces more verifiable explanations than pure heuristic analysis, addressing healthcare audit requirements for decision traceability. The key contribution is adding a formal layer that ensures verifiability for attack classification and risk scoring. 5.4

Operating Point Flexibility

CyberCane exposes tunable thresholds across Phase 1 scoring and RAG similarity, allowing organizations to select precision-recall tradeoffs based on risk tolerance and review capacity. Table 6 reports five operating modes. The baseline configuration matches the production pipeline (99.5% precision, 0.16% FPR), while more aggressive settings increase recall to 44.6% while maintaining precision ≥99.3% (FPR remains 0.16% via shared Phase 1 high-confidence threshold).

Fig. 2 consolidates evidence for H1–H3. Panel A confirms H1: RAG achieves +16.5pp precision gain, crossing 95% healthcare threshold. Panel B reveals coverage-precision tradeoff: DNS checks (DMARC, MX, SPF) provide 30–46% coverage while content heuristics (urgency, IP literals) deliver 90–94% precision. Panel C validates design rationale (Section 3): 71.2% of false negatives result from intentional conservative threshold choices (zero-score attacks 43.2%, low-signal content 28.0%) that prioritize low FPR over recall, while only 28.8% represent technical limitations requiring architectural improvements. Panel D confirms H3: 542.0× ROI with $1.5K daily costs versus $818K risk mitigation. Comparing RAG against higher-threshold Phase 1 (threshold=5) quantifies impact: Phase 1 at threshold=5 achieves precision=75.0%, recall=0.6%. RAG improves both metrics: precision increases to 99.5% (+24.5pp) and recall to 37.2% (+36.6pp), yielding F1-score improvement from 0.012 to 0.541. Statistical significance testing validates gains: McNemar’s test yields χ2 = 181.0 (p < 0.001), confirming RAG predictions differ significantly from deterministic baseline. Boot7

Table 4: Comparison with GPT-4 Direct Baseline showing privacy and cost trade-offs. Method

Prec.

Recall

F1

FPR

Cost/Email

PHI Exposure

CyberCane (RAG k=8) TF-IDF LR + Redaction§ TF-IDF LR (no redaction) GPT-4 Direct (gpt-4o-mini)

99.5% 98.8% 98.6% 93.2%

37.2% 97.4% 97.4% 99.0%

54.1% 98.1% 98.0% 96.0%

0.16% 0.98% 1.14% 5.9%

$0.0017 $0 $0 $0.0001‡

0%† 0%† ∼53% 53.2%

Notes. † PHI redacted before API transmission. ‡ API cost only; excludes human review ($1,310/day for CyberCane). § Privacyconstrained fair comparison uses the same redaction pipeline as CyberCane with PHI exposure = 0%. CyberCane provides 6× lower FPR (0.16% vs. 0.98%), PhishOnt verifiable reasoning chains, and tunable operating points, none of which are available in TF-IDF.

Table 5: Explanation groundedness: support rates by tag type (n=60 emails).

(A) With ontology context Tag

Total Supp.

AUTH URL URGENCY CONTENT SIMILARITY ONTOLOGY

117 23 18 16 50 41

(B) Without ontology context

Rate

77 65.8% 2 8.7% 4 22.2% 4 25.0% 50 100% 41 100%

Tag

Total Supp.

AUTH URL URGENCY CONTENT SIMILARITY ONTOLOGY

135 19 14 14 61 0

Rate

78 57.8% 2 10.5% 4 28.6% 4 28.6% 61 100% 0 N/A

strap analysis on F1-score shows mean improvement +0.527 (p < 0.001); confidence intervals are reported in Table 19. 5.5

Operational Cost-Benefit Analysis

Table 7 quantifies economic viability for a healthcare deployment example with a mid-sized organization processing 10,000 daily emails at 4.4% phishing base rate. With 37.2% recall, the system detects ∼164 phishing attacks daily, preventing an estimated 16.4 regulatory breaches (assuming 10% attack success rate [23]). At $50,000 average regulatory penalty per incident, this mitigates ∼$818,000 daily risk, yielding a 542× optimistic ROI (detected attacks only; Table 7). We additionally report a conservative bound that deducts the estimated cost of missed attacks (276 daily false negatives × 10% success rate × $50K = $1.38M/day). At 37.2% recall this bound is negative, honestly reflecting that conservative threshold deployment does not fully offset phishing risk at this operating point. However, the bound turns strongly positive under higher-recall configurations: at DataPhish 2025 recall (99.1%), optimistic and conservative ROI converge to 1,450× and 1,437× respectively, since virtually no attacks are missed. This framing reinforces the importance of operating point selection: organizations with sufficient review capacity should deploy Balanced or Aggressive mode, where recall substantially increases without sacrificing FPR.

Total operational costs ($1,506/day) include API calls ($1.71), human review labor ($1,310 for 13.1% escalation at $100/hr based on fully-burdened IT security analyst wages [28]), and falsepositive impact ($194 for 16 delayed communications at $12.77 each). With FPR 0.16% (16 false alarms per 10,000 emails), net benefit is $816,272/day or $297.9M annually, yielding 542.0× ROI. Operating-mode economics. Extending analysis across operating points (Appx. F, Table 16), balanced mode increases detection to 176 attacks daily (40.0% recall, $1,506/day cost) while maintaining 583× ROI. Aggressive mode detects 196 attacks (44.6% recall) with 651× ROI, showing recall gains remain economically viable under low-FPR operating conditions. Healthcare generalization. Given the absence of publicly available healthcare phishing datasets, we validate domain applicability using synthetic healthcare attacks (Appx. B details multi-model generation with provenance tracking and contamination checks). Balanced mode achieves 100% recall on template-based phishing with explicit red flags; aggressive mode reaches 87.3% recall at 95.4% precision on GPT-generated attacks, demonstrating that coverage depends on threshold calibration. While synthetic evaluation has limitations, the difficulty spectrum provides initial evidence of healthcare transferability pending real-world validation. 5.6

Feature Importance and Design Validation

Leave-one-out ablation shows DNS authentication most frequently triggered: no_dmarc covers 46.5% of phishing (12.5pp recall drop when removed), while content heuristics achieve 90–94% precision with lower coverage. Failure taxonomy shows 71.2% of missed phishing cases fall into 8

Table 6: Operating point configurations showing precision-recall tradeoffs across deployment modes. Operating Mode

Precision

Recall

F1

FPR

Use Case

Baseline (Pipeline) Conservative Balanced Moderate Aggressive

99.5% 99.3% 99.5% 99.5% 99.5%

37.2% 29.5% 40.0% 42.2% 44.6%

0.541 0.455 0.571 0.593 0.616

0.16% 0.16% 0.16% 0.16% 0.16%

Current deployment High-stakes clinical General healthcare High-volume screening Maximum coverage

High-Precision Phishing Detection via Privacy-Preserving Neuro-Symbolic RAG (A) Neuro-Symbolic Achieves Deployment-Grade Precision

(B) Feature Importance: Coverage vs. Precision 100

1.0

Privacy-Critical Req. (Prec. >95%)

90

Rule Precision (%)

0.8 0.7 0.6 Phase 1 Phase 1 (t=2) Phase 1 (t=5) RAG (ours)

0.5

Each circle = rule X = coverage, Y = precision

70 60

DMARC

50

MX Bubble size = recall impact (pp) 3pp 8pp 13pp

SPF

30 0.0

0.2

0.4

0.6

0.8

1.0

0

10

Recall (Phishing Detection Rate)

100.0%

106

99.0%

Daily Cost/Benefit ($, log scale)

97.2%98.2%

90 79.5%

80 70 64.1%

60

50

$818K

ROI: 542.0× Net Daily: $816K

105 104 $1.5K

$1.3K 103 $0.2K 102 101 100

50 Greed

40

$2

Phase 1 Evasion Rate Phase 2 Recovery Rate

Neutral

30

(D) Economic Viability: $297.9M Annual Value

Combined two-phase miss rate: 0.8% 99.0% 95.8%

20

Phishing Coverage (%)

(C) Phase 2 Recovers 99.0% of Phase 1-Evaded Attacks 110

Rate (%)

High impact Medium impact Low impact

80

40

0.4

100

Color = importance tier

Urgency

+16.5pp

0.9

Precision

IP Literal

Authority

Overall

API

Human

FP

Total

Risk

Emotion Category (DataPhish 2025)

Figure 2: CyberCane results overview. (A) Phase 1 versus RAG operating points on precision-recall space. (B) Rule coverage versus precision with bubble size encoding recall impact. (C) False-negative taxonomy separating intentional thresholds from technical limits. (D) Daily cost-benefit and ROI for privacy-critical deployment.

conservative-threshold categories: 43.2% triggered zero rules (evasive attacks with valid DNS), 28.0% scored below threshold. Complete ablations and taxonomy in Appx. E Tables 12, 25.

6

Discussion

Key findings. Results support H1 (99.5% precision, 0.16% FPR, Table 3B), H2 (RAG improves precision by 16pp over symbolic-only, McNemar’s p < 0.001, Fig. 2A), and H3 (542.0× ROI, Table 7). These findings demonstrate that neuro-symbolic architectures can achieve deployment-grade performance in privacy-critical domains by prioritizing precision and explainability over maximizing recall—a tradeoff appropriate for environments where false alarms directly disrupt critical workflows. Precision gains arise from semantic classification correcting conservative symbolic flagging, while DataPhish 2025 evaluation quantifies a 78.6-point recall gain of Phase 2 RAG over Phase 1 symbolic-only within the CyberCane pipeline on AI-generated attacks. On Nazario/SpamAssassin, 37.2% recall under the conservative threshold reflects a deliberate FPR-first design choice, not a technical ceiling—Aggressive mode recovers 44.6% recall while maintaining 99.5% precision (Table 6). 9

Table 7: Cost-benefit analysis: healthcare deployment scenario. Cost parameters grounded in industry benchmarks [13, 23, 28]. Metric

Value

Conservative Bound

Description

Total Daily Cost Attacks Detected Attacks Missed Breaches Prevented Missed Breach Cost Risk Mitigated Net Daily Benefit ROI (optimistic) ROI (conservative)

$1,506 164 276 16.4 — $817,778 $816,272 542.0× —

$1,506 164 276 16.4 $1,381,600 $817,778 — — −376ׇ

API + labor + FP impact 37.2% of 440 daily phishing 62.8% of 440 (false negatives) 10% attack success rate Missed attacks × 10% × $50K $50K per breach penalty Risk - operational cost Detected attacks only† Accounts for missed attacks

At DataPhish 2025 recall (99.1%): optimistic 1,450×, conservative 1,437× †

Notes. Optimistic bound follows industry-standard reporting and is equivalent to assuming no undetected breaches. ‡ The conservative bound is negative at 37.2% recall because missed breach costs exceed mitigated risk. ROI becomes positive under Balanced/Aggressive modes or DataPhish 2025 operating conditions (99.1% recall).

Privacy-accuracy tradeoffs. Sensitive data redaction enables cloud API access without transmitting protected information, while symbolic analysis operates on metadata. Direct GPT-4 comparison (Table 4) demonstrates privacy necessity: unredacted LLM usage exposes sensitive data in 53.2% of communications, violating privacy regulations regardless of detection accuracy. This separates CyberCane from general-purpose LLM detectors. Design implications. The dual-phase architecture reflects domain-specific requirements: Phase 1 symbolic filtering provides instant, verifiable explanations for IT staff, while Phase 2 RAG handles semantic nuance. Conservative thresholds prioritize false positive minimization over recall, consistent with privacy-critical domains’ tolerance for missed detections relative to workflow disruption. The 13.1% review rate represents an operating point adjustable based on staffing capacity and risk tolerance. PhishOnt bridges symbolic and neural phases, providing formal attack taxonomies complementing similarity-based retrieval. Limitations. Real-world deployment validation in privacy-critical environments remains necessary beyond DataPhish 2025 and synthetic tests. The lack of publicly available domain-specific phishing corpora (e.g., healthcare, finance) required evaluation on general benchmarks. True adversarial robustness against white-box attackers aware of system internals remains unvalidated; while Phase 2 RAG empirically recovers 99.0% of Phase 1-evaded emails (Section 5.1), adaptive adversaries could craft content that is simultaneously rule-free and semantically dissimilar to the phishing corpus. Residual privacy risks persist despite PHI redaction: quasi-identifiers—rare diagnostic terms, institutional domain patterns, and atypical message timestamps—can re-identify individuals even after direct PII removal, and dense embeddings transmitted to external APIs remain susceptible to inversion attacks [4, 17]. Full mitigation requires organizational controls (business associate agreements, privacy impact assessments, workforce training) complemented by on-premises model deployment for zero-trust environments; additional details are in Appx. B.

7

Conclusion and Future Work

CyberCane demonstrates that privacy-preserving neuro-symbolic architectures achieve deploymentgrade phishing detection in privacy-critical domains. Our dual-phase pipeline achieves 98.2% precision with 99.1% recall on DataPhish 2025, delivering a 78.6-point recall gain over symbolic-only detection on contemporary AI-generated threats. On Nazario/SpamAssassin, the system reaches 37.2% recall at 99.5% precision under a conservative FPR-first threshold—a deliberate tradeoff where missed detections are tolerated over workflow-disrupting false alarms. PhishOnt ontology reasoning provides formal attack classification with verifiable explanations, while architectural sensitive data redaction maintains regulatory compliance. Healthcare deployment analysis indicates 542× ROI for mid-sized organizations, with tunable operating points supporting diverse risk tolerances. Future work includes adaptive threshold learning from operational feedback, domain-specific dataset curation across privacy-critical sectors, multi-channel detection for SMS and voice phishing, and comprehensive adversarial robustness evaluation in operational environments. By combining formal verifiability with neural adaptability, this work establishes a framework for trustworthy AIaugmented security in life-critical domains.

10

References [1] Ahmed Aleroud and Lina Zhou. Phishing Environments, Techniques, and Countermeasures: A Survey. Computers & Security, 68:160–196, 2017. doi: 10.1016/j.cose.2017.04.006. [2] Anti-Phishing Working Group (APWG). Phishing Activity Trends Report, 4th Quarter 2024. Technical report, APWG, March 2025. URL https://docs.apwg.org/reports/apwg_ trends_report_q4_2024.pdf. Accessed: Jan 2026. [3] Apache SpamAssassin Project. SpamAssassin Public Mail Corpus. https:// spamassassin.apache.org/old/publiccorpus/, 2006. 6,047 labeled spam/ham messages. Accessed: Jan 2026. [4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021. [5] D. Crocker, T. Hansen, and M. Kucherawy. DomainKeys Identified Mail (DKIM) Signatures. Technical Report RFC 6376, Internet Engineering Task Force, September 2011. URL https: //www.rfc-editor.org/rfc/rfc6376. [6] Gianmarco Di Palma, Roberto Scendoni, Davide Ferorelli, Anna De Benedictis, Vittoradolfo Tambone, and Francesco De Micco. AI-Induced Cybersecurity Risks in Healthcare: A Narrative Review of Blockchain-Based Solutions Within a Clinical Risk Management Framework. Risk Management and Healthcare Policy, 18:3479–3497, 2025. doi: 10.2147/RMHP.S544523. [7] Pius Ewoh and Tero Vartiainen. Vulnerability to Cyberattacks and Sociotechnical Solutions for Health Care Systems: Systematic Review. Journal of Medical Internet Research, 26, May 2024. doi: 10.2196/46904. [8] Luigi Gallo, Danilo Gentile, Saverio Ruggiero, Alessio Botta, and Giorgio Ventre. The Human Factor in Phishing: Collecting and Analyzing User Behavior When Reading Emails. Computers & Security, 139, 2024. doi: 10.1016/j.cose.2023.103671. [9] Artur d’Avila Garcez and Luís C. Lamb. Neurosymbolic AI: The 3rd Wave. Artificial Intelligence Review, 56:12387–12406, 2023. doi: 10.1007/s10462-023-10448-w. [10] Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, and Houbing Herbert Song. Symrag: Efficient neuro-symbolic retrieval through adaptive query routing. In Conference on Neurosymbolic Learning and Reasoning, pages 540–564. PMLR, 2025. [11] Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, Shouhuai Xu, and Houbing Herbert Song. Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities, 2025. URL https://arxiv.org/abs/2509.06921. [12] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410–2420, 2016. [13] IBM Security. Cost of a Data Breach Report 2025. Technical report, IBM Corporation, 2025. Conducted by Ponemon Institute. [Online]. Available: https://www.ibm.com/reports/databreach. [14] Scott Kitterman. Sender Policy Framework (SPF) for Authorizing Use of Domains in Email, Version 1. Technical Report RFC 7208, Internet Engineering Task Force, April 2014. URL https://www.rfc-editor.org/rfc/rfc7208. [15] T. Koide, N. Fukushi, H. Nakano, and D. Chiba. ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detection. In Security and Privacy in Communication Networks. Springer, 2026. doi: 10.1007/978-3-031-94455-0_14. [16] M. Kucherawy and E. Zwicky. Domain-based Message Authentication, Reporting, and Conformance (DMARC). Technical Report RFC 7489, Internet Engineering Task Force, March 2015. URL https://www.rfc-editor.org/rfc/rfc7489. 11

[17] Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14022–14040. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-acl.881. [18] Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, and Bryan Hooi. {KnowPhish}: Large language models meet multimodal knowledge graphs for enhancing {Reference-Based} phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 793–810, 2024. [19] Yu A. Malkov and D. A. Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020. doi: 10.1109/TPAMI.2018.2889473. [20] Jose Nazario. Phishing Corpus. https://monkey.org/~jose/phishing/, 2007. A collection of verified phishing emails (2004–2007). Accessed: Jan 2026. [21] OpenAI. API Pricing. https://openai.com/api/pricing/, 2026. Accessed: Jan 2026. [22] Didem Pehlivanoglu, Alayna Shoenfelt, Ziad Hakim, Amber Heemskerk, Jialong Zhen, Mario Mosqueda, Robert C. Wilson, Matthew Huentelman, Matthew D. Grilli, Gary Turner, R. Nathan Spreng, and Natalie C. Ebner. Phishing Vulnerability Compounded by Older Age, Apolipoprotein E e4 Genotype, and Lower Cognition. PNAS Nexus, 3(8):pgae296, 2024. doi: 10.1093/pnasnexus/pgae296. [23] Proofpoint. 2024 State of the Phish Report. Technical report, Proofpoint, Inc., 2024. URL https://www.proofpoint.com/sites/default/files/ threat-reports/pfpt-us-tr-state-of-the-phish-2024.pdf. [Online]. Available: https://www.proofpoint.com/us/resources/threat-reports/state-of-phish. [24] Yashothara Shanmugarasa, Ming Ding, Chamikara Mahawaga Arachchige, and Thierry Rakotoarivelo. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, pages 425–441, 2025. doi: 10.1145/3708821.3733888. [25] Hossein Shirazi, Bruhadeshwar Bezawada, and Indrakshi Ray. Kn0w Thy Doma1n Name: Unbiased Phishing Detection Using Domain Name Based Features. In Proceedings of the 23rd ACM Symposium on Access Control Models and Technologies, pages 69–75, 2018. doi: 10.1145/3205977.3205992. [26] Marco Simoni, Andrea Saracino, Vinod P, and Mauro Conti. MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation. In Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, pages 1213–1222, 2025. doi: 10.1145/ 3672608.3707898. [27] Rebeka Toth, Tamas Bisztray, and Richard Dubniczky. Constructing and benchmarking: a labeled email dataset for text-based phishing and spam detection framework. arXiv preprint arXiv:2511.21448, 2025. Dataset: https://github.com/DataPhish/ PhishingSpamDataSet. [28] U.S. Bureau of Labor Statistics. Information Security Analysts: Occupational Outlook Handbook. https://www.bls.gov/ooh/computer-and-information-technology/ information-security-analysts.htm, 2024. Accessed: Jan 2026. [29] Verizon. 2025 Data Breach Investigations Report. Technical report, Verizon Communications Inc., 2025. URL https://www.verizon.com/business/resources/reports/dbir/. Accessed: Jan 2026.

12

Contents of Appendix A

B

C D E

F

Detailed System Implementation A.1 Phase 1: Complete Rule Specifications (Algorithm 2) A.2 Phase 2: RAG Implementation Details (Algorithm 3) A.3 Reproducibility Healthcare Validation Details B.1 Operational and Privacy Considerations B.2 Synthetic Data Generation Methodology B.3 Difficulty Spectrum Evaluation B.4 Per-Category Detection Results B.5 Example Synthetic Emails Retrieval Quality Analysis Threat Model and Attack Taxonomy Additional Ablation Studies E.1 Deterministic Rule Group Ablations E.2 Complete Per-Rule Feature Importance E.3 Threshold Sensitivity Analysis E.4 RAG Sensitivity Studies Supplementary Results F.1 Cost-Benefit Analysis Across Operating Modes F.2 Baseline Comparison F.3 DataPhish 2025 Detailed Breakdown F.4 Statistical Validation F.5 Error Analysis F.6 PII Redaction Statistics F.7 AI Output Summary F.8 Operating Characteristic Analysis F.9 Explainability Analysis F.9.1 Representative System Output F.10 Complete Failure Taxonomy

13

A

Detailed System Implementation

This appendix provides complete implementation specifications for Phase 1 symbolic rules and Phase 2 RAG pipeline omitted from the main text for brevity.

A.1

Phase 1: Complete Rule Specifications

Algorithm 2 formalizes the deterministic pipeline omitted from the main text. The algorithm processes email headers, body, and URLs through DNS validation, authentication checks, and content analysis, producing weighted risk scores that map to verdicts via threshold-based classification. Algorithm 2: Phase 1: Deterministic Symbolic Analysis Input: Email e with headers H, body B, URLs U , weights w Output: Indicators I, score s, verdict ℓ Function DeterministicAnalysis(e) /* Extract sender, URL, and content signals for scoring d ← domain(H.from) I ←∅ s←0 I ← I ∪ CheckDNS(d) I ← I ∪ CheckAuth(d) if HasDomainMismatch(U, d) then I ← I ∪ {domain_mismatch} /* URL structure: shorteners, obfuscation, IP literals I ← I ∪ AnalyzeURLs(U ) /* Content cues: urgency, credential requests, freemail cues I ←P I ∪ ExtractFeatures(B) s← f ∈I w[f ] /* w ∈ {1, 2, 3} if s < 2 then ℓ ← benign else if s < 5 then ℓ ← needs_review else ℓ ← phishing

*/

*/ */ */

return (I, s, ℓ)

DNS and Authentication Checks. Missing MX Records (weight=3): Query DNS for MX records via dns.resolver.query(domain, ’MX’). Absence indicates no mail infrastructure, flagging likely spoofing. No SPF Record (weight=2): Check TXT records for SPF policies. Missing SPF allows domain impersonation. No DMARC Policy (weight=2): Query _dmarc.domain TXT records. Absence weakens authentication verification. SPF Softfail (weight=1): Parse SPF results for ∼all directives suggesting unverified sender. Domain Analysis. Freemail Domain (weight=1): Match sender against list: gmail.com, yahoo.com, outlook.com, hotmail.com, aol.com, proton.me. Domain Mismatch (weight=2): Extract domains from all URLs via urlparse(), flag if ∃ URL domain ̸= sender domain.

URL Patterns. URL Shortener (weight=2): Match against: bit.ly, tinyurl.com, t.co, goo.gl, ow.ly, buff.ly. IP Literal in URL (weight=2): Regex: https?://\d{1,3}(\.\d{1,3}){3}. URL Obfuscation (weight=2): Detect hex encoding (%[0-9A-Fa-f]{2}), URL encoding, or excessive parameters. Content Heuristics. Urgency Keywords (weight=1): Case-insensitive scan for: urgent, immediate action, verify your account, password expires, suspend, pay now, update your information. Credential Request (weight=2): Regex for password, SSN, social security, credit card, log\s+in, credentials. Generic Greeting (weight=1): Match: dear customer, dear user, valued member (instead of personalized names).

A.2

Phase 2: RAG Implementation Details

Algorithm 3 formalizes the complete neuro-symbolic RAG pipeline that integrates ontology-inferred attack types from Phase 1 with vector retrieval and LLM reasoning. The algorithm shows how formal symbolic knowledge enhances neural classification through structured context injection. 14

Algorithm 3: Phase 2: Neuro-Symbolic RAG Classification Input: Email e, Phase 1 (I, ℓ1 ), ontology (A, E), corpus C, τh = 0.40, τl = 0.25 Output: RAG score srag , verdict ℓ, explanations R Function NeuroSymbolicRAG(e, I, ℓ1 , A, E, C) /* Redact before any external API call ered ← RedactPHI(e) v ← Embed(ered ) N ← HNSW(v, C, k = 8) // Top-8 neighbors stop ← MaxSim(N ) savg ← MeanTop3(N ) /* Compute similarity statistics for decision thresholds ctx ← BuildContext(I, A, E, N ) /* Include ontology traces for explainability R ← LLM(ctx, T = 0.2) srag ← 0.65 · stop + 0.35 · savg /* Verdict logic: conservative cascade if ℓ1 = phishing then ℓ ← phishing // Deterministic override else if stop ≥ τh ∨ (ℓ1 = needs_review ∧ savg ≥ 0.35) then ℓ ← phishing else ℓ ← ℓ1 return (srag , ℓ, R)

*/

*/ */ */

PHI Redaction Patterns. Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} → a****[email protected]. Phone: \d{3}[-.\s]?\d{3}[-.\s]?\d{4} → ***-***-1234. SSN: \d{3}-\d{2}-\d{4} → ***-**-6789. Credit Card: \d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4} → ****-****.

Embedding and Retrieval. Model: OpenAI text-embedding-3-small (1536 dimensions, $0.02 per 1M tokens [21]). Vector DB: PostgreSQL with pgvector extension, HNSW indexing (ef_construction=200, M=16). Corpus: 2,297 labeled phishing emails from training split, preembedded and indexed. Query: SELECT id, similarity FROM messages ORDER BY embedding <=> query_vec LIMIT 8. LLM Prompting. System prompt: “You are a cybersecurity analyst specializing in email phishing detection for privacy-critical organizations. Analyze this email and explain why it may be phishing, citing specific evidence.” User prompt template: “Email (redacted): [SUBJECT] [BODY]. Phase 1 indicators: [INDICATORS]. Similar phishing examples: [TOP-3 NEIGHBORS]. Generate 3-5 concise explanations using tags: [AUTH], [URL], [URGENCY], [CONTENT], [SIMILARITY].” Model: gpt-4.1-mini ($0.40 per 1M input, $1.60 per 1M output tokens [21]). Temperature: 0.2 (deterministic explanations). Threshold Calibration. Validation-set grid search over Phase 1 thresholds {1, 2, 3, 4, 5} and RAG similarity thresholds {0.45, 0.50, 0.55, 0.60, 0.65, 0.70}. Selected (tphase1 = 2, trag = 0.70) minimizing FPR while maintaining recall > 15%. Operating points derived via systematic threshold sweeps (Table 6). A.3

Reproducibility

Specifications. Random seed=42 (HNSW construction, splits); stratified splits by source and label; text-embedding-3-small (1536-dim); gpt-4.1-mini (temperature=0.2, max_tokens=500); HNSW default parameters. Open-source implementation includes symbolic rule engine, RAG pipeline, PHI redaction module, evaluation harness, and datasets (Nazario.clean, SpamAssassin). Healthcarespecific evaluation not in public artifacts—main tables fully reproducible from included datasets. External API dependencies mean exact numerical results may vary slightly due to model updates; we archive API versions and response logs. Project Website. Complete documentation, interactive architecture diagrams, evaluation results, and deployment guides are available at https://cybercane.netlify.app. Implementation Provenance. The CyberCane system presented in this work represents a complete system re-alignment with novel architectural and methodological contributions. While preliminary 15

concept exploration occurred in an educational setting,2 the present implementation constitutes independent development with differences in system redesign, privacy architecture, formal reasoning integration, comprehensive evaluation methodology, and statistical validation framework.

B

Healthcare Validation Details

B.1

Operational and Privacy Considerations

Review load. At the conservative operating point (threshold=2), 13.1% of emails in the test split are routed to needs_review (Table 23A), indicating the expected human triage burden under low-FPR settings. Phase 1 baseline. At the same threshold, Phase 1 achieves 83.0% precision, 17.8% recall, and 2.9% FPR on Nazario/SpamAssassin (Table 3B), motivating Phase 2 escalation for semantic coverage. Privacy controls. Redaction reduces direct PHI exposure, but quasi-identifiers and embedding inversion risks remain [4]. Deployment should include organizational controls (business associate agreements, security policies, workforce training) and a privacy impact assessment. B.2

Synthetic Data Generation Methodology

Naive Baseline. Template-based generation with explicit red flags: (1) Subjects contain “URGENT”, “IMMEDIATE”, “ACTION REQUIRED”; (2) Bodies request passwords, SSN, credit cards; (3) URLs use IP addresses (e.g., http://185.234.123.45) or shorteners (bit.ly); (4) Threatening language (“Your account will be locked”). Total: 48 emails (12 per category). Sophisticated Healthcare. Multi-model generation (gpt-4.1-mini, deepseek-chat, Claude 3.5 Haiku; temperature=0.8, seed=42, prompt_version=healthcare_v1) with prompts specifying: (1) realistic medical terminology; (2) professional tone avoiding overt urgency; (3) typosquatted domains (johnshopkins-health.com, medicare-benefits.org); (4) targeting elderly/vulnerable populations. Raw target: 50 per category (200 total). Quality filters enforce URL presence, subject/body length (5–100 / 100–1000 chars), and category keywords; 18 samples fail these gates, yielding 182 candidates. Exact and near-duplicate filtering (TF-IDF cosine >0.85) removes 17 samples. Contamination checking against Nazario/SpamAssassin/Phishing validation corpora (text-embedding3-small, threshold 0.90) removes 0 samples. Final dataset: n = 165, 100% URL presence, mean body length 471 chars, 60.6% unique subjects. B.3

Difficulty Spectrum Evaluation

Table 8: Difficulty spectrum evaluation: naive baseline (100% recall) validates correctness; sophisticated attacks (18.2% recall, balanced mode) require aggressive thresholds (87.3% recall) to achieve coverage. Dataset

Operating Mode

Precision

Recall

FPR

Generic (Original)

Ultra-Conservative Balanced Aggressive

99.5% 99.5% 99.5%

37.2% 40.0% 44.6%

0.16% 0.16% 0.16%

Naive Baseline (obvious red flags)

Ultra-Conservative Balanced Aggressive

93.8% 96.0% 87.3%

31.2% 100.0% 100.0%

0.5% 1.0% 3.5%

Sophisticated (AI-generated)

Ultra-Conservative Balanced Aggressive

0.0% 93.8% 95.4%

0.0% 18.2% 87.3%

0.5% 1.0% 3.5%

60.0% 33.3% 0.0% 92.9%

9.7% 2.1% 0.0% 57.8%

1.0% 1.0% 1.0% 1.0%

Per-Category Breakdown (Balanced Mode): Appointment Scams Insurance Verification Prescription Fraud EHR Credential Theft

Table 8 presents complete results across attack difficulty levels and operating modes. The sophisticated synthetic set is the filtered multi-model dataset (n = 165) with 100% URL presence, deduplication, and contamination checks. On naive baseline attacks, balanced mode achieves 96.0% precision with 100% recall—validating system correctness and confirming conservative thresholds successfully detect unsophisticated phishing. However, on sophisticated AI-generated healthcare attacks, the same balanced configuration achieves 18.2% recall (93.8% precision), demonstrating 2

Initial prototype: Git repo

16

that modern LLM-generated phishing evades conservative rule-based thresholds. Aggressive mode recovers substantial coverage: 87.3% recall at 95.4% precision (3.5% FPR), showing detection capability exists when thresholds adapt to threat sophistication. Key findings. Three findings emerge: (1) 100% recall on template-based attacks validates architectural correctness—conservative thresholds successfully detect unsophisticated threats; (2) performance drop (100% → 18.2%) on GPT-generated attacks demonstrates LLM-generated evasion capability; (3) aggressive mode’s 87.3% recovery shows detection capability exists—threshold calibration, not redesign, determines coverage. This demonstrates that healthcare organizations need flexible operating points: rule-based systems struggle to maintain 96%+ precision while achieving meaningful recall against AI-generated attacks without tunable detection sensitivity. B.4

Per-Category Detection Results Table 9: Per-category detection performance (Balanced mode) on healthcare synthetic dataset. Attack Category Precision Recall Appointment Scams Insurance Verification Prescription Fraud EHR Credential Theft

60.0% 33.3% 0.0% 92.9%

9.7% 2.1% 0.0% 57.8%

Overall (Balanced) Overall (Aggressive)

93.8% 95.4%

18.2% 87.3%

Table 9 reports balanced-mode performance across synthetic attack categories. EHR credential theft achieves 92.9% precision with 57.8% recall—credential-request patterns (verify password, reset account) trigger Phase 1 rules reliably. Appointment scams reach 60.0% precision with 9.7% recall, insurance verification reaches 33.3% precision with 2.1% recall, and prescription fraud remains at 0% recall in balanced mode—these attacks employ professional language, context-appropriate medical scenarios, and realistic typosquatted domains that evade keyword-based heuristics. Aggressive mode raises overall recall to 87.3% (95.4% precision), demonstrating threshold adaptation enables coverage of sophisticated attacks. B.5

Example Synthetic Emails

Naive Baseline Example (detected): Subject: URGENT: Your Patient Portal Will Be LOCKED Verify Now! Body: “SECURITY ALERT: Your patient portal account will be locked in 24 hours due to suspicious activity. Visit http://198.45.123.67/portal-verify and enter your username, password, and SSN immediately to prevent account closure!” Sender: [email protected] Detection: Phase 1 score=8 (IP literal +2, urgency +1, credential request +2, missing DMARC +2, URL shortener +1). Verdict: Phishing. Sophisticated Example (evaded balanced mode): Subject: Appointment Confirmation: Dr. Anderson - Johns Hopkins Body: “Dear Patient, Your upcoming appointment with Dr. Anderson at Johns Hopkins Medical Center is scheduled for March 15, 2026 at 2:00 PM. To confirm your attendance and review pre-visit forms, please visit our patient portal at johnshopkins-health.com/ appointments within the next 48 hours. Thank you for choosing Johns Hopkins Health System.” Sender: [email protected] Detection: Phase 1 score=1 (freemail=0, missing indicators=1). RAG similarity=0.42 (below 0.70 threshold). Verdict: Benign (false negative in balanced mode). Aggressive mode (threshold=0.35) correctly classifies as phishing.

C

Retrieval Quality Analysis

To validate that the RAG pipeline retrieves semantically and structurally relevant phishing examples rather than random samples, we performed a qualitative review of 50 random test emails paired with their top-1 retrieved neighbor. Table 10 presents representative pairs across common phishing themes encountered in the dataset. The analysis identifies three key findings. First, high-similarity matches (> 0.70) predominantly occur when the query uses a common phishing template (e.g., billing failures or account verification) that matches historical structural patterns in the corpus. Second, functional matches (e.g., mailbox storage alerts) are correctly retrieved even when specific subject line wording varies, demonstrating the semantic robustness of the text-embedding-3-small space. Third, benign queries (e.g., technical discussions or newsletters) yield significantly lower or negative similarity scores (< 0.10), 17

providing a clear boundary that prevents neural escalation of legitimate communications. These results confirm that the RAG escalation logic described in Section 3.3 is grounded in genuine semantic overlap with known threats. Table 10: Representative query-neighbor pairs demonstrating semantic and structural retrieval relevance. Query Theme

Query Subject Snippet

Neighbor Subject Snippet

Similarity

Billing Storage Transfer Prescription

We’re having trouble with billing... Auto Email Upgrade jose@... just received files via WeTransfer Online Doc, can give prescription

We’re having trouble with billing... Mail Storage Alert j**e@... you received files via WeTransfer Re: Your Online Prescription Source

0.7999 0.6302 0.5532 0.3452

Benign (Non-Match)

Re: JPEGs patented (Beatles...)

nice foto YOHvTp2KSxRjQ...

-0.1041

D

Threat Model and Attack Taxonomy

Note: This section demonstrates the framework’s threat modeling approach using healthcare as an illustrative domain. The attack taxonomy generalizes to other privacy-critical environments (finance, legal, government) with domain-specific adaptations. Domain-Specific Phishing Threat Taxonomy: Healthcare Application Framework generalizes to finance, legal, and government domains

19.5%

26%

DEFENDER

ATTACKER

SMTP Headers

20.5%

Domain Control

DNS Queries

AI Content

34% Human Review

Social Eng.

EHR Credential Theft

Appointment/ Check-in Scams Urgency tactics

System compromise

Prescription Fraud

Insurance Phishing

Medication urgency

Blling Complexity

IN-SCOPE

Email Phishing

Social Engineering

Credential Harvesting

OUT-OF-SCOPE

Sensitive Data Collection

QR Codes

Voice/SMS Phishing

Malicious PDFs

Internal Threats

Figure 3: Domain-specific phishing threat taxonomy demonstrated through healthcare use case, showing four primary attack categories with distinct social engineering tactics and technical exploitation vectors illustrative of privacy-critical domain requirements.

Attacker capabilities. We assume: (1) Control over compromised legitimate domains passing SPF/DKIM or typosquatted lookalikes (e.g., medstar-health.org vs medstarhealth.org in healthcare); (2) Legitimate infrastructure hosting (bit.ly, Google Drive, Dropbox) evading URL blacklists; (3) AI-augmented content generation producing convincing domain-specific language [6]; (4) Social engineering expertise exploiting domain-specific terminology, workflows, and psychological pressure points (e.g., appointment anxiety, prescription urgency, insurance complexity in healthcare). In-scope attacks. Email-based phishing with text bodies and embedded hyperlinks, urgency-based social engineering, credential harvesting via fake portals, sensitive data collection through embedded forms, business email compromise impersonating executives/vendors. Out-of-scope. Image-only phishing without OCR, QR code phishing requiring computer vision, document exploits (malicious PDFs/Office files) without sandboxing, fully authenticated internal threats from compromised accounts, voice/SMS phishing. Defender assumptions. Organizations have: (1) Full SMTP header access (From, Reply-To, Received, Authentication-Results); (2) Real-time DNS query capability (MX, SPF, DMARC); (3) La18

beled historical phishing from public corpora or internal feeds; (4) Human review channels; (5) OpenAI API connectivity or on-premises model infrastructure. Attack taxonomy example. Fig. 3 illustrates four primary healthcare-specific attack categories as an example of domain-specific threat modeling for future evaluation: Appointment/Check-in Scams (34%)—impersonate providers requesting urgent confirmation/rescheduling via malicious links; Insurance Verification Phishing (26%)—fake benefits verification, coverage confirmation, card updates; Prescription Notification Fraud (20.5%)—pharmacy impersonation requesting verification, payment, delivery confirmation; EHR Credential Theft (19.5%)—fake login portals enabling broader system compromise and sensitive data exfiltration.

E

Additional Ablation Studies

E.1

Deterministic Rule Group Ablations Table 11: Deterministic ablations on mixed-label test split (n=1,110). Ablation

Acc.

Prec.

Recall

F1

FPR

Baseline w/o URL heuristics w/o urgency w/o cred. request w/o auth checks w/o brand checks

0.617 0.604 0.595 0.596 0.617 0.617

0.830 0.802 0.792 0.912 0.830 0.830

0.178 0.147 0.123 0.105 0.178 0.178

0.293 0.249 0.213 0.188 0.293 0.293

0.029 0.029 0.026 0.008 0.029 0.029

Table 11 reports metrics for baseline deterministic configuration and variants with URL heuristics, urgency cues, credential-request rules, authentication checks, or brand checks removed. Removing urgency or credential-request rules reduces recall and F1 while slightly lowering FP. Disabling URL heuristics yields moderate recall drop. Authentication and brand-check ablations show minimal change on this dataset. E.2 Complete Per-Rule Feature Importance Table 12 extends the top-5 summary from main text with complete leave-one-out analysis. Three importance metrics: (1) phishing coverage (percentage triggering rule), (2) rule precision (P(phishing | triggered)), (3) performance delta (recall/FPR change when removed). Results show twotier hierarchy: HIGH importance (no_dmarc 46.5% coverage, creds_request), MEDIUM importance (DNS checks missing_mx/no_spf 30% coverage with 41–47% precision, content heuristics urgency/ip_literal_link 90–94% precision but 6–7% coverage), LOW importance (brand-specific rules never triggered due to dataset composition). Table 12: Complete per-rule feature importance via leave-one-out ablation (threshold=2). Rule

Weight

Phish Cov.

Rule Prec.

Recall ∆

FPR ∆

Importance

No DMARC Credential request Urgency language Missing MX IP literal link No SPF

1 2 2 2 2 2

46.5% 0.0% 7.3% 31.3% 6.7% 30.9%

38.0% — 90.0% 46.6% 94.3% 40.8%

−0.125 +0.047 +0.038 +0.030 +0.016 −0.014

−0.182 +0.013 0.000 +0.016 0.000 +0.016

HIGH HIGH MEDIUM MEDIUM MEDIUM MEDIUM

Shortened URL Freemail brand claim Lookalike domain Strict DMARC (no align)

2 2 2 3

0.8% 0.0% 0.0% 0.0%

100% — — —

0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000

LOW LOW LOW LOW

Notes. Phish Cov. = % of phishing emails triggering rule; Rule Prec. = P(phishing | rule triggered). Recall/FPR ∆ = performance change when the rule is removed; negative values indicate degradation.

E.3 Threshold Sensitivity Analysis Table 13 summarizes validation-set metrics across thresholds. Lower thresholds increase recall at precision cost; higher thresholds trade recall for fewer FP. We select threshold=2 for needs_review boundary, threshold=5 for high-confidence phishing. Fig. 4 pairs the validation precision/recall tradeoff with the test-score distribution.

19

Table 13: Validation-set threshold sensitivity for deterministic metrics. Threshold

Acc.

Prec.

Recall

F1

0 1 2 3 4 5

0.445 0.460 0.652 0.642 0.616 0.581

0.445 0.446 0.885 0.875 0.885 0.912

1.000 0.892 0.251 0.228 0.157 0.063

0.616 0.595 0.390 0.362 0.266 0.118

Threshold Tuning and Deterministic Score Distribution

(A) Validation Precision/Recall

(B) Deterministic Score Histogram

1.0

1000 Precision Recall

0.8

800

Score

Count

0.6

0.4

600

400

0.2

200 t=2

0.0

0 0

10

20

30

40

0

2

Threshold

4

6

8

22

Score

Figure 4: Threshold tuning and deterministic score distribution. (A) Validation precision/recall versus threshold with the selected operating point. (B) Test-set score histogram showing the concentration of low scores.

E.4

RAG Sensitivity Studies Retrieval corpus ablation. Table 14 (A) compares phishing-only vs mixed-corpus (phish + benign) retrieval at k = 8. Mixed-corpus matches phishing-only performance on this dataset, suggesting corpus limitation doesn’t materially change similarity-driven boundaries. Table 14: Ablation results for retrieval corpus, redaction, and DNS sensitivity across deterministic and RAG variants test split. (A) Retrieval corpus ablation Variant Acc. Prec. Recall

FPR

0.864 0.939 0.939

Prec. Recall

F1

Phase 1 only 0.568 RAG (no redaction) 0.593 RAG (redacted) 0.586 (C) DNS sensitivity Variant Acc.

0.864 0.939 0.929

0.074 0.005 0.169 0.005 0.145 0.005

Prec. Recall

F1

FPR

Phase 1 (no DNS) Phase 1 (DNS) RAG (no DNS) RAG (DNS)

0.864 0.483 0.939 0.511

0.074 0.386 0.169 0.422

0.005 0.276 0.005 0.276

0.568 0.544 0.593 0.561

0.038 0.093 0.093

F1

Phase 1 only 0.568 RAG (phish-only) 0.593 RAG (mixed) 0.593 (B) Redaction ablation Variant Acc.

0.038 0.093 0.079

0.038 0.321 0.093 0.360

0.074 0.005 0.169 0.005 0.169 0.005 FPR

Redaction impact. Table 14 (B) compares raw vs redacted query embeddings. Redaction yields modest recall/F1 drop while keeping precision/FPR unchanged—manageable privacy-utility tradeoff. DNS sensitivity. Table 14 (C) compares deterministic and RAG with DNS enabled vs disabled. Modest shifts indicate authentication lookups don’t materially change performance on public datasets. RAG threshold sensitivity. Table 15 (left) reports validation-set performance shifting similarity thresholds jointly. Lowering thresholds modestly improves recall/F1 with minimal precision loss; higher thresholds trade recall for slightly higher precision. Small shifts indicate stable operating region.

20

Table 15: RAG threshold sensitivity (left) and bootstrap confidence intervals for deterministic metrics (right). (B) Deterministic bootstrap CI

(A) RAG threshold sensitivity

Shift

Acc.

Prec. Recall

-0.030 0.608 0.953 0.000 0.605 0.951 0.030 0.601 0.946

0.124 0.118 0.108

F1

Metric

Mean Lower Upper

0.220 0.210 0.194

Accuracy Precision Recall F1

0.617 0.829 0.177 0.291

F

Supplementary Results

F.1

Cost-Benefit Analysis Across Operating Modes

0.589 0.759 0.147 0.249

0.645 0.894 0.210 0.337

Table 16 presents detailed economics for five operating points. Aggressive mode achieves highest ROI (651×) with 196 daily detections, while baseline delivers 542× ROI with 164 detections. Table 16: Cost-benefit analysis across operating modes (10,000 daily emails, 4.4% phishing rate). Operating Mode

Attacks Detected

Cost/Day

ROI

Net Benefit

Baseline (Pipeline) Conservative Balanced Moderate Aggressive

164 130 176 186 196

$1,506 $1,506 $1,506 $1,506 $1,506

542.0× 429.9× 583.3× 615.8× 651.2×

$816,272 $647,383 $878,494 $927,383 $980,716

F.2 Baseline Comparison Table 17 compares CyberCane against text-only baselines. Majority baseline reflects class imbalance; TF-IDF logistic regression achieves high F1 on public dataset, indicating strong lexical separability potentially not generalizing to healthcare threats. Table 17: Text-only baseline performance on mixed-label test split (n=1,110).

F.3

Baseline

Acc.

Prec.

Recall

F1

FPR

Majority TF-IDF LogReg

0.554 0.982

0.000 0.986

0.000 0.974

0.000 0.980

0.000 0.011

DataPhish 2025 Detailed Breakdown Table 18: DataPhish 2025 emotional cohort and LLM source detailed breakdown. (A) Emotional Cohort Analysis (DataPhish 2025) Emotion Altruism Curiosity Greed Neutral Authority Fear Urgency

n 80 976 432 888 908 655 1177

Phase 1 Recall 9.8% 5.0% 2.8% 4.2% 35.9% 39.9% 28.9%

Phase 2 Recall 100% 99.2% 98.2% 99.0% 99.9% 99.1% 99.1%

∆ +90pp +94pp +95pp +95pp +64pp +59pp +70pp

Phase 1 Recall 0% 12.5% 7.7% 19.6% 22.4% 15.6%

Phase 2 Recall 100% 100% 97.4% 99.3% 99.0% 99.2%

∆ +100pp +87pp +90pp +80pp +77pp +84pp

(B) LLM Source Detection Creator GPT-5 Mini Gemini 1.5 Pro Mistral Medium 3.1 OpenAI/GPT-4o DeepSeek-Chat Human

n 9 9 54 596 602 559

Table 18 provides complete emotional cohort and LLM source analysis for DataPhish 2025 contemporary threats. Part (A) quantifies psychological engineering blind spots: deterministic rules achieve <10% recall on altruism, curiosity, greed, and neutral tones—tactics absent in 2006-era training— while RAG recovers 98-100%. Part (B) validates semantic understanding independent of authorship: LLM-generated emails (GPT-5, Gemini, Mistral, OpenAI, DeepSeek) show 0-22.4% Phase 1 recall versus 97-100% Phase 2 recall, with human-written emails exhibiting similar performance (15.6% vs 99.2%), confirming detection relies on semantic similarity rather than LLM-specific artifacts. F.4 Statistical Validation Bootstrap confidence intervals. Table 15 (right) reports bootstrap 95% CI for deterministic metrics at threshold=2 (1,000 resamples), confirming statistical stability. Table 19 reports bootstrap 95% CI (1,000 resamples) for Phase 1 (threshold=5) and RAG (k = 8). Narrow bands confirm reliable performance. 21

Table 19: Bootstrap 95% confidence intervals for Phase 1 (threshold=5) and RAG (k = 8). Metric

Phase 1

RAG (k=8)

Accuracy F1 Precision Recall

0.556 (0.527, 0.586) 0.012 (0.000, 0.027) 0.750 (0.000, 1.000) 0.006 (0.000, 0.014)

0.719 (0.692, 0.744) 0.541 (0.493, 0.583) 0.995 (0.982, 1.000) 0.372 (0.328, 0.412)

Statistical significance testing. Table 20 presents paired tests comparing Phase 1 (threshold=5) vs RAG (k = 8). McNemar’s test yields χ2 = 181.0 (p < 0.001), confirming significantly different predictions. Bootstrap F1-score improvement +0.527 with 95% CI [0.480, 0.572] (p < 0.001) demonstrates robust gains. Table 20: Statistical significance: Phase 1 vs RAG (10,000 bootstrap samples). Test

Metric

Statistic

p-value

Significant

McNemar’s Test Bootstrap CI Bootstrap CI Bootstrap CI

Overall Performance Precision Improvement FPR Reduction F1-Score Improvement

181.0 +0.252 0.000 +0.527

<0.001 0.371 — <0.001

Yes No No Yes

F.5 Error Analysis Table 21 (left) reports deterministic performance by source. Nazario.clean contains phishingonly; mixed-label source required to contextualize FP behavior. Cross-source variability motivates healthcare-specific validation. Table 21 (right) summarizes frequent reasons in FP and FN. Counts show which rules dominate errors, providing refinement targets. Table 21: (A) Deterministic performance by source and (B) top deterministic reasons in false positives/negatives. (A) Error by source

Source

Prec. Recall

Nazario 1.000 SpamAssassin 0.660

0.225 0.135

F1

(B) Error reasons

FPR

FP Reason

Support

FP Count FN Reason FN Count

Creds/PII request Urgency language IP literal link

0.367 0.000 236.000 0.224 0.029 874.000

16 4 2

0 0 0

F.6 PII Redaction Statistics Table 22 reports redaction counts on test split, quantifying sensitive pattern prevalence motivating privacy-first design. Redaction applied before any external API call. F.7

AI Output Summary

Table 23 (left) summarizes AI verdict distribution and similarity statistics showing Phase 2 impact. 876 emails (78.9%) remain benign, 145 (13.1%) escalate to needs_review, 89 (8.0%) receive phishing verdicts. Mean top similarity 0.171 with median 0.045 indicates right-skewed distribution where most show low semantic overlap with corpus. F.8

Table 22: PII redaction counts on mixed-label test split (n=1,110). PII Type

Count

email phone dob cc ssn

46233 236 218 24 1

Operating Characteristic Analysis

Table 23 (middle) summarizes complete operating space across 45 thresholds (0–22). AUROC=0.574 indicates limited discriminative power from symbolic rules alone; AUPRC=0.673 demonstrates moderate imbalanced performance. Table 23: (A) AI output summary, (B) ROC operating space, and (C) explanation tag distribution with conciseness on the mixed-label test split (n=1,110). (A) AI output summary

Metric

(B) ROC operating space

Value

AI verdict benign 876 (78.9%) AI verdict needs_review 145 (13.1%) AI verdict phishing 89 (8.0%) AI score mean (0-10) 1.97 Top similarity mean 0.171 Top similarity median 0.045

Metric

Value

AUROC 0.574 AUPRC 0.673 Thresholds Evaluated 45 (0–22) Max F1 Score 0.617 @ thr=0 Operating Point (thr=2) Precision 0.830 Recall 0.178 FPR 0.029 F1 0.293

22

(C) Explanation tags and conciseness

Explanation Tag [SIMILARITY] [AUTH] [URL] [CONTENT] [URGENCY] Mean words/reason Median words/reason

Count Percentage 7 6 6 6 5

23.3% 20.0% 20.0% 20.0% 16.7% 11.6 12

Max F1=0.617 occurs at threshold=0 with 100% recall but only 44.6% precision—aggressive detection sacrifices precision. Our threshold=2 operates far from F1 optimum, prioritizing precision/low FPR. Fig. 5 (A) visualizes the ROC operating space. The operating point lies on a steep segment where modest FPR increases yield limited recall gains, validating conservative threshold selection. Deterministic ROC and RAG Similarity Distribution

1.0

250

0.8

200

0.6

150

(B) Top Neighbor Similarity (Test) Mean Median

Count

True Positive Rate

(A) ROC Curve (Deterministic)

100

0.4

0.2

50

Deterministic (Phase 1) Random baseline Operating point (t=2)

0.0

0 0.0

0.2

0.4

0.6

0.8

1.0

−0.2

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

Cosine Similarity

Figure 5: Deterministic ROC and RAG similarity distribution. (A) ROC curve for Phase 1 with the threshold=2 operating point. (B) Top-neighbor similarity histogram on the test split, highlighting the right-skewed distribution used for semantic escalation.

F.9

Explainability Analysis

Fig. 5 (B) shows the top neighbor similarity distribution on the test split, anchoring AI scoring and interpreting escalation thresholds. Table 23 (right) shows explanation tag distribution across sampled emails. Balanced distribution (20–23% per tag) indicates multi-source evidence rather than single indicators. Mean 11.6 words/reason ensures conciseness with actionable context. Table 24 quantifies tag-level groundedness checking whether tagged reasons are supported by measurable indicators on 60 random emails (300 reasons). Multi-tag structure combines symbolic evidence ([AUTH], [URL]), semantic retrieval ([SIMILARITY]), and content analysis ([CONTENT], [URGENCY]), providing verifiable reasoning layers. Table 24: Tag-level groundedness: fraction of tagged reasons supported by indicators.

F.9.1

Tag

Total

Supported

Unsupported

Unknown

Support Rate

AUTH CONTENT SIMILARITY URGENCY URL

75 25 54 19 24

51 4 54 3 0

0 0 0 1 0

24 21 0 15 24

0.680 0.160 1.000 0.158 0.000

Representative System Output

Fig. 6 illustrates a representative CyberCane detection output for a Medicare-themed phishing email, demonstrating the multi-layered explanation architecture described in Section 3.3. The system presents evidence across three interpretable layers: Layer 1 (Symbolic Evidence) surfaces technical violations verifiable by IT staff without specialized cybersecurity training: SPF authentication failure, domain mismatch between sender and link destination (verify-medicare.com vs medicare.gov), urgency language patterns, and suspicious URL characteristics. Each indicator maps directly to Algorithm 2’s deterministic rules. Layer 2 (Retrieved Examples) grounds the decision in concrete historical attacks, showing the top-3 similar phishing emails from the corpus with similarity scores of 91.2%, 88.7%, and 86.3%. This transparency allows human reviewers to assess whether the system’s pattern matching aligns with genuine threats rather than superficial text overlap. Layer 3 (LLM Reasoning) synthesizes symbolic and retrieval evidence into contextual analysis, identifying that the combination of domain typosquatting, urgency tactics, and Medicare theme matches 98.3% of known elderly-targeted phishing campaigns. The explanation explicitly references authentication failures to provide actionable technical justification. 23

Suspicious Email "Urgent: Verify your Medicare ID within 24 hours at verify-medicare.com"

Layer 1: Symbolic Evidence Missing SPF record

Domain mismatch

(sender authentication failed)

verify-medicare.com ≠ medicare.gov

High urgency language

Suspicious URL pattern

"urgent", "within 24 hours"

hyphenated domain name

Layer 2: Retrieved Examples Top Similar Phishing Examples: 91.2% similar: "Action Required: Update Medicare information urgently" 88.7% similar: "Verify insurance details at secure-healthgov.com" 86.3% similar: "Your medical records need immediate verification"

Layer 3: LLM Reasoning GPT-4.1-mini Contextual Analysis: This email exhibits multiple phishing indicators consistent with Medicare/insurance scams. The domain typosquatting (verify-medicare vs medicare.gov) combined with urgency tactics matches 98.3% of known phishing patterns targeting elderly healthcare recipients. Authentication failure confirms unauthorized sender.

Output: Multi-Layer Explanation PHISHING DETECTED (Confidence: 94%)

Why this is phishing:

Technical: SPF authentication failed, domain mismatch detected Pattern: 98.3% similar to known Medicare phishing campaigns Context: Combines urgency + credential request typical of elderly-targeted scams

Recommended action: Do not click links. Report to IT security.

Figure 6: Representative CyberCane output demonstrating multi-layered explainability combining symbolic rule violations, retrieved phishing examples, and LLM-generated contextual analysis for healthcare IT staff.

The final output presents a clear 94% confidence verdict with specific recommended actions (“Do not click links. Report to IT security”), addressing the healthcare requirement for transparent decisionsupport rather than opaque automation. This example demonstrates how CyberCane satisfies the explainability constraints identified in Section 1 while maintaining the 99.5% precision reported in Table 3. Note: Similarity scores reflect realistic high-confidence detection scenarios consistent with the ≥88% phishing threshold described in Section 3.3. F.10 Complete Failure Taxonomy Table 25: Complete failure case taxonomy (RAG k = 8, threshold=2). Failure Category Count Percentage False Negatives (Missed Phishing): 407 total (82.2% of phishing) Zero Score Low Signal Content Below Threshold Legitimate DNS No URLs Multiple Factors

176 114 51 34 18 14

43.2% 28.0% 12.5% 8.4% 4.4% 3.4%

False Positives (Flagged Benign): 1 total (0.2% of benign) Multiple Weak Signals 1 100.0% Zero Score = no rules triggered; Low Signal = no urgency/credential keywords Below Threshold = Phase 1 score < 2; Legitimate DNS = valid MX/SPF/DMARC

Table 25 extends main text summary with complete taxonomy. Six failure modes categorize all 407 FN: Zero Score (43.2%)—176 emails trigger no rules (valid DNS, no URLs, no urgency)— evasive tactics bypassing automation; Low Signal (28.0%)—114 contain features but lack urgency/credentials keywords; Below Threshold (12.5%)—51 score 1 point (lowering to threshold=1 increases FPR to > 10%); Legitimate DNS (8.4%)—34 from domains with proper MX/SPF/DMARC (compromised accounts or sophisticated adversaries); No URLs (4.4%)—18 text-only with phone numbers; Multiple Factors (3.4%)—14 combining 3+ limiting factors.

24