LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests Justice Owusu Agyemang1,2,3∗, Jerry John Kponyo3†, Elliot Amponsah3‡, Godfred Manu Addo Boakye3§, Kwame Opuni-Boachie Obour Agyekum2¶ 1
Sperix Labs
2
VIA Cybersecurity Lab, KNUST
3
Quantum and Assistive Technologies Lab, KNUST
arXiv:2604.12064v1 [cs.CR] 13 Apr 2026
April 15, 2026
Abstract Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed. Existing privacy tooling focuses on network-level encryption and organization-level DLP, neither of which addresses the content of prompts themselves. We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests: (A) local-only inference, (B) redaction with placeholder restoration, (C) semantic rephrasing, (D) Trusted Execution Environment hosted inference, (E) split inference, (F) fully homomorphic encryption, (G) secret sharing via multi-party computation, and (H) differential-privacy noise. We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API. We evaluate the four practical options (A, B, C, H) and their combinations across four workload classes using a ground-truth-labelled leak benchmark of 1,300 samples with 4,014 annotations. Our headline finding is that no single technique dominates: the combination A+B+C (route locally when possible, redact and rephrase the rest) achieves 0.6% combined leak on PII and 31.3% on proprietary code, with zero exact leaks on PII across 500 samples. We present a decision rule that selects the appropriate option(s) from a threat-model budget and workload characterisation. Code, benchmarks, and evaluation harness are released at https://github.com/jayluxferro/llm-redactor.
1
Introduction
Large Language Model APIs have become infrastructure for developer workflows. Coding agents, writing assistants, customer support bots, and internal research tools send millions of prompts per day to cloud LLM vendors. These prompts routinely contain user-identifying information, organisational context, proprietary code, and occasionally credentials or secrets. Once sent, those prompts may be logged for debugging, retained for policy compliance, used for model improvement, or produced in response to legal process. Existing privacy tooling addresses this only at the margins. Network-level encryption (TLS) protects content from passive observers but not from the vendor. Organisation-level DLP focuses on data at rest and in version control, not on the content of transient API requests. Secret scanners catch known patterns at commit time but cannot inspect live request traffic. What is missing is a systematic framework and reference implementation that operates in the LLM request pipeline itself. This paper surveys eight distinct techniques for privacy-preserving LLM requests, implements them in an open-source shim, and evaluates them on a common benchmark. Our contributions are: ∗
[email protected], [email protected] [email protected] ‡ [email protected] § [email protected] ¶ [email protected] †
1
• A taxonomy of eight techniques organised by their privacy property, utility cost, and practicality today (§4). • A concrete threat model specifying what each technique defends against (§2). • A reference implementation that speaks both MCP and the OpenAI-compatible HTTP surface, with every option independently togglable (§5). • A ground-truth-labelled benchmark of 1,300 prompts across four workload classes (PII-heavy prose, secret-heavy configuration, implicit-identity prose, proprietary code) with 4,014 annotated sensitive spans (§6). • Empirical leak rates, latency, and cost for each technique and combination (§7). • A decision rule for practitioners, selecting the appropriate option(s) from a threat-model budget and workload characterisation (§8). We discuss limitations in §9, including detector quality bounds, synthetic workloads, and the absence of online utility evaluation.
2
Background and Threat Model
2.1
Actors and trust boundaries
We model six actors. The user (developer) and their local coding agent are trusted; they share a laptop inside the trust boundary. llm-redactor itself runs locally and is trusted as the enforcement point. A local model (Ollama, llama.cpp) is trusted for detection, classification, and rephrasing tasks. The cloud LLM vendor (OpenAI, Anthropic, etc.) is untrusted for privacy purposes: we assume it is curious but not actively malicious—it logs requests, may retain them for training or debugging, and may be subject to subpoena, but does not selectively target our user. The cloud infrastructure provider (AWS, GCP, Azure) is similarly untrusted. A passive network observer is defeated by TLS, which is table stakes and out of scope.
2.2
Assets
We aim to protect five categories of content in outbound prompts: (1) user-identifying information (names, emails, phone numbers, addresses, employee IDs, device IDs); (2) organisation-identifying information (company names, team names, internal project codenames, customer names); (3) secrets (API keys, bearer tokens, PEM keys, passwords, SSH keys); (4) proprietary code and prose; and (5) behavioural metadata (what the user is asking, about which projects). We explicitly do not attempt to protect request timing, volume, model selection, or the fact that a request was made. We also do not protect against out-of-band context: if the vendor already knows the user’s employer via billing, redacting the company name in the prompt does not hide that fact.
2.3
Attack scenarios
We define six concrete scenarios against which we evaluate each option. S1 (vendor log exfiltration): an insider or subpoena gains access to prompt logs. S2 (training contamination): prompts are used for model training and later regurgitated. S3 (third-party telemetry): a bundled SDK ships prompt metadata to an observability vendor. S4 (timing side channel): response-time correlation de-anonymises users. S5 (placeholder leakage): typed placeholders reveal structural information. S6 (adversarial input): obfuscated PII evades the detector. Table 1 summarises which options defend against which scenarios.
2
Table 1: Defence coverage per option per attack scenario. ✓ = defended, ∼ = partial, × = not defended.
3
Option
S1
S2
S3
S4
S5
S6
A Local B Redact C Rephrase D TEE E Split F FHE G MPC H DP noise
✓ ∼ ∼ ✓ ∼ ✓ ✓ ∼
✓ ∼ ∼ ✓ ∼ ✓ ✓ ∼
✓ ∼ ∼ × × × × ∼
✓ × × × × × × ×
n/a ∼ n/a n/a n/a n/a n/a n/a
✓ × ∼ ✓ ∼ ✓ ✓ ∼
Related Work
PII detection. Presidio [15] is the de facto open-source PII detection framework, combining rule-based recognisers with spaCy NER. Commercial alternatives include AWS Comprehend, Google DLP API, and Azure Purview. All share a fundamental limitation: they operate on surface-level patterns and miss implicit identity (“the CFO whose wife works at the competitor”). Homomorphic encryption for ML. CryptoNets [9] demonstrated neural-network inference on encrypted data. Intel’s HE-Transformer [5] and Zama’s Concrete ML [24] have improved usability, but FHE inference remains 10,000–100,000× slower than plaintext for models above 100M parameters. Multi-party computation. CrypTen [13] provides a PyTorch-compatible MPC framework. SecureML [17] and MP-SPDZ [12] offer lower-level protocols. MPC inference incurs 2–3 orders of magnitude overhead and requires non-colluding servers. Split inference. SplitNN [10] partitions a model across trust boundaries. Petals [6] operationalises this for collaborative LLM inference. The key risk is activation inversion: intermediate activations can sometimes be decoded back to input tokens. TEE-based inference. Graviton [22] and Slalom [21] pioneered TEE-hosted neural networks. Apple’s Private Cloud Compute [4], Azure Confidential Computing [16], and NVIDIA’s H100 Confidential Compute [18] are the most mature current offerings. TEEs protect against co-tenants and unprivileged operators but not against hardware-level side channels or supply-chain compromise. Differential privacy for language. DP-SGD [1] provides formal training-time guarantees. Inference-time DP for prompts is less studied; the closest work applies calibrated word-level noise [8], which we adopt in Option H. Surveys. Recent surveys [7, 23] catalogue LLM privacy risks and mitigations but do not provide a common-benchmark empirical comparison. Our contribution is exactly that comparison. Positioning. Table 2 summarises how our work relates to existing systems. To our knowledge, no prior work evaluates all eight technique classes on a common benchmark with ground-truth leak rates.
4
The Eight Options
We organise the eight options by their privacy property, utility cost, and practicality. Table 3 summarises the comparison. 3
Table 2: Comparison with existing privacy tooling for LLM requests. System
Techniques covered
Common bench.
Leak rates
Combinations evaluated
Presidio [15] AWS Comprehend CrypTen [13] Concrete ML [24] Petals [6] Apple PCC [4] [8] This work
B (redact) B (redact) G (MPC) F (FHE) E (split) D (TEE) H (DP) A–H (all)
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✗ ✗ partial ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓
Table 3: Option comparison matrix.
4.1
Option
Privacy
Utility
Cost
Latency
Practical?
A Local B Redact C Rephrase D TEE E Split F FHE G MPC H DP noise
perfect partial partial strong partial math. math. formal
bounded high medium full full full full lossy
$0 low low high medium prohib. high low
low low medium low medium v.high high low
yes yes yes limited research research research niche
Option A — Local-only inference
Local inference provides perfect privacy by construction: nothing leaves the device. Our implementation uses a few-shot classifier (running on a local 3B model via Ollama [19]) to triage requests as trivial or complex. Trivial requests are answered locally; complex requests proceed to the cloud pipeline. The classifier uses the same architecture as the sibling local-splitter project’s T1 stage. The privacy guarantee is binary: if classified locally, leak rate is zero. The cost is bounded model quality—a 3B local model cannot match frontier cloud models on complex tasks.
4.2
Option B — Redaction with placeholder restoration
A local NER detector (Presidio [15] with spaCy [11] en_core_web_sm) plus regex pattern families identifies sensitive spans. Each span is replaced with a typed, stable placeholder (⟨EMAIL_1⟩). Two occurrences of the same value map to the same placeholder (coreference stability). The reverse map lives only in process memory—never persisted to disk. On the response path, placeholders are restored by exact string match. The privacy guarantee is exactly the detector’s recall: if recall is 95% on emails, 5% of emails leak. Our evaluation quantifies this per annotation kind.
4.3
Option C — Semantic rephrasing
A local model rewrites the prompt to remove identifying details while preserving the technical question. A validator checks that key technical terms survive the rewrite; if survival rate drops below 70%, the rephrase is rejected and the pipeline falls back to B-only output. Option C targets implicit identity—phrases like “the CFO of Acme Corp whose wife works at the competitor”—which have no PII-span-level markers and cannot be caught by B’s detectors.
4
4.4
Option D — TEE-hosted inference
The client sends the plaintext request to an inference endpoint running inside a Trusted Execution Environment. Before sending, the client verifies the enclave’s attestation document (a hardwaresigned measurement of the enclave’s code and configuration). We target AWS Nitro Enclaves for the paper’s demonstration. TEEs protect against co-tenants and non-privileged cloud operators but not against a compromised hardware manufacturer or side-channel attacks. The utility cost is zero: the same model runs inside the enclave.
4.5
Option E — Split inference
The first N layers of an open-weight model run locally; only intermediate activations (not tokens) are sent to a remote host. We use Petals [6] as the reference framework. The privacy risk is activation inversion [14]: research shows activations can sometimes be decoded back to the input.
4.6
Option F — Fully homomorphic encryption
The input is encrypted with a homomorphic scheme; inference runs on ciphertext. We demonstrate a small binary classifier (sensitive vs. non-sensitive) under Zama’s Concrete ML [24]. FHE inference of full LLMs remains 10,000–100,000× slower than plaintext and is not practical today for chat-scale models.
4.7
Option G — Multi-party computation
The input is secret-shared across N non-colluding servers. We demonstrate a first-layer MPC embedding lookup using CrypTen [13]. This hides token IDs from any single server but does not protect the remaining layers, which run on plaintext activations.
4.8
Option H — Differential privacy noise
Calibrated word-level noise (substitution with semantically similar alternatives) blurs residual signal. The substitution probability is p(ε) = 1/(1 + eε ). At ε = 4 (our default), ≈1.8% of eligible words are substituted per request. H is most useful as a last-line-of-defence complement to B, adding noise to content that B’s detectors could not catch.
5
System Design
llm-redactor is a single-process shim with two transport interfaces and a pipeline of independently togglable stages. Transport layer. Two parallel interfaces serve different integration patterns: (1) an MCP [3] stdio server exposing redact.transform, redact.detect, and redact.stats tools; and (2) an HTTP proxy at POST /v1/chat/completions (OpenAI-compatible [20]). Agents point OPENAI_API_BASE at the proxy and all cloud calls transparently flow through the redactor. Pipeline. The pipeline proceeds through six stages: Stage 0 (Option A): classify as local-answerable or cloud-required. Stage 1 : detect sensitive spans via regex patterns and Presidio NER. Stage 2 (Option B): replace detected spans with typed placeholders; build in-memory reverse map. Stage 3 (Option C): rephrase via local model; validate technical-term survival. Stage 4 (Option H): inject calibrated DP noise. Stage 5 : route to the cloud target (standard API, TEE endpoint, split-inference host, or FHE/MPC endpoint). Stage 6 : restore placeholders in the response via exact match against the reverse map.
5
Each stage is independently enabled via a YAML configuration file. The reverse map is perrequest, lives only in process memory, and is discarded after restoration. A crashed process loses the map; the response cannot be de-redacted. This is the correct failure mode: on-disk persistence would be a worse leakage channel than the one we prevent. Detector design. Detection uses three complementary strategies: (1) regex patterns for structured secrets (AWS keys, bearer tokens, PEM markers, API keys, emails, IPs, phone numbers); (2) Presidio [15] with spaCy [11] en_core_web_sm for NER-based PII (person names, locations, organisations); and (3) a local LM classifier for semantic sensitivity questions that surface-level patterns miss. Detectors emit Span(start, end, kind, confidence, text, source) records. Overlapping spans are deduplicated, keeping the highest-confidence match. In strict mode, lowconfidence detections (< 0.5) cause the pipeline to refuse the request rather than silently pass it through. Placeholder design. Placeholders use rare Unicode angle brackets (⟨KIND_N⟩) to avoid collision with user text that might contain literal {EMAIL_1} syntax. Two references to the same original value receive the same placeholder (coreference stability), preserving the cloud model’s ability to reason about co-references.
6
Evaluation Setup
6.1
Workloads
We construct four synthetic workload classes, each with ground-truth annotations identifying sensitive spans. All identifiers are fabricated; no real user data is used. • WL1 (PII-heavy prose, 500 samples, 1,946 annotations): natural-language documents with embedded names, emails, phone numbers, addresses, employee IDs, and SSNs. Generated from 18 templates with random PII from a fixed-seed corpus. • WL2 (Secret-heavy configuration, 300 samples, 730 annotations): .env files, YAML configs, Docker Compose files, Terraform variables, and code snippets containing API keys, AWS credentials, bearer tokens, passwords, and PEM markers. Generated from 14 templates. • WL3 (Implicit identity, 200 samples, 220 annotations): prose that identifies individuals or organisations without using PII-span-level markers (“the CFO of Acme Corp whose wife works at the competitor”). Generated from 21 templates. Annotation kind is implicit. • WL4 (Proprietary code, 300 samples, 1,118 annotations): Python, Go, SQL, GraphQL, Dockerfile, and log snippets containing internal function names, database schemas, project codenames, and embedded credentials. Generated from 11 templates. Total: 1,300 samples, 4,014 ground-truth annotations. All workloads are deterministically reproducible from a fixed random seed.
6.2
Metrics
Privacy metrics. Exact leak rate: fraction of ground-truth annotations whose text appears verbatim in the outgoing (cloud-bound) request. Partial leak rate: fraction whose text has a ≥4-char substring match in the outgoing request (excluding exact leaks). Combined leak rate: exact + partial. Utility metrics. False positive rate: fraction of detector-flagged spans that do not correspond to any ground-truth annotation (over-redaction). Quality delta is measured via judge-model A/B comparison in online mode (not reported in this offline evaluation). 6
Cost metrics.
6.3
Latency: per-sample pipeline overhead (median and p95).
Configurations evaluated
We evaluate all eight options and key combinations: Baseline (no redaction), A (local routing), B (redact, regex only), B-NER (redact with Presidio NER), B+C (redact + rephrase), B+H (redact + DP noise at ε ∈ {2, 4, 8}), D (TEE, wire-level measurement), E (split inference stub), F (FHE classifier stub), G (MPC embedding stub). The practical options (A, B, B+C, B+H) are run on all four workloads with full samples (1,300 total); the research-stage options (D–G) are run on all workloads to measure wire-level leak properties and latency, with F and G limited to 20 samples per workload due to simulated computation time.
7
Results
7.1
Detector precision and recall
The regex-only detector achieves zero exact leak rate on emails, IPs, AWS access keys, PEM markers, and bearer tokens, but misses all person names, organisation names, addresses, SSNs, employee IDs, hostnames, and passwords—kinds that lack rigid syntactic patterns. Adding Presidio NER closes the gap substantially on person names (leak rate 0.123) and organisation names (0.259), but employee IDs (0.798) and addresses (0.060) remain partially exposed.
7.2
Per-option leak rates
Table 4 and Figure 1 present the exact leak rate for each option–workload pair.
Residual Exact Leak Rate per Option per Workload Baseline B (NER) B+C A A+B A+B+C
1.0
Exact Leak Rate
0.8 0.6 0.4 0.2 0.0
WL1 PII
WL2 Secrets
Workload
WL3 Implicit
WL4 Code
Figure 1: Residual exact leak rate per option per workload. B+C achieves the lowest leak rate across all workloads. Table 5 extends the comparison to include the research-stage options. Options E, F, and G achieve 0% token-level leak by construction (tokens never leave the device in plaintext), while Option D shows 100% wire-level leak (the TEE sees plaintext, but hardware attestation provides the privacy guarantee). Key observations. (1) No single option dominates across all workloads. (2) A+B+C is the strongest practical combination, achieving 0.6% combined leak on WL1 (PII), 6.4% on WL2
7
Table 4: Exact leak rate per option per workload (lower is better). Best result per workload in bold. WL1 PII WL2 Secrets WL3 Implicit WL4 Code †
Base
A
B
B+C
B+H
B+D†
A+B
A+B+C
1.000 1.000 1.000 1.000
.063 .242 .468 .599
.070 .162 .114 .428
.068 .214 .164 .301
.103 .244 .105 .423
.070 .162 .114 .428
.000 .042 .064 .285
.000 .036 .000 .123
B+D provides additional TEE hardware attestation. All B variants use NER.
Table 5: Combined leak rate across all options and workloads. Options D–G measure wire-level token exposure, not effective privacy. † = privacy from hardware/crypto, not from redaction. Baseline B (regex) B (NER) B+C B+D (NER)† B+H (ε = 4) A A+B A+B+C D (TEE)† E (split)† F (FHE)† G (MPC)†
WL1
WL2
WL3
WL4
1.000 0.812 0.153 0.139 0.153 0.176 0.063 0.012 0.006 1.000 0.000 0.000 0.000
1.000 0.521 0.318 0.316 0.318 0.303 0.242 0.063 0.064 1.000 0.000 0.000 0.000
1.000 1.000 0.950 0.941 0.950 0.950 0.468 0.445 0.436 1.000 0.000 0.000 0.000
1.000 0.905 0.585 0.558 0.585 0.517 0.599 0.381 0.313 1.000 0.000 0.000 0.000
(secrets), and 31.3% on WL4 (code). On WL1, A+B+C achieves zero exact leaks across all 500 samples. (3) Option A alone provides 6.3% leak on WL1 via local routing of 94% of requests, but 59.9% on WL4 where only 38% of requests are locally answerable. (4) A+B (without rephrase) achieves 1.2% on WL1 and 38.1% on WL4—already a major improvement over B alone (15.3%, 58.5%) because locally-routed requests contribute zero leaks. (5) Option B+H provides marginal improvement over B alone at ε = 4, because the word-level noise mechanism cannot reliably substitute the specific words that constitute the residual leaks. (6) WL3 (implicit identity) remains the hardest workload: even A+B+C achieves only 43.6% combined leak, because implicit identity survives both redaction and rephrasing (§7, semantic leak analysis).
7.3
Option A routing analysis
Table 6 shows the fraction of requests routed locally by the T1 classifier. Table 6: Option A local routing rates. Workload WL1 PII WL2 Secrets WL3 Implicit WL4 Code
Local
Cloud
% Local
472 224 108 114
28 76 92 186
94.4% 74.7% 54.0% 38.0%
The classifier’s routing accuracy varies sharply by workload. PII-heavy prose (WL1) is overwhelmingly classified as trivial because the templates use simple natural-language structures. Code requests (WL4) are more nuanced and require cloud-level reasoning; only 38% can be served locally. For the requests that are routed locally, the leak rate is exactly zero by construction.
8
7.4
Leak rate by annotation kind
Table 7 breaks down Option B’s (with NER) exact leak rate by annotation kind on WL1, revealing which categories the detector handles well and which it misses. Table 7: Option B exact leak rate by annotation kind (WL1). Kind
Leak Rate
email phone ip_address ssn hostname employee_id address person org_name
0.000 0.000 0.000 0.000 0.000 0.000 0.060 0.123 0.259
Option B (NER) Leak Rate by Annotation Kind (WL1) table_name database_name internal_function aws_secret_key project_name org_name org_name api_key org_name password person bearer_token person implicit address ssn phone pem_private_key ip_address ip_address ip_address hostname hostname hostname employee_id email email email aws_access_key api_key 0.0
0.2
0.4
0.6
Exact Leak Rate
0.8
1.0
Figure 2: Option B leak rate by annotation kind (WL1). Green = fully detected; orange = partially detected; red = mostly missed. Figure 2 visualises this breakdown. Structured patterns (emails, phones, IPs, SSNs) are fully detected. NER catches most person names but struggles with organisation names (especially when they resemble common nouns) and employee IDs (a custom format not in Presidio’s default recognisers).
7.5
Combinations and the Pareto frontier
Figure 3 plots exact leak rate vs. latency for each configuration. The key finding is that B+C Pareto-dominates all other combinations on WL1 and WL3: it achieves the lowest leak rate at moderate latency (∼1,000 ms median, driven by the local rephrasing model). B+H adds negligible privacy improvement over B alone but costs no additional latency (the DP noise computation is sub-millisecond). The practical recommendation is a four-tier strategy:
9
Privacy-Latency Pareto Frontier (WL1 PII)
0.5
Exact Leak Rate
0.4 0.3 0.2 B+H e=4.0
0.1
A A+B+C(cloud A+B(cloud
0.0 0
200
400
600
Median Latency (ms)
800
B+C
1000
Figure 3: Privacy–latency Pareto frontier on WL1. B+C achieves the lowest leak rate at higher latency; B+H is fastest with minimal privacy gain over B alone. 1. Route locally (Option A) whenever the request is locally answerable. This eliminates cloud exposure entirely for the majority of PII-prose workloads. 2. Redact + rephrase (B+C) for complex requests that must go to the cloud. This achieves ≤7% exact leak rate on PII and ≤30% on proprietary code. 3. TEE (Option D) for content with implicit identity (§7, semantic leak analysis), where B+C’s ≥95% semantic leak rate is unacceptable. TEE provides hardware-level protection without utility loss. 4. Refuse when no option meets the threat-model budget and the content contains implicit identity that cannot be structurally removed.
7.6
Latency overhead
Table 8: Median pipeline latency per option (ms). Option A and B+H include NER; B+C includes NER + Ollama rephrasing on a 3B model. Option
WL1
WL2
WL3
WL4
B (NER) B+D (NER) B+H (ε = 4) B+C A
17.4 17.7 17.9 1,855 1,781
18.6 19.4 21.9 2,528 2,604
13.7 14.9 15.2 1,662 1,680
28.3 28.9 35.0 2,293 2,243
Table 8 reports the median latency per option. Option B and B+H add <50 ms overhead (dominated by NER initialisation amortised across requests). Option B+C adds ∼1–2 seconds per request due to the local Ollama model call for rephrasing—acceptable for privacy-first workloads but not for latency-sensitive interactive use. Option A adds ∼300 ms for the classifier call (one Ollama round-trip with a 3B model).
10
7.7
Token cost
Redaction changes the token count of outgoing requests. Somewhat counter-intuitively, Option B reduces token count because typed placeholders (⟨EMAIL_1⟩) are shorter than the values they replace (email addresses, API keys, full names). Table 9 reports the word-level token delta. Table 9: Token count change from Option B (NER) redaction. Negative = fewer tokens sent to the cloud. Workload
Original
Outgoing
∆
8,834 4,238 4,193 9,914
7,769 3,987 3,905 9,511
−12.0% −4.5% −7.1% −4.0%
WL1 PII WL2 Secrets WL3 Implicit WL4 Code
Redaction thus provides a modest cost saving (−4 to −12% fewer tokens per request), partially offsetting the latency overhead. DP noise (Option H) performs 1:1 word substitution and does not change the token count.
7.8
Utility evaluation
We measure the quality cost of redaction using a judge-model A/B comparison. For each sample, a local LLM (Qwen 3.5 4B) generates responses to both the original and redacted prompts; a judge model then selects which response better addresses the user’s question. To control for family bias, we run a cross-family variant where Qwen generates and Llama 3.2 3B judges. Table 10: Utility evaluation: judge preference for baseline (unredacted) vs. Option B (NER) responses. Workload
Judge
WL1 PII WL2 Secrets WL1 PII
Qwen (same) Qwen (same) Llama (cross)
Baseline pref.
Redacted pref.
n
78% 80% 75%
22% 20% 20%
50 50 20
Table 10 shows that redaction incurs a modest quality cost: the judge prefers the baseline response ∼75–80% of the time. The cross-family judge (Llama judging Qwen-generated text) produces similar results (75% vs. 78%), suggesting minimal family bias at this scale. The quality loss is expected: redacted prompts replace identifying details with placeholders, removing context that can help the model produce more specific answers. For privacy-first workloads, this trade-off is acceptable.
7.9
Semantic leak analysis (WL3)
The substring-based leak metrics (exact and partial) are ill-suited for WL3’s implicit-identity annotations, where the identifying information is carried by context rather than by specific token spans. We therefore introduce a semantic leak metric: a local judge model (Llama 3.2 3B via Ollama) reads each sample’s ground-truth annotations and the redacted outgoing text, then answers whether the redacted text still identifies the same individual or organisation. Table 11 reports semantic leak rates on a 20-sample subset of WL3. All three configurations fail: the semantic leak rate is ≥95%. This is the paper’s strongest negative result. Implicit identity—conveyed by role descriptions, relationship structures, and organisational context—survives both span-level redaction and local-model rephrasing. The rephrase model preserves the structural relationships (“a senior executive whose spouse works at a competitor”) because those relationships are the content of the prompt; stripping them would make the prompt useless.
11
Table 11: Semantic leak rate on WL3 (implicit identity). A local judge model determines whether the redacted text still identifies the target. Higher = worse. Option
Semantic Leak Rate
Baseline (no redaction) B (regex only) B (NER) B+C (rephrase)
0.95 0.95 0.95 1.00
This result demonstrates a fundamental limit of content-level transformations (Options B, C, H): they can remove or blur tokens, but they cannot remove meaning without destroying utility. Addressing implicit identity requires either (a) never sending the content to the cloud (Option A), (b) sending it to a trusted enclave (Option D), or (c) accepting the residual risk with informed consent.
7.10
Research-stage demonstrations
Option D (TEE). We implement a client-side attestation verifier for AWS Nitro Enclaves [2] that checks the attestation document’s structure, PCR values, and certificate chain before sending plaintext. A full end-to-end demo (vLLM inside a Nitro Enclave serving Llama-3) requires dedicated AWS infrastructure and is left as a deployment exercise. The attestation protocol adds <100 ms to the request path. Option E (split inference). We implement a protocol stub that simulates splitting a model at layer 4: the client computes random “activations” and POSTs them to a remote endpoint. In a real Petals deployment, the activations would be the output of the first 4 transformer layers. The privacy risk is activation inversion [14]; recent work shows that early-layer activations can be partially decoded, especially for short sequences. Option F (FHE classifier). We demonstrate a simulated FHE binary classifier (sensitive/nonsensitive) with realistic timing: ∼100 ms encryption, ∼5,000 ms homomorphic inference, ∼50 ms decryption. A real Concrete ML implementation would replace the simulation with actual TFHE circuits. FHE inference of a full chat model remains impractical at current performance levels. Option G (MPC embedding). We demonstrate a simulated first-layer MPC embedding lookup: token IDs are additively secret-shared across N parties, each party looks up its share of the embedding, and shares are reconstructed. Setup takes ∼200 ms; per-token compute takes ∼50 ms. A real CrypTen implementation would provide cryptographic security guarantees.
8
Discussion
8.1
Decision rule
Given a threat-model budget (maximum acceptable exact leak rate λ) and a workload characterisation, we propose the following decision rule: 1. If λ = 0 (zero tolerance): use Option A (local-only) for all requests that the local model can handle. For the remainder, use Option D (TEE) if infrastructure permits, or refuse the request. 2. If λ ≤ 0.05: use A + B + C. Route locally when possible; redact and rephrase the rest. This achieves ≤2% on PII and ≤0.5% on implicit identity. 3. If λ ≤ 0.25: use B alone (with NER). This is the cheapest option that provides meaningful protection, achieving 10.5% on PII and 24.4% on secrets. 12
4. If latency is the primary constraint: use B (NER) at <50 ms overhead. Avoid C (adds ∼1 s) unless implicit identity is a concern.
8.2
Failure modes per technique
Option B. The detector’s recall is the ceiling. After adding a custom EMP-\d{4,6} recogniser, employee IDs drop from 79.8% to 0% leak rate. Organisation names remain the hardest category at 25.9% when they resemble common nouns. This demonstrates that domain-specific regex patterns provide high-leverage improvements with minimal effort. Option C. The local model occasionally hallucinates new details or strips load-bearing technical context. Our validator rejected 33/200 rephrases on WL3 (16.5% rollback rate). On WL1, only 7/500 were rejected (1.4%), indicating that simpler prose is easier to rephrase correctly. Option H. Table 12 shows the sensitivity of B+H to the privacy parameter ε. At ε = 8 (low noise), B+H is virtually identical to B alone. At ε = 4, NER-level performance is achieved with sub-millisecond overhead. At ε = 2 (high noise), exact leak rates drop further (0.493 on WL1 without NER vs. 0.600 for B alone) but partial leak rates increase due to word-fragment preservation. DP noise is better suited to statistical workloads where per-token fidelity is not critical. Table 12: B+H exact leak rate sensitivity to ε (WL1).
8.3
ε
Exact Leak
Partial Leak
Combined
2.0 4.0 8.0
0.493 0.103 0.599
0.308 0.073 0.213
0.801 0.176 0.812
The practical–cryptographic gap
Options A, B, C, and H are deployable today on commodity hardware. Options D, E, F, and G require specialised infrastructure or remain research-stage. The gap is closing: TEE availability is expanding (Nitro, Azure CC, H100 CC), FHE performance improves ∼2×/year, and MPC frameworks are becoming more usable. We estimate that practical FHE inference for 7B models is 5–10 years away; TEE-hosted inference is available now for organisations willing to manage the infrastructure.
9
Limitations • Detector quality bounds Options B and C. We use off-the-shelf tooling (Presidio, spaCy en_core_web_sm, regex patterns) rather than a custom-trained detector. A domain-specific NER model would likely improve recall on organisation names and employee IDs. • Workloads are synthetic. All 1,300 samples are template-generated with fabricated identifiers. Real-world prompts exhibit greater diversity and would likely reveal additional detector blind spots. • Research-stage options are demonstrated, not deployed. Options E, F, and G are protocol stubs with simulated timing. Production measurements would require dedicated infrastructure. • Judge-model quality evaluation is preliminary. Our utility comparison uses a local 4B model (Qwen 3.5) for both generation and judging on a limited sample (n = 50). A larger-scale evaluation with frontier models and cross-family judging would strengthen the quality delta measurements. 13
• Partial leak metric is noisy for implicit identity. The 4-char substring match produces high partial-leak rates on WL3 because implicit annotations share common words with the rephrased text. Semantic leak rate (judge-model based) is the appropriate metric for WL3. • Non-English content. Our detector and NER model target English. Multilingual support would require additional spaCy models and locale-specific regex patterns.
10
Ethics and Responsible Disclosure
• No real user prompts are used. All workloads are synthetic with fabricated identifiers. • If we discover a vendor-specific bypass (e.g., a bundled telemetry SDK that evades our proxy), we will notify the vendor at least 30 days before publication and document the timeline. • Source, benchmarks, and evaluation harness are released under MIT at https://github.com/ jayluxferro/llm-redactor.
11
Conclusion
Practical privacy tooling for LLM requests spans eight options from “never leave the device” to “fully homomorphic inference.” Our common-benchmark evaluation of 1,300 samples across four workload classes shows that no single technique dominates. The combination A+B+C (route locally when possible, redact and rephrase the rest) is the strongest practical configuration, achieving 0.6% combined leak on PII with zero exact leaks across 500 samples. Implicit identity (WL3) remains the hardest category at 43.6%, demonstrating a fundamental limit of content-level transformations. For deployments where implicit identity is a concern, TEE-hosted inference (Option D) provides hardware-level protection without utility loss. The optimal strategy depends on the deployment’s threat-model budget: zero-tolerance deployments should combine local routing with TEE-hosted inference; most practical deployments will benefit from A+B+C (route locally when possible, redact and rephrase the rest). We release an open-source reference implementation and benchmark suite to enable reproduction and extension.
Data and Code Availability Source, workloads, ground-truth annotations, and evaluation harness are released at https:// github.com/jayluxferro/llm-redactor.
References [1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 308–318. ACM, 2016. [2] Amazon Web Services. AWS nitro enclaves, 2024. URL https://aws.amazon.com/ec2/nitro/ nitro-enclaves/. Accessed 2026-04-12. [3] Anthropic. Model context protocol specification, 2024. URL https://modelcontextprotocol. io. Accessed 2026-04-12. [4] Apple. Private cloud compute: A new frontier for AI privacy in the cloud, 2024. URL https://security.apple.com/blog/private-cloud-compute/. Accessed 2026-04-12. [5] Fabian Boemer, Yixing Lao, Rosario Cammarota, and Casimir Wierzynski. nGraph-HE: A graph compiler for deep learning on homomorphically encrypted data. In Proceedings of the 16th ACM International Conference on Computing Frontiers (CF), pages 3–13. ACM, 2019. 14
[6] Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL): System Demonstrations, pages 38–44, 2023. [7] Saptarshi Das, Anushka Dey, Arnab Pal, and Nupur Roy. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6), 2025. [8] Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. [9] Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pages 201–210. JMLR.org, 2016. [10] Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116:1–8, 2018. [11] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrialstrength natural language processing in Python. 2020. doi: 10.5281/zenodo.1212303. [12] Marcel Keller. MP-SPDZ: A versatile framework for multi-party computation. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1575–1590. ACM, 2020. [13] Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sheshadri, Zihang Zheng, et al. CrypTen: Secure multi-party computation meets machine learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021. [14] Oscar Li, Jiankai Sun, Xin Wang, Richard Gauch, Mudhakar Srivatsa, and Kuan He. Label leakage and protection in two-party split learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. [15] Microsoft. Presidio: Data protection and de-identification sdk, 2024. URL https://github. com/microsoft/presidio. Open-source framework for PII detection and anonymization. [16] Microsoft Azure. Azure confidential computing, 2024. URL https://azure.microsoft.com/ en-us/solutions/confidential-compute/. Accessed 2026-04-12. [17] Payman Mohassel and Yupeng Zhang. SecureML: A system for scalable privacy-preserving machine learning. In Proceedings of the IEEE Symposium on Security and Privacy (S&P), pages 19–38. IEEE, 2017. [18] NVIDIA. Confidential computing on H100 tensor core GPUs, 2023. URL https://www.nvidia. com/en-us/data-center/solutions/confidential-computing/. Accessed 2026-04-12. [19] Ollama. Ollama: Run large language models locally, 2024. URL https://ollama.com. Accessed 2026-04-12. [20] OpenAI. OpenAI API reference, 2024. api-reference. Accessed 2026-04-12.
URL https://platform.openai.com/docs/
[21] Florian Tramer and Dan Boneh. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
15
[22] Stavros Volos, Kapil Vaswani, and Rodrigo Bruno. Graviton: Trusted execution environments on GPUs. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 681–696. USENIX Association, 2018. [23] Duzhen Yao et al. A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2):100211, 2024. [24] Zama. Concrete ML: Privacy-preserving machine learning using fully homomorphic encryption, 2024. URL https://github.com/zama-ai/concrete-ml.
16