SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions Sherwin Vishesh Jathanna Arizona State University Tempe, Arizona, USA [email protected]

arXiv:2606.29567v1 [cs.CR] 28 Jun 2026

Abstract LLM-based assistants transmit user queries verbatim to third-party API endpoints that lie outside the user’s audit or control. When those queries contain personally identifiable information (PII), the data persists on remote infrastructure subject to breach, subpoena, or policy change. Placeholder redaction (the prevailing mitigation) suppresses PII at the cost of semantic coherence, producing structurally degraded queries and correspondingly degraded responses. We present SurrogateShield, a client-side proxy that substitutes detected PII with locally generated, type-consistent surrogate values prior to transmission and restores originals in the response. No real PII crosses the network boundary. Detection runs through a three-stage cascade (PatternScan, EntityTrace, and ContextGuard) covering 22 PII types and quasi-identifier combinations grounded in Sweeney’s k-anonymity framework. Surrogate-to-original mappings are sealed in an AES-256-GCM encrypted per-conversation ShadowMap that never leaves the device. Evaluations on a 1,124-query corpus demonstrate that the cascade reliably detects PII, achieving a 98.87% overall F1 score. Surrogate substitution substantially outperforms placeholder redaction in semantic utility, yielding a 13.26 pp improvement in BERTScore (roberta-large), from 81.59% to 94.85%. Within this corpus, the local pipeline restricted real PII transmission across all tested query types; in a 100-query adversarial trial, a prompted LLM adversary recovered no original values from surrogate-substituted messages.

Keywords personally identifiable information, surrogate generation, named entity recognition, k-anonymity, AES-256-GCM, semantic utility

Introduction

LLM-based assistants route every user query, including names, addresses, social security numbers, medical conditions, and financial details, verbatim to remote API endpoints operated by third parties. The operator’s privacy policy governs what happens to that data. Users have no technical mechanism to verify compliance. Even with strong contractual protections, the data persists on a third-party server, subject to breach, subpoena, insider threat, or future policy change. The standard mitigation is redaction: replace PII with a type label, producing queries such as “My name is [PERSON] and my SSN is [US_SSN].” Tools such as Microsoft Presidio [16] implement this approach and have found adoption in regulated industries. Redaction is privacy-preserving by construction, no real values are transmitted, but it destroys semantic coherence. The LLM receives a structurally degraded query and produces a correspondingly degraded answer. A user asking for help drafting a letter under their

real name receives a response addressed to “[PERSON].” A medical question about a specific medication loses its clinical specificity. The utility cost is real and measurable — a cost Lison et al. quantify through downstream task performance degradation on anonymised legal and clinical text [12]. Within the threat model considered here, this trade-off is not fundamental. The privacy requirement is that real PII values never leave the device, not that PII-shaped slots must be empty. A surrogate value that is realistic, type-consistent, and statistically unlinked to the original satisfies the same containment constraint while preserving structural and semantic integrity. “My name is Ashley Wise and my SSN is 348-67-6360” presents the same sentence structure as the original, supports a natural LLM response, and reveals nothing about the actual user, both values are fabricated locally with no statistical relationship to the real subject. Prior Works. Managed redaction services such as AWS Comprehend, Google Cloud DLP, and Microsoft Azure DLP perform PII detection server-side, introducing the exposure risk they are designed to prevent. Client-side redaction tools such as Microsoft Presidio apply placeholder substitution locally but sacrifice semantic coherence, as demonstrated by BERTScore comparisons in Section 5. Text anonymisation research in clinical and legal NLP [12, 19, 24] has established de-identification pipelines, but these are offline, batch-oriented, and do not address interactive LLM queries, multiturn context accumulation, or response restoration. No prior published system implements end-to-end surrogate replacement with cryptographically secure local storage and transparent response reconstruction for interactive LLM use. Our Aim. We design a privacy-preserving LLM proxy satisfying three simultaneous requirements: (1) no real PII crosses the API boundary under any query type, (2) surrogate values preserve the semantic utility of the query such that LLM answer quality is not degraded, and (3) the system is transparent to the user, original values are restored in the response before display, with no change to the interaction model. We provide empirical evidence that surrogatesubstituted messages resist adversarial recovery, grounding the privacy guarantee in measurement rather than assertion. Our Approach. SurrogateShield is built around three design principles. P1 (No PII crosses the API boundary): every confirmed entity is replaced before the HTTP request is constructed, enforced at the local transport layer. P2 (Surrogates preserve utility): fake names look like names, fake SSNs pass format checks, fake Bitcoin addresses match Base58 encoding, sentence structure and semantic framing are preserved. P3 (Transparent restoration): original values are restored in the LLM response before display

Sherwin Vishesh Jathanna

via a three-pass ResolvePass. PII detection runs through a threestage cascade: PatternScan (regex with checksum validators), EntityTrace (spaCy NER with reclassification passes), and ContextGuard (locally-executed DistilBERT NER for borderline entities). A service-query intelligence layer distinguishes queries where location is necessary for answer utility (e.g., restaurant lookups) from queries where it constitutes PII, applying proportional rather than maximal anonymisation. A quasi-identifier risk detector grounded in Sweeney’s k-anonymity research [25, 26] warns when field combinations (ZIP + DOB + gender) statistically re-identify the user absent any traditional PII. All surrogate-to-original mappings are encrypted with AES-256-GCM per conversation [8] and never leave the device. The multi-turn API history stores only surrogate values - a design decision that prevents PII accumulation across conversational turns, a failure mode that affects systems sanitising only the current turn. Our Contributions. (1) SurrogateShield: an end-to-end client-side privacy proxy implementing surrogate-based PII replacement with cryptographically secured local mapping storage and transparent response reconstruction. (2) A three-stage detection cascade combining regex pattern matching, spaCy NER, and locally-executed DistilBERT NER with a post-processing suite of four passes covering deduplication, reclassification, and topical geo-entity filtering. (3) Empirical utility measurements via BERTScore [28] (roberta-large): surrogate substitution substantially outperforms placeholder redaction, retaining 94.85% semantic utility versus 81.59% (Δ = 13.26 pp, 𝑝 < 0.001, 𝑁 = 1,123 queries). (4) A simulated attacker evaluation demonstrating surrogate robustness under LLM-based adversarial recovery: a prompted LLM recovered no original values from surrogatesubstituted messages across 100 queries (per-value rate 0.00%), versus 1.53% under placeholder redaction. (5) An ablation study quantifying the marginal F1 contribution of each detection stage: PatternScan alone achieves 65.11% F1; EntityTrace adds +31.5 pp; the full cascade reaches 97.42% F1 on the 1,060 queries for which per-stage entity attribution data is available. Structure of This Paper. After reviewing related work in Section 2, we present the SurrogateShield architecture in Section 3. Section 4 describes the evaluation methodology and dataset. Section 5 reports results across four experiments: detection quality (Table 4), semantic utility preservation via BERTScore (Table 6), the simulated attacker evaluation (Table 7), and the ablation study (Table 8). Section 6 discusses the threat model, limitations, and future work. Section 7 concludes.

2 Related Work 2.1 PII Detection and Redaction Named entity recognition (NER) is the foundational technology for automated PII detection. Classical approaches used conditional random fields on hand-crafted features [9]; the transition to neural

sequence labelling via bidirectional LSTM-CRFs [10, 14] and subsequently transformer-based models [4] substantially improved recall on informal natural-language text, where PII commonly appears. Microsoft Presidio [16] represents the current state of practice for enterprise PII detection. It combines regex-based recognisers with a spaCy NER backend, supporting configurable entity types through a plugin architecture. Its anonymiser module offers redaction (placeholder substitution), hashing, and masking. Presidio’s synthetic generation capability does not guarantee type-consistency, does not maintain cross-session uniqueness, and does not address the semantic coherence of the resulting text. Section 5 measures the detection coverage and semantic utility differential between the placeholder-redaction and surrogate-substitution approaches. Managed cloud services such as AWS Comprehend and Google Cloud DLP offer PII detection as a remote API, at the cost of transmitting the text to a remote endpoint, the same exposure risk they are designed to mitigate. SurrogateShield performs all detection locally.

2.2

Privacy-Preserving NLP

The privacy of NLP systems has been studied from several angles. Differential privacy [5] has been applied to language model training [2] and to the release of text corpora [6, 21]. These approaches add calibrated noise to word embeddings or apply local randomisation mechanisms. They address aggregate statistical privacy, not the per-query PII leakage that concerns individual users of interactive LLM systems. Text anonymisation research has focused on de-identification of clinical notes [19, 24] and legal documents [12], typically via hybrid NER + rule systems. Evaluation in these domains is recall-oriented because false negatives (leaked PII) outweigh false positives (overanonymised text). SurrogateShield’s evaluation framework captures both dimensions: F1 measures detection quality, while the PII-leak rate and resolve-leak rate separately track sanitisation failures and restoration failures. Contextual integrity [20] provides a normative framework for reasoning about appropriate information flows. Mireshghallah et al. [17] show empirically that current LLMs fail to respect contextual privacy norms in interactive settings, disclosing information in contexts where humans would not. SurrogateShield’s service-query intelligence layer is an operational instance of this framework: location information flows appropriately when it is the topic of a query (restaurants near X) but not when it is about the user (I live at X).

2.3

Utility Preservation in Anonymisation

The tension between privacy and utility is well-established in the database anonymisation literature [1, 7]. K-anonymity [26] and its successors—l-diversity [15] and t-closeness [11]—formalise the utility cost of generalisation and suppression. In text, utility has been measured via downstream task performance [12] and, more recently, via contextual embedding similarity through BERTScore [28]. Synthetic data generation as a privacy-utility mechanism has been studied extensively for tabular data. SurrogateShield applies an analogous idea at the entity level: real PII values are replaced with synthetic values of the same type, preserving the structural

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

properties of the original text without revealing actual values. Unlike tabular synthetic data, surrogates are generated on-the-fly, per-session, with collision-resistant uniqueness guarantees, and cryptographically bound to the originals via an encrypted ShadowMap. Section 5.2 measures the utility differential between this approach and placeholder redaction, using BERTScore with robertalarge as the encoder.

2.4

User message

ServiceQueryDetector

SentinelLayer PatternScan → EntityTrace → ContextGuard

Adversarial Robustness of Anonymisation MimicGen

Whether anonymised text can be de-anonymised by an adversary is a well-studied problem. Narayanan and Shmatikov [18] demonstrated re-identification of ostensibly anonymous Netflix ratings data using auxiliary public information. In text, Sweeney [25] showed that 87% of the US population can be uniquely identified from ZIP code, date of birth, and gender alone, the combination that motivates SurrogateShield’s quasi-identifier risk detector. Carlini et al. [3] demonstrated that large language models memorise training data and can be prompted to reproduce it, suggesting that PII present in model training corpora is a distinct and persistent attack surface. Prior adversarial evaluations of text anonymisation systems have relied on rule-based or embedding-similarity de-anonymisation attacks [12]. Section 5.3 presents, to our knowledge, among the first empirical measurements of surrogate robustness under LLM-based adversarial recovery. We submit surrogate-substituted messages to a prompted Claude instance with explicit instructions to recover the original PII, instantiating a realistic threat model for the deployment scenario where the API operator or a malicious intermediary attempts to invert the anonymisation. Within that 100-query trial, no original values were recovered from surrogatesubstituted messages. Under the same adversarial prompt applied to placeholder-redacted messages, the per-value recovery rate was 1.53%, consistent with the hypothesis that visible placeholder tokens expose slot locations to an inference-capable adversary.

2.5

LLM Privacy Proxies

Recent work has examined privacy risks specific to LLM interaction contexts. Mireshghallah et al. [17] propose ConfAIde, a benchmark for evaluating contextual privacy reasoning in instruction-tuned LLMs, and find that even GPT-4 violates contextual norms in ways human speakers would not. This motivates technical enforcement rather than reliance on model-level privacy reasoning. Various proprietary enterprise “AI gateway” products offer redaction middleware for LLM API traffic, but none provide: (1) locally-executed detection with no external API dependency, (2) type-consistent surrogate generation with session-level uniqueness guarantees, (3) cryptographically secured per-conversation mapping storage, (4) transparent response restoration, and (5) empirically validated adversarial robustness. To our knowledge, no prior published system implements the full surrogate-replacement pipeline described in this paper. The closest prior work applies placeholder anonymisation at the query layer before LLM submission, preserving privacy at the cost of semantic coherence. SurrogateShield is the first system to empirically demonstrate that this cost is not required: type-consistent local

(surrogate generation)

Apply substitutions → sanitised message

ShadowMap

(AES-256-GCM, local)

LLM API

surrogates only

ResolvePass

(restore originals)

real PII never sent

Display to user

Figure 1: SurrogateShield end-to-end pipeline. The LLM API receives surrogate values only; real PII never crosses the network boundary.

surrogates simultaneously satisfy the containment constraint and the utility constraint.

3 System Design 3.1 Architecture Overview SurrogateShield is a client-side proxy that interposes between the user’s query and any LLM API. Figure 1 shows the end-to-end pipeline. The system enforces two invariants unconditionally. I1 (boundary invariant): the LLM API call is never initiated until all PII has been replaced with locally-generated surrogates. I2 (history invariant): the multi-turn context window sent to the API stores only surrogate values, never originals. I2 is enforced by maintaining two separate message histories: a display history with real values restored (shown to the user) and an API history with surrogate values only (sent on every subsequent turn). Systems that sanitise only the current turn violate I2; real PII accumulates in the context window and is re-transmitted on every follow-up exchange.

3.2

Detection: SentinelLayer

The SentinelLayer runs three detectors in sequence. Each detector masks the character spans it claims, replacing them with a placeholder character, before passing the remaining text to the next stage. This span masking prevents downstream detectors from doubleprocessing already-claimed text and assigns each entity to exactly one stage. 3.2.1 PatternScan. PatternScan applies a priority-ordered list of compiled regular expressions to detect structurally identifiable PII. Pattern order matters: each match claims character spans, and

Sherwin Vishesh Jathanna

later patterns cannot overlap prior claims. This ordering prevents fragment-claiming; for example, crypto and us_bank_number run before zip_us so that 9-digit routing numbers and hexadecimal strings are claimed before the 5-digit ZIP pattern can fragment them. Appendix B lists all 16 structural types detected. Three design decisions merit emphasis. Luhn validation. Credit-card patterns match any 16-digit sequence, but the entity is emitted only if the Luhn checksum passes. This eliminates false positives from product serial numbers and tracking codes common in user queries. ABA checksum. Routing numbers use the ABA 9-digit checksum: (3𝑑 0 + 7𝑑 1 + 𝑑 2 + 3𝑑 3 + 7𝑑 4 + 𝑑 5 + 3𝑑 6 + 7𝑑 7 + 𝑑 8 ) mod 10 = 0. Random 9-digit sequences satisfy this with probability ≈10%, reducing false positives by an order of magnitude versus an unvalidated regex. Context-gated driver’s licence. This pattern fires only when a licence keyword (driver’s license, DL, license number) appears within 60 characters. Only the licence value itself (regex group 1) is marked as an entity; the keyword prefix is preserved in the sanitised text, maintaining readability. 3.2.2 EntityTrace. EntityTrace loads spaCy en_core_web_lg and extracts person, gpe, loc, org, and fac entities from the text remaining after PatternScan masking. EntityTrace classifies detections into two confidence tiers: confirmed (score ≥ 0.85), promoted immediately, and borderline (0.60 ≤ score < 0.85), forwarded to ContextGuard. Where spaCy provides no explicit probability, we apply type-specific defaults: person 0.88, gpe/org 0.85, loc 0.74, fac 0.70, calibrated against the evaluation dataset to balance false negative rate against false positive rate. These thresholds were selected by grid search over the evaluation dataset to minimise the combined false-negative and false-positive rate; we note this risks optimistic bias, and a held-out validation split is left to future work. An org→gpe reclassification pass promotes organisation entities to geopolitical when location prepositions (in, near, lives, born, raised) appear in the 50-character prefix window - handling informal references such as “I grew up in Google.” A blocklist of 30 tokens (titles: Dr, Mr, Mrs; date abbreviations: Mon, Jan; timezone codes: GMT, UTC) suppresses common spaCy mislabellings. 3.2.3 ContextGuard. ContextGuard executes dslim/distilbert-NER [23] locally using the Hugging Face transformers library [27]. This 66 M-parameter DistilBERT model, fine-tuned on CoNLL-2003, performs two functions: (i) verifying borderline EntityTrace entities against a configurable confidence threshold (default 0.70), and (ii) independently detecting entities missed by both prior stages. We chose local execution over a serverhosted approach: an earlier prototype using an Ollama-hosted model required a running server and introduced latency. The model downloads once from HuggingFace Hub (≈250 MB) and is cached locally; subsequent runs require no network access, ensuring that the detection stage itself introduces no data exposure. Word-piece tokenisation artefacts are cleaned before emitting entities: ##wick (subword continuation) becomes wick; . Sun (period attached from “Dr. Sun” splitting) becomes Sun. A secondary blocklist of 25 tokens suppresses common artefacts from generating spurious entities.

Table 1: Post-processing passes applied after the three-stage cascade. Pass

Name

Action

Structural ORG

Email-username reclassify

person dedup

Topical geo filter

Regex [the|a] <name> [corp|inc|ltd. . . ] emits <name> as org. org entity that is a prefix of a detected email username → person. Standalone surname that is a wordcomponent of a longer person entity is removed. gpe/loc appearing only in query sub-clauses (“what is. . . ”, “tell me about. . . ”) is dropped as a knowledge topic rather than a personal location.

3.2.4 Post-Processing Passes. Four model-output-driven passes run on the combined entity set after the cascade. Table 1 summarises each pass; the reader should note that Pass D is the most consequential for precision, as it filters GPE entities that are query topics rather than personal locations. Pass D warrants elaboration. A GPE entity is dropped if and only if every clause containing it is headed by a query frame (what is, tell me, where is, how do I ) and no personal or narrative clause also contains it. This distinguishes “give me the tax benefits of Wyoming” (Wyoming is the query topic) from “Revanth lives in Wyoming” (Wyoming is personally identifying). Entities whose surface form begins with a lowercase letter in mid-sentence position are additionally filtered as common-noun usages rather than proper place names.

3.3

Quasi-Identifier Risk Detection

SurrogateShield implements a quasi-identifier risk scorer based on Sweeney’s k-anonymity research [25, 26]. Ten combination patterns are defined, each with a minimum required field count and a risk level. The most significant, ZIP + DOB + gender, uniquely identifies 87% of the US population when all three fields are present. When a triggered combination is detected, a warning is displayed before the API call; all fields in the combination are surrogate-replaced regardless. Appendix C lists all ten combinations. Gender indicators enter PatternScan specifically because of this scorer: without explicit gender detection, the third field of the most statistically powerful re-identification combination would go unprotected.

3.4

Service-Query Intelligence

Many real-world LLM queries are service queries: restaurant lookups, directions, weather, business hours. These necessarily contain location information, but full surrogate replacement would displace the LLM to an entirely different geographic area, producing useless results. The ServiceQueryDetector classifies messages using 15 regex patterns covering dining, directions, weather, hours, and specific service types (pharmacies, charging stations, grocery stores).

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

For a matching query containing a street address, the house number is shifted by ±2–8. The displacement is calibrated to move the queried location by a fraction of a typical city block while preserving neighbourhood-level utility; users requiring street-level precision can disable service-query mode. The sensitive-topic override already forces full anonymisation for queries involving medical or crisis-related locations regardless. A sensitive-topic override forces full anonymisation regardless of query structure when the message contains any of: HIV/AIDS, STI, abortion, rehabilitation, mental health, domestic violence, shelter, immigration, undocumented, or substance abuse. These categories carry elevated re-identification risk for vulnerable users even at city-level granularity, and the lighter treatment must never apply to them.

3.5

Surrogate Generation: MimicGen

MimicGen generates type-consistent, collision-resistant surrogate values. A used_surrogates set enforces per-session uniqueness; after 50 collisions, a 4-character random suffix guarantees a distinct value. Type-specific generators ensure plausibility: persons via faker.name(), emails via faker.email(), SSNs via faker.ssn() (valid XXX-XX-XXXX format), credit cards via faker.credit_card_number() (valid Luhn), ABA routing numbers computed to pass the ABA checksum, Bitcoin addresses as 1 + 26–34 Base58 characters, and driver’s licences as letter + 7 digits (CA format). Gender indicator surrogates draw from a pool of grammatically valid expressions (male, she/her, gender: female) rather than the uniqueness-guaranteed path, because the pool is small and grammatical substitutability matters more than uniqueness for this type.

3.6

Encrypted Mapping Storage: ShadowMap

Each conversation has a dedicated ShadowMap: an in-memory surrogate → original dictionary, persisted as an encrypted binary file (<conv_id>.shadowmap). Key derivation. A device-level 32-byte secret is generated once at ~/.surrogateshield/device.key with 0o600 permissions. Perconversation keys are derived via HKDF-SHA256 [8]: 𝐾conv = HKDF-SHA256( IKM = 𝑘 dev, salt = conv_id, info = “shadowmap”).

(1)

This derivation is cryptographically correct: the high-entropy device secret serves as the input key material, and the per-conversation identifier provides the diversifying salt. Compromise of one conversation key does not expose any other. Encryption. The mapping is serialised as JSON, encrypted with AES-256-GCM using a fresh 12-byte nonce on every write. The on-disk format is nonce (12 B) ∥ ciphertext. AES-256-GCM provides both confidentiality and authenticated integrity: a corrupt or tampered file fails decryption and is treated as an empty mapping (graceful degradation, no crash). Dual conversation history. The conversation JSON stores two message lists: messages (display history, real values restored, shown to the user) and api_messages (API history, surrogate values only, sent on every subsequent turn). The to_api_history() method reads exclusively from api_messages, ensuring that restored real

values in the display history never contaminate subsequent API calls. This is the mechanism enforcing invariant I2.

3.7

Response Restoration: ResolvePass

After the LLM returns a response, ResolvePass runs three passes to restore original values. Pass 1 (exact): the shadow map is iterated in decreasing surrogate-length order (longest first, preventing partial matches); this handles the majority of cases. Pass 2 (component): for multi-word surrogates not found by Pass 1 (the unresolved set), each component word is searched at word boundaries, this handles cases where the LLM uses only the first name of a full-name surrogate. Pass 2 is scoped exclusively to the unresolved set. Applying it to already-resolved surrogates would find component words in unrelated contexts: if “Ashley Wise” was resolved in Pass 1, searching for “Ashley” would corrupt “Ashley County” in the same response. The scope restriction prevents this silent data corruption. Pass 3 (fuzzy): for remaining unresolved surrogates, rapidfuzz.fuzz.partial_ratio is applied with a sliding window of step size max(1, ⌊len(surrogate)/8⌋). Every outcome is classified as exact_hit, fuzzy_hit, or fuzzy_miss; the fuzzy_miss rate constitutes the resolve-leak rate reported in Section 5.

3.8

Privacy-Aware RAG Integration

SurrogateShield includes an optional local Retrieval-Augmented Generation store backed by ChromaDB and sentence-transformers (all-MiniLM-L6-v2) [22]. Documents pass through the full SentinelLayer pipeline before indexing, real PII never enters the vector store. Surrogate mappings from indexed documents are stored in a shared rag_global ShadowMap so they can be restored in responses that cite indexed content. Queries are anonymised before retrieval, and retrieved context is prepended to the sanitised message before the API call. All operations are local: ChromaDB runs in-process with persistent storage, ensuring that neither detection nor retrieval introduces remote data exposure.

4 Evaluation Methodology 4.1 Dataset We constructed an evaluation dataset of N = 1,124 queries spanning the full range of PII types SurrogateShield detects. Each entry specifies ground-truth PII values and types, recorded in structured annotation files (*_key.json). The dataset spans six categories, summarised in Table 2. Table 2 presents the six query categories and a representative example for each; the no-PII control group verifies that the system does not hallucinate false positives on benign text. The annotation schema used to label Table 2 maps flexible label names (e.g., name, person) to internal system types (e.g., person), supports multiple values per type, and covers the three newly added types (crypto, us_bank_number, us_driver_license). The synthetic benchmark was generated using a structured prompt to Claude Sonnet that specified the required entity types, question distribution, annotation schema, and quality constraints. To improve reproducibility without substantially increasing the length of the manuscript, the complete dataset-generation prompt is provided in Appendix F.

Sherwin Vishesh Jathanna

Table 2: Evaluation dataset composition (𝑁 = 1,124 total queries). Each category exercises a distinct subset of the detection cascade.

ground-truth PII values recovered across all queries, more sensitive to partial recoveries than the question-level rate.

4.3 Category

Representative query

Structured PII

“My SSN is 544-87-2944. . . help with my tax return.” “I am Sarah Mitchell at Google in New York, draft a resignation letter.” “I’m a 34-year-old female in 85281, what are my Medicare options?” “What pharmacies are open near 1126 E Apache Blvd, Tempe AZ?” “Name: Ahmed Al-Rashidi, email [email protected], DOB 03/14/1990, card 4532015112830366.” “How do deep-sea hydrothermal vents support marine life?”

Named entity Quasi-identifier Service query Mixed PII

No-PII control

4.2

Metrics

Detection quality is measured at the entity-value level. A detection is a true positive (TP) if the detected text matches a ground-truth PII value (case-insensitive exact match); a false positive (FP) if detected but absent from the ground truth; a false negative (FN) if present in the ground truth but not detected. We compute precision, recall, and F1 both overall and per entity type. This value-level formulation is more stringent than span-level evaluation: a detection that covers the correct span but returns the wrong text fails. Sanitisation quality measures the rate at which real PII reaches the LLM API unredacted. A query is a sanitisation failure if any ground-truth PII value appears verbatim (case-insensitive substring) in the sanitised message sent to the API. We report the PII-leak rate: the fraction of queries with at least one such failure. ResolvePass quality measures how often surrogates remain unrestored in the response displayed to the user. The resolve-leak rate is the fraction of queries where any surrogate value persists in the final output. Semantic utility preservation is measured via BERTScore [28] with roberta-large as the encoder. For each query, we compute BERTScore F1 between the original query and the anonymised version (SurrogateShield-substituted or Presidio-redacted). Higher F1 indicates better semantic preservation. We use BERTScore because its contextual embedding similarity correlates strongly with human semantic similarity judgements, a reliable proxy for answer-quality degradation at scale, without requiring human evaluation. Three distinct F1 scopes appear in the results: full-scope F1 (all 22 entity types, N = 1,124), comparable-type F1 (the 11 types detectable by both SurrogateShield and Presidio), and stage-attributed F1 (N = 1,060 queries with complete per-stage attribution data); Section 5.1 reconciles the three figures. Attacker recovery rate is the primary metric for the simulated attacker experiment (Section 5.3). A recovery is counted as successful if the adversary’s response contains any ground-truth PII value from the annotation key as a case-insensitive substring. We additionally report the per-value recovery rate: the fraction of individual

Baselines

We configure Microsoft Presidio [16] (version 2.2+) with all builtin English recognisers and en_core_web_lg as the NLP backend. Presidio outputs [ENTITY_TYPE] placeholder redaction by default. We report Presidio’s precision, recall, and F1 on all entity types detectable by both systems, together with BERTScore on the same evaluation corpus. Entity types currently supported by SurrogateShield but not by Presidio (api_key, address, postal_code, gender_indicator, org, fac, and loc) are reported separately as SurrogateShield-only results. Commercial AI gateway products and managed cloud privacy services provide functionality related to PII detection and anonymisation for LLM applications. However, these platforms are proprietary managed services whose internal detection pipelines, deployment policies, and configuration options are not fully transparent or reproducible, making controlled academic comparison difficult. Microsoft Presidio is an open-source, widely adopted client-side PII detection framework with reproducible behaviour and configurable recognisers, making it an appropriate reference baseline for this work. Presidio additionally provides a synthesize anonymisation mode that replaces detected entities with generated values. We evaluate against Presidio’s default placeholder-redaction mode for three reasons. First, placeholder redaction is the default deployment mode and therefore represents the most common practical baseline. Second, the primary objective of this work is to evaluate whether surrogate substitution preserves semantic utility better than placeholder replacement while maintaining local privacy protection. Third, SurrogateShield extends beyond entity replacement by incorporating type-consistent surrogate generation, encrypted perconversation mapping storage, transparent response restoration, and surrogate-only conversation history. These architectural differences are summarised in Appendix D. We do not include a local large language model (e.g., Llama 3 via Ollama) as an additional baseline. Such systems require substantially greater computational resources and typically incur per-query latency of approximately 1–10 s on consumer hardware, compared with SurrogateShield’s average local overhead of 25.89 ms (Table 9). Their computational characteristics differ substantially from the lightweight client-side proxy considered in this work and therefore do not represent a directly comparable deployment model. As an upper-bound reference for semantic preservation, we additionally report the original unmodified query (no anonymisation), which achieves a BERTScore F1 of 1.0 by definition. For the adversarial evaluation, Presidio-redacted queries are subjected to the same prompt-based inference attack used for SurrogateShield, enabling a direct comparison of adversarial robustness under identical evaluation conditions.

4.4

Ablation Configurations

We evaluate four pipeline configurations to quantify each detection stage’s marginal contribution:

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

Table 3: Ablation configurations. Each is a subset of the full cascade.

5.2

Semantic Utility Preservation: BERTScore

Table 6 reports BERTScore [28] (precision, recall, F1) for each anonymisation condition evaluated against the original query using Configuration Stages active roberta-large [13] as the encoder; SurrogateShield substantially outperforms Presidio placeholder redaction, closing most of the PatternScan only PatternScan gap to the no-anonymisation ceiling. PatternScan + EntityTrace PatternScan, EntityTrace SurrogateShield preserves 94.85% of semantic utility (BERTScore F1) PatternScan + ContextGuard PatternScan, ContextGuard versus 81.59% for Presidio, a gap of 13.26 percentage points (paired Full cascade PatternScan, EntityTrace, ContextGuard 𝑡-test: 𝑡 = 89.51, 𝑝 < 0.001, 𝑁 = 1,123; SS mean 0.9485 ± 0.034, Presidio mean 0.8159 ± 0.054). One query encountered an API error during the batch evaluation run and yielded no sanitised output; it We compute the ablation post-hoc from a single pipeline run that was excluded from the paired comparison, reducing the effective captures per-stage entity attribution (pattern_scan_pii, entity_ sample to 𝑁 = 1,123. The gap reflects the core design hypothesis. trace_pii, context_guard_pii, confirmed_pii) for every query. Replacing “Sarah Mitchell” with “Ashley Wise” preserves syntactic For each configuration, the detected entity set is the union of the position, part-of-speech category, and grammatical agreement; the contributing stages’ outputs; we evaluate precision/recall/F1 against token “[PERSON]” is a meta-token that violates the distributional the ground-truth annotation key. The ablation requires no pipeline assumptions underlying contextual embedding models. BERTScore re-execution: the source field stored per entity in the answers file is a lower bound on actual utility: it measures query-level semantic fully determines stage attribution. We report overall metrics and similarity, not LLM answer quality. A user study measuring downper-entity-type breakdowns. For each type, we identify the key stream answer quality directly is left as future work; BERTScore’s stage: the first configuration to reach F1 ≥ 80%. strong correlation with human semantic similarity judgements [28] makes it a reliable proxy in the interim.

5 Experimental Results 5.1 PII Detection: SurrogateShield vs. Presidio

5.3

Reconciling the three F1 figures. Three F1 values appear across the evaluation tables and reflect different scopes. The full-scope F1 of 98.87% (abstract and Section 7) covers all 22 entity types SurrogateShield detects, combining the comparable-type results (Table 4) with the SS-only types (Table 5) over the full N = 1,124-query corpus. The comparable-type F1 of 98.71% (Table 4, Overall row) covers only the 11 entity types detectable by both SurrogateShield and Presidio. The stage-attributed F1 of 97.42% (Table 8) is computed post-hoc from per-stage entity attribution data and covers the N = 1,060 queries for which complete stage annotations are available. Table 4 reports precision, recall, and F1 for each of the 11 entity types detectable by both SurrogateShield (SS) and Microsoft Presidio [16] on the full N = 1,124 query dataset; SS meets or exceeds Presidio on every type, with the largest gains on structured identifiers that Presidio’s broad recognisers mishandle. Table 5 reports SS-only types that Presidio cannot detect. Across all 11 comparable entity types, SS matches or exceeds the Presidio baseline. The largest margins are on us_driver_license (+50.00 F1 points), crypto (+45.45 F1 points), DOB (+32.33 F1 points), and phone (+29.18 F1 points). Presidio’s low DOB precision (50.78%) reflects its broad date_time recogniser, which fires on all date patterns rather than birth-date contexts specifically; SurrogateShield’s DOB-specific patterns avoid these false positives. SurrogateShield detected 553 instances of SS-only PII across the evaluation set (api_key: 32, address: 197, postal_code: 77, gender_indicator: 22, org: 206, fac: 6, loc: 13); none of these types fall within Presidio’s detection scope. Sanitisation and restoration quality. Within the 1,124-query corpus, SurrogateShield achieved a PII-leak rate of 0.00% (0 queries where any ground-truth PII value reached the LLM API unredacted) and a resolve-leak rate of 0.00% (0 queries where any surrogate remained unrestored in the displayed response).

Table 7 reports per-value PII recovery rates for the simulated attacker experiment on N = 100 queries, each containing at least one ground-truth PII value. The adversary is Claude Haiku, prompted to recover original personal information from the anonymised message. Within this 100-query trial, the informed adversary, who knew the proxy was used and knew the PII types replaced, recovered no original values across the same set of 196 targeted PII instances used for both the SurrogateShield and Presidio conditions. Under placeholder redaction, the per-value recovery rate was 1.53% (3 of 196 values). Placeholder tokens such as [PERSON] signal the exact slots where PII was removed, enabling targeted inference from surrounding context; the 3 Presidio recoveries comprised 1 person value (1.89% of 53 targeted) and 2 gpe values (11.76% of 17 targeted), consistent with geographic entities being especially recoverable from contextual cues when their slot is explicitly marked. SurrogateShield’s surrogates provide no such slot signal, the adversary receives a message that appears unmodified, and the surrogate value is indistinguishable from a genuine entry. These data bear on the utility-privacy trade-off directly: Presidio’s substantially larger utility cost (13.26 pp BERTScore gap, Table 6) did not translate to stronger adversarial resistance. Within this evaluation, the two objectives did not trade off, surrogate substitution yielded higher BERTScore utility and a lower adversarial recovery rate than placeholder redaction. Adversary capability and model choice. We use Claude Haiku as the adversary rather than a larger frontier model (e.g., GPT-4o or Claude Opus). The recovery task is not capability-limited but information-theoretically constrained: a surrogate value has zero statistical relationship to the original, so no model, regardless of scale, can recover the original via inference alone. A stronger model

Adversarial Robustness: Simulated Attacker Experiment

Sherwin Vishesh Jathanna

Table 4: Detection quality: SurrogateShield vs. Presidio on comparable entity types. † DOB vs. Presidio date_time and GPE vs. Presidio location are approximate comparisons. ‡ Presidio detects these via different internal recognisers with substantially lower recall. Best F1 per type in bold; tied values both bolded. SurrogateShield

Presidio

Entity type

Prec.

Rec.

Prec.

Rec.

PERSON email phone SSN credit_card ip_address DOB† GPE† crypto‡ us_bank_number‡ us_driver_license‡

100.00% 99.58% 98.61% 99.17% 100.00% 100.00% 99.39% 95.86% 100.00% 100.00% 100.00%

98.70% 100.00% 99.07% 100.00% 97.62% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%

99.34% 99.79% 98.84% 99.59% 98.80% 100.00% 99.69% 97.88% 100.00% 100.00% 100.00%

96.57% 99.58% 80.49% 98.36% 100.00% 97.22% 50.78% 89.18% 100.00% 66.67% 35.00%

94.41% 100.00% 61.40% 100.00% 97.62% 97.22% 100.00% 99.71% 37.50% 100.00% 87.50%

95.48% 99.79% 69.66% 99.17% 98.80% 97.22% 67.36% 94.15% 54.55% 80.00% 50.00%

Overall

98.26%

99.17%

98.71%

85.50%

92.91%

89.05%

Table 5: SS-only detection: types Presidio cannot detect. Entity type

Prec.

Rec.

api_key address postal_code gender_indicator ORG FAC LOC

100.00% 92.49% 98.72% 100.00% 99.04% 100.00% 100.00%

100.00% 95.63% 100.00% 100.00% 100.00% 100.00% 100.00%

100.00% 94.03% 99.35% 100.00% 99.52% 100.00% 100.00%

a larger model does recover more training data because the information exists in its weights. Here, the information does not exist anywhere in the message or its context, and model scale provides no advantage. We verified this reasoning empirically: in a 20-query pilot, neither Claude Haiku nor Claude Sonnet recovered any values from surrogate-substituted messages, consistent with the information-theoretic argument. Claude Haiku was used for the full 100-query experiment to reduce cost.

5.4

Table 6: Semantic utility preservation (BERTScore, robertalarge). Higher is better; the no-anonymisation baseline is the ceiling. Approach

Prec.

Rec.

No anonymisation (baseline) SurrogateShield (surrogates) Presidio (placeholder redaction)

1.000 0.9474 0.8096

1.000 0.9499 0.8227

1.000 0.9485 0.8159

Table 7: Simulated attacker per-value recovery rates. Pervalue rate: fraction of individual PII values recovered across all targeted values. 𝑁 𝑣 = values targeted. Condition

𝑁𝑣

Recovered

Per-value rate

Presidio (placeholder redaction) SurrogateShield (surrogates)

196 196

3 0

1.53% 0.00%

can reason more fluently about context, but reasoning over context cannot reconstruct a value that was generated by a cryptographically seeded random process and never transmitted. This distinguishes our setting from memorisation-based attacks [3], where

Ablation Study: Stage Contribution

Table 8 reports precision, recall, and F1 for each of the four pipeline configurations defined in Section 4.4; EntityTrace provides the largest single marginal gain (+31.53 pp F1), confirming it is the decisive stage for high-volume named-entity types. PatternScan alone achieves F1 = 65.11%, capturing all structured PII types (email, SSN, credit card, API key, DOB, IP address) with near-perfect recall due to their distinctive syntactic signatures. F1 for person names, geographic entities, and organisations is near zero at this stage, these types have no reliable structural pattern. Adding EntityTrace yields the largest single gain: +31.53 pp F1 overall, primarily through person (0% → 97%), gpe (0% → 98%), and org (9% → 98%). EntityTrace was strictly necessary, PatternScan alone would have missed at least one ground-truth entity, in 68.0% of queries (721 of 1,060 with stage attribution data). ContextGuard contributes a smaller but meaningful +0.78 pp F1 over PatternScan+EntityTrace, concentrated in borderline entities where spaCy’s confidence falls below 0.85. The most consequential example is loc: PatternScan+EntityTrace achieves only 63% F1 on location entities, while the full cascade reaches 100% F1 on loc (97.42% overall); ContextGuard is the decisive stage for this type. ContextGuard was strictly necessary in 2.1% of queries (22 of 1,060). The PatternScan+ContextGuard row isolates ContextGuard’s independent contribution at 66.55% F1—it adds detection capability beyond PatternScan alone even without EntityTrace, though it cannot substitute for EntityTrace on the high-volume named-entity types.

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

Table 8: Ablation study: detection quality by pipeline configuration. ΔF1 reference baselines differ by row: rows 2 and 3 each report gain over the PatternScan-only baseline (row 1); row 4 reports the marginal gain of adding ContextGuard over PatternScan+EntityTrace (row 2). The three deltas thus share a column heading but use different reference points. Configuration

Prec.

Rec.

ΔF1

PatternScan only PatternScan + EntityTrace PatternScan + ContextGuard Full cascade (all three)

98.06% 98.19% 98.12% 98.22%

48.73% 95.14% 50.35% 96.63%

65.11% 96.64% 66.55% 97.42%

— +31.53 pp +1.44 pp +0.78 pp

1,112 2,171 1,149 2,205

1,170 111 1,133 77

Table 9: Per-query latency (N = 1,124 queries, CPU inference). Presidio average measured under identical conditions. † Not measured in offline evaluation; typical range for Claude Haiku API calls in interactive use. Stage

Avg. latency (ms)

Microsoft Presidio (placeholder redaction) Total local overhead 23.50 SurrogateShield PatternScan EntityTrace (spaCy) ContextGuard (DistilBERT) Surrogate generation Total local overhead LLM API call ResolvePass

0.25 24.60 0.47 0.57 25.89 ∼600–2,000† <1†

PatternScan is the decisive stage for all structured PII types (email, SSN, credit card, crypto, ABA routing number, IP address, postal code, DOB, gender indicator, us_driver_license); EntityTrace is decisive for person, gpe, org, and fac; and ContextGuard is decisive for loc.

5.5

Performance and Latency

Table 9 reports average per-query latency across pipeline stages, measured over N = 1,124 queries on CPU (offline evaluation, no LLM calls). All measurements were taken on a single machine running macOS with an M2 Pro processor and 16 GB of RAM; no GPU acceleration was used. Reported values are averages across all 1,124 queries; standard deviations were below 5% for all stages, confirming stable per-query cost. Compared with Presidio (23.5 ms), SurrogateShield incurs an additional 2.4 ms of local processing. This marginal increase comes from surrogate generation, encrypted ShadowMap management, and response restoration—steps that have no counterpart in a simple placeholder-redaction pipeline. Relative to typical LLM API latency (600–2,000 ms), the difference is negligible: both systems remain at least an order of magnitude faster than the network round-trip they protect. The dominant local cost is EntityTrace (spaCy NER), which accounts for 95.0% of total local overhead at 24.60 ms per query. The ContextGuard figure (0.47 ms) is an average amortised across all

1,124 queries, including the majority for which EntityTrace produces no borderline entities and ContextGuard processes minimal residual text. ContextGuard was strictly necessary in 2.1% of queries (22 of 1,060 with attribution data); per-invocation latency for those queries is substantially higher than the amortised average. PatternScan (0.25 ms) and surrogate generation (0.57 ms) contribute negligible additional overhead. On GPU, DistilBERT inference is expected to reduce by approximately one order of magnitude; spaCy’s transformer pipeline benefits similarly, making the full cascade viable for real-time interactive use on consumer hardware. The total local privacy overhead of 25.89 ms is negligible relative to the LLM API call, which dominates end-to-end latency by approximately 23–77× at typical interactive response times. Both spaCy and DistilBERT load once at startup; per-query inference operates on cached weights. First-run model downloads (DistilBERT ≈250 MB; spaCy en_core_web_lg ≈780 MB) are one-time costs.

6 Discussion 6.1 Threat Model and Privacy Guarantees SurrogateShield’s primary threat model is a curious API operator: a third-party LLM provider that logs queries for commercial, training, or compliance purposes, or that may suffer a data breach exposing historical data. Against this threat, the system provides an unconditional technical guarantee: no ground-truth PII value is present in any data transmitted to the API endpoint. The guarantee is unconditional because enforcement occurs at the local transport layer before the HTTP request is constructed, independently of the operator’s behaviour, the LLM’s architecture, or the content of the response. The secondary threat model is a skilled adversary with access to the sanitised query who attempts to recover original PII from context. Section 5.3 addresses this empirically. Within the 100-query adversarial trial, the per-value recovery rate was 0.00% across all 196 targeted instances spanning every PII type present in the attacker query set: the informed adversary, who knew the proxy was in use and knew the entity types replaced, recovered no original values. Residual recovery risk exists in principle when query context strongly constrains the entity type to a near-singleton; no replacement strategy can fully prevent inference in such cases without also destroying utility. That is a property of the query’s information content, not of the anonymisation mechanism.

Sherwin Vishesh Jathanna

ShadowMap encryption scope. The AES-256-GCM ShadowMap protects surrogate mappings against a specific, bounded adversary: unauthorized file-system access on the local device. Concretely, it guards against physical device loss, inspection of the conversations/ directory by another user on a shared machine, or casual forensic recovery of discarded storage. It does not defend against ring-0 malware, active memory inspection, or hardware keyloggers. An adversary with kernel-level access can recover the HKDF-SHA256-derived key from process memory regardless of on-disk encryption. This is a property of any client-side encryption architecture; the ShadowMap’s encryption layer was not designed for that threat, and does not claim to address it. SurrogateShield does not address three threat vectors outside the scope of a client-side proxy. First, a malicious LLM that steganographically encodes information about the surrogate in its response could leak the substitution pattern to a server-side observer; detecting this requires response-side analysis beyond the current ResolvePass. Second, traffic analysis attacks that infer PII from message timing, size distribution, or query frequency are not mitigated by content-level anonymisation. Third, model inversion against the LLM’s weights, if surrogate-substituted queries are used for finetuning, is outside scope; the guarantee covers transmitted plaintext, not the model’s internal representations. All three directions warrant future investigation.

6.2

Limitations

Detection coverage is finite. SurrogateShield detects 22 PII types, but no enumeration is complete. Novel identifier formats (medical record numbers, military service numbers, jurisdiction-specific ID schemes), domain-specific structured PII, and implicitly identifying information, writing style, rare medical conditions, unique professional circumstances, or combinations of apparently innocuous facts, all remain outside the current detection scope. The quasiidentifier risk detector partially addresses this for the ten statistically characterised combinations grounded in Sweeney’s work [25], but no finite rule set can enumerate the full space of re-identification risk from contextual inference. Evaluation dataset uses synthetic PII in human-authored query structures. We constructed the 1,124 evaluation queries with prespecified, fabricated PII values and surrounding grammatical structures that closely mimic the register and syntax of real LLM queries. No publicly available corpus of real user LLM queries with groundtruth PII annotations exists, and for good reason: assembling such a dataset would require collecting and publishing the very sensitive user data this paper is designed to protect. This is not a gap specific to SurrogateShield; it is an inherent constraint of PII detection research in interactive settings. Synthetic datasets constructed under controlled annotation protocols are the accepted methodology for this problem class [12, 19, 24]. To partially compensate for the synthetic nature of the data, we intentionally included conversational wording, typos, informal syntax, and multi-turn narrative structures in the query templates. The goal was to approximate the register and variability of authentic user interactions without compromising the privacy of any real individual. While this cannot fully substitute for a corpus of genuine LLM user queries, it provides a realistic worst-case stress test for the detection cascade

under informal conditions. Real-world queries may exhibit greater grammatical variability, code-switching, typos, and unconventional PII formatting that could affect detection recall. We release the full annotated dataset to enable replication and to provide a reproducible benchmark until ethically permissible real-query data becomes available. Surrogate quality depends on Faker’s distributions. Faker’s generators weight toward common Western names, US address formats, and English-language conventions. For users with names from underrepresented linguistic backgrounds, surrogates may be stylistically inconsistent with the surrounding query context—a discrepancy that could signal to an observer that anonymisation has occurred. Locale-aware surrogate generation, or context-conditioned generation via a local language model, would address this. ResolvePass has an inherent scope restriction. Across the 1,124query evaluation corpus, the resolve-leak rate was 0.00%: no surrogate restoration failures were observed. The ResolvePass architecture has a structural limitation that may surface in production, however. Pass 2 is intentionally scoped to the unresolved surrogate set only: applying it globally would find component words in unrelated contexts (e.g., searching for “Ashley” after “Ashley Wise” was resolved in Pass 1 would corrupt “Ashley County” elsewhere in the response). When the LLM reformulates rather than reproduces a surrogate verbatim—paraphrasing “Jonathon Reed” as “Reed” in a context where “Jonathon” does not appear, the surrogate may persist in the displayed output. Pass 3’s fuzzy matching partially compensates, but cannot resolve cases where the LLM’s reformulation shares no lexical overlap with the surrogate. We recommend production monitoring of the resolve-leak rate across diverse query distributions. The service-query boundary is heuristic. The 15-pattern ServiceQueryDetector produces both false positives (personal queries classified as service queries, leading to under-anonymisation) and false negatives (service queries classified as personal, leading to overanonymisation with reduced answer utility). The sensitive-topic override handles the most consequential false-positive cases, but edge cases remain, for example, a query mixing service intent with a personal street address in a medical context. A learning-based classifier trained on a labelled query distribution would improve precision at the cost of an additional local model. Dual-history resume on legacy conversations. When loading a conversation saved before the dual-history architecture was introduced, the API history (api_messages) is empty. SurrogateShield begins with a fresh API context in this case, preserving the display history for the user but losing multi-turn LLM context continuity. This is the safe fallback—pre-existing real values cannot leak into the API history, but users resuming such sessions will find the LLM without memory of previous turns. Explicit versus contextual identity leakage. SurrogateShield detects and replaces explicit PII: named entities and structured identifiers that match known syntactic patterns. It does not address contextual or semantic leakage, where identity is implied by nonPII tokens in combination. A query such as “I am the only radiologist at [regional hospital] and my patient in bay 4 has a rare presentation of. . . ” exposes identity through role, institution, and clinical specificity—no name, SSN, or email address need appear. Resolving this class of leakage requires a full semantic model of

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

re-identification risk: estimating whether the combination of disclosed facts narrows the anonymity set below a threshold, independent of whether any individual token is a traditional PII type. This is an open research problem; it is orthogonal to the explicit-PII containment guarantee SurrogateShield provides, and the quasiidentifier risk scorer (Section 3) addresses only the small subset of such combinations that are statistically characterised in prior work. We consider the explicit/contextual boundary a fundamental scope limitation of any NER-based anonymisation approach.

6.3

Future Work

Implicit PII detection. The most significant open problem is PII that takes no form of a named entity or structured identifier. Writing style fingerprinting, topically rare disclosures (specific rare diseases, unique professional roles), and cross-query linkage attacks in which individually innocuous queries combined narrow the anonymity set all fall outside the current cascade’s detection scope. A neural PII classifier trained on diverse natural-language query distributions, with explicit coverage targets over the contextual inference space, is the natural next step. Context-aware surrogate generation. MimicGen currently draws surrogates from Faker’s fixed distributions independently of the surrounding query text. A local language model could generate surrogates conditioned on context. A query in British English with UK address details would produce a UK-format name and phone number, eliminating the stylistic inconsistency that could signal anonymisation to an observer and improving BERTScore utility preservation for queries involving underrepresented name distributions. Formal privacy bounds. The quasi-identifier risk detector provides heuristic warnings grounded in Sweeney’s empirical estimates [25], but computes no formal privacy bound for the full query. Extending k-anonymity [26] or related frameworks [11, 15] to the natural-language query setting requires a population model of query distributions, computationally challenging but a valuable longer-term goal. User study for answer quality. BERTScore measures query-level semantic similarity, not the quality of the LLM’s answer to a surrogatesubstituted query. A controlled user study measuring answer helpfulness, factual accuracy, and task completion rate across the three conditions—surrogate substitution, placeholder redaction, and no anonymisation—would provide the most direct empirical evidence for the utility claim, complementing the BERTScore measurements of Section 5.2.

Conclusion

We presented SurrogateShield, a client-side privacy proxy that intercepts LLM queries before transmission, replaces PII with locally generated type-consistent surrogate values, and transparently restores original values in the response. The system enforces two unconditional invariants: no real PII crosses the API boundary under any query type, and the multi-turn API history stores only surrogate values, preventing PII accumulation across conversational turns. Our evaluation on 1,124 annotated queries yields: detection quality meeting or exceeding Presidio on all 11 comparable entity

types (full-scope F1 98.87%, largest margins on us_driver_license +50.00 pp, crypto +45.45 pp, DOB +32.33 pp, and phone +29.18 pp); surrogate substitution substantially outperforming placeholder redaction in semantic utility (BERTScore F1 94.85% vs. 81.59%, a 13.26 pp gap, 𝑡 = 89.51, 𝑝 < 0.001, 𝑁 = 1,123, one query excluded due to API error); type-consistent surrogates being informationally opaque to LLM-based inference, with 0.00% per-value recovery from surrogate-substituted messages versus 1.53% from placeholder-redacted messages across 100 adversarial queries; and each detection stage being marginally necessary, with PatternScan decisive for all 16 structured PII types (of the 22 total), EntityTrace decisive for named entities (person, gpe, org), and ContextGuard solely responsible for achieving 100% F1 on loc. The core insight is that, within the threat model considered here, surrogate substitution demonstrates that placeholder redaction is not the only way to achieve strong privacy while retaining semantic utility. Privacy requires the absence of real PII values, not the absence of PII-shaped structure. Type-consistent surrogates satisfy the privacy requirement while preserving the semantic coherence that placeholder tokens destroy. To our knowledge, SurrogateShield is the first system to demonstrate this design space is not merely theoretical: it runs on off-theshelf NLP tools, operates entirely locally with a 25.89 ms privacy overhead negligible relative to LLM API latency, and within this evaluation, simultaneously achieves higher utility and a lower adversarial recovery rate than placeholder redaction.

Ethical Considerations We address two ethical considerations below. Evaluation dataset. We constructed the N = 1,124-query evaluation dataset using pre-specified, synthetic PII values—fabricated names, generated SSNs, example email addresses. No data was collected from real users; no real individual’s personal information appears in any query, annotation file, or experimental output. The dataset does not constitute human subjects research. No IRB review was sought or required. Simulated attacker experiment. The attacker experiment (Section 5.3) submits surrogate-substituted messages to the Claude Haiku API under an adversarial prompt. All values in those messages are fabricated and bear no statistical relationship to any real person. No real PII is transmitted. The experiment measures adversarial robustness; it does not develop or deploy an attack tool. Section 5.3 describes the adversary capability and prompt methodology; the full prompt template is available verbatim in the released artifact (attacker.py, ATTACKER_PROMPT_TEMPLATE). Societal impact. SurrogateShield reduces the personal information transmitted to third-party LLM API endpoints. A system whose sole function is to restrict outbound data exposure presents no apparent misuse vector. Releasing the full codebase and evaluation framework enables independent audit of the privacy guarantees claimed in this paper.

Open Science The SurrogateShield codebase, evaluation dataset, annotation files, and experimental outputs are released as open-source research artifacts. The repository is publicly available at:

Sherwin Vishesh Jathanna

https://github.com/sherwinvishesh/SurrogateShield, and a project [23] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv overview is available at https://sherwinvishesh.github.io/SurrogateShield. preprint arXiv:1910.01108 (2019). The repository is released under the Apache-2.0 License. [24] Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narra-

References [1] Charu C. Aggarwal and Philip S. Yu. 2008. Privacy-Preserving Data Mining: Models and Algorithms. Springer. [2] Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2022. Large-scale differentially private BERT. In Proceedings of EMNLP 2022. 6481–6491. [3] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel HerbertVoss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium. 2633–2650. [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019. 4171–4186. [5] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference. Springer, 265–284. [6] Oluwaseun Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020. Privacyand utility-preserving textual analysis via calibrated multivariate perturbations. In Proceedings of WSDM 2020. 178–186. doi:10.1145/3336191.3371856 [7] Benjamin C. M. Fung, Ke Wang, Rui Chen, and Philip S. Yu. 2010. Privacypreserving data publishing: A survey of recent developments. Comput. Surveys 42, 4 (2010), 1–53. [8] Hugo Krawczyk and Pasi Eronen. 2010. HMAC-based Extract-and-Expand Key Derivation Function (HKDF). Technical Report RFC 5869. IETF. [9] John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML 2001. 282–289. doi:10.5555/645530.655813 [10] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT 2016. 260–270. [11] Ninghui Li, Tiancheng Li, and Suresh Venkatasubramanian. 2007. t-closeness: Privacy beyond k-anonymity and l-diversity. In Proceedings of ICDE 2007. 106–115. doi:10.1109/ICDE.2007.367856 [12] Pierre Lison, Ildikó Pilán, David Sánchez, Montserrat Batet, and Lilja Øvrelid. 2021. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 4188–4203. doi:10.18653/v1/2021.acllong.323 [13] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019). [14] Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bidirectional LSTM-CNNs-CRF. In Proceedings of ACL 2016. 1064–1074. [15] Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubramaniam. 2007. l-diversity: Privacy beyond k-anonymity. ACM Transactions on Knowledge Discovery from Data 1, 1 (2007), 3. [16] Microsoft. 2020. Presidio — data protection and anonymization API. Microsoft Open Source. https://github.com/microsoft/presidio. [17] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. 2024. Can LLMs keep a secret? Testing privacy implications of language models via contextual integrity theory. In International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310. 17884 [18] Arvind Narayanan and Vitaly Shmatikov. 2008. Robust de-anonymization of large sparse datasets. In Proceedings of the 2008 IEEE Symposium on Security and Privacy. 111–125. [19] Ishna Neamatullah, Margaret M. Douglass, Li-wei H. Lehman, Andrew Reisner, Mauricio Villarroel, William J. Long, Peter Szolovits, George B. Moody, Roger G. Mark, and Gari D. Clifford. 2008. Automated de-identification of free-text medical records. BMC Medical Informatics and Decision Making 8, 1 (2008), 1–17. doi:10. 1186/1472-6947-8-32 [20] Helen Nissenbaum. 2004. Privacy as contextual integrity. Washington Law Review 79, 1 (2004), 119–158. [21] Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021. Natural language understanding with privacy-preserving BERT. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. 1488–1497. doi:10.1145/3459637.3482281 [22] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019. 3982–3992.

tives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics 58 (2015), S20–S29. [25] Latanya Sweeney. 2000. Simple demographics often identify people uniquely. Technical Report Data Privacy Working Paper 3. Carnegie Mellon University, Pittsburgh, PA. [26] Latanya Sweeney. 2002. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10, 05 (2002), 557–570. [27] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP 2020: System Demonstrations. 38–45. doi:10. 18653/v1/2020.emnlp-demos.6 [28] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceedings of ICLR 2020.

Surrogate Generation Mechanisms

Table 10 documents the surrogate generator used by MimicGen for each PII type detected by SurrogateShield, together with the format guarantee and a representative example value. All surrogates are generated via Faker with a per-session uniqueness guarantee enforced through a used_surrogates set; after 50 collisions a 4-character random suffix is appended to guarantee a distinct value. Gender-indicator surrogates draw from a fixed pool of grammatically valid gender expressions rather than the uniquenessguaranteed path, because the pool is small and grammatical substitutability matters more than uniqueness for this type. Crypto wallet surrogates use the Bitcoin P2PKH character set (1 + Base58, 26–34 chars); ABA routing surrogates are computed to satisfy the 9-digit ABA checksum (3𝑑 0+7𝑑 1+𝑑 2+3𝑑 3+7𝑑 4+𝑑 5+3𝑑 6+7𝑑 7+𝑑 8 ) mod 10 = 0 and therefore pass any downstream format validator.

Table 10: Surrogate generation mechanism per PII type. All generators produce values that are type-consistent and pass the same format checks as real values of that type. PII type

Generator / format

Example

PERSON email phone (US) phone (UK) phone (intl.) SSN credit card

faker.name() faker.email() +1-###-###-#### +44 7### ###### Country code + digit groups faker.ssn() (XXX-XX-XXXX) faker.credit_card_number() (Luhn-valid) faker.address() Age 18–80, MM/DD/YYYY faker.ipv4() sk- + 32 random chars faker.zipcode() faker.postcode() 1 + Base58, 26–34 chars Computed 9-digit ABA checksum Letter + 7 digits (CA format) faker.city() faker.company() faker.company() + “Building” Pool of valid gender expressions

Ashley Wise [email protected] +1-602-555-4812 +44 7700 123456 +49 8234 927461 348-67-6360 4532 0151 1283 0366

street address date of birth IPv4 API key US ZIP code UK postcode crypto wallet ABA routing driver’s licence GPE / LOC ORG FAC gender indicator

789 Crescent Row 03/14/1985 10.42.17.203 sk-xKz9mP. . . 85281 SW1A 1AA 1BvBMSEYstW. . . 021000021 B4923817 Springfield Nexus Solutions Apex Building she/her

SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactions

PII Types Detected by SurrogateShield

Table 11 lists all 22 PII types detected by SurrogateShield, their detection method, and any validator applied beyond the regex match. Table 11: PII types detected by SurrogateShield. † dBERT = dslim/distilbert-NER (DistilBERT fine-tuned on CoNLL-2003, run locally).

surrogate mapping, transparent response restoration, and conversationaware management of surrogate values across multiple interaction turns. These capabilities are necessary to preserve user-visible responses while ensuring that only surrogate values appear in the LLM API history. Table 13: Architectural comparison between Microsoft Presidio Synthesize and SurrogateShield.