Locale-Conditioned Few-Shot Prompting Mitigates Demonstration Regurgitation in On-Device PII Substitution with Small Language Models
arXiv:2605.13538v1 [cs.CL] 13 May 2026
Anuj Sadani Infrrd.ai [email protected]
Deepak Kumar Infrrd.ai [email protected]
May 14, 2026
Abstract Personally Identifiable Information (PII) redaction usually replaces detected entities with placeholder tokens such as [PERSON], destroying the downstream utility of the redacted text for retrieval and Named Entity Recognition (NER) training. We propose a fully on-device pipeline that substitutes PII with consistent, type-preserving fake values: a 1.5 B mixture-ofexperts token classifier (openai/privacy-filter) detects spans, a 1-bit Bonsai-1.7B Small Language Model (SLM) proposes contextual surrogates for names, addresses, and dates, and a rule-based generator (faker) handles patterned fields. We report a prompting finding more important than the quantization choice: with naive fixed three-shot demonstrations, the 1-bit SLM regurgitates demonstration outputs verbatim regardless of input; 1.58-bit Ternary-Bonsai-1.7B reproduces byte-identical failures, ruling out quantization as the cause. We fix this with locale-conditioned rotating few-shot demonstrations: a character-range heuristic picks a locale-pure pool and a per-input MD5 hash samples three demonstrations. With the fix, 482/482 unique Bonsai-1.7B calls succeed (no echoes) and produce locale-correct surrogates, although the SLM still copies from a small same-locale demonstration pool—a residual narrowness we quantify. On a 2000-document multilingual corpus, hybrid perplexity (PPL) beats faker in all six locales under a multilingual evaluator (XGLM-564M); length preservation is best-of-three in 4 of 6 locales. On downstream NER (400 train / 100 test, English), redact yields F1=0.000, faker 0.656, original 0.960; on a matched 160/40 subset including hybrid, faker (0.506) outperforms hybrid (0.346) at p < 0.001. We report this as an honest negative finding: SLM surrogates produce more natural text but a less varied training distribution, and downstream NER benefits more from variety than from naturalness. Code: https://github.com/asadani/on-device-pii-substitution.
1
Introduction
Existing Personally Identifiable Information (PII) redaction tools—Microsoft Presidio [7], the OpenAI privacy filter [10], spaCy+regex pipelines [3]—produce redacted text in which detected entities are replaced by short placeholder tokens such as [NAME] or [EMAIL]. This is adequate for compliance display purposes but fails for downstream uses that need natural-looking text: training data for fine-tuning, search indices over redacted corpora, and retrieval-augmented generation over sensitive document stores. The placeholder approach also hurts utility in measurable ways: Named Entity Recognition (NER) models trained on [PERSON]-redacted corpora generalise poorly to non-redacted text (Section 4.5), and language-model perplexity over placeholder-rich documents is dominated by the placeholder tokens themselves. Substitution—replacing each detected PII span with a realistic, fake value of the same type— is the obvious alternative, but three constraints have so far prevented its widespread adoption in privacy-sensitive pipelines: (1) the substitution must be consistent within a document, so 1
that “John Smith” → “Marcus Chen” everywhere, not five different fakes; (2) it must be type-preserving, so a Chinese name does not become a US name; and (3) it should run on-device without sending the (still-real-PII) input to a cloud LLM [9]. Recent advances in 1-bit and ternary-bit quantization (e.g., the Bonsai [11] and TernaryBonsai [12] families, and prior work on BitNet [6]) make small generative language models— hereafter Small Language Models (SLMs)—viable on commodity CPUs. We investigate whether such models can serve as the surrogate-proposer in a privacy-preserving substitution pipeline, evaluating the resulting system on five metrics (privacy, naturalness, consistency, length, and downstream NER training F1). Threat model (scope). We measure literal-string leak against ground-truth PII values; we do not consider membership-inference or linkage attacks against the substituted output. Differential-privacy guarantees on the substitution distribution are out of scope. Contributions. 1. A reproducible on-device substitution pipeline combining openai/privacy-filter, Bonsai-1.7B (Q1_0, 1-bit blockwise quantization), and faker, with all code and configurations released. 2. A quantitative comparison of redact / faker-only / hybrid substitution modes under the constraint of CPU-only inference, on 100 multi-template multilingual documents. We are not aware of a directly comparable prior result that fixes all three constraints (consistency, type-preservation, and on-device execution) together. 3. Identification and isolation of a “few-shot regurgitation” failure mode in small SLMs at extreme quantization, validated by showing that 1-bit Bonsai-1.7B and 1.58-bit Ternary-Bonsai-1.7B produce identical regurgitating outputs on the same inputs (i.e., the failure is caused by prompting, not by quantization). 4. A simple and effective fix: locale-conditioned rotating few-shot demonstrations, with deterministic per-input demo sampling that preserves cache-friendliness. The fix eliminates the qualitative regurgitation failure (482/482 Bonsai-1.7B calls succeed, no echoes) and improves naturalness PPL across all six locales. It does not, however, make hybrid surrogates a better source of NER training data than faker’s random ones at the larger sample sizes we now evaluate—a tradeoff between naturalness and training-distribution variety that we report as an honest negative finding (Section 4.5).
2
Related Work
2.1
PII detection
Microsoft Presidio [7] combines regex, spaCy NER, and rule-based recognisers. The openai/privacy-filter model [10] (1.5 B Mixture-of-Experts (MoE), 50 M active) is a finetuned encoder using BIOES (Begin/Inside/Outside/End/Single)-style token tags (popularised in NER by Ratinov and Roth [14]) over eight PII categories. Running the inference harness shipped with the model on its bundled seven-template synthetic benchmark (samples.json, N =100), we measure F1 = 0.587 (precision 0.619, recall 0.627), with the largest source of false positives being dates (44.5% of all FPs) and the largest source of false negatives being non-English addresses (these numbers are computed by us, from the same harness; see Section 4.1). We use these as the detection-side floor for all subsequent comparisons, since every mode in our experiments inherits the same detector and therefore the same recall ceiling.
2
2.2
Synthetic PII and anonymization
faker [1] is the de-facto rule-based generator for synthetic PII. Presidio’s anonymizer can swap detected spans for hash, redaction, or counter-style replacements but not consistent typepreserving fakes. Recent work has used GPT-3.5 / GPT-4 [9] to generate context-appropriate replacements; this approach violates the on-device constraint and incurs per-request cost.
2.3
Low-bit small language models
The Bonsai family (1-bit Q1_0) [11] and Ternary-Bonsai (1.58-bit Q2_0; ternary weights) [12] are Qwen3-based [13] decoder-only LMs trained for extreme quantization. We run inference via llama.cpp [2]. Prior work [6] has evaluated extreme-quantization SLMs on Question-Answering (QA)-style benchmarks; we did not find published evaluations of these models on generativesubstitution tasks where input copying or demonstration-regurgitation is a failure mode (the specific failure we document in Section 4.3).
2.4
In-context learning copy bias and demonstration sensitivity
Our few-shot regurgitation diagnosis (Section 4.3, Section 4.6) sits within a broader literature on copy bias and demonstration sensitivity in in-context learning at full LM scale. Min et al. [8] show that demonstration label correctness matters less than format and label-space distribution: models often pattern-match the surface structure of demos rather than learn the input→output mapping. Zhao et al. [16] document a systematic majority-label and recency bias in few-shot prompts and propose calibration to correct it. Lu et al. [5] show that few-shot outputs are highly sensitive to demonstration order, with performance varying by tens of points across permutations. Our finding extends these phenomena to extreme-quantization SLM scale: a 1-bit (and 1.58-bit) model under fixed three-shot prompting emits demonstration values verbatim, and our locale-conditioned rotating-demo fix is structurally a per-input randomisation of both demonstration content and ordering.
2.5
Downstream-utility evaluation
Prior anonymization work typically reports privacy and naturalness in isolation. We follow the broader synthetic-data literature in additionally measuring length preservation and withindocument entity consistency, both relevant to downstream NER training utility, and we evaluate the latter directly by training a spaCy [3] NER model on each variant of our corpus and testing on held-out original text.
3
Method
3.1
Architecture text -> privacy-filter (BIOES -> char spans) -> EntityResolver (group by canonical, label) -> propose_surrogate dispatcher: PERSON / ADDRESS / DATE -> Bonsai-1.7B EMAIL / PHONE / ACCT / URL / SECRET -> faker -> splice (R-to-L, preserve whitespace) -> output text
Detection runs once per document at ∼1 s/512 tokens on CPU. The EntityResolver groups detected spans by (canonical_lowercased, label) so that all mentions of “John Smith” share a single surrogate. Surrogates are cached per (mode, family, canonical, label) so repeated names and common patterns trigger the SLM exactly once over the corpus. 3
3.2
Surrogate proposers
For PERSON, ADDRESS, and DATE labels, we invoke Bonsai-1.7B (Q1_0) through llama-cli –single-turn. Each call uses a three-shot prompt of the form Real: <example>\nFake: <example>\n...Real: <input>\nFake:. Locale-conditioned rotating demonstrations. Naively using a small fixed set of demonstrations is catastrophic for 1-bit SLMs: the model pattern-matches the demonstration outputs and emits one of them verbatim regardless of the input. A pilot run of the pipeline with three fixed English demonstrations produced “Alice Johnson” as the surrogate for the Chinese name 杨娟, the Japanese name 山田太郎, and the German name Müller-Schulz—all collapsed to the first English demonstration value (see Section 4.3 for the complete diagnosis). Our final design avoids this with two cooperating mechanisms: 1. Locale conditioning. A lightweight character-range and keyword-based heuristic classifies each input into one of {en, de, es, ja, zh} for PERSON / ADDRESS, and one of {mdy_slash, ymd_dash, dmy_dash_mon, dmy_slash, unknown} for DATE. Each class has its own pool of 4–8 demonstrations in the matching script and date format (Appendix A). 2. Per-input rotating sampling. From the appropriate pool, three demonstrations are sampled deterministically with an MD5 hash of the input string as the random seed. This means the same entity always receives the same demonstrations (cache-friendly across documents) but different entities receive different demonstration subsets, spreading surrogate-proposal calls across the full pool rather than concentrating them on one fixed demo subset. With this strategy, Bonsai-1.7B produces locale-appropriate, format-preserving surrogates: 杨娟→李伟, 山田太郎→郑强 (Chinese fallback for kanji-only Japanese names—see Section 5), Müller-Schulz→Anna Becker, 11-Jul-1998→05-Aug-2003. We validate every response and reject empty, identity-equal, or punctuation-only outputs, falling back to faker when validation fails. For EMAIL, PHONE, ACCT, URL, and SECRET labels, we use faker directly because these fields have low contextual entropy—there is no benefit from generative reasoning over Faker().email().
3.3
Splice and whitespace
The privacy filter sometimes includes a leading whitespace in its character span. We preserve original leading and trailing whitespace when splicing surrogates back into the text to avoid cosmetic artefacts such as is[PRIVATE_PERSON] (which would inflate perplexity).
4
Experiments
4.1
Setup
Dataset. We use the openai/privacy-filter synthetic-document generator [10] (the same template engine that produced the 100-document benchmark shipped with the model card) to generate a fresh, larger 2000-document corpus (data/samples_2000.json, seed=42), covering 7 templates (1099, W-2, auto_insurance, bank_statement, invoice, mortgage_insurance, paystub) and 6 locales (en_US n=840, en_IN 320, de_DE 240, es_MX 200, ja_JP 200, zh_CN 200). Each document carries ground-truth PII values (∼6.6 per document on average). Primary metrics (Table 1) are reported on the first 100 documents (en_US 40, en_IN 21, de_DE 12, zh_CN 12, ja_JP 8, es_MX 7); the downstream-NER experiment (Section 4.5) is run on a larger 500-document English subset to give the train/test split enough power for the small-effect comparisons it requires. 4
Table 1: Primary metrics (N =100, averaged across all 7 templates and 6 locales). Naturalness PPL is computed under the multilingual XGLM-564M [4] causal LM (covering all six evaluation locales). Mode
Leak ↓
PPL ↓
Consistency ↑
Length pres. ↑
redact faker hybrid
0.249 0.249 0.249
39.4 84.0 69.9
1.000 1.000 1.000
0.979 0.976 0.982
Avg. latency 1.7 s 1.6 s 41.2 s
Models. • Detection: openai/privacy-filter, 1.5 B MoE, 50 M active, bf16, CPU [10, 15]. • SLM: Bonsai-1.7B Q1_0 [11], CPU via llama.cpp [2]. • Naturalness evaluator: XGLM-564M (∼564 M parameters, multilingual causal LM trained on 30 languages including all six locales evaluated here) [4], CPU. • Fallback surrogate generator: faker [1]. • Downstream NER: spaCy blank English pipeline [3]. Hardware.
31 GB RAM, x86-64 CPU; no GPU used.
Modes evaluated. • redact: replace each span with [LABEL] (current state of practice). • faker: replace every span with a faker value of matching type. • hybrid: Bonsai-1.7B for PERSON / ADDRESS / DATE, faker for the other five labels (our proposed pipeline).
4.2
Primary metrics
We report four per-document metrics, averaged across the corpus: 1. Leak rate: fraction of ground-truth PII strings that still appear verbatim (case-insensitive substring) in the output. Lower is better. 2. Naturalness perplexity (PPL): XGLM-564M perplexity over the output text, chunked at 1024 tokens. Lower is more natural. XGLM-564M is multilingual and covers all six evaluation locales, so cross-locale PPL numbers are directly comparable rather than biased toward Latin-script outputs. 3. Consistency rate: for entities appearing ≥ 2 times in the input, fraction where the output uses the same surrogate at every mention. Higher is better. 4. Length preservation: 1 − | len(out) − len(in) |/len(in). Closer to 1 is better. Headline result. Hybrid PPL (69.9) is lower than faker (84.0, −16.8%) under the multilingual XGLM-564M evaluator, and length preservation is the best of the three modes (0.982 vs. 0.976, 0.979). Redact’s PPL (39.4) is the lowest of the three numerically, but this is not because the redacted text is more natural—XGLM-564M simply scores the [LABEL] placeholder as a single out-of-distribution token without further compositional structure to evaluate; we discuss this caveat in Section 5. Consistency is 1.000 across all modes because surrogates are deterministically cached per (mode, family, canonical, label). 5
Privacy floor. The leak rate of 0.249 is identical across all three modes: every mode misses the same ground-truth values, consistent with the detection-side recall (≈ 0.627) we measured for the privacy filter on the same data (Section 2). The architectural choice between redact / faker / hybrid does not affect privacy at all—only what to do with the spans the filter catches. Per-locale (PPL). Hybrid is the XGLM-564M-PPL winner over faker in all six locales: en_US (66.1 vs. 80.7, −18.1%), en_IN (74.4 vs. 83.4, −10.8%), de_DE (79.3 vs. 92.5, −14.3%), es_MX (81.1 vs. 101.0, −19.7%), ja_JP (55.8 vs. 82.4, −32.3%), zh_CN (68.3 vs. 78.9, −13.4%). Because XGLM-564M is multilingual, the gap is no longer dominated by tokenisation artefacts, and the ja_JP / zh_CN advantages—where the SLM produces script-correct surrogates that the LM can score in-distribution—are now visible rather than hidden. Per-locale (length preservation). Hybrid is the length-preservation winner over faker in four of six locales (de_DE 0.979 vs. 0.965, en_IN 0.981 vs. 0.972, ja_JP 0.972 vs. 0.964, zh_CN 0.992 vs. 0.974); en_US (0.983 vs. 0.985) and es_MX (0.978 vs. 0.983) narrowly favour faker. The zh_CN gap (+0.018) is the largest of any locale, reflecting the SLM’s ability to produce length-matched Han-script addresses where faker’s random selection from its zh_CN pool varies more in length.
4.3
Few-shot regurgitation: a prompting failure, not a quantization failure
A pilot run of the pipeline using a fixed three-shot demonstration template (one English, one Japanese, one Spanish demo) revealed a striking failure mode: across all 509 unique-entity Bonsai-1.7B calls, the model produced output that was never literally identical to the input (0 echoes by our validation), but for low-resource-locale inputs the output was very often one of the few-shot demonstration values verbatim, regardless of the input: • 杨娟 (Chinese, NAMED INSURED on a zh_CN auto-insurance form) → “Alice Johnson” (the first PERSON demonstration). • 宁夏回族自治区兰州县山亭陈街B座572990 → “123 Main Street, Boston MA 02101” (the first ADDRESS demonstration). • 11-Jul-1998 (DD-Mon-YYYY) → 03/15/1985 (MM/DD/YYYY—both type and locale wrong). • 山田太郎 (Japanese), Müller-Schulz (German) → both “Alice Johnson”. Diagnosis: prompting, not quantization. Our initial hypothesis was that 1-bit quantization had degraded the model’s instruction-following. We tested this by re-running the exact same five problem prompts under 1.58-bit Ternary-Bonsai-1.7B (Q2_0) [12]—a different quantization scheme running on the same Qwen3 base architecture [13] and the same demonstrations. Ternary-Bonsai-1.7B produced byte-for-byte identical outputs to Bonsai-1.7B (杨娟 → “Alice Johnson”, 11-Jul-1998 → 03/15/1985, etc.) at roughly six times slower per-call inference. This rules out quantization as the root cause: the model is doing what any small LM asked to pattern-match Real:X\nFake:Y would do—copy one of the demonstration values from the prompt context (the first PERSON demo’s output “Alice Johnson” in our pilot) when the input does not match the demonstration’s distribution. Fix: locale-conditioned rotating few-shot demonstrations (described in Section 3.2). With character-range locale detection feeding into pools of 4–8 demonstrations per locale (and 4 demonstrations per date format), and per-input-hash-seeded sampling of 3 demonstrations per call, the same 1-bit Bonsai-1.7B produces:
6
Table 2: Average per-document latency (seconds, CPU only). Mode
Detect
Surrogate
Splice + PPL
Total
redact faker hybrid
∼1.7 ∼1.6 ∼1.5
0.00 <0.01 ∼39.8
<0.1 <0.1 <0.1
1.7 1.6 41.2
• 杨娟 → 李伟 (Chinese name in zh-pool) • 宁夏回族自治区兰州县山亭陈街B座572990 → 广东省广州市天河区珠江新城200号 (Chinese address) • 11-Jul-1998 → 05-Aug-2003 (DD-Mon-YYYY format preserved) • Müller-Schulz → Anna Becker (German name) • Hauptstraße 45, 10117 Berlin → Bahnhofstraße 7, 60313 Frankfurt • “John Smith” → “David Kim” (different US name; no longer “Alice Johnson”) The fix is dataset-free, does not require fine-tuning, and adds < 50 ms per surrogate-proposal call. All quantitative numbers in Tables 1 and 3 are produced under this fixed prompting strategy. Known limit: kanji-only Japanese names. These map to the zh pool because our locale heuristic requires kana to disambiguate (山田太郎 → 郑强 instead of a Japanese fake). A character-frequency-based classifier or an explicit “treat ambiguous CJK as both ja and zh” strategy would address this.
4.4
Latency
The 24× latency gap between hybrid and redact / faker is dominated by Bonsai-1.7B surrogate generation. Without our (canonical, label) cache, hybrid would call Bonsai-1.7B for every PII mention (∼660 calls across the 100-document corpus); with the cache, only 482 unique calls were made (482/482 succeeded, 0 echoed-or-empty, 0 errored under the locale-conditioned prompting strategy), saving ∼27% of inference time. Each Bonsai-1.7B call costs ∼7–10 s for the ∼30-token output (loading + 1.7 B model inference + cleanup). Detection cost is identical across modes because the privacy filter is invoked both for input PII detection and for residual-leak measurement.
4.5
Downstream NER utility
Setup. We test whether substitution preserves downstream training utility: an NER model trained on substituted data should approach the F1 of one trained on original data. From the 2000-document corpus (Section 4.1) we draw the English-locale subset (1159 documents, en_US + en_IN). We run the experiment at two scales: a large-scale 400 train / 100 test split for original, redact, and faker, and a matched-subset 160 train / 40 test split for all four modes including hybrid. The two-scale design lets us report the high-statistical-power comparison between the three cheap modes at the larger scale, while keeping the (CPU-bound) Bonsai-1.7B surrogate generation tractable for the four-way comparison at the smaller scale; both scales share the same stratified-by-locale split methodology. The test set is always in original form—substitution applies only to the training set. PII spans are extracted from the ground-truth pii_gt dictionary (substring search) so that substitution quality is decoupled from detection quality. A blank spaCy English NER pipeline is trained for 30 iterations with a single 7
Table 3: Span-level NER F1 on held-out original documents. Mean ± SD across 5 spaCy training seeds; the substituted training corpus is fixed per mode, only spaCy gradient initialisation varies. Top: large-scale 400 train / 100 test, three modes. Bottom: matched 160 train / 40 test subset including the hybrid mode, which requires Bonsai-1.7B surrogate generation and is therefore evaluated at the smaller scale to keep wall-clock manageable on CPU. Mode
Train spans
Precision
Recall
F1
∆F1 vs. orig.
Large-scale (400 train / 100 test): original 2592 0.947 ± 0.006 0.974 ± 0.012 0.960 ± 0.004 redact 2592 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 faker 2592 0.984 ± 0.008 0.493 ± 0.038 0.656 ± 0.033
(baseline) −0.960 −0.304
Matched subset (160 train / 40 test): original 1046 0.895 ± 0.006 redact 1046 0.000 ± 0.000 faker 1046 0.955 ± 0.017 hybrid 1046 0.909 ± 0.015
(baseline) −0.908 −0.402 −0.562
0.923 ± 0.004 0.000 ± 0.000 0.347 ± 0.052 0.215 ± 0.033
0.908 ± 0.003 0.000 ± 0.000 0.506 ± 0.056 0.346 ± 0.044
binary PII label on each variant of the train set, then evaluated against held-out original spans (label-agnostic span overlap). Redact destroys downstream utility entirely (F1 = 0.000). A NER model trained on [PRIVATE_PERSON]-style placeholders learns to predict only those placeholder tokens and never fires on real text. This is the sharpest possible motivation for substitution: redacted-text training data is, for any downstream NER consumer, effectively labelled negative-only data. The model converges (training loss falls to <2.0) but its decision boundary lives in the wrong vocabulary. Faker and hybrid both recover a substantial fraction of original F1. Type-preserving fakes—whether random (faker) or SLM-generated locale-conditioned (hybrid)—give the model enough surface variety to learn that “capitalised proper-noun-shaped two-token sequence in a name field” is a PII span. At the large scale (400 train / 100 test), faker recovers 0.656/0.960 = 68.3% of the original baseline; precision is high (0.984 ± 0.008) and recall is moderate (0.493 ± 0.038)—fake-data-trained models are conservative, preferring to miss spans rather than hallucinate them. The seed-to-seed standard deviation on faker F1 is tight at 0.033, putting all between-mode comparisons solidly above the noise floor. The hybrid–faker comparison at the matched scale: the gap is now real, not noise. On the matched 160 train / 40 test subset—where all four modes share the same train/test docs and all 1046 training PII spans—we observe hybrid F1 = 0.346 ± 0.044 versus faker F1 = 0.506 ± 0.056, a gap of −0.160 in faker’s favour. Under p Welch’s two-sample t-test (nf = nh = 5, unequal variances) this gives standard error SE = 0.0562 /5 + 0.0442 /5 = 0.032, t = 5.02, Welch–Satterthwaite degrees of freedom ≈ 7.6, and a two-tailed p < 0.001.1 The gap is well outside seed-level noise. This is an honest negative finding for the central hybrid-vs-faker comparison on downstream NER training utility, even though hybrid wins on every other metric we measure: hybrid PPL beats faker under XGLM-564M in all six locales (Table 1) and length preservation is best-of-three in four locales. The interpretation is that locale-conditioned, contextually-realistic surrogates produce more natural-looking output text but a less varied surface distribution than faker’s 1 Standard deviations in Table 3 are computed with statistics.pstdev (population SD) for consistency with our existing harness; the corresponding sample-SD-based Welch statistic is t ≈ 4.49 on ν ≈ 7.6 degrees of freedom, which still resolves at p < 0.001.
8
i.i.d. random surrogates—and downstream NER training benefits more from variety than from naturalness. We revisit this tradeoff in Section 5, quantify it in the “Why hybrid is less varied” paragraph below, and propose concrete remediations (higher-temperature decoding, larger pools, stochastic pool sampling, hybrid+faker mixed corpora) in Section 6. A pilot run with the naive fixed-three-demonstration strategy (Section 4.3, single seed) produced hybrid F1 = 0.310 vs. faker F1 = 0.427—a gap of −0.117. The matched 160/40 gap is −0.160, which is larger, not smaller, than that pilot. We read this as: locale-conditioned prompting eliminates the qualitative demonstration-regurgitation failure (482/482 unique Bonsai-1.7B calls succeed; Section 4.3), but the resulting in-distribution surrogates still yield less downstream training variety than faker’s random ones. Prompting fixes the output of the SLM; it does not turn that SLM’s output into a better training-data generator than faker for this particular task.
4.6
Surrogate distinctness: why hybrid is less varied
The “less varied surface distribution” interpretation in Section 4.5 is not just qualitative—it falls out structurally from how the SLM is prompted. Each locale-conditioned PERSON pool in Section A contains 6–8 (input, output) pairs, so the total number of demonstration strings the SLM ever sees as prompt context for a given locale is bounded by 2|pool| = 12–16 strings per locale (|pool| “Real:” input examples plus |pool| “Fake:” output examples), totalling roughly 64 demonstration strings across all five locale pools (en, de, es, ja, zh). Few-shot pattern matching at SLM scale pulls outputs strongly toward any of these visible strings (we confirm both-side copying empirically below); the achievable corpus-wide surrogate vocabulary is therefore upper-bounded by a small constant set by the prompt design, independently of the number of input documents. faker has no such ceiling: each call samples i.i.d. from each locale’s full first/last name lists (∼ 103 –104 unique values per locale). We confirm this bound empirically on the matched 160-document en-locale training corpus (the same training set as Table 3 bottom). Table 4 reports unique-surrogate counts and typetoken ratios per label per mode. For PERSON, hybrid uses only 10 distinct surrogate names across 274 mentions in 160 documents, against faker’s 31—a 3.1× narrower vocabulary, with type-token ratio 0.037 vs. 0.113. A finer-grained look at the 10 hybrid PERSON surrogates is illuminating. The five mostrepeated values are Jennifer Wong, Linda Vasquez, Michael O’Brien, David Kim, and Priya Krishnamurthy (cf. Section A). All five appear verbatim in the en demonstration pool—and notably, all five are the input-side (“Real:”) examples, not the output-side (“Fake:”) examples that the prompt nominally offers as the imitation target. The locale-conditioned fix has therefore not eliminated demonstration regurgitation; it has shifted the failure from “copy a wrong-locale demo verbatim” (Section 4.3) to “copy a same-locale demo entry verbatim,” with the model drawing from both sides of the (input, output) pairs. The measured 10 unique en PERSON surrogates fits comfortably under the 2|pool| = 16 ceiling derived above. For ADDRESS the picture is the same: 6 unique hybrid surrogates vs. 18 for faker on 162 mentions (3.0×). For the labels that use faker in both modes (EMAIL, PHONE, ACCOUNT), hybrid still produces ∼2× fewer unique values than faker (8 vs. 16, 18 vs. 29, 33 vs. 61). This residual gap is not a regurgitation effect—no SLM is involved—but a side-effect of faker’s perdocument re-seeding interacting with mode-dependent call ordering: in faker mode, Faker’s first PERSON / ADDRESS calls advance the generator state before EMAIL / PHONE / ACCOUNT are sampled, so the latter see varied state across documents; in hybrid mode the SLM handles PERSON / ADDRESS without consuming Faker state, so EMAIL / PHONE / ACCOUNT are repeatedly sampled from the just-after-seed-0 generator state and concentrate on a smaller set. The implication: the residual 2× narrowing in non-SLM labels is a separate (and fixable) issue tied to seed management, distinct from the 3× narrowing in SLM-handled labels that is driven by demonstration regurgitation. The combined effect is that the NER trainer sees only a handful of distinct “positive” PII 9
Table 4: Surrogate distinctness on the matched 160-doc en-locale training corpus (same training set as Table 3 bottom). For PERSON and ADDRESS—the two labels where hybrid uses the SLM rather than faker—hybrid produces a 3× narrower unique-surrogate vocabulary than faker, consistent with demonstration-pool regurgitation. DATE (n=20) is a small-n outlier where hybrid’s multi-format demo pools yield slightly more variety. EMAIL/PHONE/ACCOUNT use faker in both modes but show a residual ∼2× gap driven by per-document Faker re-seeding interacting with call ordering (see text), not by SLM regurgitation. Label PERSON ADDRESS DATE
Mentions
faker
hybrid
unique
TTR
unique
TTR
31 18 5
0.113 0.111 0.250
10 6 9
0.037 0.037 0.450
8 18 33
0.079 0.085 0.119
274 162 20
Faker-only labels in both modes (sanity check): EMAIL 101 16 0.158 PHONE 211 29 0.137 ACCOUNT 278 61 0.219
strings in hybrid mode, learns those specific surface forms, and fails to generalise to the held-out original documents—the mechanism behind hybrid’s lower recall in Table 3. The dominant factor is the SLM-handled labels, which are also the largest by mention count (PERSON 274, ADDRESS 162). The fix lives at the prompt level, not in the SLM weights; the Section 6 suggestions (higher-temperature decoding, larger pools, stochastic pool sampling) attack this ceiling directly.
5
Discussion and Limitations
Privacy-filter recall ceiling. The non-zero leak rate in all three modes reflects the privacy filter’s measured detection F1 of 0.587 (Section 2). A pipeline using a higher-recall detector or pattern-based fallback (e.g., regex for Social Security Numbers (SSNs) and pre-masked digits) would shift this floor for all modes equally; the relative comparison between substitution modes is unaffected. Few-shot regurgitation is a prompting issue, not a quantization issue. Section 4.3 shows that the failure occurs identically under 1-bit Bonsai-1.7B and 1.58-bit Ternary-Bonsai-1.7B. Larger or higher-precision models might suppress the symptom by attending more strongly to the instruction, but the underlying pattern-matching pull will remain at any small-LM scale. The first-line fix is at the prompt level (locale-conditioned rotating demonstrations), which eliminates the qualitative cross-locale failure of the fixed-three-shot strategy and we recommend as default practice; however, as Table 4 shows, the SLM still copies from the demonstration pool, so the residual surrogate-vocabulary ceiling is ∼ 2|pool| and the broader fix is to enlarge or stochasticise the pool itself (Section 6). No coreference resolution. “John” and “John Smith” within the same document are treated as distinct entities by surface-form grouping. A proper coreference layer would further improve consistency. Naturalness via XGLM-564M PPL. We use XGLM-564M [4], a 564 M-parameter multilingual causal LM covering all six evaluation locales, as the naturalness proxy. PPL under a generic 10
LM remains a proxy: it rewards in-distribution surface form rather than literal naturalness, and it rewards short placeholder tokens (which redact mode produces) for being predictable as standalone tokens regardless of whether the surrounding text is grammatical. A human evaluation, or a more discriminative proxy such as a fluency classifier, would refine this measurement. No explicit threat model. We measure literal-string leak, not inference attacks. A determined adversary with access to (output_text, external knowledge) could potentially link surrogates back to originals. Differential-privacy guarantees on the substitution distribution are out of scope. Sample size and statistical reporting. The downstream-NER experiment is run at two scales: a large-scale 400 train / 100 test split for original, redact, and faker, and a matched 160 train / 40 test split that additionally includes hybrid. We report mean ± SD per mode across n = 5 training seeds and apply Welch’s t-test to the substituted-mode comparisons (Table 3). All pairwise comparisons are multiple SDs apart and resolve at p < 0.001, including the hybrid vs. faker comparison at the matched scale (Section 4.5).
6
Future Work
The hybrid-vs-faker downstream-NER gap (Section 4.5) frames most of the open directions below. The shared diagnosis: locale-conditioned demonstrations fix the qualitative regurgitation failure but yield a narrower surrogate distribution than faker’s i.i.d. random sampling. Future work should attack that distribution directly. Broaden the SLM surrogate distribution. We use llama-cli with default decoding, which is close to greedy at this scale and produces the same surrogate for the same input every time (the point of the MD5 seeding, but also the source of the narrowness). Two parameter-only changes are worth ablating: (i) raising temperature / nucleus-sampling threshold to widen single-call output variance, and (ii) replacing the deterministic MD5 seed with a per-document random seed so that repeated entities still collapse within a document but the corpus-level surrogate distribution is no longer fixed. Both keep the on-device constraint and add no latency. Larger and stochastic demonstration pools. Each locale pool currently holds 4–8 (input, output) entries (Section A); the SLM regurgitates from both sides of those entries (Section 4.5), so the achievable surrogate space is bounded above by roughly 2|pool| unique strings per locale, i.e. 12–16 strings. Two extensions: scale each pool to 50–100 entries (cheap, no model changes), and stochastically rotate the pool subset itself rather than deterministically hashing the input. The empirical hybrid PERSON unique-count of 10 in Table 4 is at the ceiling of the current 2|pool| = 16 bound and is the proximate cause of the hybrid–faker NER gap. Mixed-source training corpora. A pragmatic alternative to closing the gap on the SLM side is to feed the downstream NER trainer a mix of hybrid and faker substitutions of the same documents—naturalness from hybrid, surface variety from faker. The mix ratio is a single hyperparameter. We hypothesise an F1 between the two single-source modes, but the alternative—that the mix exceeds either single source by combining faker’s surface variety with hybrid’s locale fidelity—is also plausible and is precisely what an ablation would test. Coreference-aware entity grouping. Current resolution is surface-form only; “John” and “John Smith” within the same document map to distinct entities. A lightweight coreference layer would tighten consistency and slightly reduce the number of unique surrogate-proposal calls. 11
Locale-heuristic refinements. Kanji-only Japanese names currently route to the zh pool because our character-range heuristic requires kana to disambiguate (Section 4.3). A characterfrequency or n-gram-based classifier, or an explicit “treat ambiguous CJK as both ja and zh and let the SLM pick” strategy, would address this. Higher-recall detection floor. The 0.249 leak rate (Table 1) is the privacy filter’s detection ceiling; it bounds all three modes equally. Layering a regex/rule fallback for SSNs, pre-masked digits, and locale-specific patterns on top of openai/privacy-filter is orthogonal to the substitution-mode comparison but important for production deployment. Inference-attack threat model. We measure literal-string leak only. A determined adversary with auxiliary knowledge could potentially link surrogates back to originals via membershipinference or linkage attacks. Quantitative evaluation under a formal threat model—and, separately, a differentially-private surrogate sampler—would close the gap between “no literal leak” and “no information leak.”
7
Conclusion
We built and evaluated a fully on-device PII substitution pipeline combining openai/privacy-filter, Bonsai-1.7B (Q1_0), and faker; the architecture meets its document-level objectives (consistent, length-preserving, locale-correct surrogates with multilingual-PPL gains over faker in all six evaluated locales). The most actionable contribution is methodological. We document a “few-shot regurgitation” failure mode in which a small SLM, prompted naively with a fixed three-shot demonstration template, ignores the input and emits one of the demonstration outputs verbatim—and we show by direct comparison with 1.58-bit Ternary-Bonsai-1.7B that this is a property of prompting at small-LM scale, not of 1-bit quantization. A simple locale-conditioned rotatingdemonstrations prompting strategy resolves the qualitative cross-locale failure: 482/482 unique Bonsai-1.7B surrogate proposals succeed (no echoes, no errors) under this strategy, producing locale-correct, format-preserving surrogates (杨娟 → 李伟, Müller-Schulz → Anna Becker, 11-Jul-1998 → 05-Aug-2003); the fix adds < 50 ms per surrogate proposal. Substitution outperforms [LABEL]-style redaction by an enormous margin on downstream NER training utility: at the large 400 train / 100 test scale, faker recovers F1=0.656 ± 0.033 against redact’s 0.000 ± 0.000 (a gap of many standard deviations, not a noise effect). Our negative finding for hybrid is the second methodological contribution. At the matched 160/40 scale, faker (F1=0.506) clearly beats hybrid (F1=0.346) at p < 0.001, despite hybrid winning every other metric (PPL, length preservation, locale-fidelity of generated strings). The implication: natural-looking substitution and useful-for-training substitution are different objectives. SLM surrogates collapse to a narrower in-distribution surface than faker’s i.i.d. random outputs, and downstream NER training benefits more from the variety than from the naturalness. We caution future work against treating “hybrid beats faker on PPL” as a proxy for downstream utility—they are not the same metric. All code, configurations, and the 2000-document generated evaluation corpus are released under an open license.
Reproducibility All code and the 2000-document generated evaluation set (data/samples_2000.json, produced by the openai/privacy-filter template engine at seed=42) are released at: https://github.com/asadani/on-device-pii-substitution 12
To reproduce the primary results: # Primary metrics (Table 1, ~3 hours on CPU) python eval_substitution.py --n 100 --modes redact faker hybrid \ --samples-path data/samples_2000.json \ --bonsai-size 1.7B --output results/run_v3 # Per-locale and per-template breakdowns python analyze_results.py \ --results results/run_v3/substitution_results.json \ --out results/run_v3/analysis.md # Downstream NER utility -- large scale, 3 modes (Table 2 top, ~2.5 h) python eval_ner_utility.py \ --samples-path data/samples_2000.json \ --modes original redact faker --n-max 500 \ --output results/ner_v3 # Downstream NER utility -- matched 4-mode subset (Table 2 bottom, ~1.5 h) python eval_ner_utility.py \ --samples-path data/samples_2000.json \ --modes original redact faker hybrid --n-max 200 \ --output results/ner_v3_matched # Surrogate distinctness (Table 3, ~50 min hybrid + <1s faker on CPU) python analyze_surrogate_distinctness.py --n-max 200 \ --modes faker hybrid \ --output results/surrogate_distinctness.json
Total wall-clock for the full pipeline is approximately 8–9 hours on a 31 GB single-CPU machine, dominated by hybrid-mode Bonsai-1.7B generation in the primary, NER, and distinctness experiments. The privacy-filter model (∼2.8 GB) and the XGLM-564M evaluator (∼1.1 GB) are downloaded once from Hugging Face on first run and cached locally.
Use of generative AI assistance In line with arXiv’s policy on the use of generative AI in scholarly work, the authors disclose that a large language model assistant (Anthropic’s Claude, primarily Claude Opus 4.7) was used during the preparation of this manuscript. Specifically: (i) the assistant helped explore and draft prose for the introduction, related-work, and discussion sections from the authors’ notes and bulletpoint outlines; (ii) the assistant helped scaffold LaTeX, bibliography, and table formatting; and (iii) the assistant ran the experimental scripts and reported the resulting numerical outputs back to the authors. All quantitative results in Tables 1, 2, 3, and 4 are computed by the released code (eval_substitution.py, eval_ner_utility.py, analyze_surrogate_distinctness.py) on the released artefact and are reproducible by any reader; no result was generated, projected, or hallucinated by the language model. The authors have independently reviewed all assistantdrafted text and are solely responsible for the technical claims, methodology, statistical analysis, and the final wording of the manuscript.
References [1] Daniele Faraglia and others. Faker: a Python package that generates fake data. Software repository, MIT license, https://github.com/joke2k/faker, 2024. accessed 29 April 2026. 13
[2] Georgi Gerganov and contributors. llama.cpp: Port of LLaMA models in C/C++. Software repository, https://github.com/ggerganov/llama.cpp, 2024. accessed 29 April 2026. [3] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in Python. Software framework, https: //spacy.io, 2020. accessed 29 April 2026. [4] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. arXiv preprint arXiv:2112.10668, 2022. [5] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8086–8098. Association for Computational Linguistics, 2022. [6] Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The Era of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024. [7] Microsoft. Presidio: Context aware, pluggable, and customizable data protection and de-identification sdk. Software repository, Apache-2.0 license, https://github.com/ microsoft/presidio, 2024. accessed 29 April 2026. [8] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11048–11064. Association for Computational Linguistics, 2022. [9] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [10] OpenAI. openai/privacy-filter: a 1.5 b mixture-of-experts token classifier for PII detection. Hugging Face model card, https://huggingface.co/openai/privacy-filter, 2025. accessed 29 April 2026. [11] PrismML. Bonsai-demo: a reference implementation for the bonsai family of 1-bit (q1_0) qwen3-based small language models. Software repository, https://github.com/ PrismML-Eng/Bonsai-demo, 2025. accessed 29 April 2026. [12] PrismML. Ternary-bonsai: 1.58-bit (q2_0) qwen3-based small language models for ondevice inference. Distributed via the Bonsai-Demo software repository, https://github. com/PrismML-Eng/Bonsai-demo, 2025. accessed 29 April 2026. [13] Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [14] Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155. Association for Computational Linguistics, 2009. [15] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on 14
Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics, 2020. [16] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 12697–12706, 2021.
A
Locale-conditioned demonstration pools
For full reproducibility we list the demonstration pools used by the locale-conditioned rotating few-shot strategy (Section 3.2). Per-input MD5 hashes seed a deterministic sample of three demonstrations from the appropriate pool. PERSON pool. • en: (John Carter, Marcus Chen), (Linda Vasquez, Olivia Brennan), (David Kim, Theo Pemberton), (Sarah Patel, Maya Iyer), (Robert Williams, Daniel Foster), (Priya Krishnamurthy, Nadia Subramanian), (Michael O’Brien, Patrick Donovan), (Jennifer Wong, Cynthia Park). • de: (Hans Müller, Karl Schmidt), (Anna Becker, Lena Hoffmann), (Klaus Wagner, Erik Krüger), (Ingrid Weber, Petra Neumann), (Stefan Fischer, Dietrich Bauer), (Helga Zimmermann, Brigitte Klein). • es: (Juan García, Carlos Hernández), (María Rodríguez, Ana Fernández), (Diego Sánchez, Luis Castillo), (Carmen Ortiz, Lucía Vázquez), (Roberto Jiménez, Pablo Morales), (Sofía Ramírez, Elena Aguilar). • ja: (山田太郎, 鈴木一郎), (佐藤花子, 田中美咲), (渡辺健, 高橋翔), (中村裕子, 小林由香), (加藤博之, 斎藤大輔), (井上恵美, 松本香織). • zh: (李伟, 王芳), (张敏, 刘洋), (陈杰, 黄燕), (周磊, 吴娟), (徐明, 孙丽), (郑强, 马晶). ADDRESS pool. Six entries per locale; en covers both en_US and en_IN. Examples include (Hauptstraße 45, 10117 Berlin, Lindenallee 12, 80331 München) for de, (Calle Reforma 123, 06600 CDMX, Avenida Insurgentes 456, 03100 CDMX) for es, and (北京市朝 阳区建国路1号, 上海市浦东新区世纪大道100号) for zh. Full lists are in pii_substitute.py. DATE pool. Pools per detected format: mdy_slash (5 entries), ymd_dash (4), dmy_dash_mon (4), dmy_slash (3), unknown (2 fallbacks).
15