Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox Federico Lucca Independent researcher, Trento, Italy

arXiv:2610.32955v1 [cs.CL] 23 Oct 2026

federico.lucca@gmail.com Preprint v1.1, 22 October 2026

Abstract Personal email agents decide, for every incoming message, what kind of message it is, whether it is urgent, whether a reply should be drafted, whether a task should be created, and whether the message should be ignored. In production these decisions are usually delegated to a generative LLM or to ad-hoc rules, and their error rate is rarely measured: the labels in the database were written by the system itself. We describe Foyer (formerly SecondBrain), a self-hosted personal inbox agent in daily use since 2026, and a measurement protocol—stratified sampling, a frozen snapshot of what the model sees, thread-grouped splits, secret redaction, and human labels collected in the agent’s own review loop—that produces ground truth before any training. On 231 real emails in Italian and English labelled by the owner, we compare, under the same thread-grouped 5-fold cross-validation or zero-shot on the same rows, the production keyword rules, a TF-IDF + logistic-regression baseline, three sizes of a local typed decision model (Tev1 0.8B and 4B, Nimble 9B via Ollama on CPU and on a T4), a hosted typed decision model (Bespoke Nimble v4-max) and three hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B-A4B and 31B). Three results. First, the production rules for message type score 28% against a 58% majority class: the field the dashboard displayed most prominently carried no information. Second, with about two hundred labels, a logistic regression has the highest accuracy of any system on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), at a latency three orders of magnitude lower—though it never predicts the two rarest types, and removing the sender from its input costs it four points on type; the best zero-shot systems are Gemma 4 on type (0.74) and the hosted decision model on urgency and is-noise (0.71, 0.86), with Tev1 4B on a T4 the best local option (0.69 on type, 0.35 s per decision). Third, the sender’s relationship to the owner—client, supplier, partner, service—is recovered from other emails of the same contact with 98.6% accuracy on the 63% of emails whose sender has been seen before, and 85.7% overall with a majority fallback; it belongs in the contact graph, not in the prompt. Fourth, per-email accuracy is not the product metric: when the labels are composed into the agent’s actions, the trained baseline—more accurate on every field—is also the system that most often ignores an email that needed a reply or a task (18% of such emails, against 4% for the best generative model), because it is precise and conservative while the LLMs over-act; a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system overall. A third, adversarially sampled round of 500 emails (722 in total), with eleven hypotheses pre-registered before the labels were opened, confirms seven of them and refutes four. We release the protocol, the harness and the pre-registration; the emails stay private.

1 Introduction

typically keep: the stored labels are the model’s own output. Early evidence is mixed. Classical classifiers trained on a few thousand records outperform Jev on network-traffic classification [7]; a community benchmark reports a plain 9B instruct model ahead of every decision model on triage tasks; the vendor’s own benchmark places the open models within three points of Jev [3]. None of it concerns personal, multilingual correspondence.

A growing share of what personal AI agents do is not generation but decision: which project does this email belong to, is it urgent, does it need a reply, is it noise. Generative LLMs answer these questions by producing text that is then parsed; typed decision models, introduced with TypeSafe’s Jev in September 2026 [1] and followed within weeks by open models (Nimble [2], Tev1, Clef, Strands Decider) and This paper makes three contributions. a local endpoint in Ollama [3], answer them directly with a probability distribution over caller-defined options, in one 1. A measurement protocol for a production email agent forward pass and without free text. that yields human ground truth without changing the Whether this new class of models is good enough to resystem: stratified sampling by predicted class, a frozen place the generative step in a personal agent is an empirical snapshot of the exact model input, deterministic splits question, and it cannot be answered with the data such agents keyed on record ids, and labelling in the agent’s own 1

Preprint — draft v0.6, 21 October 2026

Lucca

interface. 2. A comparison of nine systems on that ground truth— production rules, a classical baseline, local and hosted typed decision models, hosted generative LLMs—on message type, urgency, three agent switches (draft a reply, create a task, ignore) and project assignment, with calibration, latency and cost. 3. A decomposition of the agent’s labels into what is learnable from the text of one message (type, urgency, the three switches) and what is not (the sender’s relationship to the owner), with a measured alternative for the latter: inheritance from the contact graph.

leave the device.

3 The System Foyer (formerly SecondBrain) is a self-hosted personal dashboard (React + Django) that ingests Gmail and IMAP accounts, assigns each email to a project, extracts tasks and timeline events, and runs background agents. The classification pipeline (classify_emails) proceeds in stages: 1. Noise rules. Domain and pattern lists for newsletters and bot senders, with a separate robot test that a contact in the address book cannot override—a rule added after a notification address entered the address book and became permanently exempt. 2. Self-mail to a dedicated project. 3. Grouping by sender: by domain for organisations, by full address for consumer providers; groups above five messages are split by normalised subject. 4. Type and urgency by rules. At ingest (sync_gmail), each message receives a type and an urgency from a keyword heuristic (Appendix B.4): type is offerta, richiesta or aggiornamento if one of a handful of Italian/English keywords appears in subject or snippet, else altro; urgency is alta or media on a list of urgency words, else bassa. The heuristic can never produce cliente, spam or newsletter; messages later attached to a spam or newsletter project inherit that type. No model is involved in these two fields. 5. LLM pass 1 per group: a prompt with up to 8,000 characters of messages and the list of existing projects (up to 80, with human contacts), returning project name, project type, description, contacts, tasks and timeline as JSON. The LLM therefore decides project assignment, not the per-email type or urgency. 6. LLM pass 2: cross-group merge. 7. Upsert with three matching rules: exact name; same name root plus at least one shared contact; at least three shared contacts. 8. Attachments, mail labels, drafts, suggested tasks, structured contacts. Every decision is logged through a structured decision(kind, outcome, context) call with a flow id, which gives provenance for the error analysis. At the time of the study the LLM backend was a hosted gateway (OpenRouter) with a local Ollama model as fallback; the local 3B model had been demoted after timing out on CPU. The study therefore measures the production system as it actually ran, not as its documentation describes it: reading the code rather than the README revealed both that the local model was no longer primary and that type and urgency were never produced by a model at all.

2 Related Work Typed decision models. Jev [1] defined the interface: a state, a set of typed questions (choice, yes/no, score), and a distribution per question. Open models followed: Bespoke Nimble 9B, fine-tuned from Qwen3.5-9B with LoRA on answer tokens only and released with its contrastively curated data and recipe [2]—its held-out comparison places the untuned 9B base at 66% against 90% for Nimble and 93% for Jev, and its evaluation notes insist that models be compared only on the same examples under the same procedure, which we adopt; Together’s Tev1 4B/0.8B, Cloudflare’s Clef, Amazon’s Strands Decider and Perplexity’s pplx-decider; Ollama 0.35 exposed them locally through /v1/systemone [3]. The LLM-as-Jev framework shows that any causal LLM can be scored under the same convention [4]. An early evidence audit found 29 arXiv papers mentioning typed decision models within nine days of the Jev release and proposes an evaluation checklist [5], which we follow; a broader benchmark places Jev against nineteen general-purpose LLMs [6]. Decision models vs. classical classifiers. On network-flow classification, Random Forest and Extra Trees trained on 8,000 records beat Jev in every test week [7]; a community test of 14 decision models on 56 triage tasks found a plain Qwen3.5-9B ahead of all of them. We take the classical baseline as the cost floor every neural option must beat.

Email classification and personal agents. Email classification predates LLMs by three decades: rule learning for mail filing [9], folder prediction on the Enron corpus [10, 11], reply prediction—“does this message need a reply”—and attachment prediction as the first “intelligent email” tasks [12], the former being exactly our needs-reply switch, and the observation that users need a handful of broad categories rather than many folders [13], which is what our six types and three switches are. That work trained per-user classifiers on a few hundred labelled messages, which is also where 4 Measurement Protocol our baseline sits. What is new is the alternative: zero-shot The stored classification field is written by the model; decisions from a language model, generative or typed, without any per-user data—and the question of whether, for one counting it says nothing about error. We added a inbox, that alternative beats a classifier trained on labels the dataset_email command that: user can produce in an afternoon. Privacy motivates the local • samples stratified by predicted type (seven classes: richisetting: keeping the model where the data are is the premise esta, aggiornamento, offerta, cliente, altro, spam, newsletof federated and on-device learning [14], and personal correter), because 39% of stored labels are altro and 94% of spondence is the canonical example of data that should not urgencies bassa; uniform sampling would teach the prior, 2

Preprint — draft v0.6, 21 October 2026

Lucca

Table 1: Labelled corpus (231 emails after exclusions).

not the content; • freezes a snapshot of exactly what the model sees (sender, subject, body truncated to 1,500 characters, date, channel) next to the model’s prediction, so later re-runs cannot change the example; • assigns splits deterministically by thread, stratified by type: a first version keyed on sha1(id) put 11 threads across splits and gave 4 test emails a near-duplicate in train, so the unit is the thread (normalised subject + sender key), and, while the test split is below 60 emails, the primary protocol is 5-fold thread-grouped cross-validation with the baseline trained out-of-fold; • redacts secrets in the frozen body (passwords, tokens, card and account numbers) before storage—a pilot export contained a VPN password in clear text; • collects human labels in the interface: type, area, project, urgency, needs-reply, action-within-48h, is-noise; • reports, before any training, the only number that matters: how often the production model was right; • pre-registers the hypotheses of each new labelling round: for the third round, eleven hypotheses with thresholds and a verification script were committed before the labelled file was opened (one post-hoc change: a format fix in the script, not a threshold).

Type aggiornamento newsletter richiesta altro offerta spam Urgency bassa / media / alta Projects (incl. none)

n

Relationship

n

132 53 20 18 5 3

servizio cliente me monitoraggio fornitore famiglia / socio / istituzione

115 69 18 14 8 7/6/2

161 / 59 / 11 16 + 1

needs-reply yes action-48h yes / is-noise yes

60 20 / 119

ground truth is therefore a single human reviewer anchored on model proposals. Class distribution in Table 1.

Labels. Two axes were separated after the first round, because the stored type mixed what a message says with who sent it. Type (six classes: richiesta, aggiornamento, offerta, newsletter, spam, altro) describes the message. Relationship (cliente, fornitore, socio, famiglia, istituzione, servizio, monitoraggio, me) describes the sender and is assigned per contact; cliente covers every interaction that can lead to a deal, before or after a signature. Urgency has three levels. Three booleans are the agent’s switches: needs-reply (the agent drafts an answer), action-48h (the agent creates a task) and is-noise (the agents ignore the message; it is kept only In the first round the id-keyed split placed 11 of 92 threads as context). Project is the project the email belongs to as across splits and gave 4 test emails a near-duplicate in train; of October 2026; 26 rows are deliberately empty (“none”) the second round was split by thread and stratified by type because the project-manager agent has not proposed one yet. (train 159 / validation 39 / test 33), with one inherited thread still crossing. Because the fixed test split holds no alta Systems. Production rules: the keyword heuristic of Apand one offerta, every number in this paper comes from pendix B.5 (type, urgency); it has no output for the booleans thread-grouped cross-validation (trained baseline) or from and still emits cliente for 32 rows, wrong by construcall 231 rows (zero-shot systems); the fixed split is kept in tion after the label change. Classical baseline: TF-IDF the repository for future rounds. word 1–2-grams and character 3–5-grams, logistic regression with balanced class weights, 5-fold cross-validation grouped by thread (207 threads), the only trained system. Typed decision models: Tev1 0.8B on CPU (i5-1245UE), Tev1 4B and Nimble 9B (Q8) on a Kaggle T4 via Ollama /v1/systemone, and Bespoke Nimble v4-max through its API. Generative LLMs: Qwen2.5-7B-Instruct, Gemma 4 26B-A4B and Gemma 4 31B through OpenRouter, JSON mode, temperature 0. All systems received the same Italian question and one-line class definitions (Appendix B); the body was stripped of markup and URLs and truncated to 1,500 characters. Zero-shot systems were run once on all 231 rows. The hosted Gemma 4 26B is not deterministic at temperature 0 (0.774, 0.800, 0.791 on the first-round corpus); hosted figures are single-run estimates.

5 Experimental Setup Data. Three labelling rounds on the owner’s production database. Rounds 1–2 were sampled stratified by the type predicted by the production rules: 119 emails on 20 October 2026 (4 skipped as ambiguous) and 120 on 21 October; eight rows marked ignore (private family mail, empty bodies) are excluded, leaving the v2 corpus: 231 emails in 207 threads, our unbiased reference. Round 3 (500 emails, 491 kept) was sampled adversarially: all 118 emails of the unlabelled pool in which Nimble or Gemma 4 saw offerta or spam (G1), 125 on which the models disagreed on action-48h (G2), and 257 at random stratified on the baseline’s prediction (G3). The v3 corpus (722 emails, 698 threads) is therefore a stress test built on the LLMs’ presumed errors, not a sample of the natural inbox; system comparisons are reported on v2, and v3 is used for the pre-registered hypotheses, the contact-graph result and the action analysis. In the first round two model annotators proposed type and urgency and the owner confirmed or corrected each row in a spreadsheet that showed sender, subject, 700 characters of body and both proposals but not the production prediction (110/115 types and 104/115 urgencies survived review); in the second round the owner labelled from the model proposal with the same interface, then the full set was revised jointly with an assistant to apply the final label definitions below. The

Metrics. Accuracy and macro-F1 against the majority class; for systems that return probabilities, mean confidence on correct vs. incorrect decisions; latency as median per decision; cost per 1,000 emails from provider usage. With 231 rows one email is 0.43 accuracy points; offerta (5) and spam (3) make macro-F1 on type unstable, so for type we read accuracy and the F1 of the four large classes.

6 Results The production rules are worse than no classifier. The keyword heuristic that populated the type field of every email 3

Preprint — draft v0.6, 21 October 2026

Lucca

Table 2: Message type (6 classes) and urgency (3 levels), 231 Table 4: Ablations of the trained baseline (thread-grouped 5-fold emails. B = TF-IDF + logistic regression, thread-grouped 5-fold CV, same folds within the table; accuracy / macro-F1). “blind” = CV; all other systems zero-shot. 95% bootstrap intervals on accu- subject and body only, no sender. racy (2,000 resamples of rows). Latency is the median per decision. Input / features

System

Type Acc. [95% CI]

Majority class Production rules (keyword) B: TF-IDF + LogReg Tev1 0.8B, local CPU Tev1 4B, local T4 Nimble 9B Q8, local T4 Nimble v4-max, hosted Qwen2.5-7B, hosted gen. Gemma 4 26B-A4B, hosted gen. Gemma 4 31B, hosted gen.

0.571 0.277 [.22–.34] 0.788 [.74–.84] 0.623 [.56–.68] 0.688 0.619 0.701 [.65–.76] 0.571 [.51–.64] 0.740 [.68–.80] 0.732 [.67–.79]

M-F1

Urgency Acc. [95% CI] M-F1

Latency

0.121 0.205 0.474 0.284 0.386 0.326 0.456 0.286 0.542 0.510

0.697 0.684 [.62–.75] 0.736 [.68–.79] 0.251 [.20–.31] 0.450 0.411 0.714 [.66–.78] 0.437 [.38–.50] 0.628 [.56–.69] 0.645 [.58–.71]

– <1 ms <1 ms 3.8 s 0.35 s 0.58 s 0.50 s 0.48 s 0.76 s 0.72 s

0.274 0.291 0.548 0.235 0.379 0.357 0.541 0.393 0.504 0.497

TF-IDF, with sender TF-IDF, blind MiniLM-L6 (en), with sender MiniLM-L12 multilingual, with sender Majority class

needs-reply (yes 60)

action-48h (yes 20)

is-noise (yes 119)

Majority class B: TF-IDF + LogReg Tev1 0.8B, local Nimble v4-max, hosted Gemma 4 26B-A4B Gemma 4 31B

0.740 / 0.425 0.831 / 0.742 0.706 / 0.465 0.805 / 0.743 0.796 / 0.731 0.805 / 0.748

0.913 / 0.477 0.931 / 0.731 0.840 / 0.578 0.805 / 0.536 0.849 / 0.569 0.840 / 0.562

0.515 / 0.340 0.900 / 0.899 0.628 / 0.626 0.857 / 0.857 0.809 / 0.806 0.771 / 0.766

urgency

needs-reply

action-48h

is-noise

0.71 / 0.41 0.71 / 0.40 0.66 / 0.46 0.62 / 0.40 0.70

0.84 / 0.78 0.83 / 0.76 0.73 / 0.67 0.76 / 0.71 0.74

0.90 / 0.47 0.89 / 0.47 0.84 / 0.52 0.83 / 0.50 0.91

0.89 / 0.89 0.87 / 0.87 0.81 / 0.81 0.84 / 0.84 0.52

Table 5: Relationship inherited from other emails of the same contact (leave-one-out), vs. the majority class. Key same address same domain majority (servizio)

Table 3: The three agent switches, 231 emails (accuracy / macroF1). System

type 0.74 / 0.38 0.70 / 0.28 0.55 / 0.26 0.69 / 0.43 0.57

Covered

Acc. (covered)

M-F1 (covered)

Acc. (all, fallback)

146/231 (63%) 173/231 (75%) –

0.986 0.890 –

0.953 0.574 –

0.857 0.849 0.498

Which differences are real. McNemar’s test on paired per-email correctness (Table 2) separates the comparisons that matter from those that do not. On type, B is significantly better than the production rules (χ2 = 92, p < 0.01) and than Nimble (χ2 = 4.1, p < 0.05), but not than Gemma 4 26B or 31B (χ2 = 1.2, 1.8, n.s.): the trained baseline and the best generative model are tied on type. Gemma 4 26B is significantly better than Qwen2.5-7B (χ2 = 27) and Nimble than Tev1 0.8B (χ2 = 4.4). On urgency, B and Nimble are tied (χ2 = 0.2) and both beat the Gemmas (p < 0.05); B’s lead over the production rules on urgency is not significant (χ2 = 2.6). Bootstrap intervals are ±5–6 points throughout. Among zero-shot systems the ranking depends on the field. On type the generative Gemma 4 26B-A4B is best (0.74, and the best F1 on the four large classes), the hosted decision model Nimble v4-max second (0.70); on urgency and is-noise Nimble is best (0.71, 0.86) and the Gemmas fall to 0.63 and 0.81; on needs-reply Nimble and Gemma 31B tie (0.81). Nimble answers in 0.5 s at about $0.02 per 1,000 emails, Gemma 26B in 0.76 s at $0.03. Qwen2.5-7B—the same open weights a user could run locally—is at or below the majority class on both type and urgency. Giving the generative models examples does not close the gap: with three labelled examples per class in the prompt (drawn from each fold’s training part), Qwen gains one accuracy point and nine macro-F1 points on type, neither significant (p = 0.77, 0.10), and Gemma 4 26B is unchanged (0.74 to 0.73), at six to eight times the cost per email. Local decision models: size matters more than quantised depth. On a T4, Tev1 4B reaches 0.69 on type at 0.35 s per decision, above both Tev1 0.8B (0.62) and the locally served Nimble 9B in Q8 (0.62); the hosted Nimble v4-max does better (0.70), so either the Q8 quantisation or the serving stack costs roughly eight points. All local decision models fail on urgency (0.25–0.45): they assign media or alta to most messages (Tev1 0.8B: 151 and 66 of 231), where bassa is 70% of the labels. A three-level scale phrased as “how urgent” is part of the problem: rephrasing it as “what should the recipient do with this email” lifts Tev1 4B from 0.45 to 0.56 and Nimble 9B from 0.41 to 0.46, still below the majority class; the switches, phrased as actions, fare better (Table 3: Tev1 4B on a T4 reaches 0.79 / macroF1 0.74 on needs-reply). Asking the type question in English instead of Italian does not help either model (Tev1 4B 0.69

in the system reaches 0.28 accuracy against a 0.57 majority class; on urgency it sits just below the majority (0.68 vs 0.70). Because the sample was stratified on the heuristic’s own prediction, this is not an unbiased estimate of its production accuracy, but a field that performs below “always say the most common class” on a sample built to be favourable to it had never carried information. This is the paper’s first result, and it was invisible until someone counted. Two hundred labels and a logistic regression have the highest accuracy on every field—while never predicting the two rarest classes. Out-of-fold TF-IDF + logistic regression leads on type (0.79), urgency (0.74), needs-reply (0.83), action-48h (0.93), is-noise (0.90) and project assignment (0.71 on rows with a project), in under a millisecond. The claim is about accuracy: on type its macro-F1 (0.47) is below Gemma 4’s (0.54) because it never predicts offerta or spam, which have fewer threads than folds. Two ablations qualify the win (Table 4). Removing the sender from the baseline’s input costs four points on type (0.74 to 0.70 in the ablation harness, whose folds differ slightly from the main run), one to two on the switches and none on urgency: most of the advantage is textual, and the sender matters chiefly for altro, which is almost entirely the owner’s own mail (16/18 rows; no zero-shot system reaches F1 0.3 on it, B gets 0.83). Replacing TF-IDF with a sentence-embedding model (MiniLM) and the same classifier does not help: the English all-MiniLM-L6-v2 is below the majority class on type (0.55) and the multilingual paraphrase-multilingual-MiniLM-L12-v2 reaches 0.69, still below the n-gram baseline, which on this corpus benefits from character n-grams over Italian/English morphology and sender tokens. Aggregating the three rare types into one class (altro+offerta+spam, 26 rows) gives a four-class view in which macro-F1 is stable: B 0.57, Tev1 4B 0.58, Nimble 9B 0.50, against 0.57 majority accuracy. 4

Preprint — draft v0.6, 21 October 2026

Lucca

Table 6: Labels composed into actions (v2, policy “noise wins”). (drafts: precision 0.80, recall 0.47; tasks: 0.71 / 0.25), the False noise = emails needing a reply or task that were ignored, of LLMs recall more and over-act (Nimble creates 28 useless 68. tasks in 33, Gemma 19 in 24; the baseline 2 in 7). Inheriting System

Draft P / R

Task P / R

False noise

Useless tasks

B: TF-IDF + LogReg Nimble v4-max Gemma 4 26B-A4B Gemma 4 31B Tev1 0.8B

0.80 / 0.47 0.65 / 0.61 0.62 / 0.59 0.63 / 0.64 0.31 / 0.07

0.71 / 0.25 0.15 / 0.25 0.21 / 0.25 0.19 / 0.25 0.35 / 0.30

12/68 (18%) 8/68 (12%) 3/68 (4%) 5/68 (7%) 25/68 (37%)

2/7 28/33 19/24 21/26 11/17

the relationship from the contact and applying the monitoring rule raises task recall for every system (B 0.25 to 0.40, Gemma 0.25 to 0.65) at no cost in false noise; the oracle relationship adds nothing more, so the relationship is not the bottleneck. Which system is right depends on the price of a missed email versus a useless task. Making that price explicit settles it: with a useless task costing 1, a useless draft 0.5 and a missed email c, the baseline has the lowest total cost only while c < 3; from c ≈ 3 upward Gemma 4 26B is cheapest at every price, and at c = 10 its cost is less than half the baseline’s (60 vs 126 on v2). For a personal agent, where a missed client email is worth far more than a task to dismiss, that favours the generative model out of the box. But the baseline’s false-noise rate is a threshold, not a ceiling: choosing the is-noise threshold inside the cross-validation to minimise the same cost (missed email 10, email handled needlessly 1) halves it—on v3 from 28/120 to 13/120, on v2 from 12/68 to 7/68, and to 4/68 with Platt scaling—at the price of handling two to three times more noise. With that threshold the baseline’s false noise is no longer distinguishable from Nimble’s (v3: 0.108 vs 0.092, p = 0.72; v2: 0.059 vs 0.118, p = 0.34) and its total cost is lower (v3: 225 vs 250; v2: 79 vs 99), while it keeps its accuracy lead. Calibration itself is not what does the work—a cost-chosen threshold depends only on the ranking of probabilities—but it is what makes the probabilities safe to display or combine. Calibration. The hosted decision model is the bestcalibrated system on every field (ECE 0.04–0.11 on v2, 0.04–0.09 on v3), and it is deterministic (491/491 identical answers on a repeated run). The trained baseline is strongly under-confident: on type its mean confidence is 0.32 for an accuracy of 0.79 (ECE 0.47), an artefact of balanced class weights on few rows; recalibrating it inside the cross-validation (Platt or isotonic, fitted on each outer fold’s training part) cuts its ECE by a factor of 4–17 (type: 0.47 to 0.11 on v2, 0.37 to 0.05 on v3; is-noise: 0.29 to 0.03), to the same range as the hosted decision model, at a cost of five accuracy points on type when the corpus is small (231 rows) and none when it is larger (722). The local decision models are over-confident on errors (Tev1 4B 0.81 right vs 0.70 wrong on type; Nimble 9B Q8 0.88 vs 0.76). The adversarial corpus is a stress test, not a second sample. On v3 (722 emails) the baseline’s lead over Gemma 4 26B on type becomes significant (+0.136, p < 0.001), as does its lead over Nimble on every switch; the contact-graph relationship reaches 0.927 on the 73% of covered emails and 0.898 overall (coverage is inflated on v3, where G3 was drawn by sender strata and many senders recur: the v2 figures, 63% covered and 0.857 overall, are the ones to carry to production); the production rules fall to 0.17 on type; the local decision models on a T4 fall below the majority class on every field (Tev1 4B 0.52 on type, Nimble 9B Q8 0.50; 0.40 and 0.34 on urgency), as do their hosted and generative counterparts on type (Nimble v4-max 0.56, Gemma 4 26B 0.65 against 0.58). But v3 was sampled on the LLMs’ presumed errors, and the baseline was the first proposal shown to the reviewer, so these gaps are

+ relationship from contact and monitoring rule: task recall B 0.40, Nimble 0.50, Gemma 26B 0.65

Table 7: Project assignment, 16 projects + none; models see only the project names. System Majority (personale) B: TF-IDF + LogReg Tev1 0.8B Nimble v4-max Gemma 4 26B-A4B Gemma 4 31B

Acc. (all)

Acc. (with project, n=205)

none P / R

0.316 0.697 0.260 0.303 0.329 0.468

– 0.712 0.234 0.259 0.278 0.468

– 0.75 / 0.58 0.22 / 0.46 0.13 / 0.65 0.13 / 0.73 0.21 / 0.46

to 0.64, Nimble 0.62 to 0.64). action-48h is the hard switch. With 20 positives in 231, no zero-shot model beats the majority class in accuracy (0.91); all beat it in macro-F1 (0.54–0.58 vs 0.48), B reaches 0.73. This is the switch with the highest cost of error—a task created or not—and the one where a classifier needs the most examples. The relationship belongs in the contact graph. Where a contact has at least one other labelled email (146 of 231, 63%), inheriting the relationship from the same address is right 98.6% of the time; the remaining 37% of emails come from senders seen once, for whom a majority fallback gives 0.857 overall against 0.498 for the majority class (Table 5). In production the covered fraction grows with every labelled contact, and a new sender is exactly the case where the agent should ask rather than guess. Matching on domain covers more (75%) but errs more (0.890): gmail.com mixes the owner, clients and partners. As a negative control we did ask two local decision models to infer the relationship from the text of one message: Tev1 4B reaches 0.52 and Nimble 9B 0.55 against a 0.50 majority class—no better than guessing—where inheritance from the contact gives 0.86. The agent’s production logic is being changed to inherit it from the contact (Section 3). Project assignment needs descriptions, not names. Given only the sixteen project names, zero-shot models score 0.26–0.47 and over-use none (precision 0.13–0.22); the baseline, which learns from the other emails of a thread’s neighbours, reaches 0.70. A fair zero-shot comparison requires a description per project written by the owner rather than derived from the labels; it is left for the next round. Accuracy is not the product metric. The labels compose into one of four actions per email: draft a reply, create a task, ignore, or nothing, with the rule that is-noise wins. Scored that way (Table 6), the error that costs most—false noise, an email that needed a reply or a task and was ignored— inverts the ranking. On v2, of 68 emails that needed action, the trained baseline ignores 12 (18%), Nimble 8 (12%), Gemma 4 26B 3 (4%), Tev1 0.8B 25 (37%); on v3, B 23% against Nimble 9%. The baseline is precise and conservative 5

Preprint — draft v0.6, 21 October 2026

Lucca

upper bounds, not estimates. The pre-registered hypotheses give the honest summary: seven of eleven confirmed, four refuted—among the refuted, that rare commercial classes could be found by sampling where the LLMs see them (all 118 such emails yielded 3 offerta and 7 spam: the models over-predict commercial mail, the inbox simply has little of it after Gmail’s own filtering), and that the baseline would learn offerta with eight examples (its F1 stays 0). Jev was not run for lack of an API key; Nimble’s public demo was offline and the paid API was used; GPU runs were made on Kaggle’s free T4 with the same script and data.

the owner reviewed labels proposed by model annotators rather than labelling from scratch (110/115 first-round type proposals survived review), and the final label definitions were applied in a joint revision with an assistant, which may reflect anchoring as much as agreement (no independent second annotator). 231 labels give 95% confidence intervals of roughly ±6 points on accuracy, so differences below eight points between systems are not significant; the gap between the production heuristic and every other system, and between the trained baseline and the zero-shot models on is-noise, are. The sample was stratified on the production heuristic’s prediction, so its accuracy is not a production estimate. The data cannot be released; the protocol, the harness and aggregate statistics can. Round 3 was sampled on the LLMs’ presumed errors and shown with the baseline’s proposal first; v3 comparisons inherit both biases. Three gaps are known and scheduled for the next round: a second annotator blind to model proposals on a 60-email subset, with Cohen’s κ; a reweighting of the zero-shot accuracies by the production distribution of the heuristic’s predictions, to correct the stratified sample; and a project-assignment run with a one-line description per project written by the owner, since names alone are not a fair zero-shot input (Table 7). Few-shot prompting and recalibration of the baseline, raised by early reviewers, have been run and are reported above. The generative naming step for new projects was run through a hosted endpoint serving open weights (Qwen2.57B-Instruct) that are deployable locally; local latency for that step is reported on a -email subsample only. Local GPU results come from Kaggle’s free T4, not from the owner’s hardware; the locally served Nimble is the Q8 build. The hosted generative model was not deterministic across runs at temperature 0 (0.774–0.800 on type), so hosted figures should be read as single-run estimates.

7 Error Analysis From the divergences between the six systems we counted, per cause, the emails on which at least three systems err in the direction of that cause (ERRORI.md). 1. Relational classes. In the first round all 5 cliente emails were misread as richiesta or aggiornamento by at least three systems; after the label change, altro (mostly the owner’s own mail) plays the same role: no zero-shot system reaches F1 0.3 on it. Who the sender is to the owner is not in the text of one message; it lives in the contact graph (Table 5). 2. Automated senders are not the problem. We expected deploy, CI and console notifications to be misfiled; on the reviewed labels no email with an automated sender is misclassified by any system. The rule-based robot filter in front of the models already removes the hard cases, and the models handle the rest. 3. Thin text. Bodies under 20 characters are now excluded as ignore; in the first round every system failed on them on both fields. 4. Debatable labels. On 9 type labels and 5 urgency labels at least four of the six systems agree with each other against the reviewer. These are candidates for a second annotator rather than evidence of model error; at 8–12% of the corpus they bound the achievable accuracy. 5. Rare commercial classes are rare. Sampling every email in which two LLMs saw offerta or spam (118 of a 1,322-email pool) produced 3 and 7 true cases: Nimble’s precision on spam is 0.10. The models over-predict commercial mail; the inbox, filtered upstream by Gmail, has little of it. 6. Taxonomy drift. The stored type mixes message kind (request, update, offer) with relationship (client, spam), which is the root of cause 1.

9 Conclusion Two days of measurement on 231 emails showed that the field a personal agent displayed most confidently had never carried information; that two hundred human labels and a logistic regression beat every zero-shot model—decision models and generative LLMs, local and hosted—on every one of the agent’s decisions, at negligible cost; that among zero-shot options a hosted typed decision model is the best choice for the agent’s switches and a mid-size generative model for message type, with a 4B local decision model on a consumer GPU close behind; that the one label no model should be asked for, the sender’s relationship, is recovered almost perfectly from the contact graph; and that the most accurate system is also the one that most often ignores an email that needed action, so the choice between a classifier and a model is a choice about the price of errors, not about accuracy—and once that price is written down, a threshold chosen for it makes the classifier the cheapest option again. The protocol—stratified sampling, frozen snapshots, threadgrouped splits, secret redaction, counting before training— cost less than the labelling and is reusable for any personal agent. The production system is being changed accordingly: type and urgency will no longer be rule-based, the relationship will be inherited from the contact, and the three

8 Limitations Single user, single inbox, two languages. What would not transfer to another user is clear from the components: the baseline’s weights (they encode this owner’s senders and vocabulary and must be retrained from that user’s labels), the contact graph (empty for a new user until contacts recur, which is why coverage, not accuracy, is the quantity to watch), the class priors (70–82% bassa here) and any confidence threshold. What should transfer is the protocol itself, the zero-shot behaviour of the decision models, and the finding that the sender’s relationship is a graph property; the second inbox needed to test this is the obvious next study; 6

Preprint — draft v0.6, 21 October 2026

Lucca

A Fixed-split results

switches will drive the agent’s actions directly; the change is in progress at the time of writing and will be reported with the next labelling round.

The fixed thread-grouped split (train 159 / validation 39 / test 33) is recorded in the labelled JSONL and in the repository’s result files; it is not used for any figure in this paper Reproducibility. Code: dataset_email.py, because the test split holds no alta and a single offerta. It is bench_cv_pilota.py, bench_online_pilota.py, retained so that future labelling rounds can report a held-out bench_prob_pilota.py, relazione_contatto.py, number once the small classes have enough examples. bench_project.py, genera_report_v2.py; a README with every command, environment variable name B Questions, definitions and prompts (verbaand result hash; run log and result JSONs in docs/paper/. tim) GPU runs: a single-cell Kaggle notebook (T4) with the All systems used the same Italian question and definisame questions. Models: tev1:0.8b, tev1, nimble on tions, from bench_tev1_pilota.py; English glosses in Ollama 0.40; scikit-learn 1.9. brackets.

References [1] D. Almeida. Introducing System One Models & Jev. TypeSafe AI blog, 15 Sept. 2026. [2] Bespoke Labs, S. Pimpalgaonkar, P. Dalal, M. Sathiamoorthy. Nimble: data, model, recipe for an open Jev. https: //github.com/bespokelabsai/nimble, 2026. [3] Ollama. Ollama now supports Jev-style decision models. Blog, 29 Sept. 2026. [4] Y. Li, J. Wagle. LLM-as-Jev: LLMs are already Jevstyle decision models—when and how to fine-tune them. arXiv:2610.02076, 2026. [5] Typed Decision Models: An Early Evidence Audit and Evaluation Checklist. arXiv:2609.32160, 2026. [6] Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks. arXiv:2610.11978, 2026. [7] A First Glance at Jev for Network Traffic Classification: Accuracy, Processing Time, and Cost. arXiv:2610.00376, 2026. [8] Jev in the Wild: A Data-Driven Analysis of the Jev Model’s Functionality, Applications and Ecosystem. arXiv:2609.30216, 2026. [9] W. W. Cohen. Learning rules that classify e-mail. AAAI Spring Symposium on Machine Learning in Information Access, 1996.

B.1 Type. Question: “Che tipo di messaggio è questa email per il suo destinatario?” [What kind of message is this email for its recipient?] • richiesta: chi scrive chiede al destinatario qualcosa, o serve una sua azione o risposta [the sender asks something of the recipient, or an action or reply is needed] • aggiornamento: informazione, notifica, conferma o esito senza richiesta (codici di sicurezza, avvisi di accesso, prenotazioni, report di stato, thread in copia) [information, notification, confirmation or outcome with no request] • offerta: proposta commerciale mirata al destinatario (preventivo, offerta personale) [a commercial proposal aimed at the recipient] • cliente: corrispondenza con un cliente anche potenziale su un incarico professionale [correspondence with a client, including a prospective one, about a professional engagement] • newsletter: invio di massa periodico o promozionale da un brand o servizio [periodic or promotional mass mailing] • spam: commerciale non richiesto da mittente sconosciuto, o phishing [unsolicited commercial mail from an unknown sender, or phishing] • altro: tutto il resto, compresi i messaggi inviati dal destinatario stesso [everything else, including messages sent by the recipient]

[10] B. Klimt, Y. Yang. The Enron corpus: a new dataset for email classification research. ECML, 2004.

B.2 Urgency. Question: “Quanto è urgente agire su questa email?” [How urgent is it to act on this email?] alta: serve un’azione entro circa 48 ore o c’è un appuntamento in gior[11] R. Bekkerman, A. McCallum, G. Huang. Automatic catenata [action within about 48 hours, or an appointment today]; gorization of email into folders: benchmark experiments on Enron and SRI corpora. CIIR Technical Report IR-418, Uni- media: serve un’azione o una lettura attenta senza scadenza vicina [action or careful reading, no near deadline]; bassa: versity of Massachusetts Amherst, 2004. nessuna azione richiesta [no action required].

[12] M. Dredze, T. Brooks, J. Carroll, J. Magarick, J. Blitzer, F. Pereira. Intelligent email: reply and attachment prediction. B.3 Proc. IUI, Gran Canaria, 2008.

question

Decision-model request. One choice per call: {"model": m,

[13] M. Grbovic, G. Halawi, Z. Karnin, Y. Maarek. How "state": {mittente, oggetto, corpo}, many folders do you really need? Classifying email into "questions": {campo: {"type": "choice", a handful of categories. Proc. CIKM, pp. 869–878, 2014. "instructions": DOMANDA[campo], "criteria": doi:10.1145/2661829.2662018. CRITERI[campo]}}} against /v1/systemone (Ollama

locally, Bespoke’s API for Nimble). Body cleaning: style/script blocks, tags and URLs removed, whitespace collapsed, truncated to 1,500 characters.

[14] B. McMahan, E. Moore, D. Ramage, S. Hampson, B. Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. AISTATS, 2017. 7

Preprint — draft v0.6, 21 October 2026

Lucca

B.4 Generative prompt (OpenRouter). System message = question + “Scegli esattamente una etichetta. Definizioni:” + the definitions above, one per line + “Rispondi SOLO con JSON: {"tipo": "<etichetta>"}”; user message = the cleaned email as JSON; temperature 0, max_tokens 60, response_format json_object. Answers outside the vocabulary were counted as errors (1 of 230 calls for Qwen, 0 for Gemma).

unrelated to me. altro: the rest, including messages sent by me. Urgency: alta action within about 48 hours or an appointment today; media action or careful reading without a near deadline (security notices, time confirmations, failed builds); bassa no action. The annotator saw sender, subject and the first 650 characters of the body, without the production prediction; for empty or markup-only bodies it decided from subject and sender.

B.5 Production heuristic (sync_gmail._classify_email). Lower-cased subject + snippet. Type: offerta if any of {offerta, preventivo, price, quote, proposta}; else richiesta if any of {richiesta, request, help, supporto, assistenza}; else aggiornamento if any of {aggiornamento, update, status, ticket, issue}; else altro. Urgency: alta on {urgente, urgent, asap, subito, critico, critical, scadenza, deadline, entro oggi, entro domani, immediate, priorità alta, high priority, bloccante, blocking}; media on {importante, important, soon, presto, entro la settimana, this week, priorità media, follow up, follow-up, in attesa}; else bassa. The function’s own docstring reads “classificazione euristica semplice—da sostituire con Ollama quando disponibile” [simple heuristic classification—to be replaced by Ollama when available].

C Labelling procedure and guidelines Procedure. A review spreadsheet with one row per email and the columns: id, split, sender, subject, body (700 characters, whitespace normalised), date, first proposal (type, urgency, confidence, note), second proposal (type, urgency), disagreement, confirm, corrected type, corrected urgency, area, project, needs-reply, action-48h, is-noise, skip, reviewer notes. Rows were ordered with disagreements and low-confidence proposals first, then by id. The production prediction was deliberately not shown. A row counts as reviewed when confirm is set or a corrected value is given. Four rows were marked skip because the reviewer found them ambiguous, and were excluded. The reviewer went through the rows in the spreadsheet order (disagreements first) and reports filling the optional columns (area, project, needs-reply, action-48h, is-noise) on every row; the project column did not survive the import into the label table in this round and is being re-imported. The reviewer applied cliente more narrowly than the written guideline: “someone who gives me work”, i.e. an actual paying client rather than a prospective one. Guidelines given to the model annotator (translated). richiesta: the sender asks me for something, or my action or reply is needed (including an automatic notice that says “action required”). aggiornamento: information, notification, confirmation or outcome with no request: bookings, security codes, login alerts, updated calendar invitations, status reports, legal or pricing notices from a service, threads seen in copy. offerta: a commercial proposal aimed at me. cliente: correspondence with a client (including prospective) about an ongoing professional engagement. newsletter: periodic or promotional mass mailing from a brand or service I subscribed to. spam: unsolicited commercial mail from a sender 8

Related documents

• Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox#1448470
• Toollery: Scaling LLM Agents to Thousands of Skills and Tools#1036901
• One Size Does Not Fit All: Setting Inference Depth from the Questions a Deployment Actually Asks#1083613
• EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks#1115862
• Forty Questions: Identifying a Language Model from One-Word Answers#1412252