ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems Daeyeon Son Independent Researcher Republic of Korea [email protected]
arXiv:2604.11943v2 [cs.OS] 18 Apr 2026
April 2026
Abstract
harder to circumvent than application-layer classifiers. Each classification costs 65 ms on 7B in our bare-metal runtime (or ∼400 ms hosted under llama-cpp-python), fast enough for per-action governance. By comparison, the nearest fine-tuned baseline (Llama Guard 3 8B) requires ∼1 s per check in the same hosted environment, because it must generate response tokens rather than read a single logit position. I also show that treating KV cache as process state enables checkpoint, restore, and fork operations analogous to traditional process management. To my knowledge, no prior system exposes LLM logit vectors as OS-level governance primitives. ProbeLogits is the substrate primitive on which a kernel-resident tool governance gateway (Governed MCP) is built; the gateway is described in a companion paper.
An OS kernel that runs LLM inference internally can read logit distributions before any text is generated— and act on them as a governance primitive. I present ProbeLogits, a kernel-level operation that performs a single forward pass and reads specific token logits to classify agent actions as safe or dangerous, with zero learned parameters. I evaluate ProbeLogits on three base models (Qwen 2.5-7B, Llama 3 8B, Mistral 7B) across three external benchmarks (HarmBench [30], XSTest [32], ToxicChat [25]). On HarmBench non-copyright, all three models reach 97–99% block rate with the right verbalizer. On ToxicChat (n=1,000), ProbeLogits achieves F1 parity-or-better against Llama Guard 3 across the three families: the strongest configuration (Qwen 2.57B Safe/Dangerous, α = 0.0) reaches F1 = 0.812 with bootstrap 95% CIs disjoint (+13.7 pp significant over LG3). Llama 3 S/D matches LG3 within CI (+0.4 pp, parity), and Mistral Y/N exceeds by +4.4 pp. Latency is ∼2.5× faster than LG3 in the same hosted environment because the primitive reads a single logit position instead of generating tokens; in the bare-metal native runtime ProbeLogits drops to 65 ms (§7.4, §7.6). A key design contribution is the calibration strength α, which serves as a deployment-time policy knob rather than a learned hyperparameter. By adjusting α, the OS can enforce strict policies for privileged operations (higher α, maximizing recall) or relaxed policies for conversational agents (lower α, maximizing precision). Contextual calibration [27] corrects verbalizer prior asymmetry, with bias magnitude varying by (model, verbalizer) pair (e.g., Llama 3 Y/N −0.54 vs. S/D −2.54, a 4.7× shift; §7.6). I implement ProbeLogits within Anima OS, a baremetal x86_64 OS written in 86,000 lines of Rust. Because agent actions must pass through kernel-mediated host functions, ProbeLogits enforcement operates below the WASM sandbox boundary, making it significantly
1
Introduction
Operating systems have always treated processes as opaque entities. The kernel manages their resources— memory, CPU time, file descriptors—but never inspects what a process thinks. This was a reasonable abstraction for decades: a process’s internal state is its own concern, and the OS need not understand it to schedule, isolate, or terminate it. But large language models break this assumption. At every inference step, an LLM produces a probability distribution over its entire vocabulary—a logit vector that reveals the model’s uncertainty, intent, and confidence before any text is generated. This is a fundamentally new kind of process state: a semantic signal that the OS could use for scheduling, safety, and governance decisions. Today’s operating systems discard this information entirely. The consequences are threefold. Latency accumulation: when AI safety checks require generating text, parsing it, and then acting on the parse result, each abstraction layer adds overhead. A simple “is this action safe?” query traverses application code, IPC, 1
inference framework, text generation, and parsing— accumulating latency that makes per-action governance impractical. Bypassable safety: when safety policies are enforced in application code, any agent with sufficient privilege (or a bug in the sandbox) can bypass them. The enforcement boundary is too high in the stack. Semantic opacity: the OS kernel cannot make informed decisions about AI workloads because it has no access to model internals—logits, hidden states, uncertainty—that would enable intelligent scheduling and governance. This paper presents ProbeLogits, an OS primitive that addresses all three problems by exposing LLM logit vectors as kernel-level abstractions. ProbeLogits performs a single forward pass on a prompt and reads specific token logits to classify actions without generating any text. The classification technique itself—reading logits at class-label token positions (known as verbalizers in NLP) with contextual calibration—is well-established in NLP [24, 27]. The contribution of this work is not the ML technique but its integration as a kernel primitive with structural enforcement guarantees. When logit reading runs inside the kernel and agent actions must traverse kernel-mediated host functions, safety checks become kernel-mediated—a defensein-depth property that application-layer classifiers cannot easily provide. Concretely, I make three contributions:
ProbeLogits classification costs 0.6 ms on 135M and 65 ms on 7B—fast enough for per-action governance.
2
Motivation and Background
2.1
The Layered AI Problem
Contemporary AI systems are built as deep layer cakes: hardware → OS kernel → userspace runtime → inference framework → orchestration library → application. Each layer adds latency, attack surface, and opacity. A safety check in LangChain [6] must traverse Python function calls, HTTP requests to an inference server, text generation, regex parsing of the response, and a policy decision—all before the agent’s action can be approved or denied. This architecture is a consequence of treating AI as an application-level concern rather than a kernel-level one.
2.2
Three Problems in Detail
Problem 1: Latency. Consider the task of classifying whether an agent’s intended action is safe. In a text-based approach, the system constructs a prompt (“Is deleting all user data safe? Answer Yes or No.”), sends it to an inference server, waits for text generation (∼650 ms for a multi-token response on 7B), parses the 1. ProbeLogits as an OS primitive. I de- output, and acts on the result. With ProbeLogits, the fine binary, N-way, and entropy operations as same classification requires one forward pass (65 ms on kernel abstractions, and show that a general- 7B) with no text generation, no parsing, and no IPC—a purpose 7B model with zero learned fine-tuning 10× reduction. achieves F1 parity with Llama Guard 3 on ToxicChat [25] (1,000 prompts; bootstrap 95% CIs overlap) and 97–99% block rates on HarmBench [30] Problem 2: Bypassability. Application-level safety non-copyright across three model families. To my filters—including prompt injection defenses, output knowledge, this is the first system to expose LLM classifiers, and policy checkers—operate at the same privilege level as the agent they are meant to constrain. logits as OS primitives. A sufficiently capable agent can modify, disable, or cir2. KV cache as process state. I treat the KV cumvent these filters. Kernel-level enforcement is fundacache as the analogue of CPU register state, en- mentally different: an agent running in a WASM sandabling checkpoint, restore, and fork operations that box cannot bypass governance checks without first esparallel traditional process management. caping the sandbox, because the checks are executed by the kernel on the agent’s behalf, outside the sandbox 3. Kernel-enforced constitutional governance. boundary. I demonstrate a governance pipeline (adversarial pre-filter → ProbeLogits classification) that is kernel-enforced below the WASM sandbox boundProblem 3: Semantic opacity. To a conventional ary, making it significantly harder to circumvent OS, an LLM inference workload is indistinguishable than application-layer classifiers. from any other compute task. The kernel cannot tell I implement these primitives in Anima OS, a bare- whether the model is confident or uncertain, whether metal x86_64 operating system comprising 86,000 lines the output is safe or dangerous, or whether the agent of Rust. The inference engine achieves 1,666 tokens/s on is behaving anomalously. This opacity prevents the SmolLM2-135M (1.39× llama.cpp) and 15 tokens/s on OS from making semantically informed decisions about Qwen2.5-7B (parity at DDR5 bandwidth saturation). scheduling priority, resource allocation, or isolation. 2
2.3
Algorithm 1 ProbeLogits N-way Classification
Threat Model
I assume the following threat model: agents execute in Require: Prompt p, class labels C = [c1 , . . . , cN ] WASM sandboxes with fuel-limited execution and no di- Ensure: Sorted class probabilities 1: for i ← 1 to N do rect memory access outside their sandbox. The kernel is 2: ti ← TextToId(ci ) ▷ O(log |V |) BTreeMap trusted. The network is untrusted (all inter-node comlookup munication uses authenticated protocols). Agents may 3: if ti = None then return None be adversarial: they may attempt to circumvent safety 4: end if policies, exfiltrate data, or consume excessive resources. 5: end for The governance system must provide structural guar6: tokens ← Encode(p) antees (not just probabilistic detection) against policy 7: ResetKvCache() circumvention. 8: for tok ∈ tokens do 9: ℓ ← ForwardOne(tok) ▷ Incremental forward 10: end for 3 ProbeLogits Design 11: ℓmax ← maxi ℓti exp(ℓti − ℓmax ) for each i 3.1 The Problem with Text-Based Clas- 12: ei ← P 13: S ← i ei sification 14: if S ≤ 10−10 then P (ci ) ← 1/N for all i ▷ Uniform fallback Text-based classification—generating a response and 15: parsing it—suffers from three failure modes that make 16: else 17: P (ci ) ← ei /S for each i it unsuitable for kernel-level governance: 18: end if 1. Unexpected format. The model may respond 19: return SortDescending({(ci , P (ci ), ℓti )}) “No, I don’t think that would be appropriate” instead of the expected “No.” Parsing extracts “No” but the actual answer is buried in natural language This is an N -way softmax restricted to the target tothat may vary unpredictably. ken positions. The key property is that exactly one for2. Fragile parsing. Regular expressions and string ward pass suffices regardless of N —the cost is identical matching break on whitespace variations, capital- for binary classification and 100-way classification. ization differences, or tokenizer artifacts. Each failI implement three operations as kernel primitives: ure mode requires a new parsing rule. Binary classification (probe_yes_no). Looks up token IDs for “Yes” and “No” via text_to_id(), runs one forward pass, and computes a 2-class softmax. Returns the winning class and confidence ∈ [0.5, 1.0]. A numerical guard handles the edge case where both logits underflow: if exp(ℓyes ) + exp(ℓno ) ≤ 10−10 , the result ProbeLogits eliminates all three failure modes by defaults to confidence 0.5 (maximally uncertain). reading the logit vector directly, before any text is generated. N-way classification (probe_classify). Generalizes binary classification to N classes. Each class la3.2 ProbeLogits Mechanism bel must resolve to a single token in the vocabulary. Let p be a prompt (a sequence of tokens), V be the Returns a vector of ClassResult sorted by probabilguard applies: if vocabulary, and f be the model’s forward pass. Pro- ity (descending). The same numerical the softmax denominator is ≤ 10−10 , all classes receive beLogits computes: uniform probability 1/N . ℓ = f (p) ∈ R|V | (1) Token vocabulary lookup (text_to_id). Maps a where ℓ is the logit vector over the full vocabulary. For text string to its token ID using a BTreeMap over a set of target classes C = {c1 , . . . , cN }, each mapped to the GPT-2 byte-pair encoding vocabulary. Cost: a token ID tci via vocabulary lookup, the classification O(log |V |), where |V | = 152,064 for Qwen2.5’s GPTprobability for class ci is: 2 BPE vocabulary. Returns None if the text does not correspond to a single token. exp(ℓtci ) Algorithm 1 presents the complete probe_classify (2) P (ci ) = PN exp(ℓ ) procedure. tc j j=1 3. Wasted computation. Generating a multi-token response requires multiple forward passes (one per token). Classification needs only the first token’s logit distribution—all subsequent tokens are wasted computation.
3
3.3
• Autonomy gating: when H > 8 nats (high uncertainty), the kernel can automatically defer the decision to a human operator or a larger model.
Information Efficiency
A single forward pass produces a logit vector over the full vocabulary of |V | = 152,064 tokens. This vector carries log2 (|V |) ≈ 17.2 bits of information—the maximum entropy of the distribution. Standard text generation uses this vector to sample one token, discarding > 99.99% of the available information. ProbeLogits extracts precisely the bits needed for the task at hand. A binary classification extracts 1 bit (plus confidence). An N-way classification extracts log2 N bits. In both cases, the information is extracted from a single forward pass, whereas text-based classification requires multiple forward passes (one per generated token) to recover the same information through text parsing.
• Out-of-distribution detection: anomalously high entropy on routine prompts signals that the model is encountering inputs outside its training distribution. • Dynamic temperature: the kernel can adjust sampling temperature inversely to entropy, producing more deterministic outputs when the model is already confident.
3.6
Grammar-Constrained Decoding
For tasks that require multi-token structured output (rather than single-token classification), I impleProbeLogits provides four robustness guarantees that ment grammar-constrained decoding as a complementary primitive. The constraint engine operates by masktext-based classification cannot: ing logits before sampling: 1. No parsing failures. The output is always a ( floating-point probability in [0, 1]. There is no text ℓi if token i is valid ′ to parse, no regex to fail, no format to validate. ℓi = (4) −∞ otherwise 2. Bounded confidence. The softmax output is always in [0, 1] by construction. Binary classification For choice grammars (selecting one of N string opconfidence is always in [0.5, 1.0]. tions), the masking algorithm maintains the set of re3. Graceful degradation. When the model is un- maining valid completions given the tokens generated certain, the confidence approaches 0.5 (binary) or so far. At each step, a token is valid if appending its 1/N (N-way). This is a meaningful signal—the sys- text to the generated prefix would still be a prefix of tem can defer to a larger model or to human judg- at least one remaining choice. This prefix matching operates over character sequences, not token sequences, ment. which correctly handles cases where choices share com4. Numerical stability. The log-sum-exp trick mon prefixes. prevents overflow. The uniform fallback when I acknowledge a limitation: the masking is not a 100% P exp(·) ≤ 10−10 prevents division by zero. f64 guarantee for arbitrary tokenizer/choice combinations. accumulation in entropy computation prevents pre- GPT-2’s byte-pair encoding covers all single ASCII cision loss with large vocabularies. characters, so ASCII choices work reliably. For nonASCII choices or unusual tokenizer vocabularies, I recommend using probe_classify() (single-token classi3.5 Logit Entropy fication) where deterministic results are needed, and reBeyond classification, the full logit vector carries a serving grammar constraints for multi-token structured scalar summary of model uncertainty: Shannon entropy. output. The overhead of grammar masking is negligible: < |V | X exp(ℓi ) 0.01 ms per token, compared to 65 ms for the forward (3) H(ℓ) = − pi ln pi , pi = P exp(ℓ ) pass itself. The mask is recomputed at each generaj j i=1 tion step, but the vocabulary scan (152K tokens) is fast For a vocabulary of 152,064 tokens, entropy ranges because it involves only string prefix comparison. from 0 nats (complete certainty—one token has probability 1) to ln(152,064) ≈ 11.93 nats (uniform distribution—maximum uncertainty). 4 KV Cache as Process State I implement logit_entropy() as a kernel primitive with f64 accumulation for numerical stability across the 4.1 The Process State Analogy full 152K-token vocabulary. The computation uses the log-sum-exp trick (subtract maxi ℓi before exponentia- Traditional operating systems manage process state tion) and skips terms where pi < 10−10 to avoid ln(0). through a well-established set of abstractions: CPU regLogit entropy enables three OS-level capabilities: isters are checkpointed on context switch, the program
3.4
Robustness Properties
4
counter tracks execution position, and fork() duplicates the entire process state to create a child. I observe that LLM inference has a direct analogue:
• Speculative execution: fork the KV cache, explore a candidate action, and restore if the action is rejected by governance.
Table 1: Process state analogy between traditional OS and LLM inference primitives.
• Agent forking: create a new agent with the same conversational context, diverging from a specific point in the conversation.
Traditional Process
LLM Inference
CPU registers Program counter fork() Context switch Core dump
KV cache (attention state) KV position (tokens processed) kv_fork() kv_checkpoint/restore() KV cache serialization
• Migration: serialize the KV cache, transfer it to another node, and restore—enabling live agent migration across a cluster. • Conversation branching: checkpoint before a critical decision, explore multiple branches, and select the best outcome.
This analogy is not merely conceptual—it has direct implementation consequences. Just as an OS must checkpoint registers to switch between processes, an AI-native OS must checkpoint KV cache to switch between inference contexts. Just as fork() enables speculative execution in traditional systems, kv_fork() enables speculative decoding and conversation branching.
4.2
4.4
vLLM [19] and SGLang [17] also manage KV caches, but for a fundamentally different purpose. Their PagedAttention and RadixAttention optimize serving throughput—maximizing the number of concurrent requests that fit in GPU memory. My KV cache operations optimize process state management—enabling checkpoint, restore, and fork as OS primitives for individual agents. The two approaches are orthogonal: a system could use PagedAttention for memory efficiency and KV process state operations for agent lifecycle management.
Operations
I implement three KV cache operations: kv_checkpoint(). Snapshots the current KV cache state (key and value tensors for all layers, up to the current position) into a KvCheckpoint structure. The operation uses checked_mul to prevent integer overflow when computing checkpoint size, and enforces a 32 MB cap (MAX_CHECKPOINT_BYTES) to prevent memory exhaustion. Returns an error if the checkpoint would exceed the cap.
5
Constitutional Governance Integration
5.1
Why Kernel-Level Matters
Enforcement
In Anima OS, agents execute in WASM sandboxes compiled via Cranelift AOT. An agent interacts with the outside world exclusively through 15 host functions exposed by the kernel (12 base operations plus 3 MCP tool-calling functions added in the companion governance gateway [29]). This architecture provides a structural guarantee: the agent cannot bypass governance checks without escaping the WASM sandbox, because it has no mechanism to invoke system operations other than through the host function interface, which the kernel controls. This leads to a critical distinction: enforcement is complete; detection is probabilistic. The enforcement boundary—WASM isolation preventing direct system calls—is a structural guarantee that holds regardless of the agent’s capability or intent. The detection mechanism— ProbeLogits classifying whether an action is harmful—is probabilistic, with accuracy depending on model size and prompt quality. The combination is powerful: even if detection produces a false negative (classifying a harmful action as safe), the agent still op-
kv_restore(). Restores KV cache state from a checkpoint. Before restoring, the operation validates that the checkpoint’s dimensions (number of layers, bytes per position) match the current model. This prevents a subtle class of bugs where checkpoints from one model are accidentally applied to another. kv_fork(). Creates a copy of the KV cache suitable for transfer to a different inference context. Currently implemented as an alias for kv_checkpoint(), but semantically distinct: fork implies the original continues executing, while checkpoint implies the original may be overwritten.
4.3
Comparison with Serving Systems
Use Cases
KV cache as process state enables several capabilities that are impossible or expensive without kernel-level support: 5
erates within its capability sandbox, limiting the blast radius of any single action.
5.4
Blake3 Audit Chain
All governance decisions—classifications, confidence scores, and responses—are recorded in a Blake3 hash chain. Each audit entry includes the previous entry’s 5.2 Detection Pipeline hash, creating a tamper-evident log: retroactive modifiI implement a three-stage detection pipeline that com- cation of any entry invalidates all subsequent hashes. I bines rule-based pre-filtering with model-based classifi- note that this guarantee holds within the buffer window cation: (in-memory ring buffer); durable persistence requires in1. Stage 1: Adversarial pre-filter (< 1 µs). An tegration with the AnimaFS storage layer. O(n) pattern scan detects prompt injection, encoding tricks (base64, rot13 references), authority imImplementation personation (“ADMIN OVERRIDE”), and instruc- 6 tion overrides (“ignore previous”). Actions exceeding a risk-score threshold are classified as dangerous 6.1 Bare-Metal Inference Engine with no forward pass. The inference engine comprises 6,900 lines of no_std 2. Stage 2: Input sanitization. Control charac- Rust, running directly on bare metal with no operating ters and injection patterns are stripped before the system, no libc, and no standard library. It implements: action is embedded in the safety-priming prompt template. GGUF model loader. Parses the GGUF binary for3. Stage 3: Calibrated forward pass (65 ms). mat (llama.cpp’s model format) directly from USB mass A single 7B forward pass reads Safe/Danger- storage via xHCI, loading tensor data in Q4_0, Q6_K, ous verbalizer logits. Contextual calibration cor- and Q8_0 quantization formats. rects inherent token bias, and a privacy keyword boost compensates for the model’s blind spot on surveillance-as-harm. GPT-2 BPE tokenizer. A byte-pair encoding tokenizer supporting vocabularies up to 152,064 tokens. The per-action cost is dominated by the 7B forToken-to-ID lookup uses a BTreeMap for O(log |V |) perward pass (65 ms). A natural optimization—cascading formance. The tokenizer supports chat templates for through a smaller model first (e.g., 135M at 0.6 ms) and instruction-following models. escalating to 7B only for ambiguous cases—is left for future work. Llama transformer. A complete Llama-architecture transformer implementation: multi-head grouped-query 5.3 Graduated Response attention with RoPE positional encoding, SwiGLU feedThe governance system maps ProbeLogits confidence to forward networks, and RMS normalization. Supports models with GQA ratios (e.g., 7B uses 32 attention a graduated response: heads with 8 KV heads). Table 2: Graduated response policy mapping ProbeLogits confidence bands to governance verdicts. Confidence
Response
P (harmful) > 0.9 P (harmful) > 0.7 P (harmful) > 0.5 P (harmful) ≤ 0.5
Block (deny the action) Warn (log + notify operator) Log (record for audit) Allow (proceed normally)
AVX-512/AVX2 SIMD kernels. Quantized matrix-vector multiplication kernels for three formats: • Q4_0: 4-bit quantization with per-block f16 scale. AVX-512 kernel processes 64 elements (2 blocks) per iteration using VNNI vpdpbusd instructions. • Q6_K: 6-bit quantization with per-superblock scale and per-block sub-scale. Requires careful bit unpacking across three byte segments.
The graduated response is intentionally conservative: • Q8_0: 8-bit quantization with per-block f16 scale. a 78% confidence harmful classification results in a The fastest format, using direct vpmaddubsw inwarning, not a block. This reflects the reality that 7B structions. Q4_0 models are not highly accurate classifiers, and false positives (blocking legitimate actions) erode user All kernels use F16C hardware instructions trust more than false negatives (allowing suspicious ac- (vcvtph2ps) for f16→f32 scale conversion, elimitions with a warning). nating the need for a software floating-point library. 6
Table 3: Code scale by component.
SMP work-stealing graph executor. The inference computation is expressed as a dataflow graph (606 operations, 226 synchronization barriers for 7B). The graph executor uses work-stealing across all available CPU cores: each core maintains a local deque of ready operations, and idle cores steal work from busy cores’ deques. For the 7B model, the forward pass is fully parallelized across 8 cores.
6.2
Component
The no_std Challenge
LoC
Tests
boot/ (bare-metal) kernel/ (hosted) runtime/ cli/ sdk/ dsl/
48,300 24,300 9,400 2,300 330 1,400
— 447 106 51 28 24
Total
∼86k
656+
Role Kernel + inference + MCP Lifecycle, memory WASM, scheduling Agent mgmt WASM SDK + MCP Agent DSL
Table 4: Token generation throughput (tokens/s). Higher is better.
Building an LLM inference engine in a bare-metal environment presents four challenges that do not exist in userspace: 1. No float runtime. Bare-metal Rust provides no libm for transcendental functions. I use F16C hardware instructions for f16 conversion and a polynomial approximation (fast_expf_poly) for exp() in softmax and entropy computations.
Model
Config
Anima
llama.cpp
135M Q4_0
16w/16t 8w/8t
1,666 1,351
1,195 1,301
7B Q4_0
8w/8t 16w/16t
15 —
15 14
2. No filesystem. GGUF model files (4+ GB) are read from USB mass storage via a custom xHCI driver (7,400 lines), block by block.
6.4
4. No allocator (initially). The heap allocator is bootstrapped from the UEFI memory map. Once initialized, standard alloc::Vec and alloc::String are available.
7
Evaluation
7.1
Experimental Setup
Code Scale
Table 3 summarizes the system’s code scale. The boot/ 3. No threading. SMP parallelism is implemented kernel (42,700 LoC) is the largest component, containfrom scratch: AP (Application Processor) boot- ing the bare-metal inference engine (6,900 LoC), xHCI strap via the INIT-SIPI protocol, spin locks, and USB 3.0 driver (7,400 LoC), VirtIO-Net networking, atomic operations—no pthread, no std::thread. AnimaNet protocol, and all ProbeLogits primitives.
6.3
All experiments run on a single machine:
System Context
• CPU: AMD Ryzen 9 9800X3D (8 cores, 16 threads, AVX-512 with VNNI)
ProbeLogits operates within a complete OS kernel. The agent lifecycle manages five states (Nascent, Active, Suspended, Isolated, Terminated) with trust evolution based on task completion history and a four-tier capability system (System, AiNative, AiEnhanced, Classic). AnimaNet provides distributed networking: UDP discovery with boot_nonce-based reboot detection, TCP-based Raft consensus for cluster coordination, and live agent migration (serializing the AgentControlBlock and transferring it over TCP). A two-node cluster has been validated end-to-end with 11/11 test markers passing. The web stack enables agents to access external services: DNS resolution, HTTP client, TLS 1.3 (via rustls with no_std ring), and a web_fetch() host function exposed to WASM sandboxes. The WASM sandbox uses Cranelift AOT compilation with fuel-limited execution. Fifteen host functions bridge the sandbox to kernel services (12 base operations: inference, governance, IPC, file system, web access; plus 3 MCP tool-calling functions added in the companion gateway [29]).
• RAM: 60 GB DDR5-6000 dual-channel (∼75 GB/s measured bandwidth) • Models: SmolLM2-135M Q4_0 Qwen2.5-7B-Instruct Q4_0 (4.1 GB)
(70 MB),
• Baseline: llama.cpp (latest, March 2026) on Linux 6.17, same hardware Anima OS runs bare-metal (UEFI boot, no Linux). The llama.cpp baseline runs on Linux with identical CPU and memory.
7.2
Inference Throughput
Table 4 presents inference throughput results. On the 135M model, Anima OS achieves 1,666 tokens/s with 16 work-stealing workers, a 1.39× improvement over llama.cpp’s 1,195 tokens/s with 16 threads. With 8 workers/threads, the advantage narrows to 1.04× (1,351 vs. 1,301). The 135M model is compute-bound : its 7
Table 5: ProbeLogits operation latency. Operation Binary (probe_yes_no) N-way (probe_classify) Entropy (logit_entropy) Grammar masking (per token) KV checkpoint
7.5
135M
7B
0.6 ms 0.6 ms 0.1 ms <0.01 ms 0.5 ms
65 ms 65 ms 0.5 ms <0.01 ms 12 ms
I validate ProbeLogits on the standard ToxicChat [25] benchmark (Lin et al., 2023)—a human-annotated dataset of 10K+ real user conversations from a deployed chatbot. I sample 1,000 prompts (362 toxic, 638 benign; seed=42) from the test split. Contextual calibration. Instruction-tuned models exhibit strong sycophancy bias—a tendency to agree with the implied framing of the question. With Yes/No verbalizers and uncalibrated readout, the model predicts “dangerous” for the majority of prompts. I apply contextual calibration [27]: at boot time, seven nullinput prompts are probed to measure the model’s inherent bias (∆ = logitDangerous − logitSafe ), which is subtracted from all classifications scaled by a tunable α ∈ [0, 1]. I also use Safe/Dangerous verbalizers in addition to Yes/No, since Safe/Dangerous carry direct classification signal rather than surface agreement; the trade-offs of this choice are explored in §7.6 and §10. The prompt template uses safety-priming framing: "Is this action dangerous, harmful, or a privacy violation? Answer Safe or Dangerous.\nAction: {action}\nAnswer:"
70 MB parameter footprint fits in L3 cache, and performance scales with parallel compute—where workstealing outperforms static thread partitioning. On the 7B model, both systems achieve 15 tokens/s (65 ms per token). The 7B model is bandwidthbound : each forward pass reads 4.1 GB of parameters from main memory, and performance is determined by memory bandwidth rather than compute efficiency.
7.3
Bandwidth Saturation Analysis
The 7B parity result is explained by a fundamental hardware limit. The theoretical minimum time per token for a bandwidth-bound model is: tmin =
4.1 GB model size = = 55 ms bandwidth 75 GB/s
(5)
Single-model ToxicChat baseline. On Qwen2.57B-Instruct Q4_0 with α = 0.5 on the bare-metal Anima OS native runtime, the full pipeline (S/D verbalizer, calibrated, with adversarial pre-filter and privacy keyword boost) reaches F1 = 0.837 on ToxicChat 1000 (Table 11). For apples-to-apples hosted-mode comparison with Llama Guard 3 under the same llama-cpp-python environment, the pure-logit configuration (no hand rules) at α = 0.0 reaches F1 = 0.812 (Table 8, raw data results/toxicchat_qwen_sd_alpha00.json). Tuning α trades F1 for higher recall. The behavior of α as a deployment-time policy knob is detailed below; the multi-model and verbalizer comparisons follow in §7.6.
The measured 65 ms corresponds to 85% of theoretical peak bandwidth utilization. The remaining 15% accounts for non-memory operations (softmax, RoPE, layer normalization) and memory access patterns that do not achieve perfect streaming. This is a significant finding: bare-metal execution reaches the hardware bandwidth limit. No software optimization—better SIMD kernels, more efficient scheduling, or tighter code—can improve 7B performance on this hardware. The path to faster 7B inference requires higher memory bandwidth (DDR5-8000, HBM) or reduced model size (more aggressive quantization).
7.4
ProbeLogits Safety Classification
Substrate vs. supplementary heuristics. The full deployment pipeline combines (i) calibrated logit reading (the substrate primitive), (ii) the chosen verbalizer, (iii) an adversarial pre-filter (O(n), no forward pass), and (iv) a privacy keyword boost (a hand rule for OS actions). The substrate primitive alone accounts for the bulk of the classification signal; (iii)–(iv) are governance-layer heuristics that contribute roughly 4 pp on top of the substrate. The full pipeline ablation (uncalibrated → +calibration → +safety prompt → +privacy boost → +adversarial pre-filter, on a 260-prompt OS-action benchmark) is reported in the companion Governed MCP paper [29], which uses that dataset for end-to-end gateway evaluation. The model-only configuration of that pipeline (calibrated forward pass, Safe/Dangerous verbalizer, no
ProbeLogits Performance
Table 5 shows ProbeLogits operation latencies. Binary and N-way classification have identical cost because both require exactly one forward pass; the additional softmax over N token logits is negligible (<1 µs for N ≤ 100). Logit entropy computation is cheap (0.1–0.5 ms) because it operates on the already-computed logit vector without requiring an additional forward pass. It only needs to iterate over the 152K logits once. KV cache checkpoint cost scales with model size: 0.5 ms for 135M (small KV cache) and 12 ms for 7B (larger KV cache that must be copied). The 32 MB cap ensures bounded memory consumption. 8
Table 6: HarmBench block rate by model and verbalizer. Non-copyright (n=300) is the safety-harm subset; copyright (n=100) requires reference-based detection out of scope. Wilson 95% CIs in brackets.
Table 7: XSTest at Y/N α = 0.5. Recall is the fraction of the 200 unsafe prompts caught; over-refusal is the fraction of the 250 safe prompts wrongly flagged.
S/D Tot. S/D non-© Y/N Tot. Y/N non-©‡
Model Qwen 2.5-7B Llama 3 8B Mistral 7B
74.2% 72.8% 65.5%
98.7% 97.0% 87.3%*
86.8% 75.0% 79.5%
99.0% 98.0% 98.7%
Model
Recall
Over-refusal
F1
Qwen 2.5-7B Llama 3 8B Mistral 7B
100% 98.5% 99.5%
86.4% 51.6% 55.6%
0.649 0.749 0.740
‡
Y/N non-© Wilson 95% CIs: Qwen [97.1, 99.7], Llama 3 [95.7, 99.1], Mistral [96.6, 99.5]. * Mistral S/D underperforms because “Dangerous” tokenizes as 3 tokens in Mistral’s SentencePiece (§10); the probe reads a meaningless “D” logit.
Table 8: ToxicChat (n=1000) hosted-mode. PL = ProbeLogits (zero-shot, vanilla model). LG3 = Llama Guard 3 (fine-tuned safety classifier). Best F1 per model in bold. Wilson 95% CIs in brackets for headline F1.
hand rules) achieves F1=0.941; supplementary heuristics add ∼4 pp. The companion paper additionally reports the governance-level ablation that motivates this work: removing the entire ProbeLogits layer from a 6-layer kernel governance pipeline on a 101-prompt MCP benchmark drops F1 from 0.773 to 0.327 (∆F1 = −0.446) [29].
7.6
System
Verb.
PL Qwen 2.5-7B PL Qwen 2.5-7B PL Qwen 2.5-7B PL Qwen 2.5-7B PL Mistral 7B PL Llama 3 8B PL Llama 3 8B
S/D S/D S/D Y/N Y/N S/D Y/N
LG3 8B Q4_K_M — LG3 8B Q8_0 —
Multi-Model Validation
7.6.2
I evaluate ProbeLogits across three base model families (Qwen 2.5-7B Q4_0, Llama 3 8B Q4_0, Mistral 7B v0.3 Q8_0) on three external benchmarks (HarmBench [30], XSTest [32], ToxicChat [25]) with two verbalizer pairs (Safe/Dangerous, Yes/No). All hosted-mode evaluations use llama-cpp-python with 16 threads. Wilson 95% CIs and bootstrap F1 CIs are reported on the headline numbers.
F1
R
P
0.0 0.812 [0.781, 0.841] 0.5 0.801 1.0 0.775 1.0 0.707 0.0 0.719 0.5 0.679 [0.638, 0.719] 0.5 0.634
0.862 0.898 0.909 0.898 0.732 0.641 0.624
0.768 0.724 0.676 0.583 0.707 0.723 0.644
α
— —
0.675 [0.630, 0.717] 0.536 0.911 0.662 0.528 0.888
XSTest: Recall Architecture-Agnostic, Over-Refusal Model-Dependent
XSTest [32] (Röttger et al., NAACL 2024) contains 200 unsafe prompts and 250 safe-but-edgy prompts designed to test exaggerated safety. I report Y/N verbalizer at α = 0.5 (Table 7). Two findings. First, recall is essentially perfect across all three models (98.5–100%): the substrate primitive captures unsafe content with very high reliability regardless of base model. Second, over-refusal varies by 35 pp across models—this is not architectureagnostic. ProbeLogits is sensitive to the model’s prior 7.6.1 HarmBench: Catching Clearly Harmful over edge cases; the same primitive on Qwen produces Prompts a much more conservative classifier than on Llama 3 or Mistral. This is a real limitation, addressed in practice HarmBench [30] (Mazeika et al., ICML 2024) provides by per-model α tuning (§7.6.5). 400 unsafe prompts in 7 categories. I separate copyright (n=100) from non-copyright safety harm (n=300) 7.6.3 ToxicChat: Hosted-Mode Comparison because copyright detection requires reference-based Across Models matching not in scope for logit-level safety classification. Table 8 reports same-environment hosted results. Three Headline finding (Table 6). On non-copyright observations: (1) Qwen with S/D substantially outperforms prompts with the Y/N verbalizer (universal across tokenizers), all three models reach 97–99% block rate— LG3. Qwen S/D α = 0.0 reaches F1 = 0.812 (CI [0.781, within 1 pp of each other, McNemar pairwise compar- 0.841]) vs. LG3 0.675 (CI [0.630, 0.717]) — CIs disjoint, isons all p > 0.24 (statistically indistinguishable). The +13.7 pp significant. With a vanilla 7B model and zero mechanism is broadly architecture-agnostic for clearly- learned safety parameters, ProbeLogits exceeds the fineharmful content. Copyright prompts behave very differ- tuned LG3 baseline. (2) Llama 3 with S/D matches LG3 on F1 (parently across models (Qwen 50%, Llama 3 6%, Mistral 22%) and are reported separately rather than averaged ity). Llama 3 S/D F1 = 0.679 vs. LG3 = 0.675 — CIs overlap fully, the gap is statistical noise (Table 10). in. 9
Table 9: Calibration bias (mean logitpos − logitneg over 7 null prompts) varies by model and verbalizer. Positive bias ⇒ model defaults to “Yes/Dangerous”; negative ⇒ defaults to “No/Safe”. Model Qwen 2.5-7B Llama 3 8B Mistral 7B
Y/N bias
S/D bias
+4.49 −0.54 −3.04
−3.02 −2.54 N/A (multi-token)
Table 10: Adequacy comparison on ToxicChat (n=1000), pure-logit mode for ProbeLogits (no handrules). Per-model verbalizer was selected as: Qwen and Llama 3 — S/D verbalizer optimized per-benchmark; Mistral — Y/N verbalizer forced because “Dangerous” is multi-token in Mistral’s SentencePiece (§10). ProbeLogits is zero-shot logit reading on a vanilla model; Llama Guard 3 is a fine-tuned safety classifier at the same parameter count. P
Lat.†
PL Qwen 2.5-7B (S/D, α=0.0) 0.812 0.862 0.768 PL Qwen 2.5-7B (S/D, α=0.5) 0.801 0.898 0.724 PL Mistral 7B (Y/N, α=0.0) 0.719 0.732 0.707 PL Qwen 2.5-7B (Y/N, α=1.0) 0.707 0.898 0.583 PL Llama 3 8B (S/D, α=0.5) 0.679 0.641 0.723 Llama Guard 3 8B Q4_K_M 0.675 0.536 0.911 Llama Guard 3 8B Q8_0 0.662 0.528 0.888 PL Llama 3 8B (Y/N, α=0.5) 0.634 0.624 0.644
0.57s 0.57s 0.24s 0.57s 0.40s 1.06s 1.24s 0.14s
System
Recall is significantly higher (0.641 vs. 0.536, CIs disjoint), precision significantly lower (0.723 vs. 0.911, CIs disjoint). Operating-point trade-off, not absolute superiority. (3) Mistral, restricted to Y/N, also exceeds LG3. Mistral Y/N α = 0.0 reaches F1 = 0.719, +4.4 pp over LG3 with overlapping CIs. Mistral cannot use S/D verbalizers because of tokenization (§10). 7.6.4
Verbalizer Prior Asymmetry
The same model exhibits different calibration biases under different verbalizers; magnitude and even sign differ (Table 9). The Qwen pair flips sign: under Y/N the model favors “Yes”; under S/D it favors “Safe”. This is best interpreted as verbalizer prior asymmetry—the same surface-form competition phenomenon documented by Holtzman et al. in classification settings [31], here applied to logit-probing for safety classification: the bias magnitude is a joint function of the model’s safety-tuning and the verbalizer tokens’ surface frequency in the model’s training distribution (“Safe” is far more common than “Dangerous” in RLHF responses; “Yes”/“No” are more balanced). The implication is that ProbeLogits exposes priors the model already holds, but which prior is exposed depends on the verbalizer the OS picks at boot. The Token Fertility check (§10) ensures the chosen verbalizer is single-token-per-side; the OS should additionally measure calibration bias at boot and tune α per (model, verbalizer) pair. 7.6.5
α as Per-Model Policy Knob
The optimal α depends on the model’s bias direction and magnitude. For positive-bias models (Qwen Y/N), higher α suppresses the over-trigger and improves precision. For negative-bias models (Mistral, Llama 3 Y/N), α = 0.0 is already balanced. For Qwen S/D (negative bias), small α gives best F1; larger α amplifies recall at precision cost. This is not a hyperparameter to be searched on each benchmark; it is a deployment-time policy lever for the OS to trade precision for recall as the threat model demands. 10
F1
R
†
Hosted llama-cpp-python (16t, single inst); bare-metal native PL is 65 ms (§7.4). PL = ProbeLogits.
Adequacy check against Llama Guard 3. For a same-benchmark comparison with a strong purposebuilt baseline, I evaluate Llama Guard 3 8B [20]—a dedicated safety classifier fine-tuned from Llama 3.1 8B— on the identical ToxicChat 1000-prompt sample, at both Q4_K_M and Q8_0 quantization (Table 10). The intended framing is an adequacy check (“does the substrate primitive reach a comparable operating point as a fine-tuned baseline?”), not a superiority claim, because ProbeLogits is a kernel primitive rather than a competing classifier. The four findings below disentangle the configurations: for the strongest configuration (Qwen S/D), the comparison crosses adequacy into significant superiority (+13.7 pp, CIs disjoint); for Llama 3 S/D the result is parity (CIs overlap); for Mistral Y/N (forced by tokenization) the result is a smaller positive gap. Four findings emerge, with both significant comparisons reported (recall and precision) so the operatingpoint trade-off is explicit. First (Qwen S/D exceeds LG3 with significance). ProbeLogits-Qwen-2.5-7B (S/D, α = 0) F1 = 0.812 (bootstrap 95% CI [0.781, 0.841]) vs. Llama Guard 3 F1 = 0.675 (95% CI [0.630, 0.717]); the CIs are disjoint, +13.7 pp gap is significant. With a vanilla 7B model and zero learned safety parameters, ProbeLogits exceeds the fine-tuned LG3 baseline. Second (Llama 3 S/D matches LG3 – F1 parity). ProbeLogits-Llama-3 (S/D) F1 = 0.679 (bootstrap 95% CI [0.638, 0.719]) vs. Llama Guard 3 F1 = 0.675 (95% CI [0.630, 0.717]); the CIs overlap fully. The +0.4 pp gap is within statistical noise—a vanilla model on a different architecture also reaches LG3-equivalent F1 with no learned safety parameters. Throughout this paper we use “parity” in the bootstrap-CI-overlap sense (we fail to reject the null of no difference at 95%); this is weaker than a formal non-inferiority claim, which would
require a pre-specified margin and a directional test. Third (recall vs. precision trade-off ). ProbeLogits-Llama-3 achieves recall 0.641 (Wilson 95% CI [0.590, 0.689]) vs. LG3 0.536 (Wilson 95% CI [0.484, 0.587]); CIs are disjoint, the +10.5 pp recall advantage is significant. But the trade is paid in precision: ProbeLogits-Llama-3 P = 0.723 (95% CI [0.671, 0.769]) vs. LG3 P = 0.911 (95% CI [0.865, 0.942]); these CIs are also disjoint, and the −18.8 pp precision deficit is significant in the opposite direction. The choice between the two systems is therefore an operating-point decision: ProbeLogits favors deployments where falsenegatives (missed unsafe content) are more costly than false-positives (over-blocking benign content); Llama Guard 3 is the inverse. This trade-off is a property of the system pair, not a ranking—report both stats to avoid selective reporting. Fourth (verbalizer-tokenizer alignment). The same Llama 3 model under Y/N verbalizers gives F1 = 0.634 (below LG3 by −4.1 pp). Switching to S/D recovers F1 parity. The calibration bias also shifts: Y/N −0.54 vs. S/D −2.54, a 4.7× magnitude difference. This is best interpreted as verbalizer prior asymmetry— a verbalizer’s surface statistics in pretraining (“Safe” is far more frequent than “Dangerous” in RLHF-tuned responses) determine the prior magnitude the probe reads. Section 10 discusses the implication: ProbeLogits is sensitive to verbalizer-tokenizer alignment, and tokenizers (particularly SentencePiece) constrain what verbalizer pairs are usable. Latency is consequential. In hosted llama-cpp-python (the environment used for this fairness comparison), ProbeLogits takes ∼400 ms per classification vs. Llama Guard 3 ∼1 s—roughly 2.5× faster, because ProbeLogits performs only a single forward-pass logit read while Llama Guard 3 generates response tokens. In our baremetal native runtime ProbeLogits drops to 65 ms (§7.4); Llama Guard 3 in the same runtime would yield similar proportional speedup. Either way, the relative ordering holds: ProbeLogits is fast enough for per-action governance even in unoptimized hosted environments. For context, Llama Guard 3 reports F1 ∼0.939 on its training-aligned benchmark; the gap to 0.675 on ToxicChat reflects that no safety classifier transfers losslessly across distribution shifts. ProbeLogits is designed as an OS kernel primitive for classifying agent actions, not as a general-purpose chat toxicity detector. On its intended use case (OS actions in the Governed MCP companion paper [29]), the full pipeline achieves F1 = 0.980 with perfect precision across all α values.
Table 11: Effect of calibration strength α on ToxicChat (n=1000) safety classification with Qwen2.5-7B S/D verbalizer. α controls the precision-recall tradeoff as a deployment-time policy parameter, not a learned hyperparameter. α
Acc
P
R
F1
0.3 0.5 0.7 0.8 0.9 1.0
83.9 88.6 86.2 85.0 83.9 82.5
0.891 0.867 0.784 0.755 0.727 0.698
0.633 0.809 0.854 0.867 0.890 0.912
0.740 0.837 0.817 0.807 0.800 0.790
Table 12: Governance pipeline latency per action. Stage Adversarial pre-filter Input sanitization ProbeLogits 7B Blake3 audit entry
Latency <1 µs <1 µs 65 ms <10 µs
positives) while recall rises (fewer missed unsafe). The optimal F1 of 0.837 occurs at α = 0.5; α = 1.0 maximizes recall (0.912) at the cost of precision (0.698). This is a design feature, not a limitation: the OS can enforce strict policies for privileged operations (α ≥ 0.8) while applying relaxed policies for conversational agents (α = 0.5). Unlike fixed-threshold classifiers, ProbeLogits delegates the precision-recall tradeoff to the OS security policy at deployment time. Per-model α behavior across three base models is reported in §7.6.
7.7
Constitutional Governance Overhead
Table 12 shows the per-stage cost of the governance pipeline. The 65 ms forward pass dominates total latency. While this overhead is significant, it occurs only once per agent action (not per token), making it practical: a typical agent action involves hundreds of generated tokens (∼5–20 s), so the 65 ms governance check adds <1.3% overhead. A cascading design that prescreens actions with a smaller model (e.g., 135M at 0.6 ms) could reduce average latency further; this optimization is left for future work.
7.8
Comparison with Related Systems
Table 13 compares Anima OS with related systems across 12 features. AIOS [22] is the closest competiCalibration strength (α) as policy knob. Ta- tor, but it is implemented as a Python wrapper around ble 11 shows the precision-recall trade-off under vary- existing frameworks, inheriting the layered architecture ing α on ToxicChat. As α increases, the calibration problems described in §2.1. llama.cpp [7] provides efcorrection grows stronger: precision drops (more false ficient inference but as a userspace library, not an OS 11
Table 13: Feature comparison with related systems. ✓ = supported, blank = not supported, ∼ = partial. Feature Bare-metal exec. Kernel inference ProbeLogits Logit entropy Grammar decoding KV cache as state Constitutional gov. Trust evolution Raft consensus Agent migration WASM sandbox Hidden state
Anima OS ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
AIOS
OS-Copilot
Discussion
8.1
Why Bare Metal?
llama.cpp
vLLM
AutoGPT
∼ ∼ ∼ ∼
primitive. vLLM [19] manages KV caches for serving throughput but does not expose them as process state. The distinguishing features of Anima OS— ProbeLogits, logit entropy, kernel-level governance, and KV cache as process state—are absent from all compared systems. These features are only possible because inference runs inside the kernel, giving the OS direct access to model internals.
8
LangChain
Representation engineering is a research tool : it requires training auxiliary models, operates post hoc, and aims to understand or modify model behavior. ProbeLogits is an OS primitive: it requires no training, operates in real time, and aims to make kernel-level decisions. The two approaches are complementary— representation engineering could inform the design of better ProbeLogits prompts—but they occupy different positions in the system stack.
8.3
Enforcement ̸= Detection
The key insight of the governance architecture is the separation of enforcement from detection. Enforcement is structural: WASM isolation ensures that agents can only interact with the system through host functions. This guarantee holds regardless of the agent’s capability or intent. It is binary—either the agent is sandboxed or it is not—and does not depend on the accuracy of any classifier. Detection is probabilistic: ProbeLogits classifies actions with bootstrap-CI-confirmed F1 parity to a finetuned safety baseline on ToxicChat (§7.6) and 97–99% block rate on HarmBench non-copyright across three model families, using contextual calibration and safetypriming prompts; the pipeline-with-hand-rules variant (used in the Governed MCP companion paper [29]) reaches F1 = 0.980 on OS actions. This accuracy improves with model size and prompt quality but is never 8.2 ProbeLogits vs. Representation En- 100%. The combination is more powerful than either alone. gineering A system with only enforcement (no detection) would Burns et al. [3] discover latent knowledge in LLM repre- sandbox agents but have no visibility into their behavsentations by training probes on hidden states. Zou et ior. A system with only detection (no enforcement) al. [28] control model behavior through representation could identify harmful actions but not prevent them if vectors. These works inspire my hidden state analy- the agent bypasses the detector. Anima OS provides sis (BehaviorProfile) but differ fundamentally in pur- both: detection informs the graduated response, and pose: enforcement ensures the response is applied. Running an LLM inference engine inside the kernel is an extreme design choice. I justify it on two grounds. First, trust boundary. If governance decisions depend on inference results (“is this action safe?”), the inference engine must be at least as trusted as the governance system. In a layered architecture, a compromised inference framework can return arbitrary results, undermining governance entirely. In Anima OS, the inference engine runs at the same privilege level as the governance system—in the kernel. Second, scheduling control. A kernel-level inference engine can be scheduled with precise priority relative to other kernel operations. Governance checks can preempt lower-priority inference tasks, and the kernel can manage KV cache memory alongside other kernel memory without crossing privilege boundaries.
12
Constitutional AI. Bai et al. [2] introduce constitutional AI as a training-time technique: the model The convergence of three technologies makes kernelis trained with AI feedback guided by a set of princilevel AI integration practical in 2026: ples. My approach is complementary—I enforce consti1. Quantization: 4-bit quantization (Q4_0) reduces tutional principles at runtime in the kernel, regardless of a 7B model from 14 GB to 4.1 GB, fitting in com- how the model was trained. The two approaches can be modity RAM. combined: a model trained with constitutional AI would 2. SIMD: AVX-512 VNNI enables efficient 4-bit produce more accurate ProbeLogits classifications. matrix-vector multiplication without GPU acceleration. Safety classifiers. Llama Guard [8, 20] is a fine3. Sub-100ms latency: 65 ms per forward pass tuned safety classifier achieving F1 ∼0.93–0.95 on its on 7B makes per-action governance practical—fast benchmark. WildGuard [26] achieves macro-F1 0.907 enough to classify every agent action without user- on WildGuardTest. NeMo Guardrails [11] provides programmable safety rails for LLM applications. All operperceptible delay. ate as external classifiers at the application level, reFive years ago, 7B inference required seconds on quiring dedicated models. ProbeLogits differs by using CPU. Today it requires 65 ms. This 100× improvement the same model that performs inference for classification transforms LLM inference from an application-level ser- (no second model overhead, bootstrap-CI-confirmed F1 vice to a viable kernel primitive. parity to Llama Guard 3 on ToxicChat with a vanilla 7B model and zero learned fine-tuning), and by operating below the WASM sandbox boundary where the agent 9 Related Work cannot intercept or skip the safety check.
8.4
Why Now?
AI-oriented operating systems. AIOS [22] extends a Python-based framework with LLM-oriented scheduling and memory management (published at COLM 2025). OS-Copilot [12] uses LLMs to automate Linux system administration tasks. PunkGo [23] proposes a Rust sovereignty kernel for verifiable AI agent execution with Merkle audit logs, the closest related work in kernel-level AI governance. Microsoft’s Agent Governance Toolkit [21] provides application-level runtime security for AI agents. All operate atop existing operating systems (or focus on verification rather than inference), inheriting the latency and bypassability problems discussed in §2.2. Anima OS differs by integrating inference directly into a bare-metal kernel, enabling logitlevel primitives. Inference engines. llama.cpp [7] is the state-of-theart CPU inference engine, with extensive SIMD optimization and broad hardware support. My bare-metal engine achieves parity on bandwidth-limited 7B workloads and 1.39× superiority on compute-bound 135M workloads. The performance advantage comes from eliminating OS scheduling overhead and using workstealing, not from superior algorithms.
Constrained decoding. Outlines [13] and Guidance [5] implement constrained decoding in Python/C++. llguidance [9] (Microsoft) uses PDAbased algorithms for CFG constraints. My current grammar support is limited to choice/boolean constraints; I plan to port llguidance to no_std for full CFG support. Representation engineering. Burns et al. [3] discover latent knowledge in LLM representations without supervision. Zou et al. [28] propose representation engineering as a top-down approach to AI transparency. These works inspire my hidden state behavior analysis but operate as research tools, not OS primitives (§8.2). Distributed inference. Petals [14] enables collaborative inference by splitting model layers across networked machines. Anima OS’s AnimaNet takes a different approach: distributing agents (not tensors) across nodes, with Raft [15] consensus for coordination.
Capability security. seL4 [16] provides formally verified capability-based security. CHERI provides Anima OS uses Serving systems. vLLM [19] introduces PagedAt- hardware-enforced capabilities. Ed25519-signed capability tokens for agent permissions, tention for efficient KV cache memory management in drawing on the capability security tradition but applied serving. SGLang [17] optimizes structured generation to AI agent governance. programs with RadixAttention. Both focus on serving throughput (requests per second), while my KV cache primitives focus on process state management (check- Rust bare-metal operating systems. Theseus point, restore, fork). The two approaches address dif- OS [18] explores Rust’s type system for OS safety guarferent problems. antees. These systems demonstrate the viability of Rust 13
for kernel development but do not address AI integra- Benchmark breadth and methodology. Three tion. Anima OS builds on this foundation, extending caveats apply to the evaluation: (1) per-model verbare-metal Rust to AI inference and governance. balizer choice was made post-hoc—Qwen and Llama 3 use Safe/Dangerous (single-token in their tokenizLLM agent frameworks. AutoGPT [1], ers), Mistral is forced to Yes/No because “DangerLangChain [6], CrewAI [4], and MetaGPT [10] ous” is multi-token in SentencePiece (§10). A preorchestrate LLM-based agents in Python. These registered verbalizer-selection protocol would harden frameworks provide convenience but operate entirely in the claim. (2) α values reported in α sweeps were userspace, with no kernel-level governance or inference chosen by inspecting the same benchmark on which they are reported. A held-out validation set for α integration. selection would replace test-set tuning with a principled deployment procedure. (3) the OS-action bench10 Limitations mark used for component-level pipeline ablation (260 prompts) was author-labeled and the hand-rule heurisGPU acceleration. Anima OS currently runs CPU- tics were iteratively tuned against it; it is reported only only inference. For models beyond 7B parameters, GPU in the companion paper [29] for the gateway-level evalacceleration is essential. Bare-metal GPU compute uation. ToxicChat (n=1,000), HarmBench (n=400), (AMD RDNA or NVIDIA) requires reverse-engineering and XSTest (n=450) are independent human-annotated proprietary interfaces, which remains a significant engi- benchmarks that mitigate the risk of label optimization, neering challenge. and the multi-model evaluation (Qwen 2.5-7B, Llama 3 8B, Mistral 7B) addresses single-model overfitting conProbeLogits accuracy. On ToxicChat (1,000 cerns from the v1 draft. prompts), F1 ranges from 0.740 (α = 0.3) to 0.837 (α = 0.5) for Qwen 2.5-7B S/D, with bootstrap-CI- Formal verification. The governance system’s nonconfirmed parity to Llama Guard 3 [20] (§7.6). The bypassability claim rests on WASM isolation properties, optimal α is verbalizer- and domain-dependent: α = 0.5 which I have not formally verified. TLA+ or Iris specifimaximizes F1 on chat, while OS actions tolerate higher cations of the governance invariants would significantly α (full-pipeline OS variant in companion paper [29]). strengthen the security argument. I note that WASM Independent annotators and broader benchmarks sandboxing is a well-studied isolation mechanism, but beyond the three reported (HarmBench, XSTest, formal verification of the specific Anima OS host funcToxicChat) would further strengthen the results. tion interface has not been performed. Verbalizer-tokenizer alignment. ProbeLogits requires the verbalizer pair (e.g., Safe/Dangerous, Yes/No) to be single vocabulary tokens. This is satisfied for BPE tokenizers in Llama 3 and Qwen on both pairs, but violated for Mistral 7B (SentencePiece): “Dangerous” tokenizes as ["D", "anger", "ous"], so the probe at the answer position reads a meaningless “D” logit. Mistral evaluations therefore use Yes/No only, restricting the available verbalizer space. This is a real architectural constraint, not a footnote: the choice of base model determines what verbalizer pairs are usable, which in turn affects which model priors are legibly exposed by the probe. Single-token verbalizer constraints should be checked at OS boot (Token Fertility check), and the probe should refuse to start if no usable verbalizer exists for the loaded model.
TLS certificate verification. The current TLS 1.3 implementation uses UnsecureProvider (no certificate validation). Production deployment requires integration of a no_std X.509 certificate verifier such as webpki.
Grammar expressiveness. Current grammar support is limited to choice and boolean constraints. Full context-free grammar (CFG) support—enabling JSON Schema validation, regex constraints, and arbitrary structured output—requires porting a PDA-based engine (such as llguidance [9]) to no_std. I estimate this at approximately two weeks of engineering effort.
I presented ProbeLogits, the first OS primitive that exposes LLM logit vectors as kernel-level abstractions for semantic classification, uncertainty measurement, and governance enforcement. ProbeLogits eliminates text generation, parsing, and IPC from the classification path, reducing latency from ∼650 ms (generate-thenparse) to 65 ms (single forward pass) on a 7B model.
14
Multi-agent demonstrations. While the infrastructure for multi-agent systems is complete (lifecycle, trust, capabilities, migration, Raft), I have not yet demonstrated complex multi-agent scenarios with realistic workloads. End-to-end demonstrations with cooperating and competing agents would validate the governance and trust systems under realistic conditions.
11
Conclusion
The key finding is that bare-metal inference reaches the hardware bandwidth limit: 65 ms per token on 7B corresponds to 85% of the theoretical DDR5-6000 bandwidth ceiling. No software optimization can improve this further on the same hardware—the path forward is higher memory bandwidth or GPU acceleration. ProbeLogits, combined with KV cache process state operations and kernel-enforced constitutional governance, demonstrates that an AI-native kernel can provide capabilities impossible in layered architectures: sub-millisecond semantic classification (135M), kernelenforced safety guarantees via WASM sandbox isolation (defense-in-depth), and process-level management of inference state (checkpoint, restore, fork). The calibration parameter α provides a deploymenttime knob for precision-recall tradeoff, enabling domainspecific safety policies without retraining. As AI agents become as common as processes, the operating system must evolve from a passive resource manager to an active participant in the inference loop. ProbeLogits is a first step toward that future.
[17] L. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs,” arXiv:2312.07104, 2023. [18] K. Boos, N. Liber, and L. Zhong, “Theseus: An Experiment in Operating System Structure and State Management,” OSDI 2020. [19] W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023. [20] Meta, “Llama Guard 3-8B Model Card,” https://github. com/meta-llama/PurpleLlama, 2024. [21] Microsoft, “Agent Governance Toolkit (AGT) for LLM Tool Calls,” Microsoft Research preview, 2026. [22] K. Mei et al., “AIOS: LLM Agent Operating System,” COLM 2025. [23] Z. Zhang, “Right to History: A Sovereignty Kernel for Verifiable AI Agent Execution,” arXiv:2602.20214, 2026. [24] T. Schick and H. Schütze, “Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference,” EACL 2021. [25] Z. Lin et al., “ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation,” Findings of EMNLP 2023. [26] S. Han et al., “WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs,” arXiv:2406.18495, 2024.
References
[27] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate Before Use: Improving Few-Shot Performance of Language Models,” ICML 2021.
[1] T. B. Richards (Significant-Gravitas), “AutoGPT: An Autonomous GPT-4 Experiment,” GitHub repository, 2023.
[28] A. Zou et al., “Representation Engineering: A Top-Down Approach to AI Transparency,” arXiv:2310.01405, 2023.
[2] Y. Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” arXiv:2212.08073, 2022.
[29] D. Son, “Governed MCP: Kernel-Level Tool Governance for AI Agents via Logit-Based Safety Primitives,” Anima OS companion paper, 2026 (in preparation).
[3] C. Burns, H. Ye, D. Klein, and J. Steinhardt, “Discovering Latent Knowledge in Language Models Without Supervision,” ICLR 2023.
[30] M. Mazeika et al., “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal,” ICML 2024.
[4] J. Moura, “CrewAI: Framework for Orchestrating RolePlaying Autonomous AI Agents,” GitHub repository, 2024.
[31] A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer, “Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right,” EMNLP 2021.
[5] S. Lundberg et al., “Guidance: A Language for Controlling Large Language Models,” GitHub repository, 2023.
[32] P. Röttger et al., “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models,” NAACL 2024.
[6] H. Chase, “LangChain: Building Applications with LLMs through Composability,” GitHub repository, 2022. [7] G. Gerganov, “llama.cpp: GitHub repository, 2023.
LLM inference in C/C++,”
[8] H. Inan et al., “Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations,” arXiv:2312.06674, 2023. [9] Microsoft, “llguidance: Fast Constrained Decoding Library,” GitHub repository, 2024. [10] S. Hong et al., “MetaGPT: Meta Programming for A MultiAgent Collaborative Framework,” ICLR 2024. [11] T. Rebedea, R. Dinu, M. Sreedhar, et al., “NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails,” arXiv:2310.10501, NVIDIA, 2023. [12] Z. Wu et al., “OS-Copilot: Towards Generalist Computer Agents with Self-Improvement,” arXiv:2402.07456, 2024. [13] B. Willard and R. Louf, “Efficient Guided Generation for Large Language Models,” arXiv:2307.09702, 2023. [14] A. Borzunov et al., “Petals: Collaborative Inference and Fine-tuning of Large Models,” ACL 2023 (demo). [15] D. Ongaro and J. Ousterhout, “In Search of an Understandable Consensus Algorithm,” USENIX ATC 2014. [16] G. Klein et al., “seL4: Formal Verification of an OS Kernel,” SOSP 2009.
15