arXiv CSopen access

Robust and Efficient Guardrails with Latent Reasoning

Robust and Efficient Guardrails with Latent Reasoning Siddharth Sai Xiaofei Wen Muhao Chen University of California, Davis {sai,xfwe,muhchen}@ucdavis.edu

arXiv:2605.29068v1 [cs.AI] 27 May 2026

Abstract

brittle under ambiguous, adversarial, or contextdependent safety decisions. Recent explicit reasoning guardrails (Wen et al., 2025b; Liu et al., 2025) improve robustness by learning from distilled chain-of-thought (CoT) supervision (Hsieh et al., 2023; Kim et al., 2023) and generating intermediate rationales before predicting a safety label. MrGuard (Yang et al., 2025) further extends reasoning-based guardrails to multilingual safety moderation by combining synthetic multilingual supervision with curriculum-guided Group Relative Policy Optimization (GRPO) (Shao et al., 2024). However, this robustness comes at a steep computational cost. Because these models verbalize their intermediate rationales, moderation becomes a long autoregressive generation process. The additional CoT tokens substantially inflate inference time and completion-token cost, making explicit reasoning guardrails difficult to deploy in high-traffic, real-time settings (Liu et al., 2025; Sreedhar et al., 2025). Existing efficiency-oriented variants, such as shorter supervised traces or reasoning on/off switches (NVIDIA, 2025; Sreedhar et al., 2025), reduce the amount or frequency of rationale generation but still rely on explicit decoding and may sacrifice robustness. This motivates a natural question: can guardrails retain the benefits of reasoning supervision without generating reasoning tokens at inference time? We study this question through C O L AG UARD, a latent-reasoning safety guardrail that internalizes explicit safety rationales into continuous recurrent states as shown in Figure 1. Inspired by Coconut (Hao et al., 2025) and ICoT-SI (Deng et al., 2025), C O L AG UARD performs a fixed number of latent recurrent steps in place of explicit rationale generation. It first learns from CoT supervision and then progressively replaces rationale tokens with latent states, allowing the model to directly predict the safety label without autoregressive rationale generation. A practical challenge is that pretrained

Maintaining the safety of large language models (LLMs) is crucial as they are increasingly deployed in real-world applications. Existing safety guardrails typically rely on single-pass classification or, more recently, distilled reasoning. Reasoning-based guardrails significantly outperform classification-only baselines, but they incur substantial query latency and token overhead that make them impractical for highthroughput deployment. To address this challenge, we propose C O L AG UARD, a guardrail model that transfers multi-step safety reasoning into a continuous latent space through a stage-wise training curriculum, enabling direct hidden-state propagation at inference. Evaluated on ten prompt- and response-moderation settings spanning eight safety benchmarks, C O L AG UARD improves macro-F1 by 8.24 points over Llama Guard 3 and matches our explicit reasoning baseline, GuardReasoner, in macroF1 while delivering a 12.9× speedup and 22.4× reduction in token usage. Our results suggest that latent reasoning offers a practical alternative to explicit rationale generation for deployable guardrails, jointly improving safety robustness and inference efficiency rather than treating them as competing objectives.

1

Introduction

As Large Language Models (LLMs) become integral to daily and industrial applications, ensuring their alignment with human values is critical. Although alignment training methods such as RLHF (Ouyang et al., 2022; Rafailov et al., 2023) can improve model behavior, they require modifying the target model and are costly to update after deployment. External safety guardrails (Inan et al., 2023; Han et al., 2024) therefore provide a practical alternative by offloading input and output moderation to smaller, third-party models. Early guardrails typically formulate moderation as single-pass classification, which is efficient but often becomes 1

Figure 1: Overview of C O L AG UARD. Unlike explicit reasoning guardrails (left) that generate chain-of-thought tokens before assigning labels, C O L AG UARD (right) reasons through recurrent latent states, preserving moderation performance while avoiding token generation overhead and enabling 12.9× faster inference and 22.4× fewer tokens. C O L AG UARD’s stage-wise internalization curriculum (center) begins with explicit CoT supervision and progressively replaces reasoning tokens with latent states, shifting reasoning into hidden activations.

LLMs are optimized to consume token embeddings rather than recirculated contextual hidden states, which can create a distribution mismatch during latent recurrence. To reduce this mismatch, we adopt Context-Prediction Fusion (Liu et al., 2026), which combines contextual hidden-state information with predictive semantic guidance from the vocabulary embedding space. This stabilizes latent recurrence while preserving the latency and token-efficiency benefits of avoiding explicit CoT generation. In summary, this work makes three main contributions. (1) We introduce C O L AG UARD, a latentreasoning safety guardrail that internalizes explicit safety rationales through a stage-wise curriculum, enabling moderation without autoregressive rationale generation at inference time. (2) We show that C O L AG UARD preserves the robustness of explicit reasoning guardrails while substantially reducing inference cost, suggesting that reasoningbased moderation can be made practical without verbalized rationales. (3) We analyze the latent recurrence process and find that C O L AG UARD improves over vanilla Coconut, consistent with progressive safety-relevant representation shifts across latent steps that are largely absent in vanilla Coconut recurrence.

without modifying the base LLM. Early architectures such as Llama Guard (Inan et al., 2023) and WildGuard (Han et al., 2024) treated moderation as classification, followed by models like ShieldGemma (Zeng et al., 2024), Aegis (Ghosh et al., 2024), and Qwen3Guard (Zhao et al., 2025), which improved performance through broader taxonomies. The broader guardrail literature has expanded robustness through adversarially resilient moderation (Yuan et al., 2024) and structured safety knowledge (Kang and Li, 2025). Recent work further improves performance with reasoning: GuardReasoner (Liu et al., 2025) and ThinkGuard (Wen et al., 2025b) use chain-of-thought rationales (Wei et al., 2023) from expert models to improve generalization, while MrGuard (Yang et al., 2025) extends reasoning-based guardrails to multilingual moderation through synthetic multilingual supervision and curriculum-guided GRPO. Others explore efficiency trade-offs through shorter rationale traces and on/off switches (NVIDIA, 2025; Sreedhar et al., 2025; Rebedea et al., 2023). However, because these models verbalize reasoning in natural language, they incur steep autoregressive decoding costs that limit their practicality for hightraffic, real-world deployment.

2

Latent Reasoning A growing literature suggests that effective reasoning can occur within a model’s hidden states rather than through explicit tokens (Chen et al., 2025; Zhu et al., 2025; Biran et al.,

Related Work

LLM Guardrails External guardrails provide a lightweight mechanism for safety moderation 2

2024). This space includes augmenting models with "thinking" tokens (Goyal et al., 2024; Zelikman et al., 2024; Pfau et al., 2024), internalizing CoT through staged curricula (Deng et al., 2023, 2025), and feeding hidden states back as continuous input embeddings (Hao et al., 2025; Cheng and Durme, 2024; Zhu et al., 2025). However, these methods have largely been studied on mathematical and logical reasoning tasks, and directly recycling raw hidden states can become unstable at larger scales due to distribution mismatch with the token embedding manifold. Latent Thoughts Tuning (Liu et al., 2026) addresses this with a contextprediction fusion mechanism that aligns contextual hidden states with predictive signals from the vocabulary embedding space. C O L AG UARD adapts these techniques, showing that latent reasoning can drastically reduce latency costs and preserve the robustness of explicit baselines in safety moderation.

3

scaffolding. This follows prior work on chain-ofthought reasoning, step-by-step distillation, and reasoning-based safety guardrails, where intermediate rationales provide richer supervision than final labels alone (Wei et al., 2023; Hsieh et al., 2023; Kim et al., 2023; Liu et al., 2025; Wen et al., 2025b). We assume access to a reasoning-augmented guardrail corpus D = {(xi , si , ri , yi )}N i=1 , where xi is a user prompt, si is the corresponding model response, yi = (yip , yir ) contains the final prompt and response safety labels, and ri = (ri1 , ri2 , . . . , rimi ) is a step-separated safety rationale. Unlike standard label-only guardrail training, this supervision exposes the model to the reasoning underlying the final moderation decision. However, C O L AG UARD does not aim to generate these rationales at inference time. Instead, the rationales serve as targets during the initial stages of training so that the model can later compress the deliberation process into latent steps.

C O L AG UARD

We now present C O L AG UARD, a latent-reasoning guardrail framework for efficient prompt and response moderation. C O L AG UARD uses explicit safety rationales generated by expert models as training-time supervision, then progressively internalizes this step-by-step reasoning into recurrent latent states so that inference incurs only a fixed latent computation budget before decoding the safety labels. We formulate the guardrail task in §3.1, describe reasoning-augmented supervision and explicit warm-up in §3.2–§3.3, and present latent recurrence, stage-wise internalization and efficient inference in §3.4–§3.6. 3.1

3.3

The first stage (Stage 0) trains the model as an explicit reasoning guardrail. Given an instruction I, prompt x, response s, rationale r, and final label tuple y, the model is optimized to generate structured safety-relevant rationale followed by the final safety labels: Lwarm = −E(x,s,r,y)∼D log pθ (r, y | I, x, s). This warm-up follows the explicit reasoning guardrail paradigm, where models learn to verbalize intermediate safety reasoning before predicting final moderation labels (Liu et al., 2025; Wen et al., 2025b). We denote the resulting model as G0θ , from which subsequent stages progressively replace explicit rationale steps with latent recurrent steps.

Guardrail Task

Given a user prompt x and a model response s, a guardrail model Gθ predicts the safety of both the input request and the generated response (ŷ p , ŷ r ) = Gθ (x, s), where ŷ p ∈ Y denotes the predicted prompt harmfulness label, ŷ r ∈ Y denotes the predicted response harmfulness label, and Y denotes the set of safety categories in the guardrail’s policy. 3.2

Explicit Reasoning Warm-Up

3.4

Dual-Mode Latent Recurrence

To internalize reasoning, C O L AG UARD switches between two modes. In language mode, the model consumes standard token embeddings and predicts the next token autoregressively. In latent mode, the model does not consume a standard token embedding; instead, the previous hidden state is fed back as the next input representation. Let e(·) denote the token embedding function and let ht ∈ Rd be the last-layer hidden state at

Reasoning-Augmented Supervision

The central challenge is maintaining the robustness of reasoning-based guardrails without requiring that the guardrail verbalize its reasoning process at inference time. C O L AG UARD addresses this by using explicit rationales for the initial training 3

position t. For a sequence with a latent span beginning at position a and ending at position b, vanilla latent recurrence replaces the input embedding at each latent position with the previous hidden state: ( e(wt ), t < a or t > b, Et = ht−1 , a ≤ t ≤ b,

3.5

Stage-Wise Internalization

The core of C O L AG UARD is a stage-wise curriculum that progressively replaces natural-language rationale steps with recurrent latent steps. The staged replacement schedule follows prior internalization curricula showing that gradually replacing explicit reasoning tokens is more stable than removing rationales all at once (Deng et al., 2023, 2025; Hao et al., 2025). For an example with m rationale steps, we write the step-separated rationale as

where wt is the discrete token at position t outside the latent span. This follows the chain-ofcontinuous-thought formulation introduced by Hao et al. (2025), allowing the model to perform recurrent computation in continuous latent space rather than generating intermediate rationale tokens. While this latent recurrence lays the foundation of C O L AG UARD, directly feeding contextual hidden states back into a pretrained transformer creates a distribution mismatch since the base model is trained to consume token embeddings, while ht−1 is a hidden representation. To reduce the hiddenstate/token-embedding mismatch observed in latent recurrence, we adopt context-prediction fusion from Latent Thoughts Tuning (Liu et al., 2026). At each latent position, the model first computes a predictive embedding from the next-token distribution induced by the previous hidden state: X epred (ht−1 ) = p̃θ (v | ht−1 )e(v),

r = (r1 , r2 , . . . , rm ), let K denote the maximum number of reasoning steps represented by the latent budget, and define ℓk = min(k, K). At stage k, the first k rationale steps are removed and replaced with ℓk c latent positions: (r1 , . . . , rk ) → (z1 , . . . , zℓk c ), where c is the number of latent positions allocated per replaced reasoning step within the latent budget. We denote the resulting training sequence as q (k) , which contains the instruction, prompt, response, latent span, any remaining rationale steps rk+1 , . . . , rm , and the final labels y. If k ≥ m, the rationale is fully replaced and the final label tuple follows the latent span directly; however, because the latent budget is fixed, examples with more than K rationale steps may still contain explicit rationale tokens after the maximum latent stage is reached. We therefore include a final compression stage that keeps the latent span fixed at Kc positions while removing all remaining rationale steps:

v∈Vp

where Vp is the nucleus-filtered vocabulary set, and p̃θ (v | ht−1 ) is the renormalized probability distribution over this set. Structural latent-control tokens are excluded from this distribution. The recurrent input is then constructed by fusing the contextual hidden state with the predictive embedding: ẽt = αht−1 + (1 − α)epred (ht−1 ),

r1:m → (z1 , . . . , zKc ).

where α ∈ [0, 1] controls the balance between contextual continuity and semantic anchoring. Finally, a lightweight projection module maps the fused representation back into the model input space:   α = 1, ht−1 , in et = gϕ (ẽt ), α < 1 and adapter is used,   ẽt , α < 1 and no adapter is used.

This extra stage enables the absorption of residual explicit reasoning signal into the fixed latent recurrence rather than remaining decoded as text. Training optimizes only the remaining language tokens and final labels. The prompt, response, latent-control tokens, and latent positions are masked from the language-modeling loss. Let M(k) be the set of supervised token positions in q (k) . The internalization objective is

where gϕ is a trainable adapter. When α = 1, this reduces to vanilla hidden-state recurrence; when α < 1, the recurrent state is anchored by predictive information from the vocabulary embedding.

(k)

Lint = −E(x,s,r,y)∼D

X t∈M(k)

4

(k)

log pθ (qt

(k)

| q<t ).

Benchmark

(1) safety classification performance across various baselines and (2) inference efficiency against explicit reasoning guardrails.

Samples

Prompt Harmfulness Detection ToxicChat (Lin et al., 2023) OpenAI Moderation (Markov et al., 2023) Aegis Safety Test (Ghosh et al., 2024) HarmBench (Mazeika et al., 2024) WildGuardTest (Han et al., 2024)

2,853 1,680 359 239 1,756

4.1

Reasoning Augmented Dataset. We use the GuardReasonerTrain dataset (Liu et al., 2025) as the primary training source for our guardrail model. GuardReasonerTrain is a 127,000-example reasoning-augmented compilation of the following safety-focused datasets: WildGuard (Han et al., 2024), AegisSafety (Ghosh et al., 2024), BeaverTails (Ji et al., 2023), and ToxicChat (Lin et al., 2023). Each example comes with a prompt, composed of the guardrail instructions, a user input, and a user output; a multi-step reasoning trace separated into the three tasks of request moderation, refusal detection, and response moderation; and ground-truth answers for the three tasks. Using the same reasoning-augmented supervision source as GuardReasoner allows us to directly compare explicit rationale generation against latent internalization under a matched training signal. Both iCoT (Deng et al., 2025) and Coconut (Hao et al., 2025) show that replacing too many language tokens per stage can destabilize training. We therefore split reasoning traces into smaller step-level replacements, but this increases the number of training stages and overall computational cost. To reduce cost and focus on request and response safety moderation, we remove the refusal task from C O L AG UARD training supervision.

Response Harmfulness Detection HarmBench (Mazeika et al., 2024) SafeRLHF (Ji et al., 2024) BeaverTails (Ji et al., 2023) XSTest (Röttger et al., 2024) WildGuardTest (Han et al., 2024)

602 2,000 3,021 446 1,768

Table 1: Evaluation benchmarks for prompt and response harmfulness detection.

The same masked language-modeling objective is used for the final compression stage, with supervised positions restricted to the final safety-label tokens. As k increases, less of the original rationale remains in language space, forcing more of the safety decision process to be represented by latent recurrence. This objective gives the latent positions no direct textual target so that the latent states are optimized through their downstream ability to predict the remaining rationale steps and the final safety labels. 3.6

Efficient Inference

At inference time, C O L AG UARD receives only the instruction, prompt, and response. It appends a fixed latent span and performs recurrent latent computation using the fused update in §3.4. Let  ZL = ⟨start-latent⟩, z1 , . . . , zL , ⟨end-latent⟩ denote the fixed latent span, where L is the latent budget used at deployment. After the latent span, the model returns to language mode and autoregressively predicts the final prompt and response safety labels: (ŷ p , ŷ r ) = arg

max

(y p ,y r )∈Y 2

Training Details. We use separate training configurations for the explicit CoT warm-up stage and the latent internalization stages. In Stage 0, we fully fine-tune Llama 3.1 8B on GuardReasonerTrain to obtain an explicit reasoning baseline. Training is performed on 8×A100 (80GB) GPUs for 3 epochs, with per-device batch size 1, gradient accumulation 32, AdamW optimization (Loshchilov and Hutter, 2019), a cosine learning-rate schedule, and an initial learning rate of 5×10−5 . The fusion coefficient is set to α = 1.0 in this stage, so the fusion module is inactive. Starting from the Stage-0 checkpoint, we then train the stage-wise internalization curriculum. Since roughly 80% of GuardReasonerTrain examples contain at most six reasoning steps, we use six latent recurrent steps as the fixed inference budget. Each internalization stage replaces one additional reasoning step with latent states, and a final

pθ (y p , y r | I, x, s, ZL ).

Because C O L AG UARD does not generate natural-language rationales, its inference cost scales with the number of latent positions rather than the length of an explicit chain-of-thought.

4

Experimental Setup

Experiments

To evaluate C O L AG UARD, we conduct experiments on multiple safety benchmarks, comparing 5

Table 2: F1 Score (%) of Models on 5 Benchmarks of Prompt Harmfulness Detection. Bold and underlined values denote the best and runner-up. “–” denotes the result is unavailable. Method

Model Size

ToxicChat HarmBench

OpenAI Aegis WildGuard Macro Micro Mod. SafetyTest Test Avg Avg

Closed-Source Guard API GPT-4o GPT-4o+CoT o1-preview

Unknown Unknown Unknown

64.46 73.43 57.69

82.27 81.98 89.61

62.26 76.78 74.60

81.07 88.24 83.15

80.87 82.75 76.31

74.19 80.64 76.27

69.59 77.69 69.00

74.10 71.80 99.50 84.80 82.90 – 7.47 77.63 89.40 80.23 89.34 91.39 90.18

56.00 70.90 68.47 78.50 71.50 81.60 9.36 57.74 88.90 66.02 87.37 89.01 89.17

66.94 71.98 79.94 75.70 74.52 – 9.89 69.97 84.02 65.87 83.10 83.91 84.40

64.48 63.16 67.52 72.60 73.46 – 9.44 68.43 77.68 53.47 77.37 80.48 80.83

90.58 89.45

88.15 89.44

84.28 84.23

79.49 79.77

Open-Source Guard Model LLaMA Guard LLaMA Guard 2 LLaMA Guard 3 Aegis Guard Defensive Aegis Guard Permissive Aegis Guard 2.0 ShieldGemma ShieldGemma WildGuard QwQ-preview GuardReasoner GuardReasoner GuardReasoner

7B 8B 8B 7B 7B 8B 2B 9B 7B 32B 1B 3B 8B

61.60 47.10 53.12 70.00 73.00 – 6.91 67.92 70.80 34.81 72.43 78.20 78.79

67.20 94.00 98.94 77.70 70.50 – 11.81 67.96 98.90 86.73 96.31 89.10 91.86

75.80 76.10 79.69 67.50 74.70 81.00 13.89 78.58 72.10 61.58 70.06 71.87 72.00

Latent Reasoning Guardrail (Ours) C O L AG UARD (Ours) C O L AG UARD (Ours)

3B 8B

75.27 75.26

94.25 93.54

compression stage removes any remaining explicit reasoning for longer traces while preserving the same six-step latent budget. Each stage is trained for one epoch with a reset AdamW optimizer and a constant learning rate of 1 × 10−5 . During internalization, we linearly anneal the fusion coefficient from α = 1.0 to α = 0.6 over the first 200 warm-up steps. We set the fusion temperature to 1.0, top-p to 0.9, and use a fusion adapter with hidden dimension 1024. All training is conducted in bf16 precision, and checkpoints are saved after each stage. For the 3B model, we use Llama 3.2 3B as the backbone and set the internalization learning rate to 2 × 10−5 , which we found more stable for this scale. Following the implementation choice in Liu et al. (2026), we disable the fusion adapter for the 3B model because Llama 3.2 3B uses tied inputoutput embeddings. All other training settings are kept identical to the 8B configuration.

73.15 73.45

soning baseline. More details on these benchmarks can be found in Appendix A. We compare C O L AG UARD against 20 baselines spanning closed-source APIs, open-source guard models, and our primary explicit reasoning baseline. Baseline names and model sizes are reported in Tables 2 and 3; corresponding references are provided in Appendix A.2. 4.2

Results

Overall Classification Performance. Tables 2 and 3 report F1 scores on prompt and response harmfulness detection. C O L AG UARD 8B is comparable to GuardReasoner 8B, with prompt macroF1 of 84.23 vs. 84.40 and response macro-F1 of 83.33 vs. 83.13. Compared with Llama Guard 3, it improves the average macro-F1 across both tasks by 8.24 points while avoiding explicit rationale generation. At the benchmark level, C O L AG UARD 8B achieves the best F1 on WildGuardTest for both prompt and response detection (89.44 and 81.23), and ranks second on HarmBench response and SafeRLHF (86.38 and 70.49). Its lower prompt micro-F1 relative to GuardReasoner 8B (79.77 vs. 80.83) is mainly due to ToxicChat, which accounts for 41.4% of the prompt evaluation set and therefore has a large effect on the micro average.

Safety Evaluation. To assess the performance and efficiency of our guardrail model while isolating the effect of latent reasoning against explicit rationale generation under matched supervision, we evaluate on benchmarks used by Liu et al. (2025) (Table 1) and use GuardReasoner (SFT-only, without hard-sample DPO) as our primary explicit rea6

Table 3: F1 Score (%) of Models on 5 Benchmarks of Response Harmfulness Detection. Bold and underlined values denote the best and runner-up. “–” denotes the result is unavailable. Model Size

Method

HarmBench SafeRLHF BeaverTails

XSTest WildGuard Macro Micro Response Test Avg Avg

Closed-Source Guardrail API GPT-4o GPT-4o+CoT o1-preview

Unknown Unknown Unknown

56.34 65.99 76.40

64.05 65.10 66.60

78.63 82.26 79.96

65.12 86.90 74.75

65.24 71.43 50.00

65.88 74.34 69.54

69.41 74.45 69.22

82.00 90.80 87.67 52.80 60.40 86.20 65.55 73.86 64.50 72.00 90.40 83.60 94.70 45.95 90.12 91.36 94.34

50.50 66.50 70.80 49.10 56.40 77.50 20.13 47.00 45.70 60.10 76.80 63.40 75.40 17.56 74.81 79.70 78.20

60.00 71.70 71.15 59.62 61.46 – 33.79 57.60 66.32 69.34 80.04 73.48 81.00 54.64 80.78 82.49 83.13

58.27 66.99 64.97 62.79 63.55 – 27.24 55.67 65.49 66.70 78.67 76.60 77.95 57.73 79.06 80.80 81.22

94.19 92.02

77.23 81.23

82.56 83.33

80.22 81.55

Open-Source Guardrail LLaMA Guard LLaMA Guard 2 LLaMA Guard 3 Aegis Guard Defensive Aegis Guard Permissive Aegis Guard 2.0 ShieldGemma ShieldGemma HarmBench LLaMA HarmBench Mistral MD-Judge BeaverDam WildGuard QwQ-preview GuardReasoner GuardReasoner GuardReasoner

7B 8B 8B 7B 7B 8B 2B 9B 13B 7B 7B 7B 7B 32B 1B 3B 8B

52.00 77.80 85.07 62.20 60.80 – 35.36 56.44 84.30 87.00 81.60 58.40 86.30 69.65 84.75 85.66 85.47

48.40 51.60 44.36 59.30 55.90 – 16.92 47.07 60.00 52.40 64.70 72.10 64.20 62.76 68.39 69.02 70.04

67.10 71.80 67.84 74.70 73.80 – 30.97 63.61 77.10 75.20 86.70 89.90 84.40 77.26 85.84 86.72 87.60

Latent Reasoning Guardrail (Ours) C O L AG UARD C O L AG UARD

3B 8B

86.36 86.38

68.72 70.49

86.29 86.55

Table 4: Inference Efficiency and Performance Comparison. We report inference time, completion token cost, and efficiency-adjusted F1 (EA-F1). Inference is conducted on 1×H100 (80GB) GPU. EA-F1 denotes EfficiencyAdjusted F1 (Wen et al., 2025a), a normalized metric that jointly accounts for F1 score and inference speed, where higher values indicate better efficiency-performance trade-off. 3B

Metric

8B

GuardReasoner C O L AG UARD GuardReasoner C O L AG UARD Time Cost (ms/query) Token Cost (token/query) EA-F1

3801.03 281.96 0.2122

318.9 13.0 2.5041

Model Size Comparison. C O L AG UARD 3B is already competitive with GuardReasoner 3B, slightly improving both prompt macro-F1 (84.28 vs. 83.91) and response macro-F1 (82.56 vs. 82.49). Scaling to 8B mainly benefits response detection and yields better combined averages (83.78 vs. 83.42 macro; 80.66 vs. 79.86 micro), suggesting a modest but more consistent gain from the larger backbone.

4407.8 289.4 0.1838

342.0 12.9 2.3601

kens/query, a 22.4× reduction. These gains come from replacing long autoregressive CoT generation with a fixed six-step latent recurrence. C O L AG UARD also achieves much higher EA-F1 at both model sizes, showing a stronger accuracyefficiency trade-off for deployment. 4.3

Ablation Studies

Analyzing Latent Recurrence Dynamics. Recent work questions whether latent tokens in Coconut-style recurrence perform meaningful computation beyond acting as learned placeholders. Zhang et al. (2025) find that vanilla Coconut tokens form clustered embeddings with limited input sensitivity, suggesting placeholder behavior

Inference Efficiency. Table 4 shows that C O L A G UARD substantially reduces inference cost compared with GuardReasoner. At 8B, latency drops from 4,407.8 to 342.0 ms/query, a 12.9× speedup, while token usage decreases from 289.4 to 12.9 to7

h0 (start) h5 (end)

Harmful mean trajectory Unharmful mean trajectory

85

CoLaGuard h0 h0

h0 h0

h5

h5

h5 h5

h0

h0

h1

h1

h2

h2

h3

h3

h4

h4

h5 h1

h2

h3

h4

h5

h0

h1

h2

83

83.11 83.14

82 81

h3

h4

h5

79

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

83.33

82.58 81.17

Prompt Response

80

h5 h0

84.23

84

Macro F1 (%)

Vanilla COCONUT

8k

30k

Training Data Size

127k

Figure 3: Training Data Scaling. C O L AG UARD 8B prompt and response macro-F1 across training data sizes.

79.78 combined micro-F1, compared with 83.78 and 80.72 for C O L AG UARD. Context-Prediction Fusion yields clear gains that bring it to parity with the explicit reasoning baseline (+1.96 macro-F1, +0.94 micro-F1), suggesting that the more progressive latent shifts in Figure 2 may be relevant to downstream moderation performance.

Figure 2: Geometric Analysis of Latent Representations. (Top) UMAP of mean harmful/unharmful trajectories across recurrence steps h0 –h5 . (Bottom) Intra-sample cosine similarity heatmap between latent steps. Vanilla Coconut shows highly similar latent states and early label separation, while C O L AG UARD exhibits progressive class differentiation across recurrence steps.

Scaling Training Data. Figure 3 shows that C O L AG UARD 8B improves consistently with more reasoning-augmented training data. Response macro-F1 improves sharply from 8k to 30k examples (+1.97 points) but shows limited additional gain at 127k (+0.19 points). Prompt macro-F1 increases more gradually, gaining 0.53 points from 8k to 30k and 1.12 points from 30k to 127k. These trends suggest that response moderation benefits earlier from diverse supervision, while prompt moderation continues to improve at larger scale. Overall, the results show that C O L AG UARD scales reliably with training data and achieves its best performance with the full GuardReasonerTrain corpus.

from learned shortcuts. Liu et al. (2026) show that Context-Prediction Fusion mitigates inter-sample representational collapse, suggesting more expressive latent states. We extend this analysis to C O L A G UARD through WildGuardTest latent trajectories and a full-suite CPF ablation against vanilla Coconut. Figure 2 shows the average pairwise cosine similarity between latent steps (hi , hj ) across samples and mean harmful/unharmful trajectories via UMAP (McInnes et al., 2020). Vanilla Coconut exhibits uniformly high cross-step similarity, consistent with early commitment to a fixed latent state that is simply propagated forward; its harmful and unharmful trajectories are already separated at h0 , with limited additional separation in later steps. In contrast, C O L AG UARD shows noticeably lower cross-step similarity, indicating that its latent states continue to evolve throughout the recurrence rather than collapsing after the initial step. Its trajectories begin closer together and diverge progressively, suggesting that recurrence contributes to the refinement of safety-relevant representations rather than simply preserving an early decision. As an ablation of Context-Prediction Fusion, a vanilla Coconut guardrail with the same six-step latent budget reaches 81.82 combined macro-F1 and

5

Conclusion

We introduced C O L AG UARD, a latent reasoning guardrail that internalizes explicit safety reasoning through a stage-wise curriculum. Across prompt and response harmfulness detection benchmarks, C O L AG UARD matches the average macroF1 of an explicit reasoning guardrail while substantially reducing inference cost. C O L AG UARD 8B matches GuardReasoner 8B in macro-F1 while achieving 12.9× lower latency and 22.4× fewer tokens. These results show that latent reasoning is a practical path toward safety guardrails that are both robust and efficient for deployment. 8

Limitations

Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. Preprint, arXiv:2311.01460.

While C O L AG UARD demonstrates strong efficiency and competitive safety performance, several limitations remain. First, our evaluation focuses on text-based prompt and response harmfulness detection, leaving broader policy taxonomies, multilingual inputs, multimodal content, and long-horizon agent behavior for future work. Second, C O L A G UARD is trained from distilled reasoning traces and may inherit biases or coverage gaps from the underlying supervision. Finally, although our latent representation analysis suggests progressive safetyrelevant refinement, more causal interventions are needed to fully characterize how each latent step contributes to the final decision to improve interpretability of safety decisions.

Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. 2024. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts. Preprint, arXiv:2404.05993. Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. 2025. AEGIS2.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5992–6026, Albuquerque, New Mexico. Association for Computational Linguistics.

Ethics Statement

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations.

The aim of this work is to improve the reliability and efficiency of LLM safety guardrails. While latent reasoning moderation may make strong safety filters more practical in high-traffic settings, these guardrails can still produce false positives and false negatives on ambiguous or context-dependent inputs. Therefore, C O L AG UARD itself should not be considered a replacement for human oversight in real-world deployment, but rather should be used as part of a broader moderation system. The safety data used in the evaluation and training processes may contain harmful or sensitive content and should be handled with appropriate access controls and annotator-care practices.

Aaron Grattafiori and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in neural information processing systems, 37:8093–8131. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E Weston, and Yuandong Tian. 2025. Training large language model to reason in a continuous latent space. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, Toronto, Canada. Association for Computational Linguistics.

References Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. Hopping too late: Exploring the limitations of large language models on multi-hop queries. Preprint, arXiv:2406.12775. Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. 2025. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. Preprint, arXiv:2505.16782.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.

Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed chain of thought: Efficient reasoning through dense representations. Preprint, arXiv:2412.13171.

Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, and Yaodong Yang. 2024. PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human

Yuntian Deng, Yejin Choi, and Stuart Shieber. 2025. From explicit CoT to implicit CoT: Learning to internalize CoT step by step. In The Thirteenth International Conference on Learning Representations.

9

preference. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.

Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction. Preprint, arXiv:1802.03426.

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

NVIDIA. 2025. Nemotron Content Safety Reasoning 4B. https://huggingface.co/nvidia/ Nemotron-Content-Safety-Reasoning-4B. OpenAI. 2024. OpenAI o1 system card. https:// openai.com/index/openai-o1-system-card/.

Mintong Kang and Bo Li. 2025. $r^2$-guard: Robust reasoning enabled LLM guardrail via knowledgeenhanced logical reasoning. In The Thirteenth International Conference on Learning Representations.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155.

Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. Preprint, arXiv:2305.14045.

Jacob Pfau, William Merrill, and Samuel R. Bowman. 2024. Let’s think dot by dot: Hidden computation in transformer language models. Preprint, arXiv:2404.15758.

Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. Preprint, arXiv:2402.05044.

Qwen Team. 2024. QwQ: Reflect deeply on the boundaries of the unknown. https://qwenlm.github. io/blog/qwq-32b-preview/.

Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. Preprint, arXiv:2310.17389.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.

Weihao Liu, Dehai Min, and Lu Cheng. 2026. Latent thoughts tuning: Bridging context and reasoning with fused information in latent tokens. Preprint, arXiv:2602.10229.

Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. Preprint, arXiv:2310.10501.

Yue Liu, Hongcheng Gao, Shengfang Zhai, Yufei He, Jun Xia, Zhengyu Hu, Yulin Chen, Xihong Yang, Jiaheng Zhang, Stan Z. Li, Hui Xiong, and Bryan Hooi. 2025. Guardreasoner: Towards reasoning-based llm safeguards. Preprint, arXiv:2501.18492.

Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5377–5400, Mexico City, Mexico. Association for Computational Linguistics.

Llama Team. 2024. Meta Llama guard 2. https: //github.com/meta-llama/PurpleLlama/blob/ main/Llama-Guard2/MODEL_CARD.md. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. Preprint, arXiv:2208.03274.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. Preprint, arXiv:2402.04249.

Makesh Narsimhan Sreedhar, Traian Rebedea, and Christopher Parisien. 2025. Safety through reasoning: An empirical study of reasoning guardrail models. Preprint, arXiv:2505.20087.

10

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903. Xiaofei Wen, Wenjie Jacky Mo, Yanan Xie, Peng Qi, and Muhao Chen. 2025a. Towards policy-compliant agents: Learning efficient guardrails for policy violation detection. arXiv preprint arXiv:2510.03485. Xiaofei Wen, Wenxuan Zhou, Wenjie Jacky Mo, and Muhao Chen. 2025b. Thinkguard: Deliberative slow thinking leads to cautious guardrails. arXiv preprint arXiv:2502.13458. Yahan Yang, Soham Dan, Shuo Li, Dan Roth, and Insup Lee. 2025. MrGuard: A multilingual reasoning guardrail for universal LLM safety. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27377–27396, Suzhou, China. Association for Computational Linguistics. Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li. 2024. RigorLLM: Resilient guardrails for large language models against undesired content. In Proceedings of the 41st International Conference on Machine Learning (ICML). Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. 2024. Quiet-STar: Language models can teach themselves to think before speaking. In First Conference on Language Modeling. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. 2024. Shieldgemma: Generative ai content moderation based on gemma. Preprint, arXiv:2407.21772. Yuyi Zhang, Boyu Tang, Tianjie Ju, Sufeng Duan, and Gongshen Liu. 2025. Do latent tokens think? a causal and adversarial analysis of chainof-continuous-thought. Preprint, arXiv:2512.21711. Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, Baosong Yang, Chen Cheng, Jialong Tang, Jiandong Jiang, Jianwei Zhang, Jijie Xu, Ming Yan, Minmin Sun, Pei Zhang, and 24 others. 2025. Qwen3guard technical report. Preprint, arXiv:2510.14276. Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang, Yong Shan, Tianle Cai, Taylor Kergan, Assel Kembay, Andrew Smith, Chenghua Lin, Binh Nguyen, Yuqi Pan, Yuhong Chou, Zefan Cai, and 14 others. 2025. A survey on latent reasoning. Preprint, arXiv:2507.06203.

11

A

Safety Evaluation

A.1

Description of Benchmarks

To assess the performance and efficiency of our latent reasoning guardrail model, we evaluate it across eight unique safety-related benchmarks. WildGuard (Han et al., 2024): WildGuardMix is a large-scale safety moderation dataset with 92,000 labeled examples that cover both normal and adversarial prompt behaviors that come coupled with corresponding refusal and compliance responses. The WildGuardTest split is human-annotated and covers 5,000 safety labeled examples. ToxicChat (Lin et al., 2023): ToxicChat is a benchmark that includes 10,000 real user queries, leveraged as adversarial prompts for testing content moderation and toxicity detection in human-AI interactions. Aegis Safety Test 1.0 (Ghosh et al., 2024): A dataset of approximately 11,000 manually annotated examples, Aegis Safety Test 1.0 was curated with the purpose of testing LLM safety alignment in accordance with Nvidia’s content safety taxonomy. HarmBench (Mazeika et al., 2024): HarmBench is a framework that is systematically designed to address the lack of standardized evaluation frameworks in the field of automated red teaming. By leveraging various behaviors, this framework can be used to generate red-teaming test cases for evaluating the adversarial robustness of LLMs. OpenAI Moderation (Markov et al., 2023): A benchmark for assessing LLMs’ ability to detect harmful content based on OpenAI’s safety guidelines, covering violence, self-harm, and misinformation. SafeRLHF (Ji et al., 2024): A dataset of 82,000 questions with two responses each, every entry in SafeRLHF includes safety meta-labels as well as preference between the two responses. BeaverTails (Ji et al., 2023): The Beavertails dataset was introduced to further research on safety alignment in LLMs. The complete dataset includes over 300,000 question-answer pairs that are annotated with safety meta-labels and corresponding, violated safety categories. XSTest (Röttger et al., 2024): Developed to evaluate refusal behaviors and identify systematic failure modes in large language models, XSTest is comprised of 250 safe prompts across ten prompt types and contrasting 200 unsafe prompts that humanaligned models should refuse.

Baseline

Reference

Model Size

GPT-4o GPT-4o + CoT o1-preview LLaMA Guard LLaMA Guard 2 LLaMA Guard 3 Aegis Guard Defensive Aegis Guard Permissive Aegis Guard 2.0 ShieldGemma WildGuard QwQ-preview HarmBench LLaMA HarmBench Mistral MD-Judge BeaverDam GuardReasoner

OpenAI (2024) OpenAI (2024) OpenAI (2024) Inan et al. (2023) Llama Team (2024) Grattafiori et al. (2024) Ghosh et al. (2024) Ghosh et al. (2024) Ghosh et al. (2025) Zeng et al. (2024) Han et al. (2024) Qwen Team (2024) Mazeika et al. (2024) Mazeika et al. (2024) Li et al. (2024) Ji et al. (2023) Liu et al. (2025)

Unknown Unknown Unknown 7B 8B 8B 7B 7B 8B 2B / 9B 7B 32B 13B 7B 7B 7B 1B / 3B / 8B

Table 5: Baseline references and model sizes for Tables 2 and 3.

A.2

Baseline Details

Details are presented in Table 5.

12