Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defense Nataraj Agaram Sundar
Tejas Morabia
arXiv:2606.01513v1 [cs.DC] 1 Jun 2026
eBay Inc. San Jose, CA, USA
Abstract—High-stakes enterprise document generation (e.g., financial dispute narratives, compliance notices, audit summaries) demands schema correctness, policy compliance, and low-latency operation at scale. Prior to a unified guardrail layer, production systems often stitched together separate PII redaction, content moderation, and format validation steps, leading to fragmented logic, slower request paths, and higher operational cost. We present a guardrail orchestration layer for text and image inputs that couples multi-candidate generation with an explicit compliance score used for early exit. The framework runs configurable parallel “heads,” scores candidates against weighted guardrails (PII detection, content moderation, schema constraints, and domain rules), and returns the best-scoring output with selection metadata. The available operational readout reports 5 attempts within 20 s and 91 % compliance. For payments dispute defense summaries, we analyze aggregate operational scenario readouts rather than a randomized A/B test. Variable cohorts show higher count win rates than controls overall (301/659 vs. 536/1548; +11.0 pp, 95% CI [6.6, 15.5], p < 0.001) and for adjusted item-not-received cases (+7.5 pp, 95% CI [0.2, 15.7], p = 0.045). Fraud and local evidenceranking deltas are directionally positive but not statistically significant from the aggregate count data. We also report reviewer-calibrated Responsible-AI evidence-quality signals from 770 generated-evidence reviews and a 70-case OCR slice, and we document the reproducibility boundary through the request interface, scoring logic, pseudocode, and operational evidence boundary. Index Terms—multimodal generative AI, guardrails, PII detection, content moderation, structured generation, distributed systems, best-of-N sampling, payments, chargebacks
tation creates practical issues for distributed systems: sequential latency accumulation [1], inconsistent rule interpretation across products, duplicated engineering effort, and higher manual rework when failures require human triage. This paper focuses on guardrails as a first-class distributed systems component. We introduce a unified guardrail layer that (i) standardizes policy enforcement across use cases, (ii) supports multimodal inputs (text and images), and (iii) uses an explicit compliance score to drive best-of-N selection and early exit.
I. I NTRODUCTION
II. R ELATED W ORK
Enterprises increasingly rely on automated document generation for compliance, finance, and customer operations. Unlike open-ended text generation, operational documents are typically schema-constrained (JSON outputs, fixed sections, strict length caps), policy-bounded (forbidden terms, required inclusions, privacy constraints), and high-volume (tight latency and throughput requirements). Large language models (LLMs) provide strong fluency, but production deployments surface reliability and safety risks: schema violations, missing required fields, policy breaches, and leakage of personally identifiable information (PII). A common pre-deployment pattern is to bolt on independent gates—PII detection, content moderation, format validators— as separate services. While effective in isolation, this fragmen-
Safety, moderation, and privacy for LLMs. Safety work in LLM deployment spans model-level alignment, runtime moderation, and system-level governance. Instruction tuning and RLHF reduce harmful behavior at the model layer [3], while Constitutional AI replaces some human preference labels with rule-based critique [4]. Recent moderationspecific systems such as Llama Guard and Llama Guard 3 provide input/output safeguard models and compact deployment variants [15], [16]. Other recent guardrail models focus on adversarial robustness and explicit reasoning over safety categories, including RigorLLM and R2 -Guard [17], [18]. These approaches are complementary to the runtime orchestration proposed here: they can be used as individual guardrail checks inside the scoring service, but do not by
A. Contributions Unified guardrail orchestration: a configuration-first framework that “defines once and enforces everywhere,” consolidating PII, moderation, schema checks, and domain rules. • Compliance-scored best-of-N : a multi-head generation and scoring loop that returns the highest-compliance candidate, stopping early at a configurable threshold. • Multimodal support: a single guardrail surface for text and image evidence (e.g., proof-of-delivery screenshots), enabling consistent privacy and safety enforcement. • Transparent operational evaluation: aggregate scenario-level outcome comparisons with explicit non-randomized design caveats, count-based confidence intervals, effect sizes, and Responsible-AI review signals. •
themselves define a full production request path with candidate selection, metadata, and fallback routing. Programmable and policy-level guardrails. System-level governance practices such as model cards emphasize documenting intended use, limitations, and monitoring [5]; recent risk frameworks and practitioner taxonomies emphasize prompt injection, insecure output handling, data leakage, and human oversight for LLM applications [24], [25]. NeMo Guardrails shows how programmable rails can control conversational LLM applications through interpretable runtime policies [14]. Our work extends this direction to multimodal structured document generation, where the policy engine must combine PII checks, moderation, schema validation, domain rules, early-exit selection, and asynchronous observability. Prompt injection and policy bypass. Prompt injection highlights the brittleness of relying solely on prompt text to enforce constraints. Attacks can cause models to ignore developer instructions or reveal sensitive context [7]. The OWASP LLM taxonomy treats prompt injection as a first-order application security risk [25]. More recent work also shows that guardrail judgments can shift under retrieval-augmented contexts, motivating explicit monitoring of the context in which a guardrail is evaluated [23]. This motivates guardrails that are evaluated after generation as an external check, and in some cases enforced during generation via constrained decoding. Structured and constrained generation. Many enterprise use cases require strict schemas (JSON, fixed fields). Constrained decoding and grammar-based generation provide mechanisms to guarantee structure [8]. Recent guidedgeneration work reformulates constrained neural decoding through finite-state machines and grammar constraints, improving reliability for structured interfaces [19]. In practice, production systems often combine constrained formats with validation and repair loops that re-prompt or regenerate when failures occur. Best-of-N and self-consistency. Multi-sample decoding and selection is a practical inference-time strategy for improving quality without retraining. Self-consistency demonstrates that sampling multiple reasoning traces and aggregating can improve correctness [9], [10]. We apply a related principle to compliance: sample multiple candidates and select the one maximizing an explicit guardrail score. Retrieval-augmented generation and evidence grounding. Retrieval-augmented generation (RAG) conditions generation on external evidence to reduce hallucinations [11]. Payments dispute packages similarly rely on evidence such as tracking scans, AVS/CVV signals, proof-of-delivery images, and customer/seller message context. Guardrails complement RAG by enforcing evidence inclusion, banning unverifiable claims, and routing risky outputs for review. Evaluation and operational ML. Holistic evaluation frameworks such as HELM and VHELM emphasize that deployed language and vision-language systems must be measured across dimensions such as robustness, toxicity, fairness, safety, and efficiency rather than accuracy alone [20], [21].
Fig. 1. Legacy baseline pipeline. A single-pass generation step is followed by separate PII detection, moderation, and manual triage/re-prompting before a defense package is emitted.
PROMPTEVALS further highlights the practical need for assertions and guardrails in production LLM pipelines [22]. From the systems perspective, production ML systems incur hidden technical debt through glue code, pipeline complexity, and monitoring gaps [2], while tail latency dominates user experience at scale [1]. The guardrail layer is designed as a distributed component that keeps request-path enforcement synchronous while pushing telemetry aggregation and configuration updates off the critical path. III. L EGACY BASELINE : F RAGMENTED G UARDRAIL G ATES Before the unified guardrail layer, production documentgeneration flows resembled a pipeline of independent gates: an LLM produced a draft, then PII detection and content moderation were applied as separate services; failures triggered manual triage or re-prompting. This pattern has two core drawbacks. (1) Latency and retry amplification. Each additional synchronous service call adds tail latency in the request path [1]. When a gate fails, the system either blocks the request for manual review or initiates retries/re-prompts, further increasing latency and cost. (2) Inconsistent enforcement and operational overhead. Different teams implement “similar” rules (e.g., what constitutes PII leakage) in slightly different ways, leading to divergence over time. Fixes must be replicated across services, and monitoring becomes fragmented. Figures 1 and 2 illustrate the architectural difference between the pre-guardrail baseline and a unified guardrail-driven pipeline.
Fig. 2. Unified guardrail-driven pipeline. Candidate generation, guardrail scoring, best-result selection, and selection metadata are integrated into the request path. TABLE I Q UALITATIVE COMPARISON OF BASELINE VS . UNIFIED GUARDRAIL LAYER . Capability
Baseline
Unified layer
×
✓
×
✓
× ×
✓ ✓
×
✓
partial
✓
Single enforcement surface for policies Integrated PII + moderation + schema Best-of-N selection with early exit Selection metadata (attempts/latency/errors) Config-first rule iteration (no code changes) Multimodal evidence (text + image)
Fig. 3. Synchronous request path. Evidence is ingested and canonicalized; the request router selects the model and policy tier; generation heads produce candidates; the scoring service evaluates PII, moderation, schema, and domain rules; the early-exit selector returns a compliant summary or routes belowthreshold/high-risk cases to human review.
This normalization enables uniform thresholding across heterogeneous rule sets. Domain rules can be included as guardrails (e.g., “do not mention prohibited dispute terms” or “ensure a boolean is returned”) and weighted by severity.
IV. G UARDRAIL O RCHESTRATION F RAMEWORK The guardrail layer is designed as a reusable library and service that embeds a guardrail-driven selection loop into distributed generation systems. Key design principles are: configfirst policy, parallel candidate exploration, explicit compliance scoring, and operational telemetry. Figures 3 and 4 split the design into the synchronous request path and the asynchronous monitoring/configuration loop. A. Guardrail Set and Compliance Score We represent guardrails as a collection of checks G = {g1 , . . . , gm }, where each gj (y) ∈ [0, 1] measures whether candidate output y satisfies a constraint (e.g., schema validity, disallowed terms, PII leakage, toxicity). Each check has a nonnegative weight wj . Pm C(y) =
j=1 wj gj (y) Pm ∈ [0, 1], j=1 wj
CompliancePercentage(y) = 100 · C(y).
(1)
B. Best-of-N Candidate Generation with Early Exit The core runtime loop is “best-of-N under a latency budget.” For each request, the orchestrator generates candidates in parallel and scores each against Eq. (1). If a candidate meets a compliance threshold τ , the system exits early. Figure 5 illustrates the quality-control loop, and Algorithm 1 summarizes the core logic. Algorithm 2 makes the multimodal enforcement surface explicit: both text and image evidence are normalized into a single representation before scoring, ensuring that privacy and safety guardrails apply consistently across modalities. Algorithm 3 highlights the production deployment detail that is easy to miss in purely offline evaluations: enforcement and scoring must be synchronous to guarantee policy compliance, while telemetry, dashboards, and configuration updates are pushed off the critical path. Synchronous vs asynchronous updates. Validation and compliance scoring occur synchronously within the request
Algorithm 1 Guardrail-Driven Best-of-N Generation Require: input x (text/image evidence), guardrails G, weights w, threshold τ , timeout Tmax , heads H 1: y ∗ ← ∅, c∗ ← −∞, start timer 2: while elapsed < Tmax do 3: Generate H candidates in parallel: {y1 , . . . , yH } ← G ENERATE(x) 4: for each yh do 5: ch ← C(yh ) using Eq. (1) 6: if ch > c∗ then 7: y ∗ ← yh , c∗ ← ch 8: end if 9: end for 10: if c∗ ≥ τ then 11: break 12: end if 13: end while 14: return y ∗ and SelectionMetadata (attempts, latency, errors)
Fig. 4. Control and observability loop. Policy settings remain on the synchronous path, while telemetry, dashboards, drift monitoring, review outcomes, and configuration feedback are asynchronous.
Algorithm 2 Multimodal Evidence Normalization and Privacy Filtering Require: text evidence etext , optional image evidence eimg , flags (PII, moderation), redaction policy ρ 1: u ← etext 2: if eimg present then 3: u ← u⊕E XTRACT T EXT(eimg ) {OCR or vision-capable model} 4: end if 5: u ← C ANONICALIZE (u) {normalize whitespace, timestamps, locales} 6: if PII enabled then 7: (p, λ) ← D ETECT PII(u) {entities + confidence} 8: u ← R EDACT(u, p, ρ) {mask identifiers, preserve semantics} 9: end if 10: if Moderation enabled then 11: m ← M ODERATE(u) 12: if m violates policy then 13: flag violation (score penalty) 14: end if 15: end if 16: return normalized evidence u and enforcement flags
C. Integrated PII and Content Moderation
Fig. 5. Multi-head best-of-N quality control with explicit compliance scoring and early exit at threshold τ .
path to guarantee that returned outputs meet policy thresholds. Telemetry aggregation, dashboards, and configuration updates are applied asynchronously to avoid request-path latency and support safe iteration.
The unified guardrail layer treats privacy and safety checks as first-class guardrails rather than external gates. In addition to format checks (e.g., strict JSON schema) and domain-specific rules (e.g., required evidence inclusion), the compliance score includes: • PII detection: penalizes outputs that expose sensitive identifiers (names, addresses, account numbers, tracking identifiers), optionally applying redaction. • Content moderation: filters policy-violating or disallowed content using a configurable severity tier.
Algorithm 3 Synchronous Enforcement with Asynchronous Monitoring and Policy Iteration Require: request x, cached guardrail config θ (versioned), async queue Q 1: θ ← G ET C ONFIG () {fast path: cached; slow path: refresh} 2: x′ ← N ORMALIZE E VIDENCE(x) using Alg. 2 3: (y ∗ , c∗ , meta) ← B EST O F N(x′ , θ) using Alg. 1 4: return y ∗ {synchronous: only policy-compliant output is returned} 5: enqueue (meta, c∗ , θ version) to Q {asynchronous} 6: background worker: aggregate metrics; update dashboards; propose θ changes; hot-reload
Fig. 6. Aggregate run characteristics disclosed in the approved public readout: candidate attempts, latency budget, and compliance. Request-level latency percentiles, cost, fallback rate, human-review reduction, and ablation metrics are not reported in this version because they require validated request-level telemetry exports.
TABLE II E XAMPLE GUARDRAIL API SURFACE ( REQUEST FIELDS ). Field
Purpose
text image modelName piiDetection Content moderation moderation Level guardrail guardrail Level uniqueId
Evidence text (e.g., tracking events) Optional image evidence (multipart) Target model (sandbox vs production) Enable/disable PII checking and redaction Enable/disable moderation checks Moderation strictness (policy tier) Domain rule (e.g., “ensure boolean returned”) Domain strictness / threshold knob Request identifier for audit and traceability
D. Multimodal Inputs: Images and Text Payments disputes frequently include images (shipping labels, proof-of-delivery screenshots). The unified guardrail layer supports multimodal evidence by (i) extracting text via OCR or a vision-capable model, and (ii) applying the same PII and moderation guardrails across both modalities. This unifies enforcement regardless of evidence type and reduces the need for ad hoc image-handling workflows. V. I NTEGRATION S URFACE The guardrail layer is exposed as a service endpoint that accepts evidence and policy knobs as request parameters. The following fields (derived from a production-style request invocation) illustrate the integration surface: VI. E VALUATION AND R ESULTS We evaluate the unified guardrail layer along four axes: (i) evaluation design and statistical assumptions, (ii) guardrail effectiveness and operational characteristics, (iii) ResponsibleAI evidence-quality review signals, and (iv) downstream outcome impact in payments dispute defense workloads. A. Evaluation Design and Statistical Method The payments outcome readouts in Table III are operational scenario comparisons, not a randomized A/B test. The variable groups correspond to guardrail-enabled AI defensesummary or evidence-ranking scenarios; controls correspond
to operational control cohorts from the same reporting context. Because the available exports do not prove random assignment, blocking, or matched-pair construction, we interpret the observed differences as associations and avoid claiming that the guardrail system alone caused the win-rate lift. Residual confounding is possible from case mix, dispute reason, network policy, seller behavior, evidence availability, and timewindow effects. For count-based win rates, we recover approximate wins by multiplying the reported dispute counts by the reported percentages and rounding to the nearest integer. We then compute Wilson 95% confidence intervals for each rate, Newcombe/Wilson intervals for rate differences, two-proportion z-tests, relative risk (RR), and odds-ratio effect sizes. This inference is therefore appropriate for aggregate count readouts, but it should be replaced by exact item-level analyses before camera-ready submission. Amount-weighted win rates are reported as effect sizes only: the PDFs provide aggregate amount-weighted percentages but not the per-dispute dollar amounts needed for valid bootstrap, permutation, or designbased confidence intervals. B. Compliance, Latency, and Public Operational Readout The guardrail layer emits request metadata including candidate attempts, latency, compliance score, rejection reasons, fallback decisions, and configuration version. In this public version, we report only the aggregate operational readout approved for disclosure: a typical run uses 5 candidate attempts, completes within a 20 s request budget, and reaches 91 % compliance. Figure 6 summarizes these disclosed values. C. Responsible-AI Evidence-Quality Review Signals Two additional review decks provide evaluator-facing quality signals for the evidence-generation and OCR components. For generated M2M dispute evidence, the corpus contains 770 unique reviews, with 318 labeled reviews available for estimation across five reviewers. Projected to the full corpus, approximately 574/770 reviews are rated 3/5, 157/770 are rated 4–5, and 39/770 are rated 1–2; the projected mean rating is 3.17. Narrative reviewer detail appears in 74.5% of labeled
TABLE III D OWNSTREAM OUTCOME READOUTS ( VARIABLE VS . CONTROL ). “W IN %” IS BY COUNT; “W INA MT %” IS AMOUNT- WEIGHTED . Scenario
Group
Disp. Cnt
Win%
WinAmt%
Overall AI Defense Summary
Variable Control
659 1548
45.7 34.6
41.4 22.3
Fraud Defense Summary
Variable Control
501 708
50.5 48.7
45.3 38.1
INR Defense Summary (adj.)
Variable Control
152 840
30.3 22.7
26.9 19.3
Evidence Ranking (Local)
Variable Control
445 406
17.30 14.53
24.5 17.9
TABLE IV C OUNT INFERENCE AND AMOUNT- WEIGHTED EFFECT SIZES . C OUNT INFERENCE USES ROUNDED WINS FROM AGGREGATE PERCENTAGES ; AMOUNT CI S / P - VALUES REQUIRE ITEM - LEVEL DISPUTED AMOUNTS . Scenario Overall AI Summary Fraud Summary INR Summary (adj.) Evidence Ranking
Var
Ctl
Count delta [95% CI]
p
Amt / rel.
301/659 253/501 46/152 77/445
536/1548 345/708 191/840 59/406
+11.0 [6.6, 15.5] +1.8 [-3.9, 7.5] +7.5 [0.2, 15.7] +2.8 [-2.2, 7.7]
< .001 .544 .045 .270
+19.1 / 1.86x +7.2 / 1.19x +7.6 / 1.39x +6.6 / 1.37x
Fig. 7. Downstream outcome improvements (variable vs. control) in evidence ranking and AI defense-summary scenarios. These are operational cohort deltas, not randomized causal estimates.
but not statistically significant by count; adjusted item-notreceived (INR) comparisons show a +7.5 pp count delta with a confidence interval barely above zero. Figure 7 visualizes the reported outcome deltas. F. Evidence Boundary for Runtime Ablations
exports, projecting to about 574/770 reviews, while only about 77/770 contain any structured issue flag. The most common structured issues are title-not-aligned (about 29) and titleinaccurate (about 27), indicating that a flag-only dashboard would miss much of the review nuance. For OCR evidence quality, the slice contains 70 cases, five analysts, and 350 ratings. The fields include text accuracy, word/line integrity, noise vs. non-text confusion, overall OCR, evidence utility, and pass/fail. The global weighted median of per-case mean Overall OCR scores is 3.2. Pairwise ordinal agreement is moderate to strong: the highest reported weighted kappa alignments are 0.753, 0.741, and 0.722; disagreement concentrates in blur drop-out (mean case SD 0.91), cropped text loss (0.90), and UI bleed or initial-capture artifacts (0.78). A clean chat case achieves Overall OCR 4.40 ± 0.55 with pass rate 1.00, while an analyzer-error label/receipt case scores 1.00 ± 0.00 with pass rate 0.00. These signals motivate a human-review fallback path for low-quality visual evidence and analyzer failures. D. Payments Dispute Defense Summaries: Outcome Readouts Chargeback defense summaries must be concise, evidencebacked narratives that comply with payment-network rules while avoiding prohibited language and PII leakage. Table III reports scenario-level outcomes comparing AI-driven variable scenarios with control baselines. E. Evidence Ranking and Summary Generation Lifts Local evidence ranking changes show a +2.8 pp count winrate delta and +6.6 pp amount-weighted delta, but the countbased aggregate comparison is not statistically significant at the 0.05 level. AI-generated defense-summary scenarios show the largest overall count delta (+11.0 pp) and amountweighted delta (+19.1 pp). Fraud-specific deltas are positive
The public aggregate readout supports the best-of-N operating point described above, but it does not contain enough request-level records to report validated ablations for single-candidate generation, N sensitivity, earlyexit threshold sweeps, guardrail-weight perturbations, or compliance/outcome calibration. We therefore do not report latency percentiles, cost, fallback rate, human-review reduction, or ablation results in this version. Those analyses require request-level telemetry with stable identifiers, timestamps, model/token usage, guardrail-failure labels, reviewer actions, and downstream outcomes. Section VIII treats this as a limitation rather than filling the missing fields with unverified values. VII. D ISCUSSION The unified guardrail layer demonstrates a practical pattern for high-stakes generation systems: treat policy and format constraints as a composable set of guardrails, compute an explicit compliance score used for early exit, and allocate generation budget to maximize compliance under latency limits. This buffers from post-hoc filtering in two ways. First, selection is score-driven: the engine returns the best candidate observed under a budget rather than accepting/rejecting a single draft. Second, policies become configuration: changes to rules and thresholds do not require code changes, enabling faster iteration. A. Latency, Throughput, and Cost Trade-offs Best-of-N generation increases model calls but can reduce total cost by lowering manual triage and avoiding downstream failures. A simple expected-cost model is E[Cost] = c E[Ncalls ] + m E[Nmanual ],
(2)
where c is per-call model cost and m is per-item manual handling cost. The unified guardrail layer reduces E[Nmanual ]
by enforcing PII, moderation, and schema constraints before output is emitted, while bounding E[Ncalls ] through early exit at threshold τ . From a systems perspective, a key advantage is that parallel heads trade compute for lower tail risk: under fixed budgets, selecting the maximum-compliance candidate reduces the probability of returning a non-compliant output. B. Reproducibility Boundary Raw dispute records, evidence screenshots, customer messages, and production code cannot be released. To support review without exposing proprietary data, the paper specifies the request interface, compliance-score formula, evidencenormalization algorithm, best-of-N selection loop, and asynchronous telemetry pattern. A de-identified artifact package may be released separately after organizational approval; until then, this version should be read as an applied systems paper with public algorithms and aggregate outcome readouts, not as a fully reproducible public benchmark. VIII. L IMITATIONS This work has several important limitations. First, the downstream outcome readouts are aggregate, non-randomized operational comparisons. Causal interpretation would require a randomized experiment, a matched cohort design, or covariateadjusted item-level analysis. We therefore describe observed deltas as associations rather than claiming that the guardrail layer alone caused the win-rate differences. Second, the public version does not include raw proprietary payments data, evidence screenshots, customer messages, or production code. External reproducibility is therefore limited to the algorithmic design, scoring equation, pseudocode, and disclosed aggregate metrics. Third, amount-weighted win rates are reported as aggregate effect sizes only. Statistically valid confidence intervals for amount-weighted outcomes require per-dispute dollar amounts and an appropriate bootstrap, permutation, or design-based test. Fourth, this version does not report request-level latency percentiles, per-summary cost, fallback rate, human-review reduction, or ablation results. Those values require validated production telemetry exports that were not available in approved aggregate form for this manuscript version. Fifth, the system is domain-specific to payments dispute defense. The guardrail scoring pattern may transfer to other structured document-generation settings, but rule weights, thresholds, risk routing, and human-review policies must be recalibrated for each domain. The Responsible-AI review decks also identify operational risks requiring human oversight, including title drift in generated evidence, analyzer/OCR failure, cropped text loss, UI bleed, and poor handwritten/blurry-image extraction. Future work includes contextual routing to select models/guardrail settings based on case features [12], learned calibration of weights wj in Eq. (1), request-level ablations for N , τ , and guardrail weights, and counterfactual evaluation for
policy changes [13]. Another direction is safety-aware reward modeling that explicitly penalizes privacy and toxicity risks [5], [6]. IX. C ONCLUSION We presented a compliance-scored, best-of-N guardrail orchestration framework that unifies PII detection, content moderation, schema validation, and domain rules into an explicit compliance score used for early exit. In payments dispute defense-summary scenarios, the available aggregate readouts show high compliance at bounded latency and positive operational outcome associations, with statistically significant count-based deltas overall and for adjusted INR cases. The next publication-ready step is to fill the request-level ablation, percentile latency, cost, fallback, and amount-weighted inference tables from production telemetry while retaining the human-review and reproducibility limits described above. R EFERENCES [1] J. Dean and L. A. Barroso, “The tail at scale,” Communications of the ACM, vol. 56, no. 2, pp. 74–80, 2013. [2] D. Sculley et al., “Hidden technical debt in machine learning systems,” in Proc. NeurIPS (Workshop), 2015. [3] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Proc. NeurIPS, 2022. [4] Y. Bai et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv:2212.08073, 2022. [5] M. Mitchell et al., “Model cards for model reporting,” in Proc. FAT*, 2019. [6] E. M. Bender et al., “On the dangers of stochastic parrots: Can language models be too big?” in Proc. FAccT, 2021. [7] E. Perez and M. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” arXiv:2211.09527, 2022. [8] C. Hokamp and Q. Liu, “Lexically constrained decoding for sequence generation using grid beam search,” in Proc. ACL, 2017. [9] X. Wang et al., “Self-consistency improves chain of thought reasoning in language models,” arXiv:2203.11171, 2022. [10] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” arXiv:2201.11903, 2022. [11] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Proc. NeurIPS, 2020. [12] L. Li et al., “A contextual-bandit approach to personalized news article recommendation,” in Proc. WWW, 2010. [13] P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” in Proc. ICML, 2021. [14] T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen, “NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails,” arXiv:2310.10501, 2023. [15] H. Inan et al., “Llama Guard: LLM-based input-output safeguard for human-AI conversations,” arXiv:2312.06674, 2023. [16] I. Fedorov et al., “Llama Guard 3-1B-INT4: Compact and efficient safeguard for human-AI conversations,” arXiv:2411.17713, 2024. [17] Z. Yuan, Z. Xiong, Y. Zeng, N. Yu, R. Jia, D. Song, and B. Li, “RigorLLM: Resilient guardrails for large language models against undesired content,” arXiv:2403.13031, 2024. [18] M. Kang and B. Li, “R2 -Guard: Robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning,” arXiv:2407.05557, 2024. [19] B. T. Willard and R. Louf, “Efficient guided generation for large language models,” arXiv:2307.09702, 2023. [20] P. Liang et al., “Holistic evaluation of language models,” arXiv:2211.09110, 2022. [21] T. Lee et al., “VHELM: A holistic evaluation of vision language models,” arXiv:2410.07112, 2024. [22] R. Vir, S. Shankar, H. Chase, W. Fu-Hinthorn, and A. Parameswaran, “PROMPTEVALS: A dataset of assertions and guardrails for custom production large language model pipelines,” arXiv:2504.14738, 2025.
[23] Y. She, D. W. Peterson, M. M. Liu, V. Upadhyay, M. H. Chaghazardi, E. Kang, and D. Roth, “RAG makes guardrails unsafe? Investigating robustness of guardrails under RAG-style contexts,” arXiv:2510.05310, 2025. [24] National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” NIST AI 600-1, 2024. [25] OWASP Foundation, “OWASP Top 10 for Large Language Model Applications 2025,” 2025. [Online]. Available: https://owasp.org/ www-project-top-10-for-large-language-model-applications/