E NTERPRISE VAL : Q UANTIFYING THE E FFICACY, R ELIABILITY AND VALUE OF G ENERATIVE AI IN THE E NTERPRISE
arXiv:2609.21841v1 [cs.AI] 18 Sep 2026
Abbas Raza Ali * 1 Muhammad Ajmal Siddiqui 2 Moona Zahid 3
A BSTRACT Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer “what can the model do?”, whereas a deployment decision requires “is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?”. We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to R EJECT/C ONDITIONAL/S CALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation. 1
I NTRODUCTION
The central challenge for enterprise AI is no longer whether foundation models can perform valuable professional work, but whether that capability can be translated into reliable, measurable, and economically sustainable enterprise outcomes. Three developments make this question increasingly urgent. First, model capability on realistic professional work has risen steeply: on GDPval, 1,320 expert-authored tasks across 44 occupations, the best frontier models in late 2025 produced deliverables rated as good as or better than experienced professionals in just under half of blinded pairwise comparisons, and performance more than tripled between GPT-4o and GPT-5 (Patwardhan et al., 2025). Second, realised enterprise value lags far behind: an MIT study of roughly 300 deployments found about 95% of pilots with no measurable P&L impact (Challapally et al., 2025), and McKinsey’s 2025 survey found 88% of organisations using 1 Citigrounp, Inc, London, United Kingdom 2 Ernst & Young LLP, London, United Kingdom 3 NVIDIA Corporation, London, United Kingdom. Correspondence to: Abbas Raza Ali <[email protected]>.
Proceedings of the 10 th MLSys Conference, Bellevue, WA, USA, 2027. Copyright 2027 by the author(s).
AI in at least one function but only 39% attributing any EBIT impact to it (Singla et al., 2025). Third, agentic systems are arriving before the evaluation discipline needed to operate them: Gartner projects that over 40% of agentic AI projects will be cancelled by 2027 for unclear value or inadequate risk controls (Gartner, 2025), and a survey of 306 practitioners running agents in production finds reliability to be the dominant barrier, with most restricting agents to a few steps before human intervention (Pan et al., 2025). We contend that much of the distance between the first fact and the second is a measurement gap with two parts (Figure 1). Capability benchmarks are built to be portable across organisations, and that portability is exactly what prevents them from answering the questions that decide a deployment: whether a credit analyst may rely on a generated repayment analysis depends on the firm’s documents, credit policy, citation conventions and reviewer capacity. Conversely, the metrics that internal teams do compute rarely reach a governance committee in a decision-grade form: thresholds are set after the fact, sample sizes go unreported, LLM-graded scores are presented without calibration, and the link from metric to money is asserted rather than modelled.
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
Throughout we label claims as [Established] (cited prior work), [Documented] (our pilot), [Proposed] (our design) or [Hypothesis].
2
BACKGROUND AND R ELATED W ORK
[Established]
Figure 1. Two gaps between public benchmarks and enterprise decisions. Capability benchmarks measure what a model can do on portable tasks. Enterprises must also establish transfer to their own workflows and ground truth (Gap 1) and convert quality metrics into a value-and-risk case a governance body can act on (Gap 2).
EnterpriseVal (eVal) treats evaluation as a systems problem: a repeatable pipeline that takes a frozen workflow configuration and a firm-specific task set and emits an auditable gate decision, an efficacy index and a value estimate with stated uncertainty. It sits between model benchmarks such as GDPval (Patwardhan et al., 2025), HELM (Liang et al., 2023) and the agent suites τ -bench (Yao et al., 2025; Barres et al., 2025), SWE-bench (Jimenez et al., 2024) and HAL (Kapoor et al., 2025a), and the model-risk-management regimes of regulated industries (Bank of England Prudential Regulation Authority, 2023). It borrows GDPval’s blinded expert comparison, the reliability and cost metrics of the agentevaluation literature (Yao et al., 2025; Kapoor et al., 2025b; Rabanser et al., 2026), and the statistical hygiene of survey methodology, and adds decision rules and a value model that a risk committee can act on. Contributions. 1. A formal evaluation object, use case U and frozen configuration σ including human oversight, with an autonomy×consequence evaluation-intensity matrix (§3). 2. A six-family metric catalogue with operational definitions, extending pilot fidelity metrics with reliability (passˆk), assurance and oversight metrics (§4). 3. A grading and statistical protocol: blinded experts with agreement targets, calibrated LLM-as-judge with prediction-powered correction, sample-size rules, configuration freeze and harness parity (§5). 4. A two-tier gate and Efficacy Index, given as algorithms, and a value-and-risk model with the reviewer catch rate as an explicit parameter (§6). 5. Documented pilot evidence from three workflows in a global bank, reported with limitations, and the experiments needed to complete validation (§7–§9).
From capability to economically valuable tasks. Knowledge tests such as MMLU (Hendrycks et al., 2021) saturated quickly and suffer from contamination and weak construct validity (Eriksson et al., 2025; Raji et al., 2021). HELM broadened coverage but stayed model-centric (Liang et al., 2023). GDPval shifted to expert-authored, expert-graded professional deliverables and reported that reasoning effort and task context each yield measurable gains and that models complete tasks roughly 100× faster and cheaper than experts (Patwardhan et al., 2025). GDPval is organised by occupation and designed to be portable; eVal is organised by use case and designed to be firm-specific. They are complementary: GDPval shortlists model families, eVal decides deployments. Agent evaluation and the reliability gap. τ -bench introduced passˆk, success on all k attempts, and showed agents that succeed 60% of the time once may succeed only 25% of the time on all of eight tries (Yao et al., 2025). Across 24 months of frontier releases, reliability improved far less than accuracy and plateaued similarly across vendors (Rabanser et al., 2026). Accuracy without cost yields needlessly expensive agents (Kapoor et al., 2025b); HAL standardises cost logging (Kapoor et al., 2025a), and the CLEAR framework adds latency, assurance and reliability, finding that expert judgement of production success tracks a multidimensional score far better than accuracy (Mehta, 2025). KAMI’s 5.5billion-token study found public leaderboard rank a poor predictor of performance on routine enterprise tasks, with small prompt and tool-message changes moving accuracy and cost by large factors (Roig, 2025); log-level analysis is increasingly regarded as necessary (Kirgis et al., 2026). Enterprise sandboxes (Vishwakarma et al., 2025; Wang et al., 2025) remain generic across firms. A systematic review of 84 agentic-AI evaluations finds capability measured extensively while workflow integration, trust and operation over time are rarely measured (Meimandi et al., 2025), and predictive validity, whether in-sample rankings transfer, has been proposed as the standard for agent leaderboards (Patel et al., 2026). Grounded generation. Enterprise drafting is retrievalaugmented generation (Lewis et al., 2020), and its principal failure is unfaithful content (Huang et al., 2025; Ji et al., 2023). FActScore checks atomic facts against a source (Min et al., 2023); RAGAS provides reference-free faithfulness
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise full, G #
Table 1. Positioning against representative frameworks ( partial, # absent). Dimension
Firm-specific ground truth Blinded expert grading Calibrated LLM judge Reliability (passˆk) Cost / efficiency Assurance / oversight Thresholds with CIs Value-and-risk model Governance mapping
GDPval τ /HAL CLEAR KAMI RAGAS EV #
G # #
# # # #
# # #
# G # # G
# # #
G # # # #
# G # #
# # G # #
G # #
# # # # # #
and relevance (Es et al., 2024). LLM-as-judge methods correlate well with human preference (Zheng et al., 2023; Liu et al., 2023) but exhibit position, verbosity and selfpreference biases (Gu et al., 2024), and aligning judges with human criteria is itself iterative (Shankar et al., 2024). Productivity and oversight. Field studies report 14% more issues resolved per hour in customer support (Brynjolfsson et al., 2025), 40% faster professional writing (Noy & Zhang, 2023), and 12% more tasks for consultants alongside lower accuracy outside model competence - the “jagged frontier” (Dell’Acqua et al., 2023). Time saved must therefore be measured net of review, and review performance itself must be measured. Governance. In banking, quantitative methods fall under model risk management (SR 11-7, SS1/23) (Bank of England Prudential Regulation Authority, 2023); the NIST AI RMF and its GenAI Profile call for measurement, thresholds and documentation (National Institute of Standards and Technology, 2023; 2024); the EU AI Act imposes riskmanagement, data-governance, human-oversight, accuracy and logging duties on high-risk systems (European Parliament and Council of the European Union, 2024), with standalone high-risk obligations deferred to December 2027 by the 2026 Digital Omnibus (European Parliament and Council of the European Union, 2026); ISO/IEC 42001 and 42005 provide management-system and impact-assessment standards (International Organization for Standardization, 2023; 2025). No prior framework combines firm-specific ground truth, decision thresholds with uncertainty, a value model and a governance mapping (Table 1).
grader; thresholds are decisions set before results are seen; and value is modelled, not asserted. A sixth is a systems requirement: the configuration is frozen and human oversight is measured - comparisons are valid only under harness parity, and “human in the loop” is a component whose detection performance is measured, not an assumption that converts an imperfect system into a safe one. 3.2
A use case is a tuple U = ⟨T , S, G, R, a, c⟩,
T HE E NTERPRISE VAL F RAMEWORK
[Proposed]
3.1
Design principles
Five principles, each traceable to a failure in §2: evaluate the workflow, not the model; ground truth is expert-authored and firm-specific; every score carries its uncertainty and its
ti = (xi , Si , gi ) ∈ T ,
(1)
where T is a set of n tasks sampled from the workflow population, each an instruction xi , permitted sources Si and an expert gold artefact gi ; S the corpus and access policy; G the gold-set specification including the key elements K(gi ) a competent artefact must contain; R the rubric; a ∈ {A0, . . . , A4} the autonomy level and c ∈ {C1, . . . , C4} the consequence tier (§3.4). The candidate system is itself a tuple, frozen for the run: σ = ⟨M, P, Φ, U, Γ, H⟩,
(yi , τi ) = σ(xi , Si ),
(2)
with M the model(s) and sampling settings, P prompts and policies, Φ retrieval, memory and context management, U tools and permissions, Γ guardrails, verification and logging, and H human roles, checkpoints and escalation rules; τi is the trajectory of tool calls and intermediate states. Making H part of σ is deliberate: two deployments of one model with different review designs are different systems. Any change to σ is a re-evaluation trigger. The evaluation returns a metric vector m(σ, U) with confidence intervals and a gate decision. 3.3
Architecture
Figure 2 shows four layers and a governance spine. Layer 1 constructs U; Layer 2 computes metrics (§4); Layer 3 produces graded judgements with uncertainty (§5); Layer 4 applies thresholds and the value model (§6). The spine maps each artefact to MRM, NIST, EU AI Act and ISO 42001 requirements so the evaluation doubles as the model-risk evidence file (Appendix C (supplementary)). 3.4
3
Formal setting
Autonomy, consequence and evaluation intensity
Autonomy describes what a system does; consequence describes what is at stake when it is wrong. We define five autonomy levels, adapted from (Morris et al., 2024): A0 assistive Q&A; A1 grounded drafting with citations and human edits; A2 tool-using workflow (retrieval, extraction, rule checks); A3 multi-step agent with human checkpoints; A4 bounded autonomy acting on systems with post-hoc review. Each level inherits all lower-level metrics and adds
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
4
M ETRIC L AYER
[Proposed; individual metrics instantiate established constructs (Min et al., 2023; Es et al., 2024; Yao et al., 2025)]
Metrics are grouped into six families answering distinct decision questions - can I trust it (fidelity), is it useful (utility), does it save effort (efficiency), will it keep working (reliability), does it stay within the rules (assurance) and does the human catch what it misses (oversight). Units are those a reviewer inspects: sentences, cited chunks, key elements, tasks, trajectories (Table 2).
Figure 2. eVal architecture. Four evaluation layers feed a governance decision; the governance spine consumes artefacts from every layer.
Fidelity. For output y segmented into sentences, the hallucination rate is the share of assertive sentences unsupported by the permitted sources, P 1 HR(y) = |y| s∈y 1[ assert(s) ∧ ¬ supp(s | S) ], (4) a faithfulness rather than world-factuality definition (Huang et al., 2025): a true statement from a non-permitted source counts, because the workflow requires traceability. The sentence is the operational unit; the reference definition is at the level of atomic claims (Min et al., 2023), and the two are reconciled on a calibration subset because one sentence may carry several claims. Citation precision is the share of cited chunks actually used by the citing paragraph, CP = |{c : used(c, para(c))}|/|C(y)|; the pilot used a lenient criterion (any part reflected), and we recommend also reporting a strict variant and citation recall.
Figure 3. Evaluation-intensity matrix (Eq. 3). Markers place the pilot use cases: B credit memos (A1,C3), A control assessment (A2,C3), C procedure transformation (A1,C2).
families (Table 2). Orthogonally, consequence tiers run from C1 (internal, reversible) through C2 (informs judgement) and C3 (alters records or decisions) to C4 (regulatory, customer or financial harm), following MRM materiality logic (Bank of England Prudential Regulation Authority, 2023) and ISO/IEC 42005 (International Organization for Standardization, 2025). The two set the evaluation intensity
Utility. Key-element capture KEC = |K(g) ∩ K(y)|/|K(g)| is a recall measure against expert expectation; P 1 irrelevance rate IR = |y| / rel(x)] is a precisions 1[s ∈ like measure; reporting both guards against a system that captures everything by writing everything. Efficiency. Net of review: ∆eff = 1 − t̄AI /t̄manual with t̄AI = t̄gen + t̄review . Estimated baselines (practitioner recall) are adequate for prioritisation; observed baselines from timed matched tasks (Noy & Zhang, 2023; Dell’Acqua et al., 2023) are required for a value claim. Cost per successful task is recorded alongside (Kapoor et al., 2025b). Reliability.
E(a, c) =
a+c 2
∈ {1, . . . , 4},
With K attempts per task and ci successes,
(3)
with a ∈ {0..4}, c ∈ {1..4} (Figure 3). E1 requires fidelity grading on ≥100 units; E2 adds inter-rater agreement, CIs with gate decisions on conservative bounds and K ≥ 2 repeats; E3 adds passˆk with K ≥ 4, a cost-per-successfultask gate and trajectory logging; E4 adds an adversarial suite, independent validation and production monitoring. A low-autonomy, high-consequence drafting assistant is thus not under-evaluated because it “only drafts”.
\k = 1 Pn ci K , pass i=1 k n k
(5)
the unbiased estimator of Pr[all k attempts succeed] (Yao et al., 2025). We report passˆ1, 2 and 4; the decay is the reliability signature that has stagnated across frontier models (Rabanser et al., 2026). Assurance. Precision and recall of a guidance verification step against mandatory policy items (recall weighted
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise Table 2. Metric catalogue. Grading: H human, J calibrated judge, A automatic. Level: lowest autonomy level at which the metric is gated. Family
Grading
Level
Hallucination rate (Eq. 4) ↓ H,J Citation precision (lenient/strict) ↑ H,J Citation recall ↑ J Utility Key-element capture ↑ H,J Irrelevance rate ↓ H,J Completeness (scope attempted) ↑ A Efficiency Net time reduction ∆eff ↑ A/obs. Cost per successful task ↓ A Reliability passˆk (Eq. 5); dispersion ↑/↓ A,H Recovery rate after tool failure ↑ A Assurance Guidance precision / recall ↑ H Policy-violation rate ↓ A,H Escalation appropriateness ↑ H Adversarial incident count ↓ A,H Oversight Reviewer catch rate ρcatch ↑ H (study) False acceptance rate ↓ H Unplanned intervention rate ↓ A Review burden (min/task) ↓ A
A1 A1 A1 A1 A1 A2 A1 A1 A2 A3 A1 A3 A3 A3 A1 A1 A3 A1
Fidelity
Metric
Dir.
N outputs; humans h score a random subset of n ≪ N ; the reported estimate is the prediction-powered estimator (Angelopoulos et al., 2023) PN Pn θ̂PPI = N1 i=1 f (yi ) − n1 i=1 f (yi ) − h(yi ) , (6) unbiased for the human-graded quantity whatever the judge’s accuracy, with variance shrinking as the judge improves. A use case can thus evaluate hundreds of outputs while paying for tens of expert gradings without laundering judge bias into the gate. Judge prompts, model versions and calibration sets are versioned in R; trajectory scoring follows Agent-as-a-Judge (Zhuge et al., 2025) and is confined to what logs can support (Kirgis et al., 2026). Sample size and intervals. To resolve the gap δ between inner and outer thresholds for a proportion near p̂, 2 n ≥ z1−α/2 p̂(1 − p̂)/δ 2 ;
more, since a missed requirement costs more than a false alarm); for A3+ systems, policy-violation rate, escalation appropriateness and incidents under an adversarial suite of injected instructions in tool outputs (Debenedetti et al., 2024), following the OWASP agentic taxonomy (OWASP GenAI Security Project, 2025); every tool identity, permission and response is logged so the attack surface tested is the one deployed (Model Context Protocol, 2025). Oversight. Because H ∈ σ, oversight performance is measured: the reviewer catch rate ρcatch (seeded defects detected / seeded), the false acceptance rate (defective outputs accepted / defective), the unplanned intervention rate (runs needing intervention outside checkpoints / runs) and review burden (reviewer minutes per completed task).
5
(7)
for the credit-memo gap of 5 points and p̂ ≈ 0.85, n ≈ 196 units per metric. Units cluster within documents, so clusterrobust or bootstrap intervals over documents are reported; proportions near 0 or 1 use Wilson or Agresti–Coull intervals (Brown et al., 2001). Gates are taken on the conservative bound. Tasks are run K ≥ 4 times at deployment settings for A2+ (≥ 2 for A1); paired comparisons use McNemar or Wilcoxon with Holm correction. Freeze, parity and stress sets. σ is recorded and held fixed; tuning uses a development split disjoint from the graded set. When two models are compared, everything except M is identical. A small stress set probes missing or conflicting sources, “not determinable” questions, tool errors and, for A3+, injected instructions; its results are reported separately and gated for assurance only. Algorithm 1 states the procedure.
G RADING AND S TATISTICAL P ROTOCOL
[Proposed protocol composed of established methods (Patwardhan et al., 2025; Zheng et al., 2023; Cohen, 1960; Angelopoulos et al., 2023; Brown et al., 2001)]
Blinded expert grading. Graders are practitioners from the workflow, see model and human artefacts in randomised order without provenance, and score each unit against R; each task has ≥ 2 graders and a third adjudicates. Agreement is Cohen’s κ = (po − pe )/(1 − pe ) (Cohen, 1960) or Krippendorff’s α (Krippendorff, 2004); a metric enters a gate decision only if κ ≥ 0.6, otherwise the rubric, not the system, is revised. A system’s self-assessment is never a gated metric: it violates blinding and inherits judge selfpreference (Gu et al., 2024). Calibrated judge with prediction-powered correction. Expert time is the binding constraint. A judge f scores all
6
D ECISION L AYER
[Proposed]
Two-tier gating. Each gated metric carries two thresholds agreed before results are unblinded by the business owner, model owner and second-line risk function. The outer threshold θout is the minimum viable level, anchored to the harm of failure and to current compensating controls; the inner threshold θin is the target at which the use case may scale without additional controls. Between them lies a conditional zone: deploy with enhanced review, sampling or scope restriction and a dated commitment to reach θin (Figure 4). Metrics are normalised relative to their thresholds, ! σj (mj − θjout ) νj = clip , −1, νmax , (8) σj (θjin − θjout )
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
Algorithm 1 eVal evaluation of a use case Require: U = ⟨T , S, G, R, a, c⟩; candidates Σ = {σk }; charter (θout , θin , w, νmax ); E = E(a, c) Ensure: per-system gate, EI, value estimate, evidence file 1: Freeze T , G, R and each σk ; record versions 2: n ← Eq. 7; draw T stratified; build stress set Ts 3: for each σk do 4: for ti ∈ T ∪ Ts , r = 1..K(E) do 5: (yir , τir ) ← σk (xi , Si ); log τir , cost, latency 6: end for 7: Judge f scores all units; ≥ 2 blinded humans grade a random subset of size nh 8: Compute κ; if κj < 0.6 then revise Rj , regrade 9: ∀j: mj ← PPI (Eq. 6) or human mean if nh ≥ n; CIj ← Wilson / cluster bootstrap 10: passˆk (Eq. 5), dispersion; assurance metrics on Ts 11: (Gatek , EIk , bk ) ← Algorithm 2 12: Vk ← Eq. 10 with N, ch , ℓerr from charter, ρcatch from oversight study 13: end for 14: Emit scorecards; rank passing systems by EI with sensitivity band; write evidence file; register re-evaluation triggers
Algorithm 2 Two-tier gate and Efficacy Index out in Require: mj , CI− j , σj , θj , θj , wj , νmax Ensure: Gate, EI, EIG , binding metric b, flags 1: Gate ← S CALE 2: for j = 1..J do 3: νj ← Eq. 8 out 4: if CI− then j fails θj 5: Gate ← R EJECT in 6: else if CI− j fails θj and Gate ̸= R EJECT then 7: Gate ← C ONDITIONAL 8: end if 9: if CIj straddles both thresholds then 10: flag j under-sampled (Eq. 7) 11: end if 12: end for P EIG ← 13: b ←P arg minj νj ; EI ← j wj νj ; exp( j wj ln(1 + νj )) − 1 14: Recompute EI under each alternative w′ in the charter ⇒ sensitivity band 15: return Gate, EI, EIG , b, flags
file, always reported with a weight-sensitivity band and never as a substitute for the gate. P A variant penalising weak dimensions, EIG = exp( j wj ln(1 + νj )) − 1 for νj > −1, equals zero when every metric sits at θout and is pulled toward the weakest metric; divergence between EI and EIG is a cheap diagnostic of an unbalanced profile. Algorithm 2 gives the computation.
Figure 4. Two-tier gating. Decisions use the conservative CI bound; an interval straddling both thresholds calls for more samples, not a verdict.
Value-and-risk model. For a task performed N times a year at loaded hourly cost ch , with inference cost cinfer per task and expected loss ℓerr if a residual error reaches a decision, Vannual = N ch ∆t̄ + Vquality + Vavoided − N cinfer − Cintegr
with σj = ±1 for higher/lower-is-better, so νj = 0 at θout , 1 at θin , negative on failure and capped at νmax (3 in the pilot) so one exceptional metric cannot mask a weak one. With CI− j the conservative bound,
− N ch t̄review − Crework − N πerr ℓerr , πerr ≈ HR · (1 − ρcatch ),
(10) (11)
where ∆t̄ = t̄manual − t̄review , Vquality and Vavoided capture outcome improvements and losses avoided, and Cintegr , in ∀j : CI− j ⪰ θj Crework the amortised integration and redo costs; each term − out in ∀j : CI− j ⪰ θj , ∃j : CIj ≺isθjentered as a range and Vannual reported as an interval. otherwise. Review appears as a cost in Eq. 10 and as the control on (9) expected loss through ρcatch in Eq. 11, so a proposal to “save The rule is conjunctive by design: compensation across time” by shortening review is automatically charged with metrics is appropriate for ranking models, not for deciding the risk it creates. Estimating ρcatch , how often a reviewer whether a bank may rely on a system. reading a fluent draft detects an unsupported number, is the least-studied quantity in the literature; the jagged-frontier Efficacy Index. For ranking candidates or versions, EI = result (Dell’Acqua et al., 2023) suggests it may be well P w ν with charter-fixed weights encoding the risk probelow one and fall as trust rises (§9). j j j S CALE Gate(U) = C ONDITIONAL R EJECT
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise Table 3. Pilot portfolio. A: Control assess- B: Credit-memo C: Procedure ment drafting transformation Level / tier / E Inputs
Output
Models Gated metrics Status
7
A2 / C3 / E3 A1 / C3 / E2 A1 batch / C2 / E2 Process–risk– Filings, broker Existing procecontrol matrices; research, internal dures (30k corpus, control standards spreads 40k users) Per-control compli- One of 13 memo Standardised ance class + recom- sections, cited rewrite mendation Gemini 2.5 Flash / Gemini 2.5, Pipeline v1, v2 Pro Llama 4 KEC, IR, HR, guid- CP, KEC, IR, HR, ∆eff ance P/R, passˆ4 guidance P/R Charter set; grading Human-graded re- Estimated-baseline in progress sults results
Table 4. Use Case B human-graded results. ↓ lower is better. Gate column applies Eq. 9 to point estimates for illustration; CIs to be added. Metric
θ out
Citation precision (lenient) Key-element capture Overall accuracy (composite) Irrelevance rate ↓ Hallucination rate ↓ Guidance precision Guidance recall
>65 >70 >50 >55 – – <50 <55† <10 <5 >75 >80 >75 >80
†
θ in
Gemini 2.5 Llama 4 Gate 88.0 99.0 93.5 23.0 1.6 98.0 80.0
76.0 96.0 86.0 28.0 3.2 93.0 95.0
both S CALE both S CALE not gated both S CALE both S CALE both S CALE G C ONDITIONAL; L S CALE
Recorded thresholds are inverted (inner should be stricter); retained for fidelity to the pilot and flagged for the next charter.
P ILOT IN A G LOBAL BANK
[Documented; sample sizes, CIs and agreement were not recorded in the pilot and are flagged]
The framework was piloted by the innovation function of a global systemically important bank across three workflows spanning finance controls, wholesale credit and operations knowledge management (Table 3). Institution, internal platform names and identifiers are anonymised; public model names are retained. Appendix A (supplementary) gives workflow, prompt and rubric detail. 7.1
Use Case A: control design effectiveness (A2)
Issue quality assurance reviews remediation of high-severity control issues; the assessed step classifies each control in a process–risk–control matrix as compliant, partially compliant or non-compliant against a control standard and rule library. The A2 workflow parses control attributes (who, what, when, where, how), retrieves the standard, emits a percontrol judgement table and a structured export. Thresholds and the gold set were fixed before unblinding; results are pending and none are claimed. The charter stage’s most useful output was discovering that the original design asked the model to score its own hallucination and coherence; under eVal self-assessment is informational only. 7.2
Use Case B: credit-memo drafting (A1)
Analysts upload filings, research and spreads; the platform drafts one of 13 memo sections with inline citations; the analyst edits; a guidance-verification step checks the draft against credit-policy items. Table 4 and Figure 5 report human-graded results against pre-agreed thresholds. Both models clear every outer threshold. Gemini 2.5 has higher fidelity (88% CP, 1.6% HR vs. 76%, 3.2%), capture and guidance precision; Llama 4 has higher guidance recall (95% vs. 80%).
Figure 5. Use Case B against two-tier thresholds. Dashed/dotted lines mark outer/inner thresholds.
Interpretation. Applied mechanically to point estimates, Eq. 9 places Llama 4 in S CALE on all metrics and Gemini 2.5 in C ONDITIONAL because its recall of 80% does not strictly exceed 80%. This is exactly what the framework should expose rather than resolve by fiat: without an interval the distinction is meaningless and the correct action is to grade more guidance items (Figure 4). The normalised view (Figure 6) makes a second point: with equal weights and νmax = 3, EI is 2.45 for Gemini and 2.59 for Llama; with fidelity-weighted weights (0.30 HR, 0.25 CP, 0.15 each remaining) the order reverses to 2.62 vs. 2.37. Which system to prefer depends on whether the credit function values fewer unsupported statements or fewer missed guidance items, a judgement the charter fixes in advance and the index makes explicit rather than hiding in an average. The pilot did not document harness parity between the two models, so differences cannot be attributed to the models alone, and no timed efficiency comparison was run, so no efficiency result is claimed here. 7.3
Use Case C: procedure transformation (A1, batch)
Ten frequently used procedures were rewritten by two pipeline versions; owners estimated the effort to refine each output to publishable standard and the effort to transform it manually (Figure 7). Mean estimated manual effort was 27.4 h/document; refinement after v1 was 10.6 h (−61%)
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
the evidence to the framework’s own E2/E3 standard we will: report n, cluster-robust 95% CIs and κ for every metric and re-issue gates on conservative bounds; run K ≥ 4 repeats and report passˆ{1, 2, 4}; complete Use Case A; replace estimated with observed efficiency in a within-subject timed design (Noy & Zhang, 2023; Dell’Acqua et al., 2023) with four baselines (unaided, assisted, model-only, agentic with matched tooling); add KEC grading to Use Case C; calibrate a judge and report PPI estimates; re-run B under documented parity with a stress set; and populate Eq. 10 with ρcatch elicited by seeding defects into review samples.
8
Figure 6. Threshold-normalised scores νj (Use Case B). Origin = θout , unit circle = θin , cap 3. Equal-weight and fidelityweighted EI rank the systems differently.
Figure 7. Use Case C estimated effort by document and pipeline version.
and after v2 2.9 h over the seven documents with v2 outputs (−89% against the ten-document manual mean; −88% like-for-like, since the v2 subset’s manual mean is 25.0 h). The v1→v2 improvement is consistent within documents, the more credible signal. Three caveats: the baseline is estimated and clusters at three values (17, 25, 33 h), indicating coarse banding subject to recall bias; the v2 mean is on a subset; and fidelity was not graded, so a faster procedure that silently drops a regulatory step would carry negative value under Eq. 10. 7.4
Cross-case lessons and required additions
Charters force clarity: writing thresholds first exposed an inverted pair, a self-grading step and an unstated baseline. Fidelity and utility trade off across models, and the trade-off is a business judgement to be fixed in weights before results. Efficiency claims are only as good as the baseline. To bring
D ISCUSSION
What the pilot shows and does not show. [Documented] The framework is operable: three business functions in a large regulated institution defined tasks, gold artefacts, metrics and thresholds and, in two cases, produced results a governance forum could read. Frontier models wrapped in retrieval and citation scaffolding reached 1.6–3.2% hallucination and 96–99% key-element capture on genuine wholesale-credit documents under human grading, consistent with GDPval (Patwardhan et al., 2025); and standardisation effort fell by an order of magnitude. It does not yet show statistically qualified gates, repeated-run reliability, observed efficiency or realised value. Extending to agentic systems. [Proposed] As workflows are promoted to A3, a credit agent that gathers filings, computes ratios, drafts sections and reconciles them, four things change. The unit of grading becomes the trajectory as well as the artefact: plan validity, retrieval and tool-call correctness, memory reads and writes, checkpoints and final state are logged and graded (Appendix B (supplementary)), because an agent that reaches a correct memo by an incorrect path is a production liability (Kirgis et al., 2026). Reliability becomes gated, and the vendor-wide plateau (Rabanser et al., 2026) implies the outer threshold on passˆ4 will often bind. Cost is gated per successful task (Kapoor et al., 2025a;b), since reasoning modes can multiply token cost by an order of magnitude for modest gains (Roig, 2025). Assurance expands to policy violations, escalation and injection incidents (National Institute of Standards and Technology, 2024; Debenedetti et al., 2024). Multi-agent pipelines (Guo et al., 2024) additionally require each agent to be evaluated as its own use case, with end-to-end passˆk compared against the product of component values to detect interaction effects. Threats to validity. [Proposed analysis] Construct: sentence-level HR and lenient CP bias in known directions; key-element lists encode one expert view. Internal: graders were not always blind; no agreement statistics; harness parity undocumented; estimated baselines. External: one institution, one model generation, small task sets - results
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
are about the workflows, not the models, which is why a model change is a re-evaluation trigger. Statistical: without n and CIs, Table 4’s gates illustrate the rule rather than establish findings.
9
L IMITATIONS AND H YPOTHESES
The framework presumes a gold artefact can be authored, which holds for generation and classification workflows and less so for open-ended advisory tasks. Thresholds and weights are firm-specific by design, so EI is not comparable across firms; sector reference thresholds are a desirable outcome the framework enables but cannot supply. The conjunctive gate is conservative and its rejection rate grows with the number of gated metrics, a multiplicity effect the charter manages by gating only metrics mapped to a material harm. The agentic extension is specified but not yet exercised. [Hypothesis] We state testable conjectures. H1 ρcatch for flu-
ent, cited drafts is below 0.7 and declines as measured HR falls (seeded-defect blinded review study). H2 Thresholds agreed independently by peer institutions fall in a narrow band, enabling sector references (multi-institution charter on a synthetic task set). H3 A calibrated judge reaches κ ≥ 0.6 on HR and IR but not on KEC or value-add. H4 For A3 agents on document workflows, passˆ4 at θout binds more often than any fidelity metric. H5 Use cases passing the inner gate show measurable capacity or P&L effects within two quarters at a rate well above the 5% reported for pilots generally (Challapally et al., 2025) (prospective registration; the predictive-validity standard of (Patel et al., 2026)). H6 Conjunctive gating produces fewer approvals later reversed than any weighted average of the same metrics, at the cost of more C ONDITIONAL outcomes (retrospective portfolio classification). H7 passˆk declines with human-equivalent task duration (Kwa et al., 2025) unless checkpoint density scales with it. Ablations over the components of σ (retrieval, scaffold, tools, model, autonomy, review design) are a further priority, given KAMI’s finding that tool-message changes alone move accuracy and cost by large factors (Roig, 2025). We plan an open reference implementation of Algorithms 1– 2 with a schema for Eqs. 1–2 and a synthetic public task set.
10
C ONCLUSION
Enterprise GenAI has an evaluation problem before it has a capability problem. Public benchmarks show what frontier models can do; they cannot tell a bank whether a specific workflow is fit, reliable, safe and worth scaling on its own documents and under its own controls, and the internal evaluations that try rarely meet the standard a risk committee or a reviewer requires. eVal is an evaluation system for clos-
ing that gap: a formal, frozen evaluation object; a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; a grading protocol that scales expert judgement with calibrated automation without laundering its bias; a conjunctive two-tier gate on conservative bounds; and a value model that puts review effort and residual risk on the same ledger as time saved. A pilot in a global bank shows the system to be operable and yields encouraging fidelity and efficiency results, while making plain the evidence still required. We offer the framework, pilot and hypotheses in the expectation that enterprise AI evaluation will mature as model benchmarking has - through shared definitions, pre-registered thresholds and honest reporting of uncertainty.
R EFERENCES Angelopoulos, A. N., Bates, S., Fannjiang, C., and et al. Prediction-powered inference. Science, 382(6671):669– 674, 2023. Bank of England Prudential Regulation Authority. Model risk management principles for banks. Technical report, Supervisory Statement SS1/23, 2023. Barres, V., Dong, H., Ray, S., and et al. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. Brown, L. D., Cai, T. T., and DasGupta, A. Interval estimation for a binomial proportion. Statistical Science, 16(2): 101–133, 2001. Brynjolfsson, E., Li, D., and Raymond, L. Generative AI at work. The Quarterly Journal of Economics, 140(2): 889–942, 2025. Challapally, A., Pease, C., Raskar, R., and et al. The GenAI divide: State of AI in business 2025. Technical report, MIT NANDA, July 2025. Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37– 46, 1960. Debenedetti, E., Zhang, J., and et al. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 37, 2024. Dell’Acqua, F., III, E. M., Mollick, E. R., and et al. Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Technical Report Working Paper 24-013, Harvard Business School, 2023.
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
Eriksson, M., Purificato, E., Noroozian, A., and et al. Can we trust AI benchmarks? an interdisciplinary review of current issues in AI evaluation. arXiv preprint arXiv:2502.06559, 2025. A version also appears in Proceedings of AAAI/ACM AIES 2025. Es, S., James, J., Espinosa-Anke, L., and et al. RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL): System Demonstrations, 2024. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (artificial intelligence act). Official Journal of the European Union, L 2024/1689, 12 July 2024, 2024. European Parliament and Council of the European Union. Regulation (EU) 2026/1744 of 8 july 2026 amending regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (digital omnibus on AI). Official Journal of the European Union, L 2026/1744, 24 July 2026, 2026. Gartner. Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027. Press release, 25 June 2025, 2025. Gu, J., Jiang, X., Shi, Z., and et al. A survey on LLM-as-ajudge. arXiv preprint arXiv:2411.15594, 2024. Guo, T., Chen, X., Wang, Y., and et al. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI), 2024. Hendrycks, D., Burns, C., Basart, S., and et al. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. Huang, L., Yu, W., Ma, W., and et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 2025. International Organization for Standardization. ISO/IEC 42001:2023 information technology — artificial intelligence — management system, 2023. International Organization for Standardization. ISO/IEC 42005:2025 information technology — artificial intelligence — AI system impact assessment, 2025. Ji, Z., Lee, N., Frieske, R., and et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 2023.
Jimenez, C. E., Yang, J., Wettig, A., and et al. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. Kapoor, S., Stroebl, B., Kirgis, P., and et al. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. arXiv preprint arXiv:2510.11977, 2025a. A version also appears at ICLR 2026. Kapoor, S., Stroebl, B., Siegel, Z. S., and et al. AI agents that matter. Transactions on Machine Learning Research, 2025b. Preprint arXiv:2407.01502, 2024. Kirgis, P., Kapoor, S., Rabanser, S., and et al. Log analysis is necessary for credible evaluation of AI agents. arXiv preprint arXiv:2605.08545, 2026. Krippendorff, K. Content Analysis: An Introduction to Its Methodology. Sage, Thousand Oaks, CA, 2nd edition, 2004. Kwa, T., West, B., Becker, J., and et al. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Model Evaluation & Threat Research (METR). Preprint arXiv:2503.14499. Lewis, P., Perez, E., Piktus, A., and et al. Retrievalaugmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, 2020. Liang, P., Bommasani, R., Lee, T., and et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. Liu, Y., Iter, D., Xu, Y., and et al. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. Mehta, S. Beyond accuracy: A multi-dimensional framework for evaluating enterprise agentic AI systems. arXiv preprint arXiv:2511.14136, 2025. Meimandi, K. J., Aránguiz-Dias, G., Kim, G. R., and et al. The measurement imbalance in agentic AI evaluation undermines industry productivity claims. arXiv preprint arXiv:2506.02064, 2025. Min, S., Krishna, K., Lyu, X., and et al. FActScore: Finegrained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
EnterpriseVal – Supplementary Appendix
Model Context Protocol. Model context protocol specification, revision 2025-06-18. https:// modelcontextprotocol.io, 2025. Morris, M. R., Sohl-Dickstein, J., Fiedel, N., and et al. Levels of AGI for operationalizing progress on the path to AGI. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. National Institute of Standards and Technology. Artificial intelligence risk management framework (AI RMF 1.0). Technical report, NIST AI 100-1, 2023.
Singla, A., Sukharevsky, A., Yee, L., and et al. The state of AI in 2025: Agents, innovation, and transformation. Technical report, QuantumBlack, AI by McKinsey, November 2025. McKinsey Global Survey on the state of AI. Vishwakarma, H., Agarwal, A., Patil, O., and et al. Can LLMs help you at work? a sandbox for evaluating LLM agents in enterprise environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Suzhou, China, 2025. Association for Computational Linguistics. Preprint arXiv:2510.27287.
National Institute of Standards and Technology. Artificial intelligence risk management framework: Generative artificial intelligence profile. Technical report, NIST AI 600-1, 2024.
Wang, L., Yi, D., Jose, D., and et al. Enterprise large language model evaluation benchmark. arXiv preprint arXiv:2506.20274, 2025.
Noy, S. and Zhang, W. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187–192, 2023.
Yao, S., Shinn, N., Razavi, P., and et al. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), 2025.
OWASP GenAI Security Project. OWASP top 10 for agentic applications. Released 9 December 2025 (2026 edition), 2025. Pan, M. Z., Arabzadeh, N., Cogo, R., and et al. Measuring agents in production. arXiv preprint arXiv:2512.04123, 2025. Patel, D. C., Maghraoui, K. E., Lin, S., and et al. Beyond static leaderboards: Predictive validity for the evaluation of LLM agents. arXiv preprint arXiv:2606.19704, 2026. Patwardhan, T., Dias, R., Proehl, E., and et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks. arXiv preprint arXiv:2510.04374, 2025. OpenAI. A version also appears at ICLR 2026. Rabanser, S., Kapoor, S., Kirgis, P., and et al. Towards a science of AI agent reliability. arXiv preprint arXiv:2602.16666, 2026. Raji, I. D., Bender, E. M., Paullada, A., and et al. AI and the everything in the whole wide world benchmark. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. Roig, J. V. Towards a standard, enterprise-relevant agentic AI benchmark: Lessons from 5.5 billion tokens’ worth of agentic AI evaluations. arXiv preprint arXiv:2511.08042, 2025. Kamiwaza Agentic Merit Index (KAMI) v0.1. Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., and et al. Who validates the validators? aligning LLM-assisted evaluation of LLM outputs with human preferences. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), 2024.
Zheng, L., Chiang, W., Sheng, Y., and et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 36, 2023. Zhuge, M., Zhao, C., Ashley, D. R., and et al. Agent-as-ajudge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 80569–80611. PMLR, 2025. Preprint arXiv:2410.10934, 2024.
S UPPLEMENTARY A PPENDIX
A
P ILOT D ETAIL
A.1
Evaluation lifecycle
Figure A1 shows the seven-stage lifecycle that Algorithm 1 of the main paper formalises; stages 5 (grade) and 6 (gate) are where internal evaluations are typically weakest.
Table A1. What changes when a workflow becomes agentic, and what must therefore be measured. Dimension
Assistive (A0– Agentic (A3–A4) A1)
Task control
User-led
Evaluation implication
Plan validity, stopping rule, recovery Context Evidence coverage, retrieval and tool-call correctness Action Text only State-changing ac- Authorisation, reversibility, tions action-safety gates State Short-lived Memory across State consistency; memory steps poisoning tests Failure mode Wrong content Wrong content Measure separately and wrong action Human role Reviewer / edi- Approver, supervi- Review load, intervention tor sor rate, escalation quality Economic Tokens, latency End-to-end work- Cost per successful task unit flow cost incl. human effort Attack sur- Prompt content Prompt, tools, Injection, tool misuse, privface memory, inter- ilege abuse, exfiltration agent messages
A.3
System plans and decomposes Prompted / up- Dynamic retrieval loaded and tool calls
Use Case A: workflow phases
Figure A1. Evaluation lifecycle with re-evaluation feedback loop. Table A2. Use Case A: phases, AI contribution and applicable metrics. Phase
AI contribution
Metrics
Identification
A.2
Autonomy ladder
Figure A2 expands the five autonomy levels of §3.4 and the metric families each adds; Table A1 lists what changes qualitatively when a workflow becomes agentic.
Extract issue facts, stake- Extraction KEC; omission holders, corrective-action rate; citation precision context Evaluation / Assess reasonableness of re- KEC; HR; recommendation scoping mediation plans; complete- validity ness of controls Remediation de- Classify control design vs. Classification P/R vs. frozen sign standard and rule library; gold; passˆk; completeness recommend fixes Sustainability Assess whether controls are Evidence sufficiency; escalaimplemented and operating tion appropriateness
The system specification includes two guardrails that are part of the evaluation object: no analysis begins before the input file is present and the analyst explicitly instructs the run, and outputs are exported to a structured deliverable rather than free text. Thresholds for KEC, IR, HR, guidance precision/recall and classification passˆ4 were left as “to be determined” at charter until the control owner fixed them before unblinding; grading of Gemini 2.5 Flash and Pro outputs was in progress at submission. A.4 Figure A2. Autonomy levels A0–A4 and the metric families they require.
Use Case B: generalised drafting instruction
The following is the drafting instruction for the source of repayment memo section, generalised for publication. It illustrates why prompts are part of the evaluation object: the
EnterpriseVal – Supplementary Appendix
prohibition in item 4 is a policy rule whose violation is an assurance failure, and the accounting-period requirement in item 1 defines key elements the gold specification must enumerate.
A.6
Grading rubric templates
Table A4. Sentence-level fidelity and relevance rubric (Use Cases A, B). Field
Role. You are a corporate banking analyst preparing the source-of-repayment section of a credit memo for a corporate borrower seeking a revolving credit facility.
Values
Does the sentence make a checkable factual Y / N claim? Headings, questions and explicit recommendations are not assertive. For each claim, can you locate support in the Fully / ParSupported? permitted sources? Support from your own tially / Not knowledge does not count. Does the sentence contribute to the instruc- Y / N Relevant? tion for this section? Citation used? Is the cited chunk’s content actually used Strict / Le(strict) or at least reflected (lenient)? nient / Not Key elements Tick each gold key element present and cor- checklist rect; note incorrect presentations. Value-add Correct, relevant content absent from the hu- 0 / 1 / 2 man baseline.
Objective. Identify and document the primary, secondary and other sources of repayment with specific numerical values and explicit accounting periods.
1. State the accounting periods used (e.g., LTM ended [date], FY[YYYY], three-year average). 2. Align repayment sources with the debt structure, distinguishing short- and long-term tenors. 3. Use consistent currency units and cite source materials explicitly (financial statements, base-case model, appraisals). 4. Never present revolving credit facilities, undrawn lines or short-term borrowings as a source of repayment. 5. Primary source: free cash flow (CFO less capex) relative to scheduled amortisation. 6. Secondary source: refinancing capacity, collateral realisation (asset values, LTV, advance rates) or pending transactions. 7. Other sources: realistic alternatives aligned to debt tenor.
Instruction to grader
Assertive?
Guidance-verification rubric Flag correct? For each flagged guidance item, does the TP / FP draft in fact breach or omit it? Missed item? For each mandatory item not flagged, does FN / correct the draft breach or omit it? Severity Business-assigned severity of the item. Low / Med / High
B
T RAJECTORY-L EVEL E VALUATION R ECORD
Table A5. Trajectory-level record for A3–A4 systems; each layer is logged per run and graded. Layer
A.5
Use Case C: per-document effort
Planning
Table A3. Estimated effort per procedure (hours). n/a: no v2 output assessed. Doc
Title (abbrev.)
101889 102123 129826 130293 130724 130763 131377 131411 131733 131926
Security word rules Do-not-release to any caller Inactive and dormant accounts Verifications and overrides Travel suppression Vulnerable adult incident report Regulation DD Regulation E Consumer fairness policy Branch investment procedures
Mean (n=10) Mean (v2 subset, n=7)
Recorded
Metrics
Input / context Source identities, com- Completeness; pleteness, access pol- policy-violation icy applied rate
Manual
v1
v2
25 33 33 25 33 33 25 33 17 17
3 10 10 20 10 16 10 24 3 3
2 n/a n/a 2 4 n/a 1.5 8 1 2
27.4 25.0
10.6 10.4
– 2.9
Failure example Required doc missing; restricted source accessed Unsafe plan; non-termination Correct source not retrieved; stale memory Wrong entity; action outside permission
Sub-tasks, dependen- Plan validity cies, stopping rule (judge); step count Retrieval / Queries, documents, Retrieval precision; memory ranks, evidence used; evidence coverage; memory reads/writes poisoning tests Tool use Tool, arguments, au- Tool-call precision; thorisation, result, re- recovery rate; tries unauthorised-call rate Synthesis Claim–evidence link- HR; CP; KEC Unsupported age claim carried forward Human over- Checkpoint, interven- Unplanned in- Reviewer sight tion reason, action, tervention rate; misses defect; time escalation appropri- checkpoint ateness; ρcatch bypassed Final outcome Acceptance criteria; Task success; Mandatory eleobjective state check passˆk; cost per ment missing; successful task state not as claimed
EnterpriseVal – Supplementary Appendix
C
G OVERNANCE M APPING
E
VALUE M ODEL D ECOMPOSITION
Table A6. Mapping of EnterpriseVal artefacts to governance requirements. Artefact
SR 11-7 SS1/23
/ NIST AI RMF EU AI Act ISO/IEC + GenAI Profile (high-risk) 42001
Charter (Eq. 1), Model defini- Map: context, Art. 9 a, c, thresholds tion, intended tolerances mgmt use
risk Cl. 6 planning; impact assessment Gold set, sam- Development Measure: test- Art. 10 data Annex A pling frame data documen- data representa- governance data contation tiveness trols Grading records, Independent Measure: met- Art. 15 accu- Cl. 9 perforκ, judge calibra- validation ric validity, hu- racy, robust- mance evaltion evidence man evaluation ness uation Gate decision, Validation out- Manage: risk Art. 14 human Cl. 9.3 manscorecard come, condi- treatment oversight agement retions of use view Value-and-risk Materiality as- Govern: – Cl. 6.1 obmodel sessment benefit–risk jectives Re-evaluation Ongoing Manage: post- Art. 72 Cl. 10 imtriggers, monitor- monitoring, deployment post-market; provement ing change con- monitoring Art. 12 logtrol ging
D
T HRESHOLD -S ETTING P ROTOCOL
Thresholds are set in a facilitated session before any results are unblinded, with the business owner, model owner, second-line risk function and evaluation lead present. 1. Enumerate harms. For each candidate metric, describe the concrete harm of failure. Metrics with no material harm are informational and not gated. 2. Anchor θout to harm and compensating controls. At what level does residual risk, given current review design, become unacceptable? 3. Anchor θin to scale. At what level would the risk function accept lighter compensating controls? The gap should be at least the expected sampling uncertainty at the planned n (Eq. 7). 4. Fix weights and νmax , with rationale and at least one alternative weighting for sensitivity reporting. 5. Sign and version the charter. Any post-unblinding change is a recorded deviation.
Figure A3. Schematic decomposition of Eq. 11 (main paper). Bar heights are illustrative, not pilot data.
F
S CORECARD T EMPLATE
Table A7. One scorecard per use case per cycle; every field is mandatory. Use case ID; business owner; model owner; second-line reviewer Business outcome and value hypothesis Task specification T , G, R (versions) Autonomy level a, consequence tier c, intensity E(a, c) Configuration σ = ⟨M, P, Φ, U, Γ, H⟩ with versions Baseline workflow and measurement mode (estimated / observed) Gated metrics: estimate, CI, θ out , θ in , νj , grading mode, κ/α Reliability: K, passˆ1/2/4, dispersion Assurance and stress-set results Oversight: ρcatch , FAR, intervention rate, review burden Sample sizes (units, documents, tasks); under-sampled flags Gate decision and binding metric b EI and EIG with weight-sensitivity band Value-model inputs (N , ch , ℓerr , ρcatch ) and Vannual interval Observed failure modes; conditions of use; compensating controls Re-evaluation triggers and next review date