Benchmarks Are Not Validation: A System-Level View of Financial LLM Applications Burak Payzun , İrem Demirtaş , Simona Scala , İrem Demirtaş and Elena Ferretti Prometeia S.p.A. {burak.payzun, irem.demirtas, simona.scala, elena.ferretti, secil.arslan}@prometeia.com
arXiv:2607.28840v1 [cs.CL] 30 Jul 2026
Abstract Large language models are increasingly deployed in financial applications that combine retrieval, proprietary data, tool use, orchestration logic, monitoring, and human escalation. Yet evaluation often remains model-centric: benchmark scores, task accuracy, or one-off qualitative reviews are treated as evidence of readiness. In financial settings, this is insufficient. We take the position that financial LLM systems should not be approved for production based on benchmark performance alone. They require system-level validation evidence across the application stack: data, model design, retrieval and generation performance, agent behavior, governance, and implementation. Drawing on industry experience validating GenAI applications in financial institutions, we outline a multi-layer validation view and explain why hybrid evaluation is necessary. We discuss where LLM-as-a-judge methods are useful and why they require controls such as multiple judges, rubrics, agreement, and auditability checks. We also highlight failure modes poorly captured by static benchmarks, including retrieval failures, unfaithful generation, tool misuse, escalation errors, and operational instability. Our position is that financial LLM validation should be an ongoing system discipline rather than a one-time model scoring exercise. Validation should produce decision-ready evidence, not only scores. We conclude with a research agenda for system-aware benchmarks, agent trace validation, judge alignment protocols, and lifecycle validation standards.
1
Introduction
Financial LLM evaluation has improved quickly. Benchmarks such as FinBen cover financial information extraction, textual analysis, question answering, generation, risk management, forecasting, and decision-making [Xie et al., 2024]. Earlier datasets such as FinQA and ConvFinQA show that financial question answering often requires numerical and multi-step conversational reasoning over financial documents [Chen et al., 2021; Chen et al., 2022]. These benchmarks
make model comparison more systematic and expose weaknesses missed by generic evaluations, but they do not solve the validation problem faced by financial institutions. Financial institutions are moving from experimentation to real workflows. LLM systems now summarize documents, answer customer or analyst questions, extract information for credit and lending processes, support compliance reviews, and assist internal decisions. Many are no longer simple prompt-response systems; they combine retrieval-augmented generation, proprietary knowledge bases, prompt orchestration, external tools, APIs, user interfaces, monitoring layers, and sometimes multi-agent workflows. This shift changes what validation must mean. A benchmark score says little about whether a deployed system retrieves the right documents, preserves factual grounding, respects guardrails, handles sensitive data, escalates uncertain cases, resists prompt injection, calls tools safely, or remains stable after third-party model or API changes. In finance, these are not secondary concerns: they determine whether a system is reliable, auditable, and fit for purpose. Banking regulation further raises the validation bar. Validation and audit functions are increasingly expected to assess not only traditional models but also LLM-based applications and AI-driven workflows. Regulatory frameworks such as Model Risk Management (MRM) guidelines and the EU AI Act reinforce the need for governance, traceability, explainability, and continuous oversight across the lifecycle. Existing evaluation methods cover only part of this problem. Classical NLP metrics are useful for constrained tasks, but often depend on reference answers and surface similarity. Human evaluation is valuable but costly to scale. LLM-as-ajudge methods are flexible and fast, and can approximate human preferences in some open-ended settings [Zheng et al., 2023; Liu et al., 2023]; however, they introduce prompt sensitivity, judge bias, reproducibility issues, and overconfidence. No single method is sufficient. Institutions therefore need system-specific test sets, scenarios, and acceptance criteria reflecting their documents, workflows, users, risks, and regulatory constraints. This is difficult in practice: historical data may be limited, acceptable behavior may be unclear, annotation is costly, and synthetic cases can introduce bias. The gap between benchmark performance and evidence of system readiness remains especially large for new systems with few production traces or known failures.
Data Validation
Model Performance
IT Architecture & Implementation
Model Design
Model Use & Governance
Quantitative assessment
Qualitative assessment
metrics, thresholds, retrieval/generation scores, judge agreement, latency and load-test results
expert review, documentation review, governance and risk judgment
Validation evidence package: quantitative results, qualitative findings, failure modes, remediation actions, and lifecycle controls supporting approval, conditional approval, or rejection. Figure 1: Five-pillar view of system-level validation for financial LLM applications. Data, performance, and IT architecture and implementation are primarily quantitative assessment pillars; model design and model use and governance are primarily qualitative assessment pillars. The pillars are independent assessment dimensions rather than a pipeline.
We take the position that financial LLM systems should not be approved for production based on benchmark performance alone. They require system-level validation evidence across the full application stack: data, model design, retrieval and generation behavior, agentic decision logic, governance processes, and IT implementation, with human oversight calibrated to use-case risk. Validation requirements are also usecase dependent. A low-risk internal assistant, a RAG-based knowledge search system, a customer-facing chatbot, and a system supporting creditworthiness assessment have different risk profiles and validation expectations. We organize this validation view around five independent pillars: data, model design, performance, model use and governance, and IT architecture and implementation. These pillars are not a pipeline or hierarchy; they are complementary assessment dimensions. Validation evidence should also distinguish quantitative assessment, mainly supporting data, performance, and IT implementation, from qualitative assessment, mainly supporting model design and governance. Both forms of evidence feed a structured validation evidence package that supports approval, conditional approval, or rejection. Figure 1 summarizes this view. This paper makes three contributions. First, it clarifies why benchmark-centric evaluation is insufficient for deployed financial LLM systems. Second, it proposes a system-level validation view covering data, model design, performance, agent behavior, governance, and implementation. Third, it identifies research directions for financial LLM validation, including trace-level agent evaluation, auditable LLM-as-ajudge protocols, and lifecycle validation standards.
2
Related Work
Financial benchmark suites evaluate LLMs across broad task families such as information extraction, question answering, forecasting, risk management, and decision-making [Xie et al., 2024], but benchmark coverage is not deployment validation. Surveys of financial foundation models catalog open challenges, compliance, hallucination, non-stationarity, and deployment cost—across language, time-series, and visuallanguage models [Chen et al., 2025]. Benchmarking DeepSeek-R1 on financial QA shows strong accuracy but leaves a deployment gap from small single-choice datasets,
persistent hallucination under regulation, and missing multimodality [Liu et al., 2025a]. Both frame these as model-level concerns; we treat them as system-level validation requirements. RAG evaluation moves beyond final-answer scoring by separating retrieval quality, context relevance, answer relevance, and faithfulness; RAGAS, for example, provides reference-free metrics for these modular pipelines [Es et al., 2024]. This matters in finance, where applications built on internal document collections make grounding failures a material risk. LLM-as-a-judge methods can approximate human preferences in open-ended settings [Zheng et al., 2023] and improve correlation with human judgments on selected NLG tasks [Liu et al., 2023], but they exhibit position, verbosity, authority, and self-preference biases (Table 1) [Zheng et al., 2023; Chen et al., 2024]. Panels or juries of judges reduce single-model dependence [Verga et al., 2024], yet still require auditability and agreement checks. Agent evaluation is another emerging line of work. AgentBench evaluates LLMs as agents in interactive environments and identifies long-term reasoning, decision-making, and instruction following as central obstacles [Liu et al., 2025b]. StableToolBench focuses on the difficulty of stable tool-use evaluation when LLMs interact with external tools and APIs [Guo et al., 2025]. AgentDiagnose argues that final-task success leaves agent decision processes opaque and proposes trajectory-level diagnosis [Ou et al., 2025a]. This matters for finance because errors in tool selection, parameter passing, permissions, or escalation can be more important than the fluency of the final response. A useful distinction can be made between white-box, black-box, and grey-box evaluation approaches, as also shown in Table 2: • White-box evaluation: uses traces, metadata, or internal outputs to assess intermediate behavior such as tool selection, parameter correctness, evidence use, reasoning trajectories, or task decomposition. This is reflected in diagnostic datasets and benchmarks that inspect process rather than only final answers [Mialon et al., 2023a; Wang et al., 2022; Wolfson et al., 2020]. • Black-box evaluation: evaluates only the user input and final output. Typical checks include outcome correct-
Pitfall / Bias Position bias
Description Order affects ratings.
Typical Failure Mode First response favored.
Verbosity bias
Length is mistaken for quality.
Authority bias
Confident references scores. Politeness affects scores.
Verbose weak answers outrank concise accurate ones. Hallucinated citations are rewarded. Friendly responses receive inflated ratings. Provider favoritism distorts comparisons. Consensus forms around wrong answers. Same response receives inconsistent ratings. “Helpful” or “good” is interpreted inconsistently. Persuasive reasoning is rewarded when wrong. Well-written hallucinations score highly. One judge’s “4” equals another’s “2”. Repeated evaluations disagree.
Sentiment / compassion bias Self-enhancement bias
inflate
Agreement illusion
Judges prefer same-family outputs. Judges share blind spots.
Prompt sensitivity
Prompt wording changes scores.
Rubric ambiguity
Vague criteria enable subjective scoring. Reasoning presentation affects scores. Fluency is mistaken for correctness. Scales differ across judges.
Chain-of-thought bias Style-over-substance effect Calibration instability Low reproducibility
Averaging away failures
Scores vary across runs or versions. Generic judges fail in specialized domains. Judges fabricate rationales. Systems optimize to evaluator preferences. Aggregates hide edge cases.
Weak adversarial robustness Weak human alignment
Judges fail under adversarial inputs. Human correlation is incomplete.
Metric monoculture
LLM judges become the only metric.
Distribution shift Evaluator hallucination Overfitting to the judge
Finance or legal nuance is missed. Arbitrary scores appear justified. Models learn to game judges. Critical failures disappear in averages. Unsafe outputs evade detection. Judges disagree with expert reviewers. Qualitative scores replace validation.
Recommended Mitigation Randomize order; evaluate both directions. Include conciseness criteria; normalize length. Separate factual verification from style. Use domain rubrics focused on substance. Use heterogeneous evaluator panels. Measure disagreement and diversify judges. Version-control prompts; test prompt robustness. Use operationalized rubrics with examples. Blind judges to hidden reasoning when appropriate. Combine judge scores with factuality checks. Calibrate against human-rated anchors. Fix evaluator versions and decoding parameters. Use domain evaluators and SME review. Treat rationales as evidence, not truth. Rotate judges and add human spot checks. Report worst cases and category results. Red-team the evaluator. Benchmark against expert annotators. Combine LLM judges with classical, retrieval, and rule-based checks.
Table 1: Common pitfalls and shortcomings of LLM-as-a-Judge evaluation in practice. Although LLM judges provide scalable qualitative evaluation, they remain vulnerable to systematic biases, instability, and evaluator-specific failure modes.
ness, robustness to prompt transformations, repeatedrun consistency, safety, and refusal behavior. • Grey-box evaluation: combines outcome evaluation with partial internal information, such as retrieved passages, confidence signals, tool-call summaries, or escalation logs. It is useful when assessing whether the system recognized missing, conflicting, or unsafe conditions. This distinction is important in finance because a correctlooking final answer can conceal unsafe intermediate behavior, policy violations, or incorrect tool usage. The gap is therefore clear: existing work provides useful components for evaluation, but financial institutions need an integrated validation view that connects benchmark performance, RAG evaluation, judge reliability, agent traces, governance, security, and production implementation.
3
The Limits of Benchmark-Centric Evaluation
Benchmarks are useful for comparability, reporting, and model selection, but financial LLM systems fail in ways that benchmark scores do not capture. First, benchmarks usually evaluate models or tasks in isolation. Deployed financial applications also include ingestion, chunking, embedding, retrieval, prompt construction, generation, post-processing, logging, feedback collection, and escalation. A failure in any component can produce an incorrect or unsafe output even when the underlying model is strong. Second, financial tasks are context-specific. Public benchmarks may not reflect an institution’s documents, products, regulatory environment, language mix, risk appetite, or operational constraints. A model that performs well on general financial question answering may still fail on internal policies,
Validation Level Black-box outcome
Evidence Used User input and final output
Grey-box behavior
Output plus retrieved context, confidence, tool summaries, logs Full trajectory, tool calls, parameters, observations, handoffs
White-box trace
Ablation / replay
Controlled model, retrieval, prompt, and tool variants
Typical Checks Task success, correctness, refusal, escalation, tone, safety Grounding, evidence use, uncertainty handling, escalation path Tool choice, parameter validity, permissions, retries, loop detection Base-vs-system attribution, regression and migration testing
Failure Diagnosed Visible user-facing failure Misuse of partial system evidence Unsafe or inefficient intermediate behavior Root cause across components
Table 2: Agent validation should combine black-box, grey-box, white-box, and ablation-based evidence. Final-answer correctness is necessary but insufficient when financial LLM systems can retrieve, route, call tools, or execute workflows.
local banking terminology, or specific reporting templates. Third, even finance-specific benchmarks remain bounded by task format, source material, annotation strategy, answer type, and evaluation protocol. FinBen covers broad financial tasks, and FinanceBench provides open-book question answering over company filings [Xie et al., 2024; Islam et al., 2023]; however, both still convert financial work into fixed test items, reference answers, and simplified acceptance conditions. Deployment is broader: systems must operate over proprietary taxonomies, changing product definitions, multilingual documents, ambiguous requests, incomplete evidence, and downstream business processes. Benchmarks should therefore be treated as sampling instruments, not complete validation environments. They can show performance on known task families, but not whether a specific institution’s application has been validated against its operational universe, control environment, and risk appetite. Fourth, agentic systems introduce sequential failure modes. Once an LLM can call tools, route requests, invoke APIs, decide whether to escalate, or coordinate with other agents, validation must cover traces and decision sequences. Prompt-level accuracy does not guarantee safe system behavior. Fifth, rapid model replacement can invalidate prior validation. A newer model may improve benchmark accuracy while changing refusal behavior, citation style, tool-use reliability, latency, cost, calibration, or prompt sensitivity. In finance, migration to a new foundation model, embedding model, retriever, prompt template, tool schema, guardrail, or orchestration layer should be treated as a controlled change and trigger targeted regression testing against accepted cases, known failures, and high-risk scenarios. RAG systems also require component-level evaluation. Retrieval quality, context relevance, and generation behavior should be assessed separately, because upstream errors propagate into final outputs. For these reasons, benchmark-centric evaluation should be an input to validation, not validation itself.
full application lifecycle and produce a structured evidence package rather than a single score.
4
Performance validation should begin with the validation objective, not a catalogue of metrics. Institutions should first define what correct, acceptable, incomplete, unsafe, and escalation-worthy behavior means for each use case, considering the business task, users, error consequences, groundtruth availability, explainability needs, and permitted auton-
A System-Level View of Financial LLM Validation
The following subsections expand the five pillars introduced in Figure 1. The exact implementation may vary by institution and use case, but the validation logic should cover the
4.1
Data Validation
Data validation assesses the quality, relevance, safety, and structure of training data where applicable, test data, evaluation datasets, and indexed knowledge bases used in RAG systems. It should cover corpus-level statistics, language scope, duplication, cleaning, toxicity or unsafe-content screening, privacy checks, and representativeness for the intended use case. In RAG systems, the indexed corpus is part of the model’s effective knowledge environment; poor data health can cause poor retrieval, hallucinated answers, or unsafe outputs even when the generator is capable. Data augmentation can support validation through perturbations, paraphrases, entity replacements, and synthetic edge cases that test robustness, fairness, and wording sensitivity. It must also be controlled: generated or transformed data can introduce artifacts, distort meaning, or create misleading evaluation results if not documented and checked.
4.2
Model Design Validation
Model design validation asks whether the architecture and development strategy are fit for the task. It covers the foundation model, embedding model, retrieval design, prompt strategy, fine-tuning approach, model-size reduction, and human feedback or escalation mechanisms. The key question is not whether the system uses the most advanced model, but whether the design is fit for purpose. A smaller model may suit a constrained classification task; RAG may be appropriate when answers must be grounded in internal documents; fine-tuning may be justified for domain adaptation; and prompt engineering may be sufficient for simpler workflows. Validation should also check whether assumptions on task scope, user behavior, source data, escalation rules, model limitations, regulatory constraints, and third-party dependencies are documented.
4.3
Model Performance Validation
omy. Only then should evaluators select quantitative, qualitative, human, automated, or hybrid evidence. A comprehensive evaluation should cover factual correctness, grounding, completeness, consistency, uncertainty handling, refusal behavior, robustness to paraphrase or noisy input, prompt sensitivity, privacy and confidentiality, fairness and bias where relevant, operational efficiency, and downstream business impact. This is consistent with holistic evaluation, which calls for assessment across multiple scenarios and desiderata rather than isolated accuracy measures [Liang et al., 2023], and with surveys emphasizing reliable, reproducible, and robust evaluation before deployment [Laskar et al., 2024]. Validation evidence should combine complementary methods. Structured tasks may use deterministic rules; RAG systems require evidence-level checks separating retrieval from answer quality; open-ended generation may require expert review, rubrics, or controlled LLM-as-a-judge evaluation. High-risk use cases should include adversarial, edge-case, and regression tests, while production systems require monitoring because offline results may not hold under changing users, documents, prompts, providers, or market conditions. Performance should not be reported only through aggregate scores, since averages can hide severe rare failures. Reports should include category-level results, worst-case failures, error taxonomies, confidence or escalation analysis, and remediation evidence. In operational settings, results are often summarized as acceptable, requires remediation, or not acceptable. The central question is whether the system provides sufficient evidence of fitness for the intended financial use case.
4.4
LLM-as-a-Judge: Necessary but Not Sufficient
LLM-as-a-judge methods are increasingly used because they scale qualitative assessment where classical metrics are insufficient. This is relevant for financial LLMs, whose outputs are often open-ended, context-dependent, and difficult to compare with a single reference answer. A response may be correct but incomplete, fluent but insufficiently grounded, concise but missing risk information, or helpful but inappropriate under policy constraints. LLM judges can therefore provide rubric-based signals for correctness, completeness, relevance, conciseness, language quality, tone, bias, and safety. MT-Bench, Chatbot Arena, and G-Eval show that strong LLM evaluators can align with human preferences or judgments in some settings [Zheng et al., 2023; Liu et al., 2023]. In financial validation, LLM-as-a-judge should be a structured evaluator, not a source of truth. It can assess answer quality, RAG faithfulness, instruction and communication alignment, and safety or policy issues such as privacy violations, harmfulness, toxicity, defamation, inappropriate specialized advice, intellectual property concerns, unsafe tool use, and misuse. These dimensions are especially relevant in lending, compliance, AML, fraud, credit risk, and customerfacing workflows. Judge scores should be diagnostic evidence rather than aggregate benchmark results. Their value is granularity: a system may perform well on tone and fluency while failing
on factual grounding, privacy handling, escalation, or policy compliance. In practice, judge-based evaluation often uses structured rubrics, such as 1–5 scales, with explicit thresholds for dimensions such as correctness, safety, and bias, calibrated against human-reviewed examples. LLM judges should not be treated as oracles. They can be sensitive to prompt wording, rubrics, response order, verbosity, confidence, sentiment, authority signals, and model family. Prior work documents position, verbosity, and selfenhancement biases [Zheng et al., 2023], while later studies show that both human and LLM judges are vulnerable to judgment biases and perturbations [Chen et al., 2024]. Judges may reward plausible but wrong explanations or agree for the wrong reasons when they share training distributions or model-family biases. A responsible setup should include explicit rubrics, structured outputs, documented prompts, fixed evaluator versions, controlled decoding parameters, multiple and diverse judges where possible, and agreement analysis. Evaluator juries are consistent with recent work on panels of diverse models [Verga et al., 2024]. Ordinal scores may require weighted agreement, ranked outputs may require rank correlation, and high disagreement should trigger expert review rather than be averaged away. For high-risk financial use cases, judge outputs should be calibrated against human-rated anchors and periodically checked by subject-matter experts. The goal is to use LLM judges as scalable evaluators within a controlled validation process. In finance, they should be auditable, reproducible, subordinate to the validation objective, and complementary to deterministic checks, retrieval and faithfulness metrics, adversarial testing, expert review, and lifecycle monitoring.
4.5
Agent and Tool Validation
Agentic financial LLM systems require a separate validation layer because they do not only generate text. They may classify intent, retrieve evidence, select tools, call APIs, route tasks, escalate, or execute multi-step workflows. Validation should therefore begin by classifying the type and degree of agency: a low-agency RAG assistant, a tool-calling assistant, a workflow agent that updates records, and a multi-agent system have different risk profiles. Black-box evaluation is necessary but insufficient. A final answer may look correct even if the system used the wrong source, called an unnecessary tool, passed unsafe parameters, ignored permissions, or skipped escalation. Conversely, an incorrect answer may originate from the base model, retriever, tool schema, API failure, orchestration logic, memory, or guardrails. Agent validation should include trace- and componentlevel evidence. At minimum, it should examine task identification, decomposition, retrieval, tool choice, parameter validity, tool-error handling, permission boundaries, side effects, escalation of uncertain or high-risk cases, and faithfulness of the final answer. This aligns with agent benchmarks and diagnostic work on real-world assistant tasks, interactive environments, stable tool use, and trajectory-level diagnosis [Mialon et al., 2023b; Liu et al., 2024; Guo et al., 2024; Ou et al., 2025b].
A practical design should combine black-box, grey-box, and white-box checks. Black-box tests assess the final outcome. Grey-box tests use partial internal information such as retrieved passages, confidence scores, tool-call summaries, or escalation logs. White-box tests inspect full traces, including observations, tool calls, parameters, retries, failures, and handoffs. The more authority the agent has to affect real systems, the more important grey-box and white-box validation become. Ablation and replay tests help separate base-model performance from system performance. Tasks can be rerun with retrieval disabled, fixed context, mocked tool outputs, alternative model versions, or golden tool-call sequences. This attribution matters because remediation differs: model weaknesses may require prompt redesign or model replacement, while tool-use failures may require schema constraints, permissions, fallback logic, or human approval. Agent-level acceptance tests should cover normal workflow completion, ambiguous requests, missing or conflicting evidence, permission boundaries, unsafe-action prevention, escalation, repeated-run stability, loop detection, cost and latency limits, tool-failure recovery, and prompt-injection scenarios. High-impact workflows should also use canary releases, shadow mode, and human-in-the-loop review before full deployment.
4.6
Model Use, Governance, and Lifecycle Validation
Regulatory Alignment Regulatory compliance should not be treated as an external checklist applied after technical evaluation. For financial LLM systems, regulation is part of the validation objective because the relevant object of assessment is the full application stack: data, model design, retrieval and generation behavior, agent logic, governance, and implementation controls. The EU Artificial Intelligence Act provides a key regulatory baseline for LLM applications deployed in the European financial sector. Its risk-based approach requires applicationspecific classification by intended purpose, value-chain role, autonomy, affected users, and potential impact. A generic internal assistant, a policy-search RAG system, an analystsupport tool, and a system contributing to creditworthiness assessment may therefore face different obligations even if they use the same foundation model. Commission guidelines on AI-system definition, prohibited practices, and GPAI obligations further clarify the scope of the Act, while the GPAI Code of Practice supports compliance for general-purpose AI model providers. At the same time, the Digital Omnibus package shows that AI Act implementation remains a moving regulatory target, with the May 2026 provisional agreement introducing simplification measures and delayed application dates for certain high-risk obligations. For validation, these requirements translate into practical evidence rather than legal formality. The process should verify use-case classification, prohibited-use screening, provider and deployer responsibilities, documentation, auditability, traceability, logging, human oversight, escalation rules, bias and privacy controls, incident management, and revalidation triggers after material changes. Regulatory aspects
should therefore be assessed within broader controls on data, methodology, process, governance, privacy, bias, human-inthe-loop design, monitoring, and model use. These regulatory requirements should be interpreted as validation evidence requirements rather than abstract principles. Lifecycle Validation Financial LLM systems require governance after deployment. Validation should cover versioning of models, prompts, data, retrieval indexes, rules, traces, and evaluation tests, as well as monitoring, feedback collection, change management, human-in-the-loop processes, and periodic revalidation. Deployment controls such as A/B testing, shadowing, dark launches, and canary releases allow institutions to compare versions, detect degradation, and limit failure impact. Feedback mechanisms should capture both user satisfaction and substantive issues such as incorrect answers, missing evidence, unsafe content, or failed escalation. Lifecycle validation is necessary because LLM systems are unstable in ways traditional software often is not. Foundation models may be updated, deprecated, or replaced; user behavior, retrieval corpora, prompts, and regulatory expectations may change. Validation must therefore be continuous.
4.7
IT Architecture and Implementation Validation
Implementation readiness is integral to GenAI validation. A financial LLM system is not fit for purpose solely because it produces accurate answers in controlled tests; it must also be reliable, scalable, observable, secure, and maintainable in production. Validation should assess whether the architecture supports intended use under normal, peak, and exceptional conditions. This includes response-time analysis, stress and load testing, concurrency and burst testing, failure testing, dependency review, rate-limit assessment, fallback mechanisms, and thirdparty availability. For RAG and agentic systems, it should also cover retrieval and tool-call latency, number of model or tool calls per task, caching, retry logic, timeout handling, and degraded-service modes. Operational weaknesses may create model risk even when model-level performance is acceptable. Systems can fail if peak loads are not handled, latency exceeds requirements, API rate limits are reached, third-party models become unavailable, or deprecated components are not replaced through controlled change. The technology stack should therefore be reviewed for scalability, security vulnerabilities, dependency management, version compatibility, and end-of-life risks. Updates to models, APIs, libraries, vector databases, orchestration tools, or cloud components should trigger regression testing on output quality, latency, cost, and risk controls. Logs are a core validation artifact. They support auditability, incident investigation, monitoring, regulatory review, and improvement. Logging should capture prompts or prompt identifiers, retrieved context references, model and data versions, component-level response times, errors, fallback activation, tool calls, user feedback, and evaluation results, while
avoiding unnecessary privacy, confidentiality, and security risks.
5
Failure Modes That Require System-Level Validation
As supported by our empirical experience, a system-level approach is necessary because many failures arise between components rather than inside a single model. A RAG system may retrieve irrelevant but semantically similar documents, causing a confident but unsupported answer. It may retrieve the right documents but omit key conditions, or perform well on normal inputs but fail under paraphrased, noisy, multilingual, or adversarial inputs. RAG evaluation is useful because it distinguishes retrieval, context, answer relevance, and faithfulness instead of collapsing them into one final score [Es et al., 2024]. Security failures also cut across layers. Prompt injection can be introduced directly by a user or indirectly through retrieved documents. Data poisoning can affect retrieval and generation. Excessive agency can turn a model error into an unauthorized action. Poor output handling can pass unsafe content or malformed structured output to downstream systems. These risks show why guardrails are part of system-level validation, not optional add-ons. Prompt injection, retrieval poisoning, unsafe tool calls, policy evasion, and escalation errors can propagate across retrieval, generation, orchestration, and implementation. In agentic settings, they may also affect tool selection, parameter passing, permission boundaries, and downstream actions. Guardrail architectures should therefore combine preventive controls, runtime monitoring, and postgeneration validation. Validation should assess not only whether guardrails exist, but whether they remain effective under realistic and adversarial conditions. Relevant controls include prompt filtering, retrieval sanitization, output validation, permission boundaries, policy enforcement, escalation rules, human approval workflows, logging, and fallback mechanisms. These failures are hard to detect through static benchmarks because they emerge from interactions among data, retrieval, generation, agent traces, controls, monitoring, and implementation.
6
Research Agenda
System-level validation for financial LLMs remains underdeveloped. We identify several directions for the research community. First, financial LLM benchmarks should move beyond static question-answer pairs. Future benchmarks should include retrieval corpora, evidence requirements, tool-use traces, escalation scenarios, adversarial documents, and operational constraints. Second, agent validation needs better trace-level methods. Finance requires domain-specific criteria for tool selection, parameter correctness, permission compliance, escalation, and failure recovery. Third, LLM-as-a-judge protocols need standardization through reusable rubrics, judge alignment metrics, disagreement handling rules, and auditable reporting standards.
Fourth, financial-domain red-teaming should become a validation discipline covering prompt injection, retrieval poisoning, privacy leakage, misinformation, excessive agency, and unbounded consumption in realistic workflows. Fifth, lifecycle validation should be integrated into financial LLM evaluation. Model updates, prompt changes, data drift, index refreshes, and third-party dependency changes should trigger structured revalidation. Sixth, regulation-aware validation should clarify how evolving requirements, especially under the EU AI Act, can be operationalized in technical frameworks. This requires linking use-case risk classification, governance obligations, auditability, human oversight, and revalidation triggers to observable system behavior. Finally, validation research should address the gap between technical metrics and institutional decision-making. Financial institutions need outputs that support approval decisions, risk classification, remediation planning, and monitoring.
7
Conclusion
Financial LLM systems should not be validated by benchmarks alone. This becomes critical as banks deploy RAGbased, tool-using, and agentic applications in workflows such as lending, compliance, anti-money laundering, fraud monitoring, and credit risk assessment. The relevant question is not only whether a foundation model performs well on a public benchmark, but whether the deployed system is grounded, reliable, auditable, secure, and fit for its intended financial purpose. The regulatory nature of banking makes this shift unavoidable. Financial institutions operate under model risk management expectations, internal controls, and emerging AI regulations that require structured and evidence-based assessment. As LLM applications enter decision-support processes, validation and audit teams need protocols combining quantitative metrics, qualitative expert review, trace-level analysis, governance checks, and lifecycle monitoring. Evaluation should therefore move from isolated model scoring toward repeatable validation evidence that supports approval, risk classification, remediation, and oversight. This paper takes the position that financial LLM validation is a system-level discipline. The object of validation is the full application stack: data, model design, retrieval and generation behavior, agent and tool use, guardrails, governance, and IT implementation. Hybrid evaluation is necessary because classical metrics, human review, and LLM-as-a-judge each provide useful but incomplete evidence. In finance, LLM-asa-judge should be controlled, auditable, and subordinate to the validation objective. Moving beyond benchmarks is not a rejection of benchmarks. They remain useful for model comparison and initial screening. Responsible deployment in finance, however, requires evidence about the whole system under realistic, adversarial, changing, and regulated conditions. As financial institutions adopt increasingly agentic LLM systems, validation should evolve into an ongoing discipline connecting technical performance, operational resilience, governance, auditability, and regulatory fitness across the lifecycle.
References [Chen et al., 2021] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [Chen et al., 2022] Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279–6292, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. [Chen et al., 2024] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA, November 2024. Association for Computational Linguistics. [Chen et al., 2025] Liyuan Chen, Shuoling Liu, Jiangpeng Yan, Xiaoyu Wang, Henglin Liu, Chuang Li, Kecheng Jiao, Jixuan Ying, Yang Veronica Liu, Qiang Yang, and Xiu Li. Advancing financial engineering with foundation models: Progress, applications, and challenges. arXiv preprint arXiv:2507.18577, 2025. [Es et al., 2024] Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta, March 2024. Association for Computational Linguistics. [Guo et al., 2024] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable largescale benchmarking on tool learning of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11143–11156, 2024. [Guo et al., 2025] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable largescale benchmarking on tool learning of large language models, 2025. [Islam et al., 2023] Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023. [Laskar et al., 2024] Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad
Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Xiangji Huang. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13785–13816, 2024. [Liang et al., 2023] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. [Liu et al., 2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, December 2023. Association for Computational Linguistics. [Liu et al., 2024] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2024. [Liu et al., 2025a] Shuoling Liu, Liyuan Chen, Jiangpeng Yan, Yuhang Jiang, Xiaoyu Wang, Xiu Li, and Qiang Yang. When DeepSeek-R1 meets financial applications: benchmarking, opportunities, and limitations. Frontiers of Information Technology & Electronic Engineering, 26(10):1862–1870, 2025. [Liu et al., 2025b] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2025. [Mialon et al., 2023a] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023. [Mialon et al., 2023b] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: A benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023. [Ou et al., 2025a] Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. AgentDiagnose: An open toolkit for diagnosing LLM agent trajectories. In Ivan Habernal, Peter Schulam, and Jörg Tiedemann, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 207–215, Suzhou, China, November 2025. Association for Computational Linguistics.
[Ou et al., 2025b] Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. AgentDiagnose: An open toolkit for diagnosing LLM agent trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 207–215, Suzhou, China, November 2025. Association for Computational Linguistics. [Verga et al., 2024] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024. [Wang et al., 2022] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022. [Wolfson et al., 2020] Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 2020. [Xie et al., 2024] Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, and Jimin Huang. FinBen: A holistic financial benchmark for large language models. In Advances in Neural Information Processing Systems, volume 37, 2024. [Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-ajudge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, 2023.